Kudan

Search results for "{{ search.query }}"

No results found for "{{search.query}}". 
View All Results

Capturing Video

This tutorial outlines the steps necessary for setting up the KudanAR engine to capture a video of 3D content and the camera stream on iOS.

Advanced Tutorial

This tutorial assumes you've already read through the Getting Started and Marker Basics tutorials. If you have not, it is strongly recommended that you do so before you head any further.

Background Information

This section isn't necessary to the tutorial, but it is useful to know. If you simply want to get coding, skip ahead to Step 1.

KudanAR uses OpenGL to render content and video on iOS. OpenGL draws content to framebuffers, which can be used to draw content directly to screen or for more complex operations, such as rendering content offscreen to be used as textures within 3D scenes.

In order to render a scene to a video file, image data for each frame needs to be fetched after it has been drawn to our main framebuffer, which is responsible for drawing content to device screens, and encoded into a compatible video format. This is traditionally achieved by reading pixel data directly from the main framebuffer, encoding the data and writing it directly to disk. However, this method involves significant amounts of data transfer from the GPU and can significantly detract from performance when recording high resolution videos.

Fortunately, iOS provides a method of recording that removes the need to read data from the GPU by allowing content to be rendered directly into the video encoding pipeline. This is achieved through the use of the CoreVideo library and AssetWriter class, which can accept an OpenGL texture as an input to the video encoding pipeline. Rendering a scene to this offscreen texture allows the direct transfer of data into the video encoding pipeline, removing the pressure on GPU data transfer.

Implementing the ARCaptureRenderTarget Class

To capture KudanAR content to a video, we can make a new class that is a subclass of ARRenderTarget called ARCaptureRenderTarget. We can render the AR scene to this render target and capture the frames to make our video.

We'll start by creating the subclass header file, ARCaptureRenderTarget.h:

#import <KudanAR/KudanAR.h>
 
@interface ARCaptureRenderTarget : ARRenderTarget
 
@property (nonatomic) CVPixelBufferRef pixelBuffer;
@property (nonatomic) GLuint fbo;
  
@property (nonatomic, strong) AVAssetWriter *assetWriter;
@property (nonatomic, strong) AVAssetWriterInputPixelBufferAdaptor *assetWriterPixelBufferInput;

@property (nonatomic, strong) NSDate *startDate;
@property (nonatomic) BOOL isVideoFinishing;
 
@end

Here, we have properties we can use to get references to the main framebuffer, as well as the camera's pixelbuffer. These store the data we're going to record.

We also have properties for AVAssetWriter objects in order to actually write this data to a video file.

Finally, we have two properties that allow us to know when the video has started and if the video should finish recording. These give us more control over when the recording starts and stops.

Next, we need the implementation file, ARCaptureRenderTarget.m:

#import "ARCaptureRenderTarget.h";

@implementation ARCaptureRenderTarget

- (instancetype) initWithWidth:(float)width height:(float)height
{
    self = [super initWithWidth:width height:height];
 
    if (self) {
        // Account for the scaling factor associated with some iOS devices.
        self.width *= [UIScreen mainScreen].scale;
        self.height *= [UIScreen mainScreen].scale;
 
        // Set up the required render target assets.
        [self setupAssetWriter];
        [self setupFBO];
 
        _isVideoFinishing = NO;
    }
    return self;
}

- (void)setupAssetWriter {
 
    // Set up asset writer.
    NSError *outError;
 
    // Write the video file to the application's library directory, with the name "video.mp4".
    NSURL *libsURL = [[[NSFileManager defaultManager] URLsForDirectory:NSLibraryDirectory inDomains:NSUserDomainMask] lastObject];
    NSURL *outputURL = [libsURL URLByAppendingPathComponent:@"video.mp4"];
 
    // Delete a file with the same path if one exists.
    if ([[NSFileManager defaultManager] fileExistsAtPath:[outputURL path]]){
 
        [[NSFileManager defaultManager] removeItemAtURL:outputURL error:nil];
    }
 
    _assetWriter = [AVAssetWriter assetWriterWithURL:outputURL
                                            fileType:AVFileTypeQuickTimeMovie
                                               error:&outError];
 
    if (outError) {
        NSAssert(NO, @"Error creating AVAssetWriter");
    }
 
    // Set up asset writer inputs.
 
    // Set up the asset writer to encode video in the H.264 format, with the height and width equal to that
    // of the framebuffer object.
    NSDictionary *assetWriterInputAttributesDictionary =
    [NSDictionary dictionaryWithObjectsAndKeys:
     AVVideoCodecH264, AVVideoCodecKey,
     [NSNumber numberWithInt:self.width], AVVideoWidthKey,
     [NSNumber numberWithInt:self.height], AVVideoHeightKey,
     nil];
 
    AVAssetWriterInput *assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
                                                                              outputSettings:assetWriterInputAttributesDictionary];
    // Assume the input pixel buffer is in BGRA format, the iOS standard format.
    NSDictionary *sourcePixelBufferAttributesDictionary =
    [NSDictionary dictionaryWithObjectsAndKeys:
     [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
     [NSNumber numberWithInt:self.width], kCVPixelBufferWidthKey,
     [NSNumber numberWithInt:self.height], kCVPixelBufferHeightKey,
     nil];
 
    _assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterInput
                                                                                                    sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];    
    // Add the input to the writer if possible.
    if ([_assetWriter canAddInput:assetWriterInput]) {
        [_assetWriter addInput:assetWriterInput];
    } else {
        NSAssert(NO, @"Error adding asset writer input");
    }
 
    // Start the asset writer immediately for this simple example. 
    [_assetWriter startWriting];
    [_assetWriter startSessionAtSourceTime:kCMTimeZero];
 
    // Store the date when the asset writer started recording video.
    _startDate = [NSDate date];
 
    // Check the asset writer has started.
    if (_assetWriter.status == AVAssetWriterStatusFailed) {
        NSAssert(NO, @"Error starting asset writer %@", _assetWriter.error);
    }
}

- (void)setupFBO {
 
    // Make the renderer context current, necessary to create any new OpenGL objects.
    [[ARRenderer getInstance] useContext];
 
    // Create the FBO.
    glActiveTexture(GL_TEXTURE1);
    glGenFramebuffers(1, &_fbo);
    [self bindBuffer];
 
    // Create the OpenGL texture cache.
    CVOpenGLESTextureCacheRef cvTextureCache;
 
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [EAGLContext currentContext], NULL, &cvTextureCache);
 
    if (err) {
        NSAssert(NO, @"Error creating CVOpenGLESTextureCacheCreate %d", err);
    }
 
    // Create the OpenGL texture we will be rendering to.
    CVPixelBufferPoolRef pixelBufferPool = [_assetWriterPixelBufferInput pixelBufferPool];
 
    err = CVPixelBufferPoolCreatePixelBuffer (kCFAllocatorDefault, pixelBufferPool, &_pixelBuffer);
 
    if (err) {
        NSAssert(NO, @"Error creating CVPixelBufferPoolCreatePixelBuffer %d", err);
    }
 
    CVOpenGLESTextureRef renderTexture;
    CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, cvTextureCache, _pixelBuffer,
                                                  NULL, // texture attributes
                                                  GL_TEXTURE_2D,
                                                  GL_RGBA, // opengl format
                                                  (int)self.width,
                                                  (int)self.height,
                                                  GL_BGRA, // native iOS format
                                                  GL_UNSIGNED_BYTE,
                                                  0,
                                                  &renderTexture);
 
    // Attach the OpenGL texture to the framebuffer.
    glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
 
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
 
    // Create a depth buffer for correct drawing.
    GLuint depthRenderbuffer;
 
    glGenRenderbuffers(1, &depthRenderbuffer);
 
    glBindRenderbuffer(GL_RENDERBUFFER, depthRenderbuffer);
    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8_OES, self.width, self.height);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depthRenderbuffer);
 
    // Check the FBO is complete and ready for rendering
    [self checkFBO];
}

- (void)bindBuffer {
    glBindFramebuffer(GL_FRAMEBUFFER, _fbo);
}

- (void)draw {
 
    // Draw content to the framebuffer as normal.
    [super draw];
 
    // Prevent encoding of a new frame if the AVAssetWriter is not writing or if the video is completed.
    if (self.assetWriter.status != AVAssetWriterStatusWriting || _isVideoFinishing) {
        return;
    }
 
    // Wait for all OpenGL commands to finish.
    glFinish();
 
    // Lock the pixel buffer to allow it to be used for encoding.
    CVPixelBufferLockBaseAddress(_pixelBuffer, 0);
 
    // Submit the pixel buffer for the current frame to the asset writer with the correct timestamp.
    CMTime currentTime = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSinceDate:_startDate],120);
 
    if (![_assetWriterPixelBufferInput appendPixelBuffer:_pixelBuffer withPresentationTime:currentTime]) {
        NSLog(@"Problem appending pixel buffer at time: %lld", currentTime.value);
    }
 
    // Unlock the pixel buffer to free it.
    CVPixelBufferUnlockBaseAddress(_pixelBuffer, 0);
 
    // In this simple example, finish the video if it has been recording for 5 seconds.
    if (CMTimeCompare(currentTime, CMTimeMake(5, 1)) == 1) {
        _isVideoFinishing = YES;
 
        [self.assetWriter finishWritingWithCompletionHandler:^{
            NSLog(@"Finished writing video.");
        }];
    }
}

@end

The above code will record a video of the AR scene for 5 seconds after it has been initialised and outputs the resulting video to a "video.mp4" in the application's library directory. If you wish to use this code as is without modifying it understanding how it all works, you can skip ahead to Step 2.

How it all works

Overall, this class sets up an asset writer and framebuffer object using a given width and height, namely the width and height of the device's screen. We add an instance of it to the renderer alongside the other standard render targets, and the renderer draws to the framebuffer object. The data in the framebuffer is then taken by the asset writer and written to a file we designate. The result is an .mp4 file containing the captured frames in sequence.

The init method takes a width and height and uses these to determine the bounds of the recording area. It automatically accounts for any scaling, such as when an iPad is at 2x zoom in an app. This method also sets up the asset writer and framebuffer object for recording and writing to a file. Finally, it makes sure isVideoFinishing is set to NO, because if it were set to YES, it would only record for a single frame and then stop.

The setupAssetWriter method first gets a full file path to the app's library directory and appends this with the filename "video.mp4". This is the output file that will be written to. If there's already a file by that name in the directory, it will delete it and replace it with our new video, so make sure you've either moved or renamed the video before starting a new recording. You can of course change this behaviour so that it's smarter about how it deals with naming videos, perhaps by adding a timestamp to the file name.
Next, it creates a new asset writer and applies a number of settings, namely the width and height of the frames, and adds an input that has had a number of settings applied to it, notably that the image format is 32_BGRA and that the media type is video. Since our goal is to record an MP4 video, these settings will suit us just fine, but you can change these settings to experiment with other formats if you wish.
After adding the input, it starts writing immediately and notes the exact time it started. This is why the video starts recording as soon as the class initialises. If you wish to have more control over when the video starts recording, you can remove this line of code and create a new method to call.

The setupFBO method creates and sets up a new framebuffer object with OpenGL. Firstly, it makes sure it is using the correct OpenGL context, otherwise it would not be able to create a new framebuffer object.
Next, it creates the framebuffer object and binds it. Why bindBuffer is in its own separate method is explained below.
After this, it creates a core video texture cache, allowing us to directly read and write BGRA data to our framebuffer object.
Then, it creates a new texture, using the pixel buffer as a reference for its data, before binding it and attaching it to our framebuffer object. At this point, it also makes sure that both texture coordinates of the texture are using Clamp-To-Edge wrapping, which allows us to work with "Non Power of Two" textures.
Next, it creates a depth buffer to ensure the framebuffer object's data is drawn correctly. This is because Kudan's rendering enables depth testing, so we must make sure that data is not lost, otherwise our texture cache's data can be out of order, and the resulting video will look strange in places.
Lastly, it checks that the framebuffer object has been set up correctly and is ready to be used. The checkFBO method returns a bool, so it is possible to check it using an if statement and run code if the framebuffer reports that it is not complete.

The bindBuffermethod is actually an override of the same method found in ARRenderTarget. This ensures that we bind the correct framebuffer object when rendering to our capture target, otherwise we will get incorrect data. This is why it is in its own separate method.

The draw method is another override. Usually, it is called in ARRenderTarget when content is rendered to the framebuffer, so here we make sure we call our capture code afterwards to handle the encoding of each frame. Note that the draw method returns if the asset writer is still writing, so if the asset writer is having difficulty writing a frame for any reason, some frames may be dropped. It will also return if the isVideoFinishing boolean is true, so it will not attempt to capture frames unless it is supposed to be recording them, which is good for performance.
Next, it calls glFinish() to make sure OpenGL has finished drawing to the framebuffer before attempting to access it, before appending the asset writer with the current frame, adding it onto the end of the video.
Finally, it checks how long the video has been recording for and, if it is longer than the specified time, in this example 5 seconds, it will stop the recording by setting isVideoFinishing to YES. Here, you can modify the amount of time the video records for before stopping or, if you wish to have more control over when the video stops, remove this code entirely and make a new method that sets the isVideoFinishing bool to NO.

How to use ARCaptureRenderTarget

To record content from the AR scene, it is necessary to add the scene's content to the ARCaptureRenderTarget so that it is drawn every frame. This can be achieved by using the following code, which in this example is added to the setupContent of a view controller that is a subclass of ARCameraViewController:

#import "ViewController.h"

@implementation ViewController
 
- (void)setupContent
{
  // AR Content is set up here.
  
  // Create the ARCaptureRenderTarget object.
  ARCaptureRenderTarget *captureRenderTarget = [[ARCaptureRenderTarget alloc] initWithWidth:self.view.frame.size.width height:self.view.frame.size.height];

  // Add the viewports that need rendering to the the render target.
  // The camera viewport contains the camera image. The content viewport contains the 3D content.
  [captureRenderTarget addViewPort:self.cameraView.cameraViewPort];
  [captureRenderTarget addViewPort:self.cameraView.contentViewPort];

  // Add the offscreen render target to the renderer.
  [[ARRenderer getInstance] addRenderTarget:captureRenderTarget];
}
 
@end

It's that simple. You can of course attach this code to a button or touch input so that it makes a render target at runtime. You can also achieve more complex behaviour by modifying the ARCaptureRenderTarget class' implementation. For example, you could increase the recording duration, or even have it start and stop recording in response to user input, rather than being on a timer.

Capturing Video

This tutorial outlines the steps necessary for setting up the KudanAR engine to capture a video of 3D content and the camera stream on iOS.