--- Android --- Android Javascript API Android Native API --- iOS --- iOS Javascript API iOS Native API --- Windows --- UWP Native API --- Unity --- Unity Hololens Unity --- Extensions --- Flutter Cordova Xamarin --- Smart Glasses --- Epson Vuzix --- Services --- Cloud Recognition

Wikitude SDK iOS Native API

Rendering

This example shows and explain how rendering works in combination with the Wikitude SDK Native API. There are two methods of rendering available in the Wikitude Native SDK. We call them internal and external rendering. Internal means the rendering view (OpenGL ES, Metal) is setup by the Wikitude SDK and the SDK user can define custom rendering, that is executed by the Wikitude SDK. On the other hand external rendering means the SDK user sets up the rendering view on his own and integrates the Wikitude SDK into this rendering setup.

Rendering APIs

The Wikitude Native SDK for iOS currently supports the following RenderingAPIs:

• OpenGL ES 3
• OpenGL ES 2
• Metal

How the Rendering API can be selected can be seen at Rendering API selection for External Rendering or Rendering API selection for Internal Rendering.

External OpenGL ES Rendering

Most of the examples in the Wikitude Native SDK example application use external rendering, simply because it might be the preferred usage by developers. The example application comes with a simple OpenGL ES view (ExternalEAGLView) and an also very simple renderer (ExternalRenderer).

To activate external rendering you will need to use a WTOpenGLESRenderingConfiguration in combination with a WTOpenGLESRenderingMode, in this case a WTExternalOpenGLESRenderingMode to pass WTRenderingMode_External and pass that to the WTWikitudeNativeSDK method -initWithRenderingConfiguration:delegate. The most convenient way to do this is to create a preconfigured WTOpenGLESRenderingConfiguration by calling +createExternalOpenGLESRenderingConfiguration on the WTWikitudeSDK.

WTOpenGLESRenderingConfiguration *selectedConfiguration = [WTWikitudeNativeSDK createExternalOpenGLESRenderingConfiguration:self];
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:selectedConfiguration delegate:self];

This preconfigured rendering configuration will use either OpenGL ES 3.0 or OpenGL ES 2.0, depending on device specifications.

You can also create the rendering mode and rendering configuration manually, but you will have to first call +selectRenderingConfiguration on the WTWikitudeSDK before passing it, otherwise it will not be accepted. This is to make sure that the desired rendering configuration is supported by the current device.

WTExternalOpenGLESRenderingMode *externalRenderingMode = [[WTExternalOpenGLESRenderingMode alloc] initWithDelegate:self andVersion:kEAGLRenderingAPIOpenGLES2];
WTOpenGLESRenderingConfiguration *openGLESRenderingConfiguration = [[WTOpenGLESRenderingConfiguration alloc] initWithOpenGLESRenderingMode:externalRenderingMode];
WTRenderingConfiguration *selectedConfiguration = [WTWikitudeNativeSDK selectRenderingConfiguration::[NSOrderedSet orderedSetWithObjects:openGLESRenderingConfiguration, nil]];
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:selectedConfiguration delegate:self];

This approach is more verbose and only recommended if you need more control over how the modes and configurations are created.

During the Wikitude Native SDK startup phase, which is initialized with a call to -start:completion:, several methods from the WTExternalOpenGLESRenderingProtocol protocol are called. They are needed to prepare the Wikitude Native SDK for external rendering.

- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreateExternalOpenGLESUpdateHandler:(WTWikitudeOpenGLESUpdateHandler __nonnull)updateHandler
{
    self.wikitudeUpdateHandler = updateHandler;
}

- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreateExternalOpenGLESDrawHandler:(WTWikitudeOpenGLESDrawHandler __nonnull)drawHandler
{
    self.wikitudeDrawHandler = drawHandler;
}

- (EAGLContext *)eaglContextForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    if (!_sharedWikitudeEAGLCameraContext )
    {
        EAGLContext *rendererContext = [self.renderer internalContext];
        self.sharedWikitudeEAGLCameraContext = [[EAGLContext alloc] initWithAPI:[rendererContext API] sharegroup:[rendererContext sharegroup]];
    }
    return self.sharedWikitudeEAGLCameraContext;
}

- (CGRect)eaglViewSizeForExternalRenderingInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    return self.eaglView.bounds;
}

The first two methods provide block objects, that need to be called every frame from the external rendering system. Their purpose is to update internal SDK logic and draw the camera stream in OpenGL. The last one is used to get to know the current OpenGL view size in order to draw the camera frame fullscreen and with the correct aspect ratio. External rendering also requires a shared EAGLContext object that is used by the Wikitude Native SDK to issue camera related OpenGL ES calls. This way custom rendering and Wikitude rendering do not influence each other and less conflicts can occur.

An example of how those block objects can be used is the following:

if ( self.wikitudeUpdateHandler
    &&
     self.wikitudeDrawHandler )
{
    self.wikitudeUpdateHandler();
    self.wikitudeDrawHandler();
}
// ...external rendering code...

Such a snippet is typically found somewhere in the external render loop.

Rendering API selection for External OpenGL ES Rendering

For external rendering the SDK needs to know which Rendering API was used by the external rendering system so the correct rendering API calls can be made internally.

If the rendering configuration was created with a call to +createExternalOpenGLESRenderingConfiguration on the WTWikitudeSDK, the highest supported OpenGL ES version will be chosen automatically.

If the rendering configuration was created manually, the last parameter in the initialization function of the rendering mode can be used to specify the desired version.

WTExternalOpenGLESRenderingMode *externalRenderingMode = [[WTExternalOpenGLESRenderingMode alloc] initWithDelegate:self andVersion:kEAGLRenderingAPIOpenGLES3];

Internal OpenGL ES Rendering

In case no OpenGL ES rendering is already setup in the hosting Wikitude Native SDK application, the Wikitude Native SDK provides its own WTEAGLView class, which can be retrieved through WTWikitudeNativeSDK methods. To start the Wikitude Native SDK with internal rendering, you need to create the rendering configuration with WTInternalOpenGLESRenderingMode, similarly to how External Rendering is configured

WTRenderingConfiguration *renderingConfiguration = [WTWikitudeNativeSDK createInternalOpenGLESRenderingConfiguration:self];
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:renderingConfiguration delegate:self];

self.wikitudeEAGLView = [renderingConfiguration view];

After all required objects are created, the view has to be added to the view hierarchy of your application. This can either be done using Storyboards or programmatically. The Wikitude Native SDK example application does it in code.

[self.view addSubview:self.wikitudeEAGLView];
[self.wikitudeEAGLView setTranslatesAutoresizingMaskIntoConstraints:NO];

NSDictionary *views = NSDictionaryOfVariableBindings(_wikitudeEAGLView);
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"V:|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];

To do some custom OpenGL ES calls from outside the Wikitude Native SDK, custom update and draw block objects can be passed to the Wikitude Native SDK. The corresponding delegate methods are called during the startup phase.

- (WTCustomOpenGLESUpdateHandler)wikitudeNativeSDKNeedsExternalOpenGLESUpdateHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    /* Intentionally returning a nil handler here */
    return ^(){};
}

- (WTCustomOpenGLESDrawHandler)wikitudeNativeSDKNeedsExternalOpenGLESDrawHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    __weak typeof(self) weakSelf = self;

    return ^(){
        if ( weakSelf.isTracking ) {
            [weakSelf.renderableRectangle drawInContext:[EAGLContext currentContext]];
        }
    };
}

In this snippet only custom rendering is done and no update logic.

Rendering API selection for Internal Rendering

Choosing a rendering API for Internal Rendering works in the exact same way as for External Rendering.

Metal Rendering

The previous rendering examples have featured OpenGL ES, but on the majority of current iOS devices we also have the possibility to render in Metal, a rendering API which was introduced by Apple in 2014 and offers more control over the GPU than OpenGL ES does.

We provide an external and an internal rendering example for Metal analogous to the aforementioned OpenGL ES examples.

External Metal Rendering

The external Metal rendering example application comes with a simple Metal view (ExternalMetalView) and a simple renderer (ExternalMetalRenderer).

To activate external rendering you need to pass a WTMetalRenderingConfiguration to the WTWikitudeNativeSDK method -initWithRenderingConfiguration:delegate, which you get by calling the -createExternalMetalRenderingConfiguration: method of WTWikitudeNativeSDK.

WTMetalRenderingConfiguration* configuration = [WTWikitudeNativeSDK createExternalMetalRenderingConfiguration:self];
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:configuration delegate:self];

During the Wikitude Native SDK startup phase, which is initialized with a call to -start:completion:, several methods from the WTExternalMetalRenderingProtocol protocol are called. They are needed to prepare the Wikitude Native SDK for external rendering.

- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreateExternalMetalUpdateHandler:(WTWikitudeMetalUpdateHandler __nonnull)updateHandler
{
    self.wikitudeUpdateHandler = updateHandler;
}

- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreateExternalMetalDrawHandler:(WTWikitudeMetalDrawHandler __nonnull)drawHandler
{
    self.wikitudeDrawHandler = drawHandler;
}

- (id<MTLDevice>)metalDeviceForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    if (!self.mtlDevice)
    {
        self.mtlDevice = [self.renderer device];
    }
    return self.mtlDevice;
}

- (id<MTLCommandQueue>)metalCommandQueueForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    if (!self.mtlCommandQueue)
    {
        self.mtlCommandQueue = [self.renderer commandQueue];
    }
    return self.mtlCommandQueue;
}

- (CGRect)viewSizeForExternalRenderingInWikitudeNativeSDK:(WTWikitudeNativeSDK *)wikitudeNativeSDK;
{
    return self.metalView.bounds;
}

The first two methods provide block objects, that need to be called every frame from the external rendering system. Their purpose is to update internal SDK logic and draw the camera stream in Metal. The third one is used to get the current Metal view size in order to draw the camera frame fullscreen and with the correct aspect ratio. External rendering also requires an <id>MTLDevice object and an <id>MTLCommandQueue object which are used by the Wikitude Native SDK to issue camera related Metal calls. This way custom rendering and Wikitude rendering do not influence each other and less conflicts can occur.

An example of how these block objects can be used is the following:

if ( self.wikitudeUpdateHandler && self.wikitudeDrawHandler )
{
    self.wikitudeUpdateHandler();
    self.wikitudeDrawHandler([_renderer renderCommandEncoder]);
}

// ...external rendering code...

In Metal, as opposed to OpenGL ES we also need to pass the renderer's MTLRenderCommandEncoder to the wikitudeDrawMetalHandler in every frame. Since this command encoder holds all the commands of a render call, every object that needs to be rendered has to access it in order to get rendered.

Internal Metal Rendering

In case no Metal rendering is set up in the hosting Wikitude Native SDK application, the Wikitude Native SDK provides its own Metal compatible class to do so. This class is WTMetalView. An object of that class can be retrieved from a WTMetalRenderingConfiguration object. To start the Wikitude Native SDK with internal rendering, pass a configuration, created by the createInternalMetalRenderingConfiguration: method of WTWikitudeNativeSDK to -initWithRenderingConfiguration:delegate:.

WTMetalRenderingConfiguration *configuration = [WTWikitudeNativeSDK createInternalMetalRenderingConfiguration:self];

self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:configuration delegate:self];

self.wikitudeMetalView = [configuration metalView];

After that the view has to be added to the view hierarchy of your application. This can either be done using Storyboards or programmatically. The Wikitude Native SDK example application does it in code.

[self.view addSubview:self.wikitudeMetalView];
[self.wikitudeMetalView setTranslatesAutoresizingMaskIntoConstraints:NO];

NSDictionary *views = NSDictionaryOfVariableBindings(_wikitudeMetalView);
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"|[_wikitudeMetalView]|" options:0 metrics:nil views:views]];
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"V:|[_wikitudeMetalView]|" options:0 metrics:nil views:views]];

To do some custom Metal calls from outside the Wikitude Native SDK, custom update and draw block objects can be passed to the Wikitude Native SDK. The corresponding delegate methods are called during the startup phase.

- (WTCustomMetalUpdateHandler)wikitudeNativeSDKNeedsExternalMetalUpdateHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    /* Intentionally returning a nil handler here */
    return ^(){};
}

- (WTCustomMetalDrawHandler)wikitudeNativeSDKNeedsExternalMetalDrawHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
    return ^(id<MTLRenderCommandEncoder> renderCommandEncoder) {
        if (_isTracking)
        {
            [self.renderableRectangle drawWithCommandEncoder:renderCommandEncoder];
        }
    };
}

In this snippet no update logic is implemented, but only custom rendering. The method wikitudeNativeSDKNeedsExternalDrawMetalHandler: provides the internally used MTLRenderCommandEncoder. This command encoder needs to be passed to every object that should get rendered by the renderer.