Instant Tracking

The following sections detail the instant tracking feature of the Wikitude Native SDK by introducing a minimal implementation, showcasing the simplicity the Wikitude Native SDK provides.

SMART - Seamless AR Tracking

SMART is a seamless API which integrates ARKit, ARCore and Wikitude’s SLAM in a single augmented reality SDK, cross-platform, for any device. It ensures the delivery of the best possible augmented reality experience on a wider range of devices, covering 92,6% of iOS devices and about 35% of Android devices available in the market.

SMART is enabled by default but can be disabled by setting a parameter when creating an WTInstantTracker with the SMARTEnabled option. The behaviour cannot be changed during runtime.

self.instantTracker = [self.wikitudeSDK.trackerManager createInstantTracker:self configuration:^(WTInstantTrackerConfiguration *instantTrackerConfiguration) {
    instantTrackerConfiguration.SMARTEnabled = NO;
}];

To check if the device supports SMART isPlatformAssistedTrackingSupported can be used.

if ([self.wikitudeSDK.trackerManager isPlatformAssistedTrackingSupported])
{
    // device offers platform tracking capabilities (ARKit)
}

SMART provides improved tracking capabilities at the expense of control. Because of that some Wikitude SDK features are not available when platform tracking capabilities are used by enabling SMART.

Features	SMART ON and platform assisted tracking supported	SMART OFF
Improved Tracking	✓	x
Plane Orientation	x	✓
Camera Control	x	✓
Save and Load Instant Targets	x	✓
Plane Detection	x	✓

Introduction

Instant tracking is an algorithm that, contrary to those previously introduced in the Wikitude SDK, does not aim to recognize a predefined target and start the tracking procedure thereafter, but immediately start tracking in an arbitrary environment. This enables very specific use cases to be implemented.

The algorithm works in two distinct states; the first of which is the initialization state. In this state the user is required to define the origin of the tracking procedure by simply pointing the device and thereby aligning an indicator. Once the alignment is found to be satisfactory by the user (which the users needs to actively confirm), a transition to the tracking state is performed. In this state, the environment is being tracked, which allows for augmentations to be placed within the scene.

Initialization State and Tracking State

The instant tracking algorithm requires another input value to be provided in the initialization state. Specifically, the height of the tracking device above ground is required in order to accurately adjust the scale of augmentations within the scene. To this end, the example features a range input element that allows the height to be set in meters.

During the initialization, another parameter can be set which influences the alignment of the instant tracking ground plane. This ground plane is represented by the initialization indicator and can be rotated in order to start instant tracking at e.g. a wall instead of the floor. Please refer to the WTInstantTrackerConfiguration reference for detailed information.

Basic Instant Tracking

The Instant Tracking example provides a minimal implementation of the instant tracking algorithm. First we have to make some additions to our UIViewController. In order to use instant tracking, the UIViewController has to conform to the WTInstantTrackerDelegate protocol and we will need a WTInstantTracker and a WTInstantTrackingState as members.

@interface WTInstantTrackerViewController () <WTWikitudeNativeSDKDelegate, WTInstantTrackerDelegate>

@property (nonatomic, strong) WTInstantTracker                      *instantTracker;

@property (nonatomic, assign) WTInstantTrackingState                trackingState;

Initialize both the WTInstantTracker and the WTInstantTrackingState after checking if the WTWikitudeNativeSDK is running in viewDidAppear:.

self.instantTracker = [self.wikitudeSDK.trackerManager createInstantTracker:self configuration:nil];
self.trackingState = WTInstantTrackerInitializing;

A WTInstantTracker can, minimally, be instantiated with just the previously generated tracker instance, although supplying drawables to be rendered in both the initialization state and the tracking state is advisable for any practical use case. Therefore a StrokedRectangle instance is generated.

self.renderableRectangle = [[StrokedRectangle alloc] init];
self.renderableRectangle.scale = 0.45;

To use our StrokedRectangle for the initialization of WTInstantTracker, we implement the instantTracker:didChangeInitializationPose: callback function of the WTInstantTrackerDelegate.

- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didChangeInitializationPose:(nonnull WTInitializationPose *)pose
{
    [self.renderableRectangle setProjectionMatrix:pose.projection];
    [self.renderableRectangle setModelViewMatrix:pose.modelView];
}

This callback function supplies a WTInstantTracker and the current WTInitializationPose, the latter of which can be used to set the projection matrix and model view matrix of the StrokedRectangle so it is displayed properly during the initialization state.

Next we need a means to transition from one state to the other. For this task we provide the toggleInstantTrackingState: function which we conveniently call on a button click. self.trackingState holds the value of the current state.

- (IBAction)toggleInstantTrackingState:(id)sender
{
    if ( [[self.instantTrackingButton currentTitle] isEqualToString:@"Start Tracking"] )
    {
        if ( WTInstantTrackerInitializing == self.trackingState )
        {
            [self.instantTrackingButton setTitle:@"Start Initialization" forState:UIControlStateNormal];
            [self.instantTracker setActiveInstantTrackingState:WTInstantTrackerTracking];
        }
    }
    else
    {
        if ( WTInstantTrackerTracking == self.trackingState )
        {
            [self.instantTrackingButton setTitle:@"Start Tracking" forState:UIControlStateNormal];
            [self.instantTracker setActiveInstantTrackingState:WTInstantTrackerInitializing];
        }
    }
}

We now draw an orange rectangle while the instant tracker is initializing, but we want the rectangle to be blue once we're in the tracking state.

- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didChangeState:(WTInstantTrackingState)newState
{
    _trackingState = newState;

    if (_trackingState == WTInstantTrackerInitializing) {
        [self.renderableRectangle setColor:[UIColor colorWithRed:1.00f green:0.58f blue:0.16f alpha:1.0f]];
    } else {
        [self.renderableRectangle setColor:[UIColor colorWithRed:0.41f green:0.60f blue:0.76f alpha:1.0f]];
    }
}

In the WTInstantTracker's callback function instantTracker:didChangeState: we control the color of the StrokedRectangle for a better visual representation. However this only changes the color of the rectangle and not its position, which is why we need the instantTracker:didTrack: callback function to update the projection matrix and the model view matrix of the rectangle.

- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didTrack:(nonnull WTInstantTarget *)target
{
    [self.renderableRectangle setProjectionMatrix:target.projection];
    [self.renderableRectangle setModelViewMatrix:target.modelView];
}

Next, we provide the updateDeviceHeightAboveGround: function to set the deviceHeight property of the WTInstantTracker and connect it to a UISlider. While this change is, strictly speaking, not required, we strongly recommend every application to supply the device height accurately by this method or another for the Wikitude SDK to provide an accurate scale.

- (IBAction)updateDeviceHeightAboveGround:(UISlider *)sender
{
    [self.instantTracker setDeviceHeightAboveGround:@(sender.value)];
}

Lastly we have to make the height slider invisible when the state changes from initialization to tracking, so set its alpha value correctly in instantTracker:didChangeState:.

dispatch_async(dispatch_get_main_queue(), ^{
    if ( WTInstantTrackerTracking == newState )
    {
        self.deviceHeightAboveGroundSlider.alpha = 0.0;
    }
    else
    {
        self.deviceHeightAboveGroundSlider.alpha = 1.0;
    }
});

The example outlined in this section renders an orange rectangle augmentation when in initialization state as its indicator and a corresponding blue rectangle augmentation when in tracking state. While the example is quite trivial, we believe it serves the purpose of familiarizing the reader with the core concepts of instant tracking well.

Initialization State and Tracking State

Instant Scene Picking

The instant tracking feature further allows for 3D points to be queried from the underlying point cloud structure. This section is concerned with showcasing this feature based on the corresponding sample of the sample application.

To utilize this feature a 2D input position on the screen is required. A UITapGestureRecognizer is added to the view controller for that purpose.

@interface WTInstantScenePickingViewController () <WTWikitudeNativeSDKDelegate, WTInstantTrackerDelegate, WTExternalOpenGLESRenderingProtocol, UIGestureRecognizerDelegate>

@property (nonatomic, strong) UITapGestureRecognizer *tapGestureRecognizer;

_tapGestureRecognizer = [[UITapGestureRecognizer alloc] initWithTarget:self action:@selector(handleTapFrom:)];
[self.view addGestureRecognizer:_tapGestureRecognizer];
_tapGestureRecognizer.delegate = self;

The handleTapFrom:recognizer function is implemented to query the tap position, scale it using the main screen scale to convert from points to pixels and pass the resulting coordinates to the convertScreenCoordinate:toPointCloudCoordinateOnQueue:completion function. These input coordinates have their origin in the top-left corner of the screen and are within the interval [0, screen_width_in_pixels) and [0, screen_height_in_pixels). Within the completion handler function, a boolean value and a 3D point is received. The former informs on whether the operation completed successfully or not; the latter contains the result in case of success, nil in case of failure. Note that the query will fail whenever no point cloud position can be found for the input coordinate within a specific interval. For successful queries a StrokedCube instance is generated and placed at the resulting position as an augmentation.

CGPoint tapPosition = [recognizer locationInView:self.view];

CGFloat screenScale = [[UIScreen mainScreen] scale];
CGPoint tapPositionScaled = tapPosition;
tapPositionScaled.x *= screenScale;
tapPositionScaled.y *= screenScale;

[_instantTracker convertScreenCoordinate:tapPositionScaled toPointCloudCoordinateOnQueue:[NSOperationQueue currentQueue] completion:^(BOOL success, WTPoint3D* _Nullable pointCloudCoordinate) {
    if (pointCloudCoordinate) {
        StrokedCube *targetAugmentation = [[StrokedCube alloc] init];
        targetAugmentation.uniformScale = 0.05f;
        targetAugmentation.xTranslation = pointCloudCoordinate->x;
        targetAugmentation.yTranslation = pointCloudCoordinate->y;
        targetAugmentation.zTranslation = pointCloudCoordinate->z;
        [self.renderableCubes addObject:targetAugmentation];
    }
}];

Finally, running the sample allows cube augmentations to be placed on screen touch when in tracking state.

instant scene picking on the floor of the Wikitude offices

Persistent Instant Targets

The save and load instant targets feature allows for AR experiences to be persistently accessed by multiple users across devices and operating systems. Furthermore instant targets can be expanded on the fly. This section is concerned with showcasing this feature based on the corresponding samples of the sample application. This feature is not available with platform assisted tracking enabled.

Save Instant Target

To save an instant target there has to be an active InstantTracker in the tracking state and the directories of the provided path have to exist. Use the WTInstantTracker method -saveCurrentInstantTargetWithSceneName:success:error: to actually perform the save operation. The following listing explains this very detailed. Most of the code concerns the creation/verification of an directory where a file can be written to. The Wikitude SDK would override any existing file if one with the given name currently exists. To provide rich user feedback, the Wikitude SDK API offers both, a success and failure handler to update/notify the user about potential failure or success of the save operation.

- (IBAction)saveCurrentInstantTarget:(id)sender
{
    /* Create a directory where a file can be written to */
    NSArray<NSURL *> *paths = [[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask];
    NSURL *documentDirectoryRoot = [paths objectAtIndex:0];

    NSURL *instantTargetsDirectory = [documentDirectoryRoot URLByAppendingPathComponent:@"Wikitude/InstantTargets" isDirectory:YES];

    BOOL isDirectory = NO;
    if ( ![[NSFileManager defaultManager] fileExistsAtPath:[instantTargetsDirectory absoluteString] isDirectory:&isDirectory] )
    {
        NSError *directoryCreationError = nil;
        BOOL directoryCreated = [[NSFileManager defaultManager] createDirectoryAtURL:instantTargetsDirectory withIntermediateDirectories:YES attributes:nil error:&directoryCreationError];
        if ( !directoryCreated )
        {
            NSLog(@"Unable to create Document/Wikitude/InstantTargets directory inside app sandbox. Unable to save current instant target.");
            return;
        }
    }

    NSURL *instantTargetURL = [instantTargetsDirectory URLByAppendingPathComponent:@"currentInstantTarget.wto"];

    [self.navigationItem setPrompt:@"Saving instant target..."];

    [self.instantTracker saveCurrentInstantTargetWithSceneName:instantTargetURL success:^(NSString *scenePath) {
        NSLog(@"Instant target saved at path '%@'", scenePath);
        dispatch_async(dispatch_get_main_queue(), ^{
            [self.navigationItem setPrompt:nil];
        });
    } error:^(NSError *error) {
        NSLog(@"Error while saving instant target to file. '%@'", [error localizedDescription]);
        dispatch_async(dispatch_get_main_queue(), ^{
            UIAlertController *errorSavingCurrentInstantTargetController = [UIAlertController alertControllerWithTitle:@"Error saving instant target" message:[error localizedDescription] preferredStyle:UIAlertControllerStyleAlert];
            [self presentViewController:errorSavingCurrentInstantTargetController animated:YES completion:nil];
        });
    }];
}

Load Instant Target

To load an instant target there has to be an active tracker and a previously saved instant target.

The WTInstantTargetRestorationConfiguration defines the behaviour of the loaded instant target. In the example the policy is set to WTInstantTargetExpansionPolicyAllowExpansion which means that the Wikitude SDK will try to find new points and add them to the target, making tracking more robust over time.

[self.instantTracker loadExistingInstantTargetWithTargetCollectionResource:instantTargetCollectionResource restoration:^(WTInstantTargetRestorationConfiguration *configuration) {
        [configuration setInstantTargetExpansionPolicy:WTInstantTargetExpansionPolicyAllowExpansion];
    } success:^(NSString *scenePath) {
        // ...            
    } error:^(NSError *error) {
        // ...
}];

To load an previously saved instant target, a WTTargetCollectionResource has to be created from the file saved by WTInstantTracker -saveCurrentInstantTargetWithSceneName:success:error:. Please note that the NSURL API +fileURLWithPath: is used to get a URL from which an iOS application is allowed to open a file.

WTTargetCollectionResource *instantTargetCollectionResource = [[self.wikitudeSDK trackerManager] createTargetCollectionResourceFromURL:[NSURL fileURLWithPath:instantTargetFilePath]];

The following snippet shows how the initial path to a .wto file is generated.

/* Check if a instant target is stored at a predefined path */
NSString* documentsPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];

NSString *instantTargetFilePath = [documentsPath stringByAppendingPathComponent:@"Wikitude/InstantTargets/currentInstantTarget.wto"];
BOOL instantTargetFileExists = [[NSFileManager defaultManager] fileExistsAtPath:instantTargetFilePath];

After the instant target is loaded the tracker will immediately try to find and track it. Providing a Handler to receive the callbacks on a specific thread is optional.

The following listing shows the complete implementation of a potential load instant target implementation. Please note again the success and error handler that can be used to provide user feedback.

- (IBAction)loadInstantTarget:(id)sender
{
    /* Check if a instant target is stored at a predefined path */
    NSString* documentsPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];

    NSString *instantTargetFilePath = [documentsPath stringByAppendingPathComponent:@"Wikitude/InstantTargets/currentInstantTarget.wto"];
    BOOL instantTargetFileExists = [[NSFileManager defaultManager] fileExistsAtPath:instantTargetFilePath];
    if ( !instantTargetFileExists )
    {
        NSLog(@"No instant target file found at the given path '%@'", instantTargetFilePath);
        return;
    }
    else
    {
        [self.navigationItem setPrompt:@"Loading instant target..."];
        WTTargetCollectionResource *instantTargetCollectionResource = [[self.wikitudeSDK trackerManager] createTargetCollectionResourceFromURL:[NSURL fileURLWithPath:instantTargetFilePath]];
        [self.instantTracker loadExistingInstantTargetWithTargetCollectionResource:instantTargetCollectionResource restoration:^(WTInstantTargetRestorationConfiguration *configuration) {
            [configuration setInstantTargetExpansionPolicy:WTInstantTargetExpansionPolicyAllowExpansion];
        } success:^(NSString *scenePath) {
            NSLog(@"successfully restored instant target from path '%@'", instantTargetFilePath);
            dispatch_async(dispatch_get_main_queue(), ^{
                [self.navigationItem setPrompt:nil];
            });
        } error:^(NSError *error) {
            NSLog(@"Failed to load instant target from path '%@'", instantTargetFilePath);
            dispatch_async(dispatch_get_main_queue(), ^{
                UIAlertController *errorLoadingInstantTargetAlertController = [UIAlertController alertControllerWithTitle:@"Error loading instant target" message:[error localizedDescription] preferredStyle:UIAlertControllerStyleAlert];
                [self presentViewController:errorLoadingInstantTargetAlertController animated:YES completion:nil];
            });
        }];
    }
}

Plane Detection - Experimental

Once Plane detection is enabled, the Wikitude SDK will search for planar surfaces in the representation of the scene (the point cloud). Sufficient and accurate information will then produce planar surfaces, that developers can use to interact with the actual scene. Hit testing will use found planes for a better estimate of depth values in the scene. The algorithm can detect different types of planes

Horizontal Up (e.g. floor, carpet, table)
Horizontal Down (e.g. ceiling)
Vertical (walls,...)
Arbitrary (ramps, staircases,...)

Developers can choose which type of planes they are interested in and filter correspondingly.

To use plane detection the WTInstantTracker has to be created with an WTInstantTrackerConfiguration that enables plane detection. For this purpose WTInstantTrackerConfiguration -enablePlaneDetection: has to be called. The given WTPlaneDetectionConfiguration object can then be used to further define how plane detection should run. Currently plane detection is only available with SMART disabled.

weakSelf.instantTracker = [weakSelf.wikitudeSDK.trackerManager createInstantTracker:weakSelf configuration:^(WTInstantTrackerConfiguration *instantTrackerConfiguration) {
    [instantTrackerConfiguration setSMARTEnabled:NO];
    [instantTrackerConfiguration enablePlaneDetection:^(WTPlaneDetectionConfiguration *planeDetectionConfiguration) {
        planeDetectionConfiguration.planeFilter = WTPlaneFilter_HorizontalUpward | WTPlaneFilter_HorizontalDownward | WTPlaneFilter_Vertical | WTPlaneFilter_Arbitrary;
        planeDetectionConfiguration.enableConvexHull = YES;
    }];
}];

WTInstantTrackerDelegate provides methods that inform about new, updated and lost planes.

The following snippets demonstrate the usage of these protocol methods. When a plane is recognized, a new object is created that represents a plane for rendering. Its stored in a dictionary so that it can be referenced later to update its matrices.

- (void)instantTracker:(WTInstantTracker *)instantTracker didRecognizePlane:(WTPlane *)plane
{
    NSLog(@"recognized plane with id '%ld'", plane.uniqueId);

    WTPlanePolygon *planeVisualization = [[WTPlanePolygon alloc] init];
    [planeVisualization setProjectionMatrix:_lastProjection];

    UIColor* planeColor;
    switch (plane.type) {
        case WTPlaneType_HorizontalUpward:
            planeColor = [UIColor blueColor];
            break;
        case WTPlaneType_HorizontalDownward:
            planeColor = [UIColor orangeColor];
            break;
        case WTPlaneType_Vertical:
            planeColor = [UIColor colorWithRed:0.93 green:0.13 blue:0.78 alpha:1.0];
            break;
        case WTPlaneType_Arbitrary:
            planeColor = [UIColor colorWithRed:0.25 green:0.93 blue:0.13 alpha:1.0];
            break;
    }

    [planeVisualization setColor:planeColor];

    [planeVisualization setModelViewMatrix:plane.matrix];
    [self.renderablePlanes setObject:planeVisualization forKey:[NSNumber numberWithLong:plane.uniqueId]];

    [plane requestConvexHull:[WTDefaultConvexPlaneHullDataProvider convexHullDataProviderWithCompletionHandler:^(NSArray<NSValue *> * _Nonnull convexHull) {

        std::vector<WTPoint2D> points;
        points.reserve(convexHull.count);
        for (NSValue* v in convexHull) {
            points.emplace_back();
            [v getValue:&points.back()];
        }

        std::vector<GLKVector2> convertedPoints;
        convertedPoints.resize(points.size());
        for (std::size_t i = 0; i < points.size(); ++i) {
            const WTPoint2D& p = points[i];
            convertedPoints[i] = GLKVector2Make(p.x, p.y);
        }

        WTPlanePolygon *planeVisualization = [self.renderablePlanes objectForKey:[NSNumber numberWithLong:plane.uniqueId]];
        [planeVisualization setPoints:convertedPoints];
    }]];
}

Every time a plane is tracked again, it's visualization counterpart is updated with the new matrix.

- (void)instantTracker:(WTInstantTracker *)instantTracker didTrackPlane:(WTPlane *)plane
{
    NSLog(@"didTrackPlane");

    WTPlanePolygon *planeVisualization = [self.renderablePlanes objectForKey:[NSNumber numberWithLong:plane.uniqueId]];
    [planeVisualization setProjectionMatrix:_lastProjection];
    [planeVisualization setModelViewMatrix:plane.matrix];
}

In case the Wikitude SDK is longer able to track a plane, -instantTracker:didLosePlane: is called with the plane object as parameter.

- (void)instantTracker:(WTInstantTracker *)instantTracker didLosePlane:(WTPlane *)plane
{
    NSLog(@"lost plane with id '%ld'", plane.uniqueId);
    [self.renderablePlanes removeObjectForKey:[NSNumber numberWithLong:plane.uniqueId]];
}

A WTPlane can be rendered like all other tracking targets but in addition to the matrix and extents it also provides a convex hull which can be rendered directly as a triangle fan. This can be seen in -instantTracker:didRecognizePlane: above.

Scene picking on two tracked planes, one horizontal and one vertical.

Documentation

Wikitude SDK iOS Native API