Documentation

Image Recognition

Introduction

This example shows how to recognize images in the viewfinder and overlay it with images. Furthermore it shows how to recognize multiple different images and how to react on user clicks on the overlaid elements.

For a better understanding, here are some terms that will be used in the following and other section of this documentation related to augmented reality.

  • Target: A target image and its associated extracted data that is used by the tracker to recognize an image.

  • Target collection: An archive storing a collection of targets that can be recognized by the tracker. A target collection can come from two different resource types

    • Plain: A regular ZIP file containing images in plain JPG or PNG format
    • Pre-processed: Regular images that are converted into a WTC file (Wikitude Target collection) for faster processing and storing more images offline.
  • ImageTracker: The tracker analyzes the live camera image and detects the targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.

Important: Learn how to create Image Targets

Make sure to read the chapter on how to create Image Targets before using Image Recognition on your own.

Extended Recognition Range

Introduced with SDK 7.0, the Wikitude SDK Image Recognition engine can make use of HD camera frames to detect images from further away. Further away in this context means distances 3x further away, compared to not enabling this mode (e.g. A4-sized target can reach recognition distances in the area of 2.4 meters/ 8 feet). This feature is called Image Recognition Extended Range and can be activated through a setting in the ImageTracker class. The setting extendedRangeRecognition is optional and accepts the following three constants

  • ON
  • OFF
  • AUTO (default)

see WTImageRecognitionRangeExtension imageRecognitionRangeExtension

Processing a HD camera frame in the pipe-line is computationally more expensive - this means that this feature is only activated, if the phone can handle this additional effort without sacrificing user experience. The AUTO setting will determine that based on available CPU power and camera access (e.g. Camera2 API for Android) and is the default setting in the SDK.

Simple Image Recognition

In this section we will go through the WTSimpleClientTrackerViewController code, which you can find in the example applications folder Native Examples/Controller/ClientTracking. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this sample even if you are only interested in cloud recognition for example.

Please refer to the Setup Guide section of this documentation to find detailed information on how to integrate the Wikitude Native SDK into a Xcode project.

In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.

The WTWikitudeNativeSDK class is structured to be used in a standard iOS view controller as a model of the MVC pattern. This class is central starting point for anything related to the Wikitude Native SDK. It provides factory methods to create different kinds of tracker and to receive information of those through their delegate objects.

A typical usage would be to have a strong reference to a WTWikitudeNativeSDK object in a view controller and to define this view controller to be conform to the WTWikitudeNativeSDKDelegate protocol

@interface WTSimpleClientTrackerViewController () <WTWikitudeNativeSDKDelegate, WTImageTrackerDelegate>

The WTWikitudeNativeSDKDelegate protocol defines methods that can be used to react on SDK specific events. It is the chosen way to communicate e.g. runtime errors that can not be returned directly from an API call. Such errors might contain information about camera authorization status changes and inconsistent rendering setups.

To receive image tracker information about recognized/tracked or lost targets, this view controller must also conform to the WTImageTrackerDelegate protocol. All methods in the mentioned protocol are optionals thus only provide information from the SDK but don't require any action from the application. Additionally this protocol provides information about the loading status of a particular image tracker object.

The next step we will take is create an instance of the WTWikitudeNativeSDK class. This is the only object that you have to create manually. All other object will be created by factory methods of the WTWikitudeNativeSDK object.

IMPORTANT: Do not instantiate any other class out of the Wikitude Native SDK other than the WTWikitudeNativeSDK manually. All object somehow play together in a well defined manner and that why they are internally controlled by the WTWikitudeNativeSDK object and its sub components like WTTrackerManger and WTCaptureDeviceManager.

A WTWikitudeNativeSDK object can be instantiated at any time and does not depend on any particular application lifecycle event. To integrate internal processes into the hosting application, -start:completion and -stop are used. These two methods should either be called from - viewWillAppear: | - viewWillDisappear: or -viewDidAppear: | -viewDidDisappear:. To refine the startup behaviour of the Wikitude Native SDK, the first block parameter of -start:completion: contains an already created and setup object of type WTNativeSDKStartupConfiguration. Public properties of this object can be used to further define how the Wikitude Native SDK should start e.g. its camera. Typical settings from this object are the camera preset which defines in which resolution the device camera should be started or what's the desired target rendering frame rate. The second block parameter contains a BOOL indicating if the given properties could be taken into account during the final startup phase. If an error occurred, isRunning is set to NO and the given NSError object contains more information about what went wrong.

The second block parameter is also the ideal starting point to create any tracker object from the Wikitude Native SDK if the running state is YES. Image trackers work with target collection resources, that load target collections represented through a .wtc file (Wikitude Target Collection). The .wtc file can be packed into the application bundle and a file URL retrieved using the NSBundle method -URLForResource:withExtension:subdirectory: or uploaded to a server and downloaded from the Wikitude Native SDK.

First, the TargetCollectionResource is created through the createTargetCollectionResourceFromURL:completion factory method from WTWikitudeNativeSDK and when it successfully finished loading the file containing the target collection, an ImageTracker can be created with it as a parameter.

To get notified when the image tracker finished loading, its delegate method -imageTrackerDidLoadTargets: is called or -imageTracker:didFailToLoadTargets: if the .wtc could not be loaded. If the target collection resource finished successfully, but the image tracker failed to load the targets, it means that the file was loaded successfully, but the image tracker was unable to use it, either because it is corrupted, or because it is from an unsupported version, or if the file is not a target collection at all.

[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
    if ( !isRunning ) {
        NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
    }
    else
    {
        __weak typeof(self) weakSelf = self;

        NSURL *imageTrackerResourceURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
        self.targetCollectionResource = [self.wikitudeSDK.trackerManager createTargetCollectionResourceFromURL:imageTrackerResourceURL completion:^(BOOL success, NSError * _Nullable error) {
            if ( !success )
            {
                NSLog(@"Failed to load URL resource. Reason: %@", [error localizedDescription]);
            }
            else
            {
                weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:nil];
            }
        }];
    }
}];

Next we will have a closer look on how to get on tracker specific data. The WTImageTrackerDelegate provides three methods that will be called when the tracker found, lost or tracked targets.

  • -imageTracker:didRecognizeImage: is called if a new target was found in the current camera frame
  • -imageTracker:didTrackImage: is called when an already known target moved to a new position
  • -imageTracker:didLoseImage: is called when an already known target could not be found anymore in the latest camera frame

The WTImageTarget object provides information about which target changed and what changed for the target, e.g. its position which would be represented through a OpenGL like model view matrix.

- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didRecognizeTarget:(nonnull WTImageTarget *)recognizedTarget
{
    NSLog(@"recognized target '%@'", [recognizedTarget name]);
    _isTracking = YES;
}

- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didTrackTarget:(nonnull WTImageTarget *)trackedTarget
{
    [self.renderableRectangle setProjectionMatrix:trackedTarget.projection];
    [self.renderableRectangle setModelViewMatrix:trackedTarget.modelView];
}

- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didLoseTarget:(nonnull WTImageTarget *)lostTarget
{
    NSLog(@"lost target '%@'", [lostTarget name]);
    _isTracking = NO;
}


- (void)imageTrackerDidLoadTargets:(nonnull WTImageTracker *)imageTracker
{
    NSLog(@"Image tracker loaded");
}

- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didFailToLoadTargets withError:(nonnull NSError *)error
{
    NSLog(@"Unable to load image tracker. Reason: %@", [error localizedDescription]);
}

Multiple Image Targets Tracking

The following are instructions for adding multiple target tracking to the Simple 2D Client Tracking sample, which you can find in the example applications folder Native Examples/Controller/ClientTracking.

In the Simple 2D Client Tracking sample we track only one target, but this can be easily changed. For this sample we decided to track a maximum number of five targets. Setting this maximum number is merely a performance optimization that takes effect once five targets are actually tracked as the search for additional targets can be suspended. If your use case allows for a maximum number to be set, we recommend you doing so.

First, we have to make our UIViewController a WTImageTargetDelegate, so it can react to changing distances between image targets.

@interface WTMultipleTargetsTrackerViewController () <WTWikitudeNativeSDKDelegate, WTImageTrackerDelegate, WTImageTargetDelegate>

If we track multiple targets, we need multiple StrokedRectangle instances to visualize the tracking, so we get rid of renderableRectangle and create a NSMutableDictionary in its stead.

@property (nonatomic, strong) NSMutableDictionary *renderableRectangles;

In -viewDidLoad we initialize the dictionary and leave everything else as it is.

The only configuration we need to add to the Wikitude Native SDK is in -viewDidAppear:. Here we create the image tracker mainly as we used to, but add a WTImageTrackerConfiguration as the last parameter. In this configuration we tell the image tracker to set the maximum number of concurrently trackable targets to five. The other thing we have to set is the distanceChangedThreshold, a value that determines at which distance change between two targets the Wikitude Native SDK should trigger the callback function -didRecognizeChangeBetweenImageTarget:andImageTarget:.

weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:^(WTImageTrackerConfiguration *imageTrackerConfiguration) {
    imageTrackerConfiguration.maximumNumberOfConcurrentlyTrackableTargets = 5;
}];

[weakSelf.imageTracker setDistanceChangedThreshold:10.0f];

That was already the configuration of the Wikitude Native SDK. Now all we have to do is handle the rectangles. Most of the rectangle handling is just the same as in the Simple 2D Client Tracking sample with the only difference being that the changes have to be applied to either all of the StrokedRectangle instances or a single one that has to be fetched from the dictionary first. We might encounter multiple instances of the same image, so we need unique keys. With the function -getKeyForImageTarget: we create a unique key for each WTImageTarget, which is a combination of its name value and trackingId value.

When the image tracker recognizes an image, the -imageTracker:didRecognizeImage:-callback function gets called. In it we make our UIViewController the delegate of the recognized target. Furthermore we create a new StrokedRectangle instance which we add to our renderableRectangles dictionary with the name of recognizedTarget.

NSString* key = [self getKeyForImageTarget:recognizedTarget];

recognizedTarget.delegate = self;
[self.renderableRectangles setObject:[[StrokedRectangle alloc]init] forKey:key];

As we previously defined, the callback function -didRecognizeChangeBetweenImageTarget:andImageTarget: gets called when the distance between the targets changes more than a certain threshold. For demonstration purposes we set a threshold of 300.0 and change the color of the rectangles to red if they get too close to each other. If both targets are instances of the same image, we set their rectangle colors to blue;

- (void)didRecognizeChangeBetweenImageTarget:(nonnull WTImageTarget *)firstTarget andImageTarget:(nonnull WTImageTarget *)secondTarget
{
    UIColor *rectColor = [UIColor colorWithRed:1.0f green:0.58f blue:0.16f alpha:1.0f];

    if ( 300.f > [firstTarget distanceTo:secondTarget] )
    {
        if ( [firstTarget.name isEqualToString:secondTarget.name] )
        {
            rectColor = [UIColor blueColor];
        }
        else
        {
            rectColor = [UIColor redColor];
        }
    }

    NSString* firstKey = [self getKeyForImageTarget:firstTarget];
    NSString* secondKey = [self getKeyForImageTarget:secondTarget];
    [(StrokedRectangle *)[self.renderableRectangles objectForKey:firstKey] setColor:rectColor];
    [(StrokedRectangle *)[self.renderableRectangles objectForKey:secondKey] setColor:rectColor];
}

Extended Image Tracking

Based on the Simple 2D Client Tracking sample to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In the sample we simply set a wildcard * so that all targets in this tracker are extended.

weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:^(WTImageTrackerConfiguration *imageTrackerConfiguration) {
    imageTrackerConfiguration.extendedTargets = @[@"*"];
}];