Documentation

Image Recognition

Introduction

This example shows how to recognize images in the viewfinder and overlay it with images. Furthermore it shows how to recognize multiple different images and how to react on user clicks on the overlaid elements.

For a better understanding, here are some terms that will be used in the following and other section of this documentation related to augmented reality.

  • Target: A target image and its associated extracted data that is used by the tracker to recognize an image.

  • Target collection: An archive storing a collection of targets that can be recognized by the tracker. A target collection can come from two different resource types

    • Plain: A regular ZIP file containing images in plain JPG or PNG format
    • Pre-processed: Regular images that are converted into a WTC file (Wikitude Target collection) for faster processing and storing more images offline.

  • ImageTracker: The tracker analyzes the live camera image and detects the targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.

Important: Learn how to create Image Targets Make sure to read the chapter on how to create Image Targets before using Image Recognition on your own.

Extended Recognition Range

Introduced with SDK 7.0, the Wikitude SDK Image Recognition engine can make use of HD camera frames to detect images from further away. Further away in this context means distances 3x further away, compared to not enabling this mode (e.g. A4-sized target can reach recognition distances in the area of 2.4 meters/ 8 feet). This feature is called Image Recognition Extended Range and can be activated through a setting in the ImageTracker class. The setting extendedRangeRecognition is optional and accepts the following three constants

  • ON
  • OFF
  • AUTO (default)

see ImageTrackerConfiguration.ImageRecognitionRangeExtension

Processing a HD camera frame in the pipe-line is computationally more expensive - this means that this feature is only activated, if the phone can handle this additional effort without sacrificing user experience. The AUTO setting will determine that based on available CPU power and camera access (e.g. Camera2 API for Android) and is the default setting in the SDK.

Simple Image Tracking Android

In this section we will go through the code of the SimpleImageTrackingActivity, which you can find in the example application under the package com.wikitude.samples.tracking.image. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this section even if you are for example only interested in cloud recognition.

The WikitudeSDK class is structured to be used in a standard Android activity and to make use of the activities life cycle events. We will use interfaces to communicate to the WikitudeSDK which type of rendering method we would like to use and to provide the necessary callbacks for Client- and Cloud-Trackers.

First let us have a look at the declaration of the activity class. 

public class SimpleImageTrackingActivity extends Activity implements ImageTrackerListener, ExternalRendering {
...
}

We subclass the standard Android activity and implement the interfaces ImageTrackerListener and ExternalRendering. Later on when we create the instance of the WikitudeSDK we will pass the this pointer of our activity to the WikitudeSDK constructor and this way indicate our chosen type of rendering. In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.

The next step we will take is create an instance of the WikitudeSDK class. This and the NativeStartupConfiguration are the only objects you will need to create by yourself, every other object will be created by factory methods of the WikitudeSDK instance.

IMPORTANT: Do not instantiate any other class out of the Wikitude Native SDK other than the WikitudeSDK and the NativeStartupConfiguration.

We are now going through each method of the SimpleImageTrackingActivity class, starting with onCreate. In the onCreate method we instantiate an instance of the WikitudeSDK, and an instance of the NativeStartupConfiguration, which will hold our license key. If you do not have a license key yet, read this chapter on how to obtain a free trial key. After that we are ready to propagate the onCreate life cycle event to the Wikitude SDK. It is very important that you do propagate onCreate, onPause and onResume otherwise the Wikitude SDK won't be able to properly manage its resources which will lead to unexpected behavior.

After we called the WikitudeSDK onCreate method the SDK is initialized and we are now able to create a TargetCollectionResource and an ImageTracker. To do that we get the TrackerManager from the WikitudeSDK instance and call createTargetCollectionResource passing the url to the WTC file. Since we are loading an asset on the device we indicate that by starting the url with the string file:///android_asset/ and add the path to the file starting from the asset root directory.

After createTargetCollectionResource we can create the ImageTracker that will use this newly created TargetCollectionResource by calling the createImageTracker method from the TrackerManager.

To get notified when the Tracker finished loading, recognized a target and so on we register the Activity which implements ImageTrackerListener as a Listener by passing this as the second parameter to the function.

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    wikitudeSDK = new WikitudeSDK(this);
    NativeStartupConfiguration startupConfiguration = new NativeStartupConfiguration();
    startupConfiguration.setLicenseKey(WikitudeSDKConstants.WIKITUDE_SDK_KEY);
    startupConfiguration.setCameraPosition(CameraSettings.CameraPosition.BACK);
    startupConfiguration.setCameraResolution(CameraSettings.CameraResolution.AUTO);

    wikitudeSDK.onCreate(getApplicationContext(), this, startupConfiguration);

    final TargetCollectionResource targetCollectionResource = wikitudeSDK.getTrackerManager().createTargetCollectionResource("file:///android_asset/magazine.wtc");
    wikitudeSDK.getTrackerManager().createImageTracker(targetCollectionResource, this, null);
}

The next important method to look at is the onRenderExtensionCreated method. Since we decided to use the external rendering functionality by implementing ExternalRendering, the WikitudeSDK provides us with a RenderExtension. The RenderExtension interface exposes the same methods as the standard GLSurfaceView.Renderer. In your custom GLSurfaceView.Renderer the first thing to do in every method is to always call the same method on the provided RenderExtension. To be able to that we pass the RenderExtension to the constructor of our Renderer, create our SurfaceView, initialize a Driver and set our SurfaceView as our content view.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension) {
    glRenderer = new GLRenderer(renderExtension);
    view = new CustomSurfaceView(getApplicationContext(), glRenderer);
    driver = new Driver(view, 30);
    setContentView(view);
}

Next we will have a look at the ImageTrackerListener interface. The onErrorLoadingTargets method will be called like the name suggest when the Wikitude SDK wasn't able to load the targets from the target collection resource into the image tracker. This is called, for example, when the loading of the file was successful, but the contents were corrupted or somehow incompatible with the image tracker. The onTargetsLoaded method will be called once when the targets were successfully loaded into the tracker. When the image tracker first recognizes a target it will call onImageRecognized providing you with the recognized target name. When the image tracker starts tracking this target it will call onImageTracked continuously until it loses the target and finishes tracking with a call to onImageLost.

The ImageTarget object provided in the onImageTracked method contains information about the tracked target like the name, the distance to the target and most importantly the matrices which describe where on the camera frame the target was found. We create a StrokedRectangle instance when the image is recognized, update its matrices when it is tracked and remove is when it is lost. The scale values being set allow for the StrokedRectangle to adjust to the target's aspect ratio.

@Override
public void onTargetsLoaded(ImageTracker tracker) {
    Log.v(TAG, "Image tracker loaded");
}

@Override
public void onErrorLoadingTargets(ImageTracker tracker, WikitudeError error) {
    Log.v(TAG, "Unable to load image tracker. Reason: " + error.getMessage());
}

@Override
public void onImageRecognized(ImageTracker tracker, final ImageTarget target) {
    Log.v(TAG, "Recognized target " + target.getName());

    StrokedRectangle strokedRectangle = new StrokedRectangle(StrokedRectangle.Type.STANDARD);
    glRenderer.setRenderablesForKey(target.getName() + target.getUniqueId(), strokedRectangle, null);
}

@Override
public void onImageTracked(ImageTracker tracker, final ImageTarget target) {
    StrokedRectangle strokedRectangle = (StrokedRectangle)glRenderer.getRenderableForKey(target.getName() + target.getUniqueId());

    if (strokedRectangle != null) {
        strokedRectangle.projectionMatrix = target.getProjectionMatrix();
        strokedRectangle.viewMatrix = target.getViewMatrix();

        strokedRectangle.setXScale(target.getTargetScale().getX());
        strokedRectangle.setYScale(target.getTargetScale().getY());
    }
}

@Override
public void onImageLost(ImageTracker tracker, final ImageTarget target) {
    Log.v(TAG, "Lost target " + target.getName());
    glRenderer.removeRenderablesForKey(target.getName() + target.getUniqueId());
}

Multiple Image Targets Tracking

This section is concerned with extending the simple image recognition sample to allow for multiple targets to be recognized and tracked simultaneously.

Firstly, in onCreate, we change the configuration that is being passed to the TrackerManager when creating an ImageTracker. Specifically, we set the maximum number of concurrently trackable targets to 5, set the threshold to register distance changes to 10 millimeters and set the physical targets heights of the two targets being used to 252 millimeters (252mm corresponds to their height when being printed with 100% scaling on A4 paper). The limit of at most 5 targets being tracked simultaneously is merely a performance optimization that takes effect once 5 targets have actually been recognized as the search for additional ones can be suspended. If your use case allows setting this parameter, we recommend you doing so.

HashMap<String, Integer> physicalTargetImageHeights = new HashMap<>();
physicalTargetImageHeights.put("pageOne", 252);
physicalTargetImageHeights.put("pageTwo", 252);

ImageTrackerConfiguration trackerConfiguration = new ImageTrackerConfiguration();
trackerConfiguration.setMaximumNumberOfConcurrentlyTrackableTargets(5);
trackerConfiguration.setDistanceChangedThreshold(10);
trackerConfiguration.setPhysicalTargetImageHeights(physicalTargetImageHeights);

final TargetCollectionResource targetCollectionResource = wikitudeSDK.getTrackerManager().createTargetCollectionResource("file:///android_asset/magazine.wtc");
wikitudeSDK.getTrackerManager().createImageTracker(targetCollectionResource, this, trackerConfiguration);

Additionally, we alter the code such that multiple rectangle augmentations can be draw; one for each target currently being tracked. The onImageRecognized allocates a new StrokedRectangle instance and registers it with the GLRenderer instance. The key being used for registration is a combination of the image target name and a unique ID that allows for identical targets being recognized to be distinguished from one another. The onImageTracked function updates the previously registered rectangles with the matrices and scales from the input ImageTarget. The onImageLost function correspondingly removes the rectangle instance that pertains to the image target that has been lost.

@Override
public void onImageRecognized(ImageTracker tracker, final ImageTarget target) {
    [...]

    StrokedRectangle strokedRectangle = new StrokedRectangle(StrokedRectangle.Type.STANDARD);
    glRenderer.setRenderablesForKey(target.getName() + target.getUniqueId(), strokedRectangle, null);
}

@Override
public void onImageTracked(ImageTracker tracker, final ImageTarget target) {
    StrokedRectangle strokedRectangle = (StrokedRectangle)glRenderer.getRenderableForKey(target.getName() + target.getUniqueId());

    if (strokedRectangle != null) {
        strokedRectangle.projectionMatrix = target.getProjectionMatrix();
        strokedRectangle.viewMatrix = target.getViewMatrix();

        strokedRectangle.setXScale(target.getTargetScale().getX());
        strokedRectangle.setYScale(target.getTargetScale().getY());
    }
}

@Override
public void onImageLost(ImageTracker tracker, final ImageTarget target) {
    [...]

    glRenderer.removeRenderablesForKey(target.getName() + target.getUniqueId());
}

Lastly, we extend the Activity such that it can recognize a change in distance between the targets being tracked. To do so, we add a private member variable of type ImageTarget.OnDistanceBetweenTargetsListener and initialize it with a newly allocated instance of that type, overriding the onDistanceBetweenTargetsChanged function. In its function body a color is chosen based on the distance between two targets such that targets being closer than 300 millimeters get a non-orange color. Specifically, blue for two identical targets and red for two distinct targets. If any interactions between specific image targets are desired, the name contained in the image target parameters passed in would have to be used to identify such a case.

private final ImageTarget.OnDistanceBetweenTargetsListener mDistanceListener = new ImageTarget.OnDistanceBetweenTargetsListener() {
    @Override
    public void onDistanceBetweenTargetsChanged(int distance, ImageTarget firstTarget, ImageTarget secondTarget) {
        float r = 1.0f;
        float g = 0.58f;
        float b = 0.16f;

        if (distance < 300.0f) {
            if (firstTarget.getName().equals(secondTarget.getName())) {
                r = 0.0f;
                g = 0.0f;
                b = 1.0f;
            } else {
                r = 1.0f;
                g = 0.0f;
                b = 0.0f;
            }
        }

        StrokedRectangle firstStrokedRectangle = (StrokedRectangle)glRenderer.getRenderableForKey(firstTarget.getName() + firstTarget.getUniqueId());
        if (firstStrokedRectangle != null) {
            firstStrokedRectangle.setColor(r, g, b);
        }

        StrokedRectangle secondStrokedRectangle = (StrokedRectangle)glRenderer.getRenderableForKey(secondTarget.getName() + secondTarget.getUniqueId());
        if (secondStrokedRectangle != null) {
            secondStrokedRectangle.setColor(r, g, b);
        }
    }
};

To have the onDistanceBetweenTargetsChanged function called, the listener object is registered to and unregistered from the ImageTarget in onImageRecognized and onImageLost correspondingly.

@Override
public void onImageRecognized(ImageTracker tracker, final ImageTarget target) {
    [...]

    target.setOnDistanceBetweenTargetsListener(mDistanceListener);
}

@Override
public void onImageLost(ImageTracker tracker, final ImageTarget target) {
    [...]

    target.setOnDistanceBetweenTargetsListener(null);
}

Extended Image Tracking

Extended tracking is an optional mode you can set for each target separately. In this mode the Wikitude SDK will try to continue to scan the environment of the user even if the original target image is not in view anymore. So the tracking extends beyond the limits of the original target image. The performance of this feature depends on various factors like computing power of the device, background texture and objects.

Based on the previous sample, to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In this sample we simply set a wildcard * so that all targets in this tracker are extended. 

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    ...
    ImageTrackerConfiguration trackerConfiguration = new ImageTrackerConfiguration();
    trackerConfiguration.setExtendedTargets(new String[]{"*"});

    final TargetCollectionResource targetCollectionResource = wikitudeSDK.getTrackerManager().createTargetCollectionResource("file:///android_asset/iot_tracker.wtc");
    imageTracker = wikitudeSDK.getTrackerManager().createImageTracker(targetCollectionResource, this, trackerConfiguration);
}