Documentation

Client Recognition

This example shows how to recognize images in the viewfinder and overlay it with images.

For a better understanding, here are some terms that will be used in the following and other section of this documentation related to vision-based augmented reality.

• Target: A 2D target image and its associated extracted data that is used by the tracker to recognize an image.

• Target Collection: An archive storing a collection of 2D targets that can be recognized by the tracker. A target collection can hold up to 1000 targets. Target collections are stored as .wtc files

• 3D Tracking Map: A tracking map is the equivalent of a target for 3D tracking. The map contains the relevant characteristics of a three-dimensional object. In order to recognize and track 3D objects you need to record a map first and then load this map. Maps are stored as .wtm files.

• ClientTracker: The tracker analyzes the live camera image and detects the 2D targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.

Simple 2D Client Tracking Android

In this section we will go through the code of the SimpleClientTrackingActivity, which you can find in the example application under the package com.wikitude.samples.recognition.client. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this section even if you are for example only interested in cloud recognition.

The WikitudeSDK class is structured to be used in a standard Android activity and to make use of the activities life cycle events. We will use interfaces to communicate to the WikitudeSDK which type of rendering method we would like to use and to provide the necessary callbacks for Client- and Cloud-Trackers.

First let us have a look at the declaration of the activity class.

public class SimpleClientTrackingActivity extends Activity implements ClientTrackerEventListener, ExternalRendering {
...
}

We subclass the standard Android activity and implement the interfaces ClientTrackerEventListener and ExternalRendering. Later on when we create the instance of the WikitudeSDK we will pass the this pointer of our activity to the WikitudeSDK constructor and this way indicate our chosen type of rendering. In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.

The next step we will take is create an instance of the WikitudeSDK class. This and the StartupConfiguration are the only objects you will need to create by yourself, every other object will be created by factory methods of the WikitudeSDK instance.

We are now going through each method of the SimpleClientTrackingActivity class, starting with onCreate. In the onCreate method we instantiate an instance of the WikitudeSDK, and an instance of the WikitudeSDKStartupConfiguration, which will hold our license key. If you do not have a license key yet, read this chapter on how to obtain a free trial key. After that we are ready to propagate the onCreate life cycle event to the Wikitude SDK. It is very important that you do propagate onCreate, onPause and onResume otherwise the Wikitude SDK won't be able to properly manage its resources which will lead to unexpected behavior.

After we called the WikitudeSDK onCreate method the SDK is initialized and we are now able to create a ClientTracker. To do that we get the TrackerManager from the WikitudeSDK instance and call create2dClientTracker passing the url to the WTC file. Since we are loading an asset on the device we indicate that by starting the url with the string file:///android_asset/ and add the path to the file starting from the asset root directory.

To get notified when the Tracker finished loading, recognized a target and so on we register the Activity which implements ClientTrackerEventListener as a Listener on the newly created Tracker.

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    _wikitudeSDK = new WikitudeSDK(this);
    WikitudeSDKStartupConfiguration startupConfiguration = new WikitudeSDKStartupConfiguration(WikitudeSDKConstants.WIKITUDE_SDK_KEY);
    _wikitudeSDK.onCreate(getApplicationContext(), startupConfiguration);
    ClientTracker tracker = _wikitudeSDK.getTrackerManager().create2dClientTrackerFromUrl("file:///android_asset/magazine.wtc");
    tracker.registerTrackerEventListener(this);
}

The next important method to look at is the onRenderExtensionCreated method. Since we decided to use the external rendering functionality by implementing ExternalRendering, the WikitudeSDK provides us with a RenderExtension. The RenderExtension interface exposes the same methods as the standard GLSurfaceView.Renderer. In your custom GLSurfaceView.Renderer the first thing to do in every method is to always call the same method on the provided RenderExtension. To be able to that we pass the RenderExtension to the constructor of our Renderer, create our SurfaceView, initialize a Driver and set our SurfaceView as our content view.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension_) {
    _glRenderer = new GLRenderer(renderExtension_);
    _view = new CustomSurfaceView(getApplicationContext(), _glRenderer);
    _driver = new Driver(_view, 30);
    setContentView(_view);
}

Next we will have a look at the ClientTrackerEventListener interface. The onErrorLoading method will be called like the name suggest when the Wikitude SDK wasn't able to load the client tracker. The most likely cause of this to happen would be that either the path to the WTC file was incorrect or the WTC was corrupted. The onTrackerFinishedLoading method will be called once when the tracker was successfully loaded. When the client tracker first recognizes a target it will call onTargetRecognized providing you with the recognized target name. When the client tracker starts tracking this target it will call onTracking continuously until it loses the target and finishes tracking with a call to onTargetLost.

The RecognizedTarget object provided in the onTracking method contains information about the tracked target like the name, the distance to the target and most importantly the matrices which describe where on the camera frame the target was found. Since we need this information to draw on the target we pass the object to our renderer in the 'onTracking' method and remove it when we lose the target in the onTargetLost method.

@Override
public void onErrorLoading(final ClientTracker clientTracker_, final String errorMessage_) {
    Log.v(TAG, "onErrorLoading: " + errorMessage_);
}

@Override
public void onTrackerFinishedLoading(final ClientTracker clientTracker_, final String trackerFilePath_) {

}

@Override
public void onTargetRecognized(final ClientTracker tracker_, final String targetName_) {

}

@Override
public void onTracking(final ClientTracker tracker_, final RecognizedTarget recognizedTarget_) {
    _glRenderer.setCurrentlyRecognizedTarget(recognizedTarget_);
}

@Override
public void onTargetLost(final ClientTracker tracker_, final String targetName_) {
    _glRenderer.setCurrentlyRecognizedTarget(null);
}

2D Extended Client Tracking Android

Extended tracking is an optional mode you can set for each target separately. In this mode the Wikitude SDK will try to continue to scan the environment of the user even if the original target image is not in view anymore. So the tracking extends beyond the limits of the original target image. The performance of this feature depends on various factors like computing power of the device, background texture and objects.

Based on the previous sample, to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In this sample we simply set a wildcard * so that all targets in this tracker are extended.

ClientTracker tracker = _wikitudeSDK.getTrackerManager().create2dClientTracker("file:///android_asset/magazine.wtc", new String[]{"*"});

3D Client Tracking Android

In this example we will take a look at how to use the Wikitude TrackingMapRecorder and 3D ClientTracker to enable 3D Tracking. If you haven't already done so please read through the first section on 2D Tracking before continuing here. Essential concepts like how to setup rendering or how to use a Wikitude Tracker are explained there and won't be repeated here. Similar if you don't already know why we need to record a Tracking Map or what to expect from 3D Tracking in general please read through the introduction to 3D tracking.

After we setup our activity to use ExternalRendering we will initialize the user interface in the onRenderExtensionCreated callback method. Let's look at the first lines of code below.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension_) {

    ... setup GL view ...

        showStartDialog();

    ...
}

private void showStartDialog() {
    AlertDialog.Builder builder = new AlertDialog.Builder(ClientTracking3DActivity.this);
    builder.setNegativeButton("Leave Example", new DialogInterface.OnClickListener() {
        ...
    });
    builder.setPositiveButton("Start Recording", new DialogInterface.OnClickListener() {
        public void onClick(DialogInterface dialog, int id) {
            _wikitudeSDK.getTrackingMapRecorder().startRecording();
            _trackingQualityIndicator.setVisibility(View.VISIBLE);
            _stopButton.setVisibility(View.VISIBLE);
            _stopButton.setEnabled(true);
        }
    });

    ...

    dialog.show();
}

The first thing we do in the onRenderExtensionCreated method after setting up rendering is calling the private method showStartDialog. The purpose of the start dialog is to provide the user with some information about what he needs to do and a way to start the map recording. We activate the positive button of the AlertDialog by setting the title to Start Recording and passing an OnClickListener. In this OnClickListener we start the Tracking Map recording by getting the TrackingMapRecorder from the WikitudeSDK instance and calling startRecording. We also update the UI by setting the necessary components to visible. We will talk about what those components do later in this section.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension_) {

    ... setup GL view ...
    ... show start dialog ...

    _wikitudeSDK.getTrackingMapRecorder().registerTrackingMapRecorderEventListener(new TrackingMapRecorderEventListener() {
        @Override
        public void onFinishedSavingTrackingMap(File file) {
            ...
        }

        @Override
        public void onErrorSavingTrackingMap(String errorMessage) {
            ...
        }

        @Override
        public void onTrackingMapRecordingQualityChanged(final int oldTrackingQuality, final int newTrackingQuality) {
            ...
        }

        @Override
        public void onTrackingMapRecordingCanceled() {
            ...
        }
    });

    ...

The next thing we do in the onRenderExtensionCreated method is registering a TrackingMapRecorderEventListener by calling registerTrackingMapRecorderEventListener on the TrackingMapRecorder we get from the WikitudeSDK instance. The TrackingMapRecorderEventListener interface exposes several callbacks to us. In the following we will go through the implementation of each of them one by one, starting with the onFinishedSavingTrackingMap method.

_wikitudeSDK.getTrackingMapRecorder().registerTrackingMapRecorderEventListener(new TrackingMapRecorderEventListener() {
    @Override
    public void onFinishedSavingTrackingMap(File file) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                saveMessage.setVisibility(View.INVISIBLE);
                _stopButton.setVisibility(View.INVISIBLE);
            }
        });
        _wikitudeSDK.getTrackerManager().create3dClientTracker(file.getAbsolutePath()).registerTrackerEventListener(ClientTracking3DActivity.this);
    }
    ...
}

The onFinishedSavingTrackingMap callback will be called when the TrackingMapRecorder successfully finished saving the recording to file. We will react to that fact by getting the UI ready for tracking by removing all visible UI components. After we cleared the UI we get the TrackingManager from the WikitudeSDK instance and create a new 3D ClientTracker by calling create3dClientTracker passing the path to the newly created Tracking Map file. As a result of this call the WikitudeSDK will pass back to us the ClientTracker instance it just created for us. We use it right away to register our Activity as a TrackerEventListener. The implementation of the TrackerEventListener callbacks is the same as in the first example of this chapter so we won't go into detail about it here.

_wikitudeSDK.getTrackingMapRecorder().registerTrackingMapRecorderEventListener(new TrackingMapRecorderEventListener() {

    ...

    @Override
    public void onErrorSavingTrackingMap(String errorMessage) {
        Log.v(TAG, errorMessage);
    }

    ...
}

The second callback method of the TrackingMapRecorderEventListener is the onErrorSavingTrackingMap. This method will be called by the TrackingMapRecorder if there was an error saving the Tracking Map from memory to file. This could be for several reasons but since Tracking Maps can get quite large with longer recordings the most likely one is that there is not enough storage space left on the device. In this example we just log the error message but if you are using the TrackingMapRecorder in a production app, you probably want to react on the error in some way.

_wikitudeSDK.getTrackingMapRecorder().registerTrackingMapRecorderEventListener(new TrackingMapRecorderEventListener() {

    ...

    @Override
    public void onTrackingMapRecordingQualityChanged(final int oldTrackingQuality, final int newTrackingQuality) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                _currentTrackingQuality = newTrackingQuality;
                switch (newTrackingQuality) {
                    case -1:
                        _trackingQualityIndicator.setBackgroundColor(Color.parseColor("#FF3420"));
                        _trackingQualityIndicator.setText(R.string.tracking_quality_indicator_bad);
                        break;
                    case 0:
                        _trackingQualityIndicator.setBackgroundColor(Color.parseColor("#FFD900"));
                        _trackingQualityIndicator.setText(R.string.tracking_quality_indicator_average);
                        break;
                    default:
                        _trackingQualityIndicator.setBackgroundColor(Color.parseColor("#6BFF00"));
                        _trackingQualityIndicator.setText(R.string.tracking_quality_indicator_good);
                }
            }
        });
    }

    ...
}

Since it can be hard to tell when your recording reached sufficient quality to be used for tracking, the TrackingMapRecorder will update us in the onTrackingMapRecordingQualityChanged method about the current map status. It will pass -1 if the least necessary quality isn't reached yet, 0 as soon as tracking with this map will be ok but not great and something higher than 0 when the desired quality was reached. We react to all three indications by setting the trackingQualityIndicator UI component to the appropriate state.

_wikitudeSDK.getTrackingMapRecorder().registerTrackingMapRecorderEventListener(new TrackingMapRecorderEventListener() {

    ...

    @Override
    public void onTrackingMapRecordingCanceled() {
        showStartDialog();
    }

    ...
}

The last of the TrackingMapRecorderEventListener callbacks will be called by the TrackingMapRecorder after a recording was canceled. After you canceled a recording you will need to wait till this method is called to start a new map recording. This is exactly what we do in this example by showing the initial start dialog to the user.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension_) {
    ...

    _stopButton = (Button) findViewById(R.id.client_tracking_3d_stop_recording);
    _stopButton.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            _stopButton.setVisibility(View.INVISIBLE);
            _trackingQualityIndicator.setVisibility(View.INVISIBLE);
            if (_currentTrackingQuality > 0) {
                saveMessage.setVisibility(View.VISIBLE);
                deleteTemporaryTrackingMap();
                _wikitudeSDK.getTrackingMapRecorder().stopRecording(TRACKING_MAP_FILENAME);
            } else {
                showConfirmStopDialog();
            }
        }
    });

    ...
}

After we implemented all TrackingMapRecorder callbacks the last thing we do in the onRenderExtionsionCreated method is implement the OnClickListener of the "Stop Recording" button. When the "Stop Recording" button is clicked the first thing we do is clear up the UI. Then we check if the last tracking quality indicator value was already high enough for the best possible tracking quality. If it was we show a save message to the user, clean up the last recording if there is one and stop and save the recording by getting the TrackingMapRecorder and calling stopRecording passing the name of the file the map should be stored in. Please note that you only need to pass the name of the file without any path information. All recordings will be save in external storage under Android/data/com.wikitude.nativesdksampleapp/files.

If the tracking quality indicator value indicates a bad or average recording quality we show the user a confirmation dialog, by calling the private method showConfirmStopDialog.

private void showConfirmStopDialog() {
    AlertDialog.Builder builder = new AlertDialog.Builder(ClientTracking3DActivity.this);
    builder.setNegativeButton("Restart", new DialogInterface.OnClickListener() {
        @Override
        public void onClick(DialogInterface dialog, int which) {
            _wikitudeSDK.getTrackingMapRecorder().cancelRecording();
        }
    });
    builder.setPositiveButton("Continue", new DialogInterface.OnClickListener() {
        public void onClick(DialogInterface dialog, int id) {
            deleteTemporaryTrackingMap();
            _wikitudeSDK.getTrackingMapRecorder().stopRecording(TRACKING_MAP_FILENAME);
        }
    });
    builder.setMessage("In order to experience a well working 3d tracking example, please continue recording until the quality indicator says 'Good'.")
            .setTitle("Tracking Map quality not sufficient");
    AlertDialog dialog = builder.create();
    dialog.setCancelable(false);
    dialog.show();
}

In the confirm stop dialog we give the user the option to either restart the recording or continue to save the recording. If the user decides to restart we cancel the recording by calling cancelRecording on the TrackingMapRecorder. If the user decides to use the map anyway we stop the recording like above.