Documentation

Cloud Recognition

The documentation for cloud recognition is split into two parts

  1. Documentation of the server-side component (Studio and Studio API)
  2. Documentation of the SDK side implementation, which follows below in more detail
Server-side documentation - Studio

Make sure to read the documentation about Studio and Studio API when using cloud recognition feature.

This example shows how to recognize images on a cloud server and then overlay it with augmentations utilizing the ImageTracker and CloudRecognitionService classes.

For a better understanding, here are some terms that will be used in the following and other sections of this documentation related to vision-based augmented reality.

  • Target: An image and its associated extracted data that is used to recognize an image.

  • Target Collection: A group of targets that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts of Target Collections

    • On-device Target Collection: a static wtc file containing the extracted data of your images. Can consist of up to 1,000 images.
    • Cloud Target Collection: A target collection stored on the Wikitude server. See Cloud Archive below. Can consist of up to 50,000 images.
  • Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a Target Collection and is used in combination with CloudRecognitionService .

  • CloudRecognitionService: Instead of analysing and computing the live camera feed directly on the device, the CloudRecognitionService will send the image(s) taken by the camera to the Wikitude Cloud Recognition server. The server will then do the hard work of trying to match the image with your targets in the specified cloud archive. Beside the benefit of searching in large image database, using the CloudRecognitionService has also a positive impact on the general performance in most cases. Especially when using a large target collection and on older devices.

Cloud Recognition Sample

For both Cloud Recognition samples below we will use external rendering. If you don't know what that means please go through the section on rendering before starting here.

The CloudTracker is able to run in two modes, called on-click and continuous. In On-Click mode a single recognition cycle will be executed, while in continuous mode the recognition will be run repeatedly with a variable interval. You can find both examples discussed below in our sample application under the package com.wikitude.recognition.cloud.

On-Click Cloud Recognition

We will now go through the class OnClickCloudTrackingActivity of our sample application, starting with the onCreate method. In onCreate after creating an instance of the WikitudeSDK, we obtain the TrackerManager and call createCloudRecognitionService passing our authentication token and target collection id. The third parameter is a callback that lets us know when the communication with the server was established. In case of failure, the onError callback function is called. For debugging purposes we just log the error for now, but in a real world project, were you already tested your app and know it won't be some small mistake like for example a wrong target collection id, you probably want to remove the log and for example try to load the CloudRecognitionService again. In case of success, the onInitialized callback function is called and we can proceed with creating an ImageTracker by calling createImageTracker on the TrackerManager. The first parameter should be the cloud recognition service we just created, and using the second parameter we can register the activity to receive tracking events. For this to work, we implemented ImageTrackerEventListener on our activity.

public class OnClickCloudTrackingActivity extends Activity implements ImageTrackerEventListener, ExternalRendering {
    ...
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        ...
        wikitudeSDK.onCreate(getApplicationContext(), this, startupConfiguration);
        cloudRecognitionService = wikitudeSDK.getTrackerManager().createCloudRecognitionService("b277eeadc6183ab57a83b07682b3ceba", "54e4b9fe6134bb74351b2aa3", new CloudRecognitionServiceInitializationCallback() {
            @Override
            public void onInitialized() {
                wikitudeSDK.getTrackerManager().createImageTracker(cloudRecognitionService, OnClickCloudTrackingActivity.this, null);
            }

            @Override
            public void onError(WikitudeError error) {
                Log.e(TAG, "Cloud Recognition Service failed to initialize. Reason: " + error.getMessage());
            }
        });
    }

To be able to start the recognition we define a button and set an anonymous OnClickListener which calls the CloudRecognitionService recognize method.

@Override
public void onRenderExtensionCreated(final RenderExtension renderExtension) {
    ...
    Button recognizeButton = (Button) findViewById(R.id.on_click_cloud_tracking_recognize_button);
    recognizeButton.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(final View view_) {
            cloudRecognitionService.recognize(new CloudRecognitionServiceListener() {
                ...
            }
        });
    })
}

The parameter for this function provides callback methods that allow you to react to events triggered by the CloudRecognitionService. The onResponse method is called when the connection to the cloud recognition service was successful and a response was received. This response is accessible through the CloudRecognitionServiceResponse parameter and by calling the isRecognized method on it, you can determine if a target was actually recognized by the cloud recognition service. Additional information about the target can be obtained through the getTargetInformations and getMetadata functions. The onError method is called when a connection error occurred when contacting the cloud recognition server.

cloudRecognitionService.recognize(new CloudRecognitionServiceListener() {
    @Override
    public void onResponse(final CloudRecognitionServiceResponse response) {
        if (response.isRecognized()) {
            dropDownAlert.dismiss();

            // This needs to be copied since access to the response is invalid after the end of the scope
            final String targetName = response.getTargetInformationsObject().getName();
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    EditText targetInformationTextField = findViewById(R.id.on_click_cloud_tracking_info_field);
                    targetInformationTextField.setText(targetName, TextView.BufferType.NORMAL);
                    targetInformationTextField.setVisibility(View.VISIBLE);
                }
            });
        } else {
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    EditText targetInformationTextField = findViewById(R.id.on_click_cloud_tracking_info_field);
                    targetInformationTextField.setText("Recognition failed - Please try again", TextView.BufferType.NORMAL);
                    targetInformationTextField.setVisibility(View.VISIBLE);
                }
            });
        }
    }

    @Override
    public void onError(final WikitudeError error) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                Toast.makeText(SingleCloudRecognitionActivity.this, "Recognition failed - " + error.getDescription(), Toast.LENGTH_LONG).show();
            }
        });
    }
});

ImageTrackerListener defines callbacks which enable the ImageTracker to communicate its status. On one hand we receive notifications about the loading process and on the other hand we receive updates concerning the tracking process. Let's first have a look at the methods concerning the loading process. The method onTargetsLoaded will be called, when the ImageTracker has successfully loaded and initialized targets received from the CloudRecognitionService after a request. The onErrorLoadingTargets method will be called when there was a problem loading the targets from a CloudRecognitionService response in the ImageTracker. We'll receive an errorMessage, containing more information about the problem.

@Override
public void onTargetsLoaded(ImageTracker tracker) {
    Log.v(TAG, "Image tracker loaded");
}

@Override
public void onErrorLoadingTargets(ImageTracker tracker, WikitudeError error) {
    Log.v(TAG, "Unable to load image tracker. Reason: " + error.getMessage());
}

We had a look at the loading process callbacks above. Now let's move on to the callbacks concerning the tracking process. The onImageRecognized method as well as onRecognitionSuccessful will be called when the ImageTracker starts tracking one of your targets. After the call to onImageRecognized the next calls will be to onImageTracked. On these calls the ImageTracker will pass an instance of the ImageTarget class containing information about the current target. Besides general information about the target like the name of the target and distance to the target, this object also contains matrices which define the location of the target in the current frame. We create and register a StrokedRectangle instance when an image is recognized, update its matrices using the ImageTarget object when it is tracked and unregister it when the image is lost. The scale values being set allow for the StrokedRectangle to adjust to the target's aspect ratio.

@Override
public void onImageRecognized(ImageTracker tracker, final ImageTarget target) {
    Log.v(TAG, "Recognized target " + target.getName());

    StrokedRectangle strokedRectangle = new StrokedRectangle(StrokedRectangle.Type.STANDARD);
    glRenderer.setRenderablesForKey(target.getName() + target.getUniqueId(), strokedRectangle, null);
}

@Override
public void onImageTracked(ImageTracker tracker, final ImageTarget target) {
    StrokedRectangle strokedRectangle = (StrokedRectangle)glRenderer.getRenderableForKey(target.getName() + target.getUniqueId());

    if (strokedRectangle != null) {
        strokedRectangle.projectionMatrix = target.getProjectionMatrix();
        strokedRectangle.viewMatrix = target.getViewMatrix();

        strokedRectangle.setXScale(target.getTargetScale().getX());
        strokedRectangle.setYScale(target.getTargetScale().getY());
    }
}

@Override
public void onImageLost(final ImageTracker tracker, final ImageTarget target) {
    Log.v(TAG, "Lost target " + target.getName());
    glRenderer.removeRenderablesForKey(target.getName() + target.getUniqueId());
}

Continuous Cloud Recognition

On-Click recognition is useful in some particular cases, but more often than not you probably want to use continuous recognition. For continuous cloud recognition we set an interval in which the CloudRecognitionService automatically calls the recognize function.

Similar to On-Click recognition, we define a button and set an anonymous OnClickListener which calls the CloudRecognitionService startContinuousRecognition method, passing an interval in milliseconds. Since mobile internet can be quite bad, we recommend an interval between 1500 and 2000ms. The second parameter is a listener that can be used to determine if the interval we requested is too small. If the interval time elapsed and the previous request didn't finish, the onInterruption method is called with a suggested interval that should be used in the future. The third parameter is a listener of type CloudRecognitionServiceListener and works in the same way as discussed in the On-Click section.

private void startContinuousRecognition() {
    ...
    cloudRecognitionService.startContinuousRecognition(recognitionInterval, new ContinuousCloudRecognitionServiceInterruptionListener() {
        @Override
        public void onInterruption(int preferredInterval) {
            ...
        }
    }, new CloudRecognitionServiceListener() {
        @Override
        public void onResponse(final CloudRecognitionServiceResponse response) {
            ...
        }

        @Override
        public void onError(final WikitudeError error) {
            ...
        }
    });
}