Documentation

Image Recognition

Introduction

This example shows how to recognize images in the viewfinder and overlay it with images. Furthermore it shows how to recognize multiple different images and how to react on user clicks on the overlaid elements.

For a better understanding, here are some terms that will be used in the following and other section of this documentation related to augmented reality.

  • Target: A target image and its associated extracted data that is used by the tracker to recognize an image.

  • Target collection: An archive storing a collection of targets that can be recognized by the tracker. A target collection can come from two different resource types

    • Plain: A regular ZIP file containing images in plain JPG or PNG format
    • Pre-processed: Regular images that are converted into a WTC file (Wikitude Target collection) for faster processing and storing more images offline.
  • ImageTracker: The tracker analyzes the live camera image and detects the targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.

Important: Learn how to create Image Targets

Make sure to read the chapter on how to create Image Targets before using Image Recognition on your own.

Extended Recognition Range

Introduced with SDK 7.0, the Wikitude SDK Image Recognition engine can make use of HD camera frames to detect images from further away. Further away in this context means distances 3x further away, compared to not enabling this mode (e.g. A4-sized target can reach recognition distances in the area of 2.4 meters/ 8 feet). This feature is called Image Recognition Extended Range and can be activated through a setting in the ImageTracker class. The setting extendedRangeRecognition is optional and accepts the following three constants

  • ON
  • OFF
  • AUTO (default)

see ImageRecognitionRangeExtension

Processing a HD camera frame in the pipe-line is computationally more expensive - this means that this feature is only activated, if the phone can handle this additional effort without sacrificing user experience. The AUTO setting will determine that based on available CPU power and camera access (e.g. Camera2 API for Android) and is the default setting in the SDK.

Simple Image Recognition

In this section we will go through the SimpleImageTrackingPage code, which you can find in the example applications folder NativeSDKExamples\NativeSDKExamples\src\Views\ImageTracking\. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this sample even if you are only interested in cloud recognition for example.

Please refer to the Setup Guide section of this documentation to find detailed information on how to integrate the Wikitude Native SDK into a Visual Studio project.

In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering, please read through the section on rendering.

The WikitudeNativeSDK class is structured to be used in a standard XAML page and to make use of the life cycle events. We will use interfaces to communicate to the WikitudeNativeSDK which type of rendering method we would like to use and to provide the necessary callbacks for ImageTracker

To receive image tracker information about recognized/tracked or lost targets, the class must implement event listeners.

Creating the WikitudeNativeSDK and passing the renderingSystem object.

_renderingSystem = ref new ExternalRenderingSystem(device);
_sdk = ref new WikitudeNativeSDK(_renderingSystem);

Setting up the license key

Platform::String ^ licenseKey = (Platform::String ^) Application::Current->Resources->Lookup("wikitude_license_key");
_sdk->setLicenseKey(licenseKey);

Starting the SDK and loading resource file

WeakReference wr(this);
_sdk->start(ref new StartHandler([wr](bool success_, Error ^ error_) {
            SimpleImageTrackingPage ^ page = wr.Resolve<SimpleImageTrackingPage>();
            if (success_) {
                page->_resource = page->_sdk->getTrackerManager()->createTargetCollectionResource("file://./app_assets/magazine.wtc");
                page->_resource->load(ref new LoadCompletionHandler([wr](bool success_, Error ^ error_) {
                    SimpleImageTrackingPage ^ page = wr.Resolve<SimpleImageTrackingPage>();
                    if (success_) {
                        [...]
                    }
                }));
                page->_renderingSystem->surfaceChanged(page->_renderer->deviceResources()->GetOutputSize().Width, page->_renderer->deviceResources()->GetOutputSize().Height);
            }
        }));

Creating the tracker and registering the event handlers

auto imageTracker = page->_sdk->getTrackerManager()->createImageTracker(page->_resource, nullptr);
imageTracker->RecognizedTarget += ref new ImageTrackerRecognizedTargetEventHandler(page, &SimpleImageTrackingPage::imageTargetRecognized);
imageTracker->TrackedTarget += ref new ImageTrackerTrackedTargetEventHandler(page, &SimpleImageTrackingPage::imageTargetTracked);
imageTracker->LostTarget += ref new ImageTrackerLostTargetEventHandler(page, &SimpleImageTrackingPage::imageTargetLost);
page->_imageTracker = imageTracker;

Next we will have a closer look on how to get on tracker specific data. The ImageTracker provides three events that will be fired when the tracker found, lost or tracked targets.

The ImageTarget object provides information about which target changed and what changed for the target, e.g. its position which would be represented through a DirectX like model view matrix.

    void SimpleImageTrackingPage::imageTargetRecognized(wikitude::sdk::uwp::ImageTracker ^ sender_, wikitude::sdk::uwp::ImageTarget ^ target_)
    {
        dropDownAlert->dismiss();
        auto& outputSize = _renderer->deviceResources()->GetOutputSize();
        _rectangle = new StrokedRectangle(_renderer->deviceResources()->GetD3DDevice());
        _rectangle->updateVerticalFieldOfView(_sdk->getCameraManager()->getVerticalFieldOfView());
        _rectangle->updateSurfaceSize(outputSize.Width, outputSize.Height);
    }

    void SimpleImageTrackingPage::imageTargetTracked(wikitude::sdk::uwp::ImageTracker ^ sender_, wikitude::sdk::uwp::ImageTarget ^ target_)
    {
        if (_rectangle) {
            auto scaleMatrix = ref new wikitude::sdk::uwp::Matrix();
            scaleMatrix->scale(target_->getTargetScale().x, target_->getTargetScale().y, 1.f);
            auto imageMatrix = target_->getMatrix()->multiply(scaleMatrix);
            _rectangle->updateModelMatrix(imageMatrix);
        }
    }

    void SimpleImageTrackingPage::imageTargetLost(wikitude::sdk::uwp::ImageTracker ^ sender_, wikitude::sdk::uwp::ImageTarget ^ target_)
    {
        dropDownAlert->show();

        delete _rectangle;
        _rectangle = nullptr;
    }

Multiple Image Targets Tracking

The following are instructions for adding multiple target tracking to the Simple Image Recognigtion sample, which you can find in the MultipleTargetsTrackingPage sample in the NativeSDKExamples\NativeSDKExamples\src\Views\ImageTracking\ folder.

In the Simple Image Recognition sample we track only one target, but this can be easily changed. For this sample we decided to track a maximum number of five targets. Setting this maximum number is merely a performance optimization that takes effect once five targets are actually tracked as the search for additional targets can be suspended. If your use case allows for a maximum number to be set, we recommend you doing so.

First, we need to implement an event handler to receive target's ImageTarget::DistanceToTargetChanged events.

void onDistanceToTargetChanged(int distance_, wikitude::sdk::uwp::ImageTarget^ firstTarget_, wikitude::sdk::uwp::ImageTarget^ secondTarget_);

If we track multiple targets, we need multiple StrokedRectangle instances to visualize the tracking, so we get rid of _rectangle member and create a std::map of StrokedRectangles instead.

std::map<Platform::String^, StrokedRectangle*> _rectangles;

The only configuration we need to add to the Wikitude Native SDK is in OnNavigatedTo:. Here we create the image tracker mainly as we used to, but add a ImageTrackerConfiguration as the last parameter. In this configuration we tell the image tracker to set the maximum number of concurrently trackable targets to five. The other thing we have to set is the distanceChangedThreshold, a value that determines at which distance change between two targets the Wikitude Native SDK should fire the ImageTarget::DistanceToTargetChanged event.

auto configuration = ref new ImageTrackerConfiguration;
configuration->setMaximumNumberOfConcurrentlyTrackableTargets(5);
configuration->setDistanceChangedThreshold(10);

That's all for the Wikitude Native SDK configuration. Now all we need to do is to handle the rectangles. Most of the rectangle handling is just the same as in the Simple Image Tracking sample, with the only difference being that the changes have to be applied to either all of the StrokedRectangle instances or a single one that has to be fetched from the map first. We might encounter multiple instances of the same image, so we need unique keys. We create a unique key by combining target name and target Id.

When the image tracker recognizes an image, the ImageTracker::RecognizedTarget event is fired.In it, we add our ImageTarget::DistanceToTargetChanged event handler. Furthermore we create a new StrokedRectangle instance which we add to our _rectangles map with the unique key.

auto rectangle = new StrokedRectangle(_renderer->deviceResources()->GetD3DDevice());
[...]
auto targetName = target_->getName() + target_->getUniqueId();
_rectangles[targetName] = rectangle;

target_->DistanceToTargetChanged += ref new DistanceToTargetChangedHandler(this, &MultipleTargetsTrackingPage::onDistanceToTargetChanged);

As we previously defined, the callback function ImageTarget::DistanceToTargetChanged events is fired when the distance between the targets changes more than a certain threshold. For demonstration purposes we set a threshold of 10 and change the color of the rectangles to red if they get too close to each other. If both targets are instances of the same image, we set their rectangle colors to blue.

    void MultipleTargetsTrackingPage::onDistanceToTargetChanged(int distance_, ImageTarget ^ firstTarget_, ImageTarget ^ secondTarget_)
    {
        float r = 1.0f;
        float g = 0.58f;
        float b = 0.16f;

        if (distance_ < 300.0f) {
            if (firstTarget_->getName() == secondTarget_->getName()) {
                r = 0.0f;
                g = 0.0f;
                b = 1.0f;
            }
            else {
                r = 1.0f;
                g = 0.0f;
                b = 0.0f;
            }
        }

        auto firstRectangle = _rectangles[firstTarget_->getName() + firstTarget_->getUniqueId()];
        if (firstRectangle != nullptr) {
            firstRectangle->setColor(r, g, b);
        }

        auto secondRectangle = _rectangles[secondTarget_->getName() + secondTarget_->getUniqueId()];
        if (secondRectangle != nullptr) {
            secondRectangle->setColor(r, g, b);
        }
    }

Extended Image Tracking

Warning: This feature is marked as deprecated since 9.12.0 and will be removed in future releases.

Based on the Simple Image Recognition sample to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In the sample we simply set a wildcard * so that all targets in this tracker are extended.

auto configuration = ref new ImageTrackerConfiguration;
configuration->setExtendedTargets(ref new Platform::Collections::Vector<Platform::String^>{ "*" });

page->_imageTracker = page->_sdk->getTrackerManager()->createImageTracker(page->_resource, configuration);

We're also able to track the quality of the extended tracking through the ImageTracker::UpdatedExtendedTrackingQuality event to which we attach the following handler :

void ExtendedImageTrackingPage::onUpdatedExtendedTrackingQuality(wikitude::sdk::uwp::ImageTracker ^ sender_, wikitude::sdk::uwp::ImageTarget ^ target_, wikitude::sdk::uwp::ExtendedTrackingQuality oldTrackingQuality_, wikitude::sdk::uwp::ExtendedTrackingQuality newTrackingQuality_)
{
    _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
        auto resourceLoader = Windows::ApplicationModel::Resources::ResourceLoader::GetForCurrentView();
        switch (newTrackingQuality_)
        {
        case ExtendedTrackingQuality::Bad:
            trackingQualityIndicator->Background = static_cast<SolidColorBrush^>(this->Resources->Lookup("bad_red"));
            trackingQualityIndicatorText->Text = resourceLoader->GetString("TrackingQualityBad");
            break;
        case ExtendedTrackingQuality::Average:
            trackingQualityIndicator->Background = static_cast<SolidColorBrush^>(this->Resources->Lookup("avg_yellow"));
            trackingQualityIndicatorText->Text = resourceLoader->GetString("TrackingQualityAverage");
            break;
        case ExtendedTrackingQuality::Good:
            trackingQualityIndicator->Background = static_cast<SolidColorBrush^>(this->Resources->Lookup("good_green"));
            trackingQualityIndicatorText->Text = resourceLoader->GetString("TrackingQualityGood");
            break;
        }
        trackingQualityIndicator->Visibility = Windows::UI::Xaml::Visibility::Visible;
    }));
}