Documentation

Instant Tracking

The following sections detail the instant tracking feature of the Wikitude Native SDK by introducing a minimal implementation, showcasing the simplicity the Wikitude Native SDK provides.

Introduction

Instant tracking is an algorithm that, contrary to those previously introduced in the Wikitude SDK, does not aim to recognize a predefined target and start the tracking procedure thereafter, but immediately start tracking in an arbitrary environment. This enables very specific use cases to be implemented.

The algorithm works in two distinct states; the first of which is the initialization state. In this state the user is required to define the origin of the tracking procedure by simply pointing the device and thereby aligning an indicator. Once the alignment is found to be satisfactory by the user (which the users needs to actively confirm), a transition to the tracking state is performed. In this state, the environment is being tracked, which allows for augmentations to be placed within the scene.

Initialization State and Tracking State

The instant tracking algorithm requires another input value to be provided in the initialization state. Specifically, the height of the tracking device above ground is required in order to accurately adjust the scale of augmentations within the scene. To this end, the example features a range input element that allows the height to be set in meters.

During the initialization, another parameter can be set which influences the alignment of the instant tracking ground plane. This ground plane is represented by the initialization indicator and can be rotated in order to start instant tracking at e.g. a wall instead of the floor. Please refer to the InstantTrackerConfiguration reference for detailed information.

Basic Instant Tracking

Simple Instant Tracking sample code can be found in the InstantTrackingPage.* files inside the NativeSDKExamples\NativeSDKExamples\src\Views\InstantTracking\ folder.

The Instant Tracking example provides a minimal implementation of the instant tracking algorithm. In order to use instant tracking, the code has to conform to implement new event handlers for the InstantTracker object, store this tracker as a member as well as its state.

void ChangedState(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::InstantTrackingState state_);
void StartedTracking(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::InstantTarget ^ target_);
void StoppedTracking(wikitude::sdk::uwp::InstantTracker ^ tracker_);
void ChangedInitializationPose(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::InitializationPose ^ pose);
void DidTrack(wikitude::sdk::uwp::InstantTracker ^ tracker, wikitude::sdk::uwp::InstantTarget ^ target_);
wikitude::sdk::uwp::InstantTracker ^ _instantTracker;
wikitude::sdk::uwp::InstantTrackingState _currentTrackingState;

An InstantTracker can, minimally, be instantiated with just the previously generated tracker instance, although supplying drawables to be rendered in both the initialization state and the tracking state is advisable for any practical use case. Therefore a StrokedRectangle instance is generated.

_rectangle = new StrokedRectangle(_renderer->deviceResources()->GetD3DDevice());
_rectangle->updateVerticalFieldOfView(cameraManager->getVerticalFieldOfView());
_rectangle->updateSurfaceSize(outputSize.Width, outputSize.Height);

To use our StrokedRectangle for the initialization of InstantTracker, we implement the InstantTracker::ChangedInitializationPose event handler.

void InstantTrackingPage::ChangedInitializationPose(InstantTracker ^ tracker_, InitializationPose ^ pose) {
    StrokedRectangle* strokedRectangle = getRectangle();

    strokedRectangle->updateModelMatrix(pose->getMatrix());

    if (_instantTracker->canStartTracking()) {
        strokedRectangle->setColor(0.f, 1.f, 0.f);
    } else {
        strokedRectangle->setColor(1.f, 0.f, 0.f);
    }
}

This event handler supplies a InstantTracker and the current InitializationPose, the latter of which can be used to set the projection model view matrix of the StrokedRectangle so it is displayed properly during the initialization state.

Next we need a way to transition from one state to the other. For this task we attach an event hander to the Click event of the changeStateButton. Depending on the current value of _currentTrackingState we call InstantTracker::setActiveInstantTrackingState with InstantTrackingState::Initializing or InstantTrackingState::Tracking parameter.

void InstantTrackingPage::changeStateButton_Click(Platform::Object ^ sender, Windows::UI::Xaml::RoutedEventArgs ^ e) {
    if (_instantTracker) {
        switch (_currentTrackingState) {
            case InstantTrackingState::Tracking:
                _instantTracker->setActiveInstantTrackingState(InstantTrackingState::Initializing);
                break;
            case InstantTrackingState::Initializing:
                _instantTracker->setActiveInstantTrackingState(InstantTrackingState::Tracking);
                break;
        }
    }
}

We now draw a green rectangle while the instant tracker is initializing, but we want the rectangle to be orange once we're in the tracking state.

void InstantTrackingPage::ChangedState(InstantTracker ^ tracker_, InstantTrackingState state_) {
    _currentTrackingState = state_;
    _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
        if (state_ == InstantTrackingState::Tracking) {
            changeStateButton->Content = StartInitString;
            _rectangle->setColor(1.0f, 0.58f, 0.16f);
        } else {
            changeStateButton->Content = StartTrackingString;
        }
    }));
}

In the InstantTracker::ChangedState event handler, we control the color of the StrokedRectangle for a better visual representation. However this only changes the color of the rectangle and not its position, which is why we need the InstantTracker:DidTrack event handler to update the model view matrix of the rectangle.

void InstantTrackingPage::DidTrack(InstantTracker ^ tracker, InstantTarget ^ target_) {
    StrokedRectangle* strokedRectangle = getRectangle();
    strokedRectangle->updateModelMatrix(target_->getMatrix());
}

Next, we provide the heightSlider_valueChaned event handler to call the InstantTracker::setDeviceHeightAboveGround method and connect it to a Slider. While this change is, strictly speaking, not required, we strongly recommend every application to supply the device height accurately by this method or another for the Wikitude SDK to provide an accurate scale.

void InstantTrackingPage::heightSlider_valueChaned(Platform::Object ^ sender, Windows::UI::Xaml::Controls::Primitives::RangeBaseValueChangedEventArgs ^ e) {
    float height = e->NewValue;
    _instantTracker->setDeviceHeightAboveGround(height);
    std::wstringstream wss;
    wss << std::fixed << std::setprecision(2) << height;
    heightTextBlock->Text = ref new Platform::String(wss.str().data());
}

Lastly we have to make the height slider invisible when the state changes from initialization to tracking, so set its alpha value correctly in InstantTracker:ChangedState.

_dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
        if (state_ == InstantTrackingState::Tracking) {
            heightSettingsGrid->Visibility = Windows::UI::Xaml::Visibility::Collapsed;
            changeStateButton->Content = StartInitString;
            _rectangle->setColor(1.0f, 0.58f, 0.16f);
        } else {
            heightSettingsGrid->Visibility = Windows::UI::Xaml::Visibility::Visible;
            changeStateButton->Content = StartTrackingString;
        }
    }));

The example outlined in this section renders an green rectangle augmentation when in initialization state as its indicator and a corresponding green rectangle augmentation when in tracking state. While the example is quite trivial, we believe it serves the purpose of familiarizing the reader with the core concepts of instant tracking well.

Initialization State and Tracking State

Instant Scene Picking

Scene Picking sample code can be found in the ScenePickingPage.* files inside the NativeSDKExamples\NativeSDKExamples\src\Views\InstantTracking\ folder.

The instant tracking feature further allows for 3D points to be queried from the underlying point cloud structure. This section is concerned with showcasing this feature based on the corresponding sample of the sample application.

To use this feature a 2D input position on the screen is required. A Tapped event handler is added to the SwapChainPanel for that purpose.

The swapChainPanel_Tapped event handler is implemented to query the tap position, scale it using the main screen scale to convert from points to pixels and pass the resulting coordinates to the InstantTracker::convertScreenCoordinate method. These input coordinates have their origin in the top-left corner of the screen and are within the interval [0, screen_width_in_pixels) and [0, screen_height_in_pixels). Within the completion handler function, a boolean value and a 3D point is received. The former informs on whether the operation completed successfully or not; the latter contains the result in case of success, nullptr in case of failure. Note that the query will fail whenever no point cloud position can be found for the input coordinate within a specific interval. For successful queries a StrokedCube instance is generated and placed at the resulting position as an augmentation.

auto pos = e->GetPosition(dynamic_cast<Windows::UI::Xaml::UIElement^>(sender));
wikitude::sdk::uwp::Pointf point;
point.x = pos.X;
point.y = pos.Y;
_instantTracker->convertScreenCoordinate(point, ref new ScreenCoordinateConversionHandler([this](InstantTracker^ instantTracker_, bool sucess_, wikitude::sdk::uwp::Point3Df point3Df_) {
    if (sucess_) {
        auto cameraManager = _sdk->getCameraManager();

        auto cube = new StrokedCube(_renderer->deviceResources());
        cube->setScale(0.05f, 0.05f, 0.05f);
        cube->setTranslation(point3Df_.x, point3Df_.y, point3Df_.z);
        cube->updateVerticalFieldOfView(cameraManager->getVerticalFieldOfView());
        _cubes.push_back(cube);
    }
    else {
        _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
            _notification->setText("No point found at the touched position.");
        }));
    }
}));

Finally, running the sample allows cube augmentations to be placed on screen touch when in tracking state.

instant scene picking on the floor of the Wikitude offices

Persistent Instant Targets

The save and load instant targets feature allows for AR experiences to be persistently accessed by multiple users across devices and operating systems. Furthermore instant targets can be expanded on the fly. This section is concerned with showcasing this feature based on the corresponding samples of the sample application.

Save Instant Target

To save an instant target there has to be an active InstantTracker in the tracking state and the directories of the provided path have to exist. Use the InstantTracker::saveCurrentInstantTarget method to actually perform the save operation. The Wikitude SDK would overwrite any existing file if one with the given name currently exists. To provide rich user feedback, the Wikitude SDK API offers both, a success and failure handler to update/notify the user about potential failure or success of the save operation.

Platform::String^ localfolder = Windows::Storage::ApplicationData::Current->LocalFolder->Path;
_instantTracker->saveCurrentInstantTarget(localfolder + "\\savedTarget.wto",
    ref new wikitude::sdk::uwp::InstantTrackerSaveSuccessHandler([=](InstantTracker^ sender_, Platform::String^ instantTargetPath_) {
        _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
            _notification->setText("Saving was successful.");
        }));
    }),
    ref new wikitude::sdk::uwp::InstantTrackerSaveErrorHandler([=](InstantTracker^ sender_, Error^ error_) {
        _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
            _notification->setText("Saving failed: " + error_->getMessage());
        }));
    })
);

Load Instant Target

To load an instant target there has to be an active tracker and a previously saved instant target.

The InstantTargetRestorationConfiguration defines the behaviour of the loaded instant target. In the example the policy is set to InstantTargetExpansionPolicy::AllowExpansion which means that the Wikitude SDK will try to find new points and add them to the target, making tracking more robust over time.

auto instantTargetRestorationConfiguration = ref new InstantTargetRestorationConfiguration();
instantTargetRestorationConfiguration->setInstantTargetExpansionPolicy(InstantTargetExpansionPolicy::AllowExpansion);

To load an previously saved instant target, a TargetCollectionResource has to be created from the file saved by InstantTracker::saveCurrentInstantTarget method.

Platform::String^ localfolder = Windows::Storage::ApplicationData::Current->LocalFolder->Path;
auto savedTarget = _sdk->getTrackerManager()->createTargetCollectionResource("file://" + localfolder + "\\savedTarget.wto");

After the instant target is loaded the tracker will immediately try to find and track it.

The following listing shows the complete implementation of a potential load instant target implementation. Please note again the success and error handler that can be used to provide user feedback.

Platform::String^ localfolder = Windows::Storage::ApplicationData::Current->LocalFolder->Path;
auto savedTarget = _sdk->getTrackerManager()->createTargetCollectionResource("file://" + localfolder + "\\savedTarget.wto");

auto instantTargetRestorationConfiguration = ref new InstantTargetRestorationConfiguration();
instantTargetRestorationConfiguration->setInstantTargetExpansionPolicy(InstantTargetExpansionPolicy::AllowExpansion);
_instantTracker->loadExistingInstantTarget(savedTarget,instantTargetRestorationConfiguration,
    ref new wikitude::sdk::uwp::InstantTrackerLoadSuccessHandler([=](InstantTracker^ sender_, Platform::String^ instantTargetPath_) {
    _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
        _notification->setText("Loading was successful.");
    }));
}),
    ref new wikitude::sdk::uwp::InstantTrackerLoadErrorHandler([=](InstantTracker^ sender_, Error^ error_) {
    _dispatcher->RunAsync(CoreDispatcherPriority::Normal, ref new DispatchedHandler([=]() {
        _notification->setText("Loading failed: " + error_->getMessage());
    }));
}));

Plane Detection - Experimental

Once Plane detection is enabled, the Wikitude SDK will search for planar surfaces in the representation of the scene (the point cloud). Sufficient and accurate information will then produce planar surfaces, that developers can use to interact with the actual scene. Hit testing will use found planes for a better estimate of depth values in the scene. The algorithm can detect different types of planes

  • Horizontal Up (e.g. floor, carpet, table)
  • Horizontal Down (e.g. ceiling)
  • Vertical (walls,...)
  • Arbitrary (ramps, staircases,...)

Developers can choose which type of planes they are interested in and filter correspondingly.

To use plane detection the InstantTracker has to be created with a InstantTrackerConfiguration that enables plane detection. For this purpose InstantTrackerConfiguration::enablePlaneDetection method has to be called. The given PlaneDetectionConfiguration object can then be used to further define how plane detection should run.

auto planeDetectionConfiguration = ref new PlaneDetectionConfiguration();
planeDetectionConfiguration->setPlaneFilter(PlaneFilter::Any);
planeDetectionConfiguration->enableConvexHull();

auto trackerConfiguration = ref new InstantTrackerConfiguration();
trackerConfiguration->enablePlaneDetection(planeDetectionConfiguration);

InstantTracker provides events that inform about new, updated and lost planes.

The following snippets demonstrate the implementation of those event handlers.

When a plane is recognized, a new object is created that represents a plane for rendering. It's stored in a map so that it can be referenced later to update its matrices.

void NativeSDKExamples::PlaneDetectionPage::PlaneRecognized(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::Plane ^ plane_)
{
    auto planePolygon = new PlanePolygon(_renderer->deviceResources());
    planePolygon->setPoints(to_vector(plane_->getConvexHull()));
    planePolygon->updateModelMatrix(plane_->getMatrix());
    auto cameraManager = _sdk->getCameraManager();
    planePolygon->updateVerticalFieldOfView(cameraManager->getVerticalFieldOfView());
    switch (plane_->getPlaneType()) {
    case PlaneType::HorizontalUpward:
        planePolygon->setColor(0.f, 0.f, 1.f);
        break;
    case PlaneType::Vertical:
        planePolygon->setColor(0.93f, 0.13f, 0.78f);
        break;
    case PlaneType::Arbitrary:
        planePolygon->setColor(0.25f, 0.93f, 0.13f);
        break;
    }
    _planes["plane-" + plane_->getUniqueId()] = planePolygon;
}

Every time a plane is tracked again, it's visualization counterpart is updated with the new matrix.

void NativeSDKExamples::PlaneDetectionPage::PlaneTracked(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::Plane ^ plane_)
{
    auto planePolygon = _planes["plane-" + plane_->getUniqueId()];
    planePolygon->updateModelMatrix(plane_->getMatrix());
    planePolygon->setPoints(to_vector(plane_->getConvexHull()));
}

In case the Wikitude SDK is longer able to track a plane, LostPlane event is fired with the plane object as parameter.

void NativeSDKExamples::PlaneDetectionPage::PlaneLost(wikitude::sdk::uwp::InstantTracker ^ tracker_, wikitude::sdk::uwp::Plane ^ plane_)
{
    delete _planes["plane-" + plane_->getUniqueId()];
    _planes.erase("plane-" + plane_->getUniqueId());
}

A Plane can be rendered like all other tracking targets but in addition to the matrix and extents it also provides a convex hull which can be rendered directly as a triangle fan. This can be seen in the InstantTracker::RecognizedPlane event handler listed above.

Scene picking on two tracked planes, one horizontal and one vertical.