Documentation

Positionable

In combination with the Plugins API the Wikitude SDK allows for renderables defined with the JavaScript API to be positioned directly without using the built-in tracking mechanisms. It therefore allows to take advantage of the rendering capabilities of the Wikitude SDK while supplying custom tracking algorithms. This example will take you through the process of implementing such a custom algorithm and highlight the intricacies related thereto. Specifically, a marker tracking plugin is implemented using the OpenCV and ArUco libraries.

Introduction

To be able to understand the happenings of this example and utilise the AR.Positionable object, one must first understand how it is implemented in the Wikitude SDK. This section serves as a quick introduction on the topic.

Within the JavaScript API an AR.Positionable can be defined. This definition in turn invokes the instantiation of a complementary C++ object, of which a reference is provided in the updatePositionables function of the wikitude::sdk::Plugin, allowing it to be manipulated therein. A custom plugin utilising the positionable feature can therefore be implemented by deriving from said class and overriding the updatePositionables member function accordingly. After alterations have been performed by the updatePositionables function, the AR.Positionable objects are submitted for rendering each frame. Conceptually, a positionable is therefore a plugin mutable wrapper object to a renderable in the Wikitude SDK. This enables the extension of the JavaScript API though the Plugins API in a simple manner.

Prerequisites

For this example the following resources are recommended.

Plugin example

Have a look at the Plugins API example on this page if you are not familiar with it yet.

ArUco marker

If you would like to create your own ArUco markers, please refer to the utilities accompanying the ArUco library package. It can be downloaded from SourceFore.

A marker specific to the ArUco augmented reality library with ID #303.

ArUco and OpenCV documentation

If you would like to delve into the details of the tracking algorithm, the ArUco website and the OpenCV documentation pages on camera calibration and 3d reconstruction are the recommended starting points.

JavaScript implementation

Similar to the AR.ImageTrackable and AR.GeoObject, the AR.Positionable is available. It requires a string identifier and a renderable as its input parameters. For this example, an AR.Model is used. Notice that no tracker can be specified, as the tracking will be provided by the plugin instead.

var World = {
    _myPositionable: null,

    init: function initFn() {
        this.createOverlays();
    },

    createOverlays: function createOverlaysFn() {
        var myModel = new AR.Model(
            "assets/car.wt3", {
                onLoaded: this.loadingStep,
                    scale: {
                        x: 0.01,
                        y: 0.01,
                        z: 0.01
                    }
            });

        World._myPositionable = new AR.Positionable("myPositionable", {
            drawables: {
                cam: myModel
            }
        });
    }
};

World.init();

Plugin implementation

To implement a custom tracking we use the marker tracking capabilities of the ArUco library, which is based on the OpenCV library. It allows ArUco markers to be recognised within the camera frame. It additionally allows to compute their camera relative 3D position, enabling placement of the model onto the tracked marker. Although the ArUco and OpenCV libraries do most of the heavy lifting, there are quite a lot of things to be considered and done for it to work correctly. These considerations are important for most practical plugins and will be presented in the following sections.

Ultimately however, all the custom plugin has to do is set the world matrix, view matrix and projection matrix of the AR.Positionable object. How these matrices are to be set differs based on whether a 3D renderable or a 2D renderable is attached.

// transformation matrices for a 3D renderable
positionable->setWorldMatrix(identityMatrix.get());
positionable->setViewMatrix(modelViewMatrix.get());
positionable->setProjectionMatrix(projectionMatrix.get());

// transformation matrices for a 2D renderable
positionable->setWorldMatrix((projectionMatrix * modelViewMatrix).get());
positionable->setViewMatrix(identityMatrix.get());
positionable->setProjectionMatrix(identityMatrix.get());

The header file

Please see below the content of the MarkerTrackerPlugin.h file. We derive from the wikitude::sdk::ArchitectPlugin class and override the cameraFrameAvailable function.

Regarding member variables, the MarkerTracker that can be registered in _registeredMarkerTracker contains a reference to the aruco::MarkerDetector which is the main class of the aruco library and performs all the steps of the tracking algorithm..

class MarkerTrackerPlugin : public wikitude::sdk::ArchitectPlugin {
public:
    MarkerTrackerPlugin();

    /* From ArchitectPlugin */
    void initialize(const std::string& temporaryDirectory_, wikitude::sdk::PluginParameterCollection& pluginParameterCollection_) override;
    void cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) override;

    MarkerTrackerJavaScriptPluginModule* getJavaScriptPluginModule();

protected:
    void createMarkerTracker(long id_);
    void createMarkerTrackable(long id_, long markerTrackerId_, int markerId_);
    void calculateProjection(wikitude::sdk::Size<int> cameraFrameSize_);

protected:
    std::unordered_map<long, std::unique_ptr<MarkerTracker>>    _registeredMarkerTracker;
    std::unordered_map<long, std::unique_ptr<MarkerTrackable>>  _registeredMarkerTrackables;

    std::vector<int>                    _recentMarkerIDs;

    wikitude::sdk::RuntimeParameters*   _runtimeParameters;
    wikitude::sdk::CameraParameters*    _cameraParameters;

    wikitude::sdk::Matrix4              _projection;
};

The cameraFrameAvailable function

In the cameraFrameAvailable function we send the camera frame to the MarkerTracker we registered in _registeredMarkerTracker by calling processCameraFrame. There we call the _detector.detect() function which performs the marker tracking on the luminance camera frame given a set of input parameters. While most of the parameters should be self explanatory, the cameraMatrix parameter is not. It contains the data required to calculate the 3D position of the marker relative to the camera. Traditionally, the camera parameters along with distortion coefficients are precomputed by a separate camera calibration process. For the sake of this example however, the parameters are simply estimated with the specifications of the iPhone 5. While the results suffers slightly, they should suffice for this simple demonstration. Even on different devices, the application still performs well. Should this not be the case for your device, you may need to alter the focal length or CDD sensor sizes accordingly.

// calculate the focal length in pixels (fx, fy)
const float focalLengthInMillimeter = 4.12f;
const float CCDWidthInMillimeter = 4.536f;
const float CCDHeightInMillimeter = 3.416f;

const float focalLengthInPixelsX = _width * focalLengthInMillimeter / CCDWidthInMillimeter;
const float focalLengthInPixelsY = _height * focalLengthInMillimeter / CCDHeightInMillimeter;

cv::Mat cameraMatrix = cv::Mat::zeros(3, 3, CV_32F);

cameraMatrix.at<float>(0, 0) = focalLengthInPixelsX;
cameraMatrix.at<float>(1, 1) = focalLengthInPixelsY;

// calculate the frame center (cx, cy)
cameraMatrix.at<float>(0, 2) = 0.5f * _width;
cameraMatrix.at<float>(1, 2) = 0.5f * _height;

// always 1
cameraMatrix.at<float>(2, 2) = 1.0f;

const float markerSizeInMeters = 0.1f;

std::vector<aruco::Marker> markers;
_detector.detect(frameLuminance, _markers, cameraMatrix, cv::Mat(), markerSizeInMeters);

Once markers are detected, a matrix is calculated that transforms the origin into the center of the tracked marker. Note that the tracking is restricted to a specific marker ID in this case to avoid ambiguities.

double viewMatrixData[16];
for (auto& marker : _markers) {
    // consider only marker 303
    if (marker.id == 303) {
        marker.calculateExtrinsics(markerSizeInMeters, cameraMatrix, cv::Mat(), false);
        marker.glGetModelViewMatrix(viewMatrixData);
    }
}

We also have to compose a model view matrix that transforms the origin of the coordinate system into the marker center, enabling our model to be drawn on top. It is aligned such that the X-axis and Y-axis lie in the marker plane with the Z-axis being perpendicular thereto such that the positive half space is in front of the marker.

To produce this matrix several transformations have to be composed. The ArUco generated view matrix assumes a left handed coordinate system while the Wikitude SDK assumes a right handed coordinate system. To correct this discrepancy the Y-axis is flipped. As this application is intended to run on a mobile device, we need to account for the different device orientations. This is a twofold issue as is requires rotations to be applied depending on the current interface orientation and the correction of the aspect ratio for portrait orientations. Additionally, mobile devices have different screen and video capturing characteristics, therefore another corrective matrix is required to account for the aspect ratio.

wikitude::sdk::Matrix4 viewMatrix;
for (int i = 0; i < 16; ++i) {
    viewMatrixData[i] = viewMatrix[i];
}

const wikitude::sdk::Scale2D<float>& cameraToSurfaceScaling = _runtimeParameters->getCameraToSurfaceScaling();
wikitude::sdk::Matrix4 aspectRatioCorrection;
aspectRatioCorrection.scale(cameraToSurfaceScaling.x, cameraToSurfaceScaling.y, 1.0f);

for (int i = 0; i < 16; ++i) {
    viewMatrix[i] = static_cast<float>(viewMatrixData[i]);
}

// OpenCV left handed coordinate system to OpenGL right handed coordinate system
viewMatrix.scale(1.0f, -1.0f, 1.0f);


wikitude::sdk::Matrix4 modelViewMatrix;
if ( _runtimeParameters->getCameraToSurfaceAngle() == 90.f || _runtimeParameters->getCameraToSurfaceAngle() == 270.f ) { /* we need a better comparison here */
    const float aspectRatio = static_cast<float>(cameraFrameSize.width) / static_cast<float>(cameraFrameSize.height);
    wikitude::sdk::Matrix4 portraitAndUpsideDownCorrection;
    portraitAndUpsideDownCorrection.scale(aspectRatio, 1.0f / aspectRatio, 1.0f);

    modelViewMatrix *= portraitAndUpsideDownCorrection;
}

modelViewMatrix *= aspectRatioCorrection;

wikitude::sdk::Matrix4 rotation;
float rotationAngle = 360.f - _runtimeParameters->getCameraToSurfaceAngle();
modelViewMatrix *= rotation.rotateZ(rotationAngle);

modelViewMatrix *= viewMatrix;

The updatePositionables function

The updatePositionables method determines whether any markers have been newly found that were not found in the previous frame and whether any markers have been lost that were found in the previous frame. It then accordingly calls the enteredFieldOfVision and exitedFieldOfVision trigger functions, which enable use of these triggers within the JavaScript API.

void MarkerTrackerJavaScriptPluginModule::updatePositionables(const std::unordered_map<std::string, wikitude::sdk::Positionable*>& positionables_) {

    std::lock_guard<std::mutex> positionableDataReadLock(_currentPositionablesUpdateMutex);
    for ( auto& sdkPositionablePair : positionables_ ) {

        for ( auto& positionableData : _currentPositionableData ) {

            auto matchSearch = std::find_if(positionableData.begin(), positionableData.end(), [&](auto currentPair_) {
                return sdkPositionablePair.first == currentPair_.first;
            });
            if ( matchSearch != positionableData.end() ) {
                if ( matchSearch->second._state == demo::PositionableState::Recognized ) {
                    callInstance(matchSearch->second._trackableId, "_markerRecognized(" + std::to_string(matchSearch->second._marker._id) + ")");
                    sdkPositionablePair.second->enteredFieldOfVision();
                } else if ( matchSearch->second._state == demo::PositionableState::Lost ) {
                    callInstance(matchSearch->second._trackableId, "_markerLost(" + std::to_string(matchSearch->second._marker._id) + ")");
                    sdkPositionablePair.second->exitedFieldOfVision();
                }
                sdkPositionablePair.second->setViewMatrix(matchSearch->second._marker._matrix.get());
                sdkPositionablePair.second->setProjectionMatrix(_projection.get());
                sdkPositionablePair.second->setWorldMatrix(_identity.get());
            }
        }
    }
    _currentPositionableData.clear();
}

Native implementation

As the plugin instantiation and registration is covered by the Plugins API example, a detailed description on this subject is omitted here.

Running the sample with the ArUco marker provided in the resource section should present you with the car model nicely being placed on top of the marker.