Documentation

Plugins API

This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.

About Wikitude SDK Plugins

A plugin is a class, or rather a set of classes, written in C++ that allows extending the functionality of the Wikitude SDK. While there is a Plugin base class that offers some of the main functionality, a plugin itself can have multiple different optional modules that allow for more complex concepts to be implemented. The following table gives a very brief overview of the distribution of responsibilities of the plugin related classes.

class purpose
Plugin Main class of a plugin implementation. Derive from this class to create your plugin. It handles the application lifecycle and main plugin functionality. It provides access to various parameters of the Wikitude SDK and provides access to the camera frame and the recognized targets. It further owns and manages all the optional plugin modules.
ImageTrackingPluginModule Optional module to allow for custom image tracking implementations. Derive from this class to implement your own image tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
InstantTrackingPluginModule Optional module to allow for custom instant tracking implementations. Derive from this class to implement your own instant tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
ObjectTrackingPluginModule Optional module to allow for custom object tracking implementations. Derive from this class to implement your own object tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
CameraFrameInputPluginModule Optional module to allow for frame data to be input into the Wikitude SDK. Derive from this class to implement your own camera frame acquisition. The supplied frame data supplied can be processed and rendered by the Wikitude SDK.
DeviceIMUInputPluginModule Optional module to allow for sensor data to be input into the Wikitude SDK. Derive from this class to implement your own sensor data acquisition. The supplied sensor data supplied will be used by the Wikitude SDK for its tracking algorithms where applicable.
OpenGLESRenderingPluginModule Optional module to allow for custom OpenGL ES rendering. Available on iOS and Android.
MetalRenderingPluginModule Optional module to allow for custom Metal rendering. Only available on iOS.
DirectXRenderingPluginModule Optional module to allow for custom DirectX rendering. Only available on Windows.

Each of the optional modules can be registered by calling the corresponding function of the Plugin class.

An important thing to remember when working with plugins is that they need to have a unique identifier. If an attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.

Plugin Base Class

class Plugin {
public:
    Plugin(std::string identifier_);
    virtual ~Plugin();

    virtual void initialize(const std::string& temporaryDirectory_, PluginParameterCollection& pluginParameterCollection_);
    virtual void setSDKEdition(SDKEdition sdkEdition_);
    virtual void pause();
    virtual void resume(unsigned int pausedTime_);
    virtual void destroy();

    virtual void cameraFrameAvailable(common_code::ManagedCameraFrame& managedCameraFrame_) = 0;
    virtual void deviceRotationEventAvailable(const DeviceRotationEvent& deviceRotationEvent_);
    virtual void deviceOrientationEventAvailable(const DeviceOrientationEvent& deviceOrientationEvent_);
    virtual void prepareUpdate();
    virtual void update(const RecognizedTargetsBucket& recognizedTargetsBucket_) = 0;

    virtual const std::string& getIdentifier() const;

    virtual void setEnabled(bool enabled_);
    virtual bool isEnabled() const;

    virtual PluginType getPluginType() const;

    virtual bool canPerformTrackingOperationsAlongOtherPlugins();
    virtual bool canUpdateMultipleTrackingInterfacesSimultaneously();

    ImageTrackingPluginModule* getImageTrackingPluginModule() const;
    InstantTrackingPluginModule* getInstantTrackingPluginModule() const;
    ObjectTrackingPluginModule* getObjectTrackingPluginModule() const;

    CameraFrameInputPluginModule* getCameraFrameInputPluginModule() const;
    DeviceIMUInputPluginModule* getDeviceIMUInpputPluginModule() const;

    OpenGLESRenderingPluginModule* getOpenGLESRenderingPluginModule() const;
    MetalRenderingPluginModule* getMetalRenderingPluginModule() const;
    DirectXRenderingPluginModule* getDirectXRenderingPluginModule() const;

protected:
    void setImageTrackingPluginModule(std::unique_ptr<ImageTrackingPluginModule> imageTrackingPluginModule_);
    void setObjectTrackingPluginModule(std::unique_ptr<ObjectTrackingPluginModule> objectTrackingPluginModule_);
    void setInstantTrackingPluginModule(std::unique_ptr<InstantTrackingPluginModule> instantTrackingPluginModule_);

    void setCameraFrameInputPluginModule(std::unique_ptr<CameraFrameInputPluginModule> cameraFrameInputPluginModule_);
    void setDeviceIMUInputPluginModule(std::unique_ptr<DeviceIMUInputPluginModule> deviceIMUInputPluginModule_);

    void setOpenGLESRenderingPluginModule(std::unique_ptr<OpenGLESRenderingPluginModule> openGLESRenderingPluginModule_);
    void setMetalRenderingPluginModule(std::unique_ptr<MetalRenderingPluginModule> metalRenderingPluginModule_);
    void setDirectXRenderingPluginModule(std::unique_ptr<DirectXRenderingPluginModule> directXRenderingPluginModule_);

    void iterateEnabledPluginModules(std::function<void(PluginModule& activePluginModule_)> activePluginModuleIteratorHandle_);

protected:
    std::string     _identifier;
    bool            _enabled;

    mutable std::mutex          _pluginModuleAccessMutex;
    std::set<PluginModule*>     _availablePluginModules;

private:
    std::unique_ptr<ImageTrackingPluginModule> _imageTrackingModule;
    std::unique_ptr<InstantTrackingPluginModule> _instantTrackingModule;
    std::unique_ptr<ObjectTrackingPluginModule> _objectTrackingModule;

    std::unique_ptr<CameraFrameInputPluginModule>   _cameraFrameInputModule;
    std::unique_ptr<DeviceIMUInputPluginModule>     _deviceIMUInputPluginModule;

    std::unique_ptr<OpenGLESRenderingPluginModule> _openGlesRenderingModule;
    std::unique_ptr<MetalRenderingPluginModule> _metalRenderingModule;
    std::unique_ptr<DirectXRenderingPluginModule> _directXRenderingModule;
};

While we will not go over every function of the Plugin class and all the optional module classes, the following sections will present sample plugins that should convey most of the concepts and methods involved in creating your own plugin.

Information about Recognized Targets

If the Wikitude SDK is running with active image-, instant- or object recognition, the plugins API will populate the RecognizedTargetsBucket in the update method with the currently recognized targets. The plugin may then use the corresponding target objects to acquire data, most importantly the pose, and use it for further processing.

class RecognizedTargetsBucket {
public:
    virtual ~RecognizedTargetsBucket() = default;

    virtual const std::vector<ImageTarget*>& getImageTargets() const = 0;
    virtual const std::vector<ObjectTarget*>& getObjectTargets() const = 0;

    virtual const std::vector<InstantTarget*>& getInstantTargets() const = 0;
    virtual const std::vector<InitializationPose*>& getInitializationPoses() const = 0;

    virtual const std::vector<Plane*>& getPlanes() const = 0;
    virtual const Matrix4& getViewMatrix() const = 0;
};

Platform Specifics

Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin and create a std::shared_ptr from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.

To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.

Registering Plugins

The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTWikitudeNativeSDK class.

Registering a C++ Plugin

To register a C++ plugin, call the -registerPlugin: method and pass a std::shared_ptr wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.

@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];

Removing a C++ Plugin

To remove a already known C++ plugin, call either the -removePlugin: or -removeNamedPlugin: method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared when calling -registerPlugin:. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).

[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];

Barcode and QR code reader

This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.

ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.

The C++ barcode plugin is created when WTBarcodePluginViewController loads its view object.

- (void)viewDidLoad
{
    [super viewDidLoad];

    //...
    self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingConfiguration:selectedConfiguration delegate:self];
    //...

    _barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self);
}

When WTBarcodePluginViewController starts the Wikitude Native SDK using the -start:completion: method, the C++ barcode plugin is registered using the -registerPlugin: method in the completion block.

- (void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];

     //...

    [self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
    //...
    } completion:^(BOOL isRunning, NSError * __nonnull error) {
        if ( !isRunning ) {
            //...
        }
        else
        {
            //...
            [self.wikitudeSDK registerPlugin:_barcodePlugin];
        }
    }];
}

Please note that the plugin is only registered when the Wikitude Native SDK could actually start.

Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h file. To create the bar code plugin we derive our BarcodePlugin class from wikitude::sdk::Plugin and override initialize, destroy, cameraFrameAvailable and update. We also declare the following member variables: _worldNeedsUpdate, _image, _imageScanner and _presentingViewController. The _worldNeedsUpdate variable will later be used as an indicator if we need to display the recognized barcode, _image and _imageScanner are classes from zBar which we'll use to scan for bar codes and _presentingViewController will hold a pointer to the WTBarcodePluginViewController from which we call a method every time we recognize a new barcode or QR code.

#include <zbar.h>

#import <WikitudeNativeSDK/Plugin.h>
#import <WikitudeNativeSDK/ManagedCameraFrame.hpp>
#import <WikitudeNativeSDK/RecognizedTargetsBucket.h>
#import <WikitudeNativeSDK/InterfaceOrientation.hpp>

@class WTBarcodePluginViewController;

class BarcodePlugin : public wikitude::sdk::Plugin {
public:
    BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight, WTBarcodePluginViewController *presentingViewController);
    virtual ~BarcodePlugin();

    void initialize(const std::string& temporaryDirectory_, wikitude::sdk::PluginParameterCollection& pluginParameterCollection_) override;
    void destroy() override;

    void cameraFrameAvailable(wikitude::common_code::ManagedCameraFrame& cameraFrame_) override;
    void update(const wikitude::sdk::RecognizedTargetsBucket& recognizedTargetsBucket_) override;

protected:
    int                             _currentlyRecognizedBarcodes;

    zbar::Image                     _image;
    zbar::ImageScanner              _imageScanner;

    __weak WTBarcodePluginViewController   *_presentingViewController;
};

In the constructor we set _worldNeedsUpdate to zero indicating that there is no update necessary and initialize the zBar::Image member variable passing its constructor the width and height of the camera frame, the image format of Y800, set its data pointer to null and the data length to zero.

In the initialize method we configure the zbar::ImageScanner by calling setConfig, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.

void BarcodePlugin::initialize(const std::string& /* temporaryDirectory_ */, wikitude::sdk::PluginParameterCollection& /* pluginParameterCollection_ */) {
    _imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}

We react to the destroy event by setting the current data pointer of the zbar::Image member to null and length to zero.

void BarcodePlugin::destroy() {
    _image.set_data(nullptr, 0);
}

The last but most interesting method is cameraFrameAvailable. In this method we set the data of our previously initialized zbar::Image member variable to the frame data we just received and the length of the data to the corresponding size by calling set_data. We then start the scanning process by calling the scan method of our zBar::ImageScanner passing the zBar::Image member instance. The zBar::ImageScanner::scan method returns the number of detected bar codes in the image frame, we save this number in a local variable n. If n is not equal to the result of the last frame, which we saved to _worldNeedsUpdate member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was, we call the presentBarcodeResult method, passing the code content.

void BarcodePlugin::cameraFrameAvailable(wikitude::common_code::ManagedCameraFrame& cameraFrame_) {
    const wikitude::sdk::CameraFramePlane& luminancePlane = cameraFrame_.get()[0];
    _image.set_data(luminancePlane.getData(), luminancePlane.getDataSize());

    int numberOfRecognizedCodes = _imageScanner.scan(_image);
    if ( numberOfRecognizedCodes != _currentlyRecognizedBarcodes ) {
        if ( numberOfRecognizedCodes ) {
            zbar::Image::SymbolIterator symbol = _image.symbol_begin();
            [_presentingViewController presentBarCodeResult:[NSString stringWithUTF8String:std::string(symbol->get_data()).c_str()]];
        } else {
            [_presentingViewController presentBarCodeResult:nil];
        }
    }
    _currentlyRecognizedBarcodes = numberOfRecognizedCodes;
}

Back in the Objective-C class WTBarcodePluginViewController, the -presentBarCodeResult: method is implemented like the following:

- (void)presentBarCodeResult:(NSString *)barcodeScanResult
{
    NSString *uiBarcodePrompt = nil;
    if ( barcodeScanResult )
    {
        uiBarcodePrompt = [NSString stringWithFormat:@"Found '%@'", barcodeScanResult];
    }
    else
    {
        uiBarcodePrompt = @"Scanning for barcodes...";
    }

    dispatch_async(dispatch_get_main_queue(), ^{
        [self.navigationItem setPrompt:uiBarcodePrompt];
    });
}

Please note that the -presentBarcodeResult: method is not called from the main thread, meaning that we need to schedule any UI related calls on the main thread. This is done by executing a block asynchronously on the dispatch main queue.

To visualize the detected bar code, we use a NSLabel and set its text property to the detected bar code string.

Face Detection

This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.

The Face detection plugin example consists of the C++ classes FaceDetectionPlugin, FaceDetectionPluginConnector and the Objective-C class WTFaceDetectionPluginViewController. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces. FaceDetectionPluginConnector is used to connect the FaceDetectionPlugin class with the WTFaceDetectionPluginViewController. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude Native SDK example application.

WTFaceDetectionPluginViewController handles the face detection plugin creation and registration exactly as described in the previously bar code example.

A novelty in this example is a dependence on the camera frame orientation due to the face detection algorithm expecting faces to be upright. We therefore accept the camera-to-surface angle as an input from the Wikitude SDK in order to rotate the camera frame accordingly.

void FaceDetectionPlugin::initialize(const std::string& temporaryDirectory_, wikitude::sdk::PluginParameterCollection& pluginParameterCollection_) {
    wikitude::sdk::RuntimeParameters& runtimeParameters = pluginParameterCollection_.getRuntimeParameters();
    runtimeParameters.addCameraToSurfaceAngleChangedHandler(reinterpret_cast<std::uintptr_t>(this), std::bind(&FaceDetectionPlugin::cameraToSurfaceAngleChanged, this, std::placeholders::_1));

    _runtimeParameters = &runtimeParameters;
}
void FaceDetectionPlugin::cameraToSurfaceAngleChanged(float cameraToSurfaceAngle_) {
    { // auto release scope
        std::unique_lock<std::mutex>(_cameraToSurfaceAngleMutex);

        _cameraToSurfaceAngle = cameraToSurfaceAngle_;
    }

    calculateProjection(_cameraToSurfaceAngle, -1.f, 1.f, -1.f, 1.f, 0.f, 500.f);
    _observer.projectionMatrixChanged(_projectionMatrix);
}

Next we have a look at the FaceDetectionPlugin class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector to notify the view controller about the result.

if (!_isDatabaseLoaded) {
    _isDatabaseLoaded = _cascadeDetector.load(_databasePath);
    if (!_isDatabaseLoaded) {
        return;
    }
}
wikitude::sdk::Size<int> frameSize = cameraFrame_.getColorMetadata().getPixelSize();

// only one plane that contains luminance and chrominance
const unsigned luminanceDataSize = cameraFrame_.get()[0].getDataSize() * 2 / 3;

std::memcpy(_grayFrame.data, cameraFrame_.get()[0].getData(), luminanceDataSize);

cv::Mat smallImg = cv::Mat(frameSize.height * 0.5f, frameSize.width * 0.5f, CV_8UC1);


cv::resize(_grayFrame, smallImg, smallImg.size(), CV_INTER_AREA);

/* Depending on the device orientation, the camera frame needs to be rotated in order to detect faces in it */

float currentCameraToSurfaceAngle;
{ // auto release scope
    std::unique_lock<std::mutex>(_cameraToSurfaceAngleMutex);

    currentCameraToSurfaceAngle = _cameraToSurfaceAngle;
}

if (currentCameraToSurfaceAngle == 90) {
    cv::transpose(smallImg, smallImg);
    cv::flip(smallImg, smallImg, 1);
} else if (currentCameraToSurfaceAngle == 180) {
    cv::transpose(smallImg, smallImg);
    cv::flip(smallImg, smallImg, 0);
} else if (currentCameraToSurfaceAngle == 270) {
    cv::flip(smallImg, smallImg, -1);
} else if (currentCameraToSurfaceAngle == 0) {
    // nop for landscape right
}

cv::Rect crop = cv::Rect(smallImg.cols / 4, smallImg.rows / 4, smallImg.cols / 2, smallImg.rows / 2);

cv::Mat croppedImg = smallImg(crop);


_result.clear();
_cascadeDetector.detectMultiScale(croppedImg, _result, 1.1, 2, 0, cv::Size(20, 20));

if (_result.size()) {
    convertFaceRectToModelViewMatrix(croppedImg, _result.at(0), currentCameraToSurfaceAngle);
    _observer.faceDetected(_modelViewMatrix);
} else {
    _observer.faceLost();
}

To render a frame around detected faces we created an instance of the StrokedRectangle class which takes care of rendering a rectangle around faces and all targets of the also active ImageTracker. When the plugin detects, loses or recalculated the projection matrix it will call the appropriate view controller methods which we use to update the StrokedRectangle instance. If the Plugin decides it is time to render a frame around a detected face it will call -setFaceIsRecognized:atPosition:. Since the plugin will only call this method in the startRender method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.

- (void)setFaceIsRecognized:(BOOL)recognized atPosition:(const float*)modelViewMatrix
{
    self.faceDetected = recognized;
    NSString *faceDetectionInstruction = nil;
    if (recognized) {
        [self.recognizedFaceRectangle setModelViewMatrix:modelViewMatrix];
    }
    else
    {
        faceDetectionInstruction = @"Scanning for faces...";
    }
    dispatch_async(dispatch_get_main_queue(), ^{
        [self.navigationItem setPrompt:faceDetectionInstruction];
    });
}

- (void)setFaceAugmentationProjectionMatrix:(const float*)projectionMatrix
{
    [self.recognizedFaceRectangle setProjectionMatrix:projectionMatrix];
}