Plugins API

This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.

  1. About Wikitude SDK Plugins
  2. Platform Specifics
  3. Registering Plugins
  4. QR & Barcode Plugin
  5. Face Detection Plugin

About Wikitude SDK Plugins

Technically a plugin is a class, either written in C++ or Java, that is derived from the Wikitude Plugin base class. Beside lifecycle handling and options to enable and disable the plugin, the Plugin class has four main methods that you can override cameraFrameAvailable which is called each time the camera has a new frame, update which is called after each image recognition cycle as well as startRender and endRender which are called before and after the Wikitude SDK does its rendering.

The most important thing to remember when working with plugins is that they need to have a unique identifier! If the attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.

Plugin Base Class

class Plugin {
   public:
      Plugin(std::string identifier_);
      ~Plugin();
      string getIdentifier() const; // returns a unique plugin identifier
      bool processesColorCameraFrames(); // returns true if the plugins wants to process color frames instead of bw

      void setEnabled(bool enabled_);
      bool isEnabled();

      string callJavaScript(string javaScriptSnippet); // evaluates the given JavaScript snippet in the currently loaded ARchitect World context.

   protected:
      void initialize(); // called when the plugin is initially added to the Wikitude SDK
      void pause(); // called when the Wikitude SDK is paused e.g. the application state changes from active to background
      void resume(uint pausedTime_); // called when the Wikitude SDK resumes e.g. from background to active state. pausedTime represents the time in milliseconds that the plugin was not updated.
      void destroy(); // called when the plugin is removed from the Wikitude SDK

      void cameraFrameAvailable(const Frame&; cameraFrame_); // called each time the camera has a new frame
      void update(const vector<RecognizedTarget> recognizedTargets_); // called each time the Wikitude SDK renders a new frame

      void startRender(); // called before any Wikitude SDK internal rendering is done
      void endRender(); // called right after any Wikitude SDK internal rendering is done

   protected:
      string      _identifier;
      bool        _enabled;
};

With those methods in place your plugin will be able to read the full camera image for your own purpose, where the YUV image is also processed in wikitude’s computer vision engine.

Information about Recognized Targets

In case you have the wikitude SDK running with ongoing image recognition, the plugin API will populate the RecognizedTarget in the update method once an image has been recognized. The plugin can then work with class RecognizedTarget, which wraps the details of the target image in the camera view. With that you can read out the pose of the target image and use it for your purposes. Additionally the call contains the calculated distance to the recognized target

class RecognizedTarget {
   public:
      const string&    getIdentifier() const; // the identifier of the target. The identifier is defined when the target is added to a target collection
      const Mat4&      getModelViewMatrix() const; // the model view matrix that defines the transformation of the target in the camera frame (translation, rotation, scale)
      const Mat4&      getProjectionMatrix() const;
      const float      getDistanceToCamera() const; // represents the distance from the target to the camera in millimeter
};

Passing values from within the plugin to the JavaScript part of your augmented reality experience is done via the addToJavaScriptQueue() method of the Plugin class. Using this function will execute any JavaScript code in the context of your augmented reality experience.

Platform Specifics

Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin and create a std::shared_ptr from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.

To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.

Registering Plugins

The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTWikitudeNativeSDK class.

Registering a C++ Plugin

To register a C++ plugin, call the -registerPlugin: method and pass a std::shared_ptr wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.

@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];

Removing a C++ Plugin

To remove a already known C++ plugin, call either the -removePlugin: or -removeNamedPlugin: method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared when calling -registerPlugin:. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).

[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];

Barcode and QR code reader

This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.

ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.

The C++ barcode plugin is created when WTBarcodePluginViewController loads its view object.

- (void)viewDidLoad
{
    [super viewDidLoad];

    //...    
    self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
    //...

    _barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self);
}

When WTBarcodePluginViewController starts the Wikitude Native SDK using the -start:completion: method, the C++ barcode plugin is registered using the -registerPlugin: method in the completion block.

- (void)viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];

     //...

    [self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
    //...
    } completion:^(BOOL isRunning, NSError * __nonnull error) {
        if ( !isRunning ) {
            //...
        }
        else
        {
            //...
            [self.wikitudeSDK registerPlugin:_barcodePlugin];
        }
    }];
}

Please note that the plugin is only registered when the Wikitude Native SDK could actually start.

Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h file. To create the bar code plugin we derive our BarcodePlugin class from wikitude::sdk::Plugin and override initialize, destroy, cameraFrameAvailable and update. We also declare the following member variables: _worldNeedsUpdate, _image, _imageScanner and _presentingViewController. The _worldNeedsUpdate variable will later be used as an indicator if we need to display the recognized barcode, _image and _imageScanner are classes from zBar which we'll use to scan for bar codes and _presentingViewController will hold a pointer to the WTBarcodePluginViewController from which we call a method every time we recognize a new barcode or QR code.

#include <zbar.h>

#import <WikitudeNativeSDK/Plugin.h>
#import <WikitudeNativeSDK/Frame.h>
#import <WikitudeNativeSDK/RecognizedTarget.h>


@class WTBarcodePluginViewController;

class BarcodePlugin : public wikitude::sdk::Plugin {
public:
    BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight, WTBarcodePluginViewController *presentingViewController);
    virtual ~BarcodePlugin();

    virtual void initialize();
    virtual void destroy();

    virtual void cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_);
    virtual void update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_);

protected:
    int                             _worldNeedsUpdate;

    zbar::Image                     _image;
    zbar::ImageScanner              _imageScanner;

    WTBarcodePluginViewController   *_presentingViewController;
};

In the constructor we set _worldNeedsUpdate to zero indicating that there is no update necessary and initialize the zBar::Image member variable passing its constructor the width and height of the camera frame, the image format of Y800, set its data pointer to null and the data length to zero.

In the initialize method we configure the zbar::ImageScanner by calling setConfig, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.

void BarcodePlugin::initialize() {    
    _imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}

We react to the destroy event by setting the current data pointer of the zbar::Image member to null and length to zero.

void BarcodePlugin::destroy() {
    _image.set_data(nullptr, 0);
}

The last but most interesting methods are cameraFrameAvailable and update. In the cameraFrameAvailable method we set the data of our previously initialized zbar::Image member variable to the frame data we just received and the length of the data to frame width * frame height by calling set_data. We then start the scanning process by calling the scan method of our zBar::ImageScanner passing the zBar::Image member instance. The zBar::ImageScanner::scan method returns the number of detected bar codes in the image frame, we save this number in a local variable n. If n is not equal to the result of the last frame, which we saved to _worldNeedsUpdate member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was, we call the presentBarcodeResult method, passing the code content.

void BarcodePlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
    int frameWidth = cameraFrame_.getSize().width;
    int frameHeight = cameraFrame_.getSize().height;

    _image.set_data(cameraFrame_.getLuminanceData(), frameWidth * frameHeight);

    int n = _imageScanner.scan(_image);

    if ( n != _worldNeedsUpdate ) {
        if ( n ) {
            zbar::Image::SymbolIterator symbol = _image.symbol_begin();

            [_presentingViewController presentBarCodeResult:[NSString stringWithUTF8String:std::string(symbol->get_data()).c_str()]];
        }
    }

    _worldNeedsUpdate = n;
}

Back in the Objective-C class WTBarcodePluginViewController, the -presentBarCodeResult: method is implemented like the following:

- (void)presentBarCodeResult:(NSString *)barcodeScanResult
{
    if ( barcodeScanResult ) {
        dispatch_async(dispatch_get_main_queue(), ^{
            [self.barcodeLabel setText:barcodeScanResult];
        });
    }
}

Please note that the -presentBarcodeResult: method is not called from the main thread, meaning that we need to schedule any UI related calls on the main thread. This is done by executing a block asynchronously on the dispatch main queue.

To visualize the detected bar code, we use a NSLabel and set its text property to the detected bar code string.

Face Detection

This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.

The Face detection plugin example consists of the C++ classes FaceDetectionPlugin, FaceDetectionPluginConnector and the Objective-C class WTFaceDetectionPluginViewController. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces. FaceDetectionPluginConnector is used to connect the FaceDetectionPlugin class with the WTFaceDetectionPluginViewController. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude Native SDK example application.

WTFaceDetectionPluginViewController handles the face detection plugin creation and registration exactly as described in the previously bar code example. New for this example is a class method that converts UIDeviceOrientations into an int that is used by the face detection plugin to rotate the camera frame in order to always have faces in the right orientation. This class method is then used every time the device changes its orientation.

__weak typeof(self) weakSelf = self;
[[NSNotificationCenter defaultCenter] addObserverForName:UIDeviceOrientationDidChangeNotification object:[UIDevice currentDevice] queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification *note) {

        weakSelf.faceDetectionPlugin->setFlipFlag( [WTFaceDetectionPluginViewController flipFlagForDeviceOrientation:[[UIDevice currentDevice] orientation]] );
}];

Next we have a look at the FaceDetectionPlugin class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector to notify the view controller about the result. The plugin base class defines startRender and endRender, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender and again call the FaceDetectionPluginConnector instance which in turn calls the view controller. Since we do not react on the result of the Wikitude SDK image recognition we leave update blank.


... ctor/dtor ...

void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {

    wikitude::sdk::Size frameSize = cameraFrame_.getSize();
    _scaledCameraFrameWidth = cameraFrame_.getScaledWidth();
    _scaledCameraFrameHeight = cameraFrame_.getScaledHeight();

    std::memcpy(_grayFrame.data,cameraFrame_.getLuminanceData(), frameSize.height*frameSize.width*sizeof(unsigned char));

    //... Control Open CV ...        

    if ( _result.size() ) {
        convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
        _observer->faceDetected(_modelViewMatrix);
    } else {
        _observer->faceLost();
    }
}

void FaceDetectionPlugin::startRender() {
    _observer->renderDetectedFaceAugmentation();
}

void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
    /* Intentionally Left Blank */
}

//... other internally used methods ...

To render a frame around detected faces we created an instance of the StrokedRectangle class which takes care of rendering a rectangle around faces and all targets of the also active ClientTracker. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate view controller methods which we use to update the StrokedRectangle instance. If the Plugin decides it is time to render a frame around a detected face it will call -setFaceIsRecognized:atPosition:. Since the plugin will only call this method in the startRender method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.

- (void)setFaceIsRecognized:(BOOL)recognized atPosition:(const float*)modelViewMatrix
{
    self.faceDetected = recognized;
    if (recognized) {
        [self.recognizedFaceRectangle setModelViewMatrix:modelViewMatrix];
    }
}

- (void)setFaceAugmentationProjectionMatrix:(const float*)projectionMatrix
{
    [self.recognizedFaceRectangle setProjectionMatrix:projectionMatrix];
}