Documentation

Plugins API

This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.

About Wikitude SDK Plugins

A plugin is a class, or rather a set of classes, written in C++ that allows extending the functionality of the Wikitude SDK. While there is a Plugin base class that offers some of the main functionality, a plugin itself can have multiple different optional modules that allow for more complex concepts to be implemented. The following table gives a very brief overview of the distribution of responsibilities of the plugin related classes.

class purpose
Plugin Main class of a plugin implementation. Derive from this class to create your plugin. It handles the application lifecycle and main plugin functionality. It provides access to various parameters of the Wikitude SDK and provides access to the camera frame and the recognized targets. It further owns and manages all the optional plugin modules.
ImageTrackingPluginModule Optional module to allow for custom image tracking implementations. Derive from this class to implement your own image tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
InstantTrackingPluginModule Optional module to allow for custom instant tracking implementations. Derive from this class to implement your own instant tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
ObjectTrackingPluginModule Optional module to allow for custom object tracking implementations. Derive from this class to implement your own object tracking algorithm to work in conjunction with the Wikitude SDK algorithms.
CameraFrameInputPluginModule Optional module to allow for frame data to be input into the Wikitude SDK. Derive from this class to implement your own camera frame acquisition. The supplied frame data supplied can be processed and rendered by the Wikitude SDK.
DeviceIMUInputPluginModule Optional module to allow for sensor data to be input into the Wikitude SDK. Derive from this class to implement your own sensor data acquisition. The supplied sensor data supplied will be used by the Wikitude SDK for its tracking algorithms where applicable.
OpenGLESRenderingPluginModule Optional module to allow for custom OpenGL ES rendering. Available on iOS and Android.
MetalRenderingPluginModule Optional module to allow for custom Metal rendering. Only available on iOS.
DirectXRenderingPluginModule Optional module to allow for custom DirectX rendering. Only available on Windows.

Each of the optional modules can be registered by calling the corresponding function of the Plugin class.

An important thing to remember when working with plugins is that they need to have a unique identifier. If an attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.

Plugin Base Class

class Plugin {
public:
    Plugin(std::string identifier_);
    virtual ~Plugin();

    virtual void initialize(const std::string& temporaryDirectory_, PluginParameterCollection& pluginParameterCollection_);
    virtual void setSDKEdition(SDKEdition sdkEdition_);
    virtual void pause();
    virtual void resume(unsigned int pausedTime_);
    virtual void destroy();

    virtual void cameraFrameAvailable(common_code::ManagedCameraFrame& managedCameraFrame_) = 0;
    virtual void deviceRotationEventAvailable(const DeviceRotationEvent& deviceRotationEvent_);
    virtual void deviceOrientationEventAvailable(const DeviceOrientationEvent& deviceOrientationEvent_);
    virtual void prepareUpdate();
    virtual void update(const RecognizedTargetsBucket& recognizedTargetsBucket_) = 0;

    virtual const std::string& getIdentifier() const;

    virtual void setEnabled(bool enabled_);
    virtual bool isEnabled() const;

    virtual PluginType getPluginType() const;

    virtual bool canPerformTrackingOperationsAlongOtherPlugins();
    virtual bool canUpdateMultipleTrackingInterfacesSimultaneously();

    ImageTrackingPluginModule* getImageTrackingPluginModule() const;
    InstantTrackingPluginModule* getInstantTrackingPluginModule() const;
    ObjectTrackingPluginModule* getObjectTrackingPluginModule() const;

    CameraFrameInputPluginModule* getCameraFrameInputPluginModule() const;
    DeviceIMUInputPluginModule* getDeviceIMUInpputPluginModule() const;

    OpenGLESRenderingPluginModule* getOpenGLESRenderingPluginModule() const;
    MetalRenderingPluginModule* getMetalRenderingPluginModule() const;
    DirectXRenderingPluginModule* getDirectXRenderingPluginModule() const;

protected:
    void setImageTrackingPluginModule(std::unique_ptr<ImageTrackingPluginModule> imageTrackingPluginModule_);
    void setObjectTrackingPluginModule(std::unique_ptr<ObjectTrackingPluginModule> objectTrackingPluginModule_);
    void setInstantTrackingPluginModule(std::unique_ptr<InstantTrackingPluginModule> instantTrackingPluginModule_);

    void setCameraFrameInputPluginModule(std::unique_ptr<CameraFrameInputPluginModule> cameraFrameInputPluginModule_);
    void setDeviceIMUInputPluginModule(std::unique_ptr<DeviceIMUInputPluginModule> deviceIMUInputPluginModule_);

    void setOpenGLESRenderingPluginModule(std::unique_ptr<OpenGLESRenderingPluginModule> openGLESRenderingPluginModule_);
    void setMetalRenderingPluginModule(std::unique_ptr<MetalRenderingPluginModule> metalRenderingPluginModule_);
    void setDirectXRenderingPluginModule(std::unique_ptr<DirectXRenderingPluginModule> directXRenderingPluginModule_);

    void iterateEnabledPluginModules(std::function<void(PluginModule& activePluginModule_)> activePluginModuleIteratorHandle_);

protected:
    std::string     _identifier;
    bool            _enabled;

    mutable std::mutex          _pluginModuleAccessMutex;
    std::set<PluginModule*>     _availablePluginModules;

private:
    std::unique_ptr<ImageTrackingPluginModule> _imageTrackingModule;
    std::unique_ptr<InstantTrackingPluginModule> _instantTrackingModule;
    std::unique_ptr<ObjectTrackingPluginModule> _objectTrackingModule;

    std::unique_ptr<CameraFrameInputPluginModule>   _cameraFrameInputModule;
    std::unique_ptr<DeviceIMUInputPluginModule>     _deviceIMUInputPluginModule;

    std::unique_ptr<OpenGLESRenderingPluginModule> _openGlesRenderingModule;
    std::unique_ptr<MetalRenderingPluginModule> _metalRenderingModule;
    std::unique_ptr<DirectXRenderingPluginModule> _directXRenderingModule;
};

While we will not go over every function of the Plugin class and all the optional module classes, the following sections will present sample plugins that should convey most of the concepts and methods involved in creating your own plugin.

Information about Recognized Targets

If the Wikitude SDK is running with active image-, instant- or object recognition, the plugins API will populate the RecognizedTargetsBucket in the update method with the currently recognized targets. The plugin may then use the corresponding target objects to acquire data, most importantly the pose, and use it for further processing.

class RecognizedTargetsBucket {
public:
    virtual ~RecognizedTargetsBucket() = default;

    virtual const std::vector<ImageTarget*>& getImageTargets() const = 0;
    virtual const std::vector<ObjectTarget*>& getObjectTargets() const = 0;

    virtual const std::vector<InstantTarget*>& getInstantTargets() const = 0;
    virtual const std::vector<InitializationPose*>& getInitializationPoses() const = 0;

    virtual const std::vector<Plane*>& getPlanes() const = 0;
    virtual const Matrix4& getViewMatrix() const = 0;
};

Platform Specifics

To be able to use a C++ Wikitude plugin on Android, it is necessary to create a binary from the C++ code for each supported CPU architecture. To make this process as easy as possible we prepared a CMake file.

Please note that if you would like to use multiple C++ plugins in your app, you will need to package all plugins in one shared library. This is necessary because we use JNI to register C++ plugins with the Wikitude SDK and the symbol to do that has to be unique.

Android C++ Wikitude Plugin Library Build

The following CMake file shows how plugins can be added and built. The class used as Plugin has to derive from wikitude::sdk::Plugin. Note that the libarchitect.so library is not shipped with the SDK package directly, but needs to be extracted from the .aar file, either manually or automatically.

cmake_minimum_required(VERSION 3.6)

add_library(lib_architectSDK SHARED IMPORTED)
set_target_properties(lib_architectSDK PROPERTIES IMPORTED_LOCATION ${WIKITUDE_NATIVE_PATH}/${ANDROID_ABI}/libarchitect.so)

add_library(yourPlugins SHARED
    src/jniHelper.cpp
    src/JniRegistration.cpp
    src/__YOUR_PLUGIN__.cpp
)

target_include_directories(yourPlugins
    PRIVATE
    include/wikitude
    src
)

target_link_libraries(yourPlugins
    lib_architectSDK
)

This gradle snippet shows a possible way to enable automatic extraction of the Architect SDK shared library from the .aar file. It assumes in the gradle script that this code is added to be able to build and run with the java Architect SDK (as explained in Setup Guide Android API).

/* Defines the temporary path for the plugins shared library.*/
def wikitudeTmpPath = "${buildDir}/wikitude/libs"

/* Creates a new configuration to extract the shared library from the Wikitude SDK for the Plugins to link against. */
configurations { libraryExtraction }

android {
    ...

    defaultConfig {
        ...

        externalNativeBuild {
            cmake {
                arguments "-DANDROID_TOOLCHAIN=clang",
                        "-DANDROID_STL=c++_shared",
                        /* Provides the path to the extracted libarchitect.so to CMake */
                        "-DWIKITUDE_NATIVE_PATH=${wikitudeTmpPath}/jni",
                        /* Provides the path to that the built plugins library should be copied to to CMake */
                        "-DPLUGIN_OUTPUT_PATH=${project.rootDir}/plugins/src/main/jniLibs"

                cppFlags "-std=c++14 -Wno-inconsistent-missing-override"
                /* Only build for supported architectures. */
                abiFilters 'armeabi-v7a', 'arm64-v8a'
            }
        }
    }

    ...

    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
        }
    }
}

dependencies {
    ...

    /* Extract the wikitude sdk shared library from the aar. */
    libraryExtraction project(':wikitude-sdk')
}

/* Task to extract the nativesdk shared library from the aar. */
task extractNativeLibraries() {
    doFirst {
        configurations.libraryExtraction.files.each { file ->
            copy {
                from zipTree(file)
                into wikitudeTmpPath
                include "jni/**/*"
            }
        }
    }
}
tasks.whenTaskAdded {
    task ->
        if (task.name.contains("external") && !task.name.contains("Clean")) {
            /* The externalNativeBuild depends on the extraction task to be able to link properly. */
            task.dependsOn(extractNativeLibraries)
        }
}

To connect the C++ Plugin to the SDK the Java_com_wikitude_common_plugins_internal_PluginManagerInternal_createNativePlugins function has to be implemented. This function has to have three parameters(JNIEnv*, jobject and jstring) and has to return a jlongArray. The third parameter is the plugin that is being registered in case there are several plugins. The return array contains the pointers to the Plugins that should be registered as jlong. This can be seen in the JNIRegistration.cpp file in the example app code.

extern "C" JNIEXPORT jlongArray JNICALL Java_com_wikitude_common_plugins_internal_PluginManagerInternal_createNativePlugins(JNIEnv *env, jobject thisObj, jstring jPluginName) {

    env->GetJavaVM(&JavaVMResource::JAVA_VM);

    int numberOfPlugins = 1;

    jlong cPluginsArray[numberOfPlugins];

    JavaStringResource pluginName(env, jPluginName);

    if (pluginName.str == "face_detection") {
        cPluginsArray[0] = reinterpret_cast<jlong>(new FaceTrackerPlugin());
    } else if (pluginName.str == "barcode") {
        cPluginsArray[0] = reinterpret_cast<jlong>(new BarcodePlugin);
    } else if ( pluginName.str == "customcamera" ) {
        cPluginsArray[0] = reinterpret_cast<jlong>(new YUVFrameInputPlugin());
    } else if ( pluginName.str == "markertracking") {
        cPluginsArray[0] = reinterpret_cast<jlong>(new MarkerTrackerPlugin());
    } else if ( pluginName.str == "simple_input_plugin" ) {
        cPluginsArray[0] = reinterpret_cast<jlong>(new SimpleInputPlugin());
    }

    jlongArray jPluginsArray = env->NewLongArray(numberOfPlugins);
    if (jPluginsArray != nullptr) {
        env->SetLongArrayRegion(jPluginsArray, 0, numberOfPlugins, cPluginsArray);
    }

    return jPluginsArray;
}

Registering Plugins

Register C++ Plugin

To register a C++ plugin with the Wikitude SDK, call the registerNativePlugins method of the ArchitectView instance, passing the name of your plugin library. Do not add lib in front of the name or add the .so extension. Please make sure that the onCreate method of the ArchitectView was already called. The following snippet comes from the QrPluginExtension of the Wikitude SDK Example application.

@Override
public void onPostCreate() {
    architectView.registerNativePlugins("wikitudePlugins", "barcode", new ErrorCallback() {
        @Override
        public void onError(@NonNull WikitudeError error) {
            Log.e(TAG, "Plugin failed to load. Reason: " + error.getMessage());
        }
    });
}

Barcode and QR code reader

This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.

ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.

First let's have a look at the BarcodePlugin.h file. To create the bar code plugin we derive our BarcodePlugin class from the wikitude::sdk::ArchitectPlugin and BarcodeScanner::Observer classes and override cameraFrameAvailable, recognizedBarcode and lostBarcode. We also declare the following member methods and variables: getJavaScriptPluginModule() and _registeredBarcodeScanner. The getJavaScriptPluginModule() method will later be used to communicate the c++ with the js part and the _registeredBarcodeScanner variable to scan for bar codes in a camera frame.

class BarcodePlugin : public wikitude::sdk::ArchitectPlugin, public BarcodeScanner::Observer {
public:
    BarcodePlugin();
    virtual ~BarcodePlugin() = default;

    /* From ArchitectPlugin */
    void cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) override;

    /* From BarcodeScanner::Observer */
    void recognizedBarcode(long id_, const std::string& barcode_) override;
    void lostBarcode(long id_) override;

protected:
    BarcodeScannerJavaScriptPluginModule* getJavaScriptPluginModule();

protected:
    std::unordered_map<long, std::unique_ptr<BarcodeScanner>>   _registeredBarcodeScanner;
};

We now go through each method of the BarcodePlugin class, starting with the constructor. In the constructor we set a new javascript plugin module (required by the wikitude::sdk::ArchitectPlugin class) and add it as a new registered barcode scanner.

BarcodePlugin::BarcodePlugin()
:
ArchitectPlugin("com.wikitude.plugins.barcode_scanner_demo")
{
    setJavaScriptPluginModule(std::make_unique<BarcodeScannerJavaScriptPluginModule>([&](long id_) {
        _registeredBarcodeScanner.emplace(id_, std::make_unique<BarcodeScanner>(id_, *this));
    }));
}

In the cameraFrameAvailable we send a camera frame to be processed by all the registered barcode scanners we have.

void BarcodePlugin::cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) {

    for ( auto& barcodeScannerPair : _registeredBarcodeScanner ) {
        barcodeScannerPair.second->processCameraFrame(managedCameraFrame_);
    }
}

The last but most interesting methods are recognizedBarcode and lostBarcode. When we send a camera frame to be processed in cameraFrameAvailable, if the barcode scanner scans a valid barcode it calls the recognizedBarcode method which warns the js part by calling getJavaScriptPluginModule()->callBarcodeRecognizedCallback(). If a recognized barcode is not recognized anymore in the last processed frames, the scanner calls lostBarcode with the id of the barcode lost and also warns the js part by calling getJavaScriptPluginModule()->callBarcodeLostCallback.

void BarcodePlugin::recognizedBarcode(long id_, const std::string& barcode_) {
    getJavaScriptPluginModule()->callBarcodeRecognizedCallback(id_, barcode_);
}

void BarcodePlugin::lostBarcode(long id_) {
    getJavaScriptPluginModule()->callBarcodeLostCallback(id_);
}

BarcodeScannerJavaScriptPluginModule* BarcodePlugin::getJavaScriptPluginModule() {
    return dynamic_cast<BarcodeScannerJavaScriptPluginModule*>(ArchitectPlugin::getJavaScriptPluginModule());
}

Face Detection

This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.

First let's have a look at the FaceTrackerPlugin.h file. To create the face tracker plugin we derive our FaceTrackerPlugin class from the wikitude::sdk::ArchitectPlugin class and override initialize, pause and cameraFrameAvailable. We also declare the following member methods and variables: getJavaScriptPluginModule(), getOpenGLESRenderingPluginModule() and _registeredFaceTracker. The getJavaScriptPluginModule() method will later be used to communicate the c++ with the js part, the getOpenGLESRenderingPluginModule method to communicate with the rendering part and the _registeredFaceTracker variable to register the face tracker.

class FaceTrackerPlugin : public wikitude::sdk::ArchitectPlugin {
public:
    FaceTrackerPlugin();

    void initialize(const std::string& /* temporaryDirectory_ */, wikitude::sdk::PluginParameterCollection& pluginParameterCollection_) override;
    void pause() override;

    void cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) override;

    FaceTrackerJavaScriptPluginModule* getJavaScriptPluginModule();
    FaceTrackerOpenGLESRenderPluginModule* getOpenGLESRenderingPluginModule();

protected:
    std::unordered_map<long, std::unique_ptr<FaceTracker>>      _registeredFaceTracker;
};

We now go through each method of the FaceTrackerPlugin class, starting with the initialize method. Here we set a new javascript plugin module (required by the wikitude::sdk::ArchitectPlugin class) and a new face tracker opengles render plugin module.

void FaceTrackerPlugin::initialize(const std::string& temporaryDirectory_, wikitude::sdk::PluginParameterCollection& pluginParameterCollection_) {
    setJavaScriptPluginModule(std::make_unique<FaceTrackerJavaScriptPluginModule>(&pluginParameterCollection_.getRuntimeParameters(), _registeredFaceTracker, temporaryDirectory_));
    setOpenGLESRenderingPluginModule(std::make_unique<FaceTrackerOpenGLESRenderPluginModule>(&pluginParameterCollection_.getRuntimeParameters(), &pluginParameterCollection_.getCameraParameters()));
}

In the cameraFrameAvailable we send a camera frame to be processed by all the registered face tracker scanners we have. We will have to send an std::vector<wikitude::sdk::Matrix4> to the processCameraFrame as it will retrieve the necessary data to render the augmentation and then set this data in the OpenGLESRenderingPluginModule we created before in the initialize method.

void FaceTrackerPlugin::cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) {
    std::vector<wikitude::sdk::Matrix4> augmentationData;
    for ( auto& faceTrackerPair : _registeredFaceTracker ) {
        faceTrackerPair.second->processCameraFrame(managedCameraFrame_, augmentationData);
    }

    getOpenGLESRenderingPluginModule()->setAugmentationData(augmentationData);
}

The last but most interesting is the FaceTrackerOpenGLESRenderPluginModule class which will handle the rendering part of the face tracker. We have to derive the class from the wikitude::sdk::OpenGLESRenderingPluginModule and override the startRender and endRender methods. We also have to declare other methods like setAugmentationData which will be called from the FaceTrackerPlugin::cameraFrameAvailable in order to send the required data to render the augmentation.

class FaceTrackerOpenGLESRenderPluginModule : public wikitude::sdk::OpenGLESRenderingPluginModule {
public:
    FaceTrackerOpenGLESRenderPluginModule(wikitude::sdk::RuntimeParameters* runtimeParameters_, wikitude::sdk::CameraParameters* cameraParameters_);
    virtual ~FaceTrackerOpenGLESRenderPluginModule() = default;

    void startRender(wikitude::sdk::RenderableCameraFrameBucket& /* frameBucket_ */) override;
    void endRender(wikitude::sdk::RenderableCameraFrameBucket& /* frameBucket_ */) override;

    void setAugmentationData(std::vector<wikitude::sdk::Matrix4>& augmentationsData_);

    void releaseAugmentations();

protected:
    void updateRenderableAugmentations();
    void calculateProjection(float cameraToSurfaceAngle_, float cameraFrameAspectRatio_, float left, float right, float bottom, float top, float near, float far);

protected:
    std::mutex                      _updateMutex;
    wikitude::sdk::Matrix4          _projectionMatrix;
    std::vector<wikitude::sdk::Matrix4> _augmentationsData;
    std::vector<wikitude::sdk::opengl::StrokedRectangle>    _augmentations;

    wikitude::sdk::RuntimeParameters*       _runtimeParameters;
    wikitude::sdk::CameraParameters*        _cameraParameters;
};

If you are interested in the implementation details of the FaceTrackerPlugin class, you can find it in our Wikitude SDK Example Application.