--- Android --- Android Javascript API Android Native API --- iOS --- iOS Javascript API iOS Native API --- Extensions --- Cordova Titanium Unity Xamarin --- Smart Glasses --- Epson ODG Vuzix --- Services --- Cloud Recognition

Wikitude SDK Android Native API

Plugins API

This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.

About Wikitude SDK Plugins

Technically a plugin is a class, either written in C++ or Java, that is derived from the Wikitude Plugin base class. Beside lifecycle handling and options to enable and disable the plugin, the Plugin class has four main methods that you can override cameraFrameAvailable which is called each time the camera has a new frame, update which is called after each image recognition cycle as well as startRender and endRender which are called before and after the Wikitude SDK does its rendering.

The most important thing to remember when working with plugins is that they need to have a unique identifier! If the attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.

Plugin Base Class

class Plugin {
   public:
      Plugin(std::string identifier_);
      ~Plugin();
      string getIdentifier() const; // returns a unique plugin identifier
      bool processesColorCameraFrames(); // returns true if the plugins wants to process color frames instead of bw

      void setEnabled(bool enabled_);
      bool isEnabled();

      string callJavaScript(string javaScriptSnippet); // evaluates the given JavaScript snippet in the currently loaded ARchitect World context.

   protected:
      void initialize(); // called when the plugin is initially added to the Wikitude SDK
      void pause(); // called when the Wikitude SDK is paused e.g. the application state changes from active to background
      void resume(uint pausedTime_); // called when the Wikitude SDK resumes e.g. from background to active state. pausedTime represents the time in milliseconds that the plugin was not updated.
      void destroy(); // called when the plugin is removed from the Wikitude SDK

      void cameraFrameAvailable(const Frame&; cameraFrame_); // called each time the camera has a new frame
      void update(const vector<RecognizedTarget> recognizedTargets_); // called each time the Wikitude SDK renders a new frame

      void startRender(); // called before any Wikitude SDK internal rendering is done
      void endRender(); // called right after any Wikitude SDK internal rendering is done

   protected:
      string      _identifier;
      bool        _enabled;
};

With those methods in place your plugin will be able to read the full camera image for your own purpose, where the YUV image is also processed in wikitude’s computer vision engine.

Information about Recognized Targets

In case you have the wikitude SDK running with ongoing image recognition, the plugin API will populate the RecognizedTarget in the update method once an image has been recognized. The plugin can then work with class RecognizedTarget, which wraps the details of the target image in the camera view. With that you can read out the pose of the target image and use it for your purposes. Additionally the call contains the calculated distance to the recognized target

class RecognizedTarget {
   public:
      const string&    getIdentifier() const; // the identifier of the target. The identifier is defined when the target is added to a target collection
      const Mat4&      getModelViewMatrix() const; // the model view matrix that defines the transformation of the target in the camera frame (translation, rotation, scale)
      const Mat4&      getProjectionMatrix() const;
      const float      getDistanceToCamera() const; // represents the distance from the target to the camera in millimeter
};

Platform Specifics

To be able to use a C++ Wikitude plugin on Android, it is necessary to create a binary from the C++ code for each supported CPU architecture. To make this process as easy as possible we prepared an Android NDK make file and and some template code which passes your plugin to the Wikitude SDK. In the following section we discuss how you need to adapt these templates so they'll work for your plugin.

Please note that if you would like to use multiple C++ plugins in your app, you will need to package all plugins in one shared library. This is necessary because we use JNI to register C++ plugins with the Wikitude SDK and the symbol to do that has to be unique.

Android C++ Wikitude Plugin Library Build

All files needed are located under the folder PluginBuilder in the Wikitude SDK Android package. If you didn't setup the Android NDK yet, please follow the official guide.

Let's take a look at the Android.mk file, located under SDKPackageRoot/PluginBuilder/jni. The first thing we do is declare a variable containing the path to the source files relative to the make files location and set LOCAL_PATH to this location. We define where all include files are located and which files need to be compiled. Since our Example Plugin uses Android log we link android native log.

LOCAL_PATH := $(call my-dir)/..
SRC_DIR := $(LOCAL_PATH)/src

include $(CLEAR_VARS)

LOCAL_PATH := $(SRC_DIR)
include $(CLEAR_VARS)

LOCAL_MODULE := samplePlugin

LOCAL_C_INCLUDES := $(SRC_DIR)
LOCAL_SRC_FILES := __YOUR_PLUGIN__.cpp JniRegistration.cpp Plugin.cpp

LOCAL_LDLIBS += -llog

include $(BUILD_SHARED_LIBRARY)

The PluginLoader/src folder contains various src files which are needed so your plugin will compile and link correctly. Please don't modify any of them except the __YOUR_PLUGIN__.h and __YOUR_PLUGIN__.cpp. One other file that needs some slight modifications is JniRegistration.cpp shown below. Adapt the include directive and the constructor call to your plugin name and if you would like to use multiple C++ plugins, feel free to add more plugins to the cPluginsArray array but adjust the numberOfPlugins count accordingly.

If you packaged multiple plugins in one shared library but would like to instantiate only a subset of those plugins you can pass an identifier to this method when loading the library from Java. You can then decide which plugins to create depending on the value of jPluginName.

#include <jni.h>

#include "Plugin.h"
#include "__YOUR_PLUGIN__.h"

extern "C" JNIEXPORT jlongArray JNICALL Java_com_wikitude_common_plugins_internal_PluginManagerInternal_createNativePlugins(JNIEnv *env, jobject thisObj, jstring jPluginName) {

    int numberOfPlugins = 1;

    jlong cPluginsArray[numberOfPlugins];
    cPluginsArray[0] = (jlong) new __YOUR_PLUGIN__("com.example.plugin");

    jlongArray jPluginsArray = env->NewLongArray(numberOfPlugins);
    if (jPluginsArray != nullptr) {
        env->SetLongArrayRegion(jPluginsArray, 0, numberOfPlugins, cPluginsArray);
    }

    return jPluginsArray;
}

To build the plugin binary files, navigate to the jni folder and call ndk-build. A libs folder will be created containing libraries for arm7, arm64 and intel. Copy the contents of the libs folder to YourProjectRoot/app/src/main/jniLibs.

Registering Plugins

Register C++ Plugin

To register a C++ plugin with the Wikitude Native SDK, get the PluginManager from the WikitudeSDK instance and call registerNativePlugin passing the name of your library. Do not add lib in front of the name or add the .so extension. If you register your Plugin in the onCreate method of your activity, please also make sure to call the onCreate method of the WikitudeSDK first. The following snippet comes from the BarcodePluginActivity of the Wikitude Native SDK Example application.

@Override
protected void onCreate(Bundle savedInstanceState) {
    ...
    mWikitudeSDK.onCreate(getApplicationContext(), startupConfiguration);
    ...
    mWikitudeSDK.getPluginManager().registerNativePlugins("barcodePlugin");
}

Register Java Plugin

To register a Java plugin with the Wikitude Native SDK, get the PluginManager from the WikitudeSDK instance and call registerPlugin passing an instance of your Plugin.

```java @Override protected void onCreate(Bundle savedInstanceState) { ... mWikitudeSDK.onCreate(getApplicationContext(), startupConfiguration); ... mWikitudeSDK.getPluginManager().registerPlugin(new MyPlugin()); }

Barcode and QR code reader

This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.

ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.

In the BarcodePluginActivity.onCreate method we register the bar code C++ plugin by getting the PluginManager from the Wikitude SDK and calling registerNativePlugins passing the name of the native library containing our C++ plugin. Right after that we call initNative(), which we declared as a native method and implement in the C++ plugin, to initialize the JavaVM pointer hold by the native plugin. We also implement the method onBarcodeDetected to display the contents of the scanned bar code. We'll later call this method from the bar code plugin.

public class BarcodePluginActivity extends Activity implements ImageTrackerListener, ExternalRendering {

    private static String mCodeContent;
    ...

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        ...
        mWikitudeSDK.getPluginManager().registerNativePlugins("barcodePlugin");
        initNative();
    }

    ...

    public void onBarcodeDetected(final String codeContent_) {
        mCodeContent = codeContent;
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                EditText targetInformationTextField = (EditText) findViewById(R.id.barcode_plugin_info_field);
                targetInformationTextField.setText("Scan result: " + codeContent, TextView.BufferType.NORMAL);
            }
        });
    }

    private native void initNative();
}

Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h file. To create the bar code plugin we derive our BarcodePlugin class from wikitude::sdk::Plugin and override initialize, destroy, cameraFrameAvailable and update. We also declare the following member variables: _worldNeedsUpdate, _image, _imageScanner and _methodId. The _worldNeedsUpdate variable will later be used as an indicator if we need to update the View, _image and _imageScanner are classes from zBar which we'll use to scan for bar codes and _methodId will hold the method id of the onBarcodeDetected Java method.

extern JavaVM* pluginJavaVM;

class BarcodePlugin : public wikitude::sdk::Plugin {
public:
    BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight);
    virtual ~BarcodePlugin();

    virtual void initialize();
    virtual void destroy();

    virtual void cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_);
    virtual void update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_);


protected:
    int                             _worldNeedsUpdate;

    zbar::Image                     _image;
    zbar::ImageScanner              _imageScanner;

private:
    jmethodID                       _methodId;
};

We declare two variables in the global namespace one which will hold a pointer to the JavaVM and one which will hold a reference to our activity. To initialize those two variables we declared the initNative native method in the BarcodeActivity and implement it like in the following code snippet. All we do is get the pointer to the JavaVM from the JNIEnv and create a new global reference to the calling activity instance.

JavaVM* pluginJavaVM;
jobject activityObj;

extern "C" JNIEXPORT void JNICALL
Java_com_wikitude_samples_plugins_BarcodePluginActivity_initNative(JNIEnv* env, jobject obj)
{
    env->GetJavaVM(&pluginJavaVM);
    activityObj = env->NewGlobalRef(obj);
}

In the constructor we set _worldNeedsUpdate to zero indicating that there is no update necessary and initialize the zBar::Image member variable passing its constructor the width and height of the camera frame, the image format of Y800, set its data pointer to null and the data length to zero. We use the JavaVM to create an instance of JavaVMResource which is a helper class to manage the JavaVM, we provided in the file jniHelper.cpp. Next we get the Java environment from the JavaVMResource and initialize the _methodId member. In the destructor we delete the global reference to the activity object.

BarcodePlugin::BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight) :
Plugin("com.wikitude.ios.barcodePluign"),
_worldNeedsUpdate(0),
_image(cameraFrameWidth, cameraFrameHeight, "Y800", nullptr, 0)
{
    JavaVMResource vm(pluginJavaVM);
    jclass clazz = vm.env->FindClass("com/wikitude/samples/plugins/BarcodePluginActivity");
    _methodId = vm.env->GetMethodID(clazz, "onBarcodeDetected", "(Ljava/lang/String;)V");
}

BarcodePlugin::~BarcodePlugin()
{
    JavaVMResource vm(pluginJavaVM);
    vm.env->DeleteGlobalRef(activityObj);
}

In the initialize method we configure the zbar::ImageScanner by calling setConfig, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.

void BarcodePlugin::initialize() {    
    _imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}

We react to the destroy event by setting the current data pointer of the zbar::Image member to null and length to zero.

void BarcodePlugin::destroy() {
    _image.set_data(nullptr, 0);
}

The last but most interesting methods are cameraFrameAvailable and update. In the cameraFrameAvailable method we set the data of our previously initialized zbar::Image member variable to the frame data we just received and the length of the data to frame width * frame height by calling set_data. We then start the scanning process by calling the scan method of our zBar::ImageScanner passing the zBar::Image member instance. The zBar::ImageScanner::scan method returns the number of detected bar codes in the image frame, we save this number in a local variable n. If n is not equal to the result of the last frame, which we saved to _worldNeedsUpdate member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was we call the onBarcodeDetected Java method passing the code content.

void BarcodePlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
    int frameWidth = cameraFrame_.getSize().width;
    int frameHeight = cameraFrame_.getSize().height;

    _image.set_data(cameraFrame_.getLuminanceData(), frameWidth * frameHeight);

    int n = _imageScanner.scan(_image);

    if ( n != _worldNeedsUpdate ) {
        if ( n ) {

            zbar::Image::SymbolIterator symbol = _image.symbol_begin();
            JavaVMResource vm(pluginJavaVM);
            jstring codeContent = vm.env->NewStringUTF(symbol->get_data().c_str());
            vm.env->CallVoidMethod(activityObj, _methodId, codeContent);

        }
    }

    _worldNeedsUpdate = n;
}

Face Detection

This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.

The Face Detection Plugin Example consists of the C++ classes FaceDetectionPlugin, FaceDetectionPluginConnector and the Java class FaceDetectionPluginActivity. We will use OpenCV to detect faces in the current camera frame and OpenGL calls in Java to render a frame around detected faces.

The FaceDetectionPluginConnector acts as our interface between native code and Java and contains some JNI code, since JNI is not the focus of this example we won't go into detail about the implementation. If you would like to have a look at the complete code feel free to browse the source code in the Wikitude SDK release package.

We implement to Java native methods initNative and setFlipFlag. First will be used to initialize the plugin with the path to an OpenCV database, second will be used to notify the Plugin about orientation changes of the device. The other methods faceDetected, faceLost, projectionMatrixChanged and renderDetectedFaceAugmentation will be called by the Plugin to update the Java Android Activity, which controls the rendering.

extern "C" JNIEXPORT void JNICALL
Java_com_wikitude_samples_plugins_FaceDetectionPluginActivity_initNative(JNIEnv* env, jobject obj, jstring databasePath_)
{
    ...
}

extern "C" JNIEXPORT void JNICALL
Java_com_wikitude_samples_plugins_FaceDetectionPluginActivity_setFlipFlag(JNIEnv* env, jobject obj, jint flag)
{
    ...
}

... ctor / dtor ...

void FaceDetectionPluginConnector::faceDetected(const float *modelViewMatrix)
{
...
}

void FaceDetectionPluginConnector::faceLost()
{
...
}

void FaceDetectionPluginConnector::projectionMatrixChanged(const float *projectionMatrix)
{
...
}

void FaceDetectionPluginConnector::renderDetectedFaceAugmentation() {
...
}

Next we have a look at the FaceDetectionPlugin class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector to notify the Java activity about the result. The plugin base class defines startRender and endRender, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender and again call the FaceDetectionPluginConnector instance which in turn calls the Android activity. Since we do not react on the result of the Wikitude SDK image recognition we leave update blank.

... ctor/dtor ...

void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {

    ... Control Open CV ...

    if ( _result.size() ) {
        convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
        _observer->faceDetected(_modelViewMatrix);
    } else {
        _observer->faceLost();
    }
}

void FaceDetectionPlugin::startRender() {
    _observer->renderDetectedFaceAugmentation();
}

void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
}

... other internally used methods ...

In the FaceDetectionPluginActivity we override onCreate and initialize the Plugin by calling the initNative native method, passing the path to the database file. We also override onConfigurationChanged to get notify about device orientation changes and again notify the Plugin about orientation changes by calling the setFlipFlag native method. To render a frame around detected faces we created an instance of GLRendererFaceDetectionPlugin class which takes care of rendering a rectangle around faces and all targets of the also active ImageTracker. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate Java methods which we use to update the Renderer instance. If the Plugin decides it is time to render a frame around a detected face it will call renderDetectedFaceAugmentation. Since the plugin will only call this method in the startRender method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.

... imports ...

public class FaceDetectionPluginActivity extends Activity implements ImageTrackerListener, ExternalRendering {

    private WikitudeSDK mWikitudeSDK;
    private GLRendererFaceDetectionPlugin mGLRenderer;
    private File mCascadeFile;
    private FaceTarget mFaceTarget = new FaceTarget();

    ...

    @Override
    protected void onCreate(Bundle savedInstanceState) {

        ... init native sdk ...

        ... copy database file ...

        initNative(mCascadeFile.getAbsolutePath());

        ...

        setInterfaceOrientationInPlugin();
    }

    ... other lifecycle events ...

    private void setInterfaceOrientationInPlugin() {
        ...
        setFlipFlag(x);
        ...
    }

    @Override
    public void onConfigurationChanged(Configuration newConfig) {
        super.onConfigurationChanged(newConfig);
        setInterfaceOrientationInPlugin();
    }

    public void onFaceDetected(float[] modelViewMatrix) {
        mFaceTarget.viewMatrix = modelViewMatrix;
        mGLRenderer.setCurrentlyRecognizedFace(mFaceTarget);
    }

    public void onFaceLost() {
        mGLRenderer.setCurrentlyRecognizedFace(null);
    }

    public void onProjectionMatrixChanged(float[] projectionMatrix) {
        mFaceTarget.projectionMatrix = projectionMatrix;
        mGLRenderer.setCurrentlyRecognizedFace(mFaceTarget);
    }

    ... other Wikitude callbacks ...

    private native void initNative(String casecadeFilePath);
    private native void setFlipFlag(int flag);
}