Plugins API
This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.
About Wikitude SDK Plugins
A plugin is a class, or rather a set of classes, written in C++ that allows extending the functionality of the Wikitude SDK. While there is a Plugin
base class that offers some of the main functionality, a plugin itself can have multiple different optional modules that allow for more complex concepts to be implemented. The following table gives a very brief overview of the distribution of responsibilities of the plugin related classes.
class | purpose |
---|---|
Plugin |
Main class of a plugin implementation. Derive from this class to create your plugin. It handles the application lifecycle and main plugin functionality. It provides access to various parameters of the Wikitude SDK and provides access to the camera frame and the recognized targets. It further owns and manages all the optional plugin modules. |
ImageTrackingPluginModule |
Optional module to allow for custom image tracking implementations. Derive from this class to implement your own image tracking algorithm to work in conjunction with the Wikitude SDK algorithms. |
InstantTrackingPluginModule |
Optional module to allow for custom instant tracking implementations. Derive from this class to implement your own instant tracking algorithm to work in conjunction with the Wikitude SDK algorithms. |
ObjectTrackingPluginModule |
Optional module to allow for custom object tracking implementations. Derive from this class to implement your own object tracking algorithm to work in conjunction with the Wikitude SDK algorithms. |
CameraFrameInputPluginModule |
Optional module to allow for frame data to be input into the Wikitude SDK. Derive from this class to implement your own camera frame acquisition. The supplied frame data supplied can be processed and rendered by the Wikitude SDK. |
DeviceIMUInputPluginModule |
Optional module to allow for sensor data to be input into the Wikitude SDK. Derive from this class to implement your own sensor data acquisition. The supplied sensor data supplied will be used by the Wikitude SDK for its tracking algorithms where applicable. |
OpenGLESRenderingPluginModule |
Optional module to allow for custom OpenGL ES rendering. Available on iOS and Android. |
MetalRenderingPluginModule |
Optional module to allow for custom Metal rendering. Only available on iOS. |
DirectXRenderingPluginModule |
Optional module to allow for custom DirectX rendering. Only available on Windows. |
Each of the optional modules can be registered by calling the corresponding function of the Plugin
class.
An important thing to remember when working with plugins is that they need to have a unique identifier. If an attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.
Plugin Base Class
class Plugin {
public:
Plugin(std::string identifier_);
virtual ~Plugin();
virtual void initialize(const std::string& temporaryDirectory_, PluginParameterCollection& pluginParameterCollection_);
virtual void setSDKEdition(SDKEdition sdkEdition_);
virtual void pause();
virtual void resume(unsigned int pausedTime_);
virtual void destroy();
virtual void cameraFrameAvailable(common_code::ManagedCameraFrame& managedCameraFrame_) = 0;
virtual void deviceRotationEventAvailable(const DeviceRotationEvent& deviceRotationEvent_);
virtual void deviceOrientationEventAvailable(const DeviceOrientationEvent& deviceOrientationEvent_);
virtual void prepareUpdate();
virtual void update(const RecognizedTargetsBucket& recognizedTargetsBucket_) = 0;
virtual const std::string& getIdentifier() const;
virtual void setEnabled(bool enabled_);
virtual bool isEnabled() const;
virtual PluginType getPluginType() const;
virtual bool canPerformTrackingOperationsAlongOtherPlugins();
virtual bool canUpdateMultipleTrackingInterfacesSimultaneously();
ImageTrackingPluginModule* getImageTrackingPluginModule() const;
InstantTrackingPluginModule* getInstantTrackingPluginModule() const;
ObjectTrackingPluginModule* getObjectTrackingPluginModule() const;
CameraFrameInputPluginModule* getCameraFrameInputPluginModule() const;
DeviceIMUInputPluginModule* getDeviceIMUInpputPluginModule() const;
OpenGLESRenderingPluginModule* getOpenGLESRenderingPluginModule() const;
MetalRenderingPluginModule* getMetalRenderingPluginModule() const;
DirectXRenderingPluginModule* getDirectXRenderingPluginModule() const;
protected:
void setImageTrackingPluginModule(std::unique_ptr<ImageTrackingPluginModule> imageTrackingPluginModule_);
void setObjectTrackingPluginModule(std::unique_ptr<ObjectTrackingPluginModule> objectTrackingPluginModule_);
void setInstantTrackingPluginModule(std::unique_ptr<InstantTrackingPluginModule> instantTrackingPluginModule_);
void setCameraFrameInputPluginModule(std::unique_ptr<CameraFrameInputPluginModule> cameraFrameInputPluginModule_);
void setDeviceIMUInputPluginModule(std::unique_ptr<DeviceIMUInputPluginModule> deviceIMUInputPluginModule_);
void setOpenGLESRenderingPluginModule(std::unique_ptr<OpenGLESRenderingPluginModule> openGLESRenderingPluginModule_);
void setMetalRenderingPluginModule(std::unique_ptr<MetalRenderingPluginModule> metalRenderingPluginModule_);
void setDirectXRenderingPluginModule(std::unique_ptr<DirectXRenderingPluginModule> directXRenderingPluginModule_);
void iterateEnabledPluginModules(std::function<void(PluginModule& activePluginModule_)> activePluginModuleIteratorHandle_);
protected:
std::string _identifier;
bool _enabled;
mutable std::mutex _pluginModuleAccessMutex;
std::set<PluginModule*> _availablePluginModules;
private:
std::unique_ptr<ImageTrackingPluginModule> _imageTrackingModule;
std::unique_ptr<InstantTrackingPluginModule> _instantTrackingModule;
std::unique_ptr<ObjectTrackingPluginModule> _objectTrackingModule;
std::unique_ptr<CameraFrameInputPluginModule> _cameraFrameInputModule;
std::unique_ptr<DeviceIMUInputPluginModule> _deviceIMUInputPluginModule;
std::unique_ptr<OpenGLESRenderingPluginModule> _openGlesRenderingModule;
std::unique_ptr<MetalRenderingPluginModule> _metalRenderingModule;
std::unique_ptr<DirectXRenderingPluginModule> _directXRenderingModule;
};
While we will not go over every function of the Plugin
class and all the optional module classes, the following sections will present sample plugins that should convey most of the concepts and methods involved in creating your own plugin.
Information about Recognized Targets
If the Wikitude SDK is running with active image-, instant- or object recognition, the plugins API will populate the RecognizedTargetsBucket
in the update
method with the currently recognized targets. The plugin may then use the corresponding target objects to acquire data, most importantly the pose, and use it for further processing.
class RecognizedTargetsBucket {
public:
virtual ~RecognizedTargetsBucket() = default;
virtual const std::vector<ImageTarget*>& getImageTargets() const = 0;
virtual const std::vector<ObjectTarget*>& getObjectTargets() const = 0;
virtual const std::vector<InstantTarget*>& getInstantTargets() const = 0;
virtual const std::vector<InitializationPose*>& getInitializationPoses() const = 0;
virtual const std::vector<Plane*>& getPlanes() const = 0;
virtual const Matrix4& getViewMatrix() const = 0;
};
Platform Specifics
Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin
and create a std::shared_ptr
from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.
To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.
Registering Plugins
Register C++ Plugin
The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTArchitectView
class category WTArchitectView+Plugins
which is implemented in WTArchitectView+Plugins.h
. Note that this header file has to be specifically imported and is not part of the Wikitude SDK umbrella header.
Registering a C++ Plugin
To register a C++ plugin, call the -registerPlugin:
method and pass a std::shared_ptr
wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.
@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
Removing a C++ Plugin
To remove a already known C++ plugin, call either the -removePlugin:
or -removeNamedPlugin:
method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared
when calling -registerPlugin:
. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).
[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];
Barcode and QR code reader
This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.
ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.
First let's have a look at the BarcodePlugin.h
file. To create the bar code plugin we derive our BarcodePlugin
class from the wikitude::sdk::ArchitectPlugin
and BarcodeScanner::Observer
classes and override cameraFrameAvailable
, recognizedBarcode
and lostBarcode
. We also declare the following member methods and variables: getJavaScriptPluginModule()
and _registeredBarcodeScanner
. The getJavaScriptPluginModule()
method will later be used to communicate the c++ with the js part and the _registeredBarcodeScanner
variable to scan for bar codes in a camera frame.
class BarcodePlugin : public wikitude::sdk::ArchitectPlugin, public BarcodeScanner::Observer {
public:
BarcodePlugin();
virtual ~BarcodePlugin() = default;
/* From ArchitectPlugin */
void cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) override;
/* From BarcodeScanner::Observer */
void recognizedBarcode(long id_, const std::string& barcode_) override;
void lostBarcode(long id_) override;
protected:
BarcodeScannerJavaScriptPluginModule* getJavaScriptPluginModule();
protected:
std::unordered_map<long, std::unique_ptr<BarcodeScanner>> _registeredBarcodeScanner;
};
We now go through each method of the BarcodePlugin
class, starting with the constructor. In the constructor we set a new javascript plugin module (required by the wikitude::sdk::ArchitectPlugin
class) and add it as a new registered barcode scanner.
BarcodePlugin::BarcodePlugin()
:
ArchitectPlugin("com.wikitude.plugins.barcode_scanner_demo")
{
setJavaScriptPluginModule(std::make_unique<BarcodeScannerJavaScriptPluginModule>([&](long id_) {
_registeredBarcodeScanner.emplace(id_, std::make_unique<BarcodeScanner>(id_, *this));
}));
}
In the cameraFrameAvailable
we send a camera frame to be processed by all the registered barcode scanners we have.
void BarcodePlugin::cameraFrameAvailable(wikitude::sdk::ManagedCameraFrame& managedCameraFrame_) {
for ( auto& barcodeScannerPair : _registeredBarcodeScanner ) {
barcodeScannerPair.second->processCameraFrame(managedCameraFrame_);
}
}
The last but most interesting methods are recognizedBarcode
and lostBarcode
. When we send a camera frame to be processed in cameraFrameAvailable
, if the barcode scanner scans a valid barcode it calls the recognizedBarcode
method which warns the js part by calling getJavaScriptPluginModule()->callBarcodeRecognizedCallback()
. If a recognized barcode is not recognized anymore in the last processed frames, the scanner calls lostBarcode
with the id of the barcode lost and also warns the js part by calling getJavaScriptPluginModule()->callBarcodeLostCallback
.
void BarcodePlugin::recognizedBarcode(long id_, const std::string& barcode_) {
getJavaScriptPluginModule()->callBarcodeRecognizedCallback(id_, barcode_);
}
void BarcodePlugin::lostBarcode(long id_) {
getJavaScriptPluginModule()->callBarcodeLostCallback(id_);
}
BarcodeScannerJavaScriptPluginModule* BarcodePlugin::getJavaScriptPluginModule() {
return dynamic_cast<BarcodeScannerJavaScriptPluginModule*>(ArchitectPlugin::getJavaScriptPluginModule());
}
Face Detection
This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.
The Face detection plugin example consists of the C++ classes FaceDetectionPlugin
, FaceDetectionPluginConnector
and the Objective-C classes WTFaceDetectionPluginWrapper
and WTAugmentedRealityViewController+PluginLoading
. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces.
The Architect example application is based on one view controller that manages the WTArchitectView
object. Since some examples require some more Objective-C code, various class extension exist which implement this specific example code. For this example its the WTAugmentedRealityViewController+PluginLoading
class extension. The extension loads Wikitude SDK plugins based on a string identifier. Because this example requires some more Objective-C code than the previous one, a wrapper object is created which takes care about object (e.g. plugin) creation and management. This class is named WTFaceDetectionPluginWrapper
and an instance is set as a associated Objective-C object to WTAugmentedRealityViewController
(Because class extensions can only define new methods and no instance variables).
FaceDetectionPluginConnector
is used to connect the FaceDetectionPlugin
class with the WTFaceDetectionPluginWrapper
class extension. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude SDK example application.
The plugin wrapper handles the face detection plugin as described in the previously barcode example. New for this example is a class method that converts UIDeviceOrientations into an int that is used by the face detection plugin to rotate the camera frame in order to always have faces in the right orientation. This class method is then used every time the device changes its orientation.
__weak typeof(self) weakSelf = self;
[[NSNotificationCenter defaultCenter] addObserverForName:UIDeviceOrientationDidChangeNotification object:[UIDevice currentDevice] queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification *note) {
weakSelf.faceDetectionPlugin->setFlipFlag( [WTFaceDetectionPluginViewController flipFlagForDeviceOrientation:[[UIDevice currentDevice] orientation]] );
}];
Next we have a look at the FaceDetectionPlugin
class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable
method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector
to notify the view controller about the result. The plugin base class defines startRender
and endRender
, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender
and again call the FaceDetectionPluginConnector
instance which in turn calls the plugin wrapper. Since we do not react on the result of the Wikitude SDK image recognition we leave update
blank.
... ctor/dtor ...
void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
wikitude::sdk::Size frameSize = cameraFrame_.getSize();
_scaledCameraFrameWidth = cameraFrame_.getScaledWidth();
_scaledCameraFrameHeight = cameraFrame_.getScaledHeight();
std::memcpy(_grayFrame.data,cameraFrame_.getLuminanceData(), frameSize.height*frameSize.width*sizeof(unsigned char));
//... Control Open CV ...
if ( _result.size() ) {
convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
_observer->faceDetected(_modelViewMatrix);
} else {
_observer->faceLost();
}
}
void FaceDetectionPlugin::startRender() {
_observer->renderDetectedFaceAugmentation();
}
void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
/* Intentionally Left Blank */
}
//... other internally used methods ...
To render a frame around detected faces we created an instance of the StrokedRectangle
class which takes care of rendering a rectangle around faces. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate plugin wrapper methods which we use to update the StrokedRectangle
instance. If the plugin decides it is time to render a frame around a detected face it will call renderDetectedFaceAugmentation
. Since the plugin will only call this method in the startRender
method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.
void FaceDetectionPluginConnector::faceDetected(const float *modelViewMatrix) {
[_faceDetectionPluginWrapper setFaceDetected:YES];
[_faceDetectionPluginWrapper.faceAugmentation setModelViewMatrix:modelViewMatrix];
}
void FaceDetectionPluginConnector::faceLost() {
[_faceDetectionPluginWrapper setFaceDetected:NO];
}
void FaceDetectionPluginConnector::projectionMatrixChanged(const float *projectionMatrix) {
[_faceDetectionPluginWrapper.faceAugmentation setProjectionMatrix:projectionMatrix];
}
void FaceDetectionPluginConnector::renderDetectedFaceAugmentation() {
if ( [_faceDetectionPluginWrapper faceDetected] ) {
[_faceDetectionPluginWrapper.faceAugmentation drawInContext:[EAGLContext currentContext]];
}
}