Input Plugins API
This guide provides an introduction into the input plugins API of the Wikitude Native SDK and aims to familiarize the reader with its concepts and constraints. Due to the length and complexity of the corresponding example application source code, it will not be presented in its entirety. Relevant and descriptive source code examples will, however, be provided. Since the Input Plugins API is an extension to the Plugins API, we recommend familiarity with it prior to reading this guide.
About Wikitude SDK Input Plugins
The input plugins API provides a means to alter the inputs and outputs of the Wikitude Native SDK. For the input case specifically, custom frame data of arbitrary sources can be supplied as an input to the Wikitude SDK Native API for processing. Complementary, for the output case, the default rendering of the Wikitude SDK Native API can be substituted with more advanced implementations.
Input plugins can be registered before the SDK is started or while it is already running. In case it is registered before the Wikitude SDK is started, the internal Wikitude camera implementation is not started at all and the Wikitude SDK starts using the input plugin from the very beginning on. In case an input plugin is registered at runtime, the internal Wikitude SDK camera is first stopped, subsequently followed by a transition to the newly registered input plugin.
Input plugin module class
class CameraFrameInputPluginModule : public PluginModule {
public:
CameraFrameInputPluginModule() noexcept = default;
virtual ~CameraFrameInputPluginModule() = default;
/**
* Override/implement this method to know when the default platform camera is fully released and this camera frame input plugin module can safely access all platform camera resources
*/
virtual void onCameraReleased() = 0;
virtual void onCameraReleaseFailed(const sdk::Error& error_) = 0;
/**
* Default: false
*/
bool requestsCameraFrameRendering();
/* Called from the Wikitude SDK */
void registerOnPluginCameraReleasedHandler(std::function<void()> onPluginCameraReleasedHandler_);
void registerNotifyNewUnmanagedCameraFrameHandler(std::function<void(const sdk::CameraFrame& cameraFrame_)> notifyNewUnmanagedCameraFrameHandler_);
void registerCameraToSurfaceAngleChangedHandler(std::function<void(float cameraToSurfaceAngle_)> cameraToSurfaceAngleChangedHandler_);
protected:
/**
* Call this method to notify a new camera frame to the SDK
*/
void notifyNewUnmanagedCameraFrameToSDK(const sdk::CameraFrame& cameraFrame_);
/**
* Call this method to notify the SDK that this camer frame input plugin module fully released all platform camera resources.
*
*/
void notifyPluginCameraReleased();
void setCameraToSurfaceAngle(float cameraToSurfaceAngle_);
protected:
bool _requestsCameraFrameRendering = false;
private:
std::function<void()> _onPluginCameraReleasedHandler;
std::function<void(const sdk::CameraFrame&)> _notifyNewUnmanagedCameraFrameHandler;
std::function<void(float)> _cameraToSurfaceAngleChangedHandler;
};
An input plugin is simply a plugin that has a CameraFrameInputPluginModule associated with it. The module allows implementing the input related features.
Input plugin module implementation
The following code is a minimal example of how to create an input plugin and provide a camera frame to be rendered an processed. The code corresponds to the simple input plugin sample of the Wikitude SDK sample application.
class SimpleYUVCameraFrameInputPluginModule : public wikitude::sdk::CameraFrameInputPluginModule {
public:
SimpleYUVCameraFrameInputPluginModule() {
_requestsCameraFrameRendering = true;
}
void onCameraReleased() override {
}
void onCameraReleaseFailed(const wikitude::sdk::Error& error_) override {
}
void notifyNewCameraFrame(const wikitude::sdk::CameraFrame& cameraFrame_) {
notifyNewUnmanagedCameraFrameToSDK(cameraFrame_);
}
};
The _requestsCameraFrameRendering flag is used to communicate whether the frame should be rendered by the SDK or not.
The onCameraReleased and onCameraReleaseFailed function should be used to wait for the internal camera of the SDK to shut down before opening a new camera. This only applied if the input plugin is registered after the SDK has been started.
The notifyNewUnmanagedCameraFrameToSDK function is used to pass a camera frame to the SDK for rendering and processing. The CameraFrame class encapsulates the frame data and the frame meta data. The following sample code demonstrates how to create such an object.
After a camera frame has been acquired, in can be converted and forwarded using code akin to the following snippet.
const float fov = _fov;
const wikitude::sdk::Size<int> size{_width, _height};
const std::int32_t timescale = presentationTimestamp.timescale;
wikitude::sdk::ColorCameraFrameMetadata metaData(fov,
size,
wikitude::sdk::CameraPosition::Back,
wikitude::sdk::ColorSpace::YUV_420_NV21,
timescale
);
std::vector<wikitude::sdk::CameraFramePlane> planes;
wikitude::sdk::CameraFramePlane plane(&_frameData[0], static_cast<unsigned int>(frameDataSize));
planes.push_back(plane);
wikitude::sdk::CameraFrame cameraFrame(_frameId++, timevalue, metaData, planes);
static_cast<SimpleYUVCameraFrameInputPluginModule*>(_plugin->getCameraFrameInputPluginModule())->notifyNewCameraFrame(cameraFrame);
For a complete implementation of an input plugin for a specific and advanced use case, we strongly recommend looking into the custom camera example application source code. Additionally, the custom camera sample source code is an excellent starting point to build your own implementation from.