Input Plugins API
This guide provides an introduction into the input plugins API of the Wikitude Native SDK and aims to familiarize the reader with its concepts and constraints. Due to the length and complexity of the corresponding example application source code, it will not be presented in its entirety. Relevant and descriptive source code examples will, however, be provided. Since the Input Plugins API is an extension to the Plugins API, we recommend familiarity with it prior to reading this guide.
About Wikitude SDK Input Plugins
The input plugins API provides a means to alter the inputs and outputs of the Wikitude Native SDK. For the input case specifically, custom frame data of arbitrary sources can be supplied as an input to the Wikitude SDK Native API for processing. Complementary, for the output case, the default rendering of the Wikitude SDK Native API can be substituted with more advanced implementations.
Input plugins can be registered before the SDK is started or while it is already running. In case it is registered before the Wikitude SDK is started, the internal Wikitude camera implementation is not started at all and the Wikitude SDK starts using the input plugin from the very beginning on. In case an input plugin is registered at runtime, the internal Wikitude SDK camera is first stopped, subsequently followed by a transition to the newly registered input plugin.
Input plugin module class
class CameraFrameInputPluginModule : public PluginModule {
public:
CameraFrameInputPluginModule() noexcept = default;
virtual ~CameraFrameInputPluginModule() = default;
/**
* Override/implement this method to know when the default platform camera is fully released and this camera frame input plugin module can safely access all platform camera resources
*/
virtual void onCameraReleased() = 0;
virtual void onCameraReleaseFailed(const sdk::Error& error_) = 0;
/**
* Default: false
*/
bool requestsCameraFrameRendering();
/* Called from the Wikitude SDK */
void registerOnPluginCameraReleasedHandler(std::function<void()> onPluginCameraReleasedHandler_);
void registerNotifyNewUnmanagedCameraFrameHandler(std::function<void(const sdk::CameraFrame& cameraFrame_)> notifyNewUnmanagedCameraFrameHandler_);
void registerCameraToSurfaceAngleChangedHandler(std::function<void(float cameraToSurfaceAngle_)> cameraToSurfaceAngleChangedHandler_);
protected:
/**
* Call this method to notify a new camera frame to the SDK
*/
void notifyNewUnmanagedCameraFrameToSDK(const sdk::CameraFrame& cameraFrame_);
/**
* Call this method to notify the SDK that this camer frame input plugin module fully released all platform camera resources.
*
*/
void notifyPluginCameraReleased();
void setCameraToSurfaceAngle(float cameraToSurfaceAngle_);
protected:
bool _requestsCameraFrameRendering = false;
private:
std::function<void()> _onPluginCameraReleasedHandler;
std::function<void(const sdk::CameraFrame&)> _notifyNewUnmanagedCameraFrameHandler;
std::function<void(float)> _cameraToSurfaceAngleChangedHandler;
};
An input plugin is simply a plugin that has a CameraFrameInputPluginModule associated with it. The module allows implementing the input related features.
Input plugin module implementation
The following code is a minimal example of how to create an input plugin and provide a camera frame to be rendered an processed. The code corresponds to the simple input plugin sample of the Wikitude SDK sample application.
FrameInputPluginModule::FrameInputPluginModule(bool requestRendering_)
: _metadata(wikitude::sdk::ColorCameraFrameMetadata(-1, {640,480}, wikitude::sdk::CameraPosition::Back, wikitude::sdk::ColorSpace::YUV_420_888, 0))
, _frameId(-1)
{
_requestsCameraFrameRendering = requestRendering_;
}
void onCameraReleased() override {
}
void onCameraReleaseFailed(const wikitude::sdk::Error& error_) override {
}
void FrameInputPluginModule::cameraToSurfaceAngleChanged(float angle_){
CameraFrameInputPluginModule::setCameraToSurfaceAngle(angle_);
}
void FrameInputPluginModule::fieldOfViewChanged(float fov_) {
_metadata = wikitude::sdk::ColorCameraFrameMetadata(fov_, _metadata.getPixelSize(), _metadata.getCameraPosition(), _metadata.getFrameColorSpace(), _metadata.getTimestampTimescale());
}
void FrameInputPluginModule::notifyNewCameraFrameYUV_420_888(const std::vector<wikitude::sdk::CameraFramePlane>& planes_) {
notifyNewUnmanagedCameraFrameToSDK(wikitude::sdk::CameraFrame(++_frameId, 0, _metadata, planes_));
}
extern "C" JNIEXPORT void JNICALL Java_com_wikitude_samples_plugins_FrameInputPluginModule_nativeFieldOfViewChanged(JNIEnv*, jobject, jlong nativeHandle_, jfloat fov_) {
auto module = reinterpret_cast<FrameInputPluginModule*>(nativeHandle_);
module->fieldOfViewChanged(fov_);
}
extern "C" JNIEXPORT void JNICALL Java_com_wikitude_samples_plugins_FrameInputPluginModule_nativeCameraToSurfaceAngleChanged(JNIEnv*, jobject, jlong nativeHandle_, jfloat angle_) {
auto module = reinterpret_cast<FrameInputPluginModule*>(nativeHandle_);
module->cameraToSurfaceAngleChanged(angle_);
}
extern "C" JNIEXPORT void JNICALL Java_com_wikitude_samples_plugins_FrameInputPluginModule_nativeCameraReleased(JNIEnv*, jobject, jlong nativeHandle_) {
auto module = reinterpret_cast<FrameInputPluginModule*>(nativeHandle_);
module->cameraReleased();
}
The `_requestsCameraFrameRendering flag is used to communicate whether the frame should be rendered by the SDK or not.
The onCameraReleased and onCameraReleaseFailed function should be used to wait for the internal camera of the SDK to shut down before opening a new camera. This only applied if the input plugin is registered after the SDK has been started.
The notifyNewUnmanagedCameraFrameToSDK function is used to pass a camera frame to the SDK for rendering and processing. The CameraFrame class encapsulates the frame data and the frame meta data. The following sample code demonstrates how to create such an object.
The setCameraToSurfaceAngle function has is used to pass the rotation of the camera relative to the rotation of the render surface. This has to be set so that the SDK can render the camera frame correctly and provide correct matrices for tracking.
After a camera frame has been acquired, in can be converted and forwarded using code akin to the following snippet.
auto module = reinterpret_cast<FrameInputPluginModule*>(nativeHandle_);
void* luminance = env_->GetDirectBufferAddress(luminanceData_);
void* chromaBlue = env_->GetDirectBufferAddress(chromaBlueData_);
void* chromaRed = env_->GetDirectBufferAddress(chromaRedData_);
unsigned int luminanceDataSize = static_cast<unsigned int>(env_->GetDirectBufferCapacity(luminanceData_));
unsigned int chromaBlueDataSize = static_cast<unsigned int>(env_->GetDirectBufferCapacity(chromaBlueData_));
unsigned int chromaRedDataSize = static_cast<unsigned int>(env_->GetDirectBufferCapacity(chromaRedData_));
std::vector<wikitude::sdk::CameraFramePlane> planes = {
wikitude::sdk::CameraFramePlane(luminance, luminanceDataSize, 1, rowStrideLuminance_),
wikitude::sdk::CameraFramePlane(chromaBlue, chromaBlueDataSize, pixelStrideChroma_, rowStrideChroma_),
wikitude::sdk::CameraFramePlane(chromaRed, chromaRedDataSize, pixelStrideChroma_, rowStrideChroma_)
};
module->notifyNewCameraFrameYUV_420_888(planes);
For a complete implementation of an input plugin for a specific and advanced use case, we strongly recommend looking into the custom camera example application source code. Additionally, the custom camera sample source code is an excellent starting point to build your own implementation from.