Plugins API

The Plugins API provides access to the camera frame as a native pointer. This allows you to do additional processing on it, either in C#, or in native code (C/C++).

Unity interface

To enable plugins, you need to add the Plugin script to a GameObject. Then, you simply subscribe to the On Camera Frame Available event and you will be notified when a new camera frame is ready for additional processing. Please be aware that this event could be called from another thread.

You will receive as a parameter to the callback a Wikitude.CameraFrame class which contains all the information about the frame.

public class CameraFrame
{
    public long ID;
    public long ColorTimestamp;
    public ColorCameraFrameMetadata ColorMetadata;
    public List<CameraFramePlane> ColorData;
}

The ColorMetadata contains additional information about the camera frame, such as size and format.

    public struct ColorCameraFrameMetadata {
        public float HorizontalFieldOfView;
        public int Width;
        public int Height;
        public CaptureDevicePosition CameraPosition;
        public FrameColorSpace ColorSpace;
        public int TimestampScale;

    }

The ColorData list contains the raw IntPtr to the actual native memory that is used to store the frame data for each of the planes contained in the camera frame.

The native pointer is only valid during the duration of the current On Camera Frame Available call and you should never delete or change the data it's pointing to. You can pass this pointer to your own native plugins, or transfer the data in C# using Marshal.Copy functions.

using UnityEngine;
using System;
using Wikitude;
using System.Runtime.InteropServices;

public class PluginController : MonoBehaviour
{
    public void OnCameraFrameAvailable(Frame frame) {

        // Example of how to transfer data from native memory
        // to a C# array

        byte[] data = new byte[frame.DataSize];
        Marshal.Copy(frame.Data, data, 0, frame.DataSize);
    }
}

Sample explanation

The Plugins - Barcode sample shows how to integrate the popular barcode library ZXing.Net into Unity and use the Plugins API to send camera frames to it for processing.

The PluginController.cs script controls the sample. It is registered in the editor to receive OnCameraFrameAvailable. First, it creates a new thread that will be used to do the scanning on. This is required because scanning might take too long on some older devices, which would make the app seem unresponsive.

In the Update method, we only check if we have any new scanning results, and display them if that is the case.

void Update() {
    if (_scanningInProgress) {
        return;
    }

    if (_scanningResult != null) {
        ResultText.text = _scanningResult;
        _scanningResult = null;
    }
}

When the OnCameraFrameAvailable event is triggered, we first check if scanning is already in progress. If not, we convert the raw data into a Color32 array, and then we trigger a new scan.

public void OnCameraFrameAvailable(CameraFrame frame) {
    if (_scanningInProgress) {
        return;
    }

    var metadata = frame.ColorMetadata;
    var data = new Color32[metadata.Width * metadata.Height];
    if (metadata.ColorSpace == FrameColorSpace.RGBA) {
        var rawBytes = new byte[frame.ColorData[0].DataSize];
        Marshal.Copy(frame.ColorData[0].Data, rawBytes, 0, (int)frame.ColorData[0].DataSize);

        for (int i = 0; i < metadata.Width * metadata.Height; ++i) {
            data[i] = new Color32(rawBytes[i * 4], rawBytes[i * 4 + 1], rawBytes[i * 4 + 2], rawBytes[i * 4 + 3]);
        }
    } else if (metadata.ColorSpace == FrameColorSpace.RGB) {
        /* conversion code */
    } else if (metadata.ColorSpace == FrameColorSpace.YUV_420_NV12 ||
               metadata.ColorSpace == FrameColorSpace.YUV_420_NV21 ||
               metadata.ColorSpace == FrameColorSpace.YUV_420_YV12) {
        /* conversion code */
    }

    // Trigger a new scan
    Monitor.Pulse(_monitor);
}

Finally, when triggering a new scan, we call the Decode method and get the scan result back. We enable the AutoRotate option so that scanning works both in Landscape, as well as in Portraits modes.

private string Decode(Color32[] colors, int width, int height) {
    var reader = new BarcodeReader();
    reader.AutoRotate = true;
    var result = reader.Decode(colors, width, height);
    if (result != null) {
        return result.Text;
    } else {
        return null;
    }
}

Please note that the code snippets above have been simplified for clarity. Please see the full source code for more information on thread synchronization and error handling, as well as other integration details.

Documentation

Wikitude SDK Hololens

Plugins API

Unity interface

Sample explanation