Track, Save and Share: A Developer View on SDK 8

Phil

More than 6 years ago in April 2012 the very first version of the Wikitude SDK found its way to our first customers. So now, after XYZ releases in total, what can one expect from the 8th major market version of an augmented reality SDK? In short: a lot. Wikitude SDK 8 represents a change as significant as SDK 5 was when we started to offer native APIs for the Wikitude SDK. Some features in this 8th version made it necessary to rethink software architecture choices from the past and re-write major parts of the SDK to give it a structure that supports future challenges and meets the requirements of future projects.

In this article, we will share insights about the technical background of the key changes and features that are new in Wikitude SDK 8.

Instant Targets – making augmented reality persistent

Back in early 2017, the Wikitude SDK was the first SDK to provide developers with the ability to use Instant Trackers (aka markerless tracking, aka Positional tracking) for a wide range of devices. Some of this functionality is now available as part of our SMART feature, which wraps ARKit and ARCore and Instant Tracking into a single API. Positional tracking is a fantastic capability, but it is not persistent at all. Users can place content freely but will have to re-do this every time they start another session.

Wikitude’s Instant Tracking included the ability of visual re-localization right from the beginning (once tracking was lost, it would pick up again in case it detected an area already covered). With SDK 8 Wikitude now introduces the ability to save Instant Tracking sessions in the form of Instant Targets. The API allows you to serialize the current session and use this target to load as a target later on – the same way used with other trackers in the Wikitude SDK. Object Tracker recognizing Object Targets, Image Tracker recognizing Image Targets, and now, Instant Tracker recognizing Instant Targets.

Instant Targets is a 3D representation of whatever a user tracked in a session – in fact, the file format in which the information is stored is identical to the files generated for Object Tracker (wto). This could be really anything – internally we tried plain images, objects, and smaller scenes with good results.

Reworked AR engine

The introduction of Instant Targets is accompanied by a major upgrade to our Instant Tracking engine – actually to the entire 3D SLAM engine that is powering Instant Tracking, Object Tracking, and all Extended Tracking modes in the background. Anything SLAM-related was revisited and improved – the algorithms have been updated to the latest generation of SLAM algorithms and incorporate findings from recent research.

While ARKit and ARCore are deploying sensor-fusion-based systems, we saw that there is still a great potential in vision-only systems, that don’t require detailed calibration of camera and IMU. Of course, sensor-fusion systems have their advantages when it comes to fast, rotational-dominant movements like fast 180° turns.

Apart from this, the new 3D SLAM engine reaches extremely good results in tracking benchmarks even compared to ARKit and ARCore. In our tests, we could see a reduction of tracking error by up to 60% compared to SDK 7.2.

The general benefits of the new SLAM engine you will enjoy are:

• Higher precision of tracking the environment in general
• Instant Tracking more robust for pure lateral movements with low parallax at the beginning
• Tracking can cover greater areas

Going beyond Object Recognition with Extended Object Tracking

For Object Recognition SDK 8 has many changes included. Both the quality and accuracy of the initial recognition and tracking improved substantially, particularly for 360° tracking scenarios. The version of Object Recognition in SDK 8 is fundamentally better compared to our previous versions. Part of the improvements are due to the new 3D SLAM engine, but a great part is also due to the new recording process for Object Targets introduced with SDK 8.

Previously, Object Targets were created by uploading a video of the object. While this method worked properly for many of our customers, we identified weaknesses with this approach:

• Extending Object Targets was cumbersome as it required producing a new video of the entire object
• The size of the video files was limited and as a direct consequence, the video quality could only be average.
• For some objects, it is hard to create a proper video

So with SDK 8, we are introducing a different way to create Object Targets: Meet image-based conversion. Object Targets for SDK 8 are now created by uploading images of the object to Studio Manager, which will then convert the images into an Object Target.

With this approach, the quality of the resulting Object Targets will already be a lot better compared to the video-based approach (e.g. if you create an Object Target from a video and compare it with an Object Target created from still images of that very same video, the new image-based conversion will produce a more accurate Object Target).

Precise Object Targets obviously also result in higher accuracy. However, the new conversion method made it necessary to adapt the internals of the wto file format and wtos created using image-based conversion will not be backward compatible.

The new image-based conversion also makes it a lot easier to add uncovered parts by just adding images to the existing map and recreating it using the new images.

We also saw that many objects from customers consist of mirrored faces or have some sort of symmetry in it. The long sides of the toy fire truck that you find in Wikitude’s sample app are nearly identical but mirrored. The new image-based conversion has been developed, so it can detect these cases and identify symmetrical faces correctly.

Speaking of new images: this method also allows you to upload images from the object under different conditions. Using images of the same object in front of a light background and in front of a dark background (or adding variants of an object in direct sunlight and in overcast conditions) will increase the stability of recognition in changing light conditions.

Scene Recognition – tracking LARGE objects and scenes

This brings me to the next great (!) addition of Object Recognition in SDK 8 – Scene recognition. As a direct consequence of the new conversion method, suitable objects can now be considerably larger. In our tests, we mapped objects like monuments, entire house facades, and castles that cover areas over 2400 square meters.

You might already get the feeling that SDK 8 is more than just another feature release of the Wikitude SDK. With SDK 8 we re-built many parts of the SDK to make the architecture fit the growing requirements. Similar to SDK 5 more than 3 years ago, SDK 8 includes another major software architecture redesign. Under the hood of the SDK now works our internally dubbed “Universal SDK”, a platform and OS-agnostic SDK wrapping the core functionality in C++ – while the Wikitude SDK always consisted of a C++ core, we took this approach to the extreme with SDK 8.

Unity Live Preview

As a result of the Universal SDK, we can now finally offer Live Preview for Unity, for both using attached cameras or Unity Remote for testing your experiences directly within the IDE. No more building and transferring your app to an attached device for your testing. Live Preview works for all tracker types – even Instant Tracking using the Unity Remote application. Unity developers can now work using this feature on their macOS or Windows machines.

Introducing support for Windows

Windows! Finally, the Wikitude SDK will run on Windows 10 UWP computers as well. Not only for Live Preview but as a first-class citizen under the operating systems supported by Wikitude. Again, the Wikitude SDK for Windows is powered by the Universal C++ core. For the start, you can build augmented reality experiences powered by Wikitude through Unity and a Native API, so you can build pure UWP AR apps without the need to learn Unity. Our JavaScript API and support for Hololens will follow later. Given the unfortunate developments of the Windows Phone ecosystem, we decided on the initial release in which only devices with Intel™ chips will be supported. As Microsoft and ARM are gearing up to run newer Microsoft hardware also on ARM-based chips, we will monitor the market and add support for ARM-based devices later on.

Some additional cool features

The camera obviously plays a vital role in every augmented reality experience. With the Wikitude SDK, you already have several ways to control the camera streams available on a device. From changing camera (back/front) to setting focal distances, there already is a great variety available. With SDK 8 we are adding two additional items that you can manually control if needed. Developers can now tell the camera which area of the camera image should be used to calculate exposure time and set focus. In a sample, we demonstrate that usage in a tap-to-focus scenario.

With this release, Android developers can use Gradle 3 (used in Android Studio 3) without any limitations.

Endless AR possibilities

Wikitude SDK 8 is a big step for us to make the life of AR developers easier and help them create new ways of AR experiences. We are proud to announce the beta version of SDK 8 today and will continue to increase stability and performance in the upcoming weeks.

Start developing with Wikitude SDK 8

Getting started with SDK 8 has never been easier! Here’s how:

Download SDK 8 and sample app (included in the download package)

Select your license plan

Got questions? Our developers are here for you!

Help us spread the news on Twitter, Facebook, and LinkedIn using the hashtag #SDK8 and #Wikitude.