Tag: Augmented-Reality

An API (Application Programming Interface) allows applications to communicate. Serving as an access point to an app, API enables users to access a database, request and retrieve information, or even alter data on other applications.

In this article, we get you familiar with the functionality of the Wikitude Studio and explain how you can benefit from using Studio API in your AR app.

Introduction to Wikitude Studio API

Wikitude’s Studio is an easy-to-use, drag and drop, web-based AR content manager. Using Studio, you can easily create two types of targets(image targets (2D items) and object targets (3D items)) to further augment for your JS-based app. On top of that, you can add simple augmentations to test your targets and their position. 

Not sure if the image target quality is good enough? Use the rating indicating the quality of an image target as a guide. Wikitude Studio API also enables conversion of image target collections to cloud archives and their management, making it possible to work with cloud-based recognition instead of on-app recognition. You can create and host your AR projects in Studio and link them directly to your app without exporting and pushing app updates.  

What does the Studio API do? 

Studio API allows you to access all the functionality mentioned above without logging in to Wikitude Studio. You can have your app or system programmatically communicate with the engine behind Wikitude Studio. The keyword here is “programmatically,” meaning the flow enables simplified app development, design, and administration and provides more flexibility. In practical terms, it allows users to quickly scale up and integrate AR capabilities into existing architecture. 

How can Studio API benefit your business: use cases

Now, let’s see some real situations where Studio API can come in handy.

• Create the project for each of the targets your customers upload in your CMS

Studio API can be integrated into your own CMS, making it easier to maintain collections and content automatically. Say you run a printing photo service and an accompanying app. The end-customer can upload pictures and add digital content associated with that photo: a video, a song, an animation, or GIF. By scanning a printed image with the app, the customer can access an AR experience that enhances a memory or a moment captured on the photo. 

Creating the image targets, assigning augmentations to the targets, and publishing content can be managed programmatically, enabling you to design the user interaction the way you want to. 

Similar functionality could be used by a postcard service, corporate merchandise producers, and other services. 

• Easily manage image targets and have your app make updates in the background  

When working with fast-changing content, numerous images, and heavy augmentations, we discourage storing your targets and augmentations in the app. Offline recognition will force you to redeploy the app frequently and make the size of the app massive. That’s where Wikitude cloud-based recognition comes to the rescue.

Imagine a publisher (just like PAPER.plus) issuing analog books and magazines with an extra AR layer. Such a service can have one app giving access to all the AR experiences associated with each printed item. As the new books and magazines are published, the publisher simply programmatically adds fresh digital content to the server, making it available for the users in the app via cloud-based recognition. 

PAPER.plus

Wikitude cloud-based recognition provides an opportunity to work with a target collection containing up to 50,000 images. Otherwise, you are limited to 1,000 target images per active target collection, and only one group can be active at a time. This flow can lead to a longer recognition speed, and the end-user will need to switch manually between collections. The functionality could be extended to many other fields, such as education, tourism, art, and culture.

• Integrate the AR functionality with already existing architecture and automatically grab data from that closed system 

The Studio API can also be used for 3D items. By having the 3D models and the material file of a machine, a robot, or part of an assembly line, you can use that information and render images of that specific machinery. The Studio engine will automatically process those images via the API to create object targets, while the API will help position augmentations. 

Using such AR experience lets employees detect and precisely localize malfunctions on the production line by grabbing data from other parts of the system, such as live sensors, measurements, and machinery history. The factory can leverage its existing training material or repairment specifications and overlay AR instructions on the machines, reducing the time required to identify and fix issues. 

Wikitude Plugins API 

The Plugins API allows extending the Wikitude SDK by 3rd party functionality. It enables users to implement external tracking algorithms (such as image tracking, instant, and object tracking) to work in conjunction with the Wikitude SDK algorithms. Additionally, you can input camera frame data and sensor data into the Wikitude SDK, which will take care of the processing and rendering. Our compatible plugins are written in C++, Java, or ObjC and can communicate with JavaScript, Native, and Unity plugins. Please note we currently don’t provide support for the extension SDKs like Cordova, Xamarin, Flutter. 

• Integrate with OCR and code readers 

What else can you achieve? The Plugins API can trigger AR content via QR code and barcode reader or add text recognition. Our client Anyline‘s text recognition API allows apps to read text, gift cards codes, bank slips, numbers, energy meters, and much more. The company’s solutions have been used by Red Bull Mobile, PepsiCo, and The World Food Program.

Anyline barcode reader

• Build remote assistance app by leveraging Wikitude’s instant tracking  

Typically, our engine is set up to recognize targets in the camera feed. With the Plugins API, you can set specific images as input rather than grab them from the camera feed. Where does that come in handy? That is an implementation-specific for remote support solutions, where one needs to broadcast a user screen to another user. Scope AR used this functionality when launching WorkLink Remote Assistance, their AR remote assistance tool. They required a markerless tracking provider to complement the plugin they created, and we were happy to support it with our technology.

• Augment the human body 

Another use case that we’ve often encountered is adding face detection, hand detection, or body detection. To use it, you need a library specialized in one of those detection functions and plug Wikitude in it. It will take over the processing and rendering of AR content. Watch our detailed face detection sample to learn more.

Connecting with a face tracking library via the Plugins API is not the only way to create this type of AR experience in combination with Wikitude. Alternatively, you could access Face and Body tracking from ARKit or ARcore via AR Bridge in our Unity Expert Edition SDK.

Wikitude AR Bridge

As you already know, an API (Application Programming Interface) allows applications to communicate with one another. Wikitude’s AR Bridge, part of the Unity Expert Edition SDK, has similar functionality: it provides access to the tracking facilities of other native platforms, e.g., ARKit and ARCore. The AR Bridge enhances Wikitude’s image and object tracking by tapping into external tracking capabilities. There are two options:

• Internal: this is a direct communication to ARKit and ARCore maintained by Wikitude and, at the moment, it offers basic positional tracking (no plane detection or more advanced tracking); 
• Plugin: allows an indirect connection to any tracking facility, pre-existing or written by developers.  As an example, we provide integration with Unity’s AR Foundation plugin. 

We provide a ready-made plugin for Unity’s AR Foundation that developers can use immediately. The plugin uses AR Bridge to inform the SDK about tracking. All current and future AR Foundation features work similarly to what we referred to as 3rd party libraries in the Plugins API context.  

However, plugins can be developed by anyone, not just by Wikitude. For example, a company is building glasses with its tracking system and integrating with Wikitude. Since they are not using ARKit or ARCore, the internal AR Bridge won’t be a default. Instead, they can create their plugin and have this custom solution work fast inside our SDK.

Ready to start building today? Access our store to choose your package or contact our team to discuss which license is the best match for your AR project.

If you are a UX/UI designer who builds user experiences in digital environments, chances are you will be working with augmented reality sooner than you think. As AR applications rapidly break into the mainstream, making the user feel in control of a product becomes even more critical in user experience design.

This article breaks down the role that user experience design principles play in augmented reality application development, with a specific focus on UI design.

The article is based on a presentation by our senior software engineer, Gökhan Özdemir, for the “UX for AR” webinar. Watch the full recording here.

What is UX design for augmented reality? 

User Experience Design, or UX, is the process of designing a product that considers the needs of the users and makes the user flow as seamless and intuitive as possible. Good UX always starts with putting the user at the center of the design process. It also relies on the principles of human psychology and empathy.

Now, what about UX for AR?  In augmented reality apps, success means offering a great user experience through a seamless blend of hardware and software. 

Augmented reality experiences are overlaid on the real environment, so the user experience is spatial and highly contextual. It makes designing UX for AR more challenging as designers need to think through spatial experiences. Getting it wrong can mean users have a less than stellar experience – and no one wants that. 

Getting started

User design can be tricky. Designing for a new technology that is only getting traction? Even trickier! Let’s explore the role of user experience (UX) design in AR applications — how to think through your user experience as a designer and navigate the technical decisions when creating an AR app. 

You will learn how to create a compelling user experience for your AR application that considers the physical space and natural human interaction. 

Five pillars of UX design for augmented reality

Users prefer to interact with elements of an interface discreetly, not to be reminded of what the interface contains. This is different from the traditional user experience (UX) associated with conventional websites and mail applications. The UX for augmented reality (also known as 3D user interface) concept emphasizes interaction and visual interest above all else. Users are interested in entering the virtual space and are not distracted by surroundings that are not real.

Our five common UX design pillars for AR will help you define the considerations you’ll need to make when designing your UI and experiences for virtual objects.

Kick-off your design process by considering these criteria:

While it’s crucial to consider the first two pillars (environment and movement) designing for AR, the last three (onboarding, interaction, and UI) are equally crucial for both 3D and traditional 2D screen space UI.  

Environment

As augmented reality experiences are spatial and always interconnected with the real world, the environment plays a key role in the design process. The environment can be broken down into four most common categories of space, defined by the distance from the user.

Examples of AR in the intimate space include face filters (like Snapchat or Instagram), hand tracking, or hand augmentations if you use a head-mounted AR display. 

Moving to personal space, augmented reality experiences might feature real objects, people, or the area around you. Featured in the video below, you can see a learning-focused AR experience that uses educational models to animate the chemistry concepts through an interactive digital layer.

AR experience in personal space

Another example of using augmented reality in personal space is popular table-top and card games and augmented packaging. Think augmented pizza boxes or collectible cards with augmented characters that interact with each other.

Next up is the social space. If you pan the camera further away, you will target the area that can be occupied by other people, unlike in a personal space where you have more privacy. This space segment is widely used for multiplayer AR games or augmenting objects on a scale, from the furniture to monuments and buildings.

In many cases, AR experiences in public space are anchored to specific locations with enough area to place an augmentation or sites that should be tracked in AR. The mumok AR experience in Vienna is a perfect example of the AR in public space where the entire building is tracked, using the Wikitude Scene Tracking feature.

mumok AR

Movement

The success of any new product or service directly depends on how well it integrates with today’s users’ minds — both physically and psychologically. Movement makes the next UX design pillar. When you design the experience, you want to use the area around the user most of the time.

As smartphones and head-worn devices give a limited view into the environment, the designer’s primary task is to guide the user. By including the navigation elements on the screen, you will be steering the user’s gaze, helping them get around and move along the experience. 

While you are visually guiding the user, it’s essential to keep in mind not to dictate to go in specific directions. This might lead to unwanted hiccups in the experience or even cause accidents. 

Onboarding

The next pillar we are going to explore is the user onboarding. Creating user-friendly and engaging augmented reality experiences can be a challenge. It’s not enough to just put some markers around your location or overlay some information on top of an image. You need to understand what the user is looking at and how they are using it. When creating the AR experiences, keep in mind that the most important thing for your users is not accuracy but usability. 

Another factor to consider is that different devices have various technical limitations in supporting AR features. Markerless AR, for instance, would require the user to move the device around, so that computer vision algorithms could detect different feature points across multiple poses to calculate surfaces.

The scanning process takes no time for newer devices with an in-built LiDAR sensor (like iPad Pro). But for other devices, your users might appreciate a comprehensive onboarding UI. The pop-up menu or instructions should guide the user on the following steps to successfully launch and run an AR experience.

To launch a tracking algorithm, you might want to use a sketched silhouette of the desired object to provide a clue on the shape and pose to prompt the user to align the view with the real object. Read more about the Alignment Initializer feature in our documentation.

Alignment initialization

Taking the onboarding offline, sometimes the physical methods like signage are used to communicate about the AR app, provide a QR code for quick download and mark the exact standpoint for an optimal experience. 

Interaction

Once the AR experience is launched, we are transitioning to another UX design staple – interaction. During this phase, your user will benefit from the intuitive and responsive interaction. When designing for touch, you are most likely be using these most common gestures and prompts: 

Responsive interaction means taking into account the distance from the desired objects to the camera, which will define how easy or difficult it is for the user to interact with it. To facilitate the interaction with farther placed objects, consider increasing the sphere’s bounding box to make it less dependent on the distance to the camera.

Minimizing input via finger might also be a good idea, especially when designing for tablet users. As most of the tablets are held with two hands, some UI or interaction elements placed in the middle of the screen will be very hard to interact with. Instead, use gaze input like triggering intro, interactions, or buttons in the augmented space by looking at them long enough. You might know this from VR where you don’t have any controllers and experiences are mostly driven by gaze. 

Consider using accessibility features, especially if you are designing for a broader audience. This way, you let the user rotate or reset the position of an augmentation instead of walking around it.

UI (User Interface)

The final principle we want to highlight is UI, which consists of augmented and traditional screen space. Depending on the use case, you will be using them interchangeably. While UI in the augmented space boosts immersion as the user perceives it as part of the experience, screen space UI is sometimes easier to read and interact. 

Designing with humans in mind

AR can improve people’s lives simply by allowing them to experience something that wasn’t possible before. Applying UX principles to AR can help designers create experiences that are clear, integrate easily into daily life, and create powerful emotional responses.

The guidelines we’ve shared aren’t magic bullets, but they do place fundamental guidance around where designers should be focusing their attention when crafting an experience for a user of any age.

What is your take on using UX principles when designing AR experiences? Let us know via social media (Twitter, Facebook, and LinkedIn) and tag @wikitude to join the conversation.