USDZ Enabling New Experiences for ARKit

At WWDC 2018, Apple announced various improvements to its support for Augmented Reality experiences and one of those was the support of a new file format, USDZ. USDZ enables new experiences and adds new ways in which users can interact with Augmented Reality content.

I’ve written more about WWDC 2018 here.

In this article, I will dispel some myths about the USD(Z) format, talk about how it works with respect to ARKit, and point out some differences which I found while working with it and writing conversion tools for it. We will slowly dissect one of the USDZ examples provided by Apple to understand the format when applied to ARKit and QuickLook.

There’s a lot to cover, so in the next articles of this series I will explain the nuts and bolts of the USD specification and file format from a more generic point of view. Thus, you can work with it and create USD scenes programmatically via either Python or C++ from any platform, not necessarily a Mac.

What is USD(Z)?

Universal Scene Descriptor (a.k.a. USD) is:

Let’s translate that to English.

Very simply put, USD is what you would call a 3D asset file format with some special additions. One of these is the ability to package a USDC file with its textures into a USDZ file for consumption by ARKit and QuickLook.

What does all this mean for the AR industry and ARKit?

USDZ files can, for example, be shared via iMessage or you can open them directly from a webpage with Safari:

You can visit this page from Safari on iOS to try this out yourself.

The coolest thing about these two examples is that they show how QuickLook allows users to experience AR content without having to install any new app.

This is huge: imagine being able to see any product in 3D, in the real world, before you buy it from your favourite online shop, without having to install any new app! This would not be possible without USDZ’s support from Apple.

Another use I am experimenting with is to showcase/share medical 3D models easily: send them to your friends with an iOS device and they can immediately check out your work, no apps needed as well. There are a lot more use cases for USDZ and QuickLook.

USD is developed by Pixar and is used in their software, from Hydra to Presto. Pixar made its implementation open source some years ago on Github, so now anyone can use it to implement conversion tools or to integrate it in their engines and software.

A single USD file can be thought of as representing a scene, and a USD scene is a hierarchy of what Pixar calls Prims (which intuitively can be thought of as nodes).

USD performs almost the same function that, say, the gLTF or COLLADA file formats do, but compared to these two other example formats, it has both more and fewer features.

For example, one of the main differences is that gLTF is starting to become widely supported, whereas USDZ is currently only supported officially on iOS’ QuickLook.

Getting the sample USDZ asset

For the purpose of this article, I will assume that you are on MacOS and you have setup your USD Python tools to be used from the command line in your terminal as explained here after downloading them from here.

Visit this website from your browser, click on the guitar 3D model and save it somewhere sensible.

Extracting a USDZ asset without renaming it

I’ve read online that most people when they want to access the files contained in a .usdz file, they first rename it to have a .zip extension and then extract it from Finder.

Instead, we can use the command line tool unzip to extract an USDZ file directly as such:

You can now navigate to your extracted folder and directly open the .usdc file:

Before we continue, let’s understand what all these different file formats mean.

What’s the difference between USD, USDC, USDA and USDZ?

Files with .usd, .usdc and .usda extensions all represent a USD scene file, but the way they store this data is different.

Recall that a USD file can be thought of as representing a scene, and a USD scene is a hierarchy of nodes. Most of the time you will have a 1-to-1 correspondence between a USD file and a 3D model, instead of a scene; i.e. you will have one single mesh in the file.

.usda

.usda files are USD scenes stored on file in ASCII format. Hence, these files are text files containing the scene description in a manner that is human-readable. For example, the following is the .usda file for the representation of a USD scene containing a simple cube:

As you can see the format is quite readable and makes it easy to understand how the scene is organised.

However, this format is not efficient as it takes a long time for large files to be parsed into a text file. For this reason, Pixar also specifies the .usdc file format, which allows software to read USD scenes much more quickly.

.usdc

.usdc files are the binary representation of USD scenes. Hence, they are stored in binary form, and opening them with a text editor is not of much help. Here is the cube scene example again, this time after opening its .usdc equivalent with a text editor:

However, these files can be read by softwares much more quickly than their .usda equivalents. In fact, USDZ packages for QuickLook must use .usdc for the scenes to be loaded quickly on mobile devices.

.usd

.usd files can be both ASCII-based or binary files. Their extension doesn’t give away what format they are from the outside, but systems reading them can determine their format from the first bytes of the file. Hence, .usd files can be both text-based or binary based, i.e. you can rename either a .usdc or .usda file to .usd and they will work like how they were working before you renamed them.

.usdz

This format is used by QuickLook to render scenes and can be imported and used in iOS projects developed in Xcode.

In its current form, it’s more or less a zipped folder containing at most (and at least) one .usdc file and any textures which that scene uses.

You can learn how to create USDZ files from OBJs here.

Exploring the Stratocaster USDZ asset

Now that you have extracted the Strato, you can convert its .usdc file to .usda as such:

Open the Stratocaster.usda file with your favourite text editor, and boom:

We can now read the contents of the file to understand how USDZ assets are organised for consumption by QuickLook.

Open the newly created Stratocaster.usda by double clicking on it so that the changes we made to it are shown in real-time.

Materials

Just at the top you can see the definition of the various materials for the asset. Some of these use constant properties, whereas the others, such as this one, use a texture map:

For usage with QuickLook, the names of the attributes (a.k.a. the material properties) in the various pbrMat1 need to be exactly the same as shown in the example above. Changing the name of diffuseColor to diffuse in the material “StratocasterGuitar_SynthString_Material” (not the one shown above) results in the loss of albedo colour for the components using that material:

Notice the strings on the left: they are missing albedo information.

For every material attribute that uses a texture instead of a constant value, QuickLook expects a material Prim like this one:

and you also need to provide a texture sampler:

This is what the material definition looks like when taken as a whole:

Scene Graph and Meshes

Looking at the expanded Scene Graph for the Stratocaster model, one thing is immediately noticeable: seems that elements are repeating.

What’s the difference between the “splitContainer” section and the elements ending with “Material”?

If we go back to our .usda file, we can look for the text “StratocasterGuitar_ButtonTone_Material”, and here it is:

Submeshes are represented by GeomSubset Prims in USD. They are used to specify subsets of the whole model to which to apply a material exclusively.

If we try to look for any of the “splitContainer” elements, we wouldn’t find them in the file. It is because they are auto-generated from the indices of a given submesh, and they use the material they are assigned via the rel material:binding relationship.

The indices listed above for the ButtonTone element are indices into faces, instead of vertices. They refer to the global face indices for the whole guitar mesh, which can be found further up in the file:

The “StratocasterGuitar” Prim also defines the elements composing a vertex: positions, normals, texture coordinates, and etc, which are then indexed via vertex indices.

Materials in USD

The USD specification doesn’t explicitly forbid the usage of non-PBR materials, but iOS’ QuickLook doesn’t work with non-PBR materials. So, make sure that your USDZ file uses PBR materials or QuickLook will not render them properly.

Why is PBR important?

PBR stands for “Physically Based Rendering”. PBR enables a variety of enhancements on how realistically objects can be rendered. One of these is the ability to use image-based lighting, which provides more accurate ambient diffuse and specular reflections.

The differences between “traditional” shading and PBR shading can be stark:

You can learn more about PBR from here and here.

In Augmented Reality, PBR is especially relevant as it allows the 3D models being rendered to be shaded in a way which is more aware of its surroundings. For example, if you were to render a USDZ model of a shiny metallic cube, with PBR materials and QuickLook the cube would reflect the environment around it as if it were a real one:

Current Issues and Limitations with USDZ

Toolset

One of the biggest pains by far is the lack of good tools for artists to export their creations from their asset creation software to USDZ directly and without having to use the command line tools. Command line tools are not for everyone, and most of all the provided ones work with only a handful and outdated intermediate asset formats, making it hard to really exploit the functionalities of the USD format.

Interactions

In their current form, neither the USD or USDZ specifications talk about the ability for the user to interact with the 3D scene described by a USDZ file. Hence, all that’s possible to do at the moment when opening a USDZ file in QuickLook is to look at it, and nothing else. For example, tapping on it won’t trigger any actions and such actions has no way to be coded into the format anyway.

You can bake animations into the asset, but those are pre-baked and not interactive.

PBR only

You can’t use non-PBR materials when combining USDZ with QuickLook. Note that nothing stops you from declaring Attributes (in USD-speak) on your Prims which have names resembling the material properties you’d find on, say, a Lambert material, but they won’t work with Pixar’s renderer or Apple’s.

This is a limitation especially when you want to display widgets/abstract 3D objects rather than 3D models of realistic objects.

Apple products only

So far USDZ hasn’t been adopted by Android or other platforms, and hence it can only be consumed by users with iOS and MacOS devices. On other platforms, gLTF and gLB seem to be the main candidate to become the USDZ equivalent.

Moreover, QuickLook is only available on iOS. Although third-party solutions exist, there isn’t an equivalent native solution on Android. But you can be sure that if and when it arrives, it will use the gLTF format and will display the content slightly different from QuickLook.

My predictions

AR is inherently averse to the concept of apps: you don’t want to switch to a different app every time you want to access a different AR layer or service, let alone download and install an app which you’re going to use only when at a specific location.

For this reason, the fact that USDZ allows users to interact with AR content without having to install an app is a step forward in that direction, albeit a small one. To really move in the direction of an AR cloud, we need to be able to, at least, add interactions to experiences such as the ones offered by USDZ.

This calls for a solution closer to a web browser than an app store; Mozilla and others are working towards that solution with WebXR.

Given what the USD format is intended as a format to represent 3D data, but not interactions, I don’t think that Apple or Pixar will modify the specifications to introduce interactions and scripting.

Instead, Apple will either have to bite the bullet and allow users to work with something similar to the WebXR/AR or extend QuickLook/Safari to integrate scripting of the USDZ scenes available.

Apple will have to gradually introduce better tools to author USD assets if it doesn’t want to officially support gLTF. So, we might see some announcements with respect to that at WWDC 2019.

Conclusions

USDZ, when combined with ARKit and QuickLook allows you to:

• let users try out 3D models of realistic objects without having to install apps (which is great for e-commerce applications);
• send 3D models to friends via iMessage;
• package USD scenes into a portable format; and
• render objects in 3D realistically thanks to PBR;

By far the most promising usage for USDZ files is the first one on the list: users can try out products in real-life scale and with a lighting environment that makes them look the same as the final product as much as possible: