visionOS

visionOS is a new operating system developed for the Apple Vision Pro augmented and virtual reality headset or "spatial computing device" as Apple calls it. We know a limited amount about visionOS at the current time because it is in beta testing and is not widely available, but this roundup aggregates what we've learned from those who have access and what Apple has shared.

With visionOS, Apple provides a full three-dimensional interface that starts out with a Home View that is similar to the iPhone's Home Screen. It displays all of the apps that are installed on the Apple Vision Pro, with the visionOS interface able to respond dynamically to natural light and shadows to help users interpret distance and scale.

visionOS is navigated using eye movements, hand gestures, and voice control because it cannot be directly interacted with. Users can, for example, browse through apps by looking at them, and then select one with a tap of the fingers. This navigation scheme has been described as natural and intuitive by those who have been able to test Apple Vision Pro.

visionOS apps can be placed anywhere in virtual space visible to a user, and they can be resized to be ultra large or small. App windows can be stacked on one another, arranged around a room, and manipulated at will. Multiple apps can run at the same time, and it is unclear if there is an upper limit beyond what fits in the visual space. Apple describes visionOS as offering an infinite canvas, but users can also opt to focus on a single app that expands fully into the space.

The level of immersion in visionOS is adjustable. Users can either see their content in their actual surroundings through the Apple Vision Pro's cameras, or a virtual "Environment" can make the real world fade away. Apple designed several Environments like Mount Hood that can be activated as a backdrop in visionOS.

Apple adapted existing apps like FaceTime, Photos, Messages, Safari, and more to work in a 3D environment. In Photos, for example, images can be viewed larger than life, and panoramas will wrap around the user. In FaceTime, life-size tiles for each person can be positioned around the room. For the headset wearer, visionOS creates a Persona, or a digital representation that reflects face and hand movements in real time. In Messages, Safari, and other apps, users will be able to pull out 3D USDZ files to look at and manipulate virtual objects.

With content apps like Apple TV+, visionOS can display TV shows and movies on a personal movie screen that feels 100 feet wide, plus visionOS offers Apple Immersive Videos that are 180-degree high-resolution recordings. visionOS supports Spatial Audio through the Apple Vision Pro's speakers.

There is a dedicated App Store for visionOS, which will house apps and content from developers. It will also provide thousands of iPhone and iPad apps that will automatically run on visionOS and that will be adapted to the eye and hand controls. With iCloud, content is synced with iPhone, iPad, and Mac.

Bluetooth accessories like keyboards, mice, trackpads, and game controllers work with visionOS, and it also integrates with the Mac to allow the Apple Vision Pro to serve as a display for a Mac. The Mac's screen shows up in the virtual space provided by visionOS for a private, portable 4K display.

For security and authentication, visionOS supports Optic ID, an iris scanning function that works like Touch ID or Face ID. Optic ID can be used for logging into visionOS, making purchases, and logging into websites.

visionOS is an operating system built from the ground up for the Apple Vision Pro, and it is Apple's first spatial computing OS. It is unlike anything we've seen before because it is designed to be used in a virtual space, but at the same time, it is reminiscent of iOS. It's basically bringing digital content into your physical space.

There is an iOS-like Home View that serves as the main hub, and it is akin to the Home Screen. It's where you go to access all of the content that is available on the Vision Pro, including apps that you have installed.

Software runs through apps, as on macOS and iOS. So if you want to watch a movie, play a game, or get work done, you need to open the appropriate app. It's an all-new experience, but it will be familiar to iPhone and iPad users.

visionOS features a full three-dimensional interface and all of the app icons, windows, and other objects respond dynamically to natural light and shadows to help you figure out the distance and scale of the virtual elements in the room.

App Windows

Apps open up like they do on the Mac, and you have an adjustable app window that you can position anywhere in the space around you. You can make windows big or small, and have as many windows open as you want around you.

You basically have an "infinite canvas" to work with, as Apple puts it, because all of the space around you is at your disposal. Apps can be right in front of your face, pushed further back in the room, tiled on top of each other, or spread around in the room.

A single app can fill up the entire area in front of you if you want to focus on one activity like a meditation.

Environments

Apple's cameras are designed to show you your surroundings so you can place apps and other digital objects in the room around you. With the Digital Crown, though, you can be transported into an entirely virtual space that blocks out the world.

Turning the Digital Crown adjusts your level of immersion, activating what Apple calls "Environments." Environments are virtual backdrops that take you to another place, such as Mt. Hood. Apple has described these as full, 3D like places that make the real world fade away.

Available Environments include Haleakalā, Yosemite, Sky, Joshua Tree, Lake Vrangla, Mount Hood, The Moon, Beach, White Sands, Winter Light, Fall Light, Summer Light, and Spring Light.

Screen Sharing

The Vision Pro will support screen mirroring through AirPlay or FaceTime, which could allow Vision Pro headset owners to share what's on their display with others through an external monitor or TV.

Controlling visionOS is done through eye movements, hand gestures, and voice commands. You basically look at something to highlight it, and then use a gesture to interact with it. The available gesture options are listed below.

• Tap - Tapping the thumb and the index finger together signals to the headset that you want to tap on a virtual element on the display that you're looking at. Users have also described this as a pinch, and it is the equivalent to tapping on the screen of an iPhone.
• Double Tap - Tapping twice initiates a double tap gesture.
• Pinch and Hold - A pinch and a hold is similar to a tap and hold gesture, and it does things like highlighting text.
• Pinch and Drag - Pinching and dragging can be used to scroll and to move windows around. You can scroll horizontally or vertically, and if you move your hand faster, you'll be able to scroll faster.
• Zoom - Zoom is one of two main two-handed gestures. You can pinch your fingers together and pull your hands apart to zoom in, and presumably zooming out will have a pushing sort of motion. Window sizes can also be adjusted by dragging at the corners.
• Rotate - Rotate is the other two-handed gesture and based on Apple's chart, it will involve pinching the fingers together and rotating the hands to manipulate virtual objects.

Gestures will work in tandem with eye movements, and the many cameras in the Vision Pro will track where you are looking with great accuracy. Eye position will be a key factor in targeting what you want to interact with using hand gestures. As an example, looking at an app icon or on-screen element targets it and highlights it, and then you can follow up with a gesture.

Hand gestures do not need to be grand, and you can keep your hands in your lap. Apple is encouraging that, in fact, because it will keep your hands and arms from getting tired from being held in the air. You only need a tiny pinch gesture for the equivalent of a tap, because the cameras can track precise movements.

What you're looking at will let you select and manipulate objects that are both close to you and far from you, and Apple does anticipate scenarios where you might want to use larger gestures to control objects that are right in front of you. You can reach out and use your fingertips to interact with an object. For example, if you have a Safari window right in front of you, you can reach your hand out and scroll from there rather than using your fingers in your lap.

Eye and gesture-based navigation is demonstrated in onboarding videos that were pulled from the visionOS software. Apple walks users through looking at a system element and selecting it with a tap of the fingers. The headset is able to detect hand movements even when the hands are resting in a comfortable position.

Developers will be able to create custom gestures to be used with visionOS in their apps, and there are also accessibility accommodations.

There is also an air typing feature that is designed to let you type on a full virtual keyboard. There is no tactile feedback, with Apple instead focusing on visual feedback when a key is "pressed."

Input Devices

Though visionOS is meant to be controlled with hand gestures and eye movements, it does support Bluetooth mice, keyboards, and trackpads, along with game controllers for playing games.

Mice, keyboards, and trackpads can be used alongside a Mac.

Apple is reimagining many of its familiar apps for a 3D interface, including FaceTime, Photos, Messages, and Safari.

FaceTime in particular is of interest because it will show the people you're conversing with in large tiles around the room, which you can adjust. People you're chatting with will see your "Persona," a scan of your real face that reflects your facial expressions and hand movements in real time so you look like yourself rather than a person wearing a headset.

The Photos app will show you images in larger than life scale, with panoramas wrapping around you, for example, plus you can view 3D photos and videos. There's a built-in camera on the headset so you can capture video that you can watch back in full 3D.

With Messages, Safari, and other apps, users can pull out 3D USDZ files to look at and manipulate virtual objects.

iPhone and iPad Apps

There is a dedicated visionOS App Store where you can find apps created specifically for the device, but it will also allow you to download existing iPhone and iPad apps. iPhone and iPad apps will run on visionOS in a 2D environment, so you can use your favorite apps even if they're not updated for a 3D interface.

Entertainment

Apple TV+ and other apps like Disney+ will provide a variety of TV and movies that can be watched through the headset. Apple says that the effect is like a personal movie screen that feels 100 feet wide.

Apple is creating Apple Immersive Videos that are 180-degree high-resolution recordings that put you right in the action, plus 3D movies will come to life on the headset. Spatial audio is also supported for a surround sound experience.

Gaming

visionOS includes support for Apple Arcade games that can function with a connected Bluetooth gaming controller, plus Apple is working with gaming partners to create new gaming experiences for the device.

App Store

The visionOS App Store was added in the fifth visionOS beta, allowing developers with the device to try it out.

Visual Search

visionOS includes a Visual Search feature that can be used to get information about an item, detect and interact with text in the world around them, copy and paste printed text from the real world into apps, translate text between 17 different languages, and more.

Real world text that includes contact information, webpages, and unit conversions and similar information can be acted upon in visionOS. So, for example, if a printed handout has a website link in it, you can scan the link with the Vision Pro, opening up a Safari window to view the website. Or, if a recipe calls for grams and you need ounces, you can convert using the headset.

Real-time text translation will also be useful for traveling and other instances when you might want to quickly translate what you're seeing in the real world. The ‌Apple Vision Pro‌ headset will be able to automatically detect text and documents, similar to how the ‌iPhone‌ can detect text in photos and allow it to be interacted with.

Travel Mode

According to visionOS code, there is a Travel Mode that will be introduced for using Apple Vision Pro on airplanes. Not much is known about Travel Mode, but it will restrict some sensors and will require users to sit in a stationary spot to use the headset.

The Vision Pro integrates with the Mac, serving as a display for either a notebook or desktop machine. You can see all of your Mac apps in a virtual space, using a keyboard and mouse as usual to get work done.

Note that you can use the display with a Mac mini or a Mac Studio that does not have its own display. A secondary display is not required.

Rather than Face ID or Touch ID, visionOS uses Optic ID, an all new iris scanning security feature. Apple uses infrared lights and cameras inside the headset to look at the unique pattern of your iris for authentication purposes.

Optic ID can be used for logging into visionOS, making purchases, and in lieu of passwords in apps and websites.

visionOS will be available when the Apple Vision Pro launches in early 2024, with current rumors pointing to a February debut. Several visionOS betas have been introduced, but outside of Xcode, the betas are only available to developers who have a Vision Pro Developer Kit for app creation purposes.