Apple Pencil and Spatial Pen

What This Is

Apple Vision Pro does not need a stylus for everyday use — eye tracking and the system pinch gesture are usually enough. But for fine-grained input — placing a cursor inside a single character cell, drawing on the VKeyboard for swipe typing, or operating a precise UI — vmux supports a "spatial pen" model. Your pinch becomes the tip, your hand orientation becomes the angle, and vmux uses that pose to drive the cursor with sub-millimeter precision.

If you have an Apple Pencil paired with an iPad nearby, the same input pipeline accepts the iPad's pencil events when you are working in the iPhone or iPad version of vmux. On Vision Pro the "pen" is your hand.

How Vision Pro Pen Input Works

Three signals are read from each pinch gesture: position, orientation (tilt and roll), and pressure-equivalent (how long the pinch is held).

• Position is the screen-space coordinate where your gaze and pinch converge. This is the cursor location.
• Orientation is the 3D pose of your hand. vmux uses the angle between the pinch axis and "down" to compute two things: a small cursor offset (so the cursor sits where the tip of the pen would be, not where your fingers are) and a stroke-width multiplier when you are drawing.
• Pressure-equivalent is duration plus pinch tightness. vmux does not currently use this for terminal input, but it feeds into the spatial keyboard's swipe decoder.

You do not have to hold a real pen. The model just makes pinch input feel like a pen — it pivots correctly when you tilt your hand.

Where Pen Input Applies

Pen input is most visible in three places:

Plain text entry on the terminal does not use pen input — keyboard remains the fastest path. Pen input is for cursor work.

Pen vs Eye-Gaze

Eye gaze is always on. It moves the focus indicator wherever you look. Pen input layers a finer cursor on top of gaze when a pinch is active.

This is why a swipe across the spatial keyboard feels different from a series of taps — once you start a stroke, your hand drives until you let go.

Tilt Behavior

When you angle your hand, the cursor offsets in the direction of the tilt. This emulates how a real pen tip is offset from where your fingers are gripping the barrel.

• The default sensitivity is 40 points of cursor offset per unit of tilt deviation. Most people never need to change this.
• The offset is purely 2D — vmux projects the 3D tilt onto the screen plane, then translates the cursor by the projection.

For the spatial keyboard's swipe decoder, tilt also drives stroke width. A flat hand makes thin strokes. A steep tilt makes thicker strokes — useful as a visual confirmation that the system is reading your pose correctly.

Smoothing

Drag input is smoothed with an exponential moving average. The effect is what you would expect from a tablet: tiny jitters from your hand or the eye-tracker are filtered out, but real motion comes through. The smoothing factor (0.26 by default) is tuned to track quickly enough that the cursor never feels laggy on the spatial keyboard.

vmux also rejects "teleport" samples — if a single frame jumps more than 1000 points, it is discarded as a tracking glitch. This means a momentary loss of hand tracking will not jerk your cursor across the screen.

Related

• VKeyboard — the spatial keyboard, where pen input matters most
• Mouse and Selection — how vmux handles pointer events in the terminal
• Voice Commands — alternative hands-free input