OE-2.3 Hearing, Touch & Audio Displays

What this is and why it exists

Spatial audio does more for the sense of being somewhere than most people expect, and it is cheap compared with visual fidelity. That makes this unit unusually practical: it is the best return per unit of effort in the whole subject.

It also contains the villain of the field. The vestibular system is the balance organ, and when your eyes report motion that it does not, you feel ill.

That mismatch is the single biggest constraint on designing for virtual reality. It is not a curiosity to note in passing.

The vocabulary

• Auditory pathway — the route from air pressure to a usable signal.
• Localisation — knowing where a sound came from.
• Interaural difference — the difference in time or level between the two ears.
• Audio display — delivering sound positioned in space.
• Vestibular system — the balance organ of the inner ear.
• Sensory mismatch — the senses reporting different things.
• Haptic — relating to the sense of touch.
• Receptor — a structure in the skin that responds to one kind of stimulus.

The mental model

Sound comes first, physics then mechanics, before any audio hardware. It is the sense most often treated as an afterthought and it does more work than people expect. The auditory pathway plays the same role here that the visual pathway played for sight: it tells you what the hardware has to deliver.

Localisation is the practically important part. You know where a sound came from mostly from differences between your two ears. A small difference in arrival time, and a difference in level. Both are shaped by the head being in the way. Reproduce those differences and audio stops being merely present and becomes positioned. That is what conventional stereo cannot do. It has no notion of which way you are facing, so the sound world turns with your head instead of staying still.

Then the vestibular system. It reports acceleration and orientation. When the eyes report movement and the inner ear reports sitting still, the two disagree. For many people the result is nausea. Treat this as a design constraint, not a trivia item. It is why comfortable virtual reality restricts certain camera movements and why teleporting movement exists. It is also why frame rate and latency are health matters rather than polish.

Touch is the third sense the subject uses and the hardest to reproduce. Understanding the skin explains something useful. What it is made of, which receptors it carries, and which respond to pressure, vibration, stretch and temperature. Some touch sensations are cheap to fake and others are not. Vibration is inexpensive to produce and reads convincingly as texture. Sustained force is expensive, because it means physically resisting a hand. That asymmetry sets up the hardware in the next topic entirely.

What you should now be able to explain or do

Outline the auditory pathway and say what it tells hardware to deliver. Explain how you localise a sound, and what conventional stereo lacks. Say what the vestibular system reports and why a mismatch with vision causes sickness. Treat that mismatch as a constraint and name design choices that follow from it. Describe what the skin contains and say why some touch sensations are cheap to reproduce and others are not.

Check yourself