Sonic Body

Sonic Body is a research prototype and pedagogical proposition developed for a candidacy presentation at Hochschule für Musik und Tanz Köln (HfMT) in 2025 — a position focused on digital innovation in teaching. The project uses body and face tracking as a real-time synthesis interface, but the deeper argument is conceptual: tracking is not a neutral technology, it is a historically formed way of seeing. The presentation — Wie Tracking unsere Wahrnehmung verändert — placed the prototype inside a media-archaeological genealogy that runs from Marey and Muybridge to MediaPipe.

The argument

Every technical system carries a logic of perception embedded in its material. A camera does not photograph — it measures light. A pose-estimation model does not see a body — it calculates probabilities over a skeleton of points. Tracking systems write bodies into data, and in doing so they shape what a body can be and do inside a technical environment.

The media-archaeological framing makes this visible:

• Muybridge and Marey — chronophotography as the first attempt to decompose movement into measurable units. The body becomes a data source before the term existed.
• Laban notation — movement as a written language: a time-based score for the body, structurally analogous to what tracking produces automatically.
• Forsythe's Synchronous Objects — choreographic data rendered as density maps and topographies of duration. Movement as mass, not line.
• Daphne Oram / Lichttonfilm — optical sound systems that translate drawn traces into audio. The loop between body, image, and sound that Sonic Body closes in the browser.
• Cybernetics and feedback — the body not as source but as node in a feedback system. Tracking, gesture recognition, AI: all cybernetic structures.
• Paul De Marinis, The Edison Effect — laser-read wax cylinders recovering lost voices. Tracking data as sonic archive: each movement leaves an imprint, a kind of sound-fingerprint of the body.

The prototype

Sonic Body runs in the browser. MediaPipe Pose and Face Mesh produce real-time metrics — joint angles, body axes, symmetry deltas, center of mass, hand kinematics, head orientation (yaw, pitch, roll), mouth aperture, blink — which feed a declarative mapping pipeline. Each mapping chains operators (normalize, smooth, scale, quantize, clamp, exponential, threshold) that transform raw tracking data into synthesis parameters.

Seven Web Audio engines sit at the output: granular synthesis, FM with ADSR envelope, wavetable oscillator, Karplus-Strong plucked string, convolution reverb, delay, and a step-sequencer sampler. A safety layer (limiter, compressor, auto-gain) runs downstream.

The mapping layer is the compositional core. The instrument is not the synthesis engine — it is the relationship between body metric and sound parameter. FM Bite, for instance, maps hand-movement asymmetry to FM modulation index: the more asymmetric the gesture, the more aggressive the timbre. The body does not trigger sounds — it modulates them continuously.

The Lehrprobe

The presentation opened with a live demonstration: a PoseNet skeleton appearing on screen, moving with the presenter. "The machine sees only points, lines, probabilities. What do you see — a body or a data structure?" The question was not rhetorical. It framed everything that followed.

Participants were invited to stand in front of the camera and move freely, hearing how the sound responded. The FM Bite preset was demonstrated as a case study: movement as modulation, not as command.

The concluding proposition was pedagogical as much as technical: code as cultural practice. Not a question of who can programme, but of how technical systems carry history inside them — and how engaging with that history produces different kinds of participation. Teilhabe statt Kontrolle. Participation over control.

Position within the Lab

Sonic Body belongs to the Lab line Body as instrument — physical interaction as an artistic and technical problem, not a UX decision. It also marks a moment of institutional engagement: the Lab's research methods tested in a teaching context, and found to hold.