Scanning Spaces: Paradigms for Spatial Sonification and Synthesis
Ryan Michael McGee
PhD Disseration Defense
June 9th, 2015
In 1962 Karlheinz Stockhausen's "Concept of Unity in Electronic Music" introduced a connection between the parameters of intensity, duration, pitch, and timbre using an accelerating pulse train. In 1973 John Chowning discovered that complex audio spectra could be synthesized by increasing vibrato rates past 20Hz. In both cases the notion of acceleration to produce timbre was critical to discovery. Although both composers also utilized sound spatialization in their works, spatial parameters were not unified with their synthesis techniques. This dissertation examines software studies and multimedia works involving the use of spatial and visual data to produce complex sound spectra. The culmination of these experiments, Spatial Modulation Synthesis, is introduced as a novel, mathematical control paradigm for audio-visual synthesis, providing unified control of spatialization, timbre, and visual form using high-speed sound trajectories.
The unique, visual sonification and spatialization rendering paradigms of this dissertation necessitated the development of an original audio-sample-rate graphics rendering implementation, which, unlike typical multimedia frameworks, provides an exchange of audio-visual data without downsampling or interpolation.
Visual shape more important than color, color enhances shape
Stockhausen and Chowning both created a continuum between acceleration and timbre and both made extensive use of sound spatialization,
but technical limitations left spatial parameters separate from timbral control paradigms.
However, physical principles such as Doppler shift and distance-based gain-attenuation provide an intrinsic, mathematical connection between moving sound sources and FM/AM.
The symmetrical Doppler shift approximation formula ubiquitous in the field of digital audio is physically inaccurate at high-speeds over approximately 100 m/s (225 mph).
While this approximation is sufficient for existing Doppler-based effects that do not require such high-speed source motion, it prevents a fully realized connection between spatialization and high-index FM timbres.
Current multimedia software frameworks provide inadequate control rates for spatialization trajectories which limits the ability of spatial modulation to achieve high frequencies of modulation.
Sample rate spatialization choreography is necessary to achieve audio-rate frequency and amplitude modulation from simulated spatial motion.
Likewise, graphics computation at audio-sample rate is necessary to accurately visualize spatial modulation sound trajectories at full resolution.
Spatial Modulation Synthesis overcomes previous technical obstacles for accurate simulation of high-speed sound sources that can unify spatial, timbral, and visual composition.
2010 - 2012: Sound Element Spatializer
2011: Skate 1.0
2011 - 2013: Image Sonification Studies
2011: No Heritage
2012 - 2014: Seismic Sonification Studies
2014 - 2015: Spatial Modulation Synthesis and Kinetic V2
High-speed Fibonacci spirals of sound
First study producing timbral changes from spatialization (AM and panning, no Doppler)
Tedious, non-real time control and rendering via MATLAB
Spatial Amplitude Modulation via Panning
Faster Spirals Leading to Granulation
Sound Element Spatializer (2010-12)
Sound Element Spatializer (2010-12)
Real-time spatialization of an arbitrary number of simultaneous live or recorded sound sources over an arbitrary loudspeaker arrangement
Dynamic selection between multiple panning algorithms with distance cue including Doppler shift, gain attenuation, and air absorption
High-speed movement of sound sources without "zipper" noise artifacts
Flexible, precise control of sound trajectories using the OSC protocol
Robust and easy to integrate with DAW software or any audio application
Cross-platform standalone application in C++ (easy to use, no Max programming required)
Provide a link between visual and sonic worlds. Visual artists may provide trajectories for sound and musicians may provide sound for trajectories. Should encourage collaboration and exploration for both visual and sound artists