Scanning Spaces: Paradigms for Spatial Sonification and Synthesis



Ryan Michael McGee
PhD Disseration Defense
June 9th, 2015
www.spatialmodulation.com
Abstract


In 1962 Karlheinz Stockhausen's "Concept of Unity in Electronic Music" introduced a connection between the parameters of intensity, duration, pitch, and timbre using an accelerating pulse train. In 1973 John Chowning discovered that complex audio spectra could be synthesized by increasing vibrato rates past 20Hz. In both cases the notion of acceleration to produce timbre was critical to discovery. Although both composers also utilized sound spatialization in their works, spatial parameters were not unified with their synthesis techniques. This dissertation examines software studies and multimedia works involving the use of spatial and visual data to produce complex sound spectra. The culmination of these experiments, Spatial Modulation Synthesis, is introduced as a novel, mathematical control paradigm for audio-visual synthesis, providing unified control of spatialization, timbre, and visual form using high-speed sound trajectories.

The unique, visual sonification and spatialization rendering paradigms of this dissertation necessitated the development of an original audio-sample-rate graphics rendering implementation, which, unlike typical multimedia frameworks, provides an exchange of audio-visual data without downsampling or interpolation.
Presentation Outline


Background

Background
Spatialization in Electronic and Computer Music
  • Stockhausen, Gesang der Junglinge (1956)
  • Edgar Varese, Poeme electronique, Phillips Pavillion (1958)
  • Henry Brant, "Space as an Essential Aspect of Music Composition" (1967)
  • Chowning, Turenas (1972)
  • Chowning, "The Simulation of Moving Sound Sources" (1977)
  • Roads, "Spatialization of Sound Particles" (2001)

Synthesis in Electronic and Computer Music
  • Stockhausen, Kontakte (1958-1960)
  • Stockhausen, "The Concept of Unity in Electronic Music" (1962)
  • Chowning, "The Synthesis of Complex Audio Spectra by Means of Frequency Modulation" (1973)
  • Verplank, "Scanned Synthesis" (2000)
  • Wave Terrain Synthesis (top)
  • Graphic Synthesis (bottom)
Background
Sonification "data to sound to foster interpretation" (Thomas Hermann)
  • Audification - direct conversion
  • Parameter Mapping - arbitrary assignment of data values to sound parameters
    ("Sonification Handbook", Ch. 15)
  • Model Based - temporal and/or interactive mathematical model
    ("Sonification Handbook", Ch. 16)
Background
Sound Art
  • Neuhaus - sound art temporalizes space, Drive In Music
  • Edwin van der Heide, Spatial Sounds (100dB at 100km/h)

Background

Visual Music
  • Fischenger/McLaren/Whitney/Mary Ellen Bute
  • Abbado - "Perceptual Correspondences of Abstract Animation and Synthetic Sound"
    • Timbre-Shape, Perceived Spatial Location, Perceived Intensity
    • Visual shape more important than color, color enhances shape



Problem Statement

  1. Stockhausen and Chowning both created a continuum between acceleration and timbre and both made extensive use of sound spatialization, but technical limitations left spatial parameters separate from timbral control paradigms. However, physical principles such as Doppler shift and distance-based gain-attenuation provide an intrinsic, mathematical connection between moving sound sources and FM/AM.
  2. The symmetrical Doppler shift approximation formula ubiquitous in the field of digital audio is physically inaccurate at high-speeds over approximately 100 m/s (225 mph). While this approximation is sufficient for existing Doppler-based effects that do not require such high-speed source motion, it prevents a fully realized connection between spatialization and high-index FM timbres.
  3. Current multimedia software frameworks provide inadequate control rates for spatialization trajectories which limits the ability of spatial modulation to achieve high frequencies of modulation. Sample rate spatialization choreography is necessary to achieve audio-rate frequency and amplitude modulation from simulated spatial motion. Likewise, graphics computation at audio-sample rate is necessary to accurately visualize spatial modulation sound trajectories at full resolution.
    Spatial Modulation Synthesis overcomes previous technical obstacles for accurate simulation of high-speed sound sources that can unify spatial, timbral, and visual composition.
Timeline
  • 2009: W.A.N.T.S
  • 2010 - 2012: Sound Element Spatializer
  • 2011: Skate 1.0
  • 2011: SenSynth
  • 2011 - 2013: Image Sonification Studies
  • 2011: No Heritage
  • 2012 - 2014: Seismic Sonification Studies
  • 2014: Kinetic
  • 2014 - 2015: Spatial Modulation Synthesis and Kinetic V2
W.A.N.T.S. (2009)
  • High-speed Fibonacci spirals of sound
  • First study producing timbral changes from spatialization (AM and panning, no Doppler)
  • Tedious, non-real time control and rendering via MATLAB
Spatial Amplitude Modulation via Panning


Faster Spirals Leading to Granulation
Sound Element Spatializer (2010-12)

Sound Element Spatializer (2010-12)

  • Real-time spatialization of an arbitrary number of simultaneous live or recorded sound sources over an arbitrary loudspeaker arrangement
  • Dynamic selection between multiple panning algorithms with distance cue including Doppler shift, gain attenuation, and air absorption
  • High-speed movement of sound sources without "zipper" noise artifacts
  • Flexible, precise control of sound trajectories using the OSC protocol
  • Robust and easy to integrate with DAW software or any audio application
  • Cross-platform standalone application in C++ (easy to use, no Max programming required)
  • Provide a link between visual and sonic worlds. Visual artists may provide trajectories for sound and musicians may provide sound for trajectories. Should encourage collaboration and exploration for both visual and sound artists
Publication:
Skate 1.0 (2011)
  • Collaborative Installation with Electroland. Exhibited at the Los Angeles Architecture and Design Museum (July-September 2011)
  • Light trajectories controlling multiple moving sound sources via SES
  • Experimentation with fast speeds morphing timbre
SenSynth (2011)
Sonification of Spatial Data
"Kinetic Synthesizer"
Spatial Control

Publication: McGee, R., Ashbrook D., White S. "SenSynth: a Mobile Application for Dynamic Sensor to Sound Mapping" NIME 2012
Image Sonification (2011-13)
  • Shape to timbre, color not considered
  • Image data is spatial, but not temporal
  • Unified audio-visual composition
  • Image plane as spatialization map

Publication: McGee, R., Dickinson J., Legrady G. "The Voice of Sisyphus: an Image Sonification Multimedia Installation" ICAD 2012
Image Sonification (2011-2013)


Free on App Store, > 10,000 downloads, used internationally for live visual music performance and composition

Publication: McGee, R., "VOSIS: a Multi-touch Image Sonification Interface" NIME 2013
No Heritage (2011)
  • Sounds from SenSynth, image sonification experiments, and field recordings spatialized using SES with an emphasis on speed and swarming
  • High-speed spatialization like W.A.N.T.S. but now with real-time visual control and Doppler
  • Spatial Granulation technique using random motion of sound clusters
  • Octophonic Performance in Lotte-Lehmann Hall at UCSB


"Kinetic Synthesizer" from SenSynth




Granulation from Spatialization
Seismic Sonification (2012-2014)
  • 3 dimensional spatial data accelerated and filtered to produce unique timbres for seismic events
  • Inner Earth Interpreter: seismic audification studies with D.V. Rogers (2012)
  • Seismic Audification Compositions (2012):
    • Haiti: tone signatures from different seismic recording stations for a single event
    • Christchurch: creative filtering and granular manipulations of a single recording
    • Shadow Zone Shadows: multiple events spatialized according to geographic location
  • DOMUS and Seismic Spaces (2014-15)
Towards Spatial Modulation Synthesis

Typical A/V Rendering Paradigm


Unified A/V Rendering Paradigm


Kinetic (2014)

Spatial Modulation Synthesis (SM)

SM Square Modulation

SM Sinusoidal Modulation




  • Considering the symmetrical depth Doppler approximation for now...
FM via SM
SM Delay Line Artifacts

SM vs FM

Maximum Modulation Frequency at 0.1 Meter Bounds
SM vs FM

SM Other Modulators


SM Asymmetrical Bounds




SM Doppler Simulations

SM True Doppler vs FM
SM Multi-Dimensional Motion


SM AM and Granulation

SM Spatialization


  • Stereo, DBAP, VBAP, Ambisonics
  • All can result in sonic spread and additional AM
SM Control



SM Ranges


SM Demo



Download: demo VST, stereo-only, OSX 10.8 or later

Publication: McGee, R., "Spatial Modulation Synthesis" ICMC 2015

Kinetic V2 (2015)

Conclusion

  • Work Exploring Acceleration of Spatial Trajectories and Spatial Data
    • 4 Conference Proceedings Publications
    • 3 Original Compositions with Public Performances
    • 3 Software Contributions to Long-term Art Installations
    • 2 Mobile Sonification/Synthesis Apps
  • Unified Audio-Visual Rendering Paradigm
    • Sample rate trajectory control contributions to open source AlloSytem C++ suite
  • Spatial Modulation Synthesis
    • Mathematical unity of space, timbre, and visual form based on physical first principles of Doppler shift and distance-based gain attenuation using spatial controls of velocity and bounds
    • Physically Accurate Doppler rendering implementation
      • Coupling with dynamic anti-aliasing filter allows SM to achieve high FM indicies by moving up to the SOS
    • Trajectory-timbre relationships produce an intrinsic and novel visual-timbre taxonomy
    • 2 Public Exhibitions in the AlloSphere and 1 conference publication
    • Original software plug-in implementation that exists as both effect and audio-visual instrument
Future Work

  • Additional UI Features for Plug-in (listener position, orbit center, index ADSR)
  • Additional complex motion trajectories (source and listener)
  • Additional Distance Cues
  • Smaller Bounds, Higher Sampling Rates
  • Beyond SOS
  • Audience as an Audio-visual Canvas
Thank You!

  • Committee: Dr. JoAnn Kuchera-Morin, Dr. Curtis Roads, George Legrady
  • Professors/Mentors: Matthew Wright, Andres Cabrera, Stephen Travis Pope, Clarence Barlow, Marcos Novak
  • MAT Community: Reza Ali, Yuan-Yi Fan, Lance Putnam, Graham Wakefield, Karl Yerkes, Pablo Colapinto, Dennis Adderton, Myles Sciotto
  • Collaborators: Daniel Ashbrook, Johannes Girardoni, D.V. Rogers, Damon Seeley, Jatila van der Veen, Sean White
  • Fellowship: Robert W. Deutsch Foundation


Questions?



AlloSphere Demo