All Ears, All Eyes

Design Strategies for Concurrent Sonification-Visualisation of Geodata

  • PI :: PerMagnus Lindborg, PhD, Associate Professor, School of Creative Media, City University of Hong Kong
  • Co-I :: Sara Lenzi, PhD, Ikerbasque Research Fellow, University of Deusto, Spain
  • Co-I :: Paolo Ciuccarelli, Professor of Design, Northeastern University, Boston, USA
  • Researcher :: Valentina Caiola, PhD candidate, SCM, CityU

About

Sonification is the translation of data into sound. Inherently interdisciplinary, the field has seen tremendous development characterised by 1) expanding the definition to embrace aesthetics, via electroacoustic music composition; 2) professionalisation of terminology, techniques, and community-building; and 3) increased attention to visualisation. Time is ripe to focus efforts on the third point. We employ knowledge from dynamic data visualisation to improve on sonification techniques, to generate a cross-modal perception informed theoretical framework, and to determine practicable strategies for concurrent sonification-visualisation design. Project targets are: 1) a set of design guidelines, and 2) a proof-of-concept software system applied to geodata with real-life importance, such as rain and wind, pollution and traffic, forest fires and landslides. People seek to understand their physical environment. Accurate and engaging information design helps both in everyday activities and in making life-choices. Laying the research groundwork for a concurrent sonification-visualisation system for communicating environmental geodata has the potential for real-life applications with broad public appeal and societal impact. Ultimately, the goal of the project is to contribute to the digital fabric of society and improve people’s quality of life.

Funding

Stragegic Research Grant (SRG-Fd), City University of Hong Kong (2023/09–2025/08)

Publications

  • Lindborg PM, Caiola V, Chen M, Ciuccarelli P, Lenzi S (2024/09). “Re(de)fining Sonification: Project Classification Strategies in the Data Sonification Archive”. Journal of the Audio Engineering Society. DOI https://10.17743/jaes.2022.0167 , https://aes2.org/publications/elibrary-page/?id=22710
  • Lindborg, PM, Lenzi S & Chen M (2023/01). “Climate Data Sonification and Visualisation: An Analysis of Aesthetics, Characteristics, and Topics in 32 Recent Projects”. Frontiers in Psychology, section Human-Media Interaction. doi: 10.3389/fpsyg.2022.1020102, https://www.frontiersin.org/articles/10.3389/fpsyg.2022.1020102

Links

References

  • Boehringer, Jorge. “TWICE STEPPED IN STILL WATERS: SONIFICATION AND INTERDISCIPLINARITY AS ARTISTIC RESEARCH.” sensors 12 (13): 14.
  • Caiola, Valentina, Sara Lenzi, and Dina Riccò. 2022. “Audiovisual sonifications. A design map for multisensory integration in data representation.” Digital Research Society. https://doi.org/https://doi.org/10.21606/drs.2022.380.
  • Ciuccarelli, Paolo, Giorgia Lupi, and Luca Simeone. 2014. Visualizing the data city: social media as a source of knowledge for urban planning and management. Springer Science & Business Media.
  • Dubus, Gaël, and Roberto Bresin. 2013. “A systematic review of mapping strategies for the sonification of physical quantities.” PloS one 8 (12): e82491.
  • Giordano, Bruno L, Ricardo de Miranda Azevedo, Yenisel Plasencia-Calaña, Elia Formisano, and Michel Dumontier. 2022. “What do we mean with sound semantics, exactly? A survey of taxonomies and ontologies of everyday sounds.” Frontiers in Psychology 13: 964209.
  • Goudarzi, Visda. 2020. “Exploring a taxonomy of interaction in interactive sonification systems.” International Conference on Human Interaction and Emerging Technologies.
  • Hermann, Thomas, Andy Hunt, and John G Neuhoff. 2011. The sonification handbook. Logos Verlag Berlin.
  • Lenzi, Sara. 2017. While I was not there.
  • Lenzi S & Ciuccarelli P (2020). “Intentionality and design in the data sonification of social issues.” Big Data and Society.
  • Lenzi, Sara 2021. “The design of data sonification. Design processes, protocols and tools grounded in anomaly detection [Doctoral Thesis].” PhD, Design Department, Politecnico di Milano. http://hdl.handle.net/10589/177079.
  • Lenzi, Sara, and Paolo Ciuccarelli. 2020. “Intentionality and design in the data sonification of social issues.” Big Data & Society 7 (2): 2053951720944603. https://journals.sagepub.com/doi/full/10.1177/2053951720944603.
  • Lindborg, PerMagnus. 2015a. “Sound perception and design in multimodal environments.” KTH Royal Institute of Technology.
  • Lindborg, PerMagnus 2015b. LW24 [sculptural auditory display]. National Gallery, Singapore. Original edition, Commissioned for the inaugural exhibition.
  • Lindborg, PerMagnus. 2018. “Interactive Sonification of Weather Data for The Locust Wrath, a Multimedia Dance Performance.” Leonardo 51 (5): 466-474.
  • Lindborg, PerMagnus. 2019a. “What is the Color of that Electroacoustic Music?” Proceedings of the International Computer Music Conference joint with New York City Electroacoustic Music Festival., New York City.
  • Lindborg, PerMagnus. 2019b. “How do we listen?” 에밀레 Emille Journal of the Korean Electro-Acoustic Society 16: 43-49.
  • Lindborg, PerMagnus 2021. “Feeling Loki’s pain: Designing and evaluating a DIY 3D auditory display for geodata sonification.” International Conference for Auditory Displays [ICAD].
  • Lindborg, PerMagnus. 2022. Stairway to Helheim. Hong Kong & Graz, Austria: Proceedings|Catalogue of DACA (Data Art for Climate Action).
  • Lindborg, PerMagnus, Katharina Groß-Vogt, and Shauhrat S Chopra. 2023. “Data perceptualization for climate science communication.” Frontiers in Psychology 14: 1263971.
  • Vickers, Paul. 2017. “Sonification and music, music and sonification.” In The Routledge Companion to Sounding Art, 135-144. Routledge, Oxford.
  • Vickers, Paul, and Bennett Hogg. 2006. “Sonification abstraite/sonification concrete: An’aesthetic persepctive space’for classifying auditory displays in the ars musica domain.” Proceedings of the 12th International Conference on Auditory Display, London, UK, June 20 – 23, 2006.
  • Walker, Bruce N, and Michael A Nees. 2011. “Theory of sonification.” The sonification handbook (eds. Hermann, Hunt, & Neuhoff): 9-39. https://sonification.de/handbook/.
  • Worrall, David. 2019. Sonification Design. Springer.
  • Ziemer, Tim. 2023. “Sound Terminology Describing Production and Perception of Sonification.” Journal of the Audio Engineering Society.