While Visual Computing has been a dominant paradigm in Human Computer Interaction (HCI), I believe the role of sound and auditory perception is a crucial sensory mode of communication and engagement. In an increasingly complex environment we are often inundated with an overload of visual stimulation. Sound offers a complementary modality that has been highly underutilized in the design of modern computing and interaction devices. For many users, such as the visually impaired, less literate or those in complex workplace settings, it offers a crucial mode of sensing and understanding physical and digital information in a seamless auditory realm.
Over the years, I have designed experimental auditory environments for hyper-linked information navigation at Georgia Tech (Espace 2), developed multi-modal mobile audio capture and gesture-based interaction at Fuji-Xerox (Dynomite), and pioneered Wearable Audio Computing at the MIT Media Lab (Nomadic Radio). In these systems I combined novel auditory interfaces, 3D spatial sound, speech recognition and synthesis, as well as auditory context awareness (using recognition of environmental sounds) to support natural and casual everyday interaction.
I not only developed novel technologies and user interfaces for these systems, but also conducted ethnographic and usability studies, co-design and fieldwork to engage users with these systems in real-world settings; this helped me learn how such auditory computing would better fit into their lives or offer speculative user experiences that would only be adopted in the future as these technologies matured. Such was the case with Nomadic Radio, which predated the use of speech and audio interfaces that are highly prevalent in smart home devices today using Alexa, Siri, Google Assistant, and Cortana.
While these voice-based smart home devices have been widely adopted by users in their domestic and work settings, there are many critical challenges regarding privacy, natural and seamless interaction, and attention management that are not handled well by such devices and interfaces, particularly for less advanced and specialized users such as children, the elderly and disabled who continue to find such systems less suited to their needs. Hence, I plan to instigate auditory computing with a more inclusive human-centered design approach that can better address the everyday needs of a much wider user demographic. By investigating the ethical and social challenges of such settings we can also help devise far more trusting and engaging user experiences.
I plan to develop exploratory auditory computing interfaces and environments for particular usage domains, while conducting research on how a wider range of users can engage with such systems in their everyday settings. In this regard, working closely with industry firms such as Google, Nokia and Phillips would offer for greater insights and collaborations to better influence the design of existing products (say for children and disabled) and allows us to exploratory new directions in devising speculative and engaging auditory experiences.
Related Work:
Sawhney, N., Wheeler, S. and Schmandt, C., 2001. Aware Community Portals: Shared Information Appliances for Transitional Spaces. Journal of Personal and Ubiquitous Computing, 5(1), pp.66-70.
Sawhney, N. and Schmandt, C., 2000. Nomadic Radio: Speech and Audio Interaction for Contextual Messaging in Nomadic Environments. ACM Transactions on Computer-Human interaction (TOCHI), 7(3), pp.353-383.
Sawhney, N. and Schmandt, C., 1998, October. Speaking and Listening on the Run: Design for Wearable Audio Computing. In Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 108-115). IEEE.
Clarkson, B., Sawhney, N. and Pentland, A., 1998. Auditory Context Awareness via Wearable Computing. Energy, 400(600), p.20.
Schilit, B.N., Wilcox, L.D. and Sawhney, N., 1997, March. Merging the Benefits of Paper Notebooks with the Power of Computers in Dynomite. In CHI’97 Extended Abstracts on Human Factors in Computing Systems (pp. 22-23). ACM.
Sawhney, N. and Schmandt, C., 1997, October. Nomadic Radio: A Spatialized Audio Environment for Wearable Computing. In Wearable Computers, 1997. Digest of Papers, First International Symposium on (pp. 171-172). IEEE.
Wilcox, L.D., Schilit, B.N. and Sawhney, N., 1997, March. Dynomite: A Dynamically Organized Ink and Audio Notebook. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (pp. 186-193). ACM.
Sawhney, N. and Murphy, A., 1996, April. ESPACE 2: An Experimental Hyperaudio Environment. In Conference Companion on Human Factors in Computing Systems (pp. 105-106). ACM.
Copyright All Rights Reserved © 2019
Nitin Sawhney 