Student researchers work with ALS community to evolve DuoRhythmo music app for everyone

November 9, 2022

For Dr. Alper Kaya, the darkest days of his life happened about 10 years after he found out he had ALS. Based in Turkey, the ophthalmologist felt fortunate in that the disease progressed slowly, after the initial diagnosis of Amyotrophic Lateral Sclerosis in 1990. He could still perform surgeries, help patients and play classical guitar – one of the constants in his life, ever since he was a teenager. After about a decade, he lost the use of his hands to do those things he loved. He lost hope, too. At the time, it felt unbearable. But with the help of his daughter, he found other ways to make music again. Assistive technology gave him the ability to blow into a MIDI wind controller (“a hands-free electronic, breath-powered instrument”). Later, when he couldn’t close his mouth, he found other music-based software to respond to eye tracking and small head movements. He’s got a YouTube channel where he’s uploaded some of his compositions. [caption id=attachment_177843" align="aligncenter" width="1024] Dr. Alper Kaya[/caption] Most recently, he was an early tester for DuoRhythmo, a Windows app that recently released its first major update. It enables anyone, including people living with disabilities, to create music collaboratively and remotely in real-time. With it, he can create original rhythms and beats using his eyes. “Communicating through music is very important for me,” says Kaya, 61. He played in bands throughout his younger days. During high school, university and medical school, he never left music. “Music represents the color of my spirit and when I make music, it develops my spiritual life. I feel free and more creative. If you want to make music, you need an instrument and you bond with it. It’s a special relationship.” While he can no longer play instruments the way he used to, technology has given him another way to be creative and connect to music. He immediately liked the DuoRhythmo interface and being able to play hand drums (with his eyes), something he hadn’t done before, and found it very easy for a beginner like him to incorporate the irregular rhythms he grew up with in Turkish music. [caption id=attachment_177844" align="aligncenter" width="1024] Screenshot of DuoRhythmo app[/caption] What Kaya can do with the app is the culmination of years of research by students at Aalborg University Copenhagen who are studying Medialogy, a major that explores how people interact with computers. That field includes technology such as desktop apps like DuoRhythmo, as well as virtual and augmented reality. “We really like this concept of extreme constraint design, when you have a very specific target group who have limited ways of interacting with the product. And then instead of making a product and then making it accessible afterwards, we build it from the bottom up to make sure interaction comes first and then everything else comes after,” says Truls Bendik Tjemsland, one of the core group members who created the app -- students who are in the master’s leg of the Medialogy program, scheduled to graduate in summer 2023. As they were looking for projects to work on, music emerged as the focus because of its creative and collaborative nature, a means to connect people and bring them together. A collaboration between Multisensory Experience Lab at Aalborg University Copenhagen and Microsoft Research, they practiced user-centered research alongside people with ALS (PALS) and their families. [caption id=attachment_177848" align="aligncenter" width="1024] Some of the students who worked on DuoRhythmo (Photo by Frej Reosenstjerne at Danish Sound Day event showcasing DuoRhythmo in May 2022)[/caption] They pitched the idea of an app based on an accessible drum pattern, akin to a drum circle. “Through music you can get to this true pure joy of co-creation and collaboration with people living with ALS,” says Balázs András Iványi, who is studying in the Bay Area with Tjemsland for a semester before returning to Denmark. “I think this sort of collaborative aspect is like when you're making food together, that joy of doing something together. These people were able to actually play something together and make something meaningful.”Since the idea came to fruition during the pandemic, they had to test the idea remotely. They were able to find five people living with ALS all over the world. “That was a really, really interesting journey, meeting all these stakeholders and convincing people they don't have to be musically adept to be able to enjoy this experience,” Tjemsland says. In the spring of 2022, the student researchers went to a Danish convention for people with ALS. They learned several things during that event, one being that as long as they were able to use a mouse or touchscreen, people living with ALS wanted to use that, vs. an eye tracker. “We learned that along the way that we need to cater for all these other users,” Iványi continues. “As the disease progresses unfortunately, there is this continuum of inputs they can use, starting with a traditional mouse, then towards the very end an eye tracker. We wanted to overlap these as much as possible, so we have multiple input devices for every stage or can support multiple input devices for every stage of the disease.” [caption id=attachment_177851" align="aligncenter" width="1024] ALS testers helping with the development of the DuoRhythmo app in Denmark[/caption] Developers on the project, also students at the university, built the app using the Unity 3D game engine. “Quite early on, we decided that we wanted to be spatial and 3D, to have a feeling of space,” says Christian Tsalidis, who studied electronics engineering for a year in Spain before he moved to Denmark, where he’s able to combine programming skills with design. Unity was also a good choice for them, he says, because it was important to be able to release it on different platforms. “We can build for Windows, phones and/or augmented and virtual reality devices.” They looked to Microsoft researchers for inspiration. The idea of building something for play and to improve quality of life resonated with them. “There's not a lot of research allowing people with ALS to play music with other people. So this multiplayer aspect that we incorporated into DuoRhythmo was a main attraction,” says Scott Naylor, another student developer on the project. “Our whole interface was designed from the beginning with that in mind.” Currently, two people at a time can collaborate on the app. One person can modulate the effects for the speeds/beats the other is creating, and they can also decide to trade places.It's inclusive in many ways. It's not exclusively designed for people with disabilities.The students made custom buttons that are interactable with eye and head tracking, as well as those aligned with mouse and keyboard controls to open the app up to as many people as possible depending on how far they are through their diagnosis. The team found that working with Windows 11 helped them throughout the building process, as they find it’s more accessible in terms of how much more development and research is available when designing for as many as possible. Bill Buxton, a partner researcher at Microsoft, served as an advisor on DuoRhythmo. A former professional sax and synthesizer player, he pivoted to computer science and human computer interaction design decades ago. Among his numerous accomplishments, he headed the team that developed the user experience for the 3D software used to design cars. He takes a design-first perspective in each project, asking about the process and creating a culture of design that integrates all stakeholders so they can all work together contributing their expertise. [caption id=attachment_177850" align="aligncenter" width="1024] A video call on Microsoft Teams showing Microsoft advisors and Danish students working on DuoRhythmo[/caption] The design problem Buxton was interested in solving with the students was how to point and trigger things that could create music using only eyes or tongues, depending on the disabilities a person may have. Buxton and the students embrace the notion of “universal design” – such as wheelchair ramps to buildings, which came from the disability community yet benefited the general population. “For this project, the approach began with accessibility to a specific community with special needs, but the solution – likewise – brought the benefits to the general population, providing a demonstration that with appropriate design, we can have inclusive design with minimal compromise,” Buxton says. Knowing that they wanted as many people as possible to use the app – and still be valid musically – the team approached the project from the perspective of an orchestra conductor focusing on the percussion section instead of the musician that needs fingers to play the guitar or piano. “You don't have to articulate every note to make music. You can also shape and initiate phrases,” Buxton says. “You can start these percussion patterns and as you can gradually modify them, you can conduct them and change the tempo. You can start to do amazing things that are highly creative. The key thing is the choice of the patterns that you can bring into the sequence and what you can do with them, like a DJ.” In addition to assistive devices like those involving eye tracking, the app can incorporate input from a touch pad, touch screen, mouse, foot paddle or joystick. “It's inclusive in many ways,” Buxton says. “It's not exclusively designed for people with disabilities.” But ALS testers were essential in creating the app. Cathy Cummings, executive director of the International Alliance of ALS/MND Associations, helped connect the students with their future users during the development of the app.Cummings, whose mother passed from ALS in 2005 after living with the disease for five years, volunteered with ALS Ontario, where she became chairman of the board. She also sat on the board of ALS Canada and after completing board terms in 2013, she became the Alliance’s new executive director. Microsoft already had a presence on the Alliance Innovation and Technology Advisory Council and made the introductions to the students in Denmark. “I can't say enough positive things about them. They were great to work with. The students were really diligent about knowing what they wanted to do and how they wanted to proceed, and they were really interested right from the beginning with involving people with ALS so that they felt like they were actually meeting a need as opposed to something that they had dreamt up,” says Cummings, whose organization was looking at the patient journey over time and what technology they might need throughout their journey with ALS to improve their quality of life. “A lot of technologies support the very basic physiological needs, but the more interesting ones to our group of people living with ALS falls into the higher order of Maslow's hierarchy of needs: music, art, appreciation of things that improve their quality of life that aren’t directly related to their ability to cope with the day-to-day. So, the more nuanced parts of life they felt made life worth living. And it was right at the convergence of that time when we were looking at how to define that matrix that we were introduced to this fantastic team.” Cummings adds that the collaborative nature of the app also helps counter the social isolation that often results from the disease. “Once people have their basic physiological needs satisfied, life is much more than that,” says Cummings, who appreciates how the app is available to so many people through the Microsoft Store. “Anything that's built into something that's available publicly is hugely helpful. ALS/MND is a fairly rare disease, and so the exposure to students at the university level when they're working on products and services in technology to improve quality of life really helps others gain an awareness of our disease. Once someone's aware of something, they take it with them for the rest of their life.” It's important, Cummings says, for businesses to include people with disabilities as they develop products. “At any of our education seminars or our conferences each year, we see what Microsoft is bringing in the future and how they're working forwards on things like natural voice and translation,” she says. “All of those kinds of things aren't products on the market yet, but they give people a lot of hope that they're going to be able to keep their identity and be able to communicate as they progress through this disease.” Top image: Photo by Frej Reosenstjerne at Danish Sound Day event showcasing DuoRhythmo in May 2022, with original illustration by Bernardo Henning