Roz is a creature of dichotomies. Inorganic, and yet somehow alive; a chimera of wood and metal. Even the machine’s shape presents a sharp contrast. The horizontal heart of a piano bisects an imposing vertical ring of steel, as though the classical instrument were transiting a portal to the future. From this ring hang two robotic arms, poised like a bionic praying mantis lying in wait. Picture your grandmother’s spinet splayed out on a cold metal table before a robotic surgeon, and you’ll about have it.
And yet, the point of Roz isn’t what it looks like as much as what it sounds like. Because Roz, short for “Robotic Resonance,” is an instrument. The excised innards of an upright piano, strummed by felt pads attached to electronic motors and manipulated by robotic arms, is designed to emit ethereal, ambient music. But thanks to the power of artificial intelligence (AI), Roz is quickly becoming more than an instrument — it’s becoming a performer.
“It started out as an art piece and an art and music idea,” says Mark McCoin, Roz’s creator, associate professor and head of new media for the UTSA Department of Art and Art History. “Now it’s turning into an instrument, and eventually it will turn into an entity. Watching this thing morph itself through time and space is fascinating.”
But Roz isn’t quite ready to sing “Daisy Bell” just yet; even getting the machine to this stage has been the product of seven years of intense effort. Arising out of his experience using piano harps (the cast iron frame that holds a piano’s strings) in art and music exhibitions, McCoin initially envisioned Roz as a way to make his life a little easier.
“I’ve been working with piano harps for many years, performing on them and doing various kinds of installations with them,” he says. “And at some point, they’re heavy and they’re difficult to move around, and so I thought it would be great if I didn’t have to perform on it anymore.”
But having a robot perform on his behalf was more than just a matter of convenience; it presented a world of new artistic possibilities, says Nathan Wheeler, Roz’s lead technical director and a longtime collaborator of McCoin’s.
“There were things Mark couldn’t do,” Wheeler explains, “so the idea of being able to use a robot arm to move slower than he’s capable of, or stay in one place for longer durations than he’s capable of while maintaining even pressure. Achieving higher specificity and control was the main idea.”
Now, the pair are partnering with UTSA School of Data Science (SDS) faculty and Ph.D. students to give Roz even greater abilities. From the very beginning, McCoin has wanted Roz to hear itself, to understand what it’s hearing, and then adapt how it plays accordingly. In the tech field, this is called “machine listening,” and McCoin believes that the work done with Roz at the SDS will have applications that extend far beyond robotic piano harps.
“That’s what interested the School of Data Science, and that’s why we’re here now: to develop a form of artificial intelligence which will help it hear what it does as input and respond to that,” McCoin explains.
Essentially, they’re teaching Roz to improvise — something humans have been doing for millennia, but which computers are notoriously incapable of. To accomplish this, McCoin and Wheeler take turns playing Roz, improvising as they see fit. At the same time, they note various elements such as volume and density of sound, plus information like the speed of Roz’s motors and the position of its arms. All of this comes together to form a dataset of auditory and mechanical information that they’ll use to train Roz to make its own music.
Just getting to the point where McCoin and Wheeler can play Roz together in real time is a major achievement. Like all artists, Roz has navigated some less productive periods in its career. Early on, powering off the machine would cause Roz to collapse and forget where its arms were supposed to be. Unlike countless rock stars similarly afflicted, it wasn’t fast living that was doing Roz in. It was technology, specifically servo technology.
A servo, or servomotor, is an electric motor used to push, pull or rotate a mechanical component into a desired position. Any time electricity is used to move an object — such as remote-controlled toys, supermarket doors or robots — servos play a vital role. Unfortunately, most consumer-grade servomotors have traditionally been “unidirectional,” meaning they were designed only to receive information. You can transmit an instruction to the servo, but without attaching various external sensors to your mechanism, the motor has no way to send back its actual location to whatever’s controlling it. It’s a little like writing a letter and never knowing if it’s arrived.
So-called smart servos solve this problem by recording and transmitting precise position feedback. Think of it this way: If traditional servos are First-Class Mail, smart servos are FaceTime. It wasn’t until 2020 that McCoin and Wheeler had ready access to smart servos and were ultimately able to keep Roz from keeling over after a gig.
Nor were servos the only piece of tech that the pair have had to wrestle with. Unbeknownst to the duo, the original electronic motors used to spin the felt pieces that strike Roz’s strings were malfunctioning in a delightfully unexpected way, McCoin said.
“These cheap motors were bad enough that the string would literally slow the motor down to its resonant speed, so it was going in and out of tune,” McCoin explains. “When the felt would hit the string, it would slow the motor down, so you’d hear the pitch of the motor change while the string is staying in one place. So, there was all this sort of randomized counterpoint happening between the motor and the string.”
When they upgraded motors, the phenomenon stopped. On the one hand, this change improved the consistency and usability of Roz as instrument, but on the other hand, it stripped their creation of a piece of its personality.
“We lost the magic of this individuated piece,” McCoin laments. “It was singing to us!’”
In the end, however, McCoin and Wheeler decided that the tradeoffs brought about by the hardware swap were emblematic of the entire project.
“I think that’s the whole process,” McCoin says, “letting go of what we think it is, and allowing it to be what it really is.”
WATCH: Roz’s robotic arms activate the harp strings to create an evolving soundscape of shifting string harmonics.
Other challenges the pair have faced with their programmable prodigy have ranged from the metaphysical to the mundane. For example, how do you train AI to decide what’s musical when that’s an inherently subjective assessment? This, Wheeler says, has proven to be a unique hurdle.
“This is our art piece, and we want it to be representative of the things we like and prefer,” Wheeler notes, “but we don’t want it to be weighed so heavily in that direction that it just sounds like one of us playing. It’s like a lot of generative AI in that the goal is to get results that we wouldn’t otherwise get if we were just doing it ourselves.”
As for the more mundane details inherent in building an AI-powered piano harp?
“Carrying it!” McCoin says with a laugh.
“It’s very heavy,” Wheeler agrees. “You build a robot, and you don’t even get to play the thing! The robot plays, and you turn into a glorified roadie.”
While the SDS researchers working in the lab can’t make Roz lighter, they’ve proved invaluable in other ways, McCoin says.
“The students are great,” he said. “They’ve brought things to the table that we weren’t aware of, and they look at things in ways that we never would. They’ve helped us understand our ratings system in ways that make us more objective, and that’s been very helpful.”
While Roz looks to have a bright future ahead of it, both McCoin and Wheeler recognize that the use of AI to compose and perform music will forever change society. Indeed, AI-produced art is already a contentious legal issue; it’s notorious for copyright infringement and is itself ineligible for copyright registration. How these issues will be decided, as well as how we will come to conceptualize art, music and even performance, remains to be seen.
“I think there’s going to be a values shift in terms of music and production,” Wheeler says. “I think live performance and being in the presence of a thing physically is going to acquire a different value than it has currently.”
WATCH: Roz performs at the 2023 CURRENTS Santa Fe International New Media Festival.
Whether we like it or not, AI is now a part of our life, and its role will only expand in the years to come. In fact, McCoin already thinks that AI is going to be a major component of music production for coming generations.
“If I were just starting out now, I would probably dive completely into AI,” he says. “I would not learn an instrument. My instrument would be the computer and my relationship to AI.”
Even so, McCoin wrestles with how such developments will impact future generations of students as an educator.
“What I teach a lot is connecting with your environment, connecting with the world, going out and having an adventure and making art about that — having real experiences,” he said. “What happens when it’s all virtual and (it’s) not even you making it, but something else making it for you?”
As a member of the U.S. Artificial Intelligence Safety Institute Consortium, UTSA and the School of Data Science are an ideal place for ideas like Roz to grow and take shape. McCoin and Wheeler extend their heartfelt thanks to SDS faculty, staff and students for the support and input.
“We’re very grateful to the School of Data Science for recognizing the value of Roz as both an art piece and an interdisciplinary idea that could be adapted to help students here,” McCoin says. “I’m learning so much and growing in this environment. It changes the way I think about education, and how I will go back and teach my students. Being able to have this opportunity is a gift for us.”
As for Roz, McCoin has high hopes for his newly educated apparatus. He’s explored the idea of taking Roz on tour, perhaps recruiting other musicians to perform and improvise with Roz live onstage. There’s even been talk of incorporating Roz into an opera, but there’s no word yet on whether it will sport a horned helmet.
“I think it’s fascinating how this can interface with the world,” McCoin says. “It’s not stuck as an art piece, or stuck as a performative piece, or even a scientific-interest unit. It can be all of these things. And that makes it difficult for us to figure out how we’re going to put it in the world. It can be all these things — and still we have to carry it.”
