Artificial intelligence can take your bad whistling and make it sound like Mozart

Consider the sound of a human crudely whistling the theme music from the Indiana Jones movies. Then, imagine a computer listening to the whistled version of that song, and imitating it—but in the style of Bach, on an organ.

A small group of researchers from a division of Facebook have created an artificial intelligence system that’s capable of doing just that—listening to an audio file of a song and then mimicking it with new instrumentation. It’s a cover band with a synthetic brain. They call it a “universal music translation network,” and from a given music file, it can spit out music in six different styles.

It’s not just Indiana Jones and Bach. The system can take a classic, heartfelt song like “Always on My Mind,” and turn it into Mozart playing a version of that tune. “We want to mimic the human ability to hear music, and repeat it, either by whistling or playing an instrument,” says Lior Wolf, one of the co-authors of a new study describing the work. The AI isn’t reading musical notes—it’s just turning a given audio file into code and then decoding it into a new version.

“It learns internally, its own representation, which is an alternative to [musical] notes,” Wolf adds. “The best way to describe it is that it creates [a] cover version in a different musical domain, in a different style.” Those styles include Mozart, Bach, and Beethoven. The researchers put a Metallica riff into the system, and got a piano version to come out.

Yaniv Taigman, also a coathor on the study, says that they were inspired by the success of a computer vision technique called neural style transfer, which is when an image is redrawn with a new style, as if Van Gogh painted your photograph. (An app called Prisma is a good example). “We wanted basically to bring that same effect to music, which is considered by many a universal language,” Taigman says.

It’s fun to listen to—check out the video below to hear it take a string quartet by Haydn and cover it in the style of Mozart, or African drums repeated as piano music by Beethoven. At different moments, it feels both polished and rough: the Mozart symphony covering Haydn is much better than the Mozart version of the whistled Indiana Jones theme. The paper describing the research bluntly describes it this way: “Asked to convert one musical instrument to another, our network is on par or slightly worse than professional musicians.”

The development caught the eye of Adam Roberts, an AI researcher with Google who works on Magenta, a project that focuses on machine learning, art, and music. He tweeted: “The first convincing musical style transfer I’ve heard. Make sure you check out African drums and whistling to symphony!”

Facebook already uses artificial intelligence to power the language-to-language translation services you see on the site when coming across a comment in another language, but it says in a blog post explaining the new research that it doesn’t plan to use for “a specific product or feature.” The researchers will be publishing the code behind the project on Github for anyone to use.

Janelle Shane, who experiments with a form of AI called neural networks for fun on the blog aiweirdness.com—and is known for using AI to generate funny new material like weird Valentine’s Day candy heart sayings—says that if there were a tool that average people could use to try this feature, it could help spur creativity.

“I really hope that they put together some kind of demo platform where other people can play with this tool,” she says. “It does seem like there’s lots of artists who would be able to make something really cool out of this—and right now the barrier to using artificial intelligence tools, using neural network tools for art, for generating music, is a little too high for most people.”