Spotify personalizes audio experiences with machine learning

Spotify's machine learning approach

To build this machine learning approach to audio personalization, Spotify invested in data, modeling and experience building, said Tony Jebara, vice president of engineering and head of machine learning at Spotify, during a presentation at the MIT Technology Review's EmTech Digital conference on March 30.

The audio streaming provider used raw data from playlists, listening behaviors from users, content, audio analytics and information gained from what users are browsing and skipping through.

Using that data, Spotify built machine learning models to understand the similarities between specific pieces of music or podcasts, and what content users prefer.

"We use techniques like Collaborative filtering [a machine learning method used by recommendation systems to find similarities between data] as well as content modeling to figure out what tracks are similar, what artists are similar, "Jebara said. "This fingerprinting is really essentially mapping all our content, our users into a high-dimensional vector space."

The high-dimensional vector space is a system to organize and categorize the content on the audio platform. Spotify used this to make its content and users into distinct objects.

After placing users and content into categories, Spotify determines what a consumer's taste is and combines content from different artists to help create and build a personalized playlist.

Using machine learning model building has helped Spotify drive 16 million listeners to its Discover Weekly playlists and other playlists, Jebara said.

It also led the audio streaming platform to release its blending playlist, which enables consumers to combine their music choice with that of celebrities and other famous artists into one playlist.

Although Spotify relies heavily on machine learning modeling, it also uses its editors to create what it calls its "algotorial playlist."

"Those are cutting-edge cultural experts that tell us: 'Here's a new genre of music that's coming up,'" Jebara said. "We call this algotorial because you're combining algorithms with editorial knowledge."

We're not just trying to get the most likely clicked recommendation or the most likely streamed or the longest consumption in the moment. We're also trying to build a journey for users that gets into kind of a more fulfilling and enriching content diet.

Providing long-term value

While Spotify has been able to use recommendation engines and machine learning model building to help create personalized experiences for consumers, the audio platform is now interested in creating long-term satisfaction for its consumers.

"We're not just trying to get the most likely clicked recommendation or the most likely streamed or the longest consumption in the moment," Jebara said. "We're also trying to build a journey for users that gets into kind of a more fulfilling and enriching content diet."

To do this, Spotify is trying to balance the familiar with the new for listeners. This means combining older content recommendations with unexpected recommendations that will be valuable later.

To get to this point, Spotify began a new chapter of machine learning called reinforcement learning. Reinforcement learning is a machine learning training method that rewards desired results and devalues undesirable results.

Spotify started using reinforcement learning with things like contextual bandit, a machine learning framework in which algorithms evaluate different actions to learn what will provide the best outcome in a situation.

With contextual bandit, Spotify tried to generate the most clicked audio choices (playlist, podcast episode, artist page) on listeners' homepages.

Spotify is trying to find better paths in the audio landscape for consumers with reinforcement learning, Jebara said. The audio platform wants to expose listeners to newer recommendations, with hopes that consumers will like it and click on it again.

The audio platform is also building simulators to simulate how users may react to different recommendations.

Spotify builds reinforcement learning agents in the simulated world before evaluating it out on their users.

"And that's really an exciting next frontier," Jebara said. "Machine learning that's beyond just clicks and clickbait and going after the long term."