Skip to main content

UW Team wins international Amazon Alexa Prize for the design of conversational AI

November 28, 2017

The UW Sounding Board team from left to right: Hao Fang, Hao Cheng, Ari Holtzman, Professor Mari Ostendorf, Maarten Sap, Elizabeth Clark and Professor Yejin Choi (not pictured: Professor Noah Smith).

A team of researchers from University of Washington Department of Electrical Engineering (UW EE) and the Paul G. Allen School won Amazon’s international Alexa Prize, announced at AWS re:Invent 2017 in Las Vegas, Nevada. The UW team developed Sounding Board, a conversational agent designed to interact with users with engaging and informative conversation and transform how people interact with information.

The Alexa Prize is a $2.5 million university competition designed to encourage the development of conversational artificial intelligence (AI). $500,000 of this fund goes to the winning team. The challenge is to produce a socialbot — an AI agent capable of coherent conversation with humans — that is able to converse about popular topics and current events for 20 minutes.

Teams built their socialbots using the Alexa Skills Kit and receive continuous, real-world feedback from millions of Amazon customers that have interacted with them anonymously through Alexa. Sounding Board was the only North American competitor to make it to the final round.

The Sounding Board team combines expertise in natural language processing, speech technology and human-AI collaboration. The team is led by UW EE Ph.D. student Hao Fang, working with UW EE Ph.D. student Hao Cheng and Allen School Ph.D. students Elizabeth Clark, Ari Holtzman and Maarten Sap. UW EE professor Mari Ostendorf is the lead faculty advisor for the team, working in collaboration with professors Yejin Choi and Noah Smith of the Allen School’s Natural Language Processing (UW-NLP) research group.

“The students started from scratch, with no experience building a dialog system or working with Alexa skills, but together they brought a breadth of perspectives on language processing and a passion for understanding both the technical and human factors challenges of conversational AI,” Ostendorf said.

The Sounding Board design is both user-driven and content-driven. On the user side, the system aims to understand user comments in multiple dimensions, from directives to sentiment and personality, in order to best serve user interests. At the same time, the system relies on having interesting and timely things to talk about. It actively harvests online content and leverages a knowledge graph to provide connections between related topics, which can be used to steer the conversation.

“The philosophy behind developing Sounding Board is bringing a variety of relevant content into a natural conversation,” Fang said. “Ultimately, we hope Sounding Board can become a conversational gateway to online information that users enjoy talking with.”

In order to make design a conversational AI, the researchers needed to support a diverse range of users who would interact with Sounding Board.

“One of the biggest challenges we faced when designing Sounding Board was the diversity of people we interacted with,” Clark said. “We needed to handle a wide range of topics, conversational styles, and user personalities and interests.”

For the UW team, the highly collaborative environment at the UW is ideal for both getting feedback on technical ideas and for user testing. Faculty and students from across the UW-NLP community — in computer science, electrical engineering, and linguistics — provided input on the many different versions of Sounding Board as it evolved.  In addition, a key resource in system development has been access to real Alexa users nationwide.

“It is impossible to anticipate all the types of reactions and questions people will have, even the different ways that a simple yes/no question can be answered,” Ostendorf said. “Learning from actual user data is critical.”

More than 100 teams from universities in 22 countries applied to be part of the inaugural competition. The finalists were selected from among 12 semifinalists, whose socialbots were evaluated based on customer ratings of their interactions during hundreds of thousands of conversations between July 1 and August 15.

In August, Amazon’s Ashwin Ram, senior manager for Alexa AI, announced that UW’s Sounding Board team and the Alquist team from the Czech Technical University in Prague received the highest average customer ratings, earning them a place in the finals. What’s up Bot from Heriot-Watt University in Edinburgh, Scotland earned the wildcard slot. The three finalists continued to improve their socialbots, leveraging customer interactions through November 7 and Amazon selected the winner based on assessments of a panel of three judges listening to conversations with three interactors. Amazon will publish technical papers from all participating teams in the Alexa Prize Proceedings as a way of sharing their work with the broader research community.