Nuance Labs, an AI research lab building the foundation model for human expression, today announced $50 million in Series A funding led by returning investor Lightspeed Venture Partners, with participation from returning investors Accel, South Park Commons, and new investors NVIDIA and Define Ventures. The company will open a research preview of its model to the public later this year.

Founded by former Apple PhD researchers Fangchang Ma, Edward Zhang, and Karren Yang, Nuance Labs is building AI that perceives and responds in real time with facial and vocal expression to the full range of how people communicate, from words and tone to gaze, gesture, and timing. AI that only sees the transcript is missing that layer entirely, working from a partial picture of intention and what the user actually wants. When AI understands a user and actually shows it's following, people stop conforming how they express themselves for a machine. They say more, say it more freely, and more of their intention and what they mean makes it through.

Why Avatars and Face-to-face AI Have Fallen Short

Voice AI has grown because, faults and all, it's more useful than what came before. Face-to-face AI hasn't, because those systems are assembled from parts. To date, the flow has been that speech is transcribed, a large language model decides what to say, a voice reads it, and a face is animated to match. Everything human about a conversation gets lost in the handoffs, adding delay and choppiness that break the feeling of a live conversation. The result is an avatar that sits frozen while you talk, talks over you, or simply can’t keep up. Nuance Labs' model perceives and generates natively using its specialized full-duplex model to stream perception in and stream response out at the same time, showing it's following while you're still speaking.

The Nuance Labs Approach: One Full-Duplex Model for Human Conversation and Expression

Nuance Labs' model:

• Perceives the full range of how people communicate, from words and tone to gaze, gesture, and timing

• Responds in real time with facial and vocal expression

• Runs as a single full-duplex model that streams perception in and response out at the same time

• Shows with verbal and non-verbal cues it is following while the person is still conversing

“The potential for AI and avatars to enhance our lives will never be achieved while they stare at you blankly or keep interrupting you. They've never felt natural because we've been contorting ourselves and how we communicate to the machine rather than the machine adapting to us,” said Fangchang Ma, Co-founder and CEO of Nuance Labs. “The most productive collaboration comes from being able to express yourself freely, in words, tone, gesture, and expression, the way you would with a friend or close colleague, with all the nuance in the back-and-forth that turns talking into understanding. That's what we're building at Nuance Labs: AI that understands the many ways we express ourselves and responds the way a person does, in the moment.”

The Founding Team: From Apple Research to a Single-Model Approach

The founding team spent years at Apple working on how machines perceive and reconstruct people. They saw first-hand that machines had a difficult time understanding us. Faithfully reconstructing a person for telepresence remained a challenge, let alone following the interplay of verbal and non-verbal cues in a live conversation. Nuance Labs was founded to solve the whole problem of understanding and creating human conversation and expression from audio and video signals natively in a single model rather than stitching together transcription, language, speech, and animation systems that were never designed to work together.

How the Nuance Labs Foundation Model Works

The model learns from how people actually behave in conversation and operates in full-duplex fashion: streaming audiovisual perception of the user in and generating an audiovisual response simultaneously. So, real-time responsiveness is built into the architecture rather than added on top. It perceives motion and meaning as they unfold, and demonstrates understanding in the moment, even while the user is still speaking, the way people do. The result is conversation with AI that feels natural and productive, with better outcomes wherever expression drives results and AI can benefit from understanding the most important ways that people communicate., Use cases include sales and customer service, to coaching, professional training, and education, or anywhere else AI works alongside people.

“There's a massive opportunity to fix the way people interact with these transformative AI tools. Nuance Labs is building that interface, one that can see and respond to a person the way people do,” said Nnamdi Iregbulem, Partner at Lightspeed Venture Partners. “The founding team has a rare mix of technical talent and operating excellence, and they've turned that into a single model that follows you in real time. We can see this becoming a foundational layer for AI products everywhere, and this is the team to build it."

The funding will accelerate model development, expand the research team, and support the company's first public research preview, where people can finally have a face-to-face conversation with AI that reads them and answers in kind — a conversation that feels human.

Important Links:

• Sign up for access to the public research preview here: https://nuancelabs.ai/waitlist.

• Nuance Labs is hiring researchers and engineers across modeling, data, evaluation, inference, and real-time serving. To see open roles on the careers page, visit https://www.nuancelabs.ai/careers.

About Nuance Labs 

Nuance Labs is an AI research lab building the foundation model for human conversation and expression. The company builds AI that sees, listens, and understands the nuances of communication between people, and responds naturally in real time. Founded by former Apple PhD researchers Fangchang Ma, Edward Zhang, and Karren Yang and headquartered in Seattle, Nuance Labs is backed by Lightspeed, Accel, South Park Commons, NVIDIA, and Define Ventures. Learn more at https://www.nuancelabs.ai.

Media Contact

Signal & Story for Nuance Labs
jt@signalandstory.dev