NVIDIA NemotronLabs VoiceChat 11B: Open Full-Duplex Voice AI With Live Tool Calling
NVIDIA has released NemotronLabs VoiceChat 11B , an end-to-end real-time speech-to-speech model designed for natural full-duplex voice conversations. Unlike the familiar pipeline t
NVIDIA NemotronLabs VoiceChat 11B Brings Full-Duplex Speech and Live Tool Calling to Open Voice AI
Introduction
NVIDIA has released NemotronLabs VoiceChat 11B, an end-to-end real-time speech-to-speech model designed for natural full-duplex voice conversations.
Unlike the familiar pipeline that chains automatic speech recognition, a large language model, and text-to-speech together, VoiceChat handles streaming speech understanding and speech generation inside one unified architecture. The goal is to reduce the handoffs that normally add latency and make a voice assistant feel less natural.
The model can listen while a conversation is active, yield when the user interrupts, and resume naturally when the user finishes. NVIDIA reports a 448 ms smooth turn-taking latency on Full-Duplex-Bench 1.0 and about 480 ms interruption latency.
VoiceChat also introduces something especially relevant to production voice agents: live tool calling while the spoken conversation remains active. A separate output channel can emit tool-call instructions, while predefined acknowledgment messages help the agent avoid an awkward silent pause during an external API request.
The release is important, but it is still early. NVIDIA marks the model as ready for research purposes only, recommends substantial GPU memory for deployment, and documents several limitations around long conversations, noisy environments, reasoning, multi-tool use, and runaway speech.
A Unified Speech-to-Speech Architecture
A traditional real-time voice assistant often looks like this:
User speech
↓
ASR
↓
Text
↓
LLM
↓
Text response
↓
TTS
↓
Agent speech
VoiceChat moves these capabilities into a more tightly integrated model.
NVIDIA describes the 11B system as a hybrid Mamba/Transformer architecture composed of:
- A Fast Conformer speech encoder
- A Nemotron Nano v2 9B language-model backbone
- An NVIDIA TTS decoder and audio codec
- A separate channel for tool-calling scripts
The user provides speech at 16 kHz. The output includes agent text, agent speech, and user transcription, with agent audio generated at 22.05 kHz.
Full Duplex Means the Conversation Can Overlap
A half-duplex assistant effectively treats conversation like a walkie-talkie: one side speaks, then the other side responds.
A full-duplex system can model both sides of the conversation continuously. That makes interruption handling, pauses, back-and-forth exchanges, and more natural turn taking possible.
NVIDIA’s published Full-Duplex-Bench 1.0 results include:
| Metric | VoiceChat 11B Result |
|---|---|
| Smooth turn-taking TOR | 0.82 |
| Smooth turn-taking latency | 448 ms |
| User interruption TOR | 1.0 |
| User interruption latency | 480 ms |
| User interruption GPT-4o score | 4.33 |
The 448 ms figure reported in the original news article comes from the smooth turn-taking benchmark. NVIDIA summarizes the model more broadly as delivering roughly 450 ms response latency.
VoiceChat Is Designed to Yield When the User Interrupts
Interruption handling is one of the biggest differences between a voice demo and a usable conversational agent.
VoiceChat is trained for conversational turn-taking directly. When a user starts talking in the middle of the model’s response, the system can stop its spoken turn and listen.
NVIDIA’s known-limitations documentation also says the behavior is not perfect in every situation. The model can still cut off a user on a mid-sentence pause, continue speaking after it should stop, start a new turn without new input, fall into a loop, or mishandle backchanneling.
Live Tool Calling Is the Most Interesting Engineering Feature
VoiceChat 11B is NVIDIA’s first open full-duplex model with tool-calling support that is designed to maintain a natural spoken interaction while a tool is being used.
A voice agent frequently needs information that is not inside the model. For example:
What is the current status of my order?
The model may need to call an order-management API before it can answer.
VoiceChat supports acknowledgment messages for tools. A developer can define short phrases such as:
Sure, let me check that for you.
When the model decides to call the tool, the system can speak one of those phrases while the external request is handled.
A simplified flow looks like this:
User speaks
↓
VoiceChat responds
↓
Model determines a tool is needed
↓
Tool-call event sent to client
↓
Client executes the external function
↓
Tool result returned to VoiceChat
↓
VoiceChat resumes the spoken answer
Important Tool-Calling Limitations
NVIDIA recommends a maximum of about five tools per session because performance can decline when more tools are available.
The model also cannot yet reliably call several tools simultaneously.
Other limitations include incorrect tool selection, skipped tool calls, invented arguments, incorrectly spoken tool results, and answering from internal knowledge when it should have used a tool.
One especially important detail: although VoiceChat supports user interruption during normal conversation, the user cannot currently interrupt the agent during tool execution.
Tool-Calling Benchmark Results
The reported AU Harness results are:
| Tool-Calling Category | Score |
|---|---|
| Simple | 58.5% |
| Multiple | 62.5% |
| Parallel | 42.5% |
| Parallel Multiple | 27.5% |
| Irrelevance | 89.6% |
| Average | 56.1% |
On Full-Duplex-Bench v3, NVIDIA reports:
| Metric | Score |
|---|---|
| Tool selection | 82.5% |
| Argument accuracy | 44.2% |
| Pass@1 | 33% |
These numbers show why production tool calls should still be guarded by application logic and argument validation.
VoiceChat Ranks Near the Top of Open Full-Duplex Models
NVIDIA reports that VoiceChat ranks #2 among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0.
The model is optimized for the trade-off between general intelligence and natural real-time conversation. NVIDIA explicitly warns that it may perform worse than the underlying Nemotron Nano v2 language model on knowledge, instruction following, safety, and reasoning tasks.
The Model Is Trained on About 550,000 Hours of Audio
NVIDIA’s model card lists approximately 550,000 hours of audio training data.
The mixture includes both real and synthetic speech, alongside text data used for the language-model component. Named sources include Fisher, LibriVox, LibriTTS, HiFi-TTS, VCTK, PromptTTS, UltraChat, internal NVIDIA speech data, synthetic speech, Nemotron function-calling data, and PersonaPlex training data.
VoiceChat Uses a Fixed Voice
The current VoiceChat checkpoint does not support voice cloning.
NVIDIA’s repository says the released checkpoint uses one fixed voice. The purpose of this release is more focused: low-latency, full-duplex conversational behavior and tool-aware voice interaction.
Hardware Requirements Are Significant
NVIDIA’s current real-time deployment prerequisites specify:
NVIDIA GPU with at least 80 GB of VRAM
The documented container supports GPUs including NVIDIA A100, H100, RTX 6000 Pro, and B200. The broader model card also lists H200 and B100 compatibility.
For the optimized real-time container, NVIDIA currently requires x86_64, Linux, Docker, NVIDIA Container Toolkit, and compatible NVIDIA drivers.
The troubleshooting guide notes that the model itself uses approximately 66 GB of GPU memory.
No Mature Hosted Inference API Yet
As of August 11, the Hugging Face model page does not list an active inference provider for the checkpoint.
Instead, NVIDIA provides two main paths:
- Offline inference from the Hugging Face checkpoint
- Interactive streaming through an optimized NVIDIA container
The real-time container packages CUDA, Triton, vLLM, and the complete VoiceChat inference stack and exposes a bidirectional WebSocket service.
How to Run Offline Inference
Clone NVIDIA’s experimental VoiceChat branch:
git clone https://github.com/NVIDIA-NeMo/Speech.git
cd Speech
git switch nemotron-labs-voicechat
export NEMO_DIR="$(pwd)"
Create the environment:
conda create -y -n voicechat python=3.12
conda activate voicechat
pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0
pip install -e ".[all]"
pip uninstall -y nvidia-resiliency-ext
pip install transformers==4.56.0 tokenizers==0.22.0 lhotse==1.32.2 huggingface-hub==0.34.4 hf-xet==1.1.9 torchcodec==0.10.0 torch_audiomentations jinja2
pip install ninja packaging wheel einops
pip install --no-build-isolation --no-deps causal-conv1d==1.6.2.post1 mamba-ssm==2.3.2.post1
Download the checkpoint:
hf download nvidia/NVIDIA-NemotronLabs-VoiceChat-11B --local-dir /path/to/checkpoint
Run the sample:
conda activate voicechat
export NEMO_DIR=/path/to/Speech
python "$NEMO_DIR/examples/speechlm2/offline_voicechat_infer.py" --checkpoint /path/to/checkpoint --wav "$NEMO_DIR/examples/speechlm2/sample_audio/sample_general.wav" --output-dir /path/to/output
NVIDIA recommends adding enough trailing silence to custom input audio so the model has time to respond.
How to Deploy the Real-Time Container
Download the model repository:
ngc registry model download-version nim/nvidia/nemotron-labs-voicechat:1.0.0
chmod -R 777 nemotron-labs-voicechat_v1.0.0
Launch the service:
docker run -it --rm --name=nemotron-labs-voicechat --runtime=nvidia --gpus '"device=0"' --shm-size=8GB -e NIM_HTTP_API_PORT=9000 -p 9000:9000 -v $(pwd)/nemotron-labs-voicechat_v1.0.0:/data/models --entrypoint /s2s/run_s2s_server.sh nvcr.io/nim/nvidia/nemotron-labs-voicechat:latest
Check readiness:
curl 'http://localhost:9000/v1/realtime/health'
The server then exposes a bidirectional WebSocket endpoint at:
ws://localhost:9000/v1/realtime
A Simple Tool Definition
A simplified tool definition looks like this:
[
{
"name": "get_weather",
"description": "Get current weather for a city",
"ack_messages": [
"Sure, let me check that for you."
],
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string"
}
},
"required": ["city"]
}
}
]
The ack_messages field gives the system something natural to say while the tool is executing.
Research-Only Means Production Teams Should Be Careful
NVIDIA explicitly labels VoiceChat 11B as research purposes only.
The model was trained with audio context windows of no more than about two minutes, so longer conversational context may not be retained reliably.
Known issues include reasoning mistakes, hallucinations, degraded speech after several turns, repetition, garbled text, truncated speech, runaway continuation, self-talk, clarification loops, missing words in transcription, unreliable language switching, and poor behavior in noisy or highly reverberant environments.
Where VoiceChat 11B Could Be Useful
Potential research and prototype scenarios include:
Contact Centers
The agent can listen naturally, handle interruptions, call customer-service tools, and use acknowledgment messages while waiting for APIs.
In-Car Assistants
A driver can interrupt an assistant without waiting for a full spoken response to finish. The current sensitivity to background noise means automotive deployment would require additional work.
Retail and Restaurant Ordering
A voice agent can gather an order, respond to corrections, and call product or inventory systems.
Accessibility
More natural turn-taking may help hands-free interfaces and voice-first applications, though accessibility deployments require careful user testing.
Enterprise Voice Agents
Organizations can experiment with internal tools and self-hosted speech interaction while keeping the model inside their own infrastructure.
Open Model, But With Two Different Licenses
The phrase “open source” deserves some precision.
The NeMo Speech code used by VoiceChat is licensed under Apache License 2.0.
The VoiceChat model materials use the OpenMDW 1.1 license.
For that reason, it is safer to describe VoiceChat 11B as an open model or open-weight model rather than assuming the model license is identical to the Apache-licensed software around it.
Teams planning redistribution or commercial use should review the OpenMDW license directly.
What Developers Should Test Before Production
A useful evaluation plan should cover:
- Turn taking and user interruptions
- Mid-sentence pauses and backchanneling
- Noisy, echoing, and multi-speaker audio
- Wrong or missing tool arguments
- Tool timeouts and failed APIs
- Long conversations
- Runaway speech
- Hallucinations and reasoning quality
- Approval requirements for sensitive actions
- Logging, rate limits, and human escalation
A voice agent that can call tools needs the same permission controls as any other autonomous agent.
常见问题
What is NVIDIA NemotronLabs VoiceChat 11B?
NemotronLabs VoiceChat 11B is NVIDIA’s end-to-end real-time speech-to-speech model for full-duplex conversational AI. It combines streaming speech understanding, a Nemotron language-model backbone, speech generation, and a separate tool-calling channel.
How fast is VoiceChat 11B?
NVIDIA reports 448 ms smooth turn-taking latency and about 480 ms interruption latency on Full-Duplex-Bench 1.0.
Can users interrupt VoiceChat while it is speaking?
Yes, normal conversation supports barge-in and interruption handling. However, NVIDIA says users cannot currently interrupt the agent during tool execution.
Does VoiceChat 11B support function calling?
Yes. The model can emit tool calls through a separate output channel, and developers can define acknowledgment messages that are spoken while the external tool is running.
How much GPU memory does VoiceChat 11B need?
NVIDIA’s real-time container requires a GPU with at least 80 GB of VRAM. The troubleshooting documentation says the loaded model uses roughly 66 GB.
Is there a hosted VoiceChat 11B API?
NVIDIA currently documents self-hosted offline inference and an optimized real-time container. The Hugging Face model page does not currently list a hosted inference provider.
Can VoiceChat clone voices?
No. The released checkpoint uses a fixed voice and does not support voice cloning.
Is VoiceChat 11B ready for production?
NVIDIA marks the model as research-only. Developers should evaluate its limitations around context length, reasoning, tool use, noisy audio, repetition, transcription, and runaway speech before production deployment.
相关工具
- NVIDIA NemotronLabs VoiceChat 11B: Official model card, weights, benchmarks, architecture, license, and inference instructions.
- NVIDIA NeMo Speech: Official speech repository containing the VoiceChat implementation and deployment branch.
- NVIDIA VoiceChat Real-Time Container Guide: Official Docker, WebSocket, and function-calling deployment instructions.
- NVIDIA Container Toolkit: Enables Docker containers to access NVIDIA GPUs.
- vLLM: The inference engine listed in NVIDIA’s VoiceChat model card.
- VoiceBench: An open benchmark for evaluating LLM-based voice assistants.
Related Links
- VoiceChat 11B Model Card: Primary source for release date, benchmarks, training data, architecture, and research-only status.
- VoiceChat GitHub Branch: Official source code, installation instructions, hardware requirements, limitations, and model notes.
- Real-Time Deployment Prerequisites: Hardware, Linux, Docker, driver, and container-toolkit requirements.
- Real-Time Deployment Guide: NVIDIA’s container launch, health check, WebSocket, client, and tool-use documentation.
- OpenMDW 1.1 License: License governing the VoiceChat model materials.
- Full-Duplex-Bench: Benchmark used to evaluate pause handling, turn taking, and interruption behavior.
- Full-Duplex-Bench v3: Benchmark focused on full-duplex spoken interaction with tool use.
Summary
NVIDIA NemotronLabs VoiceChat 11B combines speech understanding, language modeling, speech generation, interruption handling, and tool calling inside a unified full-duplex architecture. NVIDIA reports 448 ms smooth turn-taking latency and positions the model near the top of current open full-duplex voice benchmarks.
The most interesting engineering feature is tool use. A separate output channel can trigger external functions, while acknowledgment messages keep the conversation from falling silent during API calls.
The release is not yet a turnkey production service. Real-time deployment requires at least 80 GB of GPU memory, the current checkpoint has a fixed voice, hosted inference is not broadly available, and NVIDIA documents substantial limitations in longer sessions, reasoning, noisy audio, and tool execution.
VoiceChat 11B is best viewed as an open research platform for building and studying low-latency voice agents—not yet as a finished replacement for production contact-center or consumer voice APIs.