Lowest word error rate on the Krisp Voice Isolation Benchmark

VoxLock

A voice isolation model for voice agents. It keeps the caller's voice and removes the other voices in the room before speech-to-text hears them. It takes one input, the call audio, and picks out the caller by itself from the first seconds of the call.

Result
4.95% word error rate on 265 real recordings, with 0.11 background words leaked a minute. Krisp VI 2.5 scores 5.30% and leaks 1.52.
Size
27M parameters · 74 MB as the 8-bit build · 61% of one CPU core per live call
Training
Fine-tuned from the REAL-TSE baseline on about 100 hours of simulated calls and 87 hours of real meeting crosstalk from AMI
Runs on
CPU with onnxruntime, as a Pipecat audio filter or a LiveKit Agents plugin, inside sherpa-onnx in C++, or in the browser with onnxruntime-web
Links
Paper · Live demo · Hugging Face · GitHub · PyPI

The demo

The video above is a recording. The live demo is on the write-up page. Pick a call with someone talking over the caller and flip the switch while it plays: off is the call as recorded, on is VoxLock, with what speech-to-text hears under each. Or record up to 15 seconds of your own voice and mix in a second talker. It runs through the same model in your tab, a 47 MB download the first time.

How it compares

On the Krisp benchmark the word error rate is level with Krisp's best model; the gap is within noise. The difference is what leaks through: Krisp passes 1.52 background words a minute to the transcript and VoxLock 0.11. On 134 windows of real AMI meetings that are not in the training data, it scores 10.12% against 11.85% for the best ai-coustics model.

Performance

One live call with the default 8-bit model on one CPU thread: onnxruntime 1.31.1 on a Google Cloud g2-standard-8 (Intel Xeon @ 2.20 GHz), 60 seconds of AMI meeting audio. A real-time factor below 1 keeps up with the call.

ChunkCPU per callReal-time factor
20 ms110% of a core1.10
40 ms77% of a core0.77
80 ms61% of a core0.61

20 ms chunks keep up on two threads (0.86). The delay is 24 ms of algorithmic latency plus the chunk. On an NVIDIA L4 GPU, a whole recording runs 40 times faster than real time.

Precisions

PrecisionQualityDownloadWord error rate %Background words leaked / minCPU per call, 20 ms chunks
int8 (default)99.8%74 MB5.110.12110%
int8-full99.1%71 MB5.850.04110%
fp16100%*47 MB4.95*0.11*163%
fp32100%94 MB4.950.11165%

Quality is the share of fp32's correctly transcribed words each precision keeps, on the 265 Krisp recordings. * fp16 is the float32 model with its weights rounded to float16. Its output matches float32 to over 70 dB, so it carries the float32 scores. The browser demo uses it.

The full write-up has every model's results, how the model works, the training stages and the code to use it.