Speaker Diarization Explained: How Machines Know Who Said What

Speaker diarization is the part of transcription that figures out who said what. Without it, a two-person interview reads like an unbroken monologue. With it, the same transcript reads like a play, with each voice properly attributed and every quote traceable to its source.

This post covers the pipeline behind modern diarization, why voices can be clustered reliably, the main failure modes, how to read the DER accuracy metric, and the practical steps you can take to get clean speaker labels on your recordings.

What Diarization Is and What It Is Not

Diarization answers the question "who is speaking when," not "who specifically is this person." A diarization system labels speakers as Speaker 1, Speaker 2, Speaker 3 based on acoustic signatures, not by cross-referencing a database of known identities. You tag the speakers after the fact ("Speaker 1 is Alice") and a single find-and-replace fixes every occurrence.

Two related tasks people confuse with diarization:

• Voice Activity Detection (VAD). Decides where speech is happening at all, versus silence, music, or noise. Diarization assumes VAD has already run. See how VAD works for the upstream step.
• Speaker identification. Compares an unknown voice to enrolled samples of known people ("this sounds like Alice"). Requires a database. Diarization is the simpler, more common need.

For most interview and meeting use cases, diarization without identification is enough. Identify once, update everywhere.

The Embedding-Clustering Pipeline

A standard diarization pipeline runs four stages, though modern systems implement them as a single neural pass rather than four sequential calls:

1. VAD. Strip out silence, background music, and non-speech regions.
2. Segmentation. Cut the remaining audio into short windows, typically 1.5 to 3 seconds each.
3. Embedding. Encode each segment as a high-dimensional vector capturing that segment's voice characteristics.
4. Clustering. Group segments with similar embeddings into speaker clusters and assign each cluster a label.

The output is a sequence of (start time, end time, speaker label) tuples. A separate ASR step transcribes the words; the diarization output maps words to speakers.

If you want to see where diarization fits relative to transcription as a whole, the post on how AI transcription works covers the full product pipeline from upload to formatted output.

Why Voice Embeddings Work

The trick that makes diarization possible is that voices have stable acoustic signatures even as the words change. Pitch range, formant frequencies (the resonances that distinguish vowel sounds), speaking rate, breath patterns, and phoneme habits are surprisingly consistent for any one person across a conversation.

A modern speaker embedding model, such as ECAPA-TDNN, NVIDIA NeMo's TitaNet, or x-vector architectures, takes a short audio segment and outputs a 192-512 dimensional vector. Two segments from the same speaker cluster close together in that space. Two segments from different speakers sit further apart. Agglomerative or spectral clustering algorithms handle the grouping.

The math is not the bottleneck. The bottleneck is what happens when those assumptions break down.

Where Diarization Fails

Five failure modes you will encounter in real recordings:

Speaker Overlap

When two people talk at once, most cascade-style systems pick one speaker and drop the other. The overlapping word goes to one label and the other speaker's contribution is silently lost. This is the largest single source of error in panel discussions, animated debates, and rapid-fire interviews.

End-to-end neural diarization approaches, sometimes called EEND models, treat diarization as a multi-label prediction problem and can assign a segment to two speakers simultaneously. That handles overlap better but increases system complexity. For most practical workflows, the real fix is better mic discipline.

Similar Voices

Two speakers with similar pitch, accent, and speaking rate get merged into one cluster more often than you would expect. Same-gender panels and family interviews are common trouble cases. The model has no way to know the voices are different if the embeddings overlap significantly.

Separate microphone channels per speaker solve this outright. When that is not possible, expect some manual cleanup.

Short Utterances

A one-word interjection ("Yeah," "Right," "Exactly") often lacks enough audio for a confident speaker assignment. Most systems either guess (frequently wrong) or skip the segment. In a back-and-forth interview where one person is asking short questions, the wrong-speaker problem becomes very noticeable.

Background Voices

A TV in the background, a second conversation audible through a thin wall, or audio playback from a phone are all treated as new speakers by the diarizer. Your 2-person interview suddenly shows four speakers: the two participants plus whoever is on the news broadcast. Recording in a quiet space is the cheap fix.

Speaker Count Estimation

Most systems either auto-detect the number of speakers or accept a hint. Auto-detection is imperfect. Deepgram's diarization, for example, auto-detects without requiring a count input and claims accurate results with 16 or more speakers. Pyannote accepts a speaker count as an optional hint. When a tool you are using supports the parameter and you know the count, provide it. When the tool auto-estimates, verify the first minute of output carefully.

Diarization Error Rate (DER) Explained

DER is the standard accuracy metric for diarization. The formula combines three error types:

• Speaker confusion. Right speech detected, wrong speaker assigned.
• Missed speech. Real speech labeled as silence.
• False alarm. Silence or noise labeled as speech.

DER = (false alarm + missed + speaker confusion) / total reference speech duration.

Below 10% DER is generally considered good across most real-world audio domains.

Sources: Picovoice open benchmark (VoxConverse) and independent research benchmarks (DIHARD III), July 2026.

A DER of 10% means roughly one second in ten has the wrong speaker label. For a 60-minute file, that is 6 minutes of misattributed speech. Whether that matters depends on your use case. A rough meeting summary tolerates it. A court transcript does not.

My take: the gap between the best open-source tools and the weaker commercial add-ons is enormous, over 40 percentage points at the extremes. Choosing a tool based on the underlying diarization model rather than overall brand matters more here than in almost any other transcription feature.

For context on how accuracy is measured across the broader transcription stack, see transcription accuracy explained.

Practical Steps for Clean Diarization

If you control the recording, these are the levers that matter most:

• One microphone per speaker. USB mics, lavalier mics, or headsets on each participant. This removes most diarization difficulty before it starts.
• Separate audio channels if possible. Multi-track recording (Riverside, Zencastr, or local capture on each participant's machine) lets the diarizer work channel-by-channel instead of separating a mixed stream. Channel-based attribution is essentially solved.
• Brief pauses between turns. A half-second gap helps the system detect speaker boundaries reliably.
• Quiet environment. Mute TVs, close doors, ask others in the space to stay quiet.
• Provide speaker count if supported. When you know how many speakers are in the file and the tool accepts that hint, use it.

For meeting recordings specifically, the practical guide at how to transcribe a Zoom meeting covers channel-level recording setup that makes diarization nearly perfect.

When You Get the File After the Fact

If you did not control the recording, you are working with what you have. The recovery workflow:

• Run a high-quality diarization tool. Pyannote, Deepgram with diarize_model=latest (the current recommended parameter, per Deepgram's May 2026 docs, replacing the deprecated diarize=true), or AssemblyAI with speaker_labels: true. All three offer strong accuracy on typical interview and meeting audio.
• Provide speaker count when the tool accepts it. Where the API supports this hint, it reduces estimation noise.
• Review the first minute carefully. If the labels are right at the start, they are usually right throughout. Errors cluster at the beginning when the model is calibrating speaker identities.
• Search for known confusion points. Short interjections, overlapping segments, and speaker switches are where errors concentrate.
• Tag speakers manually once. Find-and-replace handles the rest.

If you just need a clean, speaker-labeled transcript without setting up a diarization pipeline yourself, CATT's meeting transcription tool runs diarization automatically on upload.

What Diarized Transcripts Enable

A speaker-labeled transcript opens workflows that an unlabeled one cannot support:

• Per-speaker word counts. Who dominates your meetings?
• Per-speaker analysis. Sentiment by participant, topics raised by each person.
• Better LLM summaries. The model can produce "Alice argued X, Bob countered Y" instead of "the participants discussed."
• Quote extraction. Pull all of Speaker 2's contributions for a research note or customer testimonial.

For interview transcription specifically, the guide on how to transcribe an interview recording shows how speaker labels map into structured output.

When You Do Not Need Diarization

If the recording is a single speaker (a voice memo, a lecture, a solo podcast), turn diarization off. The transcript processes faster, and you avoid the small risk of a diarization error injecting a phantom speaker break into a monologue.

For everything with two or more voices, diarization is worth enabling. The cost is minimal and the readability difference on a long recording is significant.

FAQ

What is speaker diarization?

Speaker diarization segments an audio recording by speaker, labeling each segment as "Speaker 1," "Speaker 2," and so on based on voice characteristics. It answers "who is speaking when," not "who specifically is this person." That last step requires a separate speaker identification system with enrolled voice samples.

What is Diarization Error Rate (DER)?

DER is the standard accuracy metric. It sums three error types, false alarm speech (silence called speech), missed speech (real speech called silence), and speaker confusion (right speech, wrong speaker label), then divides by total reference speech duration. Below 10% DER is generally considered good. Modern commercial APIs range from roughly 11% to over 50% on the VoxConverse benchmark depending on the vendor.

How does automatic speaker diarization handle overlapping speech?

Most cascade-style systems pick one speaker per segment and drop the other voice entirely. End-to-end neural systems like EEND treat diarization as multi-label prediction, so they can flag a segment as containing two speakers simultaneously. Overlap is still the hardest failure mode. The practical fix is better mic discipline before the call starts.

Can I specify the number of speakers to improve accuracy?

Some tools let you set a speaker count (Deepgram auto-detects without needing it; pyannote supports it as a hint). When a tool supports the parameter and you know the count, providing it reduces estimation error. If you do not know or the tool auto-estimates, review the first minute of the output to catch mislabeling early.

What recordings get the best diarization results?

Clean recordings with one microphone per speaker or separate audio channels per participant. Each speaker on their own channel turns diarization into a near-trivial problem. Quiet spaces, brief pauses between turns, and no background voices also help significantly. If the recording is already mixed, a high-quality diarization API plus manual review of the first minute is the best recovery path.

Sources

• Deepgram Speaker Diarization Docs
• Picovoice Speaker Diarization Benchmark (VoxConverse)
• AssemblyAI: Top Speaker Diarization Libraries and APIs 2026
• Benchmarking Diarization Models, arxiv 2509.26177
• Pushing the Limits of End-to-End Diarization, Interspeech 2025
• Deepgram: Introducing Batch Diarization V2 (May 2026)