How to Handle Music in Transcription (2026 Honest Guide)

Music in your transcript shows up as one of two problems: garbled pseudo-lyrics where a song played, or hallucinated filler text over an instrumental section. Both stem from the same root cause, and both have practical fixes. The key insight is that none of the major transcription engines, including Whisper, Deepgram Nova-3, or AssemblyAI, have a verified dedicated music-detection layer that suppresses these segments automatically. The reliable solutions happen before transcription, not inside it.

What Your Transcript Is Actually Telling You

The symptom is almost always one of three things.

Garbled pseudo-lyrics over a vocal section. Your podcast plays a 30-second song clip for review. The transcript fills that time with partial, wrong lyrics: words from the song mixed with invented words, broken punctuation, and mid-sentence cuts where the model lost confidence.

Hallucinated filler over instrumental music. Your intro jingle is purely instrumental. The transcript produces "so," "um," "you know," or even coherent-looking fabricated sentences. Research published in May 2025 found that Whisper large-v3 transcribes "so" for over 55% of non-speech audio samples, and hallucinates on nearly 100% of environmental sound inputs. Music is not special-cased.

Partial gibberish when music plays under speech. Your host talks over a background music bed. The transcript picks up the speech but mangles words at the edges of musical phrases, because the model is trying to reconcile two audio sources at once.

Understanding which symptom you have tells you which fix to reach for first.

Why AI Engines Don't Solve This Automatically

A common assumption is that modern tools "detect and tag" music. This is not what the vendor documentation says.

Deepgram Nova-3's documentation (checked July 2026) lists real-time multilingual transcription, vocabulary customization, and keyterm prompting. No music-detection or music-suppression feature appears.

AssemblyAI's audio intelligence docs (checked July 2026) list auto chapters, sentiment analysis, entity detection, topic detection, and summarization. Music identification does not appear as a feature.

The "[Music]" label that appears in YouTube auto-captions comes from a separate classification layer Google adds to its pipeline, not from Whisper's decoder. Whisper's own output on music segments is not a clean tag.

What Whisper actually does with music is documented in its GitHub discussion threads and in published research: the model's suppress-token list causes it to skip labeling background sounds as [music], [applause], or similar, so decoding continues into non-speech audio until the model hallucinates speech. The output looks plausible, which makes it harder to spot than silence would be.

Voice activity detection (VAD) helps but does not solve this. VAD reliably removes silence. Music carries energy in the frequency bands VAD treats as speech, so musical sections commonly pass through the filter untouched. WhisperX uses VAD on by default and still produces music hallucinations.

Fix 1: Cut the Music Before Transcribing

For predictable, structural music (intros, outros, bumpers), this is the fastest and most reliable fix. You keep your original publication file unchanged. You create a separate "transcription cut" with the music removed, transcribe that, and add manual markers in the transcript where the music was.

Audacity handles this for free. Select the music segment, delete it (or silence it if you want to preserve timing), export the cleaned file, transcribe the clean file. For a 10-second podcast intro you do this once and set it as your standing workflow.

For content with music scattered throughout (a music review show, an education podcast using song examples), the workflow adds a timestamp step:

1. List every music segment with its start and end time.
2. Cut each segment out of the transcription copy.
3. Transcribe the speech-only copy.
4. Paste markers back into the transcript at the right timestamps: [Song: "Title" by Artist, 14:22-15:04].

This is more effort upfront but produces a transcript you can actually use for SEO, show notes, search, or accessibility.

Fix 2: Separate the Speech Track with Source Separation

When speech and music play simultaneously, cutting is not an option because the two are mixed together. This is the case for a host talking over a music bed, a guest speaking while a song plays in the background, or a recording from a live venue.

Source separation tools split a mixed audio file into component tracks.

Demucs v4 (Facebook Research, open source, also called htdemucs) separates audio into vocals, drums, bass, and other instruments. Pull the vocals/other track, transcribe that. Demucs v4 achieves 9.0 dB SDR on MUSDB HQ benchmarks, which translates to reasonably clean speech isolation in practice. It runs via Python or through cloud services like Replicate for teams without local GPU capacity.

Spleeter (Deezer, open source) offers 2-stem (vocals, accompaniment), 4-stem, and 5-stem separation modes. It is faster than Demucs on CPU but generally rated lower quality on musical separation tasks. For speech-over-music scenarios, either tool works.

Source separation adds processing time and introduces its own artifacts. For a one-off recording with significant background music, the quality gain is worth it. For a podcast where music only appears in brief bumpers, cutting is simpler.

Fix 3: Configure Self-Hosted Whisper to Suppress Non-Speech

If you run Whisper yourself, two parameters reduce (but do not eliminate) music hallucinations.

no_speech_threshold sets the probability above which Whisper decides a segment contains no speech and suppresses output. Raising it (toward 1.0) makes the model more aggressive about dropping segments it is unsure about. The tradeoff is that genuinely quiet speech can also get dropped.

initial_prompt shapes the model's expectations before decoding begins. Framing the audio as a podcast or interview shifts the prior toward conversational speech. The initial_prompt effect only lasts for the first 30 seconds before being overwritten by the decoding of subsequent segments, so it is more useful for short files than long-form audio.

result = model.transcribe(
    audio_file,
    no_speech_threshold=0.8,
    condition_on_previous_text=True,
    initial_prompt="The following is a podcast interview. Music sections are not spoken content."
)

This reduces hallucinated output on non-speech segments but does not reliably produce clean "[Music]" tags. For vocal music especially, the model will still attempt to transcribe lyrics. VAD preprocessing narrows the audio to likely-speech windows before decoding and is the stronger of the two approaches for large files.

These settings are relevant for self-hosted Whisper via the openai-whisper Python library or faster-whisper. Hosted APIs expose fewer parameters.

Fix 4: Post-Process to Remove Music Sections

If you cannot change the audio before transcription, the transcript itself shows recognizable signs of music contamination.

Music hallucinations tend to appear as: repetitive short phrases, unusual proper nouns, sentences that do not connect semantically to the surrounding content, or sudden shifts in punctuation density. A script or a careful read-through can flag these segments for review.

Replace the flagged sections with explicit markers:

[Intro music: 0:00-0:12]
[Song clip: 22:40-23:10]

This is cleaner than leaving garbled text in a transcript you will use for show notes, captions, or search indexing. Downstream summarization tools will treat hallucinated lyrics as content and produce wrong summaries if you leave them in.

Specific Scenarios

Podcast with Music Intro and Outro

Cut the music before transcribing. If you publish the same format every episode, create a template transcription cut that starts at the point where the host begins speaking. For podcast transcription workflows where you run multiple episodes per week, a one-time setup saves hours.

Music Review or Education Content

You will have song clips scattered throughout. Build the timestamp-list workflow described in Fix 1 into your production process. Note the timestamp of each clip before you send audio to any transcription tool. Post-transcription, you add markers in the right places. This is the only approach that produces an accurate transcript for this content type.

Recording with Background Music Bed

Source separation is the right tool if audio quality matters. If the music is low-volume and the speech is clear, a higher no_speech_threshold in self-hosted Whisper sometimes helps. Accepting slight accuracy loss on words near musical phrases is reasonable for internal use cases. For publication, separate the tracks.

YouTube Video with Music Soundtrack

Pulling the audio from a YouTube video that has music throughout and sending it directly to a transcription engine will produce significant hallucinations. YouTube transcript generation works best when the video is primarily speech-driven. For music-heavy YouTube content, source separation before transcription is the reliable path.

Worship Service or Sermon with Music

The sermon and music alternate as discrete segments. Cutting the music sections (hymns, worship songs) before transcribing is the practical approach. Add manual markers for each musical section. The spoken portions transcribe cleanly.

A Workflow for Music-Heavy Content Creators

For anyone producing content with music regularly, building the clean-cut step into the production chain is cheaper than fixing transcripts after the fact.

1. Produce two versions of every episode: the publication mix (with music) and a transcription cut (speech only).
2. Keep the transcription cut as a separate export step in your editing software.
3. Transcribe only the transcription cut.
4. Add music markers to the final transcript by hand, at the timestamps where music played.
5. Use the clean transcript for show notes, SEO, captions, and search.

For one-off recordings, the source-separation route (Fix 2) handles cases where you cannot re-edit the file.

For recorded content with minor background music at low volume, check the transcript for the hallucination patterns described above and clean them out manually. The speaker diarization explainer covers a related accuracy problem: overlapping voices cause similar garbled-output symptoms in the transcript, and the diagnosis process is comparable.

If you just need a clean transcript from a speech-only recording with no music involved, ConvertAudioToText handles the upload without requiring an account to start.

FAQ

Why does my transcript have nonsense text where the music played?

AI transcription models are trained to map audio to words. When the audio contains music, especially vocals, the model treats it as speech and tries to transcribe it. The result is garbled pseudo-lyrics, hallucinated filler, or outright fabricated phrases. No current consumer transcription service has a verified, dedicated music-detection layer that suppresses these segments reliably.

Can VAD (voice activity detection) filter out music automatically?

Partially. VAD filters silence and low-energy segments well, but it commonly fails on music because music carries energy in the frequency bands VAD treats as speech. WhisperX with VAD on by default still produces hallucinations on musical sections that slip through. VAD is a useful starting point but not a complete solution for music.

Does Whisper output a [Music] label for music segments?

Not reliably. The [Music] label you may have seen in YouTube auto-captions comes from a separate layer Google adds on top of its own ASR pipeline, not from Whisper's decoder. Whisper's own output on music ranges from hallucinated lyrics to fabricated phrases to filler words. Research published in 2025 found Whisper large-v3 hallucinates on nearly 100% of non-speech environmental audio. Source separation or manual cutting is the only reliable mitigation.

Is it worth transcribing song lyrics from a recording?

Almost never through a general ASR engine. Sung speech has altered timing, pitch, and phoneme boundaries that confuse models trained on conversational audio. You will get partial, incorrect lyrics mixed with invented words. If you genuinely need accurate lyrics, a dedicated lyrics-recognition service or manual transcription is the right tool. For most podcast and interview use cases, cutting the music section and replacing it with a marker like [Song: Title by Artist] gives a far cleaner result.

Sources

• Deepgram models and languages overview (checked July 2026): https://developers.deepgram.com/docs/models-languages-overview
• AssemblyAI audio intelligence documentation (checked July 2026): https://www.assemblyai.com/docs/guides/audio-intelligence
• Calm-Whisper: Reduce Whisper Hallucination On Non-Speech (arXiv, 2025): https://arxiv.org/html/2505.12969v1
• Whisper GitHub hallucination discussion #1606: https://github.com/openai/whisper/discussions/1606
• Demucs v4 / htdemucs overview (HuggingFace Spaces): https://huggingface.co/spaces/abidlabs/music-separation
• Spleeter by Deezer (GitHub): https://github.com/deezer/spleeter
• WhisperX VAD documentation (DeepWiki): https://deepwiki.com/m-bain/whisperX/4.1-voice-activity-detection
• Whisper initial_prompt guide (Sotto blog): https://sotto.to/blog/improve-whisper-accuracy-prompts