Skip to content

fix(audio): honor the selected transcription client without provider fallback - #2706

Open
Çağdaş Yürekli (cagdasyurekli) wants to merge 2 commits into
microsoft:mainfrom
cagdasyurekli:fix/audio-client-routing-1284-20261015
Open

Çağdaş Yürekli (cagdasyurekli) wants to merge 2 commits into
microsoft:mainfrom
cagdasyurekli:fix/audio-client-routing-1284-20261015

Conversation

@cagdasyurekli

@cagdasyurekli Çağdaş Yürekli (cagdasyurekli) commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

The built-in audio converter ignored a configured transcription client and used SpeechRecognition’s legacy Google service. Use the supplied client's callable audio.transcriptions.create endpoint with the original encoded stream, matching filename and MIME type. Once selected, provider errors propagate without cross-provider fallback.

Chat-only clients retain the existing audio behavior: the optional legacy Google path is used when its dependencies are installed, and transcription is skipped when they are absent. This avoids breaking callers that configured llm_client for image descriptions. Endpoint capability is checked before selecting a route, not after a failed provider request.

audio_model selects the transcription model or Azure deployment name, independently of the image llm_model. It defaults to whisper-1, can be set on MarkItDown(...), and can be overridden per conversion. The provider client still needs its appropriate endpoint configuration. The README documents these choices and the legacy Google upload behavior.

Addresses the configured-client part of #1284 and provides a current-main alternative to #326. It leaves the broader no-client Google-default policy question open.

Validation on Python 3.12: the original routing regression fails 8/9 cases on unchanged main; the review follow-up fails 6/7 cases on the preceding PR head. All 16 audio cases now pass, including chat-only/missing-dependency behavior, model precedence, and no fallback after provider failure. Full core: 1,066 passed, 57 skipped under existing CI/credential/optional skip conditions. Independent real OpenAI/Azure SDK MockTransport checks pass five success cases and one HTTP401 propagation case, including multipart payloads and Azure deployment URLs. No live provider calls. Black23.7 and diff checks pass.

Net change from main: +29 production, +169 tests, +25 documentation lines. No new dependencies, CLI flags, or hosted workflows.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on Windows with Python 3.13.3 and openai 3.19.2 (the Python SDK), against main (4cc9fa1) and this branch.

What works. I started a local HTTP server that answers /v1/audio/transcriptions, and I pointed a real openai.OpenAI(base_url=...) client at it. I converted test.wav, test.mp3 and test.m4a with MarkItDown(llm_client=client).

  • On this branch each file posts one multipart request. The filenames are audio.wav, audio.mp3 and audio.mp4. The content types are audio/wav, audio/mpeg and audio/mp4. The body sizes are 265 to 266 bytes larger than the three files, which fits the multipart overhead, so the whole file is sent. The markdown ends with the transcript.
  • On main no request is sent. SpeechRecognition is not installed in my environment, so the transcript step is skipped and the markdown has no transcript.

The PR's own 9 audio tests pass on the branch. With the PR's test file on main, 8 fail and 1 passes (the no-client legacy test).

The risk I found. llm_client is also how users enable image captions. A client that supports chat but has no audio endpoint now breaks audio conversion. I used a minimal object with only chat:

  • main: convert("test.wav") succeeds.
  • this branch: FileConversionException, caused by AttributeError: 'NoAudio' object has no attribute 'audio'.

I did not run an Azure client or a client whose provider has no whisper-1 model. Those would send the request and fail at the provider, and by the PR's design that error propagates with no fallback. I read that as the same kind of change for existing users, but I did not run it.

So anyone who already passes llm_client only for images, and converts audio too, gets a new failure after upgrading. The README text in the PR describes the new behavior, but it is a silent change for existing code.

Questions for the author:

  1. Should the audio path be used only when the client has an audio.transcriptions attribute, and otherwise keep the old path? That would avoid the AttributeError case above.
  2. whisper-1 is hard coded and llm_model is not used. Azure deployments need their own deployment name. Is a separate kwarg planned?

I am leaving this as a comment. The mechanics work, but I would want a decision on those two points before it merges.

@cagdasyurekli

Copy link
Copy Markdown
Contributor Author

PRABHU KIRAN VANDRANKI (@VANDRANKI), thanks for the Windows/SDK reproduction. Addressed both points in 96f9764:

  1. The audio endpoint is selected only when audio.transcriptions.create is callable. Chat-only clients retain the existing legacy path, including successful conversion without a transcript when optional legacy dependencies are absent. This is a capability choice before any request; a selected endpoint’s authentication/network/provider failure still propagates without trying Google. The README now states the Google behavior explicitly for clients without an audio endpoint.
  2. Added audio_model for the transcription model or Azure deployment name, defaulting to whisper-1. It works on the constructor and as a per-conversion override, independently of the image llm_model. The client must still be configured for the intended provider endpoint.

The seven new review cases give six failures and one default-model control pass on the previous head, then all pass with this change. All16 audio cases and the full core suite pass (1,066 passed,57 skipped). An independent offline check with actual OpenAI/Azure SDK clients verifies default/constructor/override models, Azure deployment URLs, original WAV multipart data, and HTTP401 propagation without Google; five successful requests plus one rejection were mocked, with no network calls. Black23.7 and diff checks pass. Native Windows was not rerun locally; your original format/SDK findings are preserved and the follow-up was checked on Python3.12.

@VANDRANKI

Copy link
Copy Markdown

Thanks for both changes. I re-checked 96f9764 (community check, not a maintainer review; it does not clear the merge gate). Both of my questions are answered.

What I ran, on Windows with Python 3.13.3 and openai 3.19.2, against a local HTTP server that answers the transcription endpoints. I compared main (4cc9fa1) and this head. SpeechRecognition and pydub are not installed in my environment.

case main this head
openai.OpenAI client, test.wav, test.mp3, test.m4a no request, no transcript one request each, files audio.wav, audio.mp3, audio.mp4, model whisper-1, transcript in the output
chat-only client (an object with only chat), test.wav converts, no transcript converts, no transcript, no request, no AttributeError
MarkItDown(llm_client=c, audio_model="speech-ctor") no request model speech-ctor
same, plus llm_model="img-model" no request still speech-ctor
same constructor, convert(..., audio_model="speech-call") no request speech-call
AzureOpenAI client with audio_model="my-deploy" no request path /openai/deployments/my-deploy/audio/transcriptions?api-version=2024-06-01, model my-deploy
server answers 401 no request FileConversionException caused by an AuthenticationError, one request

Tests, packages/markitdown/tests/test_module_misc.py:

  • On this head: 185 passed, 1 failed, 1 skipped. The failure is test_speech_transcription.
  • The same file copied onto main source: 13 failed. That is the 12 new audio tests plus test_speech_transcription.
  • test_speech_transcription also fails on main alone in my environment. It needs the legacy speech path, which I do not have, so it is not caused by this PR.

Limits. Because SpeechRecognition is not installed here, I could not watch the "no Google fallback after a provider error" behavior myself. In my 401 run there was only one request to my server and the error came out. The no-fallback test in the PR (test_audio_selected_client_failure_never_falls_back) passes on this head and fails on main, but it uses mocks. I did not run the full core suite, so I have not seen your 1,066 number. I did not test a client without whisper-1, or a real provider.

My two questions are resolved. I have no other open points.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants