Repository navigation
fix(audio): honor the selected transcription client without provider fallback - #2706
Çağdaş Yürekli (cagdasyurekli) wants to merge 2 commits into
Conversation
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
There was a problem hiding this comment.
I tested this on Windows with Python 3.13.3 and openai 3.19.2 (the Python SDK), against main (4cc9fa1) and this branch.
What works. I started a local HTTP server that answers /v1/audio/transcriptions, and I pointed a real openai.OpenAI(base_url=...) client at it. I converted test.wav, test.mp3 and test.m4a with MarkItDown(llm_client=client).
- On this branch each file posts one multipart request. The filenames are
audio.wav,audio.mp3andaudio.mp4. The content types areaudio/wav,audio/mpegandaudio/mp4. The body sizes are 265 to 266 bytes larger than the three files, which fits the multipart overhead, so the whole file is sent. The markdown ends with the transcript. - On main no request is sent. SpeechRecognition is not installed in my environment, so the transcript step is skipped and the markdown has no transcript.
The PR's own 9 audio tests pass on the branch. With the PR's test file on main, 8 fail and 1 passes (the no-client legacy test).
The risk I found. llm_client is also how users enable image captions. A client that supports chat but has no audio endpoint now breaks audio conversion. I used a minimal object with only chat:
- main:
convert("test.wav")succeeds. - this branch:
FileConversionException, caused byAttributeError: 'NoAudio' object has no attribute 'audio'.
I did not run an Azure client or a client whose provider has no whisper-1 model. Those would send the request and fail at the provider, and by the PR's design that error propagates with no fallback. I read that as the same kind of change for existing users, but I did not run it.
So anyone who already passes llm_client only for images, and converts audio too, gets a new failure after upgrading. The README text in the PR describes the new behavior, but it is a silent change for existing code.
Questions for the author:
- Should the audio path be used only when the client has an
audio.transcriptionsattribute, and otherwise keep the old path? That would avoid theAttributeErrorcase above. whisper-1is hard coded andllm_modelis not used. Azure deployments need their own deployment name. Is a separate kwarg planned?
I am leaving this as a comment. The mechanics work, but I would want a decision on those two points before it merges.
|
PRABHU KIRAN VANDRANKI (@VANDRANKI), thanks for the Windows/SDK reproduction. Addressed both points in
The seven new review cases give six failures and one default-model control pass on the previous head, then all pass with this change. All16 audio cases and the full core suite pass (1,066 passed,57 skipped). An independent offline check with actual OpenAI/Azure SDK clients verifies default/constructor/override models, Azure deployment URLs, original WAV multipart data, and HTTP401 propagation without Google; five successful requests plus one rejection were mocked, with no network calls. Black23.7 and diff checks pass. Native Windows was not rerun locally; your original format/SDK findings are preserved and the follow-up was checked on Python3.12. |
|
Thanks for both changes. I re-checked 96f9764 (community check, not a maintainer review; it does not clear the merge gate). Both of my questions are answered. What I ran, on Windows with Python 3.13.3 and openai 3.19.2, against a local HTTP server that answers the transcription endpoints. I compared main (4cc9fa1) and this head. SpeechRecognition and pydub are not installed in my environment.
Tests,
Limits. Because SpeechRecognition is not installed here, I could not watch the "no Google fallback after a provider error" behavior myself. In my 401 run there was only one request to my server and the error came out. The no-fallback test in the PR ( My two questions are resolved. I have no other open points. |
The built-in audio converter ignored a configured transcription client and used SpeechRecognition’s legacy Google service. Use the supplied client's callable
audio.transcriptions.createendpoint with the original encoded stream, matching filename and MIME type. Once selected, provider errors propagate without cross-provider fallback.Chat-only clients retain the existing audio behavior: the optional legacy Google path is used when its dependencies are installed, and transcription is skipped when they are absent. This avoids breaking callers that configured
llm_clientfor image descriptions. Endpoint capability is checked before selecting a route, not after a failed provider request.audio_modelselects the transcription model or Azure deployment name, independently of the imagellm_model. It defaults towhisper-1, can be set onMarkItDown(...), and can be overridden per conversion. The provider client still needs its appropriate endpoint configuration. The README documents these choices and the legacy Google upload behavior.Addresses the configured-client part of #1284 and provides a current-main alternative to #326. It leaves the broader no-client Google-default policy question open.
Validation on Python 3.12: the original routing regression fails 8/9 cases on unchanged main; the review follow-up fails 6/7 cases on the preceding PR head. All 16 audio cases now pass, including chat-only/missing-dependency behavior, model precedence, and no fallback after provider failure. Full core: 1,066 passed, 57 skipped under existing CI/credential/optional skip conditions. Independent real OpenAI/Azure SDK MockTransport checks pass five success cases and one HTTP401 propagation case, including multipart payloads and Azure deployment URLs. No live provider calls. Black23.7 and diff checks pass.
Net change from main: +29 production, +169 tests, +25 documentation lines. No new dependencies, CLI flags, or hosted workflows.