feat(ocr): add opt-in semantic PDF extraction

[Çağdaş Yürekli (cagdasyurekli)]

Summary

• add an opt-in semantic_pdf_ocr mode to the existing markitdown-ocr plugin
• render every PDF page for vision-assisted semantic Markdown extraction
• preserve headings, reading order, nested lists, and tables without changing the default converter path
• fall back to deterministic whole-document extraction when semantic OCR fails or returns empty

Addresses the remaining semantic-structure cases described in #41 and #83.

Design

This is deliberately plugin-scoped and disabled by default. PDF structure recovery is model-dependent, slower, and more expensive, so the deterministic core converter remains unchanged.

The semantic prompt asks the model to preserve source text and hierarchy and to treat document instructions as untrusted data. Constructor and per-conversion overrides are supported. Both pdfplumber and PyMuPDF rendering paths use the same prompt, and PyMuPDF resources are closed on every exit path.

Default OCR remains backward compatible with custom services that implement the original one-argument extract_text(image_stream) method.

Verification

• pytest -q packages/markitdown-ocr/tests/test_pdf_converter.py -k 'semantic or single_argument' — 8 passed, 14 deselected
• pytest -q packages/markitdown-ocr/tests/test_pdf_converter.py -k 'not test_pdf_multipage' — 21 passed, 1 deselected
• Black 23.7.0 — passed
• git diff --check — passed

The deselected multipage assertion is an unchanged baseline/dependency-version mismatch: current pdfplumber parses that fixture instead of taking the historical PyMuPDF fallback. New tests separately exercise the malformed-PDF PyMuPDF semantic path.

Limitations

Semantic output quality depends on the selected vision model. The tests verify routing, fallback, resource cleanup, option precedence, prompt constraints, and Markdown preservation with deterministic mocks; they do not claim perfect structure recovery for every arbitrary PDF.

[Çağdaş Yürekli (cagdasyurekli)]

@microsoft-github-policy-service agree

[Çağdaş Yürekli (cagdasyurekli)]

Rebased onto current main (4cc9fa1) and verified on Python 3.12. The rebase applied cleanly. Restored the mock imports required by the semantic PDF tests after the main-branch test cleanup. The OCR integration suite now passes without excluding the multipage test.

Validation: Black 23.7.0 and git diff --check pass; core pytest: 1050 passed, 57 skipped; OCR: 117 passed (repository CI skip policy).

Is this opt-in semantic PDF OCR feature still wanted? The branch is current and ready for maintainer feedback.

[Çağdaş Yürekli (cagdasyurekli)]

Full-page OCR now releases both raster copies and pdfplumber page caches after each processed page, including empty OCR and exceptions. Three real multipage regressions fail on the previous head and pass with the fix.

Validation: OCR suite 120 passed; core 1050 passed, 57 skipped. Black 23.7.0 and diff checks pass. No live model calls were used.