Repository navigation
perf(xlsx): import pandas only for spreadsheets (1.7x faster startup, 41 MB less memory) - #2711
Afonso Ribeiro (deadstrobe5) wants to merge 1 commit into
Conversation
|
@microsoft-github-policy-service agree |
Bicheng (Kenneth) (BichengWang)
left a comment
There was a problem hiding this comment.
Community review of e09bc78: no blocking findings.
I compared the spreadsheet converter with main at 4cc9fa1 in eight fresh-process checks, with pandas, openpyxl, and xlrd available or individually blocked by an import hook. Importing and initializing MarkItDown, then converting HTML, leaves all three unloaded on this head. Repeated conversions of the repository's test.xlsx and test.xls produce identical Markdown to main. With dependencies blocked, the affected converter preserves the extra-specific MissingDependencyException message and original ModuleNotFoundError cause; blocking one Excel engine still allows conversion with the other.
Validation on macOS, Python 3.12.15, pandas 3.0.6, openpyxl 3.1.5, and xlrd 2.0.2:
GITHUB_ACTIONS=1 UV_OFFLINE=1 hatch test --python 3.12 -- -q -p no:cacheprovider: 1,051 passed, 57 skipped.- OCR package tests: 108 passed, including the inherited spreadsheet repair/image hooks.
pre-commit run --all-files: passed with the repository's Black 23.7.0 hook.
The existing remote/credential/platform/tool skips apply. I did not remeasure the timing or memory figures, or run the other Python/OS combinations.
|
Thanks for the thorough independent check, especially the blocked-dependency cases. |
import markitdowncurrently imports pandas (with openpyxl and xlrd) at module load, even though only the XLSX and XLS converters use it. pandas is the slowest import in the package, so every CLI call, MCP server start or serverless cold start pays for it, including PDF, DOCX and HTML conversions.This moves the import into a small
_import_pandas()helper that both spreadsheet converters call at the start ofconvert(). Missing dependencies still raiseMissingDependencyExceptionwith the same message and the original ImportError chained. The tests that patched_xlsx_converter.pdnow patchpandas.read_excel, and a new subprocess test checks that importing markitdown doesn't load pandas, openpyxl or xlrd.Measured on CPython 3.14 (Apple M5) with fresh processes, alternating old and new:
import markitdown; MarkItDown(), base extras[all]python -m markitdown test.docxSpreadsheet conversions take the same time, since they now pay the import on first use. Output is byte-identical on all the test files and a few generated docx/pptx/html/md files. The core and OCR tests pass, black is clean.