Skip to content

perf(xlsx): import pandas only for spreadsheets (1.7x faster startup, 41 MB less memory) - #2711

Open
Afonso Ribeiro (deadstrobe5) wants to merge 1 commit into
microsoft:mainfrom
deadstrobe5:lazy-pandas-import
Open

Afonso Ribeiro (deadstrobe5) wants to merge 1 commit into
microsoft:mainfrom
deadstrobe5:lazy-pandas-import

Conversation

@deadstrobe5

Copy link
Copy Markdown

import markitdown currently imports pandas (with openpyxl and xlrd) at module load, even though only the XLSX and XLS converters use it. pandas is the slowest import in the package, so every CLI call, MCP server start or serverless cold start pays for it, including PDF, DOCX and HTML conversions.

This moves the import into a small _import_pandas() helper that both spreadsheet converters call at the start of convert(). Missing dependencies still raise MissingDependencyException with the same message and the original ImportError chained. The tests that patched _xlsx_converter.pd now patch pandas.read_excel, and a new subprocess test checks that importing markitdown doesn't load pandas, openpyxl or xlrd.

Measured on CPython 3.14 (Apple M5) with fresh processes, alternating old and new:

before after
import markitdown; MarkItDown(), base extras 304 ms 163 ms
same with [all] 362 ms 215 ms
python -m markitdown test.docx 403 ms 234 ms
peak RSS after import 144 MB 102 MB

Spreadsheet conversions take the same time, since they now pay the import on first use. Output is byte-identical on all the test files and a few generated docx/pptx/html/md files. The core and OCR tests pass, black is clean.

@deadstrobe5

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Community review of e09bc78: no blocking findings.

I compared the spreadsheet converter with main at 4cc9fa1 in eight fresh-process checks, with pandas, openpyxl, and xlrd available or individually blocked by an import hook. Importing and initializing MarkItDown, then converting HTML, leaves all three unloaded on this head. Repeated conversions of the repository's test.xlsx and test.xls produce identical Markdown to main. With dependencies blocked, the affected converter preserves the extra-specific MissingDependencyException message and original ModuleNotFoundError cause; blocking one Excel engine still allows conversion with the other.

Validation on macOS, Python 3.12.15, pandas 3.0.6, openpyxl 3.1.5, and xlrd 2.0.2:

  • GITHUB_ACTIONS=1 UV_OFFLINE=1 hatch test --python 3.12 -- -q -p no:cacheprovider: 1,051 passed, 57 skipped.
  • OCR package tests: 108 passed, including the inherited spreadsheet repair/image hooks.
  • pre-commit run --all-files: passed with the repository's Black 23.7.0 hook.

The existing remote/credential/platform/tool skips apply. I did not remeasure the timing or memory figures, or run the other Python/OS combinations.

@deadstrobe5

Copy link
Copy Markdown
Author

Thanks for the thorough independent check, especially the blocked-dependency cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants