fix(pdf): prevent merging consecutive MasterFormat partial numberings (#2114)

[Adarsh Kumar (Adarsh-mk7)]

Resolves #2114

Summary of Bug

In packages/markitdown/src/markitdown/converters/_pdf_converter.py, _merge_partial_numbering_lines() (invoked on line 587 across all PDF conversions) post-processes text to merge standalone MasterFormat partial numbers (e.g. .1, .2, .10) with their subsequent body text.

However, when consecutive partial numbers appear (e.g. .1\n.2\nContractor shall furnish all materials. or .1\n.2 Contractor...), the function unconditionally merges .1 with .2 into an invalid compound token '.1 .2', stripping the .2 section identifier and orphaning the following text.

User Impact

CSI MasterFormat documents in construction, architecture, and engineering frequently have unnumbered sections, empty item stubs, or consecutive subsection headers. MarkItDown silently corrupts the numbering into invalid tokens like .1 .2 or .1 .2 Contractor..., producing corrupted markdown for downstream LLMs and ingestion pipelines.

Reproduction Steps

from markitdown.converters._pdf_converter import _merge_partial_numbering_lines

text = ".1\n.2\nContractor shall furnish all materials.\n.3\nWork shall comply with local codes."
print(_merge_partial_numbering_lines(text))

• Expected:

  .1
  .2 Contractor shall furnish all materials.
  .3 Work shall comply with local codes.
  

• Actual:

  .1 .2
  Contractor shall furnish all materials.
  .3 Work shall comply with local codes.
  

Root Cause

In _pdf_converter.py, the lookahead search loop advances j to the next non-empty line and merges lines[i] with lines[j] as long as j < len(lines), without checking whether lines[j] is itself a partial numbering or starts with one.

Explanation of Fix

• Introduced PARTIAL_NUMBERING_PREFIX_PATTERN = re.compile(r"^\.\d+(\s+|$)") to detect both standalone and inline partial numberings.
• Added a guard in _merge_partial_numbering_lines ensuring that lines[i] only merges with lines[j] if lines[j].strip() does not match PARTIAL_NUMBERING_PREFIX_PATTERN.
• If the next line is also a partial numbering, line[i] is preserved as-is and i advances by 1, allowing subsequent items and separating blank lines to be processed normally.
• Updated function docstring and comments.

Tests Added

Added comprehensive unit and integration regression coverage in packages/markitdown/tests/test_pdf.py (TestMasterFormatPartialNumbering):

1. test_merge_partial_numbering_consecutive_standalone: Verifies two consecutive standalone numbers (.1\n.2\nText).
2. test_merge_partial_numbering_multiple_consecutive_and_trailing: Verifies three consecutive numbers (.1\n.2\n.3\nText) and trailing numbers (.1\n.2\n.3).
3. test_merge_partial_numbering_consecutive_with_blank_lines: Parameterized test verifying blank line and whitespace preservation between numbers.
4. test_merge_partial_numbering_preceding_inline_numbered_line: Verifies standalone .1 does not merge into .2 Contractor....
5. test_merge_partial_numbering_varying_digit_lengths: Parameterized test across varying digit lengths (.1, .2, .9, .10, .99, .100).
6. test_merge_partial_numbering_preserves_multiple_blank_lines_between_sections: Verifies multi-line section spacing is preserved.
7. test_end_to_end_pdf_conversion_with_consecutive_partial_numbering: End-to-end integration test through MarkItDown().convert() and PdfConverter.convert().
8. test_merge_partial_numbering_boundary_conditions: Parameterized test verifying ordinary decimals (.5 mg, 1.2), dots (.a, ..1), and text are unaffected.

Test Results

• Regression tests: 56 passed in test_pdf.py (0 failed).
• Core multi-format tests: 89 passed (CSV, HTML, EPUB, ZIP).
• Office converters: 74 passed (XLSX, PPTX).
• Clean whitespace: git diff --check reported 0 errors.

[Adarsh Kumar (Adarsh-mk7)]

@microsoft-github-policy-service agree

[PRABHU KIRAN VANDRANKI (VANDRANKI)]

Community review, so this does not clear the merge gate.

I read the diff and ran it. The fix does what #2114 asks. One edge case is worth knowing about.

What I ran (Python 3.13.3, markitdown source from a checkout via PYTHONPATH):

• On the PR branch, pytest tests/test_pdf.py: 56 passed.
• On origin/main source (4cc9fa1) with the PR's test_pdf.py, -k partial_numbering: 13 failed, 12 passed. The 13 failures are all new tests, including the end-to-end one.
• The issue input .1\n.2\nContractor shall furnish all materials.\n.3\nWork shall comply with local codes. gives .1 .2\nContractor ... on main and .1\n.2 Contractor ...\n.3 Work ... on the PR, which is the expected output in the issue.

Edge case I checked directly: the guard is PARTIAL_NUMBERING_PREFIX_PATTERN = ^\.\d+(\s+|$), so any following line that starts with a dot, digits and a space is treated as another number. That is what makes .1\n.2 Contractor ... work. It also means a body line that happens to start with a decimal fraction is no longer merged. With .1\n.5 mg of cement shall be used per bag., main returns .1 .5 mg of cement shall be used per bag. and the PR returns the two lines unchanged. I think the trade-off is reasonable because the two cases cannot be told apart from text alone. The .5 mg case in test_merge_partial_numbering_boundary_conditions does not cover this, since there it follows no partial number. Maybe add a comment noting the limit.

Other notes:

• Open PR #2113 targets the same issue. Its guard uses PARTIAL_NUMBERING_PATTERN (a bare number only), and its diff also bundles an unrelated ipynb change. I did not run #2113.
• The new if line is about 100 characters. I could not run black here (not installed in my venv), so I do not know whether pre-commit would reformat it.
• I did not run the rest of the suite beyond test_pdf.py.

[Adarsh Kumar (Adarsh-mk7)]

Thanks for the thorough review and for testing this locally across both branches! Good catch on the decimal fraction edge case (like .5 mg)—I agree that distinguishing that from numbered prefixes purely from plain text is tricky, so I'll add an inline comment documenting this limitation and boundary condition near the regex pattern. I'll also run black / the pre-commit suite locally to ensure the line length on that if statement complies with project formatting guidelines. Appreciate the feedback and the heads-up on the other PR!