XLSX/XLS: keep repeated header names instead of Total.1, Item.1

[Cindy (cindykrafft)]

pandas renames duplicate column labels, and pandas 2 removed the option to turn that off, so a header row Item | Total | Total | Item is rendered as Item | Total | Total.1 | Item.1. The Markdown table no longer matches the spreadsheet.

When a label looks renamed, this re-reads that sheet's raw first row and uses the original labels. Eleven other sheet shapes compared give identical output.

This builds on the XLSX text-cells PR; the branch contains both commits, so please merge that one first or review just the last commit.

Tests

• test_xlsx_repeated_header_names_are_kept fails on main (| Item | Total | Total.2 | Item.1 | Total.1 | NA | Unnamed: 6 |) and passes with the change.
• With both changes: packages/markitdown 1098 passed, 12 skipped; packages/markitdown-ocr 108 passed. black clean.

Disclosure: this was found, reproduced and patched by an AI agent (Claude Code), which ran the reproduction and tests quoted above. It has not been separately reviewed by a human.

[Bicheng (Kenneth) (BichengWang)]

Reviewed 20b92625a3bcd4252a4b4a4430dc74cd9e5dd1ca (both commits), with one nonblocking coverage gap:

Repeated NA-like text headers still receive suffixes. The raw reread at _xlsx_converter.py:60–63 uses pandas' default NA parsing, so _restore_header sees missing values for literal NA, N/A, and NULL headers and retains their mangled names. For example, this workbook still produces | NA | NA.1 |:

import io
import openpyxl
from markitdown import MarkItDown, StreamInfo

workbook = openpyxl.Workbook()
workbook.active.append(["NA", "NA"])
workbook.active.append([1, 2])
stream = io.BytesIO()
workbook.save(stream)
stream.seek(0)
print(MarkItDown().convert_stream(
    stream, stream_info=StreamInfo(extension=".xlsx")
).markdown)

The same reproduction yields | N/A | N/A.1 | and | NULL | NULL.1 |. The helper docstring acknowledges this fallback, but the PR's stated behavior has this exception. Consider disabling NA parsing for the header reread while retaining a separate fallback for genuinely blank header cells, with parametrized coverage for these labels. Reading only the first row would also avoid parsing the entire sheet again.

Validation on Python 3.12.15 / pandas 3.0.6: core suite 1053 passed, 57 skipped with GITHUB_ACTIONS=1; OCR suite 108 passed; pre-commit run --all-files passed. These used direct pytest runs in an offline environment. The three additional header assertions above failed as described; header-only, numeric, date, and mixed text/numeric header probes converted successfully. I did not generate new .xls fixtures.

[Cindy (cindykrafft)]

Thanks for checking this so thoroughly. You're right; that was a gap. 2cee2ab addresses it:

• The header re-read now reads only the first row (nrows=1) with na_filter=False, so NA, N/A, NULL (and nan) header text stays as written.
• Blank header cells still keep pandas' Unnamed: n names. They come back as "" with NA parsing off, and a first row narrower than the sheet (empty trailing header cells) is padded the same way.
• New parametrized test test_xlsx_repeated_na_like_header_names_are_kept (NA, N/A, NULL, nan, blank) plus test_xlsx_repeated_headers_with_a_narrow_first_row. The four text labels fail on 20b9262 and pass now.

Validation (Python 3.12, pandas 3.0.6): core suite 1059 passed, 57 skipped with GITHUB_ACTIONS=1; OCR suite 108 passed; black 23.7.0 clean on the changed files. As before, I did not add new .xls fixtures; the XLS path uses the same helper with xlrd.

Disclosure: this fix and reply were produced by an AI agent (Claude Code), which ran the reproduction and tests quoted above.

[PRABHU KIRAN VANDRANKI (VANDRANKI)]

I ran this on Windows with Python 3.13.3, pandas 3.0.6, openpyxl 3.1.5 and xlrd 2.0.2, against upstream main (4cc9fa1) and the PR head. This branch also contains the text-cells commit from #2708, which I reviewed earlier, so this covers the header change.

• tests/test_xlsx.py on the PR head: 35 passed. The same test file against main's source: 8 failed, 27 passed. Those 8 are the new text-cell, NA-marker, repeated-header and narrow-first-row cases.
• Full packages/markitdown/tests on the PR head: 1113 passed, 3 skipped, 1 failed. The failure is test_speech_transcription, a FileNotFoundError from the audio converter on this machine. I did not run that test on main.
• I compared main and the PR on 13 workbooks: dates and times, mixed columns, int and float columns, booleans, empty columns, big ints, numeric and date headers, text that looks like numbers, and an all-empty sheet. The output differs only for repeated headers (1 | 1.1 becomes 1 | 1; two equal date headers no longer get .1) and for text cells (1.50 and 2 stay 1.50 and 2; main gave 1.5 and 2.0).
• Three sheets in one workbook (repeated x x x, a plain sheet with 01, and a sheet with k k): all three come out right on the PR. Main gives x x.1 x.2 and k k.1. A stream that starts at offset 4 still gives the restored x x x header, so the re-read of the first row seeks back correctly.
• Names that already contain dots (a.b a.b, v1.2 v1.2, and a real w.1 next to w) come back as written.
• test.xls converts to a correct table through the xlrd path. I could not write an .xls with repeated headers here, so the xlrd side of the header fix is read-only. The markitdown-ocr test change (the added dtype: object kwarg) I also only read.
• git merge-tree of the #2708 head and this head merges clean.

The stacking is stated in the description. Approving.