[Muhammad Usman Mateen (usmanmateen)]

Fixes #2678

Western European text in Windows-1252 often scores exactly like cp1250 (or a non-Latin code page) in charset_normalizer, and best() then returns whichever comes first. "Café crème" came out as "Cafﻠ crﻟme" and "crème" as "crčme".

Changes

• New best_charset_match() in _charset_utils.py. When cp1252 or cp1250 ties with the best match in both chaos and coherence, it picks between them from the bytes. It takes cp1250 when the data has:

  • S or Z with caron (0x8A 0x8E 0x9A 0x9E), which are the same in both code pages;
  • a byte that cp1252 reads as a symbol but cp1250 as a letter (ą ł Ľ Ł Ą Ż ż, read as ¹ ³ ¼ £ ¥ ¯ ¿) right before a letter, where ż (¿) must also follow a letter so "¿Qué" stays Spanish;
  • ő (õ in cp1252) anywhere other than before "e", since Portuguese only has õ in "õe".

  Otherwise it takes cp1252. With no tie, it returns best() as before.

• _get_stream_info_guesses and the fallbacks in PlainTextConverter and CsvConverter use it instead of from_bytes(...).best().

• I left the fallback in OutlookMsgConverter unchanged. It only runs when a declared code page fails to decode, and its tests patch from_bytes in that module. I'm happy to switch it too if you prefer.

The trade-off is in #2678. Some short cp1250 samples now decode as cp1252, so they move from correct to wrong. This happens when the only non-ASCII letters are ones that are also Western letters (č/è, Č/È, ř/ø, ű/û), or when ą only ends words. That includes Croatian, Slovene, Bosnian, Czech and Hungarian text without the letters above. For example, "Česká republika" decodes as "Èeská republika".

Tests

• test_western_european_cp1252_text_is_decoded (French, Spanish, Portuguese, Danish) and test_western_european_cp1252_csv_is_decoded: fail on main, pass here.
• test_central_european_cp1250_text_is_still_decoded (Polish, Czech, Hungarian, Croatian): pass on main and here.
• test_central_european_cp1250_tie_is_decoded (Polish, Slovak, Hungarian samples that tie with cp1252): pass on main, would fail with the cp1252 preference alone, pass here.
• test_western_european_cp1252_lookalike_is_decoded ("As razões", "¿Qué tal?", "Smith¹", "Nº 5"): the same bytes in Western positions still decode as cp1252.
• Full suite in packages/markitdown: 1111 passed, 12 skipped. test_speech_transcription fails both on main and here, because it needs the speech recognition service.
• black 23.7.0 (the pre-commit version) leaves the changed files unchanged.

[Muhammad Usman Mateen (usmanmateen)]

@microsoft-github-policy-service agree

[PRABHU KIRAN VANDRANKI (VANDRANKI)]

Community review, not a maintainer review and it does not clear the merge gate. I read the full diff and ran it on Windows with Python 3.13 and charset-normalizer 3.5.2.

What I ran:

• With the PR's tests/test_module_misc.py applied on top of main (4cc9fa1), the 5 test_western_european_cp1252_text_is_decoded cases and test_western_european_cp1252_csv_is_decoded fail. On the PR branch they pass, and the 4 cp1250 cases pass on both.
• Full packages/markitdown suite: main 1097 passed, PR 1107 passed. The same 7 tests fail on both (test_speech_transcription and 6 test_cli_vectors cases for vectors 13 and 15). I did not look into why they fail here, so I can only say they are identical on both branches.

Extra probe, converting text through convert_stream(..., file_extension=".txt") on main and on the PR. These are my own short sentences, so treat the counts as anecdotal:

• cp1252 gains: "naïve façade", "Sjöberg ångström" and "Café crème" (latin-1 encoded) go from garbled to correct.
• cp1250 regressions: "Česká republika" and a Hungarian sentence ("Minden ember szabadnak születik és egyenlő méltósággal...") decode correctly on main but not on the PR. "Česká republika" comes out as "Èeská republika", and the Hungarian sentence gets "egyenlõ" instead of "egyenlő". Neither contains S, s, Z or z with caron, so the guard in best_charset_match does not apply, and the cp1252 tie wins.
• The PR description names Croatian, Slovene and Bosnian as the trade-off. From what I ran, Czech and Hungarian text without š/ž is affected the same way, so that might be worth listing in the description and in #2678.
• Of the 13 cp1250 strings I tried, 6 decoded correctly on main and 4 on the PR. Several others are wrong on both (for example the Polish sentence "Wszyscy ludzie rodzą się wolni...", and short strings like "Kraków" that detect as a CJK code page). This is a small sample, not a benchmark.

Not fixed by this PR, probably out of scope: in my probe, short cp1252 strings such as "Café", "Müller Straße" and "Ürün" still decode to unrelated characters (CJK or Arabic presentation forms). I did not dig into why, so I do not know whether it is the same tie.

Small things:

• _markitdown.py still has import charset_normalizer at line 18, and after this change nothing else in that file references it (grep shows only the import line). It can probably go.
• OutlookMsgConverter still uses from_bytes(...).best(). The description explains why, so I have no objection.

Verdict: the change does what it says and the new tests are meaningful (fail on main, pass here). My concern is only that the heuristic moves some cp1250 text from correct to wrong, which seems an acceptable trade if the maintainers want it, but it should be stated plainly.

[Muhammad Usman Mateen (usmanmateen)]

Thanks for the thorough review, and for running it on Windows. You were right about Czech and Hungarian, and some of it was fixable, so I pushed 9e6b615:

• On a tie, it now prefers cp1250 when the bytes point to it, not only for S or Z with caron. Two signals:
  • cp1252 would read a Central European letter as a symbol right before a letter: ą, ł, Ľ, Ł, Ą, Ż read as ¹ ³ ¼ £ ¥ ¯ (and ż as ¿, but only between two letters, so "¿Qué" stays Spanish).
  • cp1252 would read ő as õ anywhere other than before "e". Portuguese only has õ in "õe"/"ões".
• Removed the unused import charset_normalizer from _markitdown.py.
• New tests:
  • test_central_european_cp1250_tie_is_decoded: Polish, Slovak and Hungarian samples that tie. They fail on the previous commit and pass now.
  • test_western_european_cp1252_lookalike_is_decoded: "As razões", "¿Qué tal?", "Smith¹" and "Nº 5". These keep the same bytes in Western positions decoding as cp1252.

On my own set of 17 short strings, comparing main → previous commit → now:

• cp1250 decoded correctly: 5 → 0 → 4 of 8.
• cp1252 decoded correctly: 0 → 7 → 7 of 9.

Your two examples behave differently:

• "egyenlő" is fixed.
• "Česká republika" is not. Č and È are both letters there, and nothing else in the string separates Czech from French, so it stays the documented trade-off.
• Polish with ą only at the end of a word ("rodzą się") also still goes to cp1252, because "Smith¹"-style footnotes look the same.

I've updated the description and #2678 to list Czech and Hungarian alongside Croatian, Slovene and Bosnian, and to say plainly that this moves some short cp1250 text from correct to wrong.

The short cp1252 strings that still decode as CJK or Arabic ("Müller Straße", "m³") are a different case: cp1252 isn't among the detector's tied matches there. I agree that's out of scope here.

Full suite on Linux, Python 3.12: 1111 passed, 12 skipped. black 23.7.0 leaves the files unchanged.

[PRABHU KIRAN VANDRANKI (VANDRANKI)]

Thanks for the update. I re-checked 9e6b615 (community check, not a maintainer review; it does not clear the merge gate).

What I ran on Windows, Python 3.13, charset-normalizer 3.5.2, convert_stream with StreamInfo(extension=".txt"), main (4cc9fa1) against this head:

• The Hungarian sentence I raised ("Minden ember szabadnak születik és egyenlő méltósággal és jogokkal.") now decodes correctly. It decodes correctly on main and on this head. In my earlier review it was wrong on the previous commit.
• "Česká republika" is still wrong on this head ("Èeská republika"), right on main. That matches what you wrote.
• Still correct on both main and this head: "Zażółć gęślą jaźń", "Węgry i Polska, Łódź, Żółw", "Magyarország, őrült, tűz, főváros", "Slovenský jazyk, ľudia, Ľubovňa", "Příliš žluťoučký kůň úpěl ďábelské ódy".
• The cp1252 gains hold: "naïve façade", "Sjöberg ångström", "Café crème", "Pão de açúcar e põe à mesa, ações", "Größe: 5m³, Fläche 3m²" are garbled on main and correct on this head. "As razões do Nº 5, ¿Qué tal?" and "Please see footnote¹ and note²" are correct on both.
• Some short strings are wrong on both and unchanged, for example "Kraków" and "Gospodarska zajednica, šuma, žaba". I also tried a few cp1252 sentences with ¼, £, ¥ or ¿ before a letter. The ones that were wrong on main ("The café serves ¼lb burgers...", "Prix: 15¥ chez Müller...") are wrong in the same way on this head. I did not find a text that main decodes correctly and this head breaks, other than the Czech example above. That is a small hand-made set, not a benchmark.
• Tests: tests/test_module_misc.py gives 186 passed, 1 failed, 1 skipped on this head. The failure is test_speech_transcription, which also failed in my earlier run, on main as well as on the PR. With _charset_utils.py put back to the previous commit (6bfc092), the 3 new test_central_european_cp1250_tie_is_decoded cases (pl, sk, hu) fail, so they cover this change.
• import charset_normalizer is gone from _markitdown.py.

I did not run the full package suite. I did not read the PR description again, so I have not checked that it lists the Czech case. Whether to accept the remaining Czech case is for the maintainers.

[Muhammad Usman Mateen (usmanmateen)]

Hi Adam Fourney (@afourney), gentle ping on this when you have a moment. The community re-check above on Windows confirms the fixes, with the Czech trade-off already described in #2678. Happy to adjust anything.