Skip to content

fix(utils): separate HTML5 block elements and table headers in htmlToText - #4366

Merged
barjin merged 1 commit into
apify:masterfrom
SwastikTripathi:fix/html-to-text-html5-blocks
Nov 9, 2026
Merged

barjin merged 1 commit into
apify:masterfrom
SwastikTripathi:fix/html-to-text-html5-blocks

Conversation

@SwastikTripathi

Copy link
Copy Markdown
Contributor

htmlToText only adds line breaks around the block tags in BLOCK_TAGS_REGEX, and that list stops at HTML 4. Sibling HTML5 sections were glued together, which is the merging of distinct text that the function's docs say it tries to avoid:

HTML Before After
<section>Call 555-1234</section><section>9876 Main St</section> Call 555-12349876 Main St Call 555-1234\n9876 Main St
<header>Head</header><main>Body</main><footer>Foot</footer> HeadBodyFoot Head\nBody\nFoot
A<hr>B AB A\nB
<dl><dt>Term</dt><dd>Def</dd></dl> TermDef Term\nDef
<details><summary>Sum</summary>Body</details> SumBody Sum\nBody
<tr><th>H1</th><th>H2</th></tr> H1H2 H1\tH2\t

The first row shows the practical effect: social.parseHandlesFromHtml runs on this text, so a phone number at the end of one <section> runs into the next section's digits.

The list now also has the HTML5 sectioning and grouping elements a browser lays out as blocks (article, aside, section, nav, header, footer, main, hgroup, figure, figcaption, details, summary, dialog, menu), plus hr, dt, dd, legend and caption. <th> cells get the same tab separator as <td>.

Added a test with those cases to test/utils/cheerio.test.ts. None of the new tags appear in the existing inputs (including html-to-text-test-data.ts and the social tests), so their expected output doesn't change.

Verification: I couldn't run the repo's vitest suite here. I ran the new test's cases, plus several existing cases, through the published @crawlee/utils build, once as released and once with the same two-line change applied to its compiled cheerio.js (the block-tag and td code is the same as on master). As released, all 7 new cases fail. With the change, all 7 pass, both bare and wrapped in a full <html><body> document as checkHtmlToText does, and the existing cases I retyped give the same output as before. oxfmt --check passes on both files.

🤖 Generated with Claude Code

…Text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 7, 2026
@janbuchar
janbuchar requested a review from barjin November 9, 2026 09:20

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you @SwastikTripathi !

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you @SwastikTripathi !

@barjin
barjin merged commit 589621f into apify:master Nov 9, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants