Skip to content

Prototype: replace ICU4X with cleft for text analysis - #94

Open
nicoburns wants to merge 5 commits into
harfrust-0.15-plus-mainfrom
cleft-prototype
Open

nicoburns wants to merge 5 commits into
harfrust-0.15-plus-mainfrom
cleft-prototype

Conversation

@nicoburns

Copy link
Copy Markdown
Member

LLM Contributions: Code changes generated with Devin.

Prototype of replacing ICU4X with cleft (dfrg's no_std, zero-dependency Unicode crate) for text analysis, on top of linebender#898 (merged with main; the base branch harfrust-0.15-plus-main is exactly that merge, so this PR's diff is the swap alone). Not meant to merge as is — it exists to measure. Full write-up with the method and raw data: see the session attachments; summary below.

What changes:

  • Line, word and grapheme segmentation, bidi classes and paired brackets, script and emoji flags come from cleft; canonical (de)composition for font fallback comes from harfrust::unicode (harfrust 0.15). No icu_* crate remains in the dependency graph; parley_data/parley_data_gen are deleted (the few sets cleft does not expose — Cc, Variation_Selector, Emoji_Modifier(_Base) — are fixed ranges in analysis.rs).
  • CharInfo::script is a parlance::Script; DictionaryProvider (icu_provider) is replaced by cleft::Dictionaries (set_dictionaries on the builders); complex-scripts enables cleft/dictionaries. The external_dictionaries example is removed (cleft's .dafsa generator is not published yet).
  • Runs with different line-break/word-break options restart the scan at the last opportunity before the seam (LineBreaks::with_preceding) instead of carrying one character back.
  • contributes_to_shaping uses Default_Ignorable_Code_Point (minus the Hangul fillers HarfBuzz draws) in place of General_Category=Format.

Results (x86_64 Linux VM, details in the write-up):

Tests: all pass, parley_tests snapshots unchanged. Two test_analysis expectations that encoded ICU4X behaviour changed: line-break: loose before U+2010/U+2013 needs cleft's loose_hyphens to match ICU4X/CSS Text 3, and cleft's Thai/Lao dictionaries segment กก into two words where ICU4X did not (that test now checks only the \n handling it was written for).

Changelog

Changed

  • Breaking change: text analysis uses cleft instead of ICU4X. CharInfo::script is now a parlance::Script, DictionaryProvider is replaced by cleft::Dictionaries, and parley_data/parley_data_gen are removed.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/7519feeb2a15465c9b5d89742fb9a85c
Open in Devin Desktop: https://dioxus.staging.devinenterprise.com/desktop/session/7519feeb2a15465c9b5d89742fb9a85c?variant=devin-insiders

Text analysis (line, word and grapheme segmentation, bidi classes and
paired brackets, script and emoji properties) now reads cleft's tables
and segmenters instead of icu_segmenter/icu_properties, and the pairwise
canonical (de)composition used for font fallback comes from
harfrust::unicode instead of icu_normalizer.

- parley_data and parley_data_gen are removed: cleft's packed property
  words replace the PackTab table, and the few sets cleft does not expose
  (General_Category Cc, Variation_Selector, Emoji_Modifier(_Base)) are
  small fixed ranges in analysis.rs.
- CharInfo::script is now parlance::Script.
- DictionaryProvider (icu_provider based) is replaced by cleft::Dictionaries;
  the complex-scripts feature now enables cleft/dictionaries. The
  external_dictionaries example is removed pending a published generator
  for cleft's dictionary format.
- Line-break runs with different CSS options restart the scan at the last
  opportunity before the seam (LineBreaks::with_preceding) instead of
  carrying one character back.
- contributes_to_shaping uses Default_Ignorable_Code_Point (minus the
  Hangul fillers HarfBuzz draws) in place of General_Category Format.
@staging-devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant