Repository navigation
Conversation
Text analysis (line, word and grapheme segmentation, bidi classes and paired brackets, script and emoji properties) now reads cleft's tables and segmenters instead of icu_segmenter/icu_properties, and the pairwise canonical (de)composition used for font fallback comes from harfrust::unicode instead of icu_normalizer. - parley_data and parley_data_gen are removed: cleft's packed property words replace the PackTab table, and the few sets cleft does not expose (General_Category Cc, Variation_Selector, Emoji_Modifier(_Base)) are small fixed ranges in analysis.rs. - CharInfo::script is now parlance::Script. - DictionaryProvider (icu_provider based) is replaced by cleft::Dictionaries; the complex-scripts feature now enables cleft/dictionaries. The external_dictionaries example is removed pending a published generator for cleft's dictionary format. - Line-break runs with different CSS options restart the scan at the last opportunity before the seam (LineBreaks::with_preceding) instead of carrying one character back. - contributes_to_shaping uses Default_Ignorable_Code_Point (minus the Hangul fillers HarfBuzz draws) in place of General_Category Format.
|
I'll fix CI failures and address comments from users with write access that start with 'Devin'.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM Contributions: Code changes generated with Devin.
Prototype of replacing ICU4X with
cleft(dfrg's no_std, zero-dependency Unicode crate) for text analysis, on top of linebender#898 (merged withmain; the base branchharfrust-0.15-plus-mainis exactly that merge, so this PR's diff is the swap alone). Not meant to merge as is — it exists to measure. Full write-up with the method and raw data: see the session attachments; summary below.What changes:
cleft; canonical (de)composition for font fallback comes fromharfrust::unicode(harfrust 0.15). Noicu_*crate remains in the dependency graph;parley_data/parley_data_genare deleted (the few sets cleft does not expose —Cc,Variation_Selector,Emoji_Modifier(_Base)— are fixed ranges inanalysis.rs).CharInfo::scriptis aparlance::Script;DictionaryProvider(icu_provider) is replaced bycleft::Dictionaries(set_dictionarieson the builders);complex-scriptsenablescleft/dictionaries. Theexternal_dictionariesexample is removed (cleft's.dafsagenerator is not published yet).line-break/word-breakoptions restart the scan at the last opportunity before the seam (LineBreaks::with_preceding) instead of carrying one character back.contributes_to_shapingusesDefault_Ignorable_Code_Point(minus the Hangul fillers HarfBuzz draws) in place ofGeneral_Category=Format.Results (x86_64 Linux VM, details in the write-up):
tiny_skia_render, release, stripped: main 2,703 KB → Upgrade to read-fonts 0.46, skrifa 0.49, harfrust 0.15 linebender/parley#898 3,023 KB → prototype 2,974 KB; withcomplex-scripts: 6,529 KB → 6,849 KB → 5,665 KB (−1.2 MB vs Upgrade to read-fonts 0.46, skrifa 0.49, harfrust 0.15 linebender/parley#898).documentgroup from Add larger "document scale" benchmarks for each pipeline phase linebender/parley#793), build-phase groups, median: Upgrade to read-fonts 0.46, skrifa 0.49, harfrust 0.15 linebender/parley#898 vs main −16.8%, prototype vs Upgrade to read-fonts 0.46, skrifa 0.49, harfrust 0.15 linebender/parley#898 −7.8% (noise floor of a self-compare: −1.0%). Latin/Japanese −6…−10%, Arabic −1.5…−4%.analyze_textis now the largest analysis leaf).Tests: all pass, parley_tests snapshots unchanged. Two
test_analysisexpectations that encoded ICU4X behaviour changed:line-break: loosebefore U+2010/U+2013 needs cleft'sloose_hyphensto match ICU4X/CSS Text 3, and cleft's Thai/Lao dictionaries segmentกกinto two words where ICU4X did not (that test now checks only the\nhandling it was written for).Changelog
Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/7519feeb2a15465c9b5d89742fb9a85c
Open in Devin Desktop: https://dioxus.staging.devinenterprise.com/desktop/session/7519feeb2a15465c9b5d89742fb9a85c?variant=devin-insiders