[zurfyx]

Description

When a code block is highlighted again, the highlighter replaces every token between the first and the last one that changed with fresh CodeHighlightNodes, and fresh nodes have no NodeState. So any edit that changes tokens on both sides of a word (two edits in one update, an annotation added over a range that starts and ends inside tokens, a quote typed before the word that turns the rest of the line into a string) drops the NodeState of the text in between: comment ids, suggestions, any annotation an app keeps in NodeState. MarkStateExtension's own ids were put back by its edit tracking, but only the ones the text had before the update: a comment added in that update (over a range in a code block) was dropped too.

Found by the collaboration soak: a stale agent comments a range of a highlighted code line while another writer replaces a word in it; the word kept its comment but lost a second annotation (a set-valued NodeState key). It reproduces with no collaboration: the agent's own edit already dropped it, and the binding then wrote the loss.

Highlighting now keeps each character's state: $keepNodeStateOf (in @lexical/code-core, used by Prism and Shiki) gives the new tokens the NodeState their text had, splitting a token where the state changes, and the diff that decides which tokens to replace compares NodeState too. One rule: highlighting changes how code is shown, never the state its text carries.

CompositionHoldBack: a highlighted code line now keeps a remote comment the composer reads on a word it is not composing in through its own keystrokes (before, its next keystroke dropped it from its editor until the composition ended), while the hold counts the comment as held. Where the composer changes the line's type while composing, the (opt-in) self-check of that write notes the comment until the hold ends; every client then shows the same. Those 4 cases now expect exactly that diagnostic (as S7-8, the same kind of hold-versus-highlight difference); every other check of the matrix still runs on them (convergence, fresh readers, tokens, comments).

Changeset

.changeset/code-highlight-keep-char-state.md ([fix]).

Test plan

New CodePrismRetokenizeNodeState.test.ts and CodeShikiRetokenizeNodeState.test.ts: an annotated word through edits on both sides in one update, a quote typed before it, and part of a token annotated.

Before

$ pnpm exec vitest run --project unit --reporter=verbose packages/lexical-code-prism/src/__tests__/unit/CodePrismRetokenizeNodeState.test.ts packages/lexical-code-shiki/src/__tests__/unit/CodeShikiRetokenizeNodeState.test.ts
 ✓ Prism (Prism) > the annotated word is unchanged
 ✓ Prism (Prism) > edits on both sides of the word, in one update
 × Prism (Prism) > a quote typed before the word turns it into part of a string token
 × Prism (Prism) > part of a token annotated
 ✓ Shiki (Shiki) > the annotated word is unchanged
 × Shiki (Shiki) > edits on both sides of the word, in one update
 × Shiki (Shiki) > a quote typed before the word turns it into part of a string token
 × Shiki (Shiki) > part of a token annotated
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 5 ⎯⎯⎯⎯⎯⎯⎯
      Tests  5 failed | 3 passed (8)

After

$ pnpm exec vitest run --project unit --reporter=verbose packages/lexical-code-prism/src/__tests__/unit/CodePrismRetokenizeNodeState.test.ts packages/lexical-code-shiki/src/__tests__/unit/CodeShikiRetokenizeNodeState.test.ts
 ✓ Prism (Prism) > the annotated word is unchanged
 ✓ Prism (Prism) > edits on both sides of the word, in one update
 ✓ Prism (Prism) > a quote typed before the word turns it into part of a string token
 ✓ Prism (Prism) > part of a token annotated
 ✓ Shiki (Shiki) > the annotated word is unchanged
 ✓ Shiki (Shiki) > edits on both sides of the word, in one update
 ✓ Shiki (Shiki) > a quote typed before the word turns it into part of a string token
 ✓ Shiki (Shiki) > part of a token annotated
      Tests  8 passed (8)

• Unit suites of @lexical/y, the code packages, @lexical/mark, @lexical/markdown, @lexical/html, @lexical/clipboard and the playground: all pass (258 files, 9,492 passed).
• The soak repro (a stale comment over a code line while its word is replaced, in a paragraph, a code block and a code block turned into a list, both orders): all pass; 6 failed before.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
lexical	Ready	Preview	Oct 30, 2026 12:10am UTC
lexical-playground	Ready	Preview	Oct 30, 2026 12:10am UTC

[pkg-pr-new]

lexical

npm i https://pkg.pr.new/lexical@9558

@lexical/a11y

npm i https://pkg.pr.new/@lexical/a11y@9558

@lexical/clipboard

npm i https://pkg.pr.new/@lexical/clipboard@9558

@lexical/code

npm i https://pkg.pr.new/@lexical/code@9558

@lexical/code-core

npm i https://pkg.pr.new/@lexical/code-core@9558

@lexical/code-prism

npm i https://pkg.pr.new/@lexical/code-prism@9558

@lexical/code-shiki

npm i https://pkg.pr.new/@lexical/code-shiki@9558

@lexical/compiler

npm i https://pkg.pr.new/@lexical/compiler@9558

@lexical/devtools-core

npm i https://pkg.pr.new/@lexical/devtools-core@9558

@lexical/dragon

npm i https://pkg.pr.new/@lexical/dragon@9558

@lexical/eslint-plugin

npm i https://pkg.pr.new/@lexical/eslint-plugin@9558

@lexical/extension

npm i https://pkg.pr.new/@lexical/extension@9558

@lexical/fast-check

npm i https://pkg.pr.new/@lexical/fast-check@9558

@lexical/file

npm i https://pkg.pr.new/@lexical/file@9558

@lexical/hashtag

npm i https://pkg.pr.new/@lexical/hashtag@9558

@lexical/headless

npm i https://pkg.pr.new/@lexical/headless@9558

@lexical/history

npm i https://pkg.pr.new/@lexical/history@9558

@lexical/html

npm i https://pkg.pr.new/@lexical/html@9558

@lexical/internal

npm i https://pkg.pr.new/@lexical/internal@9558

@lexical/link

npm i https://pkg.pr.new/@lexical/link@9558

@lexical/list

npm i https://pkg.pr.new/@lexical/list@9558

@lexical/mark

npm i https://pkg.pr.new/@lexical/mark@9558

@lexical/markdown

npm i https://pkg.pr.new/@lexical/markdown@9558

@lexical/mdast

npm i https://pkg.pr.new/@lexical/mdast@9558

@lexical/offset

npm i https://pkg.pr.new/@lexical/offset@9558

@lexical/overflow

npm i https://pkg.pr.new/@lexical/overflow@9558

@lexical/plain-text

npm i https://pkg.pr.new/@lexical/plain-text@9558

@lexical/react

npm i https://pkg.pr.new/@lexical/react@9558

@lexical/rich-text

npm i https://pkg.pr.new/@lexical/rich-text@9558

@lexical/selection

npm i https://pkg.pr.new/@lexical/selection@9558

@lexical/table

npm i https://pkg.pr.new/@lexical/table@9558

@lexical/tailwind

npm i https://pkg.pr.new/@lexical/tailwind@9558

@lexical/text

npm i https://pkg.pr.new/@lexical/text@9558

@lexical/utils

npm i https://pkg.pr.new/@lexical/utils@9558

@lexical/y

npm i https://pkg.pr.new/@lexical/y@9558

@lexical/yjs

npm i https://pkg.pr.new/@lexical/yjs@9558

commit: c9d0a74

[zurfyx]

Converting to draft: a randomized collaboration soak (same seeds with and without this change) found a regression this change introduces. 5 of 1,500 sessions differ, all reproducing deterministically with this PR alone on main:

• 1 session loses a token: a code line whose word now keeps a comment on part of its characters is split into tokens at that boundary; a paste into the line then makes the writer re-insert the first character of that word (a new item identity), and a collaborator's concurrent edit that refers to the old character is lost later.
• 4 sessions end with a self-check difference on a set-valued NodeState member in a code line (s:suggest…).

The node splits this change adds at state boundaries go through the writer's handling of replaced text nodes, which writes a removed node's text whole or not at all, so a short split piece next to a paste is written again. Keeping the per-character state is still the right behavior; it needs the writer to keep characters across such splits first. Leaving this as a draft until then.