Keep Markdown in cleaned comments; add protoc-utils-markdown for shared reference links

CLAUDE.md

@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Build / test commands
 
-- `./gradlew build` — compiles, runs ktlint, and runs the test suite for both modules.
+- `./gradlew build` — compiles, runs ktlint, and runs the test suite for every module.
 - `./gradlew test` — runs tests only. The root `:test` task transitively triggers the recorder's GraalVM `nativeCompile` and a `publishToMavenLocal` (see *Test fixture wiring*), so a clean checkout's first run takes several minutes for the native image build.
 - `./gradlew test --tests "ServiceAndMethodWrapperTests"` — single test class. (`-i` for info logging.)
 - `./gradlew ktlintCheck` / `./gradlew ktlintFormat` — lint / autoformat. ktlint excludes everything under `build/` (workaround for the plugin not honoring generated-source exclusions). **Always run `ktlintFormat` after editing Kotlin source** so the next CI run isn't blocked on a trivial formatting failure.
@@ -17,10 +17,11 @@ The `pr.yaml` workflow just calls `build.yaml` (which runs `./gradlew build` aft
 
 ## Module layout
 
-The repo is a multi-project Gradle build with two modules:
+The repo is a multi-project Gradle build with three modules:
 
 - **Root project `:protoc-utils`** (`src/main/kotlin/com/engine/protoc/util/…`) — the published Kotlin library. Pure JVM, no native code.
 - **Subproject `:protoc-utils-recorder`** (`recorder/`) — a tiny GraalVM-native protoc plugin (single `Main.kt`) that copies stdin to a file named `code-generator-request.binpb`. It is shipped as multi-platform native binaries to Maven Central, **not** as a JVM jar.
+- **Subproject `:protoc-utils-markdown`** (`markdown/`) — a JVM library, depending on the root library and commonmark-java, that resolves proto reference links in doc comments and converts comment Markdown to plain text. Kept separate so the core library stays free of the commonmark dependency.
 
 `settings.gradle.kts` walks the tree and includes every directory containing a `build.gradle.kts` (skipping `build`, `buildSrc`, `tmp`, `scratch`, dotfile dirs, and dirs containing `.gradle_ignore`). Subproject names are derived as `:<root-name>-<relative-path>`, so the directory `recorder/` becomes `:protoc-utils-recorder`. To add a subproject, just create a directory with a `build.gradle.kts` — no manual `include(…)` needed.
 

README.md

@@ -39,6 +39,8 @@ dependencies {
 
 The companion `protoc-utils-recorder` plugin is published as multi-platform native binaries under the same group; see [`recorder/README.md`](recorder/README.md) for how to consume those.
 
+The companion `protoc-utils-markdown` library resolves proto reference links (`[Widget]`, `[Service.Rpc]`) in doc comments and converts comment Markdown to plain text; see [`markdown/README.md`](markdown/README.md).
+
 ---
 
 ## Why this exists
@@ -79,6 +81,7 @@ val comment: String? = nameElement?.location?.leadingComments?.cleaned
 | `cleaned` | The comment text with fencing characters (`//`, `/*`, `*`, etc.) stripped |
 
 The `cleaned` form is what you almost always want when generating documentation or OpenAPI descriptions.
+For ordinary `//` comments only the common leading whitespace is removed, so Markdown in the comment (a leading code span or reference link, list markers, code fences, tables) is preserved.
 
 ### Extension options
 

gradle/libs.versions.toml

@@ -2,6 +2,8 @@
 kotest = "6.1.4"
 kotlin = "2.4.20"
 protobuf = "4.34.1"
+commonmark = "0.28.0"
+slf4j = "2.0.17"
 
 # plugin versions
 google-osdetector-plugin = "1.7.3"
@@ -21,6 +23,8 @@ cyclonedx = { id = "org.cyclonedx.bom", version.ref = "cyclonedx" }
 license-report = { id = "com.github.jk1.dependency-license-report", version.ref = "jk1-dependency-license-report" }
 
 [libraries]
+commonmark = { group = "org.commonmark", name = "commonmark", version.ref = "commonmark" }
+slf4j-api = { group = "org.slf4j", name = "slf4j-api", version.ref = "slf4j" }
 protobuf-java = { group = "com.google.protobuf", name = "protobuf-java", version.ref = "protobuf" }
 
 # test dependencies

gradle/license/allowed-licenses.json

@@ -2,6 +2,8 @@
   "allowedLicenses": [
     { "moduleLicense": "Apache License, Version 2.0" },
     { "moduleLicense": "MIT License" },
-    { "moduleLicense": "The 3-Clause BSD License" }
+    { "moduleLicense": "The 2-Clause BSD License" },
+    { "moduleLicense": "The 3-Clause BSD License" },
+    { "moduleName": "org.slf4j:slf4j-api" }
   ]
 }

markdown/README.md

@@ -0,0 +1,37 @@
+# protoc-utils-markdown
+
+CommonMark helpers shared by Engine's protoc plugins for handling proto doc comments.
+
+```kotlin
+implementation("com.engine:protoc-utils-markdown:VERSION")
+```
+
+## Reference links
+
+Doc comments refer to other proto elements with CommonMark reference-link syntax: `[Widget]`, `[acme.v1.Widget]`, `[Widget.name]`, `[Status.STATUS_ACTIVE]`, `[WidgetService.GetWidget]`, or the full form `[display text][Widget]`.
+
+- `ReferenceIndex<T>` indexes every message, field, enum, enum value, service, and RPC in a set of files.
+  A plugin-supplied function maps each `ReferenceElement` to a target `T` (an href, a rendering decision, …); returning `null` leaves the element unindexed.
+  Bare names resolve against the members of the comment's own descriptor before global short names, so a comment on `message Widget` can write `[name]`.
+- `ReferenceLinkProcessor<T>` is a commonmark-java `LinkProcessor` that rewrites resolved references as links (`ReferenceRendering.Link`) or code spans (`ReferenceRendering.Code`).
+  Inline links, images, code spans, and escaped brackets are left alone.
+  Labels in the `overrides` map (parsed from repeated `referenceLink=<label>=<URL>` parameters with `ReferenceLinkProcessor.parseOverrides`) link outside the compile scope, e.g. `google.rpc.Status`.
+- `ReferenceLinkMode` selects `NONE` (no resolution), `WARN` (log failures), or `FAIL_ON_INVALID` (log and collect failures).
+  Collected `ReferenceLinkFailure`s are formatted for `CodeGeneratorResponse.error` with `ReferenceLinkFailure.summary`.
+
+```kotlin
+val index = ReferenceIndex(request.protoFiles) { element -> ReferenceRendering.Link("#${element.fqn}") }
+val processor = ReferenceLinkProcessor(index, ReferenceLinkMode.FAIL_ON_INVALID) { it }
+val parser = Parser.builder().linkProcessor(processor).build()
+
+val document = processor.withScope("acme.v1.Widget") { parser.parse(comment.cleaned) }
+val description = MarkdownText.render(document)
+
+val failures = processor.drainFailures()
+if (failures.isNotEmpty()) response.addError("my-plugin failed:\n" + ReferenceLinkFailure.summary(failures))
+```
+
+## Plain text
+
+`MarkdownText.toPlainText` flattens a parsed comment to one line of plain text for fields that cannot hold Markdown (titles, summaries), and `MarkdownText.firstSentence` cuts it to the first sentence of the first paragraph.
+Parse with the same `ReferenceLinkProcessor` first so bad references in those fields are still reported.

markdown/build.gradle.kts

@@ -0,0 +1,54 @@
+description = "CommonMark helpers for protoc plugins: proto reference-link resolution and plain-text scrubbing of doc comments."
+
+java {
+    withJavadocJar()
+    withSourcesJar()
+}
+
+dependencies {
+    api(rootProject)
+    api(libs.commonmark)
+    implementation(libs.slf4j.api)
+}
+
+publishing {
+    publications {
+        create<MavenPublication>("maven") {
+            from(components["java"])
+            artifact(layout.buildDirectory.file("reports/cyclonedx-direct/bom.json")) {
+                classifier = "cyclonedx"
+                extension = "json"
+                builtBy(tasks.named("cyclonedxDirectBom"))
+            }
+            pom {
+                name.set(project.name)
+                inceptionYear.set("2026")
+                licenses {
+                    license {
+                        name.set("Apache-2.0")
+                        url.set("https://github.com/engine-public/protoc-utils/blob/${version}/LICENSE")
+                    }
+                }
+                developers {
+                    developer {
+                        organizationUrl.set("https://github.com/engine-public")
+                    }
+                }
+                scm {
+                    connection.set("scm:git:https://github.com/engine-public/protoc-utils.git")
+                    developerConnection.set("scm:git:https://github.com/engine-public/protoc-utils.git")
+                    url.set("https://github.com/engine-public/protoc-utils")
+                }
+            }
+        }
+    }
+}
+
+afterEvaluate {
+    publishing.publications.named<MavenPublication>("maven") {
+        pom {
+            description.set(project.description)
+            url.set("https://github.com/engine-public/protoc-utils/blob/${version}/markdown/README.md")
+        }
+    }
+}

markdown/src/main/kotlin/com/engine/protoc/util/markdown/MarkdownText.kt

@@ -0,0 +1,92 @@
+package com.engine.protoc.util.markdown
+
+import org.commonmark.node.AbstractVisitor
+import org.commonmark.node.Code
+import org.commonmark.node.Image
+import org.commonmark.node.Link
+import org.commonmark.node.Node
+import org.commonmark.node.Paragraph
+import org.commonmark.node.Text
+import org.commonmark.renderer.markdown.MarkdownRenderer
+import org.commonmark.renderer.text.TextContentRenderer
+import java.text.BreakIterator
+import java.util.Locale
+
+/**
+ * Conversions of parsed doc-comment Markdown into the strings a plugin emits.
+ */
+public object MarkdownText {
+    private val defaultMarkdownRenderer: MarkdownRenderer = MarkdownRenderer.builder().build()
+    private val textRenderer: TextContentRenderer = TextContentRenderer.builder().stripNewlines(true).build()
+
+    /**
+     * Render [node] back to CommonMark, with trailing whitespace removed from every line and from
+     * the end of the result.  A hard line break written as trailing spaces therefore renders as a
+     * soft break; cleaned doc comments have already lost trailing spaces, so this only normalizes.
+     */
+    public fun render(
+        node: Node,
+        renderer: MarkdownRenderer = defaultMarkdownRenderer,
+    ): String =
+        renderer.render(node)
+            .lines()
+            .joinToString("\n") { it.trimEnd() }
+            .trimEnd()
+
+    /**
+     * Flatten [node] to a single line of plain text for fields that cannot hold Markdown (titles,
+     * summaries).  Links and images become their visible text, code spans lose their backticks,
+     * emphasis markers are dropped, and line breaks become spaces.
+     *
+     * Mutates [node]; pass a freshly parsed document.
+     */
+    public fun toPlainText(node: Node): String {
+        val replacements = mutableListOf<Node>()
+        node.accept(
+            object : AbstractVisitor() {
+                override fun visit(link: Link) {
+                    replacements += link
+                    visitChildren(link)
+                }
+
+                override fun visit(image: Image) {
+                    replacements += image
+                    visitChildren(image)
+                }
+
+                override fun visit(code: Code) {
+                    replacements += code
+                }
+            },
+        )
+        for (n in replacements) {
+            if (n is Code) {
+                n.insertBefore(Text(n.literal))
+            } else {
+                while (true) {
+                    val child = n.firstChild ?: break
+                    n.insertBefore(child)
+                }
+            }
+            n.unlink()
+        }
+        return textRenderer.render(node).trim()
+    }
+
+    /**
+     * The plain text of [node]'s first paragraph, cut to its first sentence.  A document that does
+     * not open with a paragraph (a list, heading, or code block) is flattened whole.  Sentence
+     * boundaries come from [BreakIterator], so `e.g.`, decimals, and URLs don't end a sentence.
+     *
+     * Mutates [node]; pass a freshly parsed document.
+     */
+    public fun firstSentence(node: Node): String {
+        val first = node.firstChild
+        val plain = toPlainText(if (first is Paragraph) first else node)
+        if (plain.isEmpty()) return ""
+        val iterator = BreakIterator.getSentenceInstance(Locale.ROOT)
+        iterator.setText(plain)
+        val end = iterator.next()
+        return if (end == BreakIterator.DONE) plain else plain.substring(0, end).trim()
+    }
+}

markdown/src/main/kotlin/com/engine/protoc/util/markdown/ReferenceElement.kt

@@ -0,0 +1,50 @@
+package com.engine.protoc.util.markdown
+
+import com.engine.protoc.util.GeneratedMessageWrapper
+import com.engine.protoc.util.file.FileDescriptorProtoWrapper
+
+/**
+ * A proto element that a doc-comment reference link (`[Name]`) can point at.
+ *
+ * Instances are created by [ReferenceIndex] while it walks the compile scope and are handed to the
+ * plugin's target function, which decides where (or whether) the element is addressable in that
+ * plugin's output.
+ */
+public class ReferenceElement internal constructor(
+    /** What kind of proto element this is. */
+    public val kind: Kind,
+    /** The file that declares the element. */
+    public val file: FileDescriptorProtoWrapper,
+    /** The proto package of [file], without a leading dot; empty for the default package. */
+    public val packageName: String,
+    /**
+     * The dotted name of the element relative to its package, e.g. `Outer.Inner`,
+     * `Outer.Inner.field`, `Enum.VALUE`, or `Service.Method`.
+     */
+    public val localName: String,
+    /** The containing element: the message of a field or nested type, the enum of a value, the service of an RPC. */
+    public val parent: ReferenceElement?,
+    /** The descriptor wrapper for the element. */
+    public val descriptor: GeneratedMessageWrapper<*>,
+) {
+    /** The fully-qualified name of the element, without a leading dot. */
+    public val fqn: String = if (packageName.isEmpty()) localName else "$packageName.$localName"
+
+    /** The last segment of [localName]. */
+    public val simpleName: String get() = localName.substringAfterLast('.')
+
+    /** The kinds of element a reference link can resolve to. */
+    public enum class Kind(
+        /** The human-readable label used in log and failure messages. */
+        public val label: String,
+    ) {
+        MESSAGE("message"),
+        ENUM("enum"),
+        SERVICE("service"),
+        FIELD("field"),
+        ENUM_VALUE("enum value"),
+        RPC("rpc"),
+    }
+
+    override fun toString(): String = "${kind.label} $fqn"
+}