Skip to content

Publish markdown copies of docs pages plus llms.txt and llms-full.txt - #17

Closed
alechollingsworth-engine wants to merge 1 commit into
engine-public:mainfrom
alechollingsworth-engine:seo/llms-txt-and-markdown-pages
Closed

alechollingsworth-engine wants to merge 1 commit into
engine-public:mainfrom
alechollingsworth-engine:seo/llms-txt-and-markdown-pages

Conversation

@alechollingsworth-engine

Copy link
Copy Markdown

What

Makes the docs easy for LLMs and AI agents to read. After buildSite, a new Gradle task (writeLlmsFiles) writes into build/dist/site:

  • A markdown copy of every page at the same path with .md (/webhooks.html -> /webhooks.md, / -> /index.md). 23 pages, including the generated gRPC reference.
  • llms.txt (llmstxt.org): every page in sidebar order, plus links to the OpenAPI spec and proto zip.
  • llms-full.txt: all pages concatenated in nav order.
  • <link rel="alternate" type="text/markdown"> on each markdown-sourced HTML page, plus a link to llms.txt (head_custom.html).

How

  • Reads the staged Jekyll content (build/staging/site/content), so the markdown matches what Jekyll renders. Lives in buildSrc/src/main/kotlin/LlmsFiles.kt because the github-pages gem runs Jekyll in safe mode (no custom plugins).
  • Strips kramdown-only syntax (TOC blocks, {: ...} attribute lists, markdownlint directives), drops unused reference-link definitions from the shared footer, and makes site-relative links absolute.
  • URLs are absolute against https://engine-public.github.io/engine-partner-api (override with JEKYLL_SITE_URL). omni-docs-reverse-proxy already rewrites text/* bodies, so they come out as https://omni.engine.com/docs/... in production.
  • llms.txt header copy lives in src/main/markdown/templates/_llms_header.md (underscore = not staged as a page) so the team can edit it.
  • No changes to the existing HTML content or nav.

Tested

  • ./gradlew buildSite (ENGINE_BUILD_VERSION=2.5.0, JEKYLL_BASE_URL=/engine-partner-api) in Docker: 23 .md files, llms.txt, llms-full.txt generated; every rel="alternate" link resolves to a generated file; no leftover kramdown syntax.
  • Not run locally: the full ./gradlew build (needs the Corretto 21 toolchain); CI covers it.

Why

Part of making the Omni docs at omni.engine.com/docs readable by AI assistants and agents (Growth/AEO). Follow-ups planned separately: per-page meta descriptions, intros on the section landing pages, readable gRPC page titles.

I don't have write access to this repo yet (IT ticket pending), so this comes from a fork. Would appreciate a review from whoever maintains the docs build.

🤖 Generated with Claude Code

Adds a writeLlmsFiles task that runs after buildSite and writes, from the
staged Jekyll content:
- a plain markdown copy of each page at the same path with .md
  (/webhooks.html -> /webhooks.md, / -> /index.md)
- llms.txt (https://llmstxt.org) indexing every page in nav order plus
  the OpenAPI spec and proto downloads
- llms-full.txt with every page concatenated

Kramdown-only syntax (TOC blocks, attribute lists, lint directives) is
stripped, unused reference link definitions are dropped, and site-relative
links are made absolute. Each markdown-sourced HTML page gets a
<link rel="alternate" type="text/markdown"> to its .md copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alechollingsworth-engine
alechollingsworth-engine requested a review from a team as a code owner October 6, 2026 16:57
@alechollingsworth-engine

Copy link
Copy Markdown
Author

Closing: changes should be proposed in the staging repo instead. Will reopen there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant