Merge pull request #1 from lingodotdev/dev

feat: turbopack support

Author: AleksandrSl

cmp/.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  pull_request:
+    branches: [main, dev]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5.0.1
+        # pnpm version is taken from package.json
+      - name: Install pnpm
+        uses: pnpm/action-setup@v4
+      - name: Use Node.js 20
+        uses: actions/setup-node@v6.0.0
+        with:
+          node-version: 20
+          cache: "pnpm"
+      - name: Install dependencies
+        run: pnpm install
+      - name: Lint
+        run: pnpm lint:check
+      - name: Check formatting
+        run: pnpm format:check
+
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5.0.1
+        # pnpm version is taken from package.json
+      - name: Install pnpm
+        uses: pnpm/action-setup@v4
+      - name: Use Node.js 20
+        uses: actions/setup-node@v6.0.0
+        with:
+          node-version: 20
+          cache: "pnpm"
+      - name: Install dependencies
+        run: pnpm install
+      - name: Build
+        run: pnpm build
+      - name: Test
+        run: pnpm test

cmp/.husky/pre-commit

@@ -0,0 +1 @@
+./node_modules/.bin/nano-staged

cmp/.prettierignore

@@ -0,0 +1,9 @@
+pnpm-lock.yaml
+.changeset/
+packages/cli/demo/
+build/
+dist/
+.react-router/
+.turbo/
+.next/
+CLAUDE.md

cmp/.prettierrc

@@ -0,0 +1,8 @@
+{
+  "trailingComma": "all",
+  "singleQuote": false,
+  "semi": true,
+  "printWidth": 80,
+  "tabWidth": 2,
+  "useTabs": false
+}

cmp/README.md

@@ -1 +1,10 @@
-# LingoDev Compiler
+# Lingo.dev compiler
+
+### Development
+
+`pnpm install` from project root
+`pnpm turbo dev --filter=@lingo.dev/compile` to compile and watch for compiler changes
+
+Choose the demo you want to work with and run it from the corresponding folder.
+`tsdown` in compiler is configured to cleanup the output folder before compilation, which works fine with next, but vite
+seems to be dead every time and has to be restarted.

cmp/compiler/README.md

@@ -0,0 +1,112 @@
+# @lingo.dev/compiler
+
+Official Lingo.dev compiler with automatic translation support for React applications.
+
+This package provides plugins for multiple bundlers (Vite, Webpack, Rollup, esbuild) and a Next.js loader that
+automatically transforms React components to inject translation calls. It uses a hash-based metadata system to track
+translatable text across your application.
+
+## Features
+
+- **Automatic JSX text transformation** - Automatically detects and transforms translatable text in JSX
+- **Hash-based metadata** - Generates unique hashes for each translatable text based on content, component name, and
+  file path
+- **Opt-in or automatic** - Configure whether to require `'use i18n'` directive or transform all files
+- **Multi-bundler support** - Works with Vite, Webpack, Rollup, esbuild, and Next.js
+- **Built on unplugin** - Unified plugin API across all bundlers
+- **Metadata tracking** - Maintains `.lingo/metadata.json` with all translatable content
+- **Translation server** - On-demand translation generation during development
+- **AI-powered translations** - Support for multiple LLM providers and Lingo.dev Engine
+
+## Installation
+
+```bash
+npm install @lingo.dev/compiler
+# or
+pnpm add @lingo.dev/compiler
+# or
+yarn add @lingo.dev/compiler
+```
+
+## Structure
+
+The compiler is organized into several key modules:
+
+### Core Directories
+
+#### `src/plugin/` - Build-time transformation
+
+- **`transform/`** - Babel AST transformation logic for JSX text extraction
+- **`unplugin.ts`** - Universal plugin implementation (Vite, Webpack)
+- **`next.ts`** - Next.js-specific plugin with Turbopack and Webpack support
+- **`build-translator.ts`** - Batch translation generation at build time
+- **`virtual-modules-code-generator.ts`** - Generates code for virtual modules, dev config and locale resolvers for client and server
+
+#### `src/metadata/` - Translation metadata management
+
+- **`manager.ts`** - CRUD operations for `.lingo/metadata.json`
+- Thread-safe metadata file operations with file locking
+- Manages translation entries with hash-based identifiers
+
+#### `src/translators/` - Translation provider abstraction
+
+- **`lcp/`** - Lingo.dev Engine integration
+- **`pseudotranslator/`** - Development-mode fake translator
+- **`pluralization/`** - Automatic ICU MessageFormat detection
+- **`translator-factory.ts`** - Provider selection and initialization
+
+#### `src/translation-server/` - Development server
+
+- **`translation-server.ts`** - HTTP server for on-demand translations
+- **`cli.ts`** - Standalone CLI for translation generation
+- WebSocket support for real-time dev widget updates
+- Port management (60000-60099 range)
+
+#### `src/react/` - Runtime translation hooks
+
+- **`client/`** - Client-side Context-based hooks
+- **`server/`** - Server component cache-based hooks (isomorphic)
+- **`server-only/`** - Async server-only API (`getServerTranslations`)
+- **`shared/`** - Shared utilities (RichText rendering, Context)
+- **`next/`** - Next.js-specific middleware and locale switcher
+
+#### `src/utils/` - Shared utilities
+
+- **`hash.ts`** - Stable hash generation for translation keys
+- **`config-factory.ts`** - Configuration defaults and merging
+- **`logger.ts`** - Structured logging utilities
+- **`path-helpers.ts`** - File path resolution
+
+#### `src/widget/` - Development widget
+
+- In-browser translation editor overlay for development mode
+
+### Support Directories
+
+#### `tests/` - End-to-end testing
+
+- **`e2e/`** - Playwright tests for full build workflows
+- **`fixtures/`** - Test applications (Vite, Next.js)
+- **`helpers/`** - Test utilities and assertions
+
+#### `benchmarks/` - Performance benchmarks
+
+- Translation speed benchmarks
+- Metadata I/O performance tests
+
+#### `old-docs/` - Legacy documentation
+
+- Historical design documents and notes
+
+### Entry Points
+
+- **`src/index.ts`** - Main package exports (plugins, types)
+- **`src/types.ts`** - Core TypeScript types
+
+## Contributing
+
+This is a beta package under active development. Feedback and contributions are welcome!
+
+## License
+
+ISC

cmp/compiler/Requirements.md

@@ -0,0 +1,50 @@
+## Requirements
+
+1. The core thing is DX. We want compiler to work as seamless as possible.
+2. It should work at least with turbopack (since it's next default bundler), webpack and vite.
+3. HMR should work, whenever we update the code the state of the app should be kept as it owuld be without the compiler.
+4. There should be a way to override the translations
+5. We should be able to translate the attributes, the most obvious ones are alt and aria-label, aria-description and
+   such, but we may also add a way to translate other string attributes too.
+6. We should keep translations calls to the minimum.
+7. We start with react, but it would be nice to apply the same approach to other frameworks. Like vue and astro. So the
+   system should be flexible.
+8. The translations should be possible to commit in the repo, so we can version them. Maybe we can also consider
+   conforming to the existing standard?
+
+## Overall architecture
+
+Given that we still may want translations in the dev mode, but we want to keep them to minimum and do not interfere with
+the development flow, we may consider the following approach:
+
+1. Do not translate the text unless explicitly requested.
+2. Allow using pseudolocalization for even more simple checks. It shows which part of the UI can be translated by
+   compiler, and also using varying width of the text highlights part of the interface not ready for varying languages.
+
+This leads us to the question, how to lazily request the translations in the dev mode.
+Let's work with the SPA apps as an example (since SSR apps may have more options).
+Both currently rendered components and the locale are client side.
+Cache for translations is stored in the filesystem in the project.
+So the way to receive a request from the browser to the host that will get the translations and store the cache.
+
+The most straightforward way to do this is to use the usual web server. App will send a request for translations when needed,
+server components can do the same. This adds a bit of latency but keeps the code uniform.
+
+With this approach we get couple more questions:
+
+- How to start the server
+- How to make the client aware about server's address
+- How to batch the calls, both client and server side.
+
+How to start the server
+While it sounds simple, it's not that simple if you don't want to create any external scripts.
+We have at least three environments:
+
+- Next turbopack: no startup hooks.
+- Next webpack: there is compiler.hooks.watchRun. Well, watchRun runs on every script compilation during hot reload.
+- Vite: buildStart.
+
+So next with turbopack is the most complex case.
+
+It shows which part of the UI can be translated by
+compiler - can be done easier? Add some colors