Metacarp

Metacarp is a self-hosting compiler for Carp, written in Carp. It reads Carp source and emits a C translation unit, driving the whole pipeline — module loading, macro expansion, name resolution, type inference, interface specialization, ownership and borrow checking, and C code generation — in Carp itself.

Status: self-hosting. The compiler compiles itself, and the result compiles the compiler again to byte-identical C — a fixed point. Both generations pass the reference compiler's test suite (154 entries: examples, output tests, error-rejection tests, benches). It is still not a drop-in replacement for the reference compiler (see Limitations).

Build

Requires the reference Carp compiler and a C compiler (clang). Building bootstraps the roughly 42,000 lines of compiler source through reference Carp:

carp -b --optimize main.carp

This produces ./out/carp-compiler.

metacarp is the same compiler plus the LLVM backend and the terminal REPL. It requires LLVM 21 or later with development headers and a linkable libLLVM. The session JIT also needs Clang from the same installation. Linux CI uses LLVM and Clang 22. Build it with the compiler above:

./out/carp-compiler -c "$CARP_DIR/core" --optimize -b -o out/metacarp main-metacarp.carp

The LLVM bindings use brew --prefix llvm on macOS and llvm-config --prefix elsewhere. Embedders can pass an explicit installation prefix to LLVM.setup.

Usage

carp-compiler [options] <source.carp>

  -b, --build           compile to an executable instead of C (needs Core)
  -x, --execute         compile to an executable and run it (needs Core)
  -c, --core <dir>      compile against the Carp standard library in <dir>
                        (default: $CARP_DIR/core)
  -o, --output <file>   output path — the C file, or the executable under -b
                        (default under -b: <output-directory>/<title>
                        from Project.config, out/Untitled unless set)
  --emit c|llvm         the backend (llvm only in metacarp); llvm without
                        -b/-x writes textual IR
  --optimize            build -b/-x executables with clang -O3 -D NDEBUG
  --log-memory          build -b/-x executables with LOG_MEMORY, so
                        Debug.memory-balance counts allocations
  -g, --debug           build with debug info, keeping the C beside the
                        binary so a debugger can read what it runs
  --annotate            emit inferred types of global definitions as JSON
  --ownership           emit the move/borrow/delete plan as JSON
  --library             emit a C translation unit without main (needs Core)
  --no-core             skip the implicit Core load; the source's own
                        (load "X.carp") directives still resolve in Core
  --package-override <git-reference>=<source.carp>
                        use a local package source for this invocation
                        CARP_PACKAGE_OVERRIDES supplies newline-separated defaults
  -h, --help            show this help and exit
  -v, --version         show version and exit

With no -b/-x, the C translation unit is written to standard output (or -o). Diagnostics go to standard error and name the rejecting compiler phase.

Core is found like the reference compiler finds it: --core <dir> when given, else $CARP_DIR/core. One of them is required unless --no-core is given. -b/-x need Core even then, because linking needs the runtime headers the standard library ships with.

Project.config is recorded at compile time and Project.get-config reads it back, with the reference's keys and defaults. The driver acts on title and output-directory (-b writes <output-directory>/<title>, out/Untitled by default, unless -o is given; -x builds and runs the same path) and on cflag, libflag and pkgconfigflag, which reach clang along with what add-cflag, add-lib and add-pkg add. The other keys (echo-c, target, the docs-* keys, prompt, cmod and the rest) are accepted and read back but change nothing here.

--library emits an entry-point-free translation unit for a shared or static library, rooted at the public definitions in the input source. A public definition that cannot be compiled on its own (a generic function) is an error; a private helper is reached through the roots. There is no main, so the unit initializes its globals in a load-time constructor and System.args is empty. A top-level expression that does something has nowhere to run and is an error.

Only the roots are exported: every other function and global the unit defines is static, and hidden besides, so two Carp libraries in one host keep their own code and data. Core's C headers, and any C a library includes or wraps, still define functions of their own; build a shared library with -fvisibility=hidden so those stay inside it. Static archives have no such switch, and linking two of them still clashes on what those headers define.

(load ...) resolves like the reference compiler's: the name as given (relative to the working directory), then relative to the loading file, then the Core directory, so a local file shadows the Core file it is named after. Git references install into the shared cache (~/.cache/carp/libs/...) on first use:

(load "git@github.com:carpentry-org/strbuf@0.3.0")

The examples below assume CARP_DIR points at a checkout of the reference Carp repository (its core/ is the standard library and runtime headers).

Compile a standalone program (no standard library) to C:

./out/carp-compiler --no-core examples/hello.carp \
  | clang -x c -I "$CARP_DIR/core" -o /tmp/carp-hello -
/tmp/carp-hello        # prints: OK

Compile and run a program that uses the standard library:

./out/carp-compiler -x -c "$CARP_DIR/core" examples/squares.carp
# sum of squares of the even numbers in 1..10 = 220

LLVM backend

metacarp --emit llvm shares the complete compiler front end with the C driver. It writes LLVM IR by default, or emits and links a native object under -b and -x:

./out/metacarp --emit llvm -c "$CARP_DIR/core" -o /tmp/squares.ll examples/squares.carp
./out/metacarp --emit llvm -x -c "$CARP_DIR/core" examples/squares.carp

Pass -g or --debug to preserve source texts and attach DWARF locations from the backend-neutral line map. See carp-llvm-backend for supported lowering, native linking, the persistent ORC session JIT, and its test commands.

Terminal REPL

metacarp combines the compiler and terminal REPL in one executable. With no arguments it opens the REPL; metacarp repl accepts REPL options. File arguments use the C compiler, for example metacarp -x program.carp. metacarp --help describes both modes, and metacarp repl --help lists REPL commands and keys.

The terminal REPL uses the warm session and LLVM JIT on macOS and Linux. It provides persistent definitions and typed results, mixed-input transactions, editable source history, completion, docs, highlighting, ownership and generated-code queries, and array/record inspection. Build metacarp as under Build. The reader editing API is included in the pinned carp-reader@0.5.0 dependency:

./out/metacarp
# or: ./out/metacarp repl --core "$CARP_DIR/core" --no-history

The REPL also needs libedit (part of macOS; libedit-dev on Debian and Ubuntu).

The umbrella RFC records the editor, introspection, editable-history, and inspection milestones. The Carp editor uses system libedit for editing and persistent recall, with bounded language feedback. Unrelated runtime additions preserve globals; replacements and compile-time declarations release results and reset native state.

Self-hosting

The bootstrap chain, from the repository root:

carp -b --optimize main.carp                                    # gen 1
./out/carp-compiler -c "$CARP_DIR/core" -o self.c main.carp     # gen 1 emits itself
clang -O3 -D NDEBUG -o self-cc self.c -I "$CARP_DIR/core"      # link gen 2
./self-cc -c "$CARP_DIR/core" -o self2.c main.carp              # gen 2 emits itself
cmp self.c self2.c                                              # fixed point

The canonical generation benchmark compares reference Carp, gen 1, and gen 2 on the same workload (generating C for main.carp):

CARP_BENCH_RUNS=3 ./bench/compiler-generations.sh

It reports wall time, user time, and maximum resident set size, writes the raw measurements as TSV, and checks that the C emitted by gen 1 and gen 2 reaches a fixed point. Linking each next-generation compiler is deliberately excluded from the timed region.

bench/nbody-codegen.sh checks exact output parity with reference Carp and requires the generated hot loop to materialize its stable array data pointer. It also reports reference/generated executable runtime without imposing a noise-sensitive CI timing threshold.

The assurance harness keeps the self-host honest. It builds everything with this compiler: gen 1 comes from a seed, a compiler an earlier build of this repository produced (CARP_SEED_COMPILER, else the out/carp-compiler already there), and only without a usable seed is gen 1 built through the reference, once. The reference otherwise runs only where it is the thing compared against: the expansion diff and the generation benchmark.

• scripts/run-assurance.sh bootstrap only builds gen 1.

• scripts/run-assurance.sh phase runs lint and formatting, then every phase/session suite through gen 2. Set CARP_SKIP_STYLE=1 when the two style tools are unavailable. Set CARP_PHASE_JOBS (1 to 4) to run independent test groups concurrently; CI runs this architecture-neutral group once, on every core of a Linux runner up to 4.

• scripts/run-assurance.sh self runs the reference suite, checks the self-hosted fixed point, and compares expansion behavior. Set CARP_SELF_JOBS=2 or 3 to split the reference suite across isolated workers; CI uses all two Linux or three macOS runner cores. Gen 2 is built as strict C99, must run examples/hello.carp, and is kept (with a provenance file) when CARP_FIXED_POINT_OUT names its directory. With CARP_FIXED_POINT_REUSE=1, a directory that already holds a passed check of the current revision is used as is; CI hands one job's gen 2 to the next this way.

• scripts/run-assurance.sh sanitize runs the reference suite again with every generated program built under AddressSanitizer.

• scripts/run-assurance.sh llvm builds the LLVM driver, runs its backend's tests, and requires each memory fixture to balance under both drivers.

• scripts/run-assurance.sh all runs phase and self and is the default. CI calls the same groups rather than maintaining its own command list. The self-host group executes generated programs on both x86-64 Linux and ARM64 macOS for every commit. CI caches the versioned Carp library checkouts on both architectures, the pinned Ubuntu style-tool binaries, and one seed compiler per platform, the gen 1 of the latest passing main build. A seed only builds gen 1; the fixed point still checks what the current source produces, and test outputs are never cached.

• scripts/run-carp-suite-self.sh runs the reference repository's own test suite (examples, produces-output diffs, test/*.carp, error-rejection tests, bench builds) through this compiler. It finds the reference checkout through CARP_ROOT or CARP_DIR, and CARP_COMPILER can point it at a gen-2 binary.

• scripts/diff-expansion.sh compiles and runs a front-end corpus (macros, quasiquote, gensym, dynamic evaluation) under both the reference compiler and this one and requires identical observable output.

Examples

The programs in examples/ double as checkpoints. The standalone ones need no standard library:

example	shows
simple.carp	functions, if, a registered C primitive
hello.carp	deftemplate inline C; prints OK
nominal.carp	deftype sum type, match with a wildcard field
polymorphic-nominal.carp	one polymorphic type specialized at two instances
nested-pattern.carp	nested constructor patterns
signature-nominal.carp	layout discovery from a signature, no reachable constructor
squares.carp	the full standard library (needs --core)

An intentionally small integration experiment lives in experiments/metacarp-clap: a macOS CLAP audio effect whose interface is a Carp function compiled through the warm session and LLVM JIT. It executes unsandboxed native code inside the plugin host and is not a release artifact.

How it works

Source flows through the major phases below. Each has its own directory, data model, and tests:

source registry
  -> carp-module      module loading, load-order and git-reference resolution
  -> carp-surface     lossless surface parsing
  -> carp-expand      macro expansion (with carp-ct-env / carp-ct-eval)
                      compile-time code is lowered and compiled to bytecode
  -> carp-resolve     name resolution into the core IR (carp-ir)
  -> carp-infer       Hindley–Milner type inference (carp-types)
  -> carp-specialize  interface selection and monomorphization
  -> carp-ownership   ownership planning and flow-sensitive borrow checking
  -> carp-backend     lowering and C emission (carp-c-abi for mangling)
  -> one C translation unit

Two research libraries are experimental and opt-in. They are published in this repository but are not part of the compiler pipeline: carp-ct-types (a tag lattice for the compile-time language) and carp-ct-infer (a shape check over defmacro and defndynamic bodies, invoked through CtCheck.modules). Nothing in Metacarp calls them. They are a checking phase rather than a transforming one, so a build that ignores them loses only their diagnostics. See carp-ct-infer/README.md for what they do and do not cover yet.

Supporting libraries: carp-primitives (the declarative primitive registry) and carp-graph (strongly-connected-component ordering), plus carp-source (caller-owned source identities and byte spans) and carp-session (warm, transactional notebook inference). Each phase reports failures tagged with its own name.

Ownership is real: the compiler derives delete/copy for managed deftypes (concrete and generic), runs a sound flow-sensitive borrow check (escape, move, borrow-after-move, branch joins), and emits the corresponding C cleanup. The delete plan covers let scopes, unconsumed parameters, discarded intermediates, set! overwrites, by-value match bindings (used or not), and the payloads a wildcard pattern drops — with deep deleters and copiers for nested containers. Escaping closures heap-allocate their environments.

carp-frontend owns expansion through inference. Batch compilation, ownership tooling, and library compilation use the same preparation path. Derivation lives in carp-derive; borrowed-temporary lifting lives beside ownership. Read-only capture analysis is shared by ownership closure and backend lifting, so planning does not run backend transformations. See compiler architecture for the phase and ownership boundaries.

The reusable entry point is CarpCompiler.compile-source, or CarpCompiler.compile-sources when the caller already holds an in-memory source registry. Embedders can use CarpCompiler.compile-inferred-library-roots to retain explicit source-level roots while omitting the process entry point.

Session library

carp-session keeps an inferred core resident for notebook and editor clients, then provides transactional definition updates, cell-relative typed reports, ownership queries, completion, and incremental code generation. The design deliberately leaves transport and value hosting to clients such as Lepiter and GT. carp-llvm-backend/carp-session-jit.carp adds a persistent ORC JIT whose modules reuse that semantic session. Failed cells do not replace the last published implementation. docs/carp-session.md records the original design and implementation history.

Limitations

• The host architecture and OS are stamped for the build machine (aarch64/darwin) in the --core path.
• Delete placement is scope-based, not liveness-based: values die at scope or branch exit rather than after their last use, so peak memory can exceed the reference compiler's on the same program.
• Error messages are deliberately this compiler's own; only rejection behavior matches the reference, not diagnostic text. One rejection goes further: returning a reference to a by-value parameter is an error here, while the reference accepts it and the caller reads freed memory.

Dependencies

The default compiler uses two pinned Carpentry packages, loaded as git references — carp-reader by carp-surface, strbuf by carp-backend and carp-c-abi:

(load "git@github.com:carpentry-org/carp-reader@0.5.0")
(load "git@github.com:carpentry-org/strbuf@0.3.0")

carp-session adds one, for the shared expansion snapshot:

(load "git@github.com:carpentry-org/rc@0.3.0")

The optional LLVM backend additionally loads:

(load "git@github.com:carpentry-org/llvm@0.1.0")

License

MIT. See LICENSE.