A compact, lightweight library for embedding-based document storage and retrieval.
Capsa is a Rust library that implements the retrieval component of RAG (Retrieval-Augmented Generation) systems. It provides a simple API for ingesting documents, generating embeddings, storing them in a vector database, and performing semantic search through natural language queries.
The repository also includes a fully-functional CLI tool for document indexing and semantic search.
src/,bin/: Rust storage, indexing pipeline and Capsa CLI.federation/: read-only multi-source MCP gateway and deployment tooling.indexers/: source-specific indexing scripts, shared helpers, fixture tests and provenance collected from the existing deployment work.
Indexing is a separate, explicitly configured write operation. Do not run indexers against an immutable snapshot or use the read-only federation gateway for ingestion.
Capsa uses a standard vector database approach:
- Document Chunking - Documents are split into 128-token chunks with overlap to preserve context
- Embedding Generation - Each chunk is converted to a vector representation using an embedding model (via OpenAI-compatible API)
- Vector Storage - Embeddings are stored in libSQL (Turso's fork of SQLite with vector indexing) for fast similarity search
- Semantic Query - Queries are embedded and matched against stored vectors using cosine similarity
This allows finding relevant content based on semantic meaning rather than exact keyword matches.
Pipeline writes commit a receipt in the vector database atomically with the document and its embeddings. The receipt key combines the job ID with a random 128-bit identity persisted in the queue database. If queue acknowledgement fails or the process stops between commit and acknowledgement, replay returns the original document ID without inserting another document or embeddings. A failed acknowledgement is retried with a one-second backoff, without waiting for another job; queue and vector connections use a bounded five-second SQLite busy timeout.
Existing databases are upgraded additively using CREATE TABLE IF NOT EXISTS
(capsa_queue_identity in the queue DB, capsa_pipeline_write_receipts in the vector
DB). Existing documents and pending work are preserved; historical duplicates are
not removed or assigned receipts retroactively. Ordinary public vector insert APIs
remain unconditional. Keep the identity and receipt tables when backing up/restoring
these databases: deleting them removes replay protection. Copying a queue database
preserves its identity and represents the same queue lineage, not a new queue.
This protection applies to writes made by the patched pipeline. A document committed by an older binary without a receipt cannot be identified retroactively if its queue acknowledgement is still pending. Upgrade with a drained queue, or reconcile such legacy in-flight jobs before resuming. Independently submitting the same source document again creates a new job; source-level upserts/deduplication remain the ingestion adapter's responsibility.
capsa --db-path /index/capsa/confluence.db serve --immutableUse the same embedding model/API used to create the snapshot (override --model
and --base-url as needed). --immutable conflicts with --rw. It opens SQLite
with READ_ONLY and an internally percent-encoded file: URI with immutable=1,
does not open an ingestion queue or pipeline, and never initializes WAL or
creates/migrates schema or metadata. Startup validates stored dimension/model
metadata and the existing vector tables/index against the embedding API probe.
Missing or mismatched metadata/schema fails closed; prepare older databases with
a writer before taking the snapshot.
Stop all writers and checkpoint successfully before copying/moving the main
database file. A nonempty -wal sibling is rejected because immutable mode
ignores WAL and could otherwise serve an incomplete snapshot. The source must
also be checkpointed: an absent WAL at the destination cannot prove that a
main-file-only copy is complete. The path must be a normal absolute filename,
not a caller-supplied SQLite URI. The file must not change for the entire server
process lifetime: stop the server before replacing/updating it. Immutable mode
deliberately bypasses SQLite locking/change detection, making a fixed snapshot
suitable for read-only NFS serving. No automatic copying or deployment occurs.
Compatibility defaults are unchanged: serve without --immutable exposes only
read MCP tools, but still opens a writable database, queue and pipeline; serve --rw additionally exposes ingestion. The library equivalent for a fixed snapshot
is DocumentDatabase::new_immutable; new_reader remains query-only but
initializes writable schema/WAL for compatibility.
Add Capsa to your Cargo.toml:
[dependencies]
capsa = "0.1"use capsa::{config::Config, documentdb::DocumentDatabase};
use serde_json::json;
use secrecy::SecretString;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Configure the embedding service and database
let api_key = std::env::var("CAPSA_API_KEY").ok().map(SecretString::from);
let config = Config::new(
"http://localhost:9000/v1".to_string(),
"nomic-ai/nomic-embed-text-v1.5".to_string(),
"./documents.db".to_string(),
api_key,
);
// Connect to the database
let db = DocumentDatabase::new(&config).await?;
let conn = db.connect().await?;
// Index a document
let metadata = json!({
"title": "My Document",
"author": "Author Name"
});
let doc_id = conn.insert(metadata, "Your document text here").await?;
println!("Indexed document: {}", doc_id);
// Search
let results = conn.search_topk("your query", 5).await?;
for (doc_id, metadata, start, end) in results {
println!("Found in doc {}: chars {}-{}", doc_id, start, end);
}
Ok(())
}git clone https://github.com/glguida/capsa
cd capsa
cargo build --release
# Optionally install to ~/.cargo/bin
cargo install --path .Capsa requires an embedding service with an OpenAI-compatible API. You have several options:
Option 1: llama.cpp
llama-server -m /path/to/nomic-embed-text-v1.5.Q4_K_M.gguf --embeddings --port 9000Option 2: text-embeddings-inference
For GPU/CUDA support:
docker run -p 9000:80 ghcr.io/huggingface/text-embeddings-inference:latest \
--model-id nomic-ai/nomic-embed-text-v1.5For CPU only support:
docker run -p 9000:80 ghcr.io/huggingface/text-embeddings-inference:cpu-latest \
--model-id nomic-ai/nomic-embed-text-v1.5Option 3: Any OpenAI-compatible API (remote or local)
Index documents:
capsa pdf paper.pdf
capsa yt dQw4w9WgXcQ
capsa yt --lang es VIDEO_IDQuery:
capsa ask "your question here"
capsa ask -d -k 20 "detailed query"Add a PDF document:
$ capsa pdf attention-is-all-you-need.pdf
================================================================================
PDF DOCUMENT INGESTION SYSTEM
================================================================================
FILE......: attention-is-all-you-need.pdf
EXTRACTING TEXT...
EXTRACTION COMPLETE
TEXT SIZE.: 33110 CHARACTERS
TITLE.....: Attention is All you Need
AUTHOR....: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, �ukasz Kaiser, Illia Polosukhin
INITIALIZING DATABASE CONNECTION... DONE
PROCESSING... COMPLETE
================================================================================
INGESTION COMPLETE - DOCID=000001
================================================================================
$Add a YouTube video transcript:
$ capsa yt dQw4w9WgXcQ
================================================================================
YOUTUBE TRANSCRIPT INGESTION SYSTEM
================================================================================
INPUT.....: dQw4w9WgXcQ
LANGUAGE..: en
EXTRACTING VIDEO ID...
VIDEO ID..: dQw4w9WgXcQ
FETCHING VIDEO DETAILS...
TITLE.....: Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)
AUTHOR....: Rick Astley
FETCHING TRANSCRIPT...
TRANSCRIPT FETCHED
TEXT SIZE.: 2335 CHARACTERS
LANGUAGE..: English
INITIALIZING DATABASE CONNECTION... DONE
PROCESSING... COMPLETE
================================================================================
INGESTION COMPLETE - DOCID=000002
================================================================================
$Simple query:
$ capsa ask -d -k 1 "What is the transformer architecture?"
================================================================================
DOCUMENT RETRIEVAL SYSTEM
================================================================================
QUERY.....: What is the transformer architecture?
TOP-K.....: 1
INITIALIZING DATABASE CONNECTION... DONE
================================================================================
RECORD 001 DOCID=000001 SIMILARITY= 76.70%
================================================================================
TITLE..: Attention is All you Need
AUTHOR.: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, �ukasz Kaiser, Illia Polosukhin
SUBJECT: Neural Information Processing Systems http://nips.cc/
FILE...: attention-is-all-you-need.pdf
OFFSET.: 4080-4478 (398 BYTES)
--------------------------------------------------------------------------------
CONTENT:
--------------------------------------------------------------------------------
In this work we propose the Transformer, a model architecture eschewing recurrence and instead
relying entirely on an attention mechanism to draw global dependencies between input and output.
The Transformer allows for significantly more parallelization and can reach a new state of the art in
translation quality after being trained for as little as twelve hours on eight P100 GPUs.
2 Background
--------------------------------------------------------------------------------
$
Another query, on the same database:
$ capsa ask -d -k 1 "Will you disappoint me?"
================================================================================
DOCUMENT RETRIEVAL SYSTEM
================================================================================
QUERY.....: Will you disappoint me?
TOP-K.....: 1
INITIALIZING DATABASE CONNECTION... DONE
================================================================================
RECORD 001 DOCID=000002 SIMILARITY= 54.33%
================================================================================
TITLE..: Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)
AUTHOR.: Rick Astley
OFFSET.: 511-974 (463 BYTES)
--------------------------------------------------------------------------------
CONTENT:
--------------------------------------------------------------------------------
for so long ♪ ♪ Your heart's been aching
but you're too shy to say it ♪ ♪ Inside we both know
what's been going ♪ ♪ We know the game
and we're gonna play it ♪ ♪ And if you ask me
how I'm feeling ♪ ♪ Don't tell me
you're too blind to see ♪ ♪ Never gonna give you up ♪ ♪ Never gonna let you down ♪ ♪ Never gonna run around
and desert you ♪ ♪ Never gonna make you cry ♪ ♪ Never gonna say goodbye ♪ ♪ Never gonna tell a lie
--------------------------------------------------------------------------------
$
Output with -d shows cosine similarity percentages, helping you gauge result relevance.
Available for all commands:
--base-url <url>- Embedding service URL (default:http://localhost:9000/v1)--model <name>- Model name (default:nomic-ai/nomic-embed-text-v1.5)--db-path <path>- Database path (default:./documents.db)
CAPSA_API_KEY- API key for embedding service (optional)
capsa pdf <path>Extracts PDF metadata and text, generates embeddings, and stores them in the vector database.
capsa yt [--lang <code>] <id_or_url>Downloads YouTube transcript with metadata and indexes it for semantic search.
Options:
--lang <code>- Language code (default:en)
Accepts: Video ID or full YouTube URL
capsa ask [-d] [-k <num>] "query"Query your document database using natural language.
Options:
-d- Show similarity percentages for each result-k <num>- Number of results to return (default:5)
MIT