Repository navigation
Conversation
- Core engine: LoopedLLM with iterative refinement, exit conditions - Evaluators: Threshold, Regex, JSONSchema, Length, Composite - Bayesian priors: BetaPrior, NormalPrior, AdaptivePriors with Welford's algo - Adaptive exit: BayesianExitCondition with cost-aware stopping - Providers: OpenRouter, Ollama, Mock - 28 tests, 2 examples, CI workflow - Metaorcha v1 release
- Add Any type to **kwargs in all provider complete() methods - Add type parameters to dict in MockLLMProvider.calls - Add typing.Any imports where needed
Resolves all mypy --strict CI failures by allowing explicit Any types in evaluator protocols, dict contexts, and **kwargs parameters. - Add strict = true to [tool.mypy] - Add disallow_any_explicit = false - All 28 tests pass, ruff clean
…ence - Add IntentRefiner for Bayesian intent elicitation with adaptive questioning - Add TaskOrchestrator for multi-step task decomposition and execution - Add SQLite-backed persistence (LoopStore, SQLiteBackedPriors) - Add Evaluator and ExitConditionProtocol typing protocols - Add ConsistencyEvaluator and CompletenessEvaluator for task-aware evaluation - Add CLI entry point (loopllm) with refine, run, report, tasks subcommands - 96 tests passing, mypy --strict clean, ruff clean
- FastMCP server (mcp_server.py) exposing 13 tools via stdio transport: Core: refine, run_pipeline, classify_task, analyze_prompt Elicitation: start/answer/finish multi-turn sessions Tasks: plan_tasks, verify_output Observability: report, suggest_config, list_tasks, show_task - CLI mcp-server subcommand with provider/model/db flags - __main__.py for python -m loopllm - .vscode/mcp.json for VS Code Copilot MCP config - pyproject.toml: mcp optional dep, loopllm-mcp entry point - All 96 tests pass, mypy --strict clean, ruff clean
- Remove mock provider and test references from README - Add MCP server setup (VS Code + Cursor) as primary Quick Start - Add CLI and Python library quick starts with real providers - Add full MCP tools table, env vars reference, evaluator/provider tables - Create CONTRIBUTING.md with dev setup, project structure, and guides for adding new providers and evaluators
Phase 1 — MCP Server Enhancements: - Add prompt_history table to SQLite store (schema v2 with migration) - Add _score_prompt_quality() heuristic: 5-dimension scorer (specificity, constraint clarity, context completeness, ambiguity, format spec) - Add _classify_task_type() and _estimate_complexity() heuristics - Add 3 new MCP tools: loopllm_intercept, loopllm_prompt_stats, loopllm_feedback - Add status.json writer for near-real-time VS Code extension updates - Update MCP instructions for strong interception directive - Total MCP tools: 13 → 16 Phase 2 — VS Code Extension (vscode-loopllm/): - Status bar gauge with color coding (green/yellow/red) - Sidebar dashboard with Chart.js learning curve and radar chart - File watcher on ~/.loopllm/status.json for instant updates - sql.js (WASM) for direct SQLite reading with 3s polling - TypeScript strict mode, zero compilation errors Testing: - 128 tests passing (32 new for prompt engineering features) - Tests cover scorer, classifier, complexity estimator, store CRUD, intercept routing, stats aggregation, feedback recording
…history - MCP Sampling: loopllm_refine, run_pipeline, plan_tasks, verify_output now use ctx.sample() for mid-execution LLM calls; falls back to agent_execute if client has no sampling capability - Persistent plans: PlanRegistry backed by SQLite (schema v3); plans survive server restarts via restore_from_store() on startup - New tools: loopllm_plan_list, loopllm_plan_delete, loopllm_gauge, loopllm_context_history, loopllm_context_clear (24 tools total) - AgentPassthroughProvider: all scoring/routing deterministic Python, no LLM - LocalModelLoop: score->rewrite->retry for Ollama models without tool-calling - REST serve: loopllm serve --port 8765 for local model middleware - Tests: 186 passing, schema version bumped to 3 - README: brief onboarding + contributing guide
from __future__ import annotations turns type annotations into strings. FastMCP's inspect.signature(func, eval_str=True) evaluates them in module globals, but Context was only imported inside create_mcp_server(). Add try/except import at module level so the name resolves correctly.
…te/WASM - mcp_server: write prompt history to ~/.loopllm/prompt_history.json on every intercept call (alongside SQLite), for zero-dep extension consumption - vscode extension: replace sql.js (WASM) with direct JSON file read; removes sql.js dependency entirely, no more WASM bundling complexity - dataProvider.ts: rewritten to parse prompt_history.json with Node fs only
…coring
- cli: add 'loopllm score <prompt>' subcommand that writes status.json and
prompt_history.json instantly; no MCP server or agent call needed
- reads from stdin when prompt is '-' or omitted
- --json flag for full dimension output
- extension: auto-score active editor on text selection change (debounced 650ms)
- onDidChangeTextEditorSelection: scores selected text live
- onDidSaveTextDocument: auto-scores .md/.txt/.prompt/.llm files on save
- new command 'loopllm.scoreSelection' (Cmd/Ctrl+Shift+L keybinding)
- uses child_process.execFile('loopllm score') -> status.json -> StatusWatcher
- gauge and dashboard now update from three sources:
1. agent calls loopllm_intercept (original MCP path)
2. user selects text in any editor (new)
3. user saves a prompt file (new)
Three-layer instruction stack that ensures loopllm_intercept is called before every agent response, without any user action: 1. .github/copilot-instructions.md — injected into every Copilot chat session automatically when the repo is open; instructs the model to call loopllm_intercept first on every message and follow the route 2. .vscode/settings.json — github.copilot.chat.codeGeneration.instructions setting fires the same instruction for code generation and review prompts 3. mcp_server.py FastMCP instructions field — already present; reinforces the instruction at the MCP server registration level Together these make gauge + prompt history update passively on every prompt without manual selection, saves, or @mentions.
…chema v4) - store.py: bump SCHEMA_VERSION to 4, add learned_weights table with upsert-on-id=1 row; add save/load/get_meta methods - mcp_server.py: _score_prompt_quality() loads learned weights from store, caches raw dimension values in _last_prompt_dims global - mcp_server.py: new _update_scoring_weights() runs one online SGD step after each feedback rating; clips to [0.05, 0.50] + simplex projects - mcp_server.py: _tool_feedback() calls SGD step, appends weight_update to response payload - mcp_server.py: _next_static_question() replaced with Thompson Sampling — Beta(alpha, beta) draw per candidate question type, argmax selected - tests/test_store.py: bump expected schema version 3 → 4 - README: surface loopllm_run_pipeline as primary tool, add ML spec section (SGD formula, Thompson Sampling, Beta-Binomial, Welford, rolling confidence)
… for containers - icon.svg: replace clock icon with 3 overlapping normal distribution bell curves at different peaks/heights — matches prompt quality distribution theme - statusWatcher.ts: always run a 1s polling fallback in addition to fs.watch; fs.watch (inotify) silently fails inside Linux devcontainers — the interval ensures status.json changes are picked up regardless - rebuilt and reinstalled loopllm-prompt-gauge-0.1.0.vsix
- promptLabProvider.ts: new WebviewViewProvider with textarea, real-time per-dimension bars, grade badge, gauge fill, issues + suggestions tags, Copy and 'Send to Chat' buttons - Scores via 'loopllm score --json' subprocess (direct stdout, no file round-trip) with 350ms debounce as the developer types - 'Send to Chat' copies prompt to clipboard and opens Copilot chat panel - package.json: registers loopllm.promptLab as the first (top) view in the Loop LLM activity bar container; dashboard renamed to 'History' - extension.ts: instantiates and registers PromptLabProvider
…rrules) Auto-injected as system context on repo open by each agent: - AGENTS.md: OpenAI Codex agents, Cursor, and any AGENTS.md-aware tool - CLAUDE.md: Claude Code — includes key implementation details and code-level notes on SGD, Thompson Sampling, schema v4 - .cursorrules: Cursor-specific rules format - .github/copilot-instructions.md: rewritten as full onboarding brief (was minimal, now includes tool table, routing table, learning summary) - .vscode/settings.json: add file reference to copilot-instructions.md for all Copilot instruction hook points (codeGen, review, testGen) Effect: any developer who clones this repo and opens it in Copilot, Cursor, or Claude Code gets a full briefing injected as system context automatically — no extra steps required.
Two bugs causing 'loopllm not found' in the Prompt Lab:
1. VS Code child processes spawn with minimal PATH that excludes the
Python env bin dir (/home/codespace/.python/current/bin). Fixed by
augmenting PATH in execFile env option for both triggerScore and
PromptLabProvider._runScore, with absolute-path fallback.
2. --db flag was placed after the subcommand ('score --db ...') but it
is a global flag on the parent parser — must come before the
subcommand ('--db ... score'). Fixed in both promptLabProvider.ts
and extension.ts.
- ASCII panels for Prompt Lab (live gauge, 5 dimension bars, issues/suggestions) and History (learning curve, grade distribution, SGD learned weights) - Extra PyPI + VS Code extension badges
- Title, banners, badges, and description paragraphs updated to PromptLoop - Typing SVG lines updated to reflect Bayesian/Thompson branding - PyPI badge display updated to promptloop - CLI tool names (loopllm, loopllm_*), pip package, and GitHub repo URL unchanged - Screenshot placeholder added below extension panels - Note added clarifying loopllm = underlying API, PromptLoop = project brand
…mages to README - Remove CONTRIBUTING.md (duplicate of README contributing section) - Remove docs/theises.md (untracked design notes) - .gitignore: whitelist .vscode/settings.json alongside mcp.json, add docs/ - Track .vscode/settings.json (Copilot instructions hooks) - Add img/ screenshots (Prompt Lab + History panels) - README: replace ASCII art placeholders with actual screenshots
- Delete AGENTS.md, CLAUDE.md, .cursorrules (scatter-shot, all diverged) - Create .agent/system.md — single canonical PromptLoop system prompt with full tool table, routing table, repo structure, implementation reference, scoring dimensions, and ML/learning summary - copilot-instructions.md → thin 20-line pointer to .agent/system.md - .gitignore: add *.vsix, vscode-loopllm/out/, _scratch/* with .gitkeep - Add _scratch/ as holder dir for templates/junk (contents gitignored) - Remove dead Dune/_opam entries from .gitignore (not an OCaml project)
- tests/test_plan_registry.py: remove unused variable tid0 (F841) - src/loopllm/cli.py: remove unused import os, _estimate_complexity (F401) - src/loopllm/mcp_server.py: remove unused AgentExecutionRequired, Context (F401) - src/loopllm/providers/agent.py: remove unused LLMUsage (F401) - src/loopllm/serve.py: remove unused _classify_task_type, _estimate_complexity, LengthEvaluator (F401) ruff check passes; 186 tests pass, 4 skipped.
store.py:
- Add cast to typing import
- cast(dict[str, float], json.loads(...)) for load_learned_weights return
- cast(dict[str, Any], json.loads(...)) for load_plan return
local_loop.py:
- Add cast to typing import
- str(...) wrap on _chat return (line 211)
- cast(dict[str, Any], resp.json()) for _score return (line 241)
mcp_server.py:
- Remove unused type: ignore[union-attr] (load_learned_weights guard)
- questions: list[dict[str, Any]] = [] type annotation (line 968)
- Rename refiner result to cq_list to fix type conflict (line 983)
- ctx: Context[Any, Any, Any] | None = None on 4 tool functions
pyproject.toml:
- Add [[tool.mypy.overrides]] ignore_errors for loopllm.serve
(FastAPI is optional dep; decorators cause untyped-decorator errors)
mypy --strict: 0 errors. 186 tests pass.
CI was installing .[dev] which excluded mcp>=1.0.0, causing: - import-not-found for mcp.server.fastmcp - cascading untyped-decorator errors on serve.py FastAPI routes Fix: add mcp>=1.0.0 to [dev] optional-dep group so the type stubs are available during mypy --strict in CI. The type: ignore[assignment,misc] on the ImportError fallback (Context = Any) is still valid and needed when mypy evaluates the except branch even with mcp installed.
Adds AgentLoopController + 4 MCP tools (loopllm_loop_start/step/end/status), a reproducible adaptive-vs-fixed benchmark, README dedup/accuracy, packaging metadata, governance files, Cursor config, and launch drafts. 204 tests, CI green on 3.11-3.13.
Narrow the blanket docs/ gitignore to docs/_build/ and commit the demo walkthrough + launch drafts the README links to.
Add agent_loop.svg terminal cast embedded in README, refresh launch checklist and twitter-thread visual note, sync .agent/system.md to 28 tools / 204 tests, and update PLAN.md milestone status.
Agents submit step artifacts; the MCP server scores via deterministic evaluators and a separate critic call, then min(A,B) drives stop/continue and Bayesian depth learning. Adds guard stack, async loop_step, and CDV tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove PLAN.md, .agent/, and non-public launch drafts. Keep show-hn.md updated for CDV. Fix copilot-instructions, CONTRIBUTING, and gitignore. Co-authored-by: Cursor <cursoragent@cursor.com>
Add What is PromptLoop and System at a glance, reorder sections for CDV clarity, collapse learning math into details, update counts to v0.7.0/219. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
AdaptiveStopper should_continue/route hook for LangGraph/CrewAI/AutoGen with local Channel-A verification; example + tests; PEP 668 venv/pipx install notes in README and show-hn.md; test count 226. CI green on 3.11-3.13.
Stop ignoring public repo paths (docs/, examples/, img/). Ignore local Cursor config except mcp.json, env files, and generated docs/_build/. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace developer-first Quickstart with pipx install, loopllm score, MCP config for Cursor/VS Code, and optional serve/ollama paths. Move git clone flow to Develop from source; dedupe Install section. Co-authored-by: Cursor <cursoragent@cursor.com>
Schema v5 (episodes + active_runs), keyword recall, agent-loop recovery contract (snapshot/restore/hydrate + loopllm_loop_resume), recall injection, and sampling transparency. 31 MCP tools. Not tagged or published.
Correct passed/threshold mismatch, conservative Channel B fallbacks, and ctx.sample() graceful degradation; add MCP integration test, audit docs, demo scripts, Loop Monitor extension, and PyPI publish workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
…eanup (#9) * feat(v0.9): DAG virtual sub-agents Salvage the stranded dag_scheduler.py (compile/ready/submit/merge, CDV-scored per node, cycle detection, active-run recovery) onto current main's shipped episodic memory, and register the 5 new loopllm_dag_* MCP tools (31 -> 36). loopllm_intercept's decompose route now points at loopllm_dag_compile instead of the plain synchronous plan_tasks pipeline. * feat(v0.10): per-project scoping, DAG board, audit trail, and repo cleanup Product (v0.10.0): - Per-project state scoping (project_scope.py): key ~/.loopllm/ state by git-remote/repo-root hash instead of one shared store.db, with LOOPLLM_PROJECT as an explicit override. New `loopllm paths` command. - DAG dependency board in the VS Code Loop Monitor, rendering dag runs as a kanban-style board with per-node CDV score and plain-language failure reasons. Fixed a bug where the active-runs mirror never populated the directory the extension actually reads from. - Verification audit trail: episodes are now stamped with the git commit that was HEAD when recorded (schema v6). New `loopllm audit --since <ref>` report and a board "Export audit" button. - Non-consultation signal: an undismissable Loop Monitor banner when the user has been editing but no entry-point tool has been called this session. - One-command MCP install (`loopllm install-mcp --ide ...`) merging into Cursor/VS Code/Antigravity/Claude Code configs instead of hand-edited JSON. - README now leads with the one-sentence pitch and a real DAG board demo GIF. Repo cleanup: - Trim the public tree to the loopllm package, the vscode-loopllm extension, tests, CI, and IDE-adoption files. Untrack examples/, benchmarks/, AUDIT.md, _scratch/, and scripts/audit_cdv_loop.py going forward (kept locally and in git history). README-embedded images moved from a standalone img/ gallery into .github/assets/. * fix: untrack test_benchmark.py, which imports the now-hidden benchmarks/ script CI checks out a fresh tree without benchmarks/adaptive_vs_fixed.py, so tests/test_benchmark.py's sys.path import failed collection. Same fix already-open PR #7 applied when it removed benchmarks/. * fix(v0.10): legacy store migration path + release metadata - New `loopllm migrate-legacy` command: imports the pre-v0.10 global ~/.loopllm/store.db into the current project's scoped store via a WAL-safe sqlite3 backup; refuses to overwrite an existing project store without --force; leaves the legacy file in place. Without this, per-project scoping silently orphans every existing user's learned priors and episodes. - `loopllm paths` reports a `legacy_store` field; the MCP server logs a migration hint on startup when a legacy store exists and the project store doesn't yet. - Version 0.8.0 -> 0.10.0 (pyproject + __init__); CHANGELOG sections date-stamped; README status line and stale branch notes refreshed; recall FTS5 seam notes moved from v0.10 to v0.11. - Tests: tests/test_cli_migrate_legacy.py (9), incl. a genuine schema-v5 legacy store migrating to v6 on import while the source stays untouched. * chore: sync extension package-lock version with package.json (0.3.0) * docs: fix README tool count and schema version drift Tool table was missing loopllm_loop_resume and the 5 loopllm_dag_* tools (32 -> 36, matching what test_server_registers_expected_tool_count already asserts and what the v0.9 DAG work registered). Also fixed a stale "schema v5" reference to store.py; the store has been on schema v6 (commit_sha column) since the v0.10 audit-trail work. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Both publish paths (publish.yml -> PyPI on release; vsce publish -> VS Code Marketplace) exist but have never been exercised for real. This documents the exact one-shot sequence for when that changes, including the fact that no PYPI_API_TOKEN secret is currently configured on the repo (verified via `gh secret list`) and that the extension's package.json has no top-level PNG `icon` field yet for the Marketplace listing. Also fixed a stale test-count comment in README. Co-authored-by: Cursor <cursoragent@cursor.com>
0.9 (DAG virtual sub-agents) never shipped as its own release — it landed on main in the same PR (#9) as the 0.10.0 product work, one commit later. Keeping it as a separately-dated 0.9.0 section implied a release that never happened; folding it into 0.10.0's Added list matches what was actually tagged and published. Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(v0.11): CI verification gate on the audit trail
Closes the one real portability gap in the v0.10 audit trail: episodes
live in a local ~/.loopllm/ SQLite store, which doesn't exist on a CI
runner. `loopllm audit --export <path>` writes the (optionally
--since-filtered) episode list as a portable JSON artifact instead, so
it can be committed alongside the code it verifies.
`loopllm audit-gate --since <ref> [--min-score X] [--require-verified]`
reads that artifact back and checks every non-merge commit in the
range against it. Report-only by default (always exits 0) so a team
can dogfood the report before opting into enforcement via either flag.
Ships as a reusable composite action (.github/actions/loopllm-gate)
plus a workflow that runs it against this repo's own PRs in
report-only mode — the concrete "trust + audit layer for teams" payoff
the v0.10 commit_sha/audit-trail work set up.
commits_since() gained a no_merges option (a merge commit isn't itself
something an agent wrote and verified).
Tests: tests/test_cli_audit_gate.py (export round-trip, report-only vs
enforced modes, --min-score, merge-commit exclusion, bad-ref handling)
plus a no_merges unit test in test_project_scope.py. 294 tests pass,
ruff/mypy --strict clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: remove github-context expression from action.yml description
Action metadata (action.yml) is parsed independently of the calling
workflow, so it can't resolve the `github` context — GitHub's action
manifest loader tried to evaluate the ${{ github.event... }} example
left in the `since` input's description text and failed with
"Unrecognized named-value: 'github'", breaking every job that uses
this composite action, including this repo's own loopllm-gate.yml.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The v0.11 CI verification gate merged to main but the release metadata was
never reconciled:
- Bump 0.10.0 -> 0.11.0 (pyproject, __init__).
- Promote the CI-gate entries from CHANGELOG [Unreleased] into a dated
[0.11.0] section.
- Fix the episodic-recall FTS5 seam note ("deferred to v0.11" -> planned for
v0.12), where the semantic-recall upgrade will actually land; v0.11 shipped
as the audit gate instead.
- Sync CONTRIBUTING test count to the current 294 passing.
No behavior change; build produces loopllm-0.11.0.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_bbd05dc2-40b0-447d-9ba2-fd6d0144487f) |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The v0.11 CI verification gate (#10) merged to
main, but the release metadata was never reconciled —mainstill described itself as v0.10.0. This is a metadata/docs-only pass somainis release-consistent for a cleanv0.11.0tag.0.10.0→0.11.0(pyproject.toml,src/loopllm/__init__.py).[Unreleased]into a dated## [0.11.0]section.No behavior change; no
src/logic touched.Test plan
ruff check src/ tests/cleanpytest tests/ -q— 294 passed, 4 skippedpython -m build— producesloopllm-0.11.0wheelmcp/structlog; clean in a deps-installed venv)Publishing
v0.11.0(tag, GitHub release, PyPI,.vsix) remains a manual step perRELEASING.md.Generated by Claude Code
Note
Low Risk
Documentation and version metadata only; no executable logic or API behavior changes beyond
__version__strings.Overview
Aligns main with the already-shipped v0.11 CI verification gate by bumping the package version to 0.11.0 in
pyproject.tomlandsrc/loopllm/__init__.py.CHANGELOG gets a dated
## [0.11.0]section (CI gate / audit export content promoted out of[Unreleased]); README and CONTRIBUTING pick up the v0.11.0 release line and current pytest counts.Docs that promised episodic FTS5 recall in v0.11 are retargeted to v0.12 (v0.11 was the audit gate; recall is still keyword-only)—including the module docstring in
episodes.py. No runtime or test behavior changes.Reviewed by Cursor Bugbot for commit 102c698. Bugbot is set up for automated code reviews on this repo. Configure here.