feat(query): migrate to ClickHouse + web-Lapache backend endpoints#196
Open
jacobjurek wants to merge 1 commit into
Open
feat(query): migrate to ClickHouse + web-Lapache backend endpoints#196jacobjurek wants to merge 1 commit into
jacobjurek wants to merge 1 commit into
Conversation
Brings the query service on main up to date: the ClickHouse migration plus the web-Lapache backend endpoints it builds on (these were not yet on main). ClickHouse migration: - Swap sqlalchemy + psycopg2 for clickhouse-connect (HTTP, port 8123). - connection.py: shared client; init_db creates the service's metadata tables in ClickHouse (query_log MergeTree; query_token + signal_definition ReplacingMergeTree, read with FINAL). - Signal/cluster SQL rewritten for ClickHouse: DISTINCT ON -> LIMIT BY, EXTRACT(EPOCH ...) -> integer bucketing on the timestamp micros column, AT TIME ZONE ::date -> toDate(produced_at, tz). produced_at is MATERIALIZED (queryable, absent from SELECT *), so it is always named explicitly. - Models become plain dataclasses; metadata services use clickhouse-connect. - docker-compose query block + example.env: CLICKHOUSE_* over HTTP 8123 (gr26 keeps native 9000). Supporting endpoints (pre-existing on the feature branch): - /signals/names, /clusters, /clusters/dates; raw-data decimation via max_points; cached highest-frequency anchor signal; dev reload watcher scoped to the app package. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the query service from Postgres to ClickHouse, and brings the query work it builds on up to date on
main. Scoped toquery/plus the query service's docker-compose block and theCLICKHOUSE_*entries inexample.env— no other service is touched.ClickHouse migration
sqlalchemy+psycopg2; addclickhouse-connect(HTTP interface, port 8123).connection.py: one shared client;init_dbcreates the service's own metadata tables in ClickHouse —query_log(MergeTree, append-only),query_token+signal_definition(ReplacingMergeTree, read withFINAL).DISTINCT ON→LIMIT BY,EXTRACT(EPOCH …)→ integer bucketing on thetimestampmicros column,AT TIME ZONE …::date→toDate(produced_at, tz).produced_atis a MATERIALIZED column (queryable, just hidden fromSELECT *), so it's always named explicitly.docker-composequery block +example.env:CLICKHOUSE_*over HTTP 8123 (gr26 keeps native 9000).Supporting endpoints (pre-existing on the feature branch)
/signals/names,/clusters,/clusters/dates; raw-data decimation viamax_points; cached highest-frequency anchor signal; dev reload watcher scoped to the app package.Verification
pytest→ 12/12 pass./signals(pivot + decimation via LIMIT BY),/signals/names,/clusters(gap split),/clusters/dates(tz),/token(write), token-auth on/signals(FINALread), andquery_logwritten on every/signalscall.🤖 Generated with Claude Code