Repository navigation
materialization: pre-build Filter results per partition, with UI, poller and cache fixes - #298
Merged
Merged
Conversation
The filter's timeshiftBackward/timeshiftForward handlers called timeshiftFilterLeft()/timeshiftFilterRight(), which neither the plugin nor Cacti core defines, causing an uncaught ReferenceError. Add a timeshift action that shifts date1/date2 by the selected interval and a plugin-owned timeshiftFilter() that applies it and reloads the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The time shift callback called selectmenu('refresh') unguarded, which throws
when the timespan select is not an initialized selectmenu, aborting before
applyFilter() ran. Use a guarded setCustomTimespan() helper (instance check,
as core does) for both the time shift and the date change handler.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a CLI script that runs every saved query through the parallel query for the entire range of flow partitions with NAT on and off. It registers a process with an 8 hour limit, reuses cached partitions that still match their data, and rebuilds stale or dropped ones. Run each query in its own process so reaching the run limit does not end the rest. Move fixprivate.php, import_flows.php and repair_tables.php into cli/, tombstone the old paths in manifest.json, and locate the Cacti root relative to each script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ached shard counts Add a Materialize View action to the Filters page that, after confirmation, starts cli/materialize_queries.php in the background for the selected filters; --query-id now accepts a comma separated list. Recommend a forced re-materialization when a filter is saved with a change that alters what it materializes, from both the filter form and the report page. Show the number of cached partitions with NAT and without for each filter next to its ID. run_flow_query() gains a plan-only mode that returns the cache keys without running the query, and the map query hashing is shared with the parallel query request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Partitions column after the NAT and Non-NAT shard columns so the progress of a materialization can be followed. Count only genuine raw partition tables, and use the same helper in cli/materialize_queries.php so its partition count matches the page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on of rotated partitions Add a Status column to the Filters page showing whether the automatic materialization of each filter is required, running or up to date. Each poller cycle now records the newest partition and starts cli/materialize_queries.php in the background for the partitions that went cold since the previous cycle, for every filter with NAT on and off. The script gains a --partitions option, and the cache check can be limited to those partitions. The cached shard lookup now returns the partitions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d show status pills Have cli/materialize_queries.php wait for any ALTER TABLE already running on the partitions it needs and do the alters still due before the parallel query starts, checking each partition again just before altering it. What a partition needs altered is decided by flowview_get_partition_alter(), which the schema upgrade now also uses. Skip the poller's engine conversion while an ALTER TABLE is already running on the partition, so each cycle no longer queues another rebuild of the same table. Print a parallel query timeout in words when run from a script instead of the web page's JavaScript. Render the Filters page status as a pill in the form of the device status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…per filter The Running status only looked at the materializer's own registration, so it disappeared when the script ended, even though the parallel query it started kept working. Look for the runner and shard worker processes of requests made by a script too, ignoring rows left behind by processes that are gone, and do not start a new materialization while one is running. Make the parallel query cleanup and the runner's termination handler use the names the processes register, db_query_<id> and db_shard_<id>, so a timeout actually stops them and a terminated one removes its row. Show the number of running queries of each filter in a pill beside the status pill, matched to the filter by its cache key. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… they are for Take the running parallel queries from the db_query_<id> and db_shard_<id> registrations in the process table, whoever started them, counting a registration while it is recent or its process is alive. This no longer depends on the request belonging to user 0. Match each running request to its filter by cache key and show the number of running queries per filter, with the NAT and non-NAT split in the tip. Add --status to cli/materialize_queries.php to list the running requests, the filter each is for and their progress. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…olumn Stop cli/materialize_queries.php from showing each ALTER TABLE twice: the log message was also printed, as well as the script's own progress line. The alter is still logged, and the schema upgrade still prints it. Rename the Filters page Status column to Materialization and explain in its tip what materialization is and what the pills and the Running count mean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ll and timeout The Running: N pill tip passed only the text domain to __esc() for a text with placeholders, which Cacti rejects with an ArgumentCountError and broke the Filters page whenever a query was running. The pill label and the Parallel Query timeout message had the same mistake and would have shown the domain and 0 seconds. Pass the values first, as the rest of the plugin does. Make the unit test __() and __esc() stubs treat every argument as a sprintf argument like Cacti, so a mistake like this fails a test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A shard is only cached when the range starts before the earliest flow in the partition. Flows can start before the window of their partition, but the range was padded by only a second, so the oldest partition, or a rotated one materialized on its own, was never cached and its filter stayed Required. Pad the start by a day, and report the first and last partition in the summary. Say which partitions are not materialized, with NAT and without, in the tip of the Required and Running pill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The partition still receiving flows was scanned as a full scan by a full run and its result cached, which is wrong as soon as more flows arrive, and it was counted as materialized. Decide whether a shard is a full scan in flowview_is_full_scan(), which refuses the newest partition, so no query can cache it. cli/materialize_queries.php now only materializes partitions that have rotated, and removes anything cached for the live partition before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The live partition is no longer cached, so the table and every chart of a report scanned it again with the same map query. Workers running the same map query and range against it now take a named lock and the first keeps its result for 60 seconds for the others, under map_partition 'live' so it is never counted as materialized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rs materialized, convert every cold partition The Go query service aged out every cached partition after the Cached Data Time to Live, emptying the NAT and non-NAT shards overnight, so it only expires the short lived live shard results now. The Filters page showed every filter as Running while any materialization ran, so it uses the filters the run is for. The poller only converted the 2nd to 4th newest partitions, so any it missed stayed InnoDB, so it converts every cold partition, three per cycle and one converter at a time. flowview_runner.php now loads setup.php itself instead of relying on the plugin hooks having loaded it. The bug entries of this session are removed from the CHANGELOG as nothing has been released. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…for cached shards that match no filter The report is part of the cache key, so the Report selector is disabled on the report page for a filter that has one, and the server ignores a report sent for it. The Filters page tells the cached shards that match a filter from the others by md5, and a Prune button removes the others after confirmation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tl constants from the web server cli/materialize_queries.php now handles SIGTERM and SIGINT like the flow collector: the script stops its worker and removes its process registration, and a worker cancels its parallel queries, which stops their runner and shard workers and removes their registrations. Requests are tracked per process for that. parallel_database_query_cleanup() used SIGTERM, which does not exist under the web server without pcntl, so a timed out report was a fatal error that disabled the plugin; signals are now sent through flowview_signal_process(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…p the poller converting under it Each filter a materialization works on is registered in the process table as materialize with the filter id as the task id, so filters can be materialized separately and the Filters page reads which are running from the process table instead of a setting. The poller no longer starts ALTER TABLE conversions while a filter is being materialized, as the materializer waits for any alter on the partitions it needs and the poller kept one running, and the materializer logs its progress to the Cacti log when it has no terminal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d request, and add Prune to the Filters actions A parallel query request is reused for the same query and range, and a finished one only hands back its result, so a materialization after the shard cache was lost finished at once and cached nothing. The materializer now discards a finished request of the same query first. Prune Cached Shards is also an action in the Filters page dropdown, as well as the Prune button. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Report selector is no longer shown, only a hidden input id=report that carries the value the requests use, unless the selected filter has no report and needs one. The title of a printed report named the report by the statistical index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e filter No Report control or field is on the page for a filter that has a report. The requests no longer send it, the server takes it from the filter for the page, the charts and the session, a printed report has its panel checkboxes disabled by the server, and Save no longer changes the report of a filter. A filter with no report still shows the selector as it needs one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… left behind by a timeout One parallel query for every partition can not finish within the Maximum Run Time, and a timeout leaves the request behind as scheduled, so the next run found it, started nothing and waited again. The materializer now runs only the partitions a filter has not cached, a batch at a time, with smaller batches after a timeout, and a request that is not finished, has no runner and is old enough is removed when the same query is requested again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ore filter editing Pruning is for the whole cache, so it is no longer an action of the Filters dropdown or a confirmation page. The Prune button after Clear opens a modal jQuery UI dialog saying what will be removed and kept, and its Prune button posts action=prune_shards, which only acts on a POST. Adding the prune actions had dropped the case 'edit' label, so editing a Filter from the Filters page showed the list instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nd confirm the save in a dialog Saving a Filter with a change to what is cached for it raised a warning message, which Cacti showed as 'Some Actions failed'. The Save button now asks the server whether the change is material and, if so, opens a jQuery UI dialog with Cancel and Continue before the form is posted. Once saved, a 'Save Succeeded' dialog says to start a materialization with the Materialize View action. Other saves keep Cacti's usual Save Successful message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… the Filters page Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…p jquery-multiselect before it From 1.2.32 Cacti's drop_multi renders a select2 'N Selected' control itself, and the Filter editor also made it a jquery-multiselect, so both widgets were drawn. The editor now only calls .multiselect() before 1.2.32, or when core did not decorate the select for select2, and the Protocols field shows 'All Protocols' when none are picked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The messages and documentation that name the action are renamed with it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Worker cancellation, batch partition scoping, concurrent schema preparation, and printed-report panel state have unresolved correctness defects.
4 open issues
What changed in this PR
This PR adds per-partition materialization to FlowView’s Parallel Query cache, reducing report load times for historical flow data.
Changes:
- Adds resumable materialization through a CLI, Filter actions, and automatic poller scheduling.
- Adds cache status, pruning, and save dialogs; updates report controls and time shifting.
- Revises cache expiration, process cleanup, and partition preparation, with supporting tests and documentation.
| File | Description |
|---|---|
tests/Unit/FlowviewFilterControlsTest.php |
Tests report selection and printed-report controls. |
tests/Unit/CoreFunctionsTest.php |
Adds materialization, cache, process, and UI regression coverage. |
tests/bootstrap-unit.php |
Updates translation and logging stubs. |
tests/bin/patch-coverage.php |
Updates CLI entry-point exclusions. |
service/flowview-query/scheduler.go |
Limits shard expiration to short-lived live results. |
service/flowview-query/scheduler_test.go |
Tests revised shard expiration behavior. |
service/flowview-query/README.md |
Documents cold-cache retention and live-result expiration. |
README.md |
Documents materialization commands and UI workflows. |
manifest.json |
Tracks relocated CLI utilities and obsolete paths. |
includes/arrays.php |
Adds the select2 Protocols placeholder. |
flowview.php |
Adds time shifting and enforces saved report settings. |
flowview_upgrade.php |
Reuses the shared partition-alter planner. |
flowview_runner.php |
Loads required helpers and corrects process cleanup names. |
flowview_process.php |
Converts cold partitions and schedules materialization. |
flowview_filters.php |
Adds materialization actions, status, counts, and pruning. |
cli/repair_tables.php |
Updates bootstrap paths after relocation. |
cli/materialize_queries.php |
Implements batching, preparation, status, and worker management. |
cli/import_flows.php |
Updates bootstrap paths after relocation. |
cli/fixprivate.php |
Updates bootstrap paths after relocation. |
CHANGELOG.md |
Records materialization features and CLI relocation. |
.github/copilot-instructions.md |
Documents the CLI directory layout. |
Review effort: Balanced
Every alter of a raw partition now goes through flowview_alter_partition(), which holds a per-table named lock and works out the plan after taking it, so materializers of different Filters, the poller and the schema upgrade never apply a stale plan to a table another process already altered. The poller does not wait for the lock. A materialization worker is given its batch's partitions and scans only those, instead of every partition in the padded range, and it is started with proc_open() from an argument list, not through a shell, so SIGTERM reaches it. A printed report's saved panels are made to agree with the table-only request before they are merged, so they no longer re-enable the charts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
display_domain() returned the resolved source and destination names as they were stored, and the report table printed them as HTML. They come from reverse lookups of the flow data, which whoever controls a reverse zone can set, so a crafted PTR name could inject markup. They are now escaped with html_escape(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… validate the stream override run_flow_query() returned a Filter's cache keys in plan mode, so its return value could, as far as static analysis can tell, carry request values to the report table, the summary and the CSV export. The plan is now given through an out parameter behind flowview_plan_filter(), and what run_flow_query() returns is only ever the report. The ex_addr stream override was bound to the query as it came; it is now only taken when it is an IP address, as exporter addresses are, and otherwise the Filter's own stream is used. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
xmacan
previously approved these changes
Oct 22, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The pull request gate requires every changed line that the unit suite measures to be covered. This adds tests for the Filter editor and its saves, the chart data, planning a Filter's cache keys, the parallel and serial report paths, the request and shard lifecycle, the live partition shard sharing, a query that times out, and the partition bookkeeping. To make those paths testable: - flowview_exit() wraps exit so the unit bootstrap can see it instead of the process ending; save_filter(), flowview_get_chartdata() and the parallel query timeout use it. - includes/arrays.php is measured, as it changed. - The unreachable "No valid report was selected" branch is removed, the report is known to be statistical or printed by then, and cacti.pot is regenerated. - The printed report branch inside the two map query loop, which only runs for statistical reports, is removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TheWitness
force-pushed
the
fix-toggles
branch
from
October 22, 2026 13:19
01280ad to
139592d
Compare
The web page branch of the timeout report depends on CACTI_WEB, a constant a unit test cannot change, so the report moves into flowview_report_parallel_timeout() with whether it is a web request as an argument, and both ways are tested. The cleanup of a request's map table and db_debug() are covered too, as the change moves the diff onto them. The test stub of form_start() records its arguments instead of printing them, which SonarCloud took for a reflected XSS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
xmacan
approved these changes
Oct 22, 2026
This was referenced Oct 22, 2026
cigamit
pushed a commit
to Cacti/cacti
that referenced
this pull request
Oct 22, 2026
…8442) * fix(themes): skin jQuery multiselect in the Deepness theme The Deepness main.css repainted select2 and the jQuery UI selectmenu onto its --dp-* palette but never the jQuery multiselect button or popup menu, so those fell through to the jquery-ui.css textured surfaces and the generic uppercase button rule and looked like the Dark theme. Extend main.css (the stock jquery.multiselect.css is left untouched) so the multiselect button, menu, header, hover/active/disabled labels and group labels use the same colors as the theme's select2 controls. Reported in Cacti/plugin_flowview#298. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(themes): drop the ineffective Deepness multiselect border-radius .ui-corner-all forces border-radius with !important in Deepness, and the multiselect button carries that class, so the declaration never applied. Let the button follow the theme's jQuery UI corner rule like the other widgets. Also tag the CHANGELOG entry with the pull-request number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: TheWitness <copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TheWitness
added a commit
to Cacti/cacti
that referenced
this pull request
Oct 22, 2026
…n four themes (develop) (#8443) * fix(themes): skin jQuery multiselect in Deepness and the 3.x markup in four themes Deepness: main.css repainted select2 onto its --dp-* palette but never the jQuery multiselect button or popup menu, so those fell through to the jquery-ui.css textured surfaces and the generic uppercase button rule and looked like the Dark theme. Repaint them to match the select2 controls. Cacti, Carrot, Hollyberry, Raspberry: these themes ship the old 1.x jquery.multiselect.css while include/js/jquery.multiselect.js is 3.x, so optgroup menus showed list bullets, a nested indent, compounded 0.9em option text and an unstyled group label. Add the missing 3.x rules. All changes extend each theme's main.css; the stock jquery.multiselect.css files are left untouched. Reported in Cacti/plugin_flowview#298. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(themes): satisfy the develop CHANGELOG and theme coverage gates - Move the CHANGELOG entries into changelog.d/8443.issue, as develop expects fragments rather than direct CHANGELOG edits. - Drop the Deepness multiselect border-radius; .ui-corner-all forces border-radius with !important in this theme, so it never applied. - Add Chromium computed-style tests for the new multiselect rules in Deepness, Cacti, Carrot, Hollyberry and Raspberry so the changed CSS lines are exercised by the theme correctness suite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: TheWitness <copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.





Summary
This branch adds materialization to FlowView: pre-computing each Filter's per-partition results into the Parallel Query shard cache, so reports over older data open quickly. It also fixes the bugs that turned up while running it on a large install (about 1,470 hourly partitions). The original time-shift arrow fix is still included.
What changed
Materialization
cli/materialize_queries.php(new) materializes every saved Filter, or--query-id=ID[,ID...], over the cold partitions, with NAT on and off.--partitions,--nat=on|off,--dry-run,--force,--status,--debug.ALTER TABLEand then applies the pending ones itself, using a shared planner (flowview_get_partition_alter()) thatflowview_upgrade.phpnow also uses.processesasmaterializewith the Filter id as the task id, so Filters can be materialized separately and at the same time.map_partition = 'live'.Filters page (
flowview_filters.php)New columns:
The ID column now sits right after Filter Name.
New Materialize action. It skips Filters that are already being materialized and waits 2 seconds before redirecting, so the Running state shows.
A line under the table counts the cached shards that match a Filter (by md5 cache key) and those that match none.
A Prune button, after Clear, opens a non-modal jQuery UI dialog and POSTs
action=prune_shards, which removes only the shards that match no Filter.Saving a Filter with a change that affects its cache now:
This replaces the WARN message that Cacti showed as "Some Actions failed".
Report page (
flowview.php)timeshiftFilterLeft/Right, which were undefined.Fixes found along the way
service/flowview-query/scheduler.go):expireShardCachedeleted every cached shard older than the Cached Data Time to Live (6 hours by default), which emptied the cache overnight. It now expires only the 60-secondliverows. Cold shards are removed only when their partition is dropped.db_query<id>anddb_shard<id>, but the processes register asdb_query_<id>anddb_shard_<id>, so they were never found.parallel_database_query_cleanup()usedSIGTERM, which is undefined under the web server. That was a fatal error that disabled the plugin. Signals now go throughflowview_signal_process().scheduledby a timeout made every later run wait out the full limit. Unfinished requests with no runner that are more than 2 minutes old are now removed.flowview_runner.phploadssetup.phpitself, so it no longer depends on the plugin hooks to defineflowview_connect().drop_multialready renders as select2, so the Filter editor no longer also calls.multiselect()on Protocols, which drew two controls. Earlier Cacti versions keep jquery-multiselect.__()values are now passed before theflowviewtext domain in the new strings.Housekeeping
fixprivate.php,import_flows.phpandrepair_tables.phpmoved tocli/. They are tombstoned inmanifest.jsonso upgrades remove the old copies.Reviewers should know
deepnesstheme does not skin jquery-multiselect.main.cssstyles.ui-button,.ui-selectmenu-*and select2, but not.ui-multiselect/.ui-multiselect-menu.jquery.multiselect.cssis identical to the dark theme's.Testing
php -lon every changed file;tests/bin/validate-manifest.phppasses.tests/Unit+tests/Security, 305 passed. New tests cover materialization batching, request reuse, live-shard sharing, prune, signal handling, the per-Filter registration, the save dialogs and the select2 gating.go vetandgo test ./...pass.node --check.