materialization: pre-build Filter results per partition, with UI, poller and cache fixes

.github/copilot-instructions.md

@@ -44,7 +44,11 @@ flowview/                               # Repository root (install to plugins/fl
 ├── flowview_upgrade.php                # Background schema upgrade runner
 ├── flow_collector.php                  # NetFlow v5/v9/v10 collector daemon
 ├── functions.php / functions-pre13.php # Core UI + protocol decode helpers
-├── import_flows.php                    # CLI flow file importer
+├── cli/                                # CLI utilities (importer, repair, materializer)
+│   ├── fixprivate.php                  # CLI: repair domains of private-range addresses
+│   ├── import_flows.php                # CLI flow file importer
+│   ├── materialize_queries.php         # CLI: pre-build each query's partition cache (NAT on/off)
+│   └── repair_tables.php               # CLI: widen the raw partition table columns
 ├── run_schedule.php                    # CLI scheduled-report executor
 ├── INFO                                # Plugin metadata (name, version, compat)
 ├── README.md

CHANGELOG.md

@@ -2,6 +2,28 @@
 
 --- develop ---
 
+* feature: Say which partitions a Filter is not materialized for, with NAT and without, in the tip of its Required and Running pill
+
+* feature: Show the number of queries of each Filter that are running now in a second pill, in the style of a disabled device and reading Running: N, beside the status pill on the Filters page, with how many are with NAT and without in its tip.  The process table only has the id of a running request, so a running query is matched to a Filter by the cache key its map query is stored under, which is worked out from the Filter, and there is no pill when none are running
+
+* feature: Add --status to cli/materialize_queries.php to list the parallel queries that are running and the Filter each one is for, with its progress
+
+* feature: Have cli/materialize_queries.php get the partitions ready before it starts the Parallel Query.  It waits for an ALTER TABLE that is already running on a partition it needs, such as from the schema upgrade, since a query waits for it and then ends at the Maximum Run Time, and then does the alters that are still due itself, one partition at a time with progress shown.  The partition still receiving flows is left alone.  It also notes the partitions that have no NAT columns, as they have no NAT data.  What a partition needs altered is now decided in one place, flowview_get_partition_alter(), which the schema upgrade also uses, so the ALTER it issues is unchanged
+
+* feature: Materialize the partitions that have gone cold automatically.  Each poller cycle flowview_process.php records the newest partition, which is the one still receiving flows, and starts cli/materialize_queries.php in the background for the partitions that have rotated since the previous cycle, for every Filter with NAT on and off.  Partitions that are already materialized and accurate are skipped.  When a materialization is already running, or the Parallel Query is not enabled, the record is left alone so a later cycle catches up.  The first cycle only records the newest partition, so older partitions are still materialized with cli/materialize_queries.php.  The script gained a --partitions option for this
+
+* feature: Add a Materialization column, to the right of the Filter Name and its ID on the Filters page, with a tip explaining what materialization is, showing the automatic materialization status as a pill in the form of the device status: Required (down) when rotated partitions are not materialized yet, Running (recovering) while a materialization is in progress, and Up to date (up) once every cold partition is materialized with NAT on and off.  The ID column now follows the Filter Name
+
+* feature: Show the number of partitions each Filter has stored in the Parallel Query cache, with NAT and without, as the NAT Shards and Non-NAT Shards columns next to the ID on the Filters page, along with a Partitions column holding the total number of flow partitions so the progress of a materialization can be followed; the cache keys are worked out from the Filter without running it (run_flow_query() gained a plan-only mode, and the map query hashing is shared with the parallel query request)
+
+* feature: Add a 'Materialize' action to the Filters page that, after confirmation, starts cli/materialize_queries.php in the background for the selected Filters (--query-id now accepts a comma separated list), and tells you when a materialization is already running
+
+* feature: Recommend a forced re-materialization when a Filter is saved, from either the Filter form or the report page, with a change that alters what it materializes (report type, device, exporter, template, or any address, port, interface, AS, protocol, TCP flag or TOS restriction, plus the sort field of printed reports); display settings, the time range, cutoffs and NAT mode do not count
+
+* feature: Add cli/materialize_queries.php, which runs every saved FlowView query through the parallel query for the entire range of flow partitions, once with NAT on and once with NAT off (--query-id, --nat, --dry-run and --force are optional), so the partition results are already cached when a report is opened; it registers with register_process_start() for 8 hours so only one run is active at a time, reuses cached partitions that still match their partition's data bounds, rebuilds the ones that have since gained flows or been dropped, and runs each query and NAT mode in its own process so one that reaches the Maximum Run Time does not end the rest
+
+* dev: Move fixprivate.php, import_flows.php and repair_tables.php into the plugin's cli/ directory (listed as tombstones in manifest.json so upgrades remove the old copies); they now locate the Cacti root relative to their own location
+
 * fix: Make bundled Net_DNS2 constructor nullability explicit for PHP 8.4 without changing accepted arguments
 
 * issue: Shorten the graph drilldown button title from 'View NetFlow Traffic In Range' to 'NetFlow' so it reads cleanly as a label in Cacti's graph action menu

README.md

@@ -194,11 +194,93 @@ this migration, ensure that your Cacti system, has enough space to handle all
 the flow data.  You should check the size of your existing flows, and then
 verify that you have enough space to handle the data inside of a MySQL database.
 
-Once you have done this, simply run the 'import_flows.php' script and all your
+Once you have done this, simply run the 'cli/import_flows.php' script and all your
 legacy flow data will be imported.  Remember, it's important that you define
 your partitioning scheme ahead of time, especially if you have large quantities
 of flow data streaming into the Cacti server.
 
+## Pre-Building Query Results
+
+The Parallel Query caches the result of each fully covered partition, so
+reports over older data open quickly once a partition has been scanned.  To
+build that cache ahead of time, run the following from the plugin's `cli`
+directory with the Parallel Query enabled (Max Concurrent Threads of 2 or
+more under the FlowView Settings):
+
+```
+php cli/materialize_queries.php
+```
+
+The script runs every saved query over every flow partition that has rotated,
+once with NAT on and once with NAT off.  The partition still receiving flows is
+never cached, as a result cached for it would be wrong as soon as more flows
+arrive.  Instead, the table and charts of one report share a single scan of it,
+whose result is reused for 60 seconds and then discarded.  Use `--query-id=ID[,ID...]` or
+`--nat=on|off` to narrow the run, and `--dry-run` to see what would be run.
+Partitions that are already cached and still match their data are skipped,
+while partitions that have received more flows since, or have been dropped,
+are rebuilt.  Each Filter being materialized is registered in the Cacti process
+table as `materialize`, with the id of the Filter as the task id, for up to 8
+hours, so Filters can be materialized separately and at the same time.  A Filter
+that is already being materialized is skipped, unless `--force` is used.  The
+Filters page shows only those Filters as Running.  `--status` lists the Filters
+being materialized, and the parallel queries that are running now with the
+Filter each one is for, with its progress.  When it is not run from a terminal,
+for instance from the Filters page or by the poller, its progress is written to
+the Cacti log.  Sending it SIGTERM or SIGINT stops its workers and removes
+its registrations from the process table.
+
+Only the partitions a Filter does not have cached are run, a batch of them at a
+time, each as its own Parallel Query, as one query for every partition can not
+finish within the Maximum Run Time.  A batch that does not finish is followed by
+a smaller one, and what a batch cached is kept, so a run that is stopped, or
+timed out, carries on from there when it is run again.
+
+Before it scans anything the script gets the partitions ready.  A query waits
+for an ALTER TABLE that is running on the partition it scans, and a long one
+ends the query at the Maximum Run Time.  So the script waits for an alter that
+is already running on a partition it needs, for instance from the schema
+upgrade, and then does the alters that are still due itself, one partition at a
+time.  The partition still receiving flows is left alone.  The post-NAT columns
+are not added, as that is a deliberate choice for large installs.  The script
+notes the partitions that do not have them, and they have no NAT data until
+`flowview_upgrade_nat_columns.php` is run.
+
+You can also start it from the Filters page by selecting one or more Filters
+and choosing the 'Materialize' action.  It runs in the background after
+you confirm, and the result is written to the Cacti log.  The 'NAT Shards' and
+'Non-NAT Shards' columns on that page show how many partitions of each Filter
+are already cached, with NAT on and with NAT off, out of the total shown in the
+'Partitions' column.
+
+New partitions are materialized automatically.  Each poller cycle records the
+newest partition, which is the one still receiving flows, and materializes the
+partitions that have rotated since the previous cycle for every Filter, with
+NAT on and off, in the background.  The first cycle only records the newest
+partition, so run the script once to materialize the existing ones.  The
+'Materialization' column on the Filters page shows the result for each Filter,
+as Required, Running or Up to date.  The partition still receiving flows is not
+materialized automatically, so it does not count.
+
+When you save a
+Filter in a way that changes the data it materializes, such as its report,
+device, exporter, or any address, port, interface, AS, protocol, flag or TOS
+restriction, a dialog warns you first, and you can Cancel to keep editing or
+Continue to save.  Once it is saved, a 'Save Succeeded' dialog reminds you to
+start a materialization of the Filter with the 'Materialize' action.
+
+The report of a Filter is part of what is cached for it, so it can not be
+changed on the report page: the Report selector is disabled once the Filter has
+one.  To change it, Edit the Filter and then materialize it again.  Saving a
+Filter does not remove anything from the cache.  The results of the old settings
+simply stop matching the Filter.  The line under the Filters table says how many
+cached shards match a Filter, which is told by their md5 value, and how many
+match none, such as those of a Filter before it was changed, of a deleted
+Filter, or of a report with its settings changed on the report page.  The
+'Prune' button on the Filters page, after Clear, removes the ones that match no
+Filter, after you confirm in its dialog.  It keeps everything that matches a Filter, and does nothing when
+the cache keys of a Filter that has a report can not be worked out.
+
 # Possible Bugs?
 
 If you figure out this problem, goto GitHub and create a pulls request or open

fixprivate.php → cli/fixprivate.php

@@ -23,7 +23,7 @@
  +-------------------------------------------------------------------------+
 */
 
-chdir('../../');
+chdir(__DIR__ . '/../../../');
 require('./include/cli_check.php');
 
 global $config;

import_flows.php → cli/import_flows.php

@@ -23,7 +23,7 @@
  +-------------------------------------------------------------------------+
 */
 
-chdir('../../');
+chdir(__DIR__ . '/../../../');
 require('./include/cli_check.php');
 
 global $config;