Conversation
| return cross_edges_decorated(node_id) | ||
|
|
||
| def parents_multiple(self, node_ids: np.ndarray, *, time_stamp: datetime = None): | ||
| node_ids = np.array(node_ids, dtype=NODE_ID) |
There was a problem hiding this comment.
Just saw this here (and some other places) - same as in #458: np.array will by default create a copy. np.asarray will avoid copies, if the requirements are already met.
sdorkenw
left a comment
There was a problem hiding this comment.
Overall this looks good besides the one point - a tricky one though - that I marked
| new_cx_edges_d[layer] = edges | ||
| assert np.all(edges[:, 0] == new_id) | ||
| cg.cache.cross_chunk_edges_cache[new_id] = new_cx_edges_d | ||
| entries = _update_neighbor_cross_edges( |
There was a problem hiding this comment.
I think this here can introduce problems if a neighboring node is a neighbor to multiple new_l2_ids.
_update_neighbor_cross_edges looks right to me. It writes a complete new set of L2 edges for a node. But if the same node is updated multiple times, then only the last update is reflected. Maybe the logic here takes care of this somehow but then it still introduces multiple unnecessary writes.
So, if I am correct about this, the solution would be to consolidate this call across all new_l2_ids to only make one call per neighboring node id.
| new_cx_edges_d[layer] = edges | ||
| assert np.all(edges[:, 0] == new_id) | ||
| cg.cache.cross_chunk_edges_cache[new_id] = new_cx_edges_d | ||
| entries = _update_neighbor_cross_edges( |
1ddb0a7 to
17dfc10
Compare
b13d8ec to
d90813d
Compare
c460a5a to
6a2c5da
Compare
bf90549 to
d2d9d44
Compare
280f9fe to
cc4cd46
Compare
6605bab to
dcbecd1
Compare
Drop the vendored grid/harness/lock/exit_codes for the shared cave_pipeline.distribution package so the operator and every worker compute the same chunk-scatter bijection from one source; workers inject cg_factory and layer_bounds into the generic harness. Co-Authored-By: Claude <noreply@anthropic.com>
Picks up the 1-byte chunk-done marker the ingest workers write. Co-Authored-By: Claude <noreply@anthropic.com>
meta resolution/bounds derive from the watershed info JSON; sv lookup and the seg fallback read voxels via a neuroglancer_precomputed handle. cloud-volume stays a lazy ws_cv hatch (meshing/diagnostics). Co-Authored-By: Claude <noreply@anthropic.com>
nested imports (graph_tool via a _graph_tool shim) keep graph_tool, scipy, pandas, networkx, and cloudfiles off the cold import path; first use pays the load. bump kvdbclient to 0.7.1 (drops its cloud-volume). Co-Authored-By: Claude <noreply@anthropic.com>
…ds_by_label fastremap 1.20.0 emits 6-conn boundary voxels per label natively, so the `_label_boundary_mask` axial-diff pass and the `vol *= mask` mutation both go away. The point_cloud output (and therefore downstream KDTree min-distance queries) is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
ws_ts_scale(mip) reads the target scale (non-OCDBT mip>0 read mip 0). Mesh block size derives per-axis from the watershed pyramid, so chunk_size leaves mesh_config; setup rejects an out-of-range mip. Tests mock the tensorstore watershed reads. Co-Authored-By: Claude <noreply@anthropic.com>
The pipeline entrypoint carried a verbatim copy of setup_mesh_meta and MeshConfig that still required mesh_config.chunk_size, so mesh-meta failed once chunk_size was dropped from the dataset yaml. Import the single source of truth from pychunkedgraph.meshing instead. Co-Authored-By: Claude <noreply@anthropic.com>
PCG isn't on PyPI, but the image must report the pushed tag. The image pip-installs the package (--no-deps) with the tag fed to setuptools_scm via a cloudbuild build-arg; the hand-bumped literal + bumpversion are gone. Co-Authored-By: Claude <noreply@anthropic.com>
Non-meshing modules (app routes, sv-split profiler, pipeline/ingest entrypoints) imported meshing eagerly, pulling cloudvolume on import. Nest those imports so cv loads only when meshing actually runs. Co-Authored-By: Claude <noreply@anthropic.com>
Meshing splits initial from edited roots at a timestamp boundary, but sampling one root is unreliable: skip connections spread root creation times across layers. Instead, stamp earliest_ts when the root layer is written — the explicit cell timestamp shared by every root, lifted +500ms so the boundary sits strictly above them. get_earliest_timestamp returns it pre-edit; derive_initial_ts consumes it. Migrate no longer clobbers an ingest-stamped value. Co-Authored-By: Claude <noreply@anthropic.com>
setuptools_scm rejects non-PEP-440 strings, so pushing a build-label tag (not a semver) failed the image build. Pass the version build-arg only for version-like tags; other tags build with the Dockerfile default untouched. Co-Authored-By: Claude <noreply@anthropic.com>
A table copied/restored under a new graph_id must not let its meshes alias the source's. Hinge on dynamic_mesh_dir: an explicit graph-suffixed value shares initial meshes, so re-derive only the dynamic subdir; a bare "dynamic" or unset value gives the copy a private per-graph top-level dir. Move the whole rewrite into ChunkedGraphMeta.for_copied_graph so the graph class makes a single call. Co-Authored-By: Claude <noreply@anthropic.com>
Replace setuptools_scm (built 0.0.0 without a reachable semver tag) with a committed _version.py the release workflow bumps, commits, and tags in lockstep; setup.py reads it and the image build drops the version arg. Gate the workflow's Helm-chart update behind an opt-in input and add a workflows README. Versioning is per branch: main 2.x, pcgv3 3.x. Co-Authored-By: Claude <noreply@anthropic.com>
At chunk_layer + 1 == layer_count the parent column is rank-1 and the existing layer_agreement / np.where path returns chunk_layer instead of the root. Short-circuit to layer_count so meshes traversing up from the second-to-top layer reach the root. Co-Authored-By: Claude <noreply@anthropic.com>
Supervoxel splitting with base+fork support and locks
scipy 1.17.1's Python 3.14 wheel ships without the scipy._external subpackage, so `from scipy import ndimage` raises ModuleNotFoundError and every meshing/sv-split test errors during collection. Co-Authored-By: Claude <noreply@anthropic.com>
numpy is C-ABI-bound to conda's graph-tool-base, yet pip-compile also pinned it, so the image ran conda's latest against pip's downgraded lock — a broken mixed install. Pin it once in requirements.yml; [tool.pip-tools] unsafe-package keeps pip-compile from re-adding it to requirements.txt. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Add [tool.pytest.ini_options] with testpaths so a bare `pytest` collects the suite; test configuration previously lived only in the tox command. Co-Authored-By: Claude <noreply@anthropic.com>
The fixture duplicated gen_graph with file-backed edge/component I/O but had no callers. Co-Authored-By: Claude <noreply@anthropic.com>
Replace the duplicated `datetime.now(UTC) - timedelta(days=10)` with one helper so the "safely old" edit/lineage timestamp has a single source; rename local vars that collided with the helper name. Co-Authored-By: Claude <noreply@anthropic.com>
A test graph is determined by its atomic chunks; build_graph takes only that topology and builds the full (derivable) parent hierarchy at one timestamp, replacing hand-listed add_parent_chunk scaffolding. Co-Authored-By: Claude <noreply@anthropic.com>
Supervoxels are named and given as SV(x, y, z, seg) with zeros defaulting away, so build specs carry no repeated raw coordinate tuples; build_graph returns a BuiltGraph(cg, sv, ts) namedtuple and callers reference sv by name.
Drop the unused ts from the result; callers unpack `cg, sv = build_graph(...)` and reference supervoxels by name.
Replace the per-fixture gen_graph + create_chunk + add_parent_chunk + to_label scaffolding with named-supervoxel build_graph specs. The parent hierarchy is derived from the atomic chunks rather than hand-listed, and readable SV() coordinates replace the repeated raw (x, y, z, seg) tuples. Co-Authored-By: Claude <noreply@anthropic.com>
Reference every node through readable SV coordinates — build_graph for setup, the new label(cg, SV, layer) for construction/encoding tests — instead of positional to_label tuples. Adds an assert_graph_unchanged context manager for rejected-edit atomicity; drops the dead query fixture. Co-Authored-By: Claude <noreply@anthropic.com>
Move the edit-operation tests into tests/graph/edits/ with a coverage README, and lift the duplicated split/merge/undo setup graphs into shared edits/conftest.py fixtures. Co-Authored-By: Claude <noreply@anthropic.com>
Restore the original finite affinity (a split test must exercise the same edge type); the migration had switched it to an inf cross-chunk edge. Co-Authored-By: Claude <noreply@anthropic.com>
Shrink the lock-acquire backoff where the failed acquire is only setup, lower the test lock-expiry, and skip the doomed best-effort error-artifact write that dominated the error-path test. Assertions unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
Every test parametrizes over kvdbclient_testing.backends(); the per-backend branches, the bigtable emulator bootstrap, and the hbase mock are gone, now shipped by kvdbclient. bootstrap() resolves the config class via get_config_class(). Requires kvdbclient>=0.8.0. Co-Authored-By: Claude <noreply@anthropic.com>
MaxAgeGCRulefor previous column family with supervoxel cross chunk edges; only needed during ingest and they get deleted eventually.Summary of changes in
pychunkedgraph.ingest:layer>3untilrootlayer:This assumes all chunks at lower layer have been created before creating the current layer so we can no longer queue parent chunk jobs automatically when its children chunks are complete.
We must now ingest/create one layer at a time.
Summary of changes in
pychunkedgraph.graph.edits: