Persist session ACL in snapshots; restore on session_load (MN-REQ-06.16)

CHANGELOG.md

@@ -7,7 +7,14 @@ This project uses Semantic Versioning as **interpreted for MemNet**: package `a.
 
 ## [Unreleased]
 
+### Added
+- **Session ACL in snapshots (MN-REQ-06.16)** — session save, expire save, checkpoint, and safe-upgrade snapshots carry the CapsPolicy ACL: enabled flag, per-caller grants (pin_map, mutate, WorkerWriteScope), and optional bind. The ACL is one `#@ACL:` comment line after `@SNAP`. `session load` restores it, both from `--file` and on the `--keep-id` reload path. `@SNAP` is now **version 2**, and version 2 **requires** the `#@ACL:` line: a version 2 snapshot without it fails `bad_snapshot`. Version 1 (written before the ACL was persisted) still loads, with ACL off, and prints `@ACL: absent|off`. A loaded ACL prints `@ACL: loaded|enabled=<0|1>|callers=<n>|bind=<0|1>`. A malformed ACL line fails the load instead of silently turning ACL off. The snapshot and the upgrade passport share one serialiser (`memnet.acl.acl_to_dict` / `acl_from_dict`). Readers built before this change reject `@SNAP` version 2 with `bad_snapshot` (unsupported snapshot version 2) rather than skipping the ACL line, and an upgrade manifest with `snapshot_format` 2 is refused by an older serve (`upgrade_snapshot_format`, exit 3). Verify: MN-VER-06-S14 (`tests/test_acl_snapshot.py`).
+
+- **One-time v1 backup before the first v2 write (MN-REQ-06.16)** — the first snapshot write over a format 1 file (session save, expire save, checkpoint, safe-upgrade) copies the v1 bytes to `<file>.v1.bak` in the same directory, at mode `0600`, through `memnet/atomic_file.py`. An existing backup is never overwritten, and v2-over-v2 writes make no backup. There is no dual write: the snapshot itself is always v2. **Rollback to 0.19.23** means renaming the `.v1.bak` files over their snapshots (see [`docs/operations/safe-upgrade.md`](docs/operations/safe-upgrade.md)). That loses writes made after the upgrade. Old builds refuse v2 loudly (`bad_snapshot`, `upgrade_snapshot_format`) by design.
 ### Changed
+- **Snapshot and manifest files are private and atomic (MN-REQ-06.16)** — every snapshot (session save, expire save, checkpoint, safe-upgrade), `upgrade-manifest.json`, `upgrade-blocked.json`, `upgrade-restore.json`, and `checkpoint-manifest.json` is written through one helper (`memnet/atomic_file.py`): a temp file created `0600` (`O_EXCL`), fsynced, then renamed. Files are `0600` even when they replace an older file with a looser mode. The upgrade passport is written in the same snapshot write, not appended afterwards.
+- **Bind stores a lease digest, not the lease (MN-REQ-06.16)** — `session acl-bind` keeps the SHA-256 of the lease (`memnet.acl.lease_digest`). `check_bind` hashes the presented `--lease` (or `MEMNET_LEASE`) and compares digests in constant time. The snapshot, the passport, and the in-memory session never hold the raw lease. Consequence: a lost lease cannot be read back from a snapshot; re-run `session acl-bind` with the lease. The lease must be a high-entropy secret (SHA-256 is not a password KDF). A pre-digest snapshot with a raw `lease` loads and is hashed on load.
+- **Upgrade restore keeps the ACL enabled flag as written** — a passport ACL with `enabled: false` but grants no longer comes back enabled.
 - **Invent only — ClusterRoute vs SliceHandCarry (#191 / #47 cousin)** — `MemNetTwoMoves` outside `MemNetSystem` (`MN-REQ-06.9` + `MN-REQ-06.10` / `MN-VER-06-S08`). ClusterRoute = where the session lives (`MemNetLanMcpFront`; one owner; `pin_map` / `find` SHALL NOT span backends). SliceHandCarry = explicit copy into another session (`export_pin_map` or `session_save` → LAN file copy → dest import/`session_load`; `import_slice` same-serve only). Not a live hop. `import_slice(from_url)` not shipped. tip≠face. `inventOnly=true`; `implemented=false`; no engine code; no SemVer bump. Wire: [`docs/operations/cluster-route-vs-slice-hand-carry.md`](docs/operations/cluster-route-vs-slice-hand-carry.md).
 - **Invent only — LAN MCP front over several serves (#191)** — `MemNetLanMcpFront` outside `MemNetSystem` (`MN-REQ-06.9` / `MN-VER-06-S07`). One MCP catalogue, N LAN `memnet serve` backends; `SessionOwnerRegistry` is owner (explicit pin allowed; silent hash is not sole routing). One owner per session; `pin_map` / `find` SHALL NOT span backends. Cousin of #47 (peer sid handoff), not the same invent. tip≠face. `inventOnly=true`; `implemented=false`; no engine code; no SemVer bump. Wire: [`docs/operations/memnet-lan-mcp-front.md`](docs/operations/memnet-lan-mcp-front.md).
 

docs/operations/one-session-per-document.md

@@ -47,7 +47,7 @@ Client helper: `memnet.serve.send_command(args, stdin=…, host=…, port=…)`.
 
 ## Snapshot
 
-`session save --file` writes `# memnet-snapshot-v1` via `Path.write_text` (overwrite). MemNet does **not** make that file write-once (no `O_EXCL`, no `chmod`, no immutable flag). The caller or the filesystem can.
+`session save --file` writes `# memnet-snapshot-v1` (`@SNAP` version 2) through a temp file created `0600`, fsynced, then renamed over the target (overwrite, atomic). MemNet does **not** make that file write-once (no `O_EXCL` on the target, no immutable flag). The caller or the filesystem can.
 
 Anything mutate accepts must save and reload as the same property values. Snapshot emit escapes every Python `str.splitlines()` separator (LF, CR, VT, FF, FS/GS/RS, NEL, LS, PS) plus `\` and `|`; load splits records on LF only. A 16 KiB string is refused at GQL CREATE/SET with `@ERR: limit_exceeded|value_bytes 16384/4096` (one decoded cap with leftover pipe and snapshot load). Unicode, `|`, quotes, CR, and newlines on a value under the cap round-trip. Undeclared RAM keys persist by widening the snapshot SCHEMA; extras over `max_fields` refuse `snapshot_unsaveable`.
 

docs/operations/safe-upgrade.md

@@ -35,6 +35,23 @@ A clean stat retires the manifest inside that process. A later restart does not
 
 Point the unit `ExecStart` back at the old venv and start it when the restore stat is not clean. After a clean stat the manifest is retired, so a later start does not reload those snapshots.
 
+From the build after 0.19.23 the first write of each snapshot in format 2 keeps the old bytes once. A format 1 file that is overwritten by a format 2 write is copied to `<file>.v1.bak` (mode `0600`, same directory) before the replace. A later save never overwrites that backup.
+
+Rolling back to 0.19.23 (which reads format 1 only) means restoring those backups:
+
+1. Stop the new serve. It must not write during the rollback.
+2. For each snapshot that has a `.v1.bak`, rename the backup over the snapshot, for example `mv -f upgrade-snapshots/<sid>.snap.v1.bak upgrade-snapshots/<sid>.snap`. Do the same under `checkpoint/` and for any expire or session save file.
+3. Start the old venv.
+
+**This loses every write made after the upgrade.** The session state returns to the bytes saved before the upgrade, and nothing else is merged back.
+
+Two things an old build does loudly by design:
+
+- A format 2 snapshot with no `.v1.bak` (a file first written by the new build) is refused with `bad_snapshot` (`unsupported snapshot version 2`). It is not loaded as an empty or partial session.
+- A format 2 upgrade manifest is refused by the old serve (`upgrade_snapshot_format`, exit `3`).
+
+Keep the `.v1.bak` files until the new build has run cleanly. They are not removed by the new build.
+
 ## Hosts
 
 On rpi5-syson the serve listens on `18765` (`memnet-serve.service`). `memnet-mcp` on `18766` is not part of this procedure. The Endleaf engine serve is the backend named in the gateway registry; use that unit's port and state directory. The droplet gateway is unchanged: do not edit its version pin as part of this restart.

parts/common/memnet/memnet/acl.py

@@ -24,11 +24,13 @@
 
 from __future__ import annotations
 
+import hashlib
+import hmac
 import os
 import re
 from collections.abc import Iterable
 from dataclasses import dataclass, field
-from typing import Literal
+from typing import Any, Literal
 
 from memnet.exceptions import MemNetError
 
@@ -40,6 +42,7 @@
     r"^(anchors|ids|labels|relations)=(.+)$",
     re.IGNORECASE,
 )
+_HEX64 = re.compile(r"[0-9a-f]{64}")
 
 
 def _env_truthy(name: str) -> bool:
@@ -57,6 +60,15 @@ def in_process_trusted() -> bool:
     return _env_truthy("MEMNET_SERVE_INTERNAL") or _env_truthy("MEMNET_TEST_INLINE")
 
 
+def lease_digest(lease: str) -> str:
+    """SHA-256 hex of the stripped lease. The raw lease is never stored (MN-REQ-06.16).
+
+    SHA-256 is not a password KDF, so the lease MUST be a high-entropy secret
+    (for example 128 random bits or more).
+    """
+    return hashlib.sha256(lease.strip().encode("utf-8")).hexdigest()
+
+
 def skip_bind_allowed() -> bool:
     """In-process MAY skip bind; explicit MEMNET_ACL_SKIP_BIND also opts in."""
     if _env_truthy("MEMNET_ACL_SKIP_BIND"):
@@ -100,11 +112,13 @@ def to_wire(self) -> str:
 class SessionBind:
     """Optional missionId + lease bind; when set, mutate must match.
 
-    MemNet-only (not a Neo4j/AgensGraph privilege).
+    Holds ``lease_digest(lease)``, not the lease. The caller re-presents the
+    lease on each mutate and it is hashed before comparison. MemNet-only (not a
+    Neo4j/AgensGraph privilege).
     """
 
     mission_id: str
-    lease: str
+    lease_sha256: str
 
 
 @dataclass(frozen=True)
@@ -155,13 +169,64 @@ def set_bind(self, mission_id: str, lease: str) -> None:
                 "acl_bad_bind",
                 "mission_id and lease required together",
             )
-        self.bind = SessionBind(mission_id=mid, lease=lid)
+        self.bind = SessionBind(mission_id=mid, lease_sha256=lease_digest(lid))
         self.enabled = True
 
     def clear_bind(self) -> None:
         self.bind = None
 
 
+def acl_to_dict(acl: SessionAcl | None) -> dict[str, Any]:
+    """Serialise session ACL for snapshot ``#@ACL:`` and upgrade passport (MN-REQ-06.16)."""
+    if acl is None:
+        return {"enabled": False, "callers": [], "bind": None}
+    callers = []
+    for grant in acl.callers.values():
+        scope = grant.write_scope.to_wire() if grant.write_scope else None
+        callers.append(
+            {
+                "caller": grant.caller,
+                "can_pin_map": grant.can_pin_map,
+                "can_mutate": grant.can_mutate,
+                "write_scope": scope,
+            }
+        )
+    bind = None
+    if acl.bind is not None:
+        bind = {"mission_id": acl.bind.mission_id, "lease_sha256": acl.bind.lease_sha256}
+    return {"enabled": acl.enabled, "callers": callers, "bind": bind}
+
+
+def acl_from_dict(data: dict[str, Any]) -> SessionAcl:
+    """Rebuild a session ACL from ``acl_to_dict`` output; the enabled flag is kept as written."""
+    acl = SessionAcl()
+    for row in data.get("callers") or []:
+        acl.grant(
+            str(row.get("caller") or ""),
+            can_pin_map=bool(row.get("can_pin_map", True)),
+            can_mutate=bool(row.get("can_mutate", True)),
+            write_scope=parse_write_scope(row.get("write_scope")),
+        )
+    bind = data.get("bind")
+    if bind is not None:
+        if not isinstance(bind, dict) or not str(bind.get("mission_id") or "").strip():
+            raise MemNetError("bad_snapshot", "malformed ACL bind")
+        digest = str(bind.get("lease_sha256") or "")
+        if digest:
+            if not _HEX64.fullmatch(digest):
+                raise MemNetError("bad_snapshot", "malformed ACL bind digest")
+            acl.bind = SessionBind(mission_id=str(bind["mission_id"]).strip(), lease_sha256=digest)
+        elif bind.get("lease"):
+            # Written before the digest: hash the raw lease on load.
+            acl.set_bind(str(bind["mission_id"]), str(bind["lease"]))
+        else:
+            # Dropping the bind would silently loosen the ACL.
+            raise MemNetError("bad_snapshot", "ACL bind has no lease digest")
+    # grant() and set_bind() both force enabled; the stored flag wins.
+    acl.enabled = bool(data.get("enabled"))
+    return acl
+
+
 def parse_write_scope(raw: str | None) -> WorkerWriteScope | None:
     """Parse ``anchors=a,b;ids=x;labels=TSK;relations=about,owns``."""
     if raw is None:
@@ -280,7 +345,11 @@ def check_bind(
 
     mid = (mission_id or os.environ.get("MEMNET_MISSION_ID") or "").strip()
     lid = (lease or os.environ.get("MEMNET_LEASE") or "").strip()
-    if mid == acl.bind.mission_id and lid == acl.bind.lease:
+    if (
+        mid == acl.bind.mission_id
+        and lid
+        and hmac.compare_digest(lease_digest(lid), acl.bind.lease_sha256)
+    ):
         return
 
     if not require and skip_bind_allowed():

parts/common/memnet/memnet/atomic_file.py

@@ -0,0 +1,54 @@
+"""Atomic private file writes for snapshots and manifests (MN-REQ-06.16).
+
+The temp file is created with mode 0600 (``mkstemp`` uses ``O_CREAT|O_EXCL``
+and 0600), written, fsynced, then renamed over the destination. The
+destination is never a partial write, and it is never readable by other
+users, even when it replaces an older file created with a looser mode.
+"""
+
+from __future__ import annotations
+
+import os
+import tempfile
+from pathlib import Path
+
+
+def _fsync_fd(fd: int) -> None:
+    os.fsync(fd)
+
+
+def atomic_write_bytes(path: Path, data: bytes) -> None:
+    """Write ``data`` via a 0600 temp file, fsync, and rename.
+
+    A failure before rename leaves the destination untouched and removes
+    the temp file.
+    """
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, tmp_name = tempfile.mkstemp(
+        dir=path.parent,
+        prefix=f"{path.name}.",
+        suffix=".tmp",
+    )
+    tmp = Path(tmp_name)
+    try:
+        with os.fdopen(fd, "wb") as handle:
+            handle.write(data)
+            handle.flush()
+            _fsync_fd(handle.fileno())
+        os.replace(tmp, path)
+    except OSError:
+        try:
+            tmp.unlink(missing_ok=True)
+        except OSError:
+            pass
+        raise
+    try:
+        dir_fd = os.open(path.parent, os.O_RDONLY)
+    except OSError:
+        return
+    try:
+        _fsync_fd(dir_fd)
+    except OSError:
+        pass
+    finally:
+        os.close(dir_fd)

parts/common/memnet/memnet/checkpoint.py

@@ -9,17 +9,18 @@
 
 import hashlib
 import json
-import os
 import threading
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
 
 from memnet import __version__
 from memnet.admin_usage import authenticate
+from memnet.atomic_file import atomic_write_bytes
 from memnet.exceptions import MemNetError
 from memnet.registry import list_entries
 from memnet.session import iso_timestamp
+from memnet.snapshot import backup_v1_before_v2
 from memnet.upgrade import snapshot_format_version, state_dir
 
 CHECKPOINT_DIRNAME = "checkpoint"
@@ -29,43 +30,6 @@
 _checkpoint_lock = threading.Lock()
 
 
-def _fsync_fd(fd: int) -> None:
-    os.fsync(fd)
-
-
-def atomic_write_bytes(path: Path, data: bytes) -> None:
-    """Write ``data`` via a temp file, fsync, and rename.
-
-    A failure before rename leaves the destination untouched and removes
-    the temp file. The destination is never a partial write.
-    """
-    path.parent.mkdir(parents=True, exist_ok=True)
-    tmp = path.with_name(path.name + ".tmp")
-    try:
-        with tmp.open("wb") as handle:
-            handle.write(data)
-            handle.flush()
-            _fsync_fd(handle.fileno())
-        os.replace(tmp, path)
-    except OSError:
-        try:
-            tmp.unlink(missing_ok=True)
-        except OSError:
-            pass
-        raise
-    try:
-        dir_fd = os.open(path.parent, os.O_RDONLY)
-    except OSError:
-        return
-    try:
-        try:
-            _fsync_fd(dir_fd)
-        except OSError:
-            return
-    finally:
-        os.close(dir_fd)
-
-
 def _safe_name(session_id: str) -> str:
     cleaned = "".join(ch if ch.isalnum() or ch in "._-" else "_" for ch in session_id)
     return cleaned or "session"
@@ -180,6 +144,7 @@ def _checkpoint_locked(directory: Path | None) -> CheckpointResult:
         name = f"{_safe_name(sid)}.snap"
         path = dest / name
         try:
+            backup_v1_before_v2(path)
             atomic_write_bytes(path, payload)
         except OSError as exc:
             skipped.append(

parts/common/memnet/memnet/cli.py

@@ -469,9 +469,11 @@ def session_load(
 
                 _acl_check(SimpleNamespace(acl=acl), caller, "pin_map")
     purge_expired(caps)
+    from_snapshot = False
     try:
         if file is not None:
             ss = load_snapshot(file, caps=caps, ttl_minutes=ttl, keep_id=keep_id)
+            from_snapshot = True
         else:
             sid = session or os.environ.get("MEMNET_SESSION")
             if not sid:
@@ -497,13 +499,28 @@ def session_load(
                     keep_id=True,
                     hide_path=True,
                 )
+                from_snapshot = True
         emit_session(ss.session_id, ss.meta.expires_at, str(ss.meta.ttl_minutes))
         emit_stderr(f"MEMNET_SESSION={ss.session_id}")
         emit_stat("loaded", ss.store.row_count_non_law(), str(file) if file is not None else "-")
+        if from_snapshot:
+            _emit_loaded_acl(ss.session_id)
     except MemNetError as exc:
         _handle_error(exc)
 
 
+def _emit_loaded_acl(session_id: str) -> None:
+    """Say what the snapshot's ACL restored. Absent means an older snapshot (ACL off)."""
+    entry = get_entry(session_id)
+    acl = entry.acl if entry is not None else None
+    if acl is None:
+        emit_stdout("@ACL: absent|off")
+        return
+    enabled = 1 if acl.enabled else 0
+    bind = 1 if acl.bind is not None else 0
+    emit_stdout(f"@ACL: loaded|enabled={enabled}|callers={len(acl.callers)}|bind={bind}")
+
+
 @admin_app.command("usage-report")
 def admin_usage_report(
     token: Annotated[