Roadmap 2.2: the files backend

CONTRIBUTING.md

@@ -53,7 +53,10 @@ copy is stale, and it also rejects what a token rewrite cannot translate:
 
 - `asyncio`, threads, sleeps and anything else with no token-for-token sync
   equivalent go in `src/calendaring/_concurrency.py`, written once per mode,
-  with an entry in the replacement map (`agather` -> `gather`);
+  with an entry in the replacement map (`agather` -> `gather`); blocking
+  I/O, such as the files backend's, goes through `await ato_thread(call, ...)`,
+  which runs it in a worker thread in the async mode and directly in the sync
+  one (`to_thread`);
 - a third-party `Async*` name, such as `httpx.AsyncClient`, would become a
   `SyncClient` that does not exist: import it through a mode-free module and
   map it in the replacement map, as `_caldav_io.py` does for caldav's

docs/design/API_DESIGN.md

@@ -288,6 +288,19 @@ class AsyncBackend:
   Part 2](PRIOR_ART_AND_DECISIONS.md#part-2-standards)); `http(s):` → probe, which is [roadmap 2.3](ROADMAP.md#23-icalendar-feed-backend-read-only-http)'s "feed or CalDAV" detection (until then, `http(s):` means CalDAV, so an `https:` feed needs `kind="feed"`);
   RFC 6764 discovery for a bare domain comes from `caldav`. Trackers cannot
   be detected and need `kind=`.
+- The files backend picks its layout from the path (roadmap 2.2): a path
+  ending in `.ics` is a single file, one collection, created on the first
+  write; a directory with subdirectories is a root whose subdirectories are
+  the collections, as vdirsyncer writes them; a directory holding `.ics`
+  files or a `displayname` is one vdir collection (what khal's `path =`
+  names); an empty directory is a root.  Only a root can create and delete
+  collections.  A path that does not exist (other than a new `.ics` file in
+  an existing directory), or a file that is not `.ics`, is a
+  `ConfigurationError`.  The vdir spec has no
+  place for the components a collection holds (CalDAV's
+  `supported-calendar-component-set`), so a collection made for fewer than
+  all of them records them in a hidden `.calendaring-components` file,
+  which other tools ignore.
 - *Renamed 2026-10-19 from `connect`:* for CalDAV, files and feeds
   the factory does no network I/O (caldav's `DAVClient()` only builds an
   object, and opening a file connects to nothing), so "connect"
@@ -460,9 +473,12 @@ interval after the actual completion" (org-mode's `.+1w`). That reads into
 `RRULE` something the RFC does not say, and is dropped. The next start
 follows the `RRULE`, and the interval-from-completion behaviour applies only
 when an `X-` property asks for it
-([recurring-ical-events issue 292](https://github.com/niccokunzmann/python-recurring-ical-events/issues/292)).
-The `icalendar` manipulation goes to `recurring_ical_events`, and the I/O
-stays here ([§10 Open questions](#10-open-questions), Q3).
+(proposed in
+[recurring-ical-events issue 292](https://github.com/niccokunzmann/python-recurring-ical-events/issues/292)).
+The `icalendar` manipulation goes to
+[edit-icalendar](https://github.com/pycalendar/edit-icalendar), and the I/O
+stays here ([§10 Open questions](#10-open-questions), Q3).  Until then,
+every backend declares `task.complete-recurring` `UNSUPPORTED`.
 
 **`relatives`** fetches the objects an item's `RELATED-TO` points at (and,
 for `PARENT`, the children pointing back), as caldav's `get_relatives()`
@@ -1053,8 +1069,10 @@ up `_complete_recurring_<mode>`, and that method is spelled
 spelling, and the mismatch is reported as
 [caldav issue 735](https://github.com/python-caldav/caldav/issues/735).)
 Completing the occurrence is `icalendar` manipulation, which goes to
-`recurring_ical_events`; fetching and storing stay here
-([§10 Open questions](#10-open-questions), Q3).
+[edit-icalendar](https://github.com/pycalendar/edit-icalendar); fetching and
+storing stay here ([§10 Open questions](#10-open-questions), Q3).  Until
+edit-icalendar has it, every backend declares `task.complete-recurring`
+`UNSUPPORTED`, the CalDAV backend included.
 
 [§9 Migrating from caldav](#9-migrating-from-caldav) compares the whole of
 caldav's API with this one.
@@ -1649,6 +1667,14 @@ marked as not blocking; none blocks
    a draft as of 2026-10-12 ([§3.7](#37-recurrence)). Its README also
    covers editing and deleting single occurrences, so the merge above may
    move there too, despite "no extra package" earlier in this question.
+   *Decided by the author, 2026-10-23:* all of it, the occurrence merge
+   and recurring-task completion included, goes to edit-icalendar, which
+   supersedes recurring-ical-events issue 292.  Until edit-icalendar has
+   it, every backend declares `recurrence.edit-occurrence`,
+   `recurrence.edit-this-and-future` and `task.complete-recurring`
+   `UNSUPPORTED`, the CalDAV backend included: it neither copies nor
+   calls caldav's code for them, so the "interval after completion"
+   guess above is not inherited either.
 4. *Decided, 2026-10-10:* **`verify=True`** on writes is in from the first
    release ([§5.3 What happens when the caller asks for something unsupported](#53-what-happens-when-the-caller-asks-for-something-unsupported)).
 5. *Resolved, 2026-10-09:* assignees are `ATTENDEE`s with the tracker's
@@ -1844,7 +1870,8 @@ video calls. What it changed:
 the item base class is `Item` and carries no iCalendar properties of
 its own ([§3.1](#31-the-base-a-typed-view-over-icalendar)); what an item may
 hold is spelled out ([What an item contains](#what-an-item-contains)); the
-occurrence merge goes to `recurring_ical_events` (Q3); and change tracking is
+occurrence merge goes to `recurring_ical_events` (Q3; moved to edit-icalendar
+on 2026-10-23); and change tracking is
 Q13, which decided on 2026-10-12 how writes are tracked (a tracked subclass
 of the `icalendar` components, pending icalendar issue 1914; replaced on
 2026-10-15 by unconditional saves), and [§12](#12-beyond-the-funded-scope-implicit-synchronization)

docs/design/ROADMAP.md

@@ -293,10 +293,10 @@ Notes from doing it:
   `verify` are written once in the base class.
 - Two features were added to [§5.1](API_DESIGN.md#51-the-table):
   `recurrence.edit-occurrence` (saving one occurrence) and
-  `task.complete-recurring`. The reference backend declares both
-  `UNSUPPORTED` until the merge lands in `recurring_ical_events`
-  ([API_DESIGN Q3](API_DESIGN.md#10-open-questions)); the CalDAV backend
-  (2.1) can declare them `FULL` by calling caldav's code.
+  `task.complete-recurring`. Every backend, CalDAV included, declares
+  both `UNSUPPORTED` until
+  [edit-icalendar](https://github.com/pycalendar/edit-icalendar) has them
+  ([API_DESIGN Q3](API_DESIGN.md#10-open-questions)).
 - Sorting reads `icalendar_searcher.Searcher._sort_keys`, a private
   attribute; `test_search_sorts` catches a rename, and a public accessor
   belongs in icalendar-searcher.
@@ -554,13 +554,18 @@ The README promises that the API works identically against local files and remot
 servers. That promise is what makes this more than a `glob()` call.
 
 **Tasks:**
-- [ ] Support both layouts: one directory of `.ics` files, and a single file
-- [ ] UID-to-filename mapping, including collisions and unsafe characters
-- [ ] Atomic writes and file locking
-- [ ] An ETag surrogate (mtime and/or content hash) for collision detection
-- [ ] A sync-token surrogate for change detection
-- [ ] Search via `icalendar-searcher`, so filtering behaves as it does server-side
-- [ ] Full CRUD, passing the conformance suite
+- [x] Support both layouts: one directory of `.ics` files, and a single file
+      (and a root directory of vdir collections, as vdirsyncer writes them)
+- [x] UID-to-filename mapping, including collisions and unsafe characters
+- [x] Atomic writes and file locking (a lock only for the single file, whose
+      writes are a read-modify-write of the whole file)
+- [x] An ETag surrogate (mtime and/or content hash) for collision detection
+- [x] A sync-token surrogate for change detection
+- [x] Search via `icalendar-searcher`, so filtering behaves as it does server-side
+- [x] Full CRUD, passing the conformance suite.  Occurrence edits and
+      recurring-task completion stay `UNSUPPORTED` until
+      [edit-icalendar](https://github.com/pycalendar/edit-icalendar) has
+      them (API_DESIGN Q3)
 
 ---
 

news/+files-backend.feature.rst

@@ -0,0 +1 @@
+A files backend: ``Backend.open("~/.calendars")``, a ``file:`` URL, or a section with a path as its ``url``.  It reads and writes a vdir as vdirsyncer, khal and todoman lay it out (a root of collection directories, or one collection directory) and a single ``.ics`` file, with atomic writes, a lock around each write to a single file, and etags that catch a file edited behind its back.  In the async mode, file I/O runs in worker threads.  It passes the conformance suite in all three layouts.

pyproject.toml

@@ -24,6 +24,8 @@ classifiers = [
 ]
 dependencies = [
   "calendaring-config>=0.1",
+  # The files backend's lock on a single .ics file, on every platform.
+  "filelock>=3.12",
   "icalendar>=7.3",
   "icalendar-searcher>=1.1",
   "recurring-ical-events>=3.8",

src/calendaring/__init__.py

@@ -55,6 +55,9 @@
 # Register the reference backend's kind ("memory") in both modes.
 from . import memory as _memory  # noqa: E402, F401  isort: skip
 
+# The files backend ("files"): plain paths and file: URLs.
+from . import files as _files  # noqa: E402, F401  isort: skip
+
 # The CalDAV backend ("caldav"), when the caldav extra is installed.  Checked rather than
 # caught, so that an ImportError inside calendaring's own modules is not mistaken for it.
 if _find_spec("caldav") is not None:

src/calendaring/_async/files.py

@@ -0,0 +1,211 @@
+# Hand-written source; tools/generate_sync.py derives ../_sync/files.py from this file.
+"""The files backend (roadmap 2.2): a vdir, one collection directory, or one ``.ics`` file.
+
+``AsyncBackend.open(path)`` picks the layout from the path (``_files_store.layout``):
+a path ending in ``.ics`` is a single file, a directory holding objects or a
+``displayname`` is one collection, any other directory is a root of
+collections, one subdirectory each.  Every file operation runs in a worker
+thread in the async mode, and directly in the sync one.
+"""
+
+from __future__ import annotations
+
+import os
+from collections.abc import Iterable, Sequence
+from pathlib import Path
+from typing import Any, ClassVar
+from urllib.parse import urlsplit
+from urllib.request import url2pathname
+
+import icalendar
+from icalendar_searcher import Searcher
+
+from .._concurrency import ato_thread
+from .._files_store import Entry, IcsFile, Vdir, collections, layout, make_collection
+from ..capabilities import EDIT_ICALENDAR_PENDING, NO_EVENTS_FOR_TIME_LOG, Capabilities, Feature, Support
+from ..errors import ConfigurationError, InvalidDataError
+from ..items import _parse
+from ..loss import LossPolicy
+from ..types import BrokenObject, Component, StoredObject
+from .backend import AsyncBackend
+from .collection import AsyncCollection
+
+__all__ = ["AsyncFilesBackend", "AsyncFilesCollection"]
+
+F, E, U = Support.FULL, Support.EMULATED, Support.UNSUPPORTED
+
+
+def collection_capabilities(single_file: bool, components: frozenset[Component]) -> Capabilities:
+    """What a collection does; a single file is re-serialised on every write and names objects by UID."""
+    table: dict[Feature, Any] = {
+        **dict.fromkeys(Feature, F),
+        Feature.SEARCH_SERVER_SIDE: E,
+        Feature.SEARCH_TIME_RANGE: E,
+        Feature.SEARCH_TEXT: E,
+        Feature.CHANGES: {"level": E, "method": "etag-listing"},
+        Feature.WRITE_CONDITIONAL: {"level": E, "note": "the etag is compared, then the file replaced"},
+        Feature.MOVE: {"level": E, "note": "added to the target, then deleted here"},
+        Feature.IDENTITY_FOREIGN_ID: {"level": U, "note": "the UID is native"},
+        Feature.RECURRENCE_EDIT_OCCURRENCE: {"level": U, "note": EDIT_ICALENDAR_PENDING},
+        Feature.RECURRENCE_EDIT_THIS_AND_FUTURE: U,
+        Feature.TASK_COMPLETE_RECURRING: {"level": U, "note": EDIT_ICALENDAR_PENDING},
+    }
+    if single_file:
+        table.update(
+            {
+                Feature.CHANGES_KNOWN: E,
+                Feature.WRITE_WIRE: {"level": U, "note": "the file is re-serialised on every write"},
+                Feature.IDENTITY_CLIENT_NATIVE_ID: {"level": U, "note": "an object is named by its UID"},
+                Feature.COLLECTION_SET_NAME: U,
+            }
+        )
+    if Component.EVENT not in components:  # time entries are VEVENTs in the item's own collection
+        table[Feature.TIME_LOG] = {"level": U, "note": NO_EVENTS_FOR_TIME_LOG}
+    for feature in (Feature.CREATE_COLLECTION, Feature.DELETE_COLLECTION):
+        del table[feature]
+    return Capabilities(table)
+
+
+def _stored(entry: Entry, *, written: bool = False) -> StoredObject:
+    return StoredObject(entry.native_id, entry.etag, _parse(entry.data), None if written else entry.data, entry)
+
+
+def _listed(entry: Entry) -> StoredObject | BrokenObject:
+    try:
+        return _stored(entry)
+    except InvalidDataError as exc:
+        return BrokenObject(entry.native_id, entry.etag, entry.data, exc)
+
+
+class AsyncFilesCollection(AsyncCollection):
+    _backend: AsyncFilesBackend
+
+    def __init__(
+        self,
+        backend: AsyncFilesBackend,
+        store: Vdir | IcsFile,
+        *,
+        name: str | None,
+        color: str | None,
+        components: frozenset[Component],
+    ):
+        self.store = store
+        super().__init__(
+            backend,
+            id=store.path.name,
+            name=name,
+            color=color,
+            components=components,
+            capabilities=collection_capabilities(isinstance(store, IcsFile), components),
+            native=store,
+        )
+
+    async def _query(self, searcher: Searcher | None) -> Sequence[StoredObject | BrokenObject]:
+        return [_listed(e) for e in await ato_thread(self.store.entries)]
+
+    async def _fetch(self, native_id: str) -> StoredObject:
+        return _stored(await ato_thread(self.store.read, native_id))
+
+    async def _fetch_many(self, native_ids: Iterable[str]) -> list[StoredObject | BrokenObject]:
+        return [_listed(e) for e in await ato_thread(self.store.read_many, list(native_ids))]
+
+    async def _list_etags(self) -> dict[str, str]:
+        return await ato_thread(self.store.etags)
+
+    def _token_scope(self) -> str:
+        return f"files:{os.path.abspath(self.store.path)}"  # the path, not an id two backends may share
+
+    async def _create(self, calendar: icalendar.Calendar, *, native_id: str | None, wire: bytes | None) -> StoredObject:
+        data = wire if wire is not None else calendar.to_ical()
+        return _stored(await ato_thread(self.store.create, data, native_id=native_id), written=True)
+
+    async def _update(
+        self, native_id: str, calendar: icalendar.Calendar, *, etag: str | None, wire: bytes | None
+    ) -> StoredObject:
+        data = wire if wire is not None else calendar.to_ical()
+        return _stored(await ato_thread(self.store.update, native_id, data, etag=etag), written=True)
+
+    async def _delete(self, native_id: str, *, etag: str | None) -> None:
+        await ato_thread(self.store.delete, native_id, etag=etag)
+
+    async def _set_name(self, name: str) -> None:
+        assert isinstance(self.store, Vdir)  # a single file declares collection.set-name UNSUPPORTED
+        await ato_thread(self.store.set_name, name)
+
+    async def _delete_collection(self) -> None:
+        assert isinstance(self.store, Vdir)  # only a root declares delete-collection
+        await ato_thread(self.store.remove)
+
+    def _wrap_native(self, native_item: object) -> StoredObject:
+        if isinstance(native_item, Entry):
+            return _stored(native_item)
+        return super()._wrap_native(native_item)
+
+
+class AsyncFilesBackend(AsyncBackend):
+    """``await AsyncBackend.open("~/.calendars")``, which finds the layout in a worker thread.
+
+    Constructing it directly, ``AsyncFilesBackend("~/.calendars")``, looks at
+    the path on the spot: fine in a script, not inside a running event loop.
+    """
+
+    kind = "files"
+    collection_class: ClassVar[type[AsyncFilesCollection]] = AsyncFilesCollection
+
+    def __init__(
+        self,
+        path: str | Path,
+        *,
+        id: str | None = None,
+        loss: LossPolicy | None = None,
+        verify: bool | None = None,
+        _layout: str | None = None,
+    ) -> None:
+        self.path = Path(path).expanduser()
+        self.layout = _layout or layout(self.path)
+        level = F if self.layout == "root" else U
+        super().__init__(
+            id=id or str(self.path),
+            capabilities=Capabilities({Feature.CREATE_COLLECTION: level, Feature.DELETE_COLLECTION: level}),
+            native=self.path,
+            loss=loss,
+            verify=verify,
+        )
+
+    @classmethod
+    async def _open(cls, url: str, **options: Any) -> AsyncFilesBackend:
+        for unused in ("username", "password", "token", "session"):
+            options.pop(unused, None)
+        parts = urlsplit(url)
+        path = Path(url2pathname(parts.path) if parts.scheme == "file" else url).expanduser()
+        try:
+            return cls(path, _layout=await ato_thread(layout, path), **options)
+        except TypeError as exc:
+            raise ConfigurationError(f"the files backend does not take that option: {exc}") from None
+
+    def _stores(self) -> list[tuple[Vdir | IcsFile, str | None, str | None, frozenset[Component]]]:
+        """Each collection's store, name, colour and components: blocking, so run in a worker thread."""
+        stores: list[Vdir | IcsFile]
+        if self.layout == "file":
+            stores = [IcsFile(self.path)]
+        elif self.layout == "collection":
+            stores = [Vdir(self.path)]
+        else:
+            stores = [Vdir(p) for p in collections(self.path)]
+        return [
+            (s, s.name, s.color, s.components) if isinstance(s, Vdir) else (s, s.name, None, frozenset(Component))
+            for s in stores
+        ]
+
+    async def collections(self) -> list[AsyncCollection]:
+        return [
+            self.collection_class(self, store, name=name, color=color, components=components)
+            for store, name, color, components in await ato_thread(self._stores)
+        ]
+
+    async def _create_collection(self, name: str, components: frozenset[Component]) -> AsyncCollection:
+        path = await ato_thread(make_collection, self.path, name, components)
+        return self.collection_class(self, Vdir(path), name=name, color=None, components=components)
+
+    def __repr__(self) -> str:
+        return f"<{type(self).__name__} {self.path} ({self.layout})>"