Merge pull request #19 from nuclearcat/improve-housekeeping

Improve housekeeping

Author: nuclearcat

README.md

@@ -5,6 +5,7 @@ It supports multiple backends: Azure Blob Storage and local filesystem, to provi
 It caches the files in a local directory and serves them from there.
 Range requests are supported, but only for start offset, end limit is not implemented yet.
 Files are protected by upload locking to prevent concurrent uploads to the same path.
+To limit disk usage, a background housekeeping loop samples disk space every five minutes, keeps the cache below 1,000,000 items, and deletes the oldest cached files in configurable batches (default 100,000 files) whenever the file-count or disk-space thresholds are exceeded. A housekeeping summary is logged every five minutes showing the current free space, cache size, and any clean-up actions taken.
 
 ## Configuration
 
@@ -41,6 +42,18 @@ name = "user@email.com"
 prefixes = ["/allowed/path"]
 ```
 
+### Cache Housekeeping
+
+Cache housekeeping enforces a hard limit of 1,000,000 cached artifacts (`*.content` files) and frees disk space whenever free capacity drops below 12% (stopping once it rises above 13%). Files are always deleted in least-recently-updated order. The size of each deletion batch is configurable via the optional `[cache]` section:
+
+```toml
+[cache]
+# Number of cached files deleted per housekeeping iteration when limits are exceeded
+cleanup_chunk_size = 100000  # Defaults to 100000
+```
+
+Larger chunk sizes reclaim space faster when the cache is far above the limit, while smaller chunks reduce the amount of data removed per iteration.
+
 ## Creating user tokens
 
 The server uses JWT token based authentication. The token is passed in the `Authorization` header as a Bearer token.

config-local.toml

@@ -3,3 +3,6 @@ jwt_secret="test-secret-for-local-storage-testing"
 
 [local]
 storage_path="./test-storage"
+
+[cache]
+cleanup_chunk_size=100000

docs/_index.md

@@ -90,7 +90,7 @@ curl https://files.kernelci.org/metrics
 
 - **JWT Authentication**: Secure token-based authentication for uploads
 - **Multiple Storage Backends**: Currently supports Azure Blob Storage with extensible driver architecture
-- **Local Caching**: Files are cached locally with automatic cleanup when disk space is low
+- **Local Caching**: Files are cached locally with automatic cleanup rules; housekeeping enforces a hard limit of 1,000,000 cached entries, deletes the oldest files in configurable batches (default 100,000 files), and frees disk space whenever available space falls below 12%
 - **Range Request Support**: Partial content downloads using HTTP range requests
 - **File Locking**: Prevents concurrent uploads to the same file path
 - **Prometheus Metrics**: System monitoring and metrics collection
@@ -157,6 +157,17 @@ name = "admin@example.com"
 prefixes = [""]  # Empty prefix allows access to all paths
 ```
 
+### Cache Housekeeping
+
+The cache directory is capped at 1,000,000 cached artifacts (`*.content` files). When the limit is exceeded—or when disk space drops below 12%—the housekeeping worker deletes the oldest entries in batches. The batch size defaults to 100,000 files and can be overridden in the configuration file:
+
+```toml
+[cache]
+cleanup_chunk_size = 100000
+```
+
+Raising the value makes each cleanup iteration more aggressive; lowering it favors smaller, more frequent deletions.
+
 ## Environment Variables
 
 - `STORAGE_DEBUG`: Enable debug logging