Skip to content

Commit cacb538

Browse files
nuclearcatnuclearcat
committed
Documentation update for current code
Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
1 parent 95ac486 commit cacb538

2 files changed

Lines changed: 153 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ This is a simple storage server that supports file upload and download, with tok
44
It supports multiple backends, currently only Azure Blob is supported, to provide user transparent storage.
55
It caches the files in a local directory and serves them from there.
66
Range requests are supported, but only for start offset, end limit is not implemented yet.
7+
Files are protected by upload locking to prevent concurrent uploads to the same path.
78

89
## Configuration
910

@@ -16,6 +17,11 @@ account=""
1617
key=""
1718
container=""
1819
sastoken=""
20+
21+
# User upload permissions (optional)
22+
[[users]]
23+
name = "user@email.com"
24+
prefixes = ["/allowed/path"]
1925
```
2026

2127
## Creating user tokens
@@ -24,23 +30,53 @@ The server uses JWT token based authentication. The token is passed in the `Auth
2430
JWT secret is configured in the `config.toml` file.
2531

2632
```bash
27-
./kernelci-storage -generate_jwt_token user@email.com
33+
# Generate JWT secret
34+
./kernelci-storage --generate-jwt-secret
35+
36+
# Generate JWT token for a user
37+
./kernelci-storage --generate-jwt-token user@email.com
2838
```
29-
This will generate a JWT token for the user.
39+
The first command generates a JWT secret for your configuration, the second generates a token for the specified user.
3040

3141
### Testing Token Validity
3242

3343
You can verify if a token is valid using the `/v1/checkauth` endpoint:
3444

3545
```bash
36-
curl -X GET http://localhost:3000/v1/checkauth \
46+
curl -X GET https://localhost:3000/v1/checkauth \
3747
-H "Authorization: Bearer <JWT_TOKEN>"
3848
```
3949

4050
**Responses:**
4151
- `200 OK` with body `Authorized: user@email.com` - Token is valid
4252
- `401 Unauthorized` with body `Unauthorized` - Token is invalid or missing
4353

54+
## Environment Variables
55+
56+
- `KCI_STORAGE_CONFIG` - Override config file path (defaults to config.toml)
57+
- `STORAGE_DEBUG` - Enable debug logging
58+
4459
## API
4560

46-
See [docs](docs/) for the API documentation.
61+
### Endpoints
62+
63+
- `GET /` - Server status
64+
- `POST /v1/file` or `POST /upload` - File upload (requires JWT)
65+
- `GET /*filepath` - File download (public, supports range requests)
66+
- `GET /v1/checkauth` - Validate JWT token
67+
- `GET /v1/list` - List all files (public)
68+
- `GET /metrics` - Prometheus metrics endpoint
69+
70+
The server supports large file uploads up to 4GB and includes upload conflict protection to prevent concurrent uploads to the same file path. Downloads have a 30-second timeout for acquiring file locks.
71+
72+
### Metrics
73+
74+
The `/metrics` endpoint provides Prometheus-compatible metrics including:
75+
- `storage_free_space` - Available disk space
76+
- `storage_total_space` - Total disk space
77+
78+
Both metrics include hostname, diskname, and mount_point labels.
79+
80+
## API Documentation
81+
82+
See [docs](docs/) for detailed API documentation.

‎docs/_index.md‎

Lines changed: 113 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -4,47 +4,101 @@ date: 2025-08-06
44
description: KernelCI's Storage API documentation
55
---
66

7-
We have simple storage server that supports file upload and download, with JWT token based authentication.
7+
We have a simple Proxy server that supports file upload and download, with JWT token-based authentication. The server supports multiple storage backends, file caching, range requests, and provides Prometheus metrics.
88

99
## API Endpoints
1010

11-
The API is quite simple with upload through a `/upload` endpoint and download through a GET to the file url.
12-
1311
### Upload
1412

15-
`POST /upload`
13+
**`POST /upload`** or **`POST /v1/file`**
1614

17-
Upload a file to the server.
15+
Upload a file to the server. Requires JWT authentication.
1816

19-
field path: the path to store the file in the server.
20-
field file0: the file to upload.
17+
**Form fields:**
18+
- `path`: The path to store the file in the server
19+
- `file0`: The file to upload
2120

2221
### Download
2322

24-
`GET /path/to/file`
23+
**`GET /*filepath`**
24+
25+
Download a file from the server. Supports range requests for partial downloads.
26+
27+
### Authentication Check
28+
29+
**`GET /v1/checkauth`**
30+
31+
Validate a JWT token. Returns the authenticated user's email if valid.
32+
33+
### List Files
34+
35+
**`GET /v1/list`**
36+
37+
List all files in the storage (public endpoint).
2538

26-
Download a file from the server.
39+
### Server Status
40+
41+
**`GET /`**
42+
43+
Returns server status: "KernelCI Proxy Server"
44+
45+
### Metrics
46+
47+
**`GET /metrics`**
48+
49+
Prometheus metrics endpoint showing storage space and system information.
2750

2851
### Request a token
2952

3053
Ask the kernelci-sysadmin team for a token.
3154

32-
### Testing upload and download using curl
55+
### Testing with curl
3356

3457
```bash
35-
# Upload
36-
curl -X POST http://files.kernelci.org/upload \
58+
# Upload a file
59+
curl -X POST https://files.kernelci.org/upload \
60+
-H "Authorization: Bearer <JWT_TOKEN>" \
61+
-F "path=testfolder" \
62+
-F "file0=@local_folder/local_file"
63+
64+
# Alternative upload endpoint
65+
curl -X POST https://files.kernelci.org/v1/file \
3766
-H "Authorization: Bearer <JWT_TOKEN>" \
3867
-F "path=testfolder" \
3968
-F "file0=@local_folder/local_file"
4069

41-
# File will be uploaded to Azure Blob Storage as testfolder/local_folder/local_file
70+
# File will be uploaded as testfolder/local_folder/local_file
71+
72+
# Download a file
73+
curl https://files.kernelci.org/testfolder/local_folder/local_file
74+
75+
# Download with range request (partial content)
76+
curl -H "Range: bytes=0-1023" https://files.kernelci.org/testfolder/local_folder/local_file
77+
78+
# Check authentication
79+
curl -X GET https://files.kernelci.org/v1/checkauth \
80+
-H "Authorization: Bearer <JWT_TOKEN>"
4281

43-
# Download
44-
curl http://files.kernelci.org/testfolder/local_folder/local_file
82+
# List all files
83+
curl https://files.kernelci.org/v1/list
84+
85+
# Get metrics
86+
curl https://files.kernelci.org/metrics
4587
```
4688

47-
## Principle of operation
89+
## Features
90+
91+
- **JWT Authentication**: Secure token-based authentication for uploads
92+
- **Multiple Storage Backends**: Currently supports Azure Blob Storage with extensible driver architecture
93+
- **Local Caching**: Files are cached locally with automatic cleanup when disk space is low
94+
- **Range Request Support**: Partial content downloads using HTTP range requests
95+
- **File Locking**: Prevents concurrent uploads to the same file path
96+
- **Prometheus Metrics**: System monitoring and metrics collection
97+
- **User Access Control**: Configurable path-based upload permissions per user
98+
- **Content-Type Detection**: Automatic MIME type detection based on file extensions
99+
- **HTTP Caching Headers**: ETag and Last-Modified support for efficient caching
100+
101+
## Architecture
48102

49103
```mermaid
50104
sequenceDiagram
@@ -54,21 +108,56 @@ sequenceDiagram
54108
participant Azure Blob Storage
55109
56110
%% Upload Flow
57-
User->>Proxy Server: Upload File (POST Request)
58-
Proxy Server->>Azure Blob Storage: Store File in Cloud
59-
Azure Blob Storage-->>Proxy Server: Confirmation
60-
Proxy Server-->>User: Upload Successful
111+
User->>Proxy Server: Upload File (POST /upload or /v1/file)
112+
Note over Proxy Server: JWT Authentication & Path Permissions Check
113+
Proxy Server->>Proxy Server: Acquire File Lock
114+
Proxy Server->>Azure Blob Storage: Store File with Chunked Upload
115+
Proxy Server->>Local Cache: Cache File Locally
116+
Azure Blob Storage-->>Proxy Server: Upload Confirmation
117+
Proxy Server-->>User: Upload Successful (200 OK)
61118
62119
%% Download Flow
63-
User->>Proxy Server: Request File (GET Request)
120+
User->>Proxy Server: Request File (GET /*filepath)
121+
Proxy Server->>Proxy Server: Acquire File Lock (with timeout)
64122
Proxy Server->>Local Cache: Check if File Exists
65123
alt File Found in Cache
66124
Local Cache-->>Proxy Server: Return Cached File
67-
Proxy Server-->>User: Send File
125+
Proxy Server-->>User: Send File (with Range Support)
68126
else File Not in Cache
69127
Proxy Server->>Azure Blob Storage: Fetch File from Cloud
70128
Azure Blob Storage-->>Proxy Server: Send File Data
71-
Proxy Server->>Local Cache: Save File Locally
72-
Proxy Server-->>User: Send File
129+
Proxy Server->>Local Cache: Save File Locally (SHA-512 based filename)
130+
Proxy Server-->>User: Send File (with Range Support)
73131
end
74132
```
133+
134+
## Configuration
135+
136+
The server supports configuration through a `config.toml` file with the following structure:
137+
138+
```toml
139+
# Storage backend driver (defaults to "azure")
140+
driver = "azure"
141+
jwt_secret = "your-jwt-secret-here"
142+
143+
# Azure Blob Storage configuration
144+
[azure]
145+
account = "your-storage-account"
146+
key = "your-storage-key"
147+
container = "your-container-name"
148+
sastoken = "?sv=2022-11-02&ss=b..."
149+
150+
# User access control
151+
[[users]]
152+
name = "user@example.com"
153+
prefixes = ["/allowed/path1", "/allowed/path2"]
154+
155+
[[users]]
156+
name = "admin@example.com"
157+
prefixes = [""] # Empty prefix allows access to all paths
158+
```
159+
160+
## Environment Variables
161+
162+
- `STORAGE_DEBUG`: Enable debug logging
163+
- `KCI_STORAGE_CONFIG`: Override default config file path

0 commit comments

Comments
 (0)