Skip to content

Commit 1d45e52

Browse files
nuclearcatnuclearcat
authored
Merge pull request #13 from nuclearcat/docs-update
Docs update
2 parents 95ac486 + 1f216b5 commit 1d45e52

2 files changed

Lines changed: 171 additions & 29 deletions

File tree

‎README.md‎

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,44 @@
11
# KernelCI Storage Server
22

33
This is a simple storage server that supports file upload and download, with token based authentication.
4-
It supports multiple backends, currently only Azure Blob is supported, to provide user transparent storage.
4+
It supports multiple backends: Azure Blob Storage and local filesystem, to provide user transparent storage.
55
It caches the files in a local directory and serves them from there.
66
Range requests are supported, but only for start offset, end limit is not implemented yet.
7+
Files are protected by upload locking to prevent concurrent uploads to the same path.
78

89
## Configuration
910

1011
The server is configured using toml configuration file, the default configuration file is `config.toml`.
1112

13+
### Azure Backend Configuration
14+
1215
```toml
16+
driver="azure" # Optional, defaults to "azure"
1317
jwt_secret="JWT_SECRET"
1418
[azure]
1519
account=""
1620
key=""
1721
container=""
1822
sastoken=""
23+
24+
# User upload permissions (optional)
25+
[[users]]
26+
name = "user@email.com"
27+
prefixes = ["/allowed/path"]
28+
```
29+
30+
### Local Filesystem Backend Configuration
31+
32+
```toml
33+
driver="local"
34+
jwt_secret="JWT_SECRET"
35+
[local]
36+
storage_path="./storage" # Optional, defaults to "./storage"
37+
38+
# User upload permissions (optional)
39+
[[users]]
40+
name = "user@email.com"
41+
prefixes = ["/allowed/path"]
1942
```
2043

2144
## Creating user tokens
@@ -24,23 +47,53 @@ The server uses JWT token based authentication. The token is passed in the `Auth
2447
JWT secret is configured in the `config.toml` file.
2548

2649
```bash
27-
./kernelci-storage -generate_jwt_token user@email.com
50+
# Generate JWT secret
51+
./kernelci-storage --generate-jwt-secret
52+
53+
# Generate JWT token for a user
54+
./kernelci-storage --generate-jwt-token user@email.com
2855
```
29-
This will generate a JWT token for the user.
56+
The first command generates a JWT secret for your configuration, the second generates a token for the specified user.
3057

3158
### Testing Token Validity
3259

3360
You can verify if a token is valid using the `/v1/checkauth` endpoint:
3461

3562
```bash
36-
curl -X GET http://localhost:3000/v1/checkauth \
63+
curl -X GET https://localhost:3000/v1/checkauth \
3764
-H "Authorization: Bearer <JWT_TOKEN>"
3865
```
3966

4067
**Responses:**
4168
- `200 OK` with body `Authorized: user@email.com` - Token is valid
4269
- `401 Unauthorized` with body `Unauthorized` - Token is invalid or missing
4370

71+
## Environment Variables
72+
73+
- `KCI_STORAGE_CONFIG` - Override config file path (defaults to config.toml)
74+
- `STORAGE_DEBUG` - Enable debug logging
75+
4476
## API
4577

46-
See [docs](docs/) for the API documentation.
78+
### Endpoints
79+
80+
- `GET /` - Server status
81+
- `POST /v1/file` or `POST /upload` - File upload (requires JWT)
82+
- `GET /*filepath` - File download (public, supports range requests)
83+
- `GET /v1/checkauth` - Validate JWT token
84+
- `GET /v1/list` - List all files (public)
85+
- `GET /metrics` - Prometheus metrics endpoint
86+
87+
The server supports large file uploads up to 4GB and includes upload conflict protection to prevent concurrent uploads to the same file path. Downloads have a 30-second timeout for acquiring file locks.
88+
89+
### Metrics
90+
91+
The `/metrics` endpoint provides Prometheus-compatible metrics including:
92+
- `storage_free_space` - Available disk space
93+
- `storage_total_space` - Total disk space
94+
95+
Both metrics include hostname, diskname, and mount_point labels.
96+
97+
## API Documentation
98+
99+
See [docs](docs/) for detailed API documentation.

‎docs/_index.md‎

Lines changed: 113 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -4,47 +4,101 @@ date: 2025-08-06
44
description: KernelCI's Storage API documentation
55
---
66

7-
We have simple storage server that supports file upload and download, with JWT token based authentication.
7+
We have a simple Proxy server that supports file upload and download, with JWT token-based authentication. The server supports multiple storage backends, file caching, range requests, and provides Prometheus metrics.
88

99
## API Endpoints
1010

11-
The API is quite simple with upload through a `/upload` endpoint and download through a GET to the file url.
12-
1311
### Upload
1412

15-
`POST /upload`
13+
**`POST /upload`** or **`POST /v1/file`**
1614

17-
Upload a file to the server.
15+
Upload a file to the server. Requires JWT authentication.
1816

19-
field path: the path to store the file in the server.
20-
field file0: the file to upload.
17+
**Form fields:**
18+
- `path`: The path to store the file in the server
19+
- `file0`: The file to upload
2120

2221
### Download
2322

24-
`GET /path/to/file`
23+
**`GET /*filepath`**
24+
25+
Download a file from the server. Supports range requests for partial downloads.
26+
27+
### Authentication Check
28+
29+
**`GET /v1/checkauth`**
30+
31+
Validate a JWT token. Returns the authenticated user's email if valid.
32+
33+
### List Files
34+
35+
**`GET /v1/list`**
36+
37+
List all files in the storage (public endpoint).
2538

26-
Download a file from the server.
39+
### Server Status
40+
41+
**`GET /`**
42+
43+
Returns server status: "KernelCI Proxy Server"
44+
45+
### Metrics
46+
47+
**`GET /metrics`**
48+
49+
Prometheus metrics endpoint showing storage space and system information.
2750

2851
### Request a token
2952

3053
Ask the kernelci-sysadmin team for a token.
3154

32-
### Testing upload and download using curl
55+
### Testing with curl
3356

3457
```bash
35-
# Upload
36-
curl -X POST http://files.kernelci.org/upload \
58+
# Upload a file
59+
curl -X POST https://files.kernelci.org/upload \
60+
-H "Authorization: Bearer <JWT_TOKEN>" \
61+
-F "path=testfolder" \
62+
-F "file0=@local_folder/local_file"
63+
64+
# Alternative upload endpoint
65+
curl -X POST https://files.kernelci.org/v1/file \
3766
-H "Authorization: Bearer <JWT_TOKEN>" \
3867
-F "path=testfolder" \
3968
-F "file0=@local_folder/local_file"
4069

41-
# File will be uploaded to Azure Blob Storage as testfolder/local_folder/local_file
70+
# File will be uploaded as testfolder/local_folder/local_file
71+
72+
# Download a file
73+
curl https://files.kernelci.org/testfolder/local_folder/local_file
74+
75+
# Download with range request (partial content)
76+
curl -H "Range: bytes=0-1023" https://files.kernelci.org/testfolder/local_folder/local_file
77+
78+
# Check authentication
79+
curl -X GET https://files.kernelci.org/v1/checkauth \
80+
-H "Authorization: Bearer <JWT_TOKEN>"
4281

43-
# Download
44-
curl http://files.kernelci.org/testfolder/local_folder/local_file
82+
# List all files
83+
curl https://files.kernelci.org/v1/list
84+
85+
# Get metrics
86+
curl https://files.kernelci.org/metrics
4587
```
4688

47-
## Principle of operation
89+
## Features
90+
91+
- **JWT Authentication**: Secure token-based authentication for uploads
92+
- **Multiple Storage Backends**: Currently supports Azure Blob Storage with extensible driver architecture
93+
- **Local Caching**: Files are cached locally with automatic cleanup when disk space is low
94+
- **Range Request Support**: Partial content downloads using HTTP range requests
95+
- **File Locking**: Prevents concurrent uploads to the same file path
96+
- **Prometheus Metrics**: System monitoring and metrics collection
97+
- **User Access Control**: Configurable path-based upload permissions per user
98+
- **Content-Type Detection**: Automatic MIME type detection based on file extensions
99+
- **HTTP Caching Headers**: ETag and Last-Modified support for efficient caching
100+
101+
## Architecture
48102

49103
```mermaid
50104
sequenceDiagram
@@ -54,21 +108,56 @@ sequenceDiagram
54108
participant Azure Blob Storage
55109
56110
%% Upload Flow
57-
User->>Proxy Server: Upload File (POST Request)
58-
Proxy Server->>Azure Blob Storage: Store File in Cloud
59-
Azure Blob Storage-->>Proxy Server: Confirmation
60-
Proxy Server-->>User: Upload Successful
111+
User->>Proxy Server: Upload File (POST /upload or /v1/file)
112+
Note over Proxy Server: JWT Authentication & Path Permissions Check
113+
Proxy Server->>Proxy Server: Acquire File Lock
114+
Proxy Server->>Azure Blob Storage: Store File with Chunked Upload
115+
Proxy Server->>Local Cache: Cache File Locally
116+
Azure Blob Storage-->>Proxy Server: Upload Confirmation
117+
Proxy Server-->>User: Upload Successful (200 OK)
61118
62119
%% Download Flow
63-
User->>Proxy Server: Request File (GET Request)
120+
User->>Proxy Server: Request File (GET /*filepath)
121+
Proxy Server->>Proxy Server: Acquire File Lock (with timeout)
64122
Proxy Server->>Local Cache: Check if File Exists
65123
alt File Found in Cache
66124
Local Cache-->>Proxy Server: Return Cached File
67-
Proxy Server-->>User: Send File
125+
Proxy Server-->>User: Send File (with Range Support)
68126
else File Not in Cache
69127
Proxy Server->>Azure Blob Storage: Fetch File from Cloud
70128
Azure Blob Storage-->>Proxy Server: Send File Data
71-
Proxy Server->>Local Cache: Save File Locally
72-
Proxy Server-->>User: Send File
129+
Proxy Server->>Local Cache: Save File Locally (SHA-512 based filename)
130+
Proxy Server-->>User: Send File (with Range Support)
73131
end
74132
```
133+
134+
## Configuration
135+
136+
The server supports configuration through a `config.toml` file with the following structure:
137+
138+
```toml
139+
# Storage backend driver (defaults to "azure")
140+
driver = "azure"
141+
jwt_secret = "your-jwt-secret-here"
142+
143+
# Azure Blob Storage configuration
144+
[azure]
145+
account = "your-storage-account"
146+
key = "your-storage-key"
147+
container = "your-container-name"
148+
sastoken = "?sv=2022-11-02&ss=b..."
149+
150+
# User access control
151+
[[users]]
152+
name = "user@example.com"
153+
prefixes = ["/allowed/path1", "/allowed/path2"]
154+
155+
[[users]]
156+
name = "admin@example.com"
157+
prefixes = [""] # Empty prefix allows access to all paths
158+
```
159+
160+
## Environment Variables
161+
162+
- `STORAGE_DEBUG`: Enable debug logging
163+
- `KCI_STORAGE_CONFIG`: Override default config file path

0 commit comments

Comments
 (0)