Skip to content

Commit cc98168

Browse files
authored
RHIDP-12949 Llama stack library mode changes (#2742)
* llama stack vector_store operations in lightspeed. Enhancing URL fetch securities. Update request/response schemas Signed-off-by: Lucas <lyoon@redhat.com> * migrating from llama stack server changes Signed-off-by: Lucas <lyoon@redhat.com> * code cleanup Signed-off-by: Lucas <lyoon@redhat.com> * cleanup Signed-off-by: Lucas <lyoon@redhat.com> * Notebooks -> notebooks Signed-off-by: Lucas <lyoon@redhat.com> * Notebooks -> notebooks Signed-off-by: Lucas <lyoon@redhat.com> --------- Signed-off-by: Lucas <lyoon@redhat.com>
1 parent 7f580a5 commit cc98168

25 files changed

Lines changed: 1934 additions & 1151 deletions
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@red-hat-developer-hub/backstage-plugin-lightspeed-backend': minor
3+
'@red-hat-developer-hub/backstage-plugin-lightspeed': minor
4+
---
5+
6+
Migrated AI Notebooks from direct Llama Stack server to Lightspeed-Core integration

workspaces/lightspeed/app-config.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,15 @@ organization:
1818

1919
# Disable AI Notebooks feature by default
2020
lightspeed:
21-
aiNotebooks:
21+
notebooks:
2222
enabled: false
23+
queryDefaults:
24+
model: redhataillama-31-8b-instruct
25+
provider_id: vllm
26+
sessionDefaults:
27+
provider_id: notebooks
28+
embedding_model: ${LLAMA_STACK_EMBEDDING_MODEL}
29+
embedding_dimension: 768
2330

2431
backend:
2532
# Used for enabling authentication, secret is shared by all backend plugins

workspaces/lightspeed/plugins/lightspeed-backend/README.md

Lines changed: 62 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,10 @@ For user-facing feature documentation, see the [Lightspeed Frontend README](../l
8989

9090
#### Prerequisites
9191

92-
AI Notebooks requires a **Llama Stack service** to be running. Llama Stack provides the vector database, embeddings, and RAG capabilities.
92+
AI Notebooks requires:
93+
94+
- **Lightspeed Core service** to be running (provides the backend API proxy)
95+
- **Llama Stack service** to be accessible from Lightspeed Core (provides vector database, embeddings, and RAG capabilities)
9396

9497
For Llama Stack setup and configuration, refer to the [Llama Stack documentation](https://github.com/llamastack/llama-stack).
9598

@@ -99,21 +102,23 @@ To enable AI Notebooks, add the following configuration to your `app-config.yaml
99102

100103
```yaml
101104
lightspeed:
102-
aiNotebooks:
103-
enabled: true # Enable AI Notebooks feature (default: false)
105+
servicePort: 8080 # Optional: Lightspeed Core service port (default: 8080)
104106
105-
# Required when enabled: Llama Stack service configuration
106-
llamaStack:
107-
port: 8321 # Llama Stack API endpoint (required, commonly 8321)
107+
notebooks:
108+
enabled: false # Enable AI Notebooks feature (default: false)
108109
109-
# Optional embedding configuration
110-
embeddingModel: sentence-transformers/nomic-ai/nomic-embed-text-v1.5 # (default shown)
111-
embeddingDimension: 768 # Embedding vector dimension (default: 768)
112-
vectorIo:
113-
providerId: rhdh-docs # Vector store provider ID (default: rhdh-docs)
110+
# Required: Query defaults for RAG queries
111+
# Both model and provider_id must be configured together
112+
queryDefaults:
113+
model: llama3.1-8b-instruct # Model to use for answering queries
114+
provider_id: ollama # AI provider for the query model
114115
115-
# Optional: File processing timeout (default: 30000ms = 30 seconds)
116-
fileProcessingTimeoutMs: 30000
116+
# Required: Session defaults for creating vector stores
117+
# All three fields are required when Notebooks is enabled
118+
sessionDefaults:
119+
provider_id: notebooks # Vector store provider ID (must match Llama Stack config)
120+
embedding_model: sentence-transformers/all-mpnet-base-v2 # Model for generating embeddings
121+
embedding_dimension: 768 # Embedding vector dimension (must match model output)
117122
118123
# Optional: Chunking strategy for document processing
119124
chunkingStrategy:
@@ -125,15 +130,37 @@ lightspeed:
125130

126131
**Configuration Options**:
127132

128-
- **`enabled`**: Enable or disable the AI Notebooks feature (default: `false`)
129-
- **`llamaStack.port`**: Port of the Llama Stack service (default: `8321`)
130-
- **`llamaStack.embeddingModel`**: Model used for generating embeddings (default: `sentence-transformers/nomic-ai/nomic-embed-text-v1.5`)
131-
- **`llamaStack.embeddingDimension`**: Dimension of embedding vectors (default: `768`)
132-
- **`llamaStack.vectorIo.providerId`**: Vector store provider in Llama Stack config (default: `rhdh-docs`)
133-
- **`fileProcessingTimeoutMs`**: Timeout for file processing in milliseconds (default: `30000`)
134-
- **`chunkingStrategy.type`**: Document chunking strategy - `auto` (automatic) or `static` (fixed size) (default: `auto`)
135-
- **`chunkingStrategy.maxChunkSizeTokens`**: Maximum chunk size in tokens for static chunking (default: `512`)
136-
- **`chunkingStrategy.chunkOverlapTokens`**: Token overlap between chunks for static chunking (default: `50`)
133+
**Core Settings**:
134+
135+
- **`lightspeed.servicePort`** _(optional)_: Port where Lightspeed Core service is running (default: `8080`). The backend connects to Lightspeed Core at `http://0.0.0.0:{servicePort}` to proxy vector store operations.
136+
137+
**Notebooks Settings**:
138+
139+
- **`Notebooks.enabled`** _(optional)_: Enable or disable the AI Notebooks feature (default: `false`)
140+
141+
**Query Defaults** _(required when enabled)_:
142+
143+
- **`queryDefaults.model`** _(required)_: The LLM model to use for answering RAG queries. Must be available in the configured provider.
144+
- **`queryDefaults.provider_id`** _(required)_: The AI provider identifier for the query model (e.g., `ollama`, `vllm`). Both `model` and `provider_id` must be configured together.
145+
146+
**Session Defaults** _(required when enabled)_:
147+
148+
- **`sessionDefaults.provider_id`** _(required)_: Vector store provider identifier. Must match a provider configured in your Llama Stack instance (e.g., `notebooks`, `chromadb`). This determines where document embeddings are stored.
149+
- **`sessionDefaults.embedding_model`** _(required)_: The embedding model to use for converting documents to vectors (e.g., `sentence-transformers/all-mpnet-base-v2`). Must be available in Llama Stack.
150+
- **`sessionDefaults.embedding_dimension`** _(required)_: Dimension of the embedding vectors produced by the embedding model. Must match the model's output dimension (commonly `768`, `384`, or `1536`).
151+
152+
**Chunking Strategy** _(optional)_:
153+
154+
- **`chunkingStrategy.type`** _(optional)_: Document chunking strategy - `auto` (automatic, default) or `static` (fixed size)
155+
- **`chunkingStrategy.maxChunkSizeTokens`** _(optional)_: Maximum chunk size in tokens for static chunking (default: `512`)
156+
- **`chunkingStrategy.chunkOverlapTokens`** _(optional)_: Token overlap between chunks for static chunking (default: `50`)
157+
158+
**Where to Find These Values**:
159+
160+
- **Provider IDs**: Check your Llama Stack configuration file for configured providers (both for models and vector stores)
161+
- **Model names**: Available models are listed in your Llama Stack provider configuration
162+
- **Embedding dimensions**: Refer to the embedding model's documentation (e.g., `all-mpnet-base-v2` outputs 768 dimensions)
163+
- **Lightspeed Core port**: Check your Lightspeed Core service deployment configuration
137164

138165
#### API Endpoints
139166

@@ -144,20 +171,26 @@ When enabled, AI Notebooks exposes the following REST API endpoints:
144171

145172
- **Sessions**:
146173
- `POST /lightspeed/ai-notebooks/v1/sessions` - Create a new session
147-
- `GET /lightspeed/ai-notebooks/v1/sessions` - List all sessions
148-
- `GET /lightspeed/ai-notebooks/v1/sessions/:sessionId` - Get session details
149-
- `PUT /lightspeed/ai-notebooks/v1/sessions/:sessionId` - Update session
174+
- `GET /lightspeed/ai-notebooks/v1/sessions` - List all sessions for the current user
175+
- `PUT /lightspeed/ai-notebooks/v1/sessions/:sessionId` - Update session details
150176
- `DELETE /lightspeed/ai-notebooks/v1/sessions/:sessionId` - Delete session
151177

152178
- **Documents**:
153-
- `POST /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents/upload` - Upload document
154-
- `GET /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents` - List documents
155-
- `PUT /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents/:documentId` - Update document
156-
- `DELETE /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents/:documentId` - Delete document
179+
- `PUT /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents` - Upload or update a document (multipart/form-data)
180+
- `GET /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents` - List all documents in a session
181+
- `GET /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents/:documentId/status` - Get document processing status
182+
- `DELETE /lightspeed/ai-notebooks/v1/sessions/:sessionId/documents/:documentId` - Delete a document
157183

158184
- **Queries**:
159185
- `POST /lightspeed/ai-notebooks/v1/sessions/:sessionId/query` - Query documents with RAG
160186

187+
**Notes**:
188+
189+
- All endpoints require authentication (user context is automatically provided by Backstage)
190+
- All `/v1/*` endpoints require the `lightspeed.notebooks.use` permission
191+
- Document endpoints verify session ownership before allowing operations
192+
- `documentId` in paths is the document title (URL-encoded for special characters)
193+
161194
#### Permission Framework Support for AI Notebooks
162195

163196
When RBAC is enabled, users need the following permission to use AI Notebooks:
Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
/*
2+
* Copyright Red Hat, Inc.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
import { http, HttpResponse, type HttpHandler } from 'msw';
18+
19+
// Use port 7007 for tests (matches notebooksRouter.test.ts config)
20+
const TEST_LIGHTSPEED_SERVICE_PORT = 7007;
21+
export const LIGHTSPEED_CORE_ADDR = `http://0.0.0.0:${TEST_LIGHTSPEED_SERVICE_PORT}`;
22+
23+
// Mock session data
24+
export const mockSession1 = {
25+
session_id: 'session-1',
26+
name: 'Test Session 1',
27+
user_id: 'user:default/guest',
28+
description: 'Test description',
29+
created_at: '2024-01-01T00:00:00.000Z',
30+
updated_at: '2024-01-01T00:00:00.000Z',
31+
metadata: {
32+
embedding_model: 'sentence-transformers/nomic-ai/nomic-embed-text-v1.5',
33+
embedding_dimension: 768,
34+
provider_id: 'notebooks',
35+
conversation_id: null,
36+
},
37+
};
38+
39+
export const mockSession2 = {
40+
session_id: 'session-2',
41+
name: 'Test Session 2',
42+
user_id: 'user:default/guest',
43+
description: 'Another test',
44+
created_at: '2024-01-02T00:00:00.000Z',
45+
updated_at: '2024-01-02T00:00:00.000Z',
46+
metadata: {
47+
embedding_model: 'sentence-transformers/nomic-ai/nomic-embed-text-v1.5',
48+
embedding_dimension: 768,
49+
provider_id: 'notebooks',
50+
conversation_id: 'conv-1',
51+
},
52+
};
53+
54+
export const mockFile1 = {
55+
id: 'file-1',
56+
created_at: 1704067200,
57+
status: 'completed' as const,
58+
attributes: {
59+
document_id: 'test-document',
60+
user_id: 'user:default/guest',
61+
title: 'Test Document',
62+
session_id: 'session-1',
63+
source_type: 'text',
64+
created_at: '2024-01-01T00:00:00.000Z',
65+
},
66+
};
67+
68+
// In-memory storage for tests
69+
const vectorStores = new Map<string, any>();
70+
const files = new Map<string, any>();
71+
const vectorStoreFiles = new Map<string, any[]>();
72+
73+
export function resetMockStorage() {
74+
vectorStores.clear();
75+
files.clear();
76+
vectorStoreFiles.clear();
77+
}
78+
79+
/**
80+
* MSW handlers for lightspeed-core vector store endpoints
81+
* These mock the endpoints created in lightspeed-core that proxy to llama stack
82+
*/
83+
export const lightspeedCoreHandlers: HttpHandler[] = [
84+
// Create vector store
85+
http.post(`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores`, async ({ request }) => {
86+
const body = (await request.json()) as any;
87+
const id = `vs-${Date.now()}`;
88+
const vectorStore = {
89+
id,
90+
name: body.name,
91+
embedding_model: body.embedding_model,
92+
embedding_dimension: body.embedding_dimension,
93+
provider_id: body.provider_id,
94+
metadata: body.metadata || {},
95+
};
96+
vectorStores.set(id, vectorStore);
97+
vectorStoreFiles.set(id, []);
98+
return HttpResponse.json(vectorStore);
99+
}),
100+
101+
// Get vector store
102+
http.get(`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id`, ({ params }) => {
103+
const { id } = params;
104+
const vectorStore = vectorStores.get(id as string);
105+
if (!vectorStore) {
106+
return HttpResponse.json(
107+
{ detail: 'Vector store not found' },
108+
{ status: 404 },
109+
);
110+
}
111+
return HttpResponse.json(vectorStore);
112+
}),
113+
114+
// Update vector store
115+
http.put(
116+
`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id`,
117+
async ({ params, request }) => {
118+
const { id } = params;
119+
const vectorStore = vectorStores.get(id as string);
120+
if (!vectorStore) {
121+
return HttpResponse.json(
122+
{ detail: 'Vector store not found' },
123+
{ status: 404 },
124+
);
125+
}
126+
const body = (await request.json()) as any;
127+
const updated = {
128+
...vectorStore,
129+
metadata: body.metadata || vectorStore.metadata,
130+
};
131+
vectorStores.set(id as string, updated);
132+
return HttpResponse.json(updated);
133+
},
134+
),
135+
136+
// Delete vector store
137+
http.delete(`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id`, ({ params }) => {
138+
const { id } = params;
139+
if (!vectorStores.has(id as string)) {
140+
return HttpResponse.json(
141+
{ detail: 'Vector store not found' },
142+
{ status: 404 },
143+
);
144+
}
145+
vectorStores.delete(id as string);
146+
vectorStoreFiles.delete(id as string);
147+
return HttpResponse.json({ deleted: true });
148+
}),
149+
150+
// List vector stores
151+
http.get(`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores`, () => {
152+
const data = Array.from(vectorStores.values());
153+
return HttpResponse.json({ data });
154+
}),
155+
156+
// Upload file
157+
http.post(`${LIGHTSPEED_CORE_ADDR}/v1/files`, async () => {
158+
const fileId = `file-${Date.now()}`;
159+
const file = {
160+
id: fileId,
161+
created_at: Date.now(),
162+
purpose: 'assistants',
163+
};
164+
files.set(fileId, file);
165+
return HttpResponse.json(file);
166+
}),
167+
168+
// Add file to vector store
169+
http.post(
170+
`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id/files`,
171+
async ({ params, request }) => {
172+
const { id } = params;
173+
const vectorStore = vectorStores.get(id as string);
174+
if (!vectorStore) {
175+
return HttpResponse.json(
176+
{ detail: 'Vector store not found' },
177+
{ status: 404 },
178+
);
179+
}
180+
181+
const body = (await request.json()) as any;
182+
const vectorStoreFile = {
183+
id: body.file_id,
184+
status: 'completed' as const,
185+
created_at: Date.now(),
186+
chunks_count: 1,
187+
attributes: body.attributes || {},
188+
};
189+
190+
const storeFiles = vectorStoreFiles.get(id as string) || [];
191+
storeFiles.push(vectorStoreFile);
192+
vectorStoreFiles.set(id as string, storeFiles);
193+
194+
return HttpResponse.json(vectorStoreFile);
195+
},
196+
),
197+
198+
// List files in vector store
199+
http.get(
200+
`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id/files`,
201+
({ params }) => {
202+
const { id } = params;
203+
const storeFiles = vectorStoreFiles.get(id as string) || [];
204+
return HttpResponse.json({ data: storeFiles });
205+
},
206+
),
207+
208+
// Get file from vector store
209+
http.get(
210+
`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id/files/:fileId`,
211+
({ params }) => {
212+
const { id, fileId } = params;
213+
const storeFiles = vectorStoreFiles.get(id as string) || [];
214+
const file = storeFiles.find(f => f.id === fileId);
215+
if (!file) {
216+
return HttpResponse.json({ detail: 'File not found' }, { status: 404 });
217+
}
218+
return HttpResponse.json(file);
219+
},
220+
),
221+
222+
// Delete file from vector store
223+
http.delete(
224+
`${LIGHTSPEED_CORE_ADDR}/v1/vector-stores/:id/files/:fileId`,
225+
({ params }) => {
226+
const { id, fileId } = params;
227+
const storeFiles = vectorStoreFiles.get(id as string) || [];
228+
const fileIndex = storeFiles.findIndex(f => f.id === fileId);
229+
if (fileIndex === -1) {
230+
return HttpResponse.json({ detail: 'File not found' }, { status: 404 });
231+
}
232+
storeFiles.splice(fileIndex, 1);
233+
vectorStoreFiles.set(id as string, storeFiles);
234+
return HttpResponse.json({ deleted: true });
235+
},
236+
),
237+
];

0 commit comments

Comments
 (0)