feat(q-dev): add logging data ingestion and enrich Kiro dashboards (#8767)

* feat(q-dev): add logging data ingestion and enrich Kiro dashboards

Add support for ingesting S3 logging data (GenerateAssistantResponse and
GenerateCompletions events) into new database tables, and enrich all three
Kiro Grafana dashboards with additional metrics.

Changes:
- New models: QDevChatLog and QDevCompletionLog for logging event data
- New extractor: s3_logging_extractor.go parses JSON.gz logging files
- Updated S3 collector to also handle .json.gz files
- Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions)
- New dashboard: "Kiro AI Activity Insights" with 10 panels including
  model usage distribution, active hours, conversation depth, feature
  adoption (Steering/Spec), file type usage, and prompt/response trends
- Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration,
  and Dev (Agentic) metric panels
- Fixed "Kiro Usage Dashboard" per-user table to sort by user_id
- Migration script for new tables

* fix(q-dev): use separate base path for logging S3 prefixes

Logging data lives under a different S3 prefix ("logging/") than user
report data ("user-report/"). Add LoggingBasePath option (defaults to
"logging") so logging prefixes are constructed correctly.

* fix(q-dev): auto-scan logging path without extra config

Kiro exports to two well-known S3 prefixes in the same bucket:
- user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports)
- logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs)

When AccountId is set, automatically scan both paths. The "logging"
prefix is hardcoded since it's a standard Kiro export convention.
No additional configuration needed.

* fix(q-dev): update scope tooltip to mention logging data scanning

* fix(q-dev): fix scope ID routing and CSV/JSON file separation

Three fixes:
1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope
   IDs containing "/" (e.g. "034362076319/2026") work in URL paths
2. CSV extractor now filters for .csv files only, preventing it from
   trying to parse .json.gz logging files as CSV
3. Frontend scope API calls now encodeURIComponent(scopeId) for safe
   URL encoding

* fix(q-dev): resolve *scopeId route conflict with dispatcher pattern

The catch-all *scopeId route conflicts with *scopeId/latest-sync-state.
Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a
GetScopeDispatcher that checks for /latest-sync-state suffix and
dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId.

* fix(q-dev): use URL-safe scope ID format (underscore separator)

Scope IDs like "034362076319/2026" break URL routing because "/" is a
path separator. Change ID format to "034362076319_2026" (underscore)
when AccountId is set. The Prefix field still uses "/" for S3 path
matching. Revert to standard :scopeId routes since IDs are now safe.

Note: existing scopes need to be recreated after this change.

* fix(q-dev): use NoPKModel instead of Model in archived logging models

archived.Model only has ID+timestamps, missing RawDataOrigin fields
(_raw_data_params etc.) that common.NoPKModel includes. This caused
"Unknown column '_raw_data_params'" errors at runtime.

* fix(q-dev): fix GROUP BY in per-user table to merge display_name variants

Remove display_name from GROUP BY so same user_id with different
display_name values gets merged. Use MAX(display_name) in SELECT.

* fix(q-dev): normalize logging user IDs to match CSV short UUID format

Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report
uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to
one user_id across both data sources.

* fix(q-dev): normalize user IDs in CSV extractors and sort table DESC

Apply normalizeUserId to both createUserReportData and
createUserDataWithDisplayName so user_report CSV data also strips
the "d-{directoryId}." prefix. Change per-user table sort to
ORDER BY user_id DESC.

* style(q-dev): fix gofmt formatting in chat_log models

* perf(q-dev): parallelize logging S3 downloads and batch DB writes

Optimize logging extractor performance:
- 10 goroutine workers for parallel S3 file downloads
- Batch 50 files per DB transaction instead of 1-per-file
- sync.Map cache for display name resolution (avoid repeated IAM calls)
- Parse records in memory during download, write all at once

This should improve throughput from ~1.5 files/sec to ~15+ files/sec
for typical logging file sizes.

* fix(q-dev): check tx.Rollback error return to satisfy errcheck lint

* feat(q-dev): add per-user model usage table and models column

Add "Per-User Model Usage" table (panel 11) showing each user's
request count and avg prompt/response length per model_id. Also add
"Models Used" column to the Per-User Activity table.

* fix(q-dev): remove per-user model usage table, keep models column only

* feat(q-dev): add Kiro Executive Dashboard with cross-source analytics

New dashboard "Kiro Executive Dashboard" with 12 panels covering:
- KPIs: WAU, credits efficiency, acceptance rate, steering adoption
- Trends: weekly active users, new vs returning users
- Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec
- Cost: credits pace vs projected monthly, idle power users
- Quality: acceptance rate trends, code review findings, test generation
- Efficiency: per-user productivity table with credits/line ratio

Correlates data across user_report (credits), user_data (code metrics),
and chat_log (interaction patterns) for holistic Kiro usage insights.

* fix(q-dev): fix pie charts to show per-row slices instead of single total

Set reduceOptions.values=true so Grafana treats each SQL result row as
a separate pie slice. Fixes Model Usage Distribution, File Type Usage,
Kiro Feature Adoption, and Active File Types pie charts.

* fix(q-dev): cast Hour to string for Active Hours bar chart x-axis

* fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues

1. qdev_user_report Panel 4 (Subscription Tier Distribution): set
   reduceOptions.values=true to show per-tier slices
2. qdev_user_data Panel 6 (User Interactions): remove display_name
   from GROUP BY, use MAX(display_name) to merge same user

* fix(q-dev): prevent data inflation in user_report JOIN user_data

user_report has multiple rows per (user_id, date) due to client_type
(KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date).
A direct JOIN causes user_data metrics to be counted multiple times.

Fix: pre-aggregate user_report by (user_id, date) in a subquery before
joining, so the JOIN is always 1:1.

Affects: Credits Efficiency stat and User Productivity table.

Author: warren830

backend/plugins/q_dev/api/s3_slice_api.go

@@ -60,62 +60,21 @@ func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, er
 }
 
 // GetScope returns a single scope record
-// @Summary get a Q Developer scope
-// @Description get a Q Developer scope
-// @Tags plugins/q_dev
-// @Param connectionId path int true "connection ID"
-// @Param scopeId path string true "scope id"
-// @Param blueprints query bool false "include blueprint references"
-// @Success 200  {object} ScopeDetail
-// @Failure 400  {object} shared.ApiBody "Bad Request"
-// @Failure 500  {object} shared.ApiBody "Internal Error"
-// @Router /plugins/q_dev/connections/{connectionId}/scopes/{scopeId} [GET]
 func GetScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
 	return dsHelper.ScopeApi.GetScopeDetail(input)
 }
 
 // PatchScope updates a scope record
-// @Summary patch a Q Developer scope
-// @Description patch a Q Developer scope
-// @Tags plugins/q_dev
-// @Accept application/json
-// @Param connectionId path int true "connection ID"
-// @Param scopeId path string true "scope id"
-// @Param scope body models.QDevS3Slice true "json"
-// @Success 200  {object} models.QDevS3Slice
-// @Failure 400  {object} shared.ApiBody "Bad Request"
-// @Failure 500  {object} shared.ApiBody "Internal Error"
-// @Router /plugins/q_dev/connections/{connectionId}/scopes/{scopeId} [PATCH]
 func PatchScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
 	return dsHelper.ScopeApi.Patch(input)
 }
 
 // DeleteScope removes a scope and optionally associated data.
-// @Summary delete a Q Developer scope
-// @Description delete Q Developer scope data
-// @Tags plugins/q_dev
-// @Param connectionId path int true "connection ID"
-// @Param scopeId path string true "scope id"
-// @Param delete_data_only query bool false "Only delete scope data"
-// @Success 200
-// @Failure 400  {object} shared.ApiBody "Bad Request"
-// @Failure 409  {object} srvhelper.DsRefs "References exist to this scope"
-// @Failure 500  {object} shared.ApiBody "Internal Error"
-// @Router /plugins/q_dev/connections/{connectionId}/scopes/{scopeId} [DELETE]
 func DeleteScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
 	return dsHelper.ScopeApi.Delete(input)
 }
 
 // GetScopeLatestSyncState returns scope sync state info
-// @Summary latest sync state for a Q Developer scope
-// @Description get latest sync state for a Q Developer scope
-// @Tags plugins/q_dev
-// @Param connectionId path int true "connection ID"
-// @Param scopeId path string true "scope id"
-// @Success 200
-// @Failure 400  {object} shared.ApiBody "Bad Request"
-// @Failure 500  {object} shared.ApiBody "Internal Error"
-// @Router /plugins/q_dev/connections/{connectionId}/scopes/{scopeId}/latest-sync-state [GET]
 func GetScopeLatestSyncState(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
 	return dsHelper.ScopeApi.GetScopeLatestSyncState(input)
 }

backend/plugins/q_dev/impl/impl.go

@@ -58,6 +58,8 @@ func (p QDev) GetTablesInfo() []dal.Tabler {
 		&models.QDevS3FileMeta{},
 		&models.QDevS3Slice{},
 		&models.QDevUserReport{},
+		&models.QDevChatLog{},
+		&models.QDevCompletionLog{},
 	}
 }
 
@@ -85,6 +87,7 @@ func (p QDev) SubTaskMetas() []plugin.SubTaskMeta {
 	return []plugin.SubTaskMeta{
 		tasks.CollectQDevS3FilesMeta,
 		tasks.ExtractQDevS3DataMeta,
+		tasks.ExtractQDevLoggingDataMeta,
 	}
 }
 
@@ -127,10 +130,21 @@ func (p QDev) PrepareTaskData(taskCtx plugin.TaskContext, options map[string]int
 		if op.Month != nil {
 			timePart = fmt.Sprintf("%04d/%02d", op.Year, *op.Month)
 		}
-		base := fmt.Sprintf("%s/AWSLogs/%s/KiroLogs", op.BasePath, op.AccountId)
+		// Kiro exports data to two well-known S3 prefixes:
+		//   {basePath}/AWSLogs/{accountId}/KiroLogs/  — user report CSVs
+		//   logging/AWSLogs/{accountId}/KiroLogs/      — interaction logs (JSON.gz)
+		// When basePath is empty, default to "user-report" for CSV data.
+		reportBase := op.BasePath
+		if reportBase == "" {
+			reportBase = "user-report"
+		}
+		csvBase := fmt.Sprintf("%s/AWSLogs/%s/KiroLogs", reportBase, op.AccountId)
+		logBase := fmt.Sprintf("logging/AWSLogs/%s/KiroLogs", op.AccountId)
 		s3Prefixes = []string{
-			fmt.Sprintf("%s/by_user_analytic/%s/%s", base, region, timePart),
-			fmt.Sprintf("%s/user_report/%s/%s", base, region, timePart),
+			fmt.Sprintf("%s/by_user_analytic/%s/%s", csvBase, region, timePart),
+			fmt.Sprintf("%s/user_report/%s/%s", csvBase, region, timePart),
+			fmt.Sprintf("%s/GenerateAssistantResponse/%s/%s", logBase, region, timePart),
+			fmt.Sprintf("%s/GenerateCompletions/%s/%s", logBase, region, timePart),
 		}
 	} else {
 		// Legacy scope: use S3Prefix directly

backend/plugins/q_dev/impl/impl_test.go

@@ -34,11 +34,11 @@ func TestQDev_BasicPluginMethods(t *testing.T) {
 
 	// Test table info
 	tables := plugin.GetTablesInfo()
-	assert.Len(t, tables, 5)
+	assert.Len(t, tables, 7)
 
 	// Test subtask metas
 	subtasks := plugin.SubTaskMetas()
-	assert.Len(t, subtasks, 2)
+	assert.Len(t, subtasks, 3)
 
 	// Test API resources
 	apiResources := plugin.ApiResources()

backend/plugins/q_dev/models/chat_log.go

@@ -0,0 +1,51 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package models
+
+import (
+	"time"
+
+	"github.com/apache/incubator-devlake/core/models/common"
+)
+
+// QDevChatLog stores parsed data from GenerateAssistantResponse logging events
+type QDevChatLog struct {
+	common.NoPKModel
+	ConnectionId        uint64    `gorm:"primaryKey"`
+	ScopeId             string    `gorm:"primaryKey;type:varchar(255)" json:"scopeId"`
+	RequestId           string    `gorm:"primaryKey;type:varchar(255)" json:"requestId"`
+	UserId              string    `gorm:"index;type:varchar(255)" json:"userId"`
+	DisplayName         string    `gorm:"type:varchar(255)" json:"displayName"`
+	Timestamp           time.Time `gorm:"index" json:"timestamp"`
+	ChatTriggerType     string    `gorm:"type:varchar(50)" json:"chatTriggerType"`
+	HasCustomization    bool      `json:"hasCustomization"`
+	ConversationId      string    `gorm:"type:varchar(255)" json:"conversationId"`
+	UtteranceId         string    `gorm:"type:varchar(255)" json:"utteranceId"`
+	ModelId             string    `gorm:"type:varchar(100)" json:"modelId"`
+	PromptLength        int       `json:"promptLength"`
+	ResponseLength      int       `json:"responseLength"`
+	OpenFileCount       int       `json:"openFileCount"`
+	ActiveFileName      string    `gorm:"type:varchar(512)" json:"activeFileName"`
+	ActiveFileExtension string    `gorm:"type:varchar(50)" json:"activeFileExtension"`
+	HasSteering         bool      `json:"hasSteering"`
+	IsSpecMode          bool      `json:"isSpecMode"`
+}
+
+func (QDevChatLog) TableName() string {
+	return "_tool_q_dev_chat_log"
+}

backend/plugins/q_dev/models/completion_log.go

@@ -0,0 +1,43 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package models
+
+import (
+	"time"
+
+	"github.com/apache/incubator-devlake/core/models/common"
+)
+
+// QDevCompletionLog stores parsed data from GenerateCompletions logging events
+type QDevCompletionLog struct {
+	common.NoPKModel
+	ConnectionId     uint64    `gorm:"primaryKey"`
+	ScopeId          string    `gorm:"primaryKey;type:varchar(255)" json:"scopeId"`
+	RequestId        string    `gorm:"primaryKey;type:varchar(255)" json:"requestId"`
+	UserId           string    `gorm:"index;type:varchar(255)" json:"userId"`
+	DisplayName      string    `gorm:"type:varchar(255)" json:"displayName"`
+	Timestamp        time.Time `gorm:"index" json:"timestamp"`
+	FileName         string    `gorm:"type:varchar(512)" json:"fileName"`
+	FileExtension    string    `gorm:"type:varchar(50)" json:"fileExtension"`
+	HasCustomization bool      `json:"hasCustomization"`
+	CompletionsCount int       `json:"completionsCount"`
+}
+
+func (QDevCompletionLog) TableName() string {
+	return "_tool_q_dev_completion_log"
+}

backend/plugins/q_dev/models/migrationscripts/20260314_add_logging_tables.go

@@ -0,0 +1,43 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package migrationscripts
+
+import (
+	"github.com/apache/incubator-devlake/core/context"
+	"github.com/apache/incubator-devlake/core/errors"
+	"github.com/apache/incubator-devlake/helpers/migrationhelper"
+	"github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived"
+)
+
+type addLoggingTables struct{}
+
+func (*addLoggingTables) Up(basicRes context.BasicRes) errors.Error {
+	return migrationhelper.AutoMigrateTables(
+		basicRes,
+		&archived.QDevChatLog{},
+		&archived.QDevCompletionLog{},
+	)
+}
+
+func (*addLoggingTables) Version() uint64 {
+	return 20260314000001
+}
+
+func (*addLoggingTables) Name() string {
+	return "Add chat_log and completion_log tables for Kiro logging data"
+}

backend/plugins/q_dev/models/migrationscripts/archived/chat_log.go

@@ -0,0 +1,50 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package archived
+
+import (
+	"time"
+
+	"github.com/apache/incubator-devlake/core/models/migrationscripts/archived"
+)
+
+type QDevChatLog struct {
+	archived.NoPKModel
+	ConnectionId        uint64    `gorm:"primaryKey"`
+	ScopeId             string    `gorm:"primaryKey;type:varchar(255)" json:"scopeId"`
+	RequestId           string    `gorm:"primaryKey;type:varchar(255)" json:"requestId"`
+	UserId              string    `gorm:"index;type:varchar(255)" json:"userId"`
+	DisplayName         string    `gorm:"type:varchar(255)" json:"displayName"`
+	Timestamp           time.Time `gorm:"index" json:"timestamp"`
+	ChatTriggerType     string    `gorm:"type:varchar(50)" json:"chatTriggerType"`
+	HasCustomization    bool      `json:"hasCustomization"`
+	ConversationId      string    `gorm:"type:varchar(255)" json:"conversationId"`
+	UtteranceId         string    `gorm:"type:varchar(255)" json:"utteranceId"`
+	ModelId             string    `gorm:"type:varchar(100)" json:"modelId"`
+	PromptLength        int       `json:"promptLength"`
+	ResponseLength      int       `json:"responseLength"`
+	OpenFileCount       int       `json:"openFileCount"`
+	ActiveFileName      string    `gorm:"type:varchar(512)" json:"activeFileName"`
+	ActiveFileExtension string    `gorm:"type:varchar(50)" json:"activeFileExtension"`
+	HasSteering         bool      `json:"hasSteering"`
+	IsSpecMode          bool      `json:"isSpecMode"`
+}
+
+func (QDevChatLog) TableName() string {
+	return "_tool_q_dev_chat_log"
+}

backend/plugins/q_dev/models/migrationscripts/archived/completion_log.go

@@ -0,0 +1,42 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package archived
+
+import (
+	"time"
+
+	"github.com/apache/incubator-devlake/core/models/migrationscripts/archived"
+)
+
+type QDevCompletionLog struct {
+	archived.NoPKModel
+	ConnectionId     uint64    `gorm:"primaryKey"`
+	ScopeId          string    `gorm:"primaryKey;type:varchar(255)" json:"scopeId"`
+	RequestId        string    `gorm:"primaryKey;type:varchar(255)" json:"requestId"`
+	UserId           string    `gorm:"index;type:varchar(255)" json:"userId"`
+	DisplayName      string    `gorm:"type:varchar(255)" json:"displayName"`
+	Timestamp        time.Time `gorm:"index" json:"timestamp"`
+	FileName         string    `gorm:"type:varchar(512)" json:"fileName"`
+	FileExtension    string    `gorm:"type:varchar(50)" json:"fileExtension"`
+	HasCustomization bool      `json:"hasCustomization"`
+	CompletionsCount int       `json:"completionsCount"`
+}
+
+func (QDevCompletionLog) TableName() string {
+	return "_tool_q_dev_completion_log"
+}

backend/plugins/q_dev/models/migrationscripts/register.go

@@ -35,5 +35,6 @@ func All() []plugin.MigrationScript {
 		new(addAccountIdToS3Slice),
 		new(fixDedupUserTables),
 		new(resetS3FileMetaProcessed),
+		new(addLoggingTables),
 	}
 }

backend/plugins/q_dev/models/s3_slice.go

@@ -99,7 +99,16 @@ func (s *QDevS3Slice) normalize(strict bool) error {
 	}
 
 	if s.Id == "" {
-		s.Id = s.Prefix
+		if s.AccountId != "" {
+			// Use URL-safe ID: account_year or account_year_month
+			if s.Month != nil {
+				s.Id = fmt.Sprintf("%s_%04d_%02d", s.AccountId, s.Year, *s.Month)
+			} else {
+				s.Id = fmt.Sprintf("%s_%04d", s.AccountId, s.Year)
+			}
+		} else {
+			s.Id = s.Prefix
+		}
 	}
 
 	if s.AccountId != "" {