feat(x2a): bulk CSV project creation (#2579)

* feat(x2a): bulk CSV project creation

Signed-off-by: Marek Libra <marek.libra@gmail.com>

* add sample-project.csv

Signed-off-by: Marek Libra <marek.libra@gmail.com>

---------

Signed-off-by: Marek Libra <marek.libra@gmail.com>

Author: mareklibra

workspaces/x2a/.changeset/wise-pianos-shine.md

@@ -0,0 +1,8 @@
+---
+'@red-hat-developer-hub/backstage-plugin-scaffolder-backend-module-x2a': patch
+'@red-hat-developer-hub/backstage-plugin-x2a-common': patch
+'@red-hat-developer-hub/backstage-plugin-x2a': patch
+'@red-hat-developer-hub/backstage-plugin-x2a-backend': patch
+---
+
+The user can newly bulk-create conversion projects from an uploaded CSV file.

workspaces/x2a/README.md

@@ -301,3 +301,7 @@ Loaded Kubernetes configuration from ~/.kube/config
   - `migrations/` - Database migrations
 - `plugins/x2a-common/` - Shared code between frontend and backend
   - `client/src/schema/openapi/generated/` - Generated client-side API code
+
+## CSV Bulk Project Import
+
+See [CSV Bulk Project Import](./docs/csv-bulk-import.md) for the CSV file format, an example, repository URL conventions, and the `RepoAuthentication` scaffolder extension.

workspaces/x2a/docs/csv-bulk-import.md

@@ -0,0 +1,132 @@
+# CSV Bulk Project Import
+
+The CSV bulk import lets the user create multiple conversion projects at once by uploading a single CSV file.
+
+## How to Access
+
+1. Open the Backstage instance and navigate to `/create`.
+2. Select the **Chef-to-Ansible Conversion Project** template (`chef-conversion-project-template`).
+3. On the first page, choose **CSV upload** as the input method.
+4. Upload the CSV file and proceed through the wizard.
+
+The wizard asks for authentication with each SCM provider (GitHub, GitLab, Bitbucket) referenced in the CSV. Projects are created sequentially with the same permission checks as if the user had created each one individually.
+
+## CSV File Format
+
+The file must be UTF-8 encoded with a header row. Column order does not matter, but header names must match exactly.
+
+### Required Columns
+
+| Column             | Description                                                                                               |
+| ------------------ | --------------------------------------------------------------------------------------------------------- |
+| `name`             | Unique project name                                                                                       |
+| `abbreviation`     | Short project identifier, 1-5 alphanumeric characters matching `^([a-zA-Z][a-zA-Z0-9]*)(-[a-zA-Z0-9]+)*$` |
+| `sourceRepoUrl`    | URL of the repository containing the Chef cookbook to convert                                             |
+| `sourceRepoBranch` | Branch to read from in the source repository                                                              |
+| `targetRepoBranch` | Branch to write converted Ansible output to                                                               |
+
+### Optional Columns
+
+| Column          | Description                                                                              |
+| --------------- | ---------------------------------------------------------------------------------------- |
+| `description`   | Project description (defaults to empty)                                                  |
+| `ownedByGroup`  | Backstage group that owns the project. When empty, the signed-in user becomes the owner. |
+| `targetRepoUrl` | Repository for converted output. Defaults to `sourceRepoUrl` when empty.                 |
+
+No extra columns are allowed -- the import will reject unknown headers.
+
+### Repeatable Import
+
+The CSV import is designed to be run repeatedly with the same or an updated file. Projects whose name already exists are **skipped** (not duplicated) and counted as "skipped" in the results summary.
+
+A typical workflow for a large import:
+
+1. Upload the CSV. Some projects succeed, some may fail (e.g. due to a missing repository or a typo).
+2. Review the results. The summary shows how many succeeded, failed, and were skipped.
+3. Fix the issues - correct the CSV rows that failed and, if a partially-created project needs to be recreated, delete it from the application first.
+4. Re-upload the corrected CSV. Already-created projects are skipped automatically. Only the new or corrected rows are processed.
+
+### Repository URL Format
+
+Both `sourceRepoUrl` and `targetRepoUrl` accept two formats. All URLs are normalized to HTTPS clone URLs before being stored.
+
+**Plain HTTPS URLs** (standard clone URLs):
+
+| Provider  | Format                                 |
+| --------- | -------------------------------------- |
+| GitHub    | `https://github.com/owner/repo`        |
+| GitLab    | `https://gitlab.com/owner/repo`        |
+| Bitbucket | `https://bitbucket.org/workspace/repo` |
+
+**Backstage RepoUrlPicker format** (query-parameter style, without `https://`):
+
+| Provider  | Format                                                           |
+| --------- | ---------------------------------------------------------------- |
+| GitHub    | `github.com?owner=myuser&repo=myrepo`                            |
+| GitLab    | `gitlab.com?owner=myuser&repo=myrepo`                            |
+| Bitbucket | `bitbucket.org?workspace=myworkspace&project=myproj&repo=myrepo` |
+
+For Bitbucket, the `project` parameter is organizational metadata and is not part of the clone URL. Only `workspace` and `repo` are used.
+
+For self-hosted instances (e.g. GitHub Enterprise, self-hosted GitLab), the corresponding host should be used in place of the public domain. The host must be listed in the `integrations:` section of `app-config.yaml` so the plugin can detect the correct SCM provider. See [SCM Provider Detection](../README.md#scm-provider-detection).
+
+### Example
+
+```csv
+name,abbreviation,sourceRepoUrl,sourceRepoBranch,targetRepoUrl,targetRepoBranch,description,ownedByGroup
+web-app,wapp,https://github.com/myorg/web-app-chef,main,https://github.com/myorg/web-app-ansible,main,Convert web app cookbook,team-platform
+db-setup,dbset,gitlab.com?owner=myorg&repo=db-chef,develop,gitlab.com?owner=myorg&repo=db-ansible,main,,
+cache-svc,cache,bitbucket.org?workspace=myws&project=x2a&repo=cache-chef,main,,main,Cache service conversion,
+```
+
+Notes on the example:
+
+- Row 1 (`web-app`): uses plain HTTPS URLs.
+- Row 2 (`db-setup`): uses RepoUrlPicker-style URLs for GitLab. `description` and `ownedByGroup` are left empty.
+- Row 3 (`cache-svc`): uses RepoUrlPicker-style URL for Bitbucket. `targetRepoUrl` is empty, so the source repository is used as the target.
+
+### CSV file template
+
+Download a [sample CSV file](../plugins/x2a-backend/public/sample-projects.csv) with all supported headers.
+
+At runtime, the file is served at `/x2a/download/sample-projects.csv` (via the frontend plugin route).
+
+## RepoAuthentication Scaffolder Extension
+
+When using CSV import, the template uses the `RepoAuthentication` custom scaffolder field to collect OAuth tokens for each SCM provider found in the CSV. This replaces the standard `RepoUrlPicker` used in manual mode.
+
+### How It Works
+
+1. The extension parses the uploaded CSV and identifies all distinct SCM providers across source and target URLs.
+2. It opens an OAuth dialog for each provider for the user to authenticate.
+3. Tokens are stored as scaffolder secrets with the prefix `OAUTH_TOKEN_` (e.g. `OAUTH_TOKEN_github`, `OAUTH_TOKEN_gitlab`, `OAUTH_TOKEN_bitbucket`).
+4. The wizard blocks progression until all required providers are authenticated.
+
+### Using in a Template
+
+The extension is registered as `RepoAuthentication` and is available as a `ui:field`. It must reference the CSV field via `ui:options.csvFieldName`:
+
+```yaml
+properties:
+  repoAuthentication:
+    type: string
+    description: Provide login to all the SCMs relevant for the source CSV.
+    ui:field: RepoAuthentication
+    ui:options:
+      csvFieldName: csvContent
+```
+
+### Registering the Extension
+
+In a standard Backstage app, import and add the extension so the scaffolder can find it:
+
+```typescript
+import { RepoAuthenticationExtension } from '@red-hat-developer-hub/backstage-plugin-x2a';
+
+// In the App component, alongside <ScaffolderPage>:
+<ScaffolderFieldExtensions>
+  <RepoAuthenticationExtension />
+</ScaffolderFieldExtensions>
+```
+
+For Red Hat Developer Hub (RHDH) with dynamic plugins, the extension is registered via configuration instead. See [Providing custom Scaffolder field extensions](https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.9/html/installing_and_viewing_plugins_in_red_hat_developer_hub/assembly-front-end-plugin-wiring.adoc_rhdh-extensions-plugins#con-providing-custom-scaffolder-field-extensions.adoc_assembly-front-end-plugin-wiring) in the RHDH documentation.

workspaces/x2a/packages/app/package.json

@@ -38,6 +38,7 @@
     "@backstage/plugin-org": "^0.6.43",
     "@backstage/plugin-permission-react": "^0.4.36",
     "@backstage/plugin-scaffolder": "^1.34.0",
+    "@backstage/plugin-scaffolder-react": "^1.20.0",
     "@backstage/plugin-search": "^1.4.29",
     "@backstage/plugin-search-react": "^1.9.3",
     "@backstage/plugin-signals": "^0.0.22",

workspaces/x2a/packages/app/src/App.tsx

@@ -25,6 +25,7 @@ import {
   catalogImportPlugin,
 } from '@backstage/plugin-catalog-import';
 import { ScaffolderPage, scaffolderPlugin } from '@backstage/plugin-scaffolder';
+import { ScaffolderFieldExtensions } from '@backstage/plugin-scaffolder-react';
 import { orgPlugin } from '@backstage/plugin-org';
 import { SearchPage } from '@backstage/plugin-search';
 import {
@@ -55,6 +56,7 @@ import { SignalsDisplay } from '@backstage/plugin-signals';
 import {
   X2APage,
   x2aPluginTranslations,
+  RepoAuthenticationExtension,
 } from '@red-hat-developer-hub/backstage-plugin-x2a';
 import {
   bitbucketAuthApiRef,
@@ -135,7 +137,6 @@ const routes = (
         <ReportIssue />
       </TechDocsAddons>
     </Route>
-    <Route path="/create" element={<ScaffolderPage />} />
     <Route path="/api-docs" element={<ApiExplorerPage />} />
     <Route
       path="/catalog-import"
@@ -152,6 +153,15 @@ const routes = (
     <Route path="/settings" element={<UserSettingsPage />} />
     <Route path="/catalog-graph" element={<CatalogGraphPage />} />
     <Route path="/notifications" element={<NotificationsPage />} />
+
+    {/* At RHDH runtime, this is replaced by dynamicPlugin configuration:
+      https://docs.redhat.com/en/documentation/red_hat_developer_hub/1.9/html/installing_and_viewing_plugins_in_red_hat_developer_hub/assembly-front-end-plugin-wiring.adoc_rhdh-extensions-plugins#con-providing-custom-scaffolder-field-extensions.adoc_assembly-front-end-plugin-wiring
+    */}
+    <Route path="/create" element={<ScaffolderPage />}>
+      <ScaffolderFieldExtensions>
+        <RepoAuthenticationExtension />
+      </ScaffolderFieldExtensions>
+    </Route>
   </FlatRoutes>
 );
 

workspaces/x2a/plugins/scaffolder-backend-module-x2a/src/actions/createAndInitProject.ts

@@ -0,0 +1,120 @@
+/*
+ * Copyright Red Hat, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {
+  DefaultApiClient,
+  Project,
+  ProjectsPost,
+  ProjectsProjectIdRunPost200Response,
+  normalizeRepoUrl,
+  ScmProviderName,
+} from '@red-hat-developer-hub/backstage-plugin-x2a-common';
+import type { ActionLogger } from './createProjectAction';
+
+export type CreateAndInitProjectParams = {
+  api: DefaultApiClient;
+  row: ProjectsPost['body'];
+  sourceRepoToken: string;
+  targetRepoToken: string;
+  userPrompt?: string;
+  backstageToken?: string;
+  hostProviderMap: Map<string, ScmProviderName>;
+  logger: ActionLogger;
+};
+
+export const createAndInitProject = async (
+  params: CreateAndInitProjectParams,
+): Promise<{ projectId: string; initJobId: string }> => {
+  const {
+    api,
+    row,
+    sourceRepoToken,
+    targetRepoToken,
+    userPrompt,
+    backstageToken: token,
+    logger,
+  } = params;
+
+  const body: ProjectsPost['body'] = {
+    name: row.name,
+    description: row.description ?? '',
+    abbreviation: row.abbreviation,
+    ownedByGroup: row.ownedByGroup?.trim() || undefined,
+    sourceRepoUrl: normalizeRepoUrl(row.sourceRepoUrl),
+    targetRepoUrl: normalizeRepoUrl(row.targetRepoUrl),
+    sourceRepoBranch: row.sourceRepoBranch,
+    targetRepoBranch: row.targetRepoBranch,
+  };
+
+  logger.info(`Creating project "${row.name}" (${JSON.stringify(body)})`);
+
+  let project: Project;
+  try {
+    const response = await api.projectsPost({ body }, { token });
+    if (!response.ok) {
+      const error = (await response.json()) as { message?: string };
+      logger.error(
+        `Project "${row.name}" creation failed (status ${response.status}): ${JSON.stringify(error)}`,
+      );
+      throw new Error(error?.message ?? JSON.stringify(error));
+    }
+    project = await response.json();
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    logger.error(`Error creating project "${row.name}": ${message}`);
+    throw error;
+  }
+
+  logger.info(
+    `Project "${row.name}" created with id ${project.id}, triggering init-phase`,
+  );
+
+  let initResponseData: ProjectsProjectIdRunPost200Response;
+  try {
+    const initResponse = await api.projectsProjectIdRunPost(
+      {
+        path: { projectId: project.id },
+        body: {
+          sourceRepoAuth: { token: sourceRepoToken },
+          targetRepoAuth: { token: targetRepoToken },
+          userPrompt,
+        },
+      },
+      { token },
+    );
+
+    if (!initResponse.ok) {
+      const error = (await initResponse.json()) as { message?: string };
+      logger.error(
+        `Init-phase for project "${row.name}" (${project.id}) failed (status ${initResponse.status}): ${JSON.stringify(error)}`,
+      );
+      throw new Error(error?.message ?? JSON.stringify(error));
+    }
+
+    initResponseData = await initResponse.json();
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    logger.error(
+      `Error triggering init-phase for project "${row.name}" (${project.id}): ${message}`,
+    );
+    throw error;
+  }
+
+  logger.info(
+    `Init-phase triggered for project "${row.name}" (${project.id}), jobId: ${initResponseData.jobId}`,
+  );
+
+  return { projectId: project.id, initJobId: initResponseData.jobId };
+};