Skip to content

feat(storage): add GCS parity and public file URI semantics#1509

Open
kkopanidis wants to merge 2 commits into
mainfrom
feat/storage-gcs-provider-rewrite
Open

feat(storage): add GCS parity and public file URI semantics#1509
kkopanidis wants to merge 2 commits into
mainfrom
feat/storage-gcs-provider-rewrite

Conversation

@kkopanidis

@kkopanidis kkopanidis commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Summary

This expands the original GCS provider rewrite into a broader public-file URL contract for the storage module.

GCS provider parity:

  • Implements the missing IStorageProvider methods for containers, folders, files, and signed URLs.
  • Fixes exists so it returns the provider result instead of always returning true.
  • Supports service account key file, inline JSON (google.serviceAccountKeyJson), and Application Default Credentials.
  • Uses Uniform Bucket-Level Access and IAM (roles/storage.objectViewer for allUsers) for public buckets.

Public file URL semantics:

  • Adds File.uri as a stable Conduit-relative path (/storage/getFileUrl/:id).
  • Keeps File.url / sourceUrl as direct provider/CDN URLs only when both the container and file are public.
  • Allows public files in private containers by storing uri only and signing provider URLs on demand through Conduit.
  • Rejects private files in public containers, including omitted isPublic values.
  • Removes unsupported 99/100-year signed URL branches from GCS, Azure, Aliyun, and AWS getPublicUrl handling.
  • Adds a migration to backfill uri and clear stale direct URLs for public files in private containers.

Validation

  • npm run build in modules/storage
  • npm run build in libraries/grpc-sdk
  • Focused ESLint on touched storage TypeScript files
  • Composer 2.5 deslop pass
  • Composer 2.5 verification pass

Manual test plan

  • Configure storage provider google with a GCS bucket or emulator.
  • Verify GCS auth via key file path, inline JSON, and ADC.
  • Public container + public file stores sourceUrl, url, and uri.
  • Public container + omitted/false isPublic throws a clear validation error.
  • Private container + public file stores uri only; unauthenticated GET /storage/getFileUrl/:id returns a fresh signed URL.
  • Private container + private file still requires auth and signs on demand.
  • Folder/container/file lifecycle methods work for GCS, including exists returning false for missing files.
  • Existing public files are migrated with uri, and private-container public files have stale direct URLs cleared.

Replace the deprecated Google Cloud Storage provider with a full
implementation aligned to AWS S3 and Azure, including UBLA-safe IAM
public access, v4 signed URLs, ADC/inline JSON auth, and fixes for
deleteContainer, deleteFolder, exists, getSignedUrl, and folder markers.
Store stable Conduit-relative URIs for public files so private-container
public files resolve through Storage without unsupported long-lived provider
signatures.
@kkopanidis kkopanidis changed the title feat(storage): rewrite GCS provider to IStorageProvider parity feat(storage): add GCS parity and public file URI semantics Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant