Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions guac/blob-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,46 @@ Alternatively, set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable:
export GOOGLE_APPLICATION_CREDENTIALS=/secret/sa.json
guaccollect gcs my-bucket
```

## Cloud-agnostic blob collector

The `guaccollect blob` command accepts a single Go Cloud storage URL and
collects documents from S3, GCS, Azure Blob Storage, or a local filesystem. It
is a cloud-agnostic alternative to the provider-specific commands above:

```bash
guaccollect blob "s3://my-bucket?region=us-east-1"
guaccollect blob "gs://my-bucket"
guaccollect blob "azblob://my-container"
guaccollect blob "file:///path/to/sboms"
```

Use the cloud provider's supported environment-based authentication to access
the store. See the [Go Cloud blob guide](https://gocloud.dev/howto/blob/) for
provider-specific URL and credential configuration.

### Scope collection and cap object size

By default, the blob collector reads every object in the selected store. It does
not select files by document type: every in-scope object is passed on for
ingestion. Use a bucket or prefix that contains documents GUAC can ingest.

- `--blob-prefix` limits collection to keys beginning with the supplied prefix,
such as `sboms/`.
- `--blob-max-object-size` sets the maximum object size to read, in bytes.
Larger objects are logged and skipped. A value of `0` uses the collector's
built-in size limit; it does not disable the limit.

To collect only objects under `sboms/` with a 50 MiB per-object cap:

```bash
guaccollect blob --blob-prefix sboms/ --blob-max-object-size 52428800 \
"s3://my-bucket?region=us-east-1"
```

To periodically poll the store, use `--service-poll` and an interval:

```bash
guaccollect blob --service-poll --interval 5m \
"s3://my-bucket?region=us-east-1"
```
Loading