Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ The customer-facing CLI for the tracebloc declarative ingestion path. Wraps the

## Status

**v0.3.0 is released** — the latest stable [release](https://github.com/tracebloc/cli/releases/latest), cut from `develop`. It builds on v0.2.0's guided `dataset push` and `dataset rm` with a new `dataset list` command plus home-screen / output polish (clearer copy, guided-first framing). The binary implements `version`, `completion`, `ingest validate`, `cluster info`, and the full `dataset push` / `dataset list` / `dataset rm` flow — local schema validation, cluster discovery, data staging, submission, and Job watching, end to end.
**v0.3.0 is released** — the latest stable [release](https://github.com/tracebloc/cli/releases/latest), cut from `develop`. It builds on v0.2.0's guided `data ingest` and `dataset rm` with a new `dataset list` command plus home-screen / output polish (clearer copy, guided-first framing). The binary implements `version`, `completion`, `data validate`, `cluster info`, and the full `data ingest` / `dataset list` / `dataset rm` flow — local schema validation, cluster discovery, data staging, submission, and Job watching, end to end.

`dataset push` covers **9 of 10 task categories**: `image_classification`, `object_detection`, `keypoint_detection`, `text_classification`, `masked_language_modeling`, `tabular_classification`, `tabular_regression`, `time_series_forecasting`, and `time_to_event_prediction`. `semantic_segmentation` is pending mask-sidecar support upstream ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)); `instance_segmentation` is not yet implemented.
`data ingest` covers **9 of 10 task categories**: `image_classification`, `object_detection`, `keypoint_detection`, `text_classification`, `masked_language_modeling`, `tabular_classification`, `tabular_regression`, `time_series_forecasting`, and `time_to_event_prediction`. `semantic_segmentation` is pending mask-sidecar support upstream ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)); `instance_segmentation` is not yet implemented.

The release pipeline ships [`v0.3.0`](https://github.com/tracebloc/cli/releases/latest) as **cosign-signed, multi-arch binaries** — Linux (`amd64`, `arm64`, `386`, `arm`), macOS (`amd64`, `arm64`), and Windows (`amd64`, `arm64`) — each with `SHA256SUMS` and the install scripts. Install via [Customer experience](#customer-experience) or [build from source](#building-from-source). (A Homebrew tap and the `install.tracebloc.io` vanity URL are later follow-ups; the GitHub release URL serves installs today.)

Expand All @@ -24,7 +24,7 @@ The CLI is a sibling interface to the chart, not a replacement. Both translate t
Customer interfaces (pick one or many):
┌─────────────────────────────────────────────────┐
│ Web UI Studio for clicking through │ ← future
│ CLI `tracebloc dataset push ./data` │ ← this repo
│ CLI `tracebloc data ingest ./data` │ ← this repo
│ Python SDK `IngestConfig(...).submit()` │ ← future
│ K8s CRD `kubectl apply` Ingestion CR │ ← future
│ Helm chart tracebloc/ingestor │ ← today
Expand All @@ -50,9 +50,10 @@ The protocol — the v1 schema + the POST endpoint — is the stable point. Ever
curl -fsSL https://github.com/tracebloc/cli/releases/latest/download/install.sh | sh
# Install — Windows (PowerShell)
irm https://github.com/tracebloc/cli/releases/latest/download/install.ps1 | iex
# The installer also creates a short alias: tb works everywhere tracebloc does.

# Per dataset
tracebloc dataset push ./my-data \
tracebloc data ingest ./my-data \
--table cats_dogs_train \
--category image_classification \
--intent train \
Expand Down Expand Up @@ -108,15 +109,15 @@ All v0.1 phases are merged:
| Phase | Ticket | What | Status |
|---|---|---|---|
| 0 | [#148](https://github.com/tracebloc/client/issues/148) | Repo bootstrap + Go module + CI + `tracebloc version` | ✅ |
| 1 | [#149](https://github.com/tracebloc/client/issues/149) | Embed `ingest.v1.json` + `tracebloc ingest validate <path>` (local-only) | ✅ |
| 1 | [#149](https://github.com/tracebloc/client/issues/149) | Embed `ingest.v1.json` + `tracebloc data validate <path>` (local-only) | ✅ |
| 2 | [#150](https://github.com/tracebloc/client/issues/150) | Cluster discovery + ingestor SA token via TokenRequest | ✅ |
| 3 | [#151](https://github.com/tracebloc/client/issues/151) | Stage data into the shared PVC via ephemeral Pod | ✅ |
| 4 | [#152](https://github.com/tracebloc/client/issues/152) | Submit to jobs-manager + watch ingestor Job + summary | ✅ |
| 5 | [#153](https://github.com/tracebloc/client/issues/153) | GitHub Releases + install.sh distribution (Homebrew tap deferred) | ✅ — [`v0.1.0`](https://github.com/tracebloc/cli/releases/tag/v0.1.0) released (stable, 8-platform) |

Beyond the original phases, `dataset push` was widened from image-classification-only to 9 of 10 modalities, and the test suite gained unit-coverage wins plus a kind-based integration harness for the real-I/O seams.
Beyond the original phases, `data ingest` was widened from image-classification-only to 9 of 10 modalities, and the test suite gained unit-coverage wins plus a kind-based integration harness for the real-I/O seams.

**v0.2.0** added a friendlier guided `dataset push` and `dataset rm` on the home screen (#44, #47). **v0.3.0** added the `dataset list` command (#53) plus home-screen / output-spacing polish and feedback-copy refinements (#52, #56). **Next:** cloud-source ingestion (S3/GCS/HTTPS) for datasets above the 1 GiB local cap; `semantic_segmentation` ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)) and `instance_segmentation`. Smaller follow-ups are tracked as [open issues](https://github.com/tracebloc/cli/issues).
**v0.2.0** added a friendlier guided `data ingest` and `dataset rm` on the home screen (#44, #47). **v0.3.0** added the `dataset list` command (#53) plus home-screen / output-spacing polish and feedback-copy refinements (#52, #56). **Next:** cloud-source ingestion (S3/GCS/HTTPS) for datasets above the 1 GiB local cap; `semantic_segmentation` ([data-ingestors#136](https://github.com/tracebloc/data-ingestors/issues/136)) and `instance_segmentation`. Smaller follow-ups are tracked as [open issues](https://github.com/tracebloc/cli/issues).

Epic: [tracebloc/client#147](https://github.com/tracebloc/client/issues/147).

Expand Down
2 changes: 1 addition & 1 deletion internal/cli/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ var readInClusterClient = cluster.DiscoverInClusterClient
func newClientCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "client",
Short: "Provision and manage tracebloc clients (machines)",
Short: "Provision and manage the clients in your account",
Long: `Provision a tracebloc client for this machine and list/select clients
in your account. Requires sign-in first (` + "`tracebloc login`" + `).`,
}
Expand Down
51 changes: 34 additions & 17 deletions internal/cli/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"time"

Expand All @@ -28,8 +29,8 @@ func newClusterCmd() *cobra.Command {
Long: `Commands for inspecting the Kubernetes cluster the CLI is
configured to talk to.

Use ` + "`cluster info`" + ` to verify which cluster, namespace, and parent
tracebloc release the next ` + "`data ingest`" + ` will target. Useful as a
Use ` + "`cluster info`" + ` to verify which cluster, namespace, and
client the next ` + "`data ingest`" + ` will target. Useful as a
pre-flight before doing anything destructive (e.g. ingesting into
the wrong cluster).`,
}
Expand Down Expand Up @@ -64,13 +65,13 @@ func newClusterInfoCmd() *cobra.Command {

cmd := &cobra.Command{
Use: "info",
Short: "Show the cluster, namespace, parent release, and ingestor SA token state",
Long: `Discovers the tracebloc parent client release in the configured
Short: "Show the cluster, namespace, client install, and ingestor token state",
Long: `Discovers the tracebloc client installed in the configured
cluster + namespace and prints:

• Which kubeconfig context the CLI used
• The namespace it resolved to
• The parent release name + chart version + appVersion
• The client's release name + chart version + appVersion
• The jobs-manager Service the next data ingest would POST to
• The ingestor ServiceAccount the post-install hook would auth as
• The cluster's configured INGESTOR_IMAGE_DIGEST default
Expand All @@ -84,7 +85,7 @@ token I expect" without leaking it to terminal scrollback.

Exit codes:
0 cluster discovered + token mintable; CLI is ready
4 cluster reachable but no tracebloc parent release found
4 cluster reachable but no tracebloc client found
5 cluster reachable + release found but no usable SA token`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
Expand All @@ -103,10 +104,10 @@ Exit codes:
cmd.Flags().StringVar(&contextOverride, "context", "",
"name of the kubeconfig context to use (default: kubeconfig's current-context)")
cmd.Flags().StringVarP(&nsOverride, "namespace", "n", "",
"namespace where the parent tracebloc/client release is installed (default: the context's namespace, or 'default')")
"namespace where your tracebloc client is installed (default: the context's namespace, or 'default')")
cmd.Flags().StringVar(&ingestorSAName, "ingestor-sa", "",
"override the ingestor ServiceAccount name (default: \"ingestor\", the chart default; "+
"set this if you customized `ingestionAuthz.serviceAccountName` in the parent client chart)")
"set this if you customized `ingestionAuthz.serviceAccountName` in your client's install)")
cmd.Flags().Int64Var(&tokenExpiry, "token-expiry-seconds", 600,
"requested SA token expiration in seconds (default 600 = 10 min; ignored for static-secret fallback)")

Expand All @@ -122,11 +123,17 @@ func runClusterInfo(
) error {
p.Banner("tracebloc", "cluster diagnostics")

resolved, err := cluster.Load(cluster.KubeconfigOptions{
// Bind the active client's namespace exactly like the data commands do,
// so this pre-flight targets what `data ingest` will actually target —
// and so the multi-client "set your active client" remediation works
// here too, not just on the data path.
opts := cluster.KubeconfigOptions{
Path: kubeconfigPath,
Context: contextOverride,
Namespace: nsOverride,
})
}
binding := bindActiveClientNamespace(&opts)
resolved, err := cluster.Load(opts)
if err != nil {
// Kubeconfig errors are exit-code-3 territory (file/parse
// problem, same conceptual class as `ingest validate`'s
Expand All @@ -142,17 +149,27 @@ func runClusterInfo(
p.Section("Kubeconfig")
p.Field("context", resolved.Context)
p.Field("server", resolved.ServerURL)
p.Field("namespace", resolved.Namespace)

// Discover the parent release.
release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace)
// Discover the client's release — with the cluster-wide fallback scan
// when the namespace is just the kubeconfig default, so diagnostics find
// the client in its slug namespace instead of dead-ending on "default".
release, nsUsed, err := discoverRelease(ctx, p, cs, resolved.Namespace, binding.allowScan())
if err != nil {
// 4 = "cluster reachable, but no tracebloc release here."
// 4 = "cluster reachable, but no tracebloc client here."
// Distinct from the kubeconfig error (3) so callers can
// branch: 3 means "fix your kubeconfig", 4 means "install
// the parent chart first".
// branch: 3 means "fix your kubeconfig", 4 means "no client
// installed on this cluster". A binding miss gets the §7.3
// "runs elsewhere" explanation, same as the data commands.
if errors.Is(err, cluster.ErrNoParentRelease) {
return binding.explain(&exitError{code: 4, err: &noParentReleaseError{err}})
}
return &exitError{code: 4, err: err}
}
resolved.Namespace = nsUsed
// Printed after discovery so it reflects the namespace the scan actually
// retargeted to — this pre-flight's whole job is to report what the next
// `data ingest` will target, so it must not show the pre-scan default.
p.Field("namespace", resolved.Namespace)

// Apply the SA-name override here. Discovery doesn't read the
// name from the cluster (see #7); customers with a non-default
Expand All @@ -161,7 +178,7 @@ func runClusterInfo(
release.IngestorSAName = ingestorSAOverride
}

p.Section("Parent release")
p.Section("Client install")
p.Field("name", release.ReleaseName)
p.Field("chart version", release.ChartVersion)
p.Field("app version", release.AppVersion)
Expand Down
53 changes: 50 additions & 3 deletions internal/cli/clustertarget.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ import (
"context"
"errors"
"fmt"
"strings"

"k8s.io/client-go/kubernetes"

"github.com/tracebloc/cli/internal/cluster"
"github.com/tracebloc/cli/internal/config"
"github.com/tracebloc/cli/internal/ui"
)

// noParentReleaseError marks the exit-4 case where the reached cluster
Expand Down Expand Up @@ -42,7 +44,7 @@ type clusterTarget struct {
// `cluster doctor` is deliberately NOT a caller — it has a different exit
// contract (2/3 escalation, with discovery reported as a check Result rather
// than a hard error).
func resolveClusterTarget(ctx context.Context, opts cluster.KubeconfigOptions, needPVC bool) (*clusterTarget, error) {
func resolveClusterTarget(ctx context.Context, p *ui.Printer, opts cluster.KubeconfigOptions, b activeClientBinding, needPVC bool) (*clusterTarget, error) {
resolved, err := cluster.Load(opts)
if err != nil {
return nil, &exitError{code: 3, err: fmt.Errorf("loading kubeconfig: %w", err)}
Expand All @@ -51,7 +53,12 @@ func resolveClusterTarget(ctx context.Context, opts cluster.KubeconfigOptions, n
if err != nil {
return nil, &exitError{code: 3, err: err}
}
release, err := cluster.DiscoverParentRelease(ctx, cs, resolved.Namespace)
// The cluster-wide fallback scan only engages when the target namespace is
// the kubeconfig's default — i.e. nobody chose it: not the user (explicit
// --namespace/--context) and not the active-client binding. A binding miss
// must NOT silently redirect to some other client (§7.5 — that could be a
// different machine's client); it keeps the §7.3 "runs elsewhere" message.
release, nsUsed, err := discoverRelease(ctx, p, cs, resolved.Namespace, b.allowScan())
if err != nil {
// Only a genuine "namespace has no release" maps to the §7.3
// "runs elsewhere" rewrite; an API/RBAC list failure or an
Expand All @@ -61,6 +68,10 @@ func resolveClusterTarget(ctx context.Context, opts cluster.KubeconfigOptions, n
}
return nil, &exitError{code: 4, err: err}
}
// The scan may have retargeted discovery to the namespace that actually
// hosts the client; everything downstream (PVC discovery, dataset listing,
// prints) keys on Resolved.Namespace, so it must follow.
resolved.Namespace = nsUsed
t := &clusterTarget{Resolved: resolved, Clientset: cs, Release: release}
if needPVC {
pvc, err := cluster.DiscoverSharedPVC(ctx, cs, resolved.Namespace)
Expand All @@ -72,12 +83,41 @@ func resolveClusterTarget(ctx context.Context, opts cluster.KubeconfigOptions, n
return t, nil
}

// discoverRelease wraps DiscoverParentRelease with the cluster-wide fallback
// scan: when allowScan is set and the target namespace hosts no client, every
// namespace is scanned for one. Exactly one → target it, with a visible note
// (never a silent redirect); several → name them and ask the user to pick;
// none, or a scan failure (e.g. RBAC forbids the cluster-wide list) → the
// original discovery error stands. Returns the namespace actually used.
func discoverRelease(ctx context.Context, p *ui.Printer, cs kubernetes.Interface, namespace string, allowScan bool) (*cluster.ParentRelease, string, error) {
release, err := cluster.DiscoverParentRelease(ctx, cs, namespace)
if err == nil || !allowScan || !errors.Is(err, cluster.ErrNoParentRelease) {
return release, namespace, err
}
found, scanErr := cluster.FindClientNamespaces(ctx, cs)
if scanErr != nil || len(found) == 0 {
return nil, namespace, err
}
if len(found) > 1 {
return nil, namespace, fmt.Errorf(
"%w in namespace %q, but tracebloc clients are running in: %s. "+
"Pass --namespace to pick one, or set your active client with `tracebloc client use`.",
cluster.ErrNoParentRelease, namespace, strings.Join(found, ", "))
}
if p != nil {
p.Infof("No client in namespace %q — using the one in %q (override with --namespace).", namespace, found[0])
}
release, err = cluster.DiscoverParentRelease(ctx, cs, found[0])
return release, found[0], err
}

// activeClientBinding records that a data command defaulted its target
// namespace to the active client's cached namespace (§7.3), so a subsequent
// "no release here" failure can be explained as "the active client runs
// elsewhere" rather than a bare discovery error.
type activeClientBinding struct {
applied bool
explicit bool // user pinned --namespace/--context themselves
name string
namespace string
}
Expand All @@ -89,7 +129,7 @@ type activeClientBinding struct {
// backward compatible for anyone who hasn't run `client use`/`create`.
func bindActiveClientNamespace(opts *cluster.KubeconfigOptions) activeClientBinding {
if opts.Namespace != "" || opts.Context != "" {
return activeClientBinding{} // user was explicit — don't second-guess
return activeClientBinding{explicit: true} // user was explicit — don't second-guess
}
cfg, err := config.Load()
if err != nil {
Expand All @@ -103,6 +143,13 @@ func bindActiveClientNamespace(opts *cluster.KubeconfigOptions) activeClientBind
return activeClientBinding{applied: true, name: p.ActiveClientName, namespace: p.ActiveClientNamespace}
}

// allowScan reports whether the cluster-wide fallback scan may engage: only
// when the target namespace is the kubeconfig's default — i.e. nobody chose
// it. An explicit --namespace/--context is never second-guessed, and a
// binding miss must NOT silently retarget to some other client (§7.5 — it
// could be a different machine's); it keeps the §7.3 "runs elsewhere" message.
func (b activeClientBinding) allowScan() bool { return !b.applied && !b.explicit }

// explain rewrites a "no tracebloc release in namespace" failure (exit 4) into
// §7.3's "client runs on another machine" guidance when the target namespace
// came from the active-client binding: the cluster the kubeconfig reaches
Expand Down
Loading
Loading