diff --git a/GIT_PLAN.md b/GIT_PLAN.md index add85a1..c8a3558 100644 --- a/GIT_PLAN.md +++ b/GIT_PLAN.md @@ -11,9 +11,9 @@ rules remain outside the Rust core. - Classic receive-pack push, branches, annotated tags, stale-write rejection, fast-forward policy, connectivity validation, and malformed-input rejection. - Atomic publication of refs and the sparse object catalog. -- Arbitrary configured repository paths with isolated WAL identities and an - explicit SHA-1 or SHA-256 format. Adding a repository requires configuration, - not source changes. +- Arbitrary repository paths with isolated WAL identities and an explicit + SHA-1 or SHA-256 format. Authorized API creation requires no per-repository + configuration or restart; named access policies remain optional overrides. - Per-repository reader, writer, and administrator permissions; read-only mode; a persisted default branch; and recovery without a local repository cache. - Automatic maintenance that prunes unreachable Git objects and invokes bounded @@ -100,9 +100,12 @@ in #39. The declarative repository map supplies canonical nested paths, stable WAL identities, immutable object formats, default branches and independent read/write/admin groups. Duplicate identities, aliases and malformed paths are -rejected. Unknown repositories fail closed. Reads and administration open -existing WALs; an authorized first-push discovery can materialize a configured -repository before accepting a pack. +rejected. A wildcard policy admits authenticated API creation of new names; +missing repositories remain absent on reads, administration and wildcard push. +Explicitly provisioned names also support authorized first-push discovery. +The format and default branch are published through the existing repository WAL. +Scheduled maintenance currently requires named entries; dynamic discovery is +separate work. Remote tests added a nested SHA-256 repository through configuration while preserving the component artifact. Read discovery returned 404 with no durable diff --git a/HANDOFF.md b/HANDOFF.md index 87079e2..3ed5f06 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -58,9 +58,12 @@ The Git service supports unchanged SHA-1 and SHA-256 clients, protocol-v2 clone and have-aware fetch, shallow history, classic push, branches, tags, access control, cold recovery, automatic tail checkpoints, and explicit maintenance. Refs and its sparse object catalog publish atomically through one WAL commit. -Local MinIO tests pass with unmodified go-git. Repository paths, formats, default -branches and per-action groups now come from configuration. Only an authorized -writer can materialize a configured repository; reads open existing state. +Local MinIO tests pass with unmodified go-git. Repository creation is available through an authenticated API without per-name +configuration. A wildcard access policy supplies defaults; exact entries replace +it. Canonical names derive isolated WAL identities, and the creation publication +persists the format and default branch. Reads and wildcard pushes only open +existing Git roots; explicitly configured names also retain first-push provisioning. +The host maintenance worker still visits only named configuration entries. Cognito access-token validation and a separate administration-only machine client are wired. Password mode is the explicit local default and requires a password. Public EC2 HTTPS qualification has passed real Cognito browser login, diff --git a/examples/git/README.md b/examples/git/README.md index f3aac85..f80d8da 100644 --- a/examples/git/README.md +++ b/examples/git/README.md @@ -141,7 +141,7 @@ All values are Spin variables. The defaults target the local MinIO setup above. | `wal_collection_candidates` | `1000` | Maximum entries in a new deletion plan, capped by the durable graph limit | | `wal_max_collection_objects` | `100000` | Complete publication graph and distinct collection live-object bound; shared paths count separately for publication | | `wal_recover_retentions_after_drain` | `false` | Exclusive lost-retention recovery mode | -| `git_repositories` | two demo entries | JSON map of paths, WAL identities, formats, default branches and permissions | +| `git_repositories` | default policy and two demo entries | JSON access policies; optional fixed identities, formats and default branches | | `git_auth_mode` | `password` | `password`, `cognito`, or explicit local `anonymous` mode | | `git_password` | empty | Required password for local password mode | | `git_cognito_issuer` | empty | HTTPS Cognito user-pool issuer | @@ -186,17 +186,43 @@ configured limit high enough to read them; lowering it blocks those objects. ## Repositories and permissions -Repository names come from configuration, including nested names. For example: +Create a repository through the API, then use ordinary Git. Names can be nested, +and the `.git` suffix is optional; both URLs select the same repository. + +```sh +curl --fail-with-body -u git:local-git-password \ + -H 'Content-Type: application/json' \ + -d '{"format":"sha1","default_branch":"main"}' \ + http://127.0.0.1:19100/team/project/create +git push http://127.0.0.1:19100/team/project HEAD:main +``` + +Creation requires write permission and publishes the format and default branch +through the repository's WAL. Defaults are SHA-1 and `main`; choose SHA-256 for a +SHA-256 client repository. A successful creation returns 201; an existing +repository returns 409. A pending publication returns 503: check the repository +before retrying. Reads and ordinary pushes to a missing name do not create it. +The two explicitly configured demo repositories also support first-push setup. + +One `"*"` policy grants access to new names without a configuration edit or +restart. For Cognito, configure the groups once: ```toml -git_repositories = '{"team/project.git":{"log_id":"team-project","format":"sha1","default_branch":"main","read_groups":["developers"],"write_groups":["developers"],"admin_groups":["operators"]}}' +git_repositories = '{"*":{"read_groups":["developers"],"write_groups":["developers"],"admin_groups":["operators"]},"team/private":{"read_groups":["private-team"],"write_groups":["private-team"]}}' ``` -Each name has its own WAL identity. Duplicate identities, ambiguous names and -invalid formats are rejected. Add another entry and reload Spin to provision a -repository without rebuilding. Keep each identity stable. The first authorized -push discovery persists its format and default branch; subsequent configuration -cannot reinterpret that stored format or change its default branch. +An exact entry replaces the whole default policy; omitted groups deny access. +An omitted `log_id` derives an isolated identity from the canonical name, so +adding a permission override preserves the existing repository. Explicit IDs +must be unique and cannot use the reserved `auto-` prefix. An optional `format` +pins that policy to SHA-1 or SHA-256; otherwise readers recover the stored format. +Creation settings cannot change permissions or reinterpret an existing repository. +The local shared-password mode grants the password holder access to all names; +anonymous mode allows public creation and writes. + +The optional host maintenance worker currently visits named configuration entries. +Repositories created under `"*"` still checkpoint their log during pushes and expose +the same maintenance endpoints, but need an operator to schedule collection by name. Cognito mode checks signed access tokens, issuer, client, expiry and scope before opening storage. Git supplies the access token as its Basic password through an diff --git a/examples/git/codec.go b/examples/git/codec.go index d5e0b32..b5c8011 100644 --- a/examples/git/codec.go +++ b/examples/git/codec.go @@ -25,16 +25,20 @@ func decodeRoot(data []byte, format config.ObjectFormat, children int) (rootMeta if err := decodeMetadata(data, &meta); err != nil { return meta, err } - if !meta.Validated || meta.Format != format || len(meta.Buckets) != children { + validFormat := meta.Format == config.SHA1 || meta.Format == config.SHA256 + if !meta.Validated || !validFormat || len(meta.Buckets) != children { return meta, fmt.Errorf("invalid repository root") } + if format != "" && meta.Format != format { + return meta, fmt.Errorf("%w: repository format differs from stored state", config.ErrInvalidObjectFormat) + } if meta.Head != "" { head := plumbing.ReferenceName(meta.Head) if !head.IsBranch() || head.Validate() != nil { return meta, fmt.Errorf("invalid repository head") } } - if err := validateRefs(format, meta.Refs); err != nil { + if err := validateRefs(meta.Format, meta.Refs); err != nil { return meta, err } for i, prefix := range meta.Buckets { diff --git a/examples/git/codec_test.go b/examples/git/codec_test.go index f811f69..bced052 100644 --- a/examples/git/codec_test.go +++ b/examples/git/codec_test.go @@ -21,6 +21,31 @@ type closeProbe struct { closeErr error } +func TestRootInfersOnlyAuthenticatedSupportedFormat(t *testing.T) { + for _, format := range []config.ObjectFormat{config.SHA1, config.SHA256} { + data, err := json.Marshal(rootMeta{Validated: true, Format: format, Head: "refs/heads/main", Refs: map[string]string{ + "refs/heads/main": strings.Repeat("1", format.HexSize()), + }}) + if err != nil { + t.Fatal(err) + } + root, err := decodeRoot(data, "", 0) + if err != nil || root.Format != format { + t.Fatalf("stored format not recovered: %v", err) + } + other := config.SHA1 + if format == other { + other = config.SHA256 + } + if _, err := decodeRoot(data, other, 0); !errors.Is(err, config.ErrInvalidObjectFormat) { + t.Fatal("configured format mismatch accepted") + } + } + if _, err := decodeRoot([]byte(`{"Validated":true,"Format":"sha512","Refs":{}}`), "", 0); err == nil { + t.Fatal("unsupported stored format accepted") + } +} + func (r *closeProbe) Close() error { r.closed = true; return r.closeErr } func TestLooseRoundTripAndValidation(t *testing.T) { diff --git a/examples/git/main.go b/examples/git/main.go index 2be44cc..a4b8740 100644 --- a/examples/git/main.go +++ b/examples/git/main.go @@ -133,6 +133,7 @@ func serve(response http.ResponseWriter, r *http.Request) { return } service, method := route.Service, route.Method + creating := service == "create" maintenance := service == "maintenance" collect := service == "collect" pruneInvalidRefs := service == "prune-invalid-refs" @@ -150,7 +151,7 @@ func serve(response http.ResponseWriter, r *http.Request) { http.Error(response, message, status) return } - if limits.readOnly && (service == transport.ReceivePackService || maintenance || collect || pruneInvalidRefs) { + if limits.readOnly && (route.Action == gitWrite || maintenance || collect || pruneInvalidRefs) { http.Error(response, "repository is read-only", http.StatusForbidden) return } @@ -164,15 +165,26 @@ func serve(response http.ResponseWriter, r *http.Request) { http.Error(response, err.Error(), operationStatus(err)) return } + if creating { + route.Repository, e = repositoryCreation(r.Body, route.Repository) + if e != nil { + status := operationStatus(e) + if status == http.StatusInternalServerError { + status = http.StatusBadRequest + } + http.Error(response, e.Error(), status) + return + } + } + settings, e := walSettings(getConfig, route.Repository.LogID, limits) if e != nil { http.Error(response, e.Error(), http.StatusInternalServerError) return } session, e := unwrap(func() wt.Result[*wal.Session, wal.Failure] { - // First-push discovery requires an empty repository advertisement. Only - // a configured repository and an authorized writer may create its head. - if route.Action == gitWrite { + // Only creation and explicitly provisioned writer requests may open a log. + if creating || route.Action == gitWrite && route.Configured { return wal.Open(settings) } return wal.OpenExisting(settings) @@ -274,7 +286,11 @@ func serve(response http.ResponseWriter, r *http.Request) { s, e := retryOpenStore(open, refresh) if e != nil { log.Printf("git request setup failed stage=open-store: %v", e) - http.Error(w, "Git storage unavailable", operationStatus(e)) + status := operationStatus(e) + if creating && errors.Is(e, config.ErrInvalidObjectFormat) { + status = http.StatusConflict + } + http.Error(w, "Git storage unavailable", status) return } defer func() { @@ -282,8 +298,33 @@ func serve(response http.ResponseWriter, r *http.Request) { s.Close() } }() + if creating { + if s.stateRoot != nil { + http.Error(w, "repository already exists", http.StatusConflict) + return + } + if err := s.publish(s.meta.Refs); err != nil { + var pending *pendingError + if errors.As(err, &pending) { + http.Error(w, "repository creation pending; check the repository before retrying", http.StatusServiceUnavailable) + return + } + status := operationStatus(err) + if errors.Is(err, errPublicationConflict) { + status = http.StatusConflict + } + http.Error(w, "repository creation was not confirmed", status) + return + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusCreated) + _ = json.NewEncoder(w).Encode(struct { + Repository string `json:"repository"` + }{route.Name}) + return + } if s.stateRoot == nil { - if route.Action != gitWrite { + if route.Action != gitWrite || !route.Configured { http.NotFound(w, r) return } diff --git a/examples/git/qualification/aws/hosting.tf b/examples/git/qualification/aws/hosting.tf index f94f928..bbde410 100644 --- a/examples/git/qualification/aws/hosting.tf +++ b/examples/git/qualification/aws/hosting.tf @@ -49,10 +49,10 @@ variable "host_git_max_object_bytes" { } variable "host_repositories" { - description = "Explicit repository provisioning and independent group permissions." + description = "Repository policies; '*' supplies defaults for API-created repositories." type = map(object({ - log_id = string - format = string + log_id = optional(string, "") + format = optional(string, "") default_branch = optional(string, "main") read_groups = optional(list(string), []) write_groups = optional(list(string), []) @@ -60,8 +60,8 @@ variable "host_repositories" { })) default = {} validation { - condition = alltrue([for repo in var.host_repositories : contains(["sha1", "sha256"], repo.format)]) - error_message = "Repository format must be sha1 or sha256." + condition = alltrue([for repo in var.host_repositories : contains(["", "sha1", "sha256"], repo.format)]) + error_message = "Repository format must be empty, sha1 or sha256." } } @@ -183,7 +183,7 @@ locals { token_url = "${local.cognito_login_origin}/oauth2/token" service_url = "http://127.0.0.1:8081" validation_url = "http://127.0.0.1:3000/_validate_backend" - repositories = sort(keys(var.host_repositories)) + repositories = sort([for name in keys(var.host_repositories) : name if name != "*"]) budget_seconds = var.host_maintenance_budget_seconds pause_seconds = var.host_maintenance_pause_seconds })) @@ -192,7 +192,7 @@ locals { retry_after = max(1, length(var.host_repositories) * var.host_maintenance_pause_seconds) admin_paths = jsonencode(flatten([for name in sort(keys(var.host_repositories)) : [ for operation in ["maintenance", "collect", "prune-invalid-refs", "recover-retentions-after-drain"] : "/${name}/${operation}" - ]])) + ] if name != "*"])) }) : "" } diff --git a/examples/git/repositories.go b/examples/git/repositories.go index aa4a317..631d376 100644 --- a/examples/git/repositories.go +++ b/examples/git/repositories.go @@ -2,6 +2,7 @@ package main import ( "bytes" + "crypto/sha256" "encoding/json" "errors" "fmt" @@ -35,8 +36,7 @@ type repositoryConfig struct { repositoryAccess } -// GIT_REPOSITORIES is an object keyed by canonical names such as team/project.git. -// Missing configuration provides no repositories. There are no implicit aliases. +// Exact policies override "*". Bare names and their .git URLs identify one log. func loadRepositories(getenv func(string) string) (map[string]repositoryConfig, error) { text := getenv("GIT_REPOSITORIES") if len(text) > repositoriesConfigBytes || !utf8.ValidString(text) { @@ -52,17 +52,32 @@ func loadRepositories(getenv func(string) string) (map[string]repositoryConfig, } identities := map[string]string{} for name, encoded := range entries { - if !validRepositoryName(name) { + if name != "*" && !validRepositoryName(name) { return nil, fmt.Errorf("GIT_REPOSITORIES: noncanonical repository name %q", name) } + if name != "*" { + name = canonicalRepositoryName(name) + } + if _, exists := repositories[name]; exists { + return nil, fmt.Errorf("duplicate repository name %q", name) + } repository, err := parseRepository(encoded) if err != nil { return nil, fmt.Errorf("GIT_REPOSITORIES: repository %q: %w", name, err) } - if other, exists := identities[repository.LogID]; exists { - return nil, fmt.Errorf("repositories %q and %q share log_id %q", other, name, repository.LogID) + if name == "*" { + if repository.LogID != "" { + return nil, errors.New("wildcard policy must not specify log_id") + } + } else { + if repository.LogID == "" { + repository.LogID = automaticRepositoryID(name) + } + if other, exists := identities[repository.LogID]; exists { + return nil, fmt.Errorf("repositories %q and %q share log_id %q", other, name, repository.LogID) + } + identities[repository.LogID] = name } - identities[repository.LogID] = name repositories[name] = repository } return repositories, nil @@ -88,10 +103,11 @@ func parseRepository(encoded []byte) (repositoryConfig, error) { return repository, err } validID := repositoryLogID.MatchString(repository.LogID) - if !validID || repository.LogID == "." || repository.LogID == ".." { + if repository.LogID != "" && (!validID || repository.LogID == "." || repository.LogID == ".." || + strings.HasPrefix(repository.LogID, "auto-")) { return repository, errors.New("log_id must satisfy the WAL log identifier contract") } - if repository.Format != config.SHA1 && repository.Format != config.SHA256 { + if repository.Format != "" && repository.Format != config.SHA1 && repository.Format != config.SHA256 { return repository, config.ErrInvalidObjectFormat } if repository.DefaultBranch != "" { @@ -109,6 +125,43 @@ func parseRepository(encoded []byte) (repositoryConfig, error) { return repository, nil } +func repositoryCreation(body io.Reader, policy repositoryConfig) (repositoryConfig, error) { + data, err := io.ReadAll(io.LimitReader(body, 4097)) + if err != nil { + return policy, err + } + if len(data) > 4096 { + return policy, errObjectLimit + } + fields, err := uniqueRepositoryObject(data) + if err != nil { + return policy, err + } + for name := range fields { + if name != "format" && name != "default_branch" { + return policy, fmt.Errorf("unknown creation field %q", name) + } + } + creation, err := parseRepository(data) + if err != nil { + return policy, err + } + if creation.Format == "" { + creation.Format = policy.Format + } + if creation.Format == "" { + creation.Format = config.SHA1 + } + if policy.Format != "" && policy.Format != creation.Format { + return policy, config.ErrInvalidObjectFormat + } + policy.Format = creation.Format + if creation.DefaultBranch != "" { + policy.DefaultBranch = creation.DefaultBranch + } + return policy, nil +} + // Decode only an object, rejecting duplicate keys before normal JSON decoding // could silently overwrite them. Both repository names and fields use this path. func uniqueRepositoryObject(encoded []byte) (map[string]json.RawMessage, error) { @@ -146,7 +199,7 @@ func uniqueRepositoryObject(encoded []byte) (map[string]json.RawMessage, error) } func validRepositoryName(name string) bool { - if !strings.HasSuffix(name, ".git") || strings.Contains(name, `\`) { + if len(name) == 0 || len(canonicalRepositoryName(name)) > 4096 || name == "*" || strings.Contains(name, `\`) { return false } path := &url.URL{Path: name} @@ -162,9 +215,21 @@ func validRepositoryName(name string) bool { return segments[len(segments)-1] != ".git" } +func canonicalRepositoryName(name string) string { + if !strings.HasSuffix(name, ".git") { + return name + ".git" + } + return name +} + +func automaticRepositoryID(name string) string { + return fmt.Sprintf("auto-%x", sha256.Sum256([]byte(name))) +} + type repositoryRoute struct { Name string Repository repositoryConfig + Configured bool Service string Method string Action gitAction @@ -183,7 +248,7 @@ func resolveRepository(repositories map[string]repositoryConfig, r *http.Request } for _, service := range []string{ "info/refs", transport.UploadPackService, transport.ReceivePackService, - "maintenance", "collect", "prune-invalid-refs", "recover-retentions-after-drain", + "create", "maintenance", "collect", "prune-invalid-refs", "recover-retentions-after-drain", } { suffix := "/" + service if strings.HasSuffix(r.URL.Path, suffix) { @@ -192,8 +257,17 @@ func resolveRepository(repositories map[string]repositoryConfig, r *http.Request break } } + if !validRepositoryName(route.Name) { + return repositoryRoute{}, errRepositoryNotFound + } + route.Name = canonicalRepositoryName(route.Name) repository, ok := repositories[route.Name] - if !ok || !validRepositoryName(route.Name) { + route.Configured = ok + if !ok { + repository, ok = repositories["*"] + repository.LogID = automaticRepositoryID(route.Name) + } + if !ok { return repositoryRoute{}, errRepositoryNotFound } route.Repository = repository @@ -215,7 +289,7 @@ func resolveRepository(repositories map[string]repositoryConfig, r *http.Request switch route.Service { case transport.UploadPackService: route.Action = gitRead - case transport.ReceivePackService: + case transport.ReceivePackService, "create": route.Action = gitWrite default: route.Action = gitAdmin diff --git a/examples/git/repositories_test.go b/examples/git/repositories_test.go index ca1279c..5748d78 100644 --- a/examples/git/repositories_test.go +++ b/examples/git/repositories_test.go @@ -78,8 +78,6 @@ func TestLoadRepositoriesRejectsInvalidConfiguration(t *testing.T) { "a.git":{"log_id":"same","format":"sha1"}, "b.git":{"log_id":"same","format":"sha256"} }`}, - {name: "missing identity", text: `{"r.git":{"format":"sha1"}}`}, - {name: "missing format", text: `{"r.git":{"log_id":"one"}}`}, {name: "invalid format", text: `{"r.git":{"log_id":"one","format":"SHA1"}}`}, {name: "null format", text: `{"r.git":{"log_id":"one","format":null}}`}, {name: "path identity", text: `{"r.git":{"log_id":"a/b","format":"sha1"}}`}, @@ -109,7 +107,7 @@ func TestLoadRepositoriesRejectsInvalidConfiguration(t *testing.T) { func TestLoadRepositoriesRejectsNoncanonicalNames(t *testing.T) { t.Parallel() for _, name := range []string{ - "", "r", ".git", "team/.git", "/r.git", "r.git/", "team//r.git", "./r.git", + "", ".git", "team/.git", "/r.git", "r.git/", "team//r.git", "./r.git", "team/./r.git", "team/../r.git", "../r.git", "r%2egit", `team\r.git`, "r?.git", "r#.git", "r space.git", "résumé.git", } { @@ -182,6 +180,7 @@ func TestResolveRepositorySelectsServiceAndAction(t *testing.T) { service: transport.ReceivePackService, action: gitWrite}, {name: "read RPC", method: http.MethodPost, path: "git-upload-pack", service: transport.UploadPackService, action: gitRead}, + {name: "creation", method: http.MethodPost, path: "create", service: "create", action: gitWrite}, {name: "write RPC", method: http.MethodPost, path: "git-receive-pack", service: transport.ReceivePackService, action: gitWrite}, {name: "logical maintenance", method: http.MethodPost, path: "maintenance", service: "maintenance", action: gitAdmin}, @@ -285,3 +284,88 @@ func TestRepositoryRoutePolicyIsIndependentPerRepositoryAndAction(t *testing.T) }) } } + +func TestAutomaticRepositoryPolicyAndIdentity(t *testing.T) { + if !validRepositoryName(strings.Repeat("a", 4092)) || validRepositoryName(strings.Repeat("a", 4093)) { + t.Fatal("name limit must include the optional .git suffix") + } + repositories, err := repositoriesFromText(`{"*":{"write_groups":["writers"]},"team/private":{}}`) + if err != nil { + t.Fatal(err) + } + var first repositoryRoute + for _, name := range []string{"team/project", "team/project.git"} { + route, err := resolveRepository(repositories, httptest.NewRequest(http.MethodPost, "/"+name+"/create", nil)) + if err != nil { + t.Fatal(err) + } + if route.Configured || route.Name != "team/project.git" || route.Action != gitWrite || route.Repository.Format != "" { + t.Fatalf("unexpected automatic route: %+v", route) + } + if first.Name != "" && first.Repository.LogID != route.Repository.LogID { + t.Fatal("URL aliases select different logs") + } + first = route + } + principal := gitPrincipal{subject: "writer", groups: []string{"writers"}} + if !principal.Allows(first.Repository.repositoryAccess, gitWrite) || principal.Allows(first.Repository.repositoryAccess, gitRead) { + t.Fatal("automatic policy lost independent permissions") + } + private, err := resolveRepository(repositories, httptest.NewRequest(http.MethodPost, "/team/private.git/create", nil)) + if err != nil { + t.Fatal(err) + } + if principal.Allows(private.Repository.repositoryAccess, gitWrite) || private.Repository.LogID == first.Repository.LogID { + t.Fatal("exact denial or repository isolation lost") + } + for _, name := range []string{"*", "team/../project", "team//project", "%70roject"} { + if _, err := resolveRepository(repositories, httptest.NewRequest(http.MethodPost, "/"+name+"/create", nil)); err == nil { + t.Fatalf("wildcard accepted invalid name %q", name) + } + } + // Adding a permission override must preserve the automatically created log. + override, err := repositoriesFromText(`{"team/project":{"read_groups":["readers"]}}`) + if err != nil || override["team/project.git"].LogID != first.Repository.LogID { + t.Fatal("permission override moved storage") + } + for _, text := range []string{ + `{"*":{"log_id":"shared"}}`, + `{"project":{},"project.git":{}}`, + `{"project":{"log_id":"auto-forbidden"}}`, + } { + if _, err := repositoriesFromText(text); err == nil { + t.Fatalf("accepted ambiguous policy: %s", text) + } + } +} + +func TestRepositoryCreationSettings(t *testing.T) { + policy := repositoryConfig{LogID: "isolated", DefaultBranch: "release", repositoryAccess: repositoryAccess{WriteGroups: []string{"writers"}}} + for _, format := range []config.ObjectFormat{config.SHA1, config.SHA256} { + created, err := repositoryCreation(strings.NewReader(`{"format":"`+string(format)+`","default_branch":"main"}`), policy) + if err != nil { + t.Fatal(err) + } + if created.Format != format || created.LogID != policy.LogID || created.DefaultBranch != "main" || + len(created.WriteGroups) != 1 || created.WriteGroups[0] != "writers" { + t.Fatalf("creation changed policy: %+v", created) + } + } + created, err := repositoryCreation(strings.NewReader(`{}`), policy) + if err != nil || created.Format != config.SHA1 || created.DefaultBranch != "release" { + t.Fatal("creation defaults lost") + } + for _, text := range []string{ + `{"format":"sha512"}`, `{"format":null}`, `{"format":"sha1","format":"sha256"}`, + `{"default_branch":"../main"}`, `{"write_groups":["everyone"]}`, `{"log_id":"other"}`, + `{} {}`, strings.Repeat(" ", 4097), + } { + if _, err := repositoryCreation(strings.NewReader(text), policy); err == nil { + t.Fatalf("invalid creation accepted: %s", text) + } + } + policy.Format = config.SHA256 + if _, err := repositoryCreation(strings.NewReader(`{"format":"sha1"}`), policy); err == nil { + t.Fatal("creation bypassed configured format") + } +} diff --git a/examples/git/spin.toml b/examples/git/spin.toml index 49a978c..581d4fa 100644 --- a/examples/git/spin.toml +++ b/examples/git/spin.toml @@ -15,7 +15,7 @@ wal_secret_key = { default = "", secret = true } wal_session_token = { default = "", secret = true } wal_max_collection_objects = { default = "100000" } wal_recover_retentions_after_drain = { default = "false" } -git_repositories = { default = '{"sha1.git":{"log_id":"repo-sha1","format":"sha1"},"sha256.git":{"log_id":"repo-sha256","format":"sha256"}}' } +git_repositories = { default = '{"*":{},"sha1.git":{"log_id":"repo-sha1","format":"sha1"},"sha256.git":{"log_id":"repo-sha256","format":"sha256"}}' } git_auth_mode = { default = "password" } git_cognito_issuer = { default = "" } git_cognito_client_id = { default = "" } diff --git a/examples/git/store.go b/examples/git/store.go index cfedc56..e1b33ed 100644 --- a/examples/git/store.go +++ b/examples/git/store.go @@ -12,6 +12,7 @@ import ( "strings" "github.com/go-git/go-git/v6/plumbing" + "github.com/go-git/go-git/v6/plumbing/format/config" "github.com/go-git/go-git/v6/plumbing/format/objfile" "github.com/go-git/go-git/v6/plumbing/storer" "github.com/go-git/go-git/v6/storage" @@ -46,12 +47,7 @@ func openStore(ctx context.Context, session *wal.Session, repository repositoryC return nil, err } format := repository.Format - mem := memory.NewStorage(memory.WithObjectFormat(format)) - // Memory storage returns its owned configuration; no save is needed. - cfg, _ := mem.Config() - // Reuse stored deltas without comparing object contents to generate new ones. - cfg.Pack.Window = 1 - s := &store{ctx: ctx, limits: limits, Storer: mem, session: session, buckets: map[string]*wal.Object{}, loaded: map[*wal.Object]radixNode[indexed, *wal.Object]{}, pending: map[string]indexed{}} + s := &store{ctx: ctx, limits: limits, session: session, buckets: map[string]*wal.Object{}, loaded: map[*wal.Object]radixNode[indexed, *wal.Object]{}, pending: map[string]indexed{}} defer func() { if result == nil { s.Close() @@ -78,10 +74,17 @@ func openStore(ctx context.Context, session *wal.Session, repository repositoryC for i, key := range s.meta.Buckets { s.buckets[key] = root.Objects[i] } - for name, id := range s.meta.Refs { - if e := s.Storer.SetReference(plumbing.NewHashReference(plumbing.ReferenceName(name), plumbing.NewHash(id))); e != nil { - return nil, e - } + } + if s.meta.Format == "" { + s.meta.Format = config.SHA1 + } + s.Storer = memory.NewStorage(memory.WithObjectFormat(s.meta.Format)) + cfg, _ := s.Storer.Config() + // Reuse stored deltas without comparing object contents to generate new ones. + cfg.Pack.Window = 1 + for name, id := range s.meta.Refs { + if e := s.Storer.SetReference(plumbing.NewHashReference(plumbing.ReferenceName(name), plumbing.NewHash(id))); e != nil { + return nil, e } } // An empty repository has no unchecked objects. Existing roots must certify validation. diff --git a/examples/git/tests/creation_test.go b/examples/git/tests/creation_test.go new file mode 100644 index 0000000..60a8b78 --- /dev/null +++ b/examples/git/tests/creation_test.go @@ -0,0 +1,131 @@ +package tests + +import ( + "bytes" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestRepositoryCreation(t *testing.T) { + endpoint := os.Getenv("GIT_PROBE_URL") + if endpoint == "" || os.Getenv("GIT_PROBE_CREATE") != "1" { + t.Skip("set GIT_PROBE_URL and GIT_PROBE_CREATE=1 for a local service with wildcard policy") + } + request := func(method, url, password, body string, want int) { + t.Helper() + r, err := http.NewRequest(method, url, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + r.SetBasicAuth("git", password) + r.Header.Set("Content-Type", "application/json") + response, err := http.DefaultClient.Do(r) + if err != nil { + t.Fatal(err) + } + defer response.Body.Close() + data, err := io.ReadAll(response.Body) + if err != nil || response.StatusCode != want { + t.Fatalf("%s: HTTP %d, want %d: %s (%v)", url, response.StatusCode, want, data, err) + } + } + for _, format := range []string{"sha1", "sha256"} { + t.Run(format, func(t *testing.T) { + name := fmt.Sprintf("created/%s-%d", format, time.Now().UnixNano()) + url := strings.TrimRight(endpoint, "/") + "/" + name + password := os.Getenv("GIT_PROBE_PASSWORD") + settings := `{"format":"` + format + `","default_branch":"release"}` + request(http.MethodPost, url+"/create", "wrong-password", settings, 401) + request(http.MethodGet, url+"/info/refs?service=git-upload-pack", password, "", 404) + request(http.MethodGet, url+"/info/refs?service=git-receive-pack", password, "", 404) + request(http.MethodPost, url+"/maintenance", password, "{}", 404) + request(http.MethodPost, url+"/create", password, strings.Repeat(" ", 4097), 413) + request(http.MethodPost, url+"/create", password, `{"log_id":"override"}`, 400) + request(http.MethodPost, url+"/create", password, settings, 201) + request(http.MethodPost, url+".git/create", password, settings, 409) + other := "sha1" + if format == other { + other = "sha256" + } + request(http.MethodPost, url+"/create", password, `{"format":"`+other+`"}`, 409) + root := t.TempDir() + git(t, nil, "init", "--object-format="+format, "-b", "release", root) + write(t, filepath.Join(root, "README.md"), []byte(name)) + git(t, nil, "-C", root, "add", ".") + git(t, nil, "-C", root, "commit", "-m", "created through API") + git(t, nil, "-C", root, "push", url, "HEAD:refs/heads/release") + clone := filepath.Join(t.TempDir(), "clone") + git(t, nil, "-c", "protocol.version=2", "clone", url+".git", clone) + git(t, nil, "-C", clone, "fsck", "--full") + if got := git(t, nil, "-C", clone, "symbolic-ref", "--short", "HEAD"); strings.TrimSpace(string(got)) != "release" { + t.Fatal("created default branch was lost") + } + if got := git(t, nil, "-C", clone, "show", "HEAD:README.md"); !bytes.Equal(got, []byte(name)) { + t.Fatal("created repository data differs") + } + }) + } + t.Run("concurrent creators", func(t *testing.T) { + url := fmt.Sprintf("%s/created/race-%d", strings.TrimRight(endpoint, "/"), time.Now().UnixNano()) + type result struct { + format string + status int + err error + } + results := make(chan result, 2) + for _, format := range []string{"sha1", "sha256"} { + go func() { + r, err := http.NewRequest(http.MethodPost, url+"/create", strings.NewReader(`{"format":"`+format+`"}`)) + if err != nil { + results <- result{format: format, err: err} + return + } + r.SetBasicAuth("git", os.Getenv("GIT_PROBE_PASSWORD")) + response, err := http.DefaultClient.Do(r) + status := 0 + if err == nil { + status = response.StatusCode + _, _ = io.Copy(io.Discard, response.Body) + err = response.Body.Close() + } + results <- result{format: format, status: status, err: err} + }() + } + winner := "" + for range 2 { + r := <-results + if r.err != nil || (r.status != 201 && r.status != 409) { + t.Fatalf("concurrent creation: %+v", r) + } + if r.status == 201 { + if winner != "" { + t.Fatal("two creations committed") + } + winner = r.format + } + } + if winner == "" { + t.Fatal("no creation committed") + } + r, err := http.NewRequest(http.MethodGet, url+"/info/refs?service=git-receive-pack", nil) + if err != nil { + t.Fatal(err) + } + r.SetBasicAuth("git", os.Getenv("GIT_PROBE_PASSWORD")) + response, err := http.DefaultClient.Do(r) + if err != nil { + t.Fatal(err) + } + defer response.Body.Close() + data, err := io.ReadAll(response.Body) + if err != nil || response.StatusCode != 200 || !bytes.Contains(data, []byte("object-format="+winner)) { + t.Fatal("stored format does not match winning creator") + } + }) +}