Skip to content

Commit efbc453

Browse files
authored
feat(config): Add fingerprint ignore tags to some config fields (#46)
1 parent d08fdc0 commit efbc453

File tree

8 files changed

+179
-10
lines changed

8 files changed

+179
-10
lines changed

.github/instructions/go.instructions.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,18 @@ description: "Instructions for working on the azldev Go codebase. IMPORTANT: Alw
9696
}
9797
```
9898

99+
## Component Fingerprinting`fingerprint:"-"` Tags
100+
101+
Structs in `internal/projectconfig/` are hashed by `hashstructure.Hash()` to detect component changes. Fields **included by default** (safe: false positive > false negative).
102+
103+
When adding a new field to a fingerprinted struct, ask: **"Does changing this field change the build output?"**
104+
- **Yes** (build flags, spec source, defines, etc.) → do nothing, included automatically.
105+
- **No** (human docs, scheduling hints, CI policy, metadata, back-references) → add `fingerprint:"-"` to the struct tag and register the exclusion in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
106+
107+
If a parent struct field is already excluded (e.g. `Failure ComponentBuildFailureConfig ... fingerprint:"-"`), do **not** also tag the inner struct's fields — `hashstructure` skips the entire subtree.
108+
109+
Run `mage unit` to verify — the guard test will catch unregistered exclusions or missing tags.
110+
99111
### Cmdline Returns
100112
101113
CLI commands should return meaningful structured results. azldev has output formatting helpers to facilitate this (for example, `RunFunc*` wrappers handle formatting, so callers typically should not call `reflectable.FormatValue` directly).
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Component Identity & Change Detection
2+
3+
`azldev` computes a fingerprint for each component based on its config and sources. This enables:
4+
5+
- **Change detection**: identify which components have changed between two commits or branches, even if the change is a non-obvious config inheritance.
6+
- **Build optimization**: only rebuild changed components and their dependents, skipping unchanged ones.
7+
- **Automatic release bumping**: increment the release tag of changed packages automatically, and include the commits that caused the change in the changelog.
8+
9+
## Fingerprint Inputs
10+
11+
A component's fingerprint is a SHA256 combining:
12+
13+
1. **Config hash**`hashstructure.Hash()` of the resolved `ComponentConfig` (after all merging). Fields tagged `fingerprint:"-"` are excluded.
14+
2. **Source identity** — content hash for local specs (all files in the spec directory), commit hash for upstream.
15+
3. **Overlay file hashes** — SHA256 of each file referenced by overlay `Source` fields.
16+
4. **Distro name + version**
17+
5. **Manual release bump counter** — increments with each manual release bump, ensuring a new fingerprint even if there are no config or source changes.
18+
19+
Global change propagation works automatically: the fingerprint operates on the fully-merged config, so a change to a distro or group default changes the resolved config of every inheriting component.
20+
21+
## `fingerprint:"-"` Tag System
22+
23+
The `hashstructure` library uses `TagName: "fingerprint"`. Untagged fields are **included by default** (safe default: false positive > false negative).
24+
25+
A guard test (`TestAllFingerprintedFieldsHaveDecision`) reflects over all fingerprinted structs and maintains a bi-directional allowlist of exclusions. It fails if a `fingerprint:"-"` tag is added without registering it, or if a registered exclusion's tag is removed.
26+
27+
### Adding a New Config Field
28+
29+
1. Add the field to the struct in `internal/projectconfig/`.
30+
2. **If NOT a build input**: add `fingerprint:"-"` to the struct tag and register it in `expectedExclusions` in `internal/projectconfig/fingerprint_test.go`.
31+
3. **If a build input**: do nothing — included by default.
32+
4. Run `mage unit`.
33+
34+
### Adding a New Source Type
35+
36+
1. Implement `SourceIdentityProvider` on your provider (see `ResolveLocalSourceIdentity` in `localidentity.go` for a simple example).
37+
2. Add a case to `sourceManager.ResolveSourceIdentity()` in `sourcemanager.go`.
38+
3. Add tests in `identityprovider_test.go`.
39+
40+
## Known Limitations
41+
42+
- It is difficult to determine WHY a diff occurred (e.g., which specific field changed) since the fingerprint is a single opaque hash. The JSON output includes an `inputs` breakdown (`configHash`, `sourceIdentity`, `overlayFileHashes`, etc.) that can help narrow it down by comparing the two identity files manually.

internal/projectconfig/build.go

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ type CheckConfig struct {
1313
// Skip indicates whether the %check section should be disabled for this component.
1414
Skip bool `toml:"skip,omitempty" json:"skip,omitempty" jsonschema:"title=Skip check,description=Disables the %check section by prepending 'exit 0' when set to true"`
1515
// SkipReason provides a required justification when Skip is true.
16-
SkipReason string `toml:"skip_reason,omitempty" json:"skipReason,omitempty" jsonschema:"title=Skip reason,description=Required justification for skipping the %check section"`
16+
SkipReason string `toml:"skip_reason,omitempty" json:"skipReason,omitempty" jsonschema:"title=Skip reason,description=Required justification for skipping the %check section" fingerprint:"-"`
1717
}
1818

1919
// Validate checks that required fields are set when Skip is true.
@@ -43,9 +43,9 @@ type ComponentBuildConfig struct {
4343
// Check section configuration.
4444
Check CheckConfig `toml:"check,omitempty" json:"check,omitempty" jsonschema:"title=Check configuration,description=Configuration for the %check section"`
4545
// Failure configuration and policy for this component's build.
46-
Failure ComponentBuildFailureConfig `toml:"failure,omitempty" json:"failure,omitempty" jsonschema:"title=Build failure configuration,description=Configuration and policy regarding build failures for this component."`
46+
Failure ComponentBuildFailureConfig `toml:"failure,omitempty" json:"failure,omitempty" jsonschema:"title=Build failure configuration,description=Configuration and policy regarding build failures for this component." fingerprint:"-"`
4747
// Hints for how or when to build the component; must not be required for correctness of builds.
48-
Hints ComponentBuildHints `toml:"hints,omitempty" json:"hints,omitempty" jsonschema:"title=Build hints,description=Non-essential hints for how or when to build the component."`
48+
Hints ComponentBuildHints `toml:"hints,omitempty" json:"hints,omitempty" jsonschema:"title=Build hints,description=Non-essential hints for how or when to build the component." fingerprint:"-"`
4949
}
5050

5151
// ComponentBuildFailureConfig encapsulates configuration and policy regarding a component's

internal/projectconfig/component.go

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ type Origin struct {
4949
// SourceFileReference encapsulates a reference to a specific source file artifact.
5050
type SourceFileReference struct {
5151
// Reference to the component to which the source file belongs.
52-
Component ComponentReference `toml:"-" json:"-"`
52+
Component ComponentReference `toml:"-" json:"-" fingerprint:"-"`
5353

5454
// Name of the source file; must be non-empty.
5555
Filename string `toml:"filename" json:"filename"`
@@ -61,7 +61,7 @@ type SourceFileReference struct {
6161
HashType fileutils.HashType `toml:"hash-type,omitempty" json:"hashType,omitempty" jsonschema:"enum=SHA256,enum=SHA512,title=Hash type,description=Hash algorithm used for the hash value"`
6262

6363
// Origin for this source file. When omitted, the file is resolved via the lookaside cache.
64-
Origin Origin `toml:"origin,omitempty" json:"origin,omitempty"`
64+
Origin Origin `toml:"origin,omitempty" json:"origin,omitempty" fingerprint:"-"`
6565
}
6666

6767
// Defines a component group. Component groups are logical groupings of components (see [ComponentConfig]).
@@ -111,11 +111,11 @@ func (g ComponentGroupConfig) WithAbsolutePaths(referenceDir string) ComponentGr
111111
// Defines a component.
112112
type ComponentConfig struct {
113113
// The component's name; not actually present in serialized files.
114-
Name string `toml:"-" json:"name" table:",sortkey"`
114+
Name string `toml:"-" json:"name" table:",sortkey" fingerprint:"-"`
115115

116116
// Reference to the source config file that this definition came from; not present
117117
// in serialized files.
118-
SourceConfigFile *ConfigFile `toml:"-" json:"-" table:"-"`
118+
SourceConfigFile *ConfigFile `toml:"-" json:"-" table:"-" fingerprint:"-"`
119119

120120
// RenderedSpecDir is the output directory for this component's rendered spec files.
121121
// Derived at resolve time from the project's rendered-specs-dir setting; not present

internal/projectconfig/distro.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ type DistroReference struct {
1818
// Version of the referenced distro.
1919
Version string `toml:"version,omitempty" json:"version,omitempty" jsonschema:"title=Version,description=Version of the referenced distro"`
2020
// Snapshot date/time for source code if specified components will use source as it existed at this time.
21-
Snapshot string `toml:"snapshot,omitempty" json:"snapshot,omitempty" jsonschema:"format=date-time,title=Snapshot,description=If specified use source code as it existed at this date/time"`
21+
Snapshot string `toml:"snapshot,omitempty" json:"snapshot,omitempty" jsonschema:"format=date-time,title=Snapshot,description=If specified use source code as it existed at this date/time" fingerprint:"-"`
2222
}
2323

2424
// Implements the [Stringer] interface for [DistroReference].
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
package projectconfig_test
5+
6+
import (
7+
"reflect"
8+
"testing"
9+
10+
"github.com/microsoft/azure-linux-dev-tools/internal/projectconfig"
11+
"github.com/stretchr/testify/assert"
12+
)
13+
14+
// TestAllFingerprintedFieldsHaveDecision verifies that every field in every
15+
// fingerprinted struct has been consciously categorized as either included
16+
// (no fingerprint tag) or excluded (`fingerprint:"-"`).
17+
//
18+
// This test serves two purposes:
19+
// 1. It ensures that newly added fields default to **included** in the fingerprint
20+
// (the safe default — you get a false positive, never a false negative).
21+
// 2. It catches accidental removal of `fingerprint:"-"` tags from excluded fields,
22+
// since all exclusions are tracked in expectedExclusions.
23+
func TestAllFingerprintedFieldsHaveDecision(t *testing.T) {
24+
// All struct types whose fields participate in component fingerprinting.
25+
// When adding a new struct that feeds into the fingerprint, add it here.
26+
fingerprintedStructs := []reflect.Type{
27+
reflect.TypeFor[projectconfig.ComponentConfig](),
28+
reflect.TypeFor[projectconfig.ComponentBuildConfig](),
29+
reflect.TypeFor[projectconfig.CheckConfig](),
30+
reflect.TypeFor[projectconfig.PackageConfig](),
31+
reflect.TypeFor[projectconfig.ComponentOverlay](),
32+
reflect.TypeFor[projectconfig.SpecSource](),
33+
reflect.TypeFor[projectconfig.DistroReference](),
34+
reflect.TypeFor[projectconfig.SourceFileReference](),
35+
}
36+
37+
// Maps "StructName.FieldName" for every field that should carry a
38+
// `fingerprint:"-"` tag. Catches accidental tag removal.
39+
//
40+
// Each entry documents WHY the field is excluded from the fingerprint:
41+
expectedExclusions := map[string]bool{
42+
// ComponentConfig.Name — metadata, already the map key in project config.
43+
"ComponentConfig.Name": true,
44+
// ComponentConfig.SourceConfigFile — internal bookkeeping reference, not a build input.
45+
"ComponentConfig.SourceConfigFile": true,
46+
47+
// ComponentBuildConfig.Failure — CI policy (expected failure tracking), not a build input.
48+
"ComponentBuildConfig.Failure": true,
49+
// ComponentBuildConfig.Hints — scheduling hints (e.g. expensive), not a build input.
50+
"ComponentBuildConfig.Hints": true,
51+
52+
// CheckConfig.SkipReason — human documentation for why check is skipped, not a build input.
53+
"CheckConfig.SkipReason": true,
54+
55+
// PackageConfig.Publish — post-build routing (where to publish), not a build input.
56+
"PackageConfig.Publish": true,
57+
58+
// ComponentOverlay.Description — human-readable documentation for the overlay.
59+
"ComponentOverlay.Description": true,
60+
61+
// SourceFileReference.Component — back-reference to parent, not a build input.
62+
"SourceFileReference.Component": true,
63+
64+
// DistroReference.Snapshot — snapshot timestamp is not a build input; the resolved
65+
// upstream commit hash (captured separately via SourceIdentity) is what matters.
66+
// Excluding this prevents a snapshot bump from marking all upstream components as changed.
67+
"DistroReference.Snapshot": true,
68+
69+
// SourceFileReference.Origin — download location metadata (URI, type), not a build input.
70+
// The file content is already captured by Filename + Hash; changing a CDN URL should not
71+
// trigger a rebuild.
72+
"SourceFileReference.Origin": true,
73+
}
74+
75+
// Collect all actual exclusions found via reflection, and flag invalid tag values.
76+
actualExclusions := make(map[string]bool)
77+
78+
for _, st := range fingerprintedStructs {
79+
for i := range st.NumField() {
80+
field := st.Field(i)
81+
key := st.Name() + "." + field.Name
82+
83+
tag := field.Tag.Get("fingerprint")
84+
85+
switch tag {
86+
case "":
87+
// No tag — included by default (the safe default).
88+
case "-":
89+
actualExclusions[key] = true
90+
default:
91+
// hashstructure only recognises "" (include) and "-" (exclude).
92+
// Any other value is silently treated as included, which is
93+
// almost certainly a typo.
94+
assert.Failf(t, "invalid fingerprint tag",
95+
"field %q has unrecognised fingerprint tag value %q — "+
96+
"only `fingerprint:\"-\"` (exclude) is valid; "+
97+
"remove the tag to include the field", key, tag)
98+
}
99+
}
100+
}
101+
102+
// Verify every expected exclusion is actually present.
103+
for key := range expectedExclusions {
104+
assert.Truef(t, actualExclusions[key],
105+
"expected field %q to have `fingerprint:\"-\"` tag, but it does not — "+
106+
"was the tag accidentally removed?", key)
107+
}
108+
109+
// Verify no unexpected exclusions exist.
110+
for key := range actualExclusions {
111+
assert.Truef(t, expectedExclusions[key],
112+
"field %q has `fingerprint:\"-\"` tag but is not in expectedExclusions — "+
113+
"add it to expectedExclusions if the exclusion is intentional", key)
114+
}
115+
}

internal/projectconfig/overlay.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ type ComponentOverlay struct {
1717
// The type of overlay to apply.
1818
Type ComponentOverlayType `toml:"type" json:"type" validate:"required" jsonschema:"enum=spec-add-tag,enum=spec-insert-tag,enum=spec-set-tag,enum=spec-update-tag,enum=spec-remove-tag,enum=spec-prepend-lines,enum=spec-append-lines,enum=spec-search-replace,enum=spec-remove-section,enum=patch-add,enum=patch-remove,enum=file-prepend-lines,enum=file-search-replace,enum=file-add,enum=file-remove,enum=file-rename,title=Overlay type,description=The type of overlay to apply"`
1919
// Human readable description of overlay; primarily present to document the need for the change.
20-
Description string `toml:"description,omitempty" json:"description,omitempty" jsonschema:"title=Description,description=Human readable description of overlay"`
20+
Description string `toml:"description,omitempty" json:"description,omitempty" jsonschema:"title=Description,description=Human readable description of overlay" fingerprint:"-"`
2121

2222
// For overlays that apply to non-spec files, indicates the filename. For overlays that can
2323
// apply to multiple files, supports glob patterns (including globstar).

internal/projectconfig/package.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ type PackagePublishConfig struct {
2424
// Currently only publish settings are supported; additional fields may be added in the future.
2525
type PackageConfig struct {
2626
// Publish holds the publish settings for this package.
27-
Publish PackagePublishConfig `toml:"publish,omitempty" json:"publish,omitempty" jsonschema:"title=Publish settings,description=Publishing settings for this binary package"`
27+
Publish PackagePublishConfig `toml:"publish,omitempty" json:"publish,omitempty" jsonschema:"title=Publish settings,description=Publishing settings for this binary package" fingerprint:"-"`
2828
}
2929

3030
// MergeUpdatesFrom updates the package config with non-zero values from other.

0 commit comments

Comments
 (0)