Skip to content

Make agent sampling configurable - #131

Merged
ylzhang2447 merged 2 commits into
codex/issue-129-harbor-adapterfrom
issue-129-sampling-settings
Oct 19, 2026
Merged

ylzhang2447 merged 2 commits into
codex/issue-129-harbor-adapterfrom
issue-129-sampling-settings

Conversation

@weiddeng

Copy link
Copy Markdown
Collaborator

Stacked on #130; I'll retarget it to main once #130 merges. Refs #129.

Benchmarks compare agents at the same sampling settings. S-Code always samples at temperature 0. It starts with 8,192 output tokens per model call and doubles that after a cut-off response, up to 32,768. This change adds settings to match another agent, for example the Terminus-2 baseline's temperature of 0.6 and 4,096 output tokens.

Changes

  • New settings: model.temperature (S_CODE_MODEL_TEMPERATURE, 0 to 2) and model.max_output_tokens (S_CODE_MODEL_MAX_OUTPUT_TOKENS, 1 to 1,000,000).
  • An explicit output cap stays fixed instead of doubling.
  • The Harbor adapter passes both through as the temperature and max_output_tokens agent kwargs, and records them in provenance.

Backward compatibility

Nothing changes unless these are set: temperature stays 0, and the output budget grows as before. Saved agent runs from before this change still load, because the new fixed-cap flag defaults to off.

Testing

  • cargo fmt and cargo clippy -D warnings are clean.
  • Tests pass for config (27), agent-core (61), evals and the daemon binary. New tests cover the settings' bounds and check that a fixed cap never grows.
  • Adapter tests pass (18), including new tests for the kwargs.

Benchmarks compare agents at fixed sampling settings. S-Code always samples
at temperature 0. It starts with 8,192 output tokens per model call and
doubles that after a cut-off response, up to 32,768.

Add model.temperature (S_CODE_MODEL_TEMPERATURE, 0 to 2) and
model.max_output_tokens (S_CODE_MODEL_MAX_OUTPUT_TOKENS, 1 to 1000000).
When they are unset, behavior is unchanged. An explicit output cap stays
fixed instead of doubling. The Harbor adapter passes both through as the
temperature and max_output_tokens kwargs and records them in provenance.

Refs #129

Signed-off-by: Wei Deng <wdeng@wustl.edu>
@ylzhang2447

Copy link
Copy Markdown
Collaborator

Looks good. The bounds match between the Rust config (0..=2, 1..=1_000_000) and the Harbor adapter, defaults are preserved (temperature 0, 8192 with doubling), and tests cover the config validation, the fixed-cap no-doubling path, and the adapter kwargs.

One optional point: I didn't see a test that a configured value actually reaches the request. with_model_sampling -> execute_turn wires temperature/max_output_tokens via unwrap_or; a small test that a set value lands in the AgentRunRequest would close the config-to-request gap. Minor, since it's a direct unwrap_or.

The existing tests cover the sampling settings' validation and the agent's
fixed output cap, but none followed a configured value to the request the
model receives. Add one: it runs a turn without the settings and checks
the defaults (temperature 0, 8,192 output tokens), then with temperature
0.6 and max_output_tokens 4,096 and checks those values arrive.

Refs #129

Signed-off-by: Wei Deng <wdeng@wustl.edu>
@weiddeng

Copy link
Copy Markdown
Collaborator Author

Added in 71e08c8: configured_sampling_settings_reach_the_model_request runs a turn with and without the settings and checks the request the model provider receives: temperature 0 and 8,192 tokens by default, 0.6 and 4,096 when configured. Could you approve if it looks good?

@ylzhang2447

Copy link
Copy Markdown
Collaborator

The new configured_sampling_settings_reach_the_model_request test covers exactly the config-to-request path I meant. LGTM.

@ylzhang2447
ylzhang2447 merged commit a0722df into codex/issue-129-harbor-adapter Oct 19, 2026
21 of 23 checks passed
@ylzhang2447
ylzhang2447 deleted the issue-129-sampling-settings branch October 19, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants