Repository navigation
Make agent sampling configurable - #131
Conversation
Benchmarks compare agents at fixed sampling settings. S-Code always samples at temperature 0. It starts with 8,192 output tokens per model call and doubles that after a cut-off response, up to 32,768. Add model.temperature (S_CODE_MODEL_TEMPERATURE, 0 to 2) and model.max_output_tokens (S_CODE_MODEL_MAX_OUTPUT_TOKENS, 1 to 1000000). When they are unset, behavior is unchanged. An explicit output cap stays fixed instead of doubling. The Harbor adapter passes both through as the temperature and max_output_tokens kwargs and records them in provenance. Refs #129 Signed-off-by: Wei Deng <wdeng@wustl.edu>
|
Looks good. The bounds match between the Rust config (0..=2, 1..=1_000_000) and the Harbor adapter, defaults are preserved (temperature 0, 8192 with doubling), and tests cover the config validation, the fixed-cap no-doubling path, and the adapter kwargs. One optional point: I didn't see a test that a configured value actually reaches the request. with_model_sampling -> execute_turn wires temperature/max_output_tokens via unwrap_or; a small test that a set value lands in the AgentRunRequest would close the config-to-request gap. Minor, since it's a direct unwrap_or. |
The existing tests cover the sampling settings' validation and the agent's fixed output cap, but none followed a configured value to the request the model receives. Add one: it runs a turn without the settings and checks the defaults (temperature 0, 8,192 output tokens), then with temperature 0.6 and max_output_tokens 4,096 and checks those values arrive. Refs #129 Signed-off-by: Wei Deng <wdeng@wustl.edu>
|
Added in 71e08c8: configured_sampling_settings_reach_the_model_request runs a turn with and without the settings and checks the request the model provider receives: temperature 0 and 8,192 tokens by default, 0.6 and 4,096 when configured. Could you approve if it looks good? |
|
The new configured_sampling_settings_reach_the_model_request test covers exactly the config-to-request path I meant. LGTM. |
a0722df
into
codex/issue-129-harbor-adapter
Stacked on #130; I'll retarget it to
mainonce #130 merges. Refs #129.Benchmarks compare agents at the same sampling settings. S-Code always samples at temperature 0. It starts with 8,192 output tokens per model call and doubles that after a cut-off response, up to 32,768. This change adds settings to match another agent, for example the Terminus-2 baseline's temperature of 0.6 and 4,096 output tokens.
Changes
model.temperature(S_CODE_MODEL_TEMPERATURE, 0 to 2) andmodel.max_output_tokens(S_CODE_MODEL_MAX_OUTPUT_TOKENS, 1 to 1,000,000).temperatureandmax_output_tokensagent kwargs, and records them in provenance.Backward compatibility
Nothing changes unless these are set: temperature stays 0, and the output budget grows as before. Saved agent runs from before this change still load, because the new fixed-cap flag defaults to off.
Testing
cargo fmtandcargo clippy -D warningsare clean.