Skip to content

feat(extras): add experimental decider and llm operations - #1218

Merged
ParidelPooya merged 31 commits into
mainfrom
feat/extras-ai
Nov 16, 2026
Merged

ParidelPooya merged 31 commits into
mainfrom
feat/extras-ai

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Brings the parked /ai work to main: #1160 (decider() and llm()) and #1213 (SageMaker and Lambda transports), merged with current main, plus fixes for the open review threads on #1213.

What it adds

@aws/durable-execution-sdk-js-extras/ai. Every export is marked @experimental.

Operation Step name Subtype Returns
decider().check decider.check DeciderCheck boolean
decider().classify decider.classify DeciderClassify one choice
decider().label decider.label DeciderLabel the choices that apply
decider().score decider.score DeciderScore 0 to 1
llm().ask llm.ask LlmAsk text
Decision model host Use Permission
Any server endpoint none
SageMaker transport: sageMakerTransport({ endpointName }) sagemaker:InvokeEndpoint
Lambda transport: lambdaTransport({ functionName, qualifier }) lambda:InvokeFunction
AgentCore Runtime transport: agentCoreTransport({ runtimeArn }) bedrock-agentcore:InvokeAgentRuntime

The four AWS SDK clients are optional peer dependencies, loaded on first use. Reference: packages/aws-durable-execution-sdk-js-extras/docs/ai.md.

Fixes on top of #1213

  1. esbuild bundles. esbuild leaves a missing package to run time only when .catch() sits directly on the literal import(). The lazySdk helper in feat(extras): add SageMaker and Lambda transports for decider() #1213 moved it into the helper. So bundling /ai with esbuild failed unless all four clients were installed. Each import() now has its own .catch(sdkLoadError(...)). bundle.test.ts bundles a copy of src/ai where no client can be resolved. It fails with the feat(extras): add SageMaker and Lambda transports for decider() #1213 code and passes now.
  2. One default-client cache. lazyClient() replaces the three copies (hln33).
  3. Default client test. Each transport creates one default client on first use and reuses it.
  4. README typo (hln33), now in docs/ai.md.

Changes from merging main

  • The ai tests now run on vitest.
  • The AI reference moved from the README to docs/ai.md, as main did for docs/microvm.md. The README links to it.
  • package-dependencies.test.ts lists the four ai clients. It checks that the package root and /ai load without them.

Testing

  • Extras npm test (tsc and vitest): 514 passed, 1 skipped (the live test, which needs AI_LIVE=1).
  • npm run build and npm run lint pass.
  • Not tested against a real SageMaker endpoint or Lambda function. There is no example in the examples package, because an example needs a deployed decision model.

Python counterpart: aws/aws-durable-execution-sdk-python#928.

Design note

Problem. A durable function often needs one typed decision (yes or no, one of N, a score) or one short text answer from a model. Calling a model inside context.step by hand repeats the same work in every function: prompt shape, reply parsing, retries, the invocation deadline, and checkpoint size.

Cause. The core SDK gives steps, not model calls. No existing library gives model calls that are durable steps on Lambda. Typed-answer libraries are not durable. Durable platforms ship model layers only for their own runtimes.

Options considered.

Option Result
1. Document a context.step + client pattern only No new API. Every function repeats parsing, retries and the deadline.
2. One ai() namespace with all operations One import. A decision model and a text model share one config, so a default model is ambiguous.
3. decider() and llm() factories under /ai (chosen) Config is set once per factory. Two deciders can differ. Fakes are easy in tests.
4. One decider() config key per host (agentCore, sageMaker, ...) Each new host changes DeciderConfig.
5. endpoint or one transport (chosen) A new host, such as Bedrock later, is one new export. decider() does not change.

Removed before release. The agentCore config key (now agentCoreTransport), and AgentCoreInvokeClient (now AwsSdkClient). Neither was released.

Defaults. llm() uses a pinned Bedrock model ID, moved forward in later releases. decider() has no default host, because no managed host exists yet. It uses STRANDS_DECIDER_ENDPOINT when no option is set. Step names and subtypes are fixed, because a change breaks executions in flight.

Experimental. Every export is @experimental.

Add llm(), which returns ask, check, classify, label, and score. Each call is
one durable step that returns a typed answer.

check, classify, label, and score use a decision model such as Strands Decider
when one is configured, and Amazon Bedrock structured output otherwise. ask
uses a Bedrock text model. A decision model answer below unsureBelow goes to
the text model.
check, classify, label, and score now need a decision model. Without one
they reject with a TypeError before any durable operation. The Bedrock
structured-output fallback and unsureBelow are removed.

The default ask model moves to Claude Haiku 5.5, the newest Haiku on
Bedrock. ask sends a temperature only when set, because Haiku 5.5 rejects
one. classify accepts at most 255 options and score at most 10 levels, the
decision model's limits, and the TSDoc of each operation lists its limits.
check, classify, label, and score use a decision model, not a large
language model. So they move to decider(), and llm() keeps only ask. Both
factories live in the new @aws/durable-execution-sdk-js-extras/ai subpath,
which replaces /llm.

Step names become decider.check, decider.classify, decider.label,
decider.score, and llm.ask. Subtypes move to AiOperationSubType:
DeciderCheck, DeciderClassify, DeciderLabel, DeciderScore, and LlmAsk.
Errors extend AiError: DeciderError, DeciderInvalidResponseError, and
LlmMaxTokensError. decider() throws when it has no endpoint, transport, or
STRANDS_DECIDER_ENDPOINT.
- Retry a request that the invocation deadline stops: Bedrock and decider
  aborts become AiTimeoutError, which defaultAiRetryStrategy retries.
- Fail ask with LlmStoppedError when the reply stops for a reason other
  than its end, such as a guardrail. The guardrail message is no longer
  returned as the answer.
- Document that the decision model refuses an input over its window with
  HTTP 422, which is not retried. The server no longer cuts it.
- Validate the decider endpoint URL, keep its query and credentials out of
  checkpointed errors, and do not follow redirects.
- Report a non-JSON decider reply or a score outside the scale as
  DeciderInvalidResponseError.
- Escape every case and spacing of the closing input tag in ask.
- Keep config values when a per-call override field is undefined.
- Name the classify and label lists `choices`, and accept a plain string
  array as a score scale.
- Refuse a decider endpoint with a user name or password. fetch() rejects
  such a URL with an error that repeats the password, and the error was
  checkpointed and retried. Credentials belong in headers.
- Name only the cause's type in the "could not be reached" message, because
  the cause's message can repeat the full URL. The cause stays on the error.
- Wrap only the abort itself as AiTimeoutError. A real error that arrives
  after the deadline, such as HTTP 401, keeps its message and its retry
  decision.
- Retry LlmStoppedError for malformed_model_output and malformed_tool_use,
  where a new reply can succeed.
- Escape every "</input" prefix in the ask input, whatever follows it.
- Stop re-exporting ./ai from the package root. The ai operations are
  imported from @aws/durable-execution-sdk-js-extras/ai only, as the README
  shows. Without this, a root import of microvm pulled in the Bedrock client
  import, and esbuild failed when the optional peer was not installed.
- Load @aws-sdk/client-bedrock-runtime through one import() with .catch, so
  a bundler leaves a missing package to run time. decider() then works
  without it, and ask fails with an AiError that names the package.
- Name the system error code, such as ECONNREFUSED, in the checkpointed
  "could not be reached" message. Only the message is checkpointed, and the
  code holds no URL.
- Drop the signal.reason check in isAbort. The abort error names already
  cover it.
ask said "needs the package. Install it" for any import() failure. It now
says that only for MODULE_NOT_FOUND and ERR_MODULE_NOT_FOUND, and otherwise
says the package could not be loaded, with the original error as the cause.
Tests cover both cases and check that decider() still works without the
package.
decider({ agentCore: { runtimeArn } }) sends each request through the AWS
SDK's InvokeAgentRuntime, signed with the function's credentials. AgentCore
Runtime forwards it to the server's POST /invocations route, which the
Strands Decider server answers like /v1/systemone.

The call assumes the model is deployed and ready, and does not check it. A
non-2xx status from the server becomes DeciderError with that status, a
reply that is not JSON becomes DeciderInvalidResponseError, AWS SDK errors
reach the retry strategy unchanged, and a deadline abort becomes
AiTimeoutError.

@aws-sdk/client-bedrock-agentcore is a new optional peer dependency, loaded
on first use like the Bedrock Runtime client. decider() takes only one of
agentCore, endpoint, and transport.
Stacked on the parked `feat/extras-llm-operations` branch (#1160). It is
not for main until Bedrock supports decider models.

## Why

A Strands Decider can run on a server, on SageMaker, on Lambda, or on
AgentCore Runtime. The [Strands Decider deploy
guides](https://github.com/strands-labs/strands-decider/tree/main/deploy)
show that every host takes the same request body. Only the way the call
is sent and signed changes. So each host is one transport factory, and
`decider()` does not change when a host is added.

## What changes

| Host | Use | AWS call | Permission |
|---|---|---|---|
| Any server | `endpoint` | `fetch` to `/v1/systemone` | none |
| SageMaker | `transport: sageMakerTransport({ endpointName })` |
`InvokeEndpoint` | `sagemaker:InvokeEndpoint` |
| Lambda | `transport: lambdaTransport({ functionName, qualifier })` |
`Invoke`, request as the event | `lambda:InvokeFunction` |
| AgentCore Runtime | `transport: agentCoreTransport({ runtimeArn })` |
`InvokeAgentRuntime` | `bedrock-agentcore:InvokeAgentRuntime` |

- `agentCoreTransport` replaces the `agentCore` config key. `decider()`
now takes only `endpoint` or `transport`.
- `AgentCoreInvokeClient` is renamed to `AwsSdkClient`, because all
three transports use it.
- `@aws-sdk/client-sagemaker-runtime` and `@aws-sdk/client-lambda` are
new optional peer dependencies, loaded on first use.
- SageMaker `ModelError` becomes a `DeciderError` with the server
status. So a 422 is not retried.
- The Lambda function fails the invocation with `ValueError` or
`ValidationError` when it refuses a request. That becomes a
`DeciderError` with status 422. Any other function error is retried.
- `defaultAiRetryStrategy` also retries the Lambda and SageMaker
throttling and server errors.

Bedrock support will be one more transport when the API exists.

## Testing

- `npm test` in the extras package: 515 passed, 1 skipped. New
fake-client tests cover each transport, its error mapping and its
missing-SDK error.
- The built `dist/ai.mjs` keeps all four AWS SDK imports as runtime
`import()`.
- Not tested against a real SageMaker endpoint or Lambda function.
# Conflicts:
#	package-lock.json
#	packages/aws-durable-execution-sdk-js-extras/README.md
#	packages/aws-durable-execution-sdk-js-extras/package.json
#	packages/aws-durable-execution-sdk-js-extras/rollup.config.mjs
…o vitest

esbuild leaves a missing package to run time only when a .catch() sits
directly on its literal import(). The lazySdk helper from #1213 moved
the .catch() into the helper. So a customer who bundled /ai with esbuild
needed all four AWS SDK clients installed, even with only an endpoint.

- Each transport and llm() now attach sdkLoadError() with .catch()
  directly on their own import(). once() caches the loaded module.
- lazyClient() replaces the three copies of the default-client cache.
- sdkLoadError() searches the whole cause chain for the missing-module
  code, because a loader can wrap the error.
- bundle.test.ts bundles a copy of src/ai with esbuild in a directory
  without node_modules. It fails with the #1213 code.
- missing-sdk.test.ts checks that each transport creates one default
  client on first use and reuses it.

main moved the package to vitest and to the compute/microvm subpath. So
the ai tests now use vitest, the AI reference moved to docs/ai.md as
main did for docs/microvm.md, and package-dependencies.test.ts lists
the four ai clients as optional peers that load on first use.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 11, 2026 03:04 — with GitHub Actions Active
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 11, 2026 03:05 — with GitHub Actions Active
@github-actions

This comment has been minimized.

…fetch replies in ai tests

The lockfile from the merge bumped shared AWS SDK and Smithy packages.
It now matches main, plus the two new clients at 3.1164.0.

On Node.js 22, reading a Response body inside a step of the local test
runner with skipTime runs out of memory. So the ai tests return a plain
object from the fetch mock.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 11, 2026 04:25 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The lookup of getRemainingTimeInMillis ran before the try, so a getter
that throws failed the call. It now runs inside the try, as in the core
SDK. A new test also covers the ERR_MODULE_NOT_FOUND code.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 12, 2026 23:01 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 13, 2026 17:49 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@yaythomas

Copy link
Copy Markdown
Contributor

The Python counterpart has already merged. Its corresponding follow-up is tracked in Python SDK #934: fix AI operation execution boundaries.

Fixes for yaythomas's review.

- check and label reject a probability outside 0 to 1, and clamp one
  within 0.01, as the Python SDK does.
- decider() rejects headers together with transport.
- decider and llm check the step name (1 to 256 printable ASCII
  characters) before they create the step.
- Each operation serializes its input once and passes that string into
  the step.
- A core SDK whose step context has no signal fails the call with
  AiError instead of losing cancellation without a sign.
- The DeciderTransport docs say a transport must stop when abortSignal
  aborts.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 13, 2026 21:12 — with GitHub Actions Active
@github-actions

This comment has been minimized.

# Conflicts:
#	package-lock.json
#	packages/aws-durable-execution-sdk-js-extras/package.json
#	packages/aws-durable-execution-sdk-js-extras/rollup.config.mjs
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 16, 2026 20:35 — with GitHub Actions Active
@github-actions

This comment has been minimized.

yaythomas
yaythomas previously approved these changes Nov 16, 2026

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@github-actions

Copy link
Copy Markdown

Codex AI review

Two new findings and one existing deferred coverage gap:

  • packages/aws-durable-execution-sdk-js-extras/docs/ai.md:56
  • packages/aws-durable-execution-sdk-js-extras/e2e/ai-function.ts:24
  • packages/aws-durable-execution-sdk-js-extras/src/ai/decider.ts:186

Static inspection only; no tests or AWS calls were run.

Suppressed repeats

  • arf_v1_wtialnnqe77h4bgd3bczn2rttq — deferred (maintainer verdict)

Reviewed commit 172a043b837f6188edc10964165c9e1a39431732. Workflow run

@ParidelPooya
ParidelPooya merged commit 2057d89 into main Nov 16, 2026
52 checks passed
@ParidelPooya
ParidelPooya deleted the feat/extras-ai branch November 16, 2026 21:15

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 172a043b Deployed Nov 16, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #3282
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants