[ROBO-5722] Add Python client and server for UiPath.Ipc

[eduard-dumitru]

Summary

Adds a Python client and server for UiPath.Ipc that speaks the same wire protocol as the .NET package, plus the CI/CD plumbing to build, test, and publish it. Published and resolvable on the project-scoped uipath-ipc-deps feed. The Python package version tracks UiPath.CoreIpc (<Version>2.5.1</Version>), stamped to a PEP 440 local-version string per CI run (e.g. 2.5.1+<build>) by src/CI/stamp-python-version.py — so pyproject.toml's version is a placeholder the pipeline overwrites.

Scope — landed incrementally, each in its own commit:

1. Client — dynamic-proxy RPC over Named Pipe / TCP.
2. Bidirectional callbacks — the client hosts callback services the server invokes.
3. Server — IpcServer hosts services that a .NET or Python client calls.
4. Handler reach-back — message.client.get_callback(...), the Python analog of .NET's m.Client.GetCallback<T>().

Streams (upload / download) remain explicitly out of scope.

Python client (src/Clients/python/uipath-ipc/)

• Pure-Python, asyncio-based; talks the same wire to any UiPath.Ipc peer.
• Transports: NamedPipe (Win + POSIX via /tmp/CoreFxPipe_*) and TCP — both client and server halves — with bounded retry on FileNotFoundError to ride out the peer's accept-loop warm-up.
• IpcClient(transport=..., callbacks={IClientCallback: impl}): lazy connect, dynamic __getattr__ proxy via get_proxy(IContract), auto-reconnect on transport drop, optional request_timeout.
• RemoteException preserves message / type_name / stack_trace / inner chain, sets __cause__ for traceback display.
• Cancellation is task-based; no CancellationToken parameter on signatures. The proxy forwards CancellationRequest on the wire when its task is cancelled; peer-initiated cancellation aborts the matching in-flight handler.
• Wheel ships as py3-none-any with PEP 561 py.typed.

Callback (server → client) support

• IpcClient(callbacks={Contract: instance}) registers handler objects keyed by interface name.
• IpcConnection dispatches incoming REQUEST frames: looks up the endpoint by interface name, resolves the method by name, json.loads each parameter individually (matching the .NET convention), invokes (awaitable or sync), and writes a Response back. A write lock keeps concurrent outbound frames aligned.
• Handler exceptions wire back as RemoteException. Peer-initiated CancellationRequest cancels the matching handler task; the emitted error mimics .NET's OperationCanceledException.
• Callback methods must NOT declare CancellationToken parameters — the caller doesn't include CT in the wire Parameters array (matches the existing .NET IComputingCallback convention).

Server — IpcServer (NEW)

• IpcServer(transport, services={IContract: impl}, request_timeout=None): listens via a ServerTransport and wraps each accepted client in the existing symmetric IpcConnection, whose callbacks dict is the hosted-service set. Duck-typed dispatch by contract __name__ (the instance need not inherit the contract).
• Server transports: TcpServerTransport (asyncio.start_server) and NamedPipeServerTransport (start_serving_pipe on Windows, start_unix_server at /tmp/CoreFxPipe_<name> on POSIX).
• start / serve_forever / aclose + async context manager; connection_count introspection. Live connections are pruned via a new IpcConnection.add_close_callback hook that fires once on explicit close or peer disconnect.
• aclose closes connections before awaiting the listener, so Python 3.12+'s Server.wait_closed() (which blocks until active connections finish) doesn't hang.

Handler reach-back — Message.Client / GetCallback (NEW)

• message.py: Message / Message[T] + an IClient protocol. A service or callback method opts into a handle on its caller by declaring a Message parameter; the wire never carries it (mirrors .NET's trailing-Message convention — the caller sends the same args either way).
• IpcConnection.get_callback(Contract) returns a proxy bound to that connection, so a handler can call the peer back mid-request — the inverse direction of the in-flight call. Python analog of m.Client.GetCallback<T>(). Reached as m.client.get_callback(IContract).
• Dispatch (_invoke_callback) does signature-aware binding: wire args fill the real parameters, a Message (.client = the connection, .request_timeout carried through) is injected at its slot; the no-Message fast path is unchanged and cached per function. Works symmetrically for client-hosted callbacks and server-hosted services, since both run through the one IpcConnection.

Dedicated .NET test server (src/IpcSample.PythonClientTestServer/)

Purpose-built for the Python integration suite — net8.0, console logging on, stable READY pipe=<name> startup marker. Hosts IComputingService / ISystemService (callback-free) plus ICallbackTester, whose handlers call m.Client.GetCallback<IClientCallback>() to exercise the server→client callback path against the real .NET Message.Client reach-back.

Wire-shape tests

tests/wire/test_dotnet_compatibility.py asserts our serialized JSON against the .NET schema literally (field sets, types, TimeoutInSeconds is a non-null double, etc.) — catches a class of regression at unit-test time that previously only surfaced when the live integration suite ran. This came out of finding (and fixing) an actual mismatch: TimeoutInSeconds: null made Newtonsoft drop the entire Request silently.

CI / pipeline redesign

• Parameterized publish: publishNuGet, publishNpm, publishPyPI (all default false). Default CI runs build + test only. Old behavior of always-pushing NuGet on every branch and always-popping an approval prompt is gone — and a rejected approval no longer leaves the run as Failed.
• Parameterized build (opt-out): buildNuGet, buildNpm, buildPython (all default true).
• Per-technology Build stages so NuGet / NPM / Python race to the finish line independently.
• reuseArtifactsFromBuildId parameter lets a Publish stage replay against a prior successful build, skipping Build entirely.
• Job names shifted from environment-centric (".NET on Windows", "node.js on Ubuntu") to deliverable-centric ("NuGet — .NET on Windows", "NPM — Node + Web on Linux (test-only)", "Python — Windows"). The (test-only) marker disambiguates the Linux matrix runs from the artifact-producing Windows ones.
• New environments NuGet-Packages and PyPI-Packages mirror the existing NPM-Packages approval shape (same approver, same 12h timeout).

Feed swap + Supply Chain Guard bypass

• npm install customFeed moved from the org-level npm-packages (managed outside CoreIpc) to a project-scoped uipath-ipc-deps (CoreIpc-owned, mirrors npmjs.org + PyPI + NuGet).
• Pipeline-level SCG_KILL_SWITCH=true to bypass the org-wide Aikido Safe Chain shim, which was interfering with npm tarball downloads. Comments in azp-start.yaml and azp-nodejs.yaml document the trail (with #devops Slack links) so the next person on this doesn't redo the dead-ends.

NPM publish

• Now writes to uipath-ipc-deps first (uses the pipeline's built-in identity — no PAT). GitHub Packages stays wired up as continueOnError: true until the platform team ships the post-Mini-Shai-Hulud pipeline-auth replacement (tracked in Liviu Bud's #dev thread).
• Runs of Publish_NPM finish as "Succeeded with issues" until the GitHub side resolves — packages still ship to uipath-ipc-deps.

Review round — robustness fixes + @ipc_cancellable

A review pass (14 threads): six correctness fixes, each with a red→green regression test; seven items deliberately declined (undefined-behaviour / systemic-misuse, reasoned in-thread); plus a new @ipc_cancellable contract marker. Suite is now 234 unit + 22 .NET-interop tests, green.

Fixes

• dict serialization is string-keyed only — to_wire no longer transforms non-string keys; JSON keys are strings and the .NET contracts never use non-string-keyed dictionaries (380ac9d).
• IpcServer single-use, lifecycle under one lock — start() after aclose() raises (matches .NET's ObjectDisposedException); start()/aclose() mutate _disposed/_handle under one lock so they can't race (3930249, 31f4ee8).
• No connection leak on shutdown — a client accepted while aclose() is tearing down is rejected+closed instead of being served past close (957b32e).
• aclose() can't hang on a non-reading peer — aborts the transport instead of awaiting a graceful flush, matching .NET's synchronous Connection.Dispose() and the TS client's socket.destroy() (18b3876).
• Resilient type-hint resolution — one unresolvable annotation (e.g. a TYPE_CHECKING-only type) degrades only that name to Any; stdlib get_type_hints is all-or-nothing and previously de-typed the whole method/dataclass (d44ed8a).
• Cancellation delivered on cancel-then-close — the CancellationRequest is sent inline at cancel time, so a cancellation unwinding through async with reaches the peer before teardown, matching .NET initiating the send before dispose (95b2150).

@ipc_cancellable (4d16e7e, 1783f63, 801a34f, af91a0b) — a contract-method marker for methods whose .NET counterpart ends in a CancellationToken (which Python signatures omit, since cancellation is task-based). It gates cancellation propagation: only marked methods emit a CancellationRequest when cancelled and honor an inbound one; an unmarked call cancels locally only. No effect on the request's arguments. Documented in the README and applied to the integration-suite contracts.

Declined by design — bad inbound arg → opaque crash; value-typed empty/null Data → None; error-reply build on a cyclic exception cause; unguarded result decode on contract/wire mismatch; reused request id; variadic *args element decoding; and missing-required-arg (kept as a loud TypeError rather than mirroring .NET's null-fill/NRE). All undefined-behaviour or systemic-misuse — not worth defensive code.

Test plan

• CI green on Windows + Linux for Build (all jobs).
• 111 non-.NET + 11 .NET-interop Python tests pass (122 total) locally (pytest) and in CI.
• Client RPC over Named Pipe + TCP, incl. RemoteException propagation and task-based cancellation.
• Server→client callbacks: 3 round-trips against the .NET ICallbackTester (real m.Client.GetCallback<IClientCallback>()) — test_dotnet_interop.py#L136-L183.
• Server (IpcServer): TCP + named-pipe loopback (Python client ↔ Python server), lifecycle, connection-count pruning — test_ipc_server.py.
• Handler reach-back: full-duplex re-entrancy E2E — client → server handler → get_callback → client callback → back — test_ipc_server.py#L196-L242; plus Message-injection / get_callback units — test_message_injection.py.
• Close-callback hook (server connection pruning) — test_connection.py#L133-L173.
• Publish_PyPI publishes the wheel to uipath-ipc-deps, version stamped from UiPath.CoreIpc + build number (PEP 440 local segment, e.g. 2.5.1+<build>).
• Consumer-side install round-trips: uipath-robot-client (separate repo, own venv, index-url = uipath-ipc-deps) pins uipath-ipc>=2.5.1+20260611.6 and its 100-test suite passes against the published wheel.
• .NET-client ↔ Python-IpcServer interop (reverse of IpcSample.PythonClientTestServer): a real .NET client drives the Python server — direct calls, reach-back into a .NET-hosted callback, and a RemoteException round-trip — test_dotnet_client_interop.py + IpcSample.PythonServerTestClient. Surfaced two dispatch fixes: bind wire args to the handler signature (tolerate a .NET client's trailing CancellationToken), and close the writer on peer disconnect (transport-leak fix).
• High-effort code-review pass (7 finder angles) — fixed a real bug (serve_forever() returned immediately for Windows named pipes, tearing the server down), made Message injection keyword-based so keyword-only / Optional[Message] params inject correctly, unified connection teardown (now cancels in-flight handlers on peer disconnect), and dropped the _ConnectionInvoker shim. 4 regression tests added.
• (deferred) Pre-existing .NET flake SystemTestsOverTcp.NotPassingAnOptionalMessage_ShouldWork (tight 800ms timeout, pre-dates this PR). Tracked separately.
• (deferred) Restore GitHub Packages NPM publish + remove continueOnError once platform team ships sanctioned pipeline auth.
• (deferred) Re-enable Safe Chain Guard once DevOps fixes the underlying shim.
• Deleted src/Clients/python/_attempt0/ (the reference port; shipping impl is src/Clients/python/uipath-ipc/).

🤖 Generated with Claude Code

[Radu-Niculae]

A value-typed return answered with empty/null Data silently becomes None, where .NET throws.

This short-circuits any falsy resp.data to None before the declared return type is consulted. Reachable with no contract mismatch: a Python server method declared -> int whose impl returns None ships Data=null (one-way is keyed on the impl's own annotation — connection.py L170 — and data = None if result is None … L691). The client gets None; count + 1 then raises TypeError far from the call.

.NET deserializes ""/null into a non-nullable value type by throwing JsonSerializationException at the call (Dtos.cs L49 → Deserialize(Data ?? "", typeof(TResult))) — fail-loud. (The legit -> None void path → Data="" → null stays parity and is fine.)

Repro (3.10, loopback): GetCount() -> int with an impl that returns None → client gets None; count + 1 raises TypeError.

Fix: gate the early None on the return hint — return None only for None/Optional/object/Any/str, otherwise raise (or surface a clear decode error).

[eduard-dumitru]

great find, thanks

[eduard-dumitru]

Will not fix, UiPath.Ipc is a kit where we don't hold the hand of end-users due to the dynamic nature of their language/runtime.

This is about returned values, but the problem exists for arg-param transfers and also for class fields within those returned values/args.

Contract mismatches in general between Ipc counterparts are not supported (all bets are off).
Also, if Python's dynamic nature makes it so that regardless of the type hint, the method actually returns something incompatible along the way, then that will caused undefined errors (acceptable).

The same already goes for TypeScript where the type declarations are tentative, and through careful linting and avoiding bugs, things work, but the system doesn't validate anything. Any function that in the typescript definition vows to return a number can actually choose at runtime to returns something else, and the failure will be late, in the callee, after the JSON string was transported.

[Radu-Niculae]

A missing trailing arg for a required param raises TypeError (handler never runs), where .NET fills default(T) and invokes it.

On wire exhaustion the wire branch simply doesn't append, relying on the param having a Python default. If a missing trailing param has none, method(*pos, **kwargs) raises TypeError: missing … argument → returned as an Error, handler never invoked. This bites the forward-compat case: an older client omitting an arg the server later added.

.NET fills every missing trailing slot with default(T) (null for reference types, 0/false for value types) and invokes the handler (SetOptionalArguments, Server.cs L237; GetDefaultValue, Helpers.cs L52).

Repro (3.10): Add(a, b) called with [42] → TypeError, handler not run; control with a Python default works.

Should this match .NET? Filling a required-but-missing wire param with the type's zero value (else None) instead of skipping would align the forward-compat behavior — or is a hard error here the intended contract?

[eduard-dumitru]

systemic missuses shouldn't be treated with too much care

[eduard-dumitru]

.NET doesn't fill every missing trailing slot, but is actually wacky in behavior:

• for CancellationToken it does fill it with the default
• for any other value type it fails with NRE
• for all reference types, it silently provides null

This sillyness isn't something to lean towards, and I'd very much keep the TypeError, until we decide to revamp .NET too and then align all 3 runtimes.