Skip to content

feat(openfeature): emit server-side EVP flagevaluation#3984

Draft
leoromanovsky wants to merge 20 commits into
masterfrom
leo.romanovsky/ffl-2446-evp-flagevaluation-php
Draft

feat(openfeature): emit server-side EVP flagevaluation#3984
leoromanovsky wants to merge 20 commits into
masterfrom
leo.romanovsky/ffl-2446-evp-flagevaluation-php

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Motivation

PHP server-side flag evaluations need to emit aggregated EVP flagevaluation payloads through the native Rust/C and libdatadog sidecar path while preserving the existing OTel metric and exposure paths. The emitted events must match the live flageval-worker contract: schema-visible aggregation dimensions only, no OpenFeature reason, bounded context before buffering, and targeting_rule.key only when real targeting-rule metadata exists.

Changes

  • Adds native PHP EVP flagevaluation aggregation using backend-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, real targeting-rule key when present, targeting key, and bounded context.
  • Prunes evaluation context before it participates in aggregation keys or queued snapshots.
  • Keeps OpenFeature reason out of EVP payloads and aggregation keys.
  • Preserves visible variant/allocation/error dimensions when overflow folds into degraded events.
  • Flushes the aggregated flagevaluation batch before exposure and OTel metric sidecar actions at request shutdown.
  • Splits native-to-sidecar flagevaluation batches into bounded 512-event IPC chunks.
  • Bumps libdatadog to 3350d4b55c0815143e656f334967e4e4b6dd11f5, which contains the shared flagevaluation sidecar delivery fixes.

Decisions

  • EVP cardinality is defined only by fields the worker receives; reason is not a hidden aggregate key.
  • Degraded events omit targeting key and context but keep the visible dimensions needed for backend counts.
  • PHP keeps the sidecar delivery architecture instead of adding a direct EVP HTTP writer to the extension.
  • System-test validation covers the production per-flag degradation threshold and exercises overflow past the full-fidelity cap.
  • This PR remains draft until the cross-SDK rollout and review pass are complete.

Validation Evidence

Validated with PHP SDK commit 9093fb9439fb34a69b0926df3c315435557e9abc, libdatadog commit 3350d4b55c0815143e656f334967e4e4b6dd11f5, and the merged system-tests flagevaluation baseline plus the local singular-path/overflow contract fix.

  • RUSTC_BOOTSTRAP=1 cargo check -p datadog-php - PASS.
  • RUSTC_BOOTSTRAP=1 cargo test -p datadog-php ffe -- --nocapture - PASS, 23 passed.
  • ./tooling/bin/build-debug-artifact gnu-aarch64-8.0-nts /Users/leo.romanovsky/gsd-workspaces/flag-evaluations-cross-sdk/system-tests-clean-php/binaries - PASS; produced dd-library-php-1.21.0-aarch64-linux-gnu.tar.gz.
  • SYSTEM_TEST_BUILD_TIMEOUT=1800 ./build.sh --library php --weblog-variant apache-mod-8.0 from system-tests-clean-php - PASS on 2026-06-20.
  • TEST_LIBRARY=php WEBLOG_VARIANT=apache-mod-8.0 ./run.sh +l php FEATURE_FLAGGING_AND_EXPERIMENTATION -k "test_flag_eval_evp" - PASS on 2026-06-20, 8 passed, 2627 deselected in 60.08s with Library: php@1.21.0.
  • Degradation payload evidence from the passing system-test run: payloads=30, events=10195, total evaluation_count=12708; evp-degradation-flag emitted 10001 rows with total evaluation_count=12000, including one degraded row with evaluation_count=2000 and omitted targeting_key/context.
  • Real-backend dogfooding used local PHP builds for app-php7 and app-php8-openfeature with DD_EVP_PROXY_CONFIG_ADDITIONAL_ENDPOINTS={}; both returned 5/5 successful evaluations for ffe-dogfooding-string-flag.
  • Real-backend targeting-key prefixes for EVP lookup:
    • server-evp-e2e-php7-1781931066-user-
    • server-evp-e2e-php8-openfeature-1781931066-user-
  • Mock intake stayed empty during real-backend dogfooding: { "total_count": null, "by_path": {} }.

…ith PREP-01 libdatadog

- Enable 'flagevaluation-evp' feature on datadog-ffe dep (FfeFlagEvaluationBatch type now compiled)
- Fix components-rs/bytes.rs: update 4x VecMap::remove() -> remove_slow() for libdatadog compat post-commit 74284cac7 (VecMap API renamed); this unblocks compilation against the PREP-01 libdatadog ref
…patch

- Two-tier aggregation in components-rs/ffe.rs: full→degraded→drop-counted
  with caps GLOBAL_CAP=131072/PER_FLAG_CAP=10000/DEGRADED_CAP=32768
- Killswitch DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default: on) via
  evp_enabled() in Rust and isEvpEnabled() in EvaluationMetricRecorder.php
- ddog_ffe_flush_flag_evaluation_batch() Rust C-export dispatches
  SidecarAction::FfeFlagEvaluationBatch via sidecar_blocking::enqueue_actions
- ddtrace_ffe_flush_flag_evaluation_batch() C wrapper in tracer/ffe.c
  mirrors existing exposure/metric flush pattern with sidecar globals
- RSHUTDOWN call added in tracer/ddtrace.c after existing flush calls
- 11 Rust unit tests covering both tiers, overflow, drain, killswitch
…EVP aggregator race

ddog_ffe_evaluate() records into the global EVP_AGGREGATOR; without
EVP_TEST_LOCK the test ran concurrently with degraded_tier_overflow
tests, causing dropped_degraded_overflow to be 2 instead of 1.
… + regen Cargo.lock

Points dd-trace-php's libdatadog submodule at the local PREP-01 commit
containing the flagevaluation EVP emitter (FfeFlagEvaluationBatch), so
components-rs builds against it via the datadog-ffe path dep with the
flagevaluation-evp feature. NOTE: 89a2ba7fc is local/unpushed — re-point
to the merged upstream libdatadog SHA before any PR.
The Rust C-export ddog_ffe_flush_flag_evaluation_batch (components-rs/ffe.rs)
was added without a matching prototype in the committed cbindgen header
components-rs/datadog.h. tracer/ffe.c calls it, so PHP8's stricter toolchain
fails with -Werror=implicit-function-declaration (ddtrace.so link Error 2).
PHP7 only warned and linked, masking the bug. Prototype matches the Rust
signature (SidecarTransport**/InstanceId*/QueueId*/CharSlice x3).
…ow drops

The full-tier EVP flagevaluation drain previously emitted context: None and
drained the degraded-overflow drop count silently.

- Full tier now carries the pruned evaluation context (shared prune_context
  bounds: <=256 fields, string values >256 bytes skipped) plus context.dd.service,
  matching the degraded tier's cap enforcement. The pruned context is captured
  once per bucket at insertion and carried verbatim into the drained event.
- The degraded-tier overflow drop counter is read-and-reset at drain and logged
  via tracing::warn when non-zero, so an undersized degradedCap is observable
  instead of a silent loss of legitimate counts.
…low surfacing

- ddog_ffe_evaluate_populates_evp_aggregator_for_flush / _respects_killswitch:
  drive the real FFI entry point ddog_ffe_evaluate (the function the PHP/C layer
  calls) and assert it feeds the aggregator that the sidecar flush drains, closing
  the 'unit-green but emits nothing' gap that earlier tests left uncovered.
- full_tier_event_carries_pruned_context / _prunes_oversized_string_values /
  _empty_context_emits_no_context_object: assert the full tier carries the pruned
  context and enforces the field/value bounds.
- drain_resets_degraded_overflow_drop_counter: assert drain reads-and-resets the
  observable overflow drop counter.
…ncode-safe wire + reliable enqueue)

Bump the libdatadog submodule to the bincode-safe flagevaluation fix (DataDog/libdatadog#2117): the worker->sidecar IPC is bincode, which the old serde_json::Value + skip_serializing_if wire types could not deserialize, so the sidecar silently dropped every batch.

- Stringify the pruned full-tier context (JSON object string) at drain so the bincode wire stays plain; the sidecar flusher re-expands it into a JSON object for the POST.

- Use sidecar_blocking::enqueue_actions_reliable for the one-shot RSHUTDOWN flush.
@datadog-official

datadog-official Bot commented Jun 14, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 17 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | ASAN test_c with multiple observers: [8.3]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | check-big-regressions   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.1]   View in Datadog   GitLab

View all 17 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 54.08% (-0.04%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9093fb9 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jun 16, 2026

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2026-06-20 02:12:53

Comparing candidate commit 9093fb9 in PR branch leo.romanovsky/ffl-2446-evp-flagevaluation-php with baseline commit 8f132ce in branch master.

Some scenarios are present only in baseline or only in candidate runs. If you didn't create or remove some scenarios in your branch, this maybe a sign of crashed benchmarks 💥💥💥
Check Gitlab CI job log to find if any benchmark has crashed.

Scenarios present only in candidate:

  • FlagEvaluationBench/benchEvaluateSplit
  • FlagEvaluationBench/benchEvaluateDistinctContexts
  • FlagEvaluationBench/benchEvaluateTargetingMatch-opcache
  • FlagEvaluationBench/benchEvaluateWithoutCounting
  • FlagEvaluationBench/benchEvaluateTargetingMatch
  • FlagEvaluationBench/benchEvaluateDistinctContexts-opcache
  • FlagEvaluationBench/benchEvaluateWithoutCounting-opcache
  • FlagEvaluationBench/benchEvaluateSplit-opcache

Found 0 performance improvements and 1 performance regressions! Performance is the same for 193 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:SpanBench/benchOpenTelemetryInteroperability

  • 🟥 execution_time [+195.869µs; +199.038µs] or [+100.859%; +102.491%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant