Skip to content

chore(otel): Ensure daemonset uses local policy#41

Merged
jsingleton-dev merged 1 commit into
ctrlplanedev:mainfrom
jsingleton-dev:jsingleton/fix-ds-internal-traffic-policy
May 11, 2026
Merged

chore(otel): Ensure daemonset uses local policy#41
jsingleton-dev merged 1 commit into
ctrlplanedev:mainfrom
jsingleton-dev:jsingleton/fix-ds-internal-traffic-policy

Conversation

@jsingleton-dev
Copy link
Copy Markdown
Collaborator

@jsingleton-dev jsingleton-dev commented May 8, 2026

We run OTEL as a DaemonSet. When spans hit the collector, it uses the pod's IP as a cache key to lookup the pods information and get all the details like namespace, deployment, etc. We set the OTEL collector endpoint to OTEL_EXPORTER_OTLP_ENDPOINT=http://ctrlplane-otel:4318 which hits the Service. By default, the Service traffic policy will load balance traffic to pods on different nodes. So when a span from node A hits a OTEL pod in node B, the pod IP is a cache miss and we don't get the enriched details. In that case, the spans look like this (see only pod IP is there)

Using Default Cluster mode

Screenshot 2026-05-08 at 2 37 05 PM

When fixed to use Local traffic policy, the Service sends traffic to the pods on the same node as the caller. This is more efficient and causes the cache to properly return the pod data. See the results here with full k8s info

Using Correct Local mode

Screenshot 2026-05-08 at 2 36 30 PM

This PR defaults traffic policy to Local when it runs as a DaemonSet as is the proper config.

Summary by CodeRabbit

  • Chores
    • Updated ctrlplane chart to version 1.1.3
    • Updated OTEL chart to version 0.2.2
    • Optimized OTEL service traffic routing for DaemonSet deployments

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 8, 2026

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds conditional internalTrafficPolicy: Local configuration to the OTEL Kubernetes Service template when deployed as a DaemonSet, includes three test cases validating this behavior across different workload configurations, and bumps both parent and subchart versions.

Changes

OTEL Service Internal Traffic Policy Feature

Layer / File(s) Summary
Service Template Conditional Logic
charts/ctrlplane/charts/otel/templates/service.yaml
OTEL Service template conditionally sets spec.internalTrafficPolicy to Local when workload.kind resolves to "DaemonSet".
Test Suite Coverage
charts/ctrlplane/tests/otel_internal_traffic_policy_test.yaml
New test suite with three test cases: validates internalTrafficPolicy is Local in default DaemonSet mode, remains Local when explicitly set to DaemonSet, and is absent when workload.kind is Deployment.
Chart Version Updates
charts/ctrlplane/Chart.yaml, charts/ctrlplane/charts/otel/Chart.yaml
Parent chart version bumped from 1.1.2 to 1.1.3; OTEL subchart version bumped from 0.2.1 to 0.2.2.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • ctrlplanedev/ctrlcharts#38: Both PRs modify the otel Helm chart to conditionally change manifests based on the same .Values.workload.kind (DaemonSet vs Deployment)—the retrieved PR adds workload.kind and deployment/templating changes, while the main PR uses that same value to set Service internalTrafficPolicy and add tests.
  • ctrlplanedev/ctrlcharts#40: Both PRs modify the same OTEL Service Helm template (charts/ctrlplane/charts/otel/templates/service.yaml).

Suggested reviewers

  • jsbroks
  • adityachoudhari26
  • zacharyblasczyk

Poem

🐇 A hop and a test, DaemonSets blessed,
Local traffic flows, no eastward quest,
Service now knows which path is best—
Charts bumped up, the code has passed the test!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately reflects the main changes: it updates the OTEL chart version and adds internalTrafficPolicy: Local configuration for DaemonSet deployments.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
charts/ctrlplane/charts/otel/Chart.yaml (1)

1-8: ⚡ Quick win

Add kubeVersion to Chart.yaml to declare minimum Kubernetes compatibility.
This chart uses spec.internalTrafficPolicy (in service.yaml), which is GA only in Kubernetes v1.26+. Declaring kubeVersion: ">=1.26.0" prevents installations on incompatible clusters.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@charts/ctrlplane/charts/otel/Chart.yaml` around lines 1 - 8, Chart.yaml is
missing a kubeVersion declaration which allows installs on clusters older than
v1.26 where spec.internalTrafficPolicy is not GA; update the Chart.yaml for the
otel chart by adding a kubeVersion: ">=1.26.0" entry (next to
apiVersion/name/version/appVersion) to declare minimum Kubernetes compatibility
so installations are prevented on incompatible clusters; note the chart uses
spec.internalTrafficPolicy in service.yaml so target Kubernetes v1.26+.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@charts/ctrlplane/charts/otel/Chart.yaml`:
- Around line 1-8: Chart.yaml is missing a kubeVersion declaration which allows
installs on clusters older than v1.26 where spec.internalTrafficPolicy is not
GA; update the Chart.yaml for the otel chart by adding a kubeVersion: ">=1.26.0"
entry (next to apiVersion/name/version/appVersion) to declare minimum Kubernetes
compatibility so installations are prevented on incompatible clusters; note the
chart uses spec.internalTrafficPolicy in service.yaml so target Kubernetes
v1.26+.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 47fbf96e-6458-4c2c-8cc1-5a66fe793e9a

📥 Commits

Reviewing files that changed from the base of the PR and between 402df97 and 4c18b48.

⛔ Files ignored due to path filters (1)
  • charts/ctrlplane/Chart.lock is excluded by !**/*.lock
📒 Files selected for processing (4)
  • charts/ctrlplane/Chart.yaml
  • charts/ctrlplane/charts/otel/Chart.yaml
  • charts/ctrlplane/charts/otel/templates/service.yaml
  • charts/ctrlplane/tests/otel_internal_traffic_policy_test.yaml

@jsingleton-dev
Copy link
Copy Markdown
Collaborator Author

Nitpick comments:
In @charts/ctrlplane/charts/otel/Chart.yaml:

  • Around line 1-8: Chart.yaml is missing a kubeVersion declaration which allows
    installs on clusters older than v1.26 where spec.internalTrafficPolicy is not
    GA; update the Chart.yaml for the otel chart by adding a kubeVersion: ">=1.26.0"
    entry (next to apiVersion/name/version/appVersion) to declare minimum Kubernetes
    compatibility so installations are prevented on incompatible clusters; note the
    chart uses spec.internalTrafficPolicy in service.yaml so target Kubernetes
    v1.26+.

Eh I checked and this was in beta from 1.22 which was EOL 4 years ago. Given we don't have version constraints in this repo already I don't think this is the trigger for adding them. Open to it if others disagree

@jsingleton-dev jsingleton-dev merged commit 8b19638 into ctrlplanedev:main May 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants