You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Upgrade nginx configuration to reduce 104 error - #176
This pull request introduces several improvements to the Nginx deployment and adds tooling for testing and mocking inference servers. The main themes are: enhancing Nginx's robustness and scalability under high concurrency, and providing tools for local testing and benchmarking of inference endpoints.
Nginx configuration and deployment improvements:
Increased worker_connections from 1024 to 65535 in nginx.conf.template to support more simultaneous connections, improving scalability.
Added backlog=4096 to both HTTP and HTTPS listen directives and set the system's net.core.somaxconn to 4096 in run.sh.template to match, reducing connection drops under heavy load. [1][2][3]
Introduced a model_proxy_upstream block with keepalive settings for efficient connection reuse and automatic stale-connection retry, mitigating connection resets under high concurrency.
Updated proxy settings for /model-proxy/ and /job-server/ locations: set HTTP/1.1, cleared Connection header, and enabled retry on connection-level errors for POST endpoints, increasing reliability. [1][2]
Updated the Kubernetes deployment to grant the container NET_ADMIN capability, allowing it to set the listen backlog at runtime.
Tooling for local testing and benchmarking:
Added tools/mock_inference_server.py, a FastAPI-based mock inference server that simulates OpenAI-compatible chat completions with configurable delays and token counts for local/dev testing.
Added tools/test_inference.py, a concurrent benchmarking script to stress-test inference endpoints, reporting per-model and overall statistics, and logging failed requests.
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR hardens the Nginx deployment to reduce connection resets under high concurrency (e.g., 104 errors) and introduces local tooling to mock and benchmark inference endpoints.
Changes:
Increased Nginx connection capacity and listen backlog; added upstream keepalive/retry tuning for model-proxy and job-server routes.
Updated Kubernetes deployment and startup script to attempt raising somaxconn.
Added a mock OpenAI-compatible inference server and an async benchmarking script.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
File
Description
tools/test_inference.py
Adds an aiohttp-based concurrent benchmarking tool for inference endpoints.
tools/mock_inference_server.py
Adds a FastAPI mock server for OpenAI-style chat completions (streaming + non-streaming).
src/pylon/deploy/pylon.yaml.template
Grants additional Linux capabilities to support runtime networking/sysctl tuning.
src/pylon/deploy/pylon-config/run.sh.template
Attempts to raise net.core.somaxconn to match configured Nginx backlog.
src/pylon/deploy/pylon-config/nginx.conf.template
Increases worker connection limits, adds backlog tuning, and introduces a keepalive upstream for model-proxy.
Preserve backend URL schemes and TLS SNI, report runtime connection limits without extra capabilities, and clarify retry guarantees. Stabilize mock streaming metadata and make load-generator connection reuse configurable, with regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Set the model-proxy Host header to the backend authority, keep mock token usage consistent, and replace batch barriers with a fixed request worker pool. Cover header rendering, usage totals, slot refill, request counts, and invalid load arguments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve DNS refresh for hostname-based MODEL_PROXY_URI values. This upstream resolves its hostname when Nginx loads, unlike the previous variable-based proxy_pass, which used the configured resolver. After a DNS address change, requests remain pinned to the old backend until a reload; an initially unresolved hostname can also prevent gateway startup. Enable dynamic upstream resolution with a shared zone, the server's resolve parameter, and a reachable resolver supported by the deployed Nginx version. Update the rendering assertion for the new server directive.
Emit terminal completion chunk with finish_reason stop
tools/mock_inference_server.py:107
Emit a terminal completion chunk before [DONE]. Every JSON chunk currently has finish_reason: null, so clients inspecting that field never receive a normal completion reason; an empty response emits no completion metadata at all. Send a final chunk with the same ID and timestamp, an empty delta, and finish_reason: "stop". Update the streaming test to check that terminal chunk separately from the content chunks.
Use a shared upstream zone with resolver-backed address updates. Emit a terminal stop chunk before DONE for both empty and nonempty mock streams, with regression coverage for stream metadata and dynamic DNS directives.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore job-server behavior to the dev baseline and narrow tuning to model-proxy connection reuse and the original capacity settings. Rely on nginx 1.30.4 defaults for proxy HTTP version, Connection headers, and keepalive lifetime settings. Add coverage preserving the existing job-server behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Invalid token bounds let the mock server start but cause every completion request to fail.
0 open findings
Previously missed (1)
In code that hasn't changed since last review
Validate token bounds before starting the server
tools/mock_inference_server.py:159
Passing --min-tokens 100 with the default --max-tokens 50 starts the server, but every chat request then raises ValueError in random.randint() and returns HTTP 500. Validate 0 <= min_tokens <= max_tokens with parser.error() before updating the configuration or starting Uvicorn so invalid bounds fail immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements to the Nginx deployment and adds tooling for testing and mocking inference servers. The main themes are: enhancing Nginx's robustness and scalability under high concurrency, and providing tools for local testing and benchmarking of inference endpoints.
Nginx configuration and deployment improvements:
worker_connectionsfrom 1024 to 65535 innginx.conf.templateto support more simultaneous connections, improving scalability.backlog=4096to both HTTP and HTTPSlistendirectives and set the system'snet.core.somaxconnto 4096 inrun.sh.templateto match, reducing connection drops under heavy load. [1] [2] [3]model_proxy_upstreamblock with keepalive settings for efficient connection reuse and automatic stale-connection retry, mitigating connection resets under high concurrency./model-proxy/and/job-server/locations: set HTTP/1.1, clearedConnectionheader, and enabled retry on connection-level errors for POST endpoints, increasing reliability. [1] [2]NET_ADMINcapability, allowing it to set the listen backlog at runtime.Tooling for local testing and benchmarking:
tools/mock_inference_server.py, a FastAPI-based mock inference server that simulates OpenAI-compatible chat completions with configurable delays and token counts for local/dev testing.tools/test_inference.py, a concurrent benchmarking script to stress-test inference endpoints, reporting per-model and overall statistics, and logging failed requests.