Skip to content

Commit c8187a2

Browse files
authored
Merge pull request #32 from ApodexAI/fix/tokenizer-daemon-thread-import
fix(tokenizer): keep the tiktoken daemon thread out of the dynamic linker
2 parents 53bd2cf + e3f6c42 commit c8187a2

5 files changed

Lines changed: 317 additions & 23 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,42 @@ the GitHub Release body, so a release with no entry here fails.
77

88
Versioning follows [docs/versioning.md](docs/versioning.md).
99

10+
## [0.8.1] - 2026-09-05
11+
12+
### Fixed
13+
14+
- `runtime/loop/tokenizer.py` no longer runs `import tiktoken` on the daemon
15+
thread it spawns. The import is hoisted to the calling thread; only
16+
`tiktoken.get_encoding()` stays on the thread. An `atexit` hook now joins an
17+
in-flight load (1 s cap), following `providers/nonblocking_stream.py`.
18+
19+
Rationale: `tiktoken._tiktoken` is a Rust extension, so the import `dlopen`s a
20+
shared object, and CPython kills daemon threads mid-flight at finalization
21+
(`pthread_exit` at the next GIL acquisition). Killed inside the dynamic linker
22+
that is not survivable — losing glibc's `_dl_load_lock` surfaces as a later
23+
SIGSEGV, unwinding through a Rust / `extern "C"` frame calls `abort()`. A
24+
2026-09-05 investigation in ApodexHarness found the thread parked inside that
25+
import at interpreter exit on *every* short run (the import takes 24-29 ms and
26+
the first `get_encoding_nonblocking()` call lands late in a run), with two
27+
observed deaths *after* a fully correct protocol stream: `-11` in
28+
`test_stateless_across_invocations`, `-6` in `test_serve_subprocess_e2e`. The
29+
window is wider wherever `TIKTOKEN_CACHE_DIR` is unset, because the load then
30+
takes the no-timeout network path.
31+
32+
The module's loop-protection intent is unchanged: what it defends against is a
33+
minutes-long, no-timeout HTTP fetch inside `get_encoding()`, and that is still
34+
on the daemon thread. The import is local and needs no network.
35+
36+
### Consumer action
37+
38+
None required — `get_encoding_nonblocking()` keeps its signature and its
39+
contract (returns `None` while loading; callers still fall back to a heuristic).
40+
One timing difference worth knowing: the *first* call for an encoding name now
41+
spends 24-29 ms importing tiktoken on the calling thread, where before it
42+
returned instantly. In exchange the encoder becomes available sooner, which
43+
shortens the heuristic-fallback window `tokens.py` documents as the opening
44+
turns of every process.
45+
1046
## [0.8.0] - 2026-09-05
1147

1248
### Added

‎agent_core/runtime/loop/tokenizer.py‎

Lines changed: 86 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,34 @@
1717
``None``; callers fall back to a chars/4 heuristic until the encoder
1818
lands. The loop thread NEVER blocks on tiktoken, cache-baked or not.
1919
20-
NOTE: ``llm_client.py`` keeps an equivalent private copy (with an
21-
``o200k_base`` → ``cl100k_base`` preference) that predates this module
22-
and is left untouched to avoid churning tested code. New callers should
23-
use :func:`get_encoding_nonblocking`.
20+
What is deliberately NOT on that daemon thread is ``import tiktoken``.
21+
``tiktoken._tiktoken`` is a Rust extension, so the import ``dlopen``s a
22+
shared object; CPython kills daemon threads mid-flight during
23+
finalization (``pthread_exit`` at the next GIL acquisition), and being
24+
killed inside the dynamic linker is not survivable — losing glibc's
25+
``_dl_load_lock`` shows up as a later SIGSEGV, and unwinding through a
26+
Rust / ``extern "C"`` frame calls ``abort()``. The 2026-09-05
27+
investigation found the thread parked inside that import at
28+
interpreter-exit on *every* short run, with two observed deaths after a
29+
fully correct protocol stream (``-11`` in
30+
``test_stateless_across_invocations``, ``-6`` in
31+
``test_serve_subprocess_e2e``). The import is a local dlopen — 24-29 ms,
32+
no network, nothing the docstring above is defending against — so it
33+
belongs on the caller thread. Only ``get_encoding()`` (140 ms+, and
34+
unbounded on a cache miss) needs the thread.
35+
36+
Belt and braces: an ``atexit`` hook joins any in-flight init, following
37+
``providers/nonblocking_stream.py``. CPython runs ``atexit`` callbacks
38+
before it starts killing daemon threads, so that is the last point at
39+
which the load can be drained cleanly.
2440
"""
2541

2642
from __future__ import annotations
2743

44+
import atexit
2845
import logging
2946
import threading
47+
import time
3048
from typing import Any
3149

3250
logger = logging.getLogger(__name__)
@@ -43,39 +61,86 @@
4361
_encoders: dict[str, Any] = {}
4462
_lock = threading.Lock()
4563

64+
# In-flight init threads, keyed by encoding name, so ``_join_pending`` can
65+
# drain them at exit. Each thread removes its own entry when it finishes.
66+
_threads: dict[str, threading.Thread] = {}
67+
_atexit_registered = False
68+
69+
# Long enough for a cache-hit ``get_encoding`` (~140 ms) to land, short
70+
# enough that a wedged network fetch cannot hold up process exit. A load
71+
# that misses this deadline is left where it is — the thread no longer
72+
# touches the dynamic linker, which is what made the mid-flight kill
73+
# dangerous in the first place.
74+
_JOIN_TIMEOUT_S = 1.0
75+
76+
77+
def _join_pending() -> None:
78+
"""atexit: drain in-flight inits before the interpreter kills them."""
79+
with _lock:
80+
pending = list(_threads.values())
81+
deadline = time.monotonic() + _JOIN_TIMEOUT_S
82+
for thread in pending:
83+
remaining = deadline - time.monotonic()
84+
if remaining <= 0:
85+
break
86+
thread.join(timeout=remaining)
87+
4688

47-
def _load(name: str) -> None:
48-
"""Blocking tiktoken init — only ever runs on a daemon thread."""
89+
def _load(name: str, tiktoken: Any) -> None:
90+
"""Blocking ``get_encoding`` — only ever runs on a daemon thread.
91+
92+
``tiktoken`` is passed in already imported: this function must not
93+
import anything, see the module docstring.
94+
"""
4995
enc: Any
5096
try:
51-
import tiktoken # pyright: ignore[reportMissingImports]
52-
5397
enc = tiktoken.get_encoding(name) # pyright: ignore[reportUnknownMemberType,reportUnknownVariableType]
54-
except Exception: # not installed / fetch failure / bad name
98+
except Exception: # fetch failure / bad name
5599
enc = False
56100
logger.debug("tiktoken encoding %r unavailable; using heuristic", name)
57101
with _lock:
58102
_encoders[name] = enc
103+
_threads.pop(name, None)
59104

60105

61106
def get_encoding_nonblocking(name: str = "cl100k_base") -> Any | None:
62107
"""Return the cached tiktoken encoder for ``name`` without ever blocking.
63108
64-
The first call schedules a daemon-thread init and returns ``None``;
65-
later calls return the encoder once it has loaded, or ``None`` while
66-
it is still loading. Returns ``None`` permanently when tiktoken is
67-
unavailable — callers MUST fall back to a heuristic on ``None``.
109+
The first call for a name imports tiktoken on the calling thread (a
110+
local dlopen, no network), schedules the encoder init on a daemon
111+
thread and returns ``None``; later calls return the encoder once it
112+
has loaded, or ``None`` while it is still loading. Returns ``None``
113+
permanently when tiktoken is unavailable — callers MUST fall back to
114+
a heuristic on ``None``.
68115
"""
116+
global _atexit_registered
117+
69118
enc = _encoders.get(name, _MISSING)
70119
if enc is not _MISSING:
71120
return enc or None # None (loading) and False (failed) both collapse to None
121+
122+
# On the caller thread, deliberately — never on the daemon thread.
123+
try:
124+
import tiktoken # pyright: ignore[reportMissingImports]
125+
except Exception: # not installed
126+
with _lock:
127+
_encoders[name] = False
128+
logger.debug("tiktoken unavailable; using heuristic")
129+
return None
130+
72131
with _lock:
73-
if _encoders.get(name, _MISSING) is _MISSING: # still unclaimed under the lock
74-
_encoders[name] = None # mark loading so concurrent callers don't re-spawn
75-
threading.Thread(
76-
target=_load,
77-
args=(name,),
78-
name=f"tiktoken-init-{name}",
79-
daemon=True,
80-
).start()
132+
if _encoders.get(name, _MISSING) is not _MISSING: # claimed while we imported
133+
return _encoders[name] or None
134+
_encoders[name] = None # mark loading so concurrent callers don't re-spawn
135+
thread = threading.Thread(
136+
target=_load,
137+
args=(name, tiktoken),
138+
name=f"tiktoken-init-{name}",
139+
daemon=True,
140+
)
141+
_threads[name] = thread
142+
if not _atexit_registered:
143+
atexit.register(_join_pending)
144+
_atexit_registered = True
145+
thread.start()
81146
return None

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "apodex-agent-core"
3-
version = "0.8.0"
3+
version = "0.8.1"
44
description = "Shared, product-neutral runtime primitives for Apodex agents"
55
readme = "README.md"
66
license = "Apache-2.0"
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
"""The tiktoken loader must not leave a daemon thread inside an import.
2+
3+
``import tiktoken`` dlopens a Rust extension. A daemon thread killed
4+
mid-import at interpreter finalization takes glibc's ``_dl_load_lock``
5+
with it (later SIGSEGV) or unwinds through an ``extern "C"`` frame
6+
(``abort()``); both were observed in production as a signal death
7+
*after* a fully correct protocol stream. So the import belongs on the
8+
caller thread and only ``get_encoding`` on the daemon thread — these
9+
tests pin which thread runs which, and that the exit hook drains the
10+
load.
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import importlib.util
16+
import sys
17+
import threading
18+
import types
19+
from typing import Any
20+
21+
import pytest
22+
23+
from agent_core.runtime.loop import tokenizer
24+
25+
26+
class _RecordingLoader:
27+
"""Stands in for the real ``tiktoken``, recording the loading thread."""
28+
29+
def __init__(self, gate: threading.Event | None = None) -> None:
30+
self.import_thread: str | None = None
31+
self.encode_thread: str | None = None
32+
self.encoder = object()
33+
self._gate = gate
34+
35+
# -- meta_path finder -------------------------------------------------
36+
def find_spec(self, fullname: str, path: Any = None, target: Any = None) -> Any:
37+
if fullname != "tiktoken":
38+
return None
39+
return importlib.util.spec_from_loader(fullname, self)
40+
41+
def create_module(self, spec: Any) -> types.ModuleType:
42+
return types.ModuleType(spec.name)
43+
44+
def exec_module(self, module: types.ModuleType) -> None:
45+
self.import_thread = threading.current_thread().name
46+
module.get_encoding = self._get_encoding # pyright: ignore[reportAttributeAccessIssue]
47+
48+
# -- the fake tiktoken API -------------------------------------------
49+
def _get_encoding(self, name: str) -> object:
50+
self.encode_thread = threading.current_thread().name
51+
if self._gate is not None:
52+
self._gate.wait(timeout=5.0)
53+
return self.encoder
54+
55+
56+
@pytest.fixture
57+
def clean_tokenizer():
58+
"""Reset the module cache and unhook any real/fake tiktoken."""
59+
saved_module = sys.modules.pop("tiktoken", None)
60+
saved_meta = list(sys.meta_path)
61+
saved_atexit_registered = tokenizer._atexit_registered
62+
tokenizer._encoders.clear()
63+
tokenizer._threads.clear()
64+
tokenizer._atexit_registered = True # don't leak a real atexit hook per test
65+
try:
66+
yield
67+
finally:
68+
tokenizer._join_pending()
69+
tokenizer._encoders.clear()
70+
tokenizer._threads.clear()
71+
tokenizer._atexit_registered = saved_atexit_registered
72+
sys.meta_path[:] = saved_meta
73+
sys.modules.pop("tiktoken", None)
74+
if saved_module is not None:
75+
sys.modules["tiktoken"] = saved_module
76+
77+
78+
def test_import_runs_on_caller_thread_get_encoding_on_daemon(clean_tokenizer) -> None:
79+
loader = _RecordingLoader()
80+
sys.meta_path.insert(0, loader)
81+
82+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is None
83+
# The import must already have happened, synchronously, right here.
84+
assert loader.import_thread == threading.current_thread().name
85+
assert "tiktoken" in sys.modules
86+
87+
tokenizer._join_pending()
88+
assert loader.encode_thread == "tiktoken-init-cl100k_base"
89+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is loader.encoder
90+
91+
92+
def test_atexit_hook_registers_once_and_drains_the_in_flight_load(
93+
clean_tokenizer, monkeypatch: pytest.MonkeyPatch
94+
) -> None:
95+
gate = threading.Event()
96+
loader = _RecordingLoader(gate=gate)
97+
sys.meta_path.insert(0, loader)
98+
registered: list[Any] = []
99+
monkeypatch.setattr(tokenizer.atexit, "register", registered.append)
100+
tokenizer._atexit_registered = False
101+
102+
tokenizer.get_encoding_nonblocking("cl100k_base")
103+
tokenizer.get_encoding_nonblocking("o200k_base")
104+
assert "cl100k_base" in tokenizer._threads # in flight
105+
assert registered == [tokenizer._join_pending]
106+
107+
gate.set()
108+
registered[0]()
109+
assert tokenizer._threads == {} # thread finished and deregistered itself
110+
assert tokenizer._encoders["cl100k_base"] is loader.encoder
111+
112+
113+
def test_atexit_join_timeout_is_shared_across_threads(
114+
clean_tokenizer, monkeypatch: pytest.MonkeyPatch
115+
) -> None:
116+
class _Clock:
117+
now = 10.0
118+
119+
def monotonic(self) -> float:
120+
return self.now
121+
122+
class _PendingThread:
123+
def __init__(self, clock: _Clock) -> None:
124+
self.clock = clock
125+
self.timeouts: list[float] = []
126+
127+
def join(self, timeout: float | None = None) -> None:
128+
assert timeout is not None
129+
self.timeouts.append(timeout)
130+
self.clock.now += 0.4
131+
132+
clock = _Clock()
133+
pending = [_PendingThread(clock) for _ in range(4)]
134+
tokenizer._threads.update({str(i): thread for i, thread in enumerate(pending)}) # type: ignore[arg-type]
135+
monkeypatch.setattr(tokenizer.time, "monotonic", clock.monotonic)
136+
137+
tokenizer._join_pending()
138+
139+
assert [thread.timeouts for thread in pending] == [
140+
[pytest.approx(1.0)],
141+
[pytest.approx(0.6)],
142+
[pytest.approx(0.2)],
143+
[],
144+
]
145+
146+
147+
def test_caller_never_blocks_on_a_wedged_get_encoding(clean_tokenizer) -> None:
148+
gate = threading.Event()
149+
loader = _RecordingLoader(gate=gate)
150+
sys.meta_path.insert(0, loader)
151+
try:
152+
# Returns immediately even though get_encoding is parked.
153+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is None
154+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is None
155+
finally:
156+
gate.set()
157+
158+
159+
def test_missing_tiktoken_is_terminal_and_spawns_no_thread(clean_tokenizer) -> None:
160+
class _Blocker:
161+
def find_spec(self, fullname: str, path: Any = None, target: Any = None) -> Any:
162+
if fullname == "tiktoken":
163+
raise ImportError("no tiktoken")
164+
return None
165+
166+
sys.meta_path.insert(0, _Blocker())
167+
168+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is None
169+
assert tokenizer._threads == {}
170+
assert tokenizer._encoders["cl100k_base"] is False
171+
assert tokenizer.get_encoding_nonblocking("cl100k_base") is None
172+
173+
174+
def test_concurrent_callers_spawn_one_thread(clean_tokenizer) -> None:
175+
gate = threading.Event()
176+
loader = _RecordingLoader(gate=gate)
177+
sys.meta_path.insert(0, loader)
178+
try:
179+
start = threading.Barrier(4)
180+
181+
def call() -> None:
182+
start.wait(timeout=5.0)
183+
tokenizer.get_encoding_nonblocking("cl100k_base")
184+
185+
callers = [threading.Thread(target=call) for _ in range(4)]
186+
for t in callers:
187+
t.start()
188+
for t in callers:
189+
t.join(timeout=5.0)
190+
191+
assert len(tokenizer._threads) == 1
192+
finally:
193+
gate.set()

‎uv.lock‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)