Repository navigation
Webserver timeout randomly #64212
Description
Apache Airflow version
3.1.8
If "Other Airflow 3 version" selected, which one?
No response
What happened?
I have a dag scheduled every 10 minutes. The 3rd one always fails ....
On worker side I can see some timeouts :
2026-03-25T10:30:01.055984Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:01.056166Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
--- Supervised process Last chance exception handler ---
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112, in run
result = _execute_task(context=context, ti=ti, log=log)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1523, in _execute_task
result = ctx.run(execute, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 252, in execute
return_value = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 228, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 251, in execute_callable
return runner.run(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/callback_runner.py", line 82, in run
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/dags/live/dags/dag_netbox_devices_maintenance.py", line 69, in link_vms
xcom_value = context["ti"].xcom_pull(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 366, in xcom_pull
values = XCom.get_all(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/xcom.py", line 304, in get_all
msg = SUPERVISOR_COMMS.send(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 206, in send
return self._get_response()
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 269, in _get_response
frame = self._read_frame()
^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 230, in _read_frame
len_bytes = self.socket.recv(4)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/timeout.py", line 37, in handle_timeout
self.log.error("Process timed out", pid=os.getpid())
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
return self._proxy_to_logger(name, event, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
return getattr(self._logger, method_name)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1186, in run
log.exception("Task failed with exception")
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception
return self.error(event, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
return self._proxy_to_logger(name, event, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
return getattr(self._logger, method_name)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1697, in main
state, _, error = run(ti, context, log)
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1191, in run
f"ti.finish.{ti.dag_id}.{ti.task_id}.{state.value}",
^^^^^
UnboundLocalError: cannot access local variable 'state' where it is not associated with a value
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 388, in _fork_main
target()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main
main()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1704, in main
log.exception("Top level error")
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception
return self.error(event, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
return self._proxy_to_logger(name, event, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
return getattr(self._logger, method_name)(*args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe
2026-03-25T10:30:01.425867Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:02.130218Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:02.130481Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325006, now - 4993/1325174, monotonic - 420551.394657526, last_heartbeat_sent - 420531.392953123, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:02.130650Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:30:02.197240Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:06.425046Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:07.196062Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:10.981992Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.982286Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.982729Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.983217Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:11.057156Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.057421Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.057769Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.058240Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:11.425105Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:12.197500Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:14.092457Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '569d87ec-8b31-3e7b-8124-30a33342604d'} ticket:4198e42b-4207-4725-9e0d-f1eba84c01bb] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:16.425168Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:17.196067Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:20.984262Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:20.996950Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bcd4a090> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:20.997314Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.000576Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc65aa50> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.000826Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001151Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001307Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001478Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001610Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.059241Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.080066Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af044f80> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.080298Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.083181Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af023d10> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.083411Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.083728Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.083900Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.084075Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.084207Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.425544Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:22.132032Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:22.132399Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325174, now - 4994/1325333, monotonic - 420571.396551106, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:22.197758Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:26.427229Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:27.196171Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:27.871188Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ec90397f-5794-3113-80d2-7121be96b7b6'} ticket:88299c2e-82b9-4e03-98fd-0323ac37d8fe] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:31.012018Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012272Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012521Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012828Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:31.094629Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.094944Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.095310Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.095613Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:31.425628Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:32.197584Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:36.425520Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:37.196474Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:41.014457Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.018476Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc673cb0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.018686Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.021958Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc672ff0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.022183Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022500Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022699Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022894Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.023032Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.096446Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.099030Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0a090> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.099257Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.102609Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0ae70> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.102853Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103176Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103333Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103505Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103636Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.413066Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b96a3d63-ef49-35b5-87e7-8a7c5f29634b'} ticket:9d6bfcc8-db46-42d9-94df-52b8394f82e9] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:41.425611Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:42.134000Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:42.134280Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325333, now - 4994/1325498, monotonic - 420591.398436219, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:42.134482Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:30:42.197596Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:46.426644Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:47.197483Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:51.024749Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025049Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025313Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025579Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:51.114102Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.114455Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.114779Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.115225Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:51.425463Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:52.197710Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:55.024679Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '24170662-5cfd-39ac-8495-ffa8773da31f'} ticket:4a9d24f7-1bd0-4491-b1bc-1fea82954d40] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:56.426185Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:57.196374Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:01.026491Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.029477Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6816a0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.029706Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.033288Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc680dd0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.033580Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034072Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034313Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034606Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034778Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.116463Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.118559Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee27800> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.118876Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.123325Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee26840> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.123614Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124083Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124279Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124484Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124695Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.425581Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:02.136276Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:31:02.136610Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325498, now - 4995/1325657, monotonic - 420611.400776736, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:31:02.197911Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:06.425600Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:07.196376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:08.547138Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ad3a4b26-88db-32b9-8b72-c6257848e8b6'} ticket:02fcf9a7-2d2e-467c-b411-9e45e8eb1413] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:11.045252Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.045532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.045762Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.046066Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:31:11.135225Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.135532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.135781Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.136095Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:31:11.425077Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:12.197613Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:16.425981Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:17.197156Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:21.046940Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.049988Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6820c0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.050205Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.053581Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc681a00> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.053835Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054143Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054293Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054460Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054587Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.137066Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.139022Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af206ed0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.139237Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.142489Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af205b20> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.142716Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143039Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143200Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143370Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143501Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.426376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:22.110421Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b8885937-29a4-366c-8887-e2a0601f26f9'} ticket:667cb126-3953-471d-8711-07996e013455] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:22.137485Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:31:22.137742Z [debug ] heartbeat_tick : Prev sent/recv: 4995/1325657, now - 4995/1325822, monotonic - 420631.401909165, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:31:22.137919Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:31:22.196920Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:26.426201Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:27.196677Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:31.065029Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.065299Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.065533Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.118060Z [error ] Task execute_workload[51cc03c0-8c9c-4afc-aad0-bd5527c889b1] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
return self._connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
) = self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
event = self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
data = self._network_stream.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read
with map_exceptions(exc_map):
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise
exit_code = process.wait()
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait
self._monitor_subprocess()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess
alive = self._service_subprocess(max_wait_time=max_wait_time) is None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess
need_more = socket_handler(key.fileobj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb
gen.send(request)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests
self._handle_request(msg, log, request.id)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request
xcoms = self.client.xcoms.get_sequence_slice(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice
resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
with map_httpcore_exceptions():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out
2026-03-25T10:31:31.153931Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.154179Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.154394Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.206917Z [error ] Task execute_workload[eedcb4b9-06ba-4366-8e07-cf16ad60bd25] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
resp = self._pool.handle_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
raise exc from None
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
response = connection.handle_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
return self._connection.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
) = self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
event = self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
data = self._network_stream.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read
with map_exceptions(exc_map):
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
supervise(
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise
exit_code = process.wait()
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait
self._monitor_subprocess()
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess
alive = self._service_subprocess(max_wait_time=max_wait_time) is None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess
need_more = socket_handler(key.fileobj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb
gen.send(request)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests
self._handle_request(msg, log, request.id)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request
xcoms = self.client.xcoms.get_sequence_slice(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice
resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get
return self.request(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
response = transport.handle_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
with map_httpcore_exceptions():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out
2026-03-25T10:31:31.425181Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:32.197312Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
On webserver side, nothing at all, evenr with DEBUG log level :( . 90% on my dags works fine, but sometimes, I have some timeout on the webserver. I don't know how to debug it properly.
What you think should happen instead?
no timeout :D.
How to reproduce
No sure, the dag is pretty straight forward, the dag starts logging things, and "nothing".
Operating System
Ubuntu 24.04.4 LTS
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.22.0
apache-airflow-providers-celery==3.17.0
apache-airflow-providers-cncf-kubernetes==10.13.0
apache-airflow-providers-common-compat==1.14.0
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-messaging==2.0.2
apache-airflow-providers-common-sql==1.32.0
apache-airflow-providers-docker==4.5.2
apache-airflow-providers-elasticsearch==6.5.0
apache-airflow-providers-fab==3.4.0
apache-airflow-providers-ftp==3.14.1
apache-airflow-providers-git==0.2.4
apache-airflow-providers-google==20.0.0
apache-airflow-providers-grpc==3.9.2
apache-airflow-providers-hashicorp==4.5.0
apache-airflow-providers-http==6.0.0
apache-airflow-providers-microsoft-azure==13.0.0
apache-airflow-providers-mysql==6.5.0
apache-airflow-providers-odbc==4.12.0
apache-airflow-providers-openlineage==2.11.0
apache-airflow-providers-postgres==6.6.0
apache-airflow-providers-redis==4.4.2
apache-airflow-providers-sendgrid==4.2.1
apache-airflow-providers-sftp==5.7.0
apache-airflow-providers-slack==9.7.0
apache-airflow-providers-smtp==2.4.2
apache-airflow-providers-snowflake==6.10.0
apache-airflow-providers-ssh==4.3.1
apache-airflow-providers-standard==1.12.0
Deployment
Docker-Compose
Deployment details
No response
Anything else?
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://....
AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://....
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__LOGGING__LOGGING_LEVEL: DEBUG
AIRFLOW__CORE__DAGS_FOLDER: "/opt/airflow/dags/live/dags"
AIRFLOW__CORE__LAZY_LOAD_PLUGINS: 'false'
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'False'
AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT: 43200
AIRFLOW__CORE__DEFAULT_TIMEZONE: Europe/Paris
AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_FAILURE: 'true'
AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_RETRY: 'true'
AIRFLOW__SMTP__SMTP_HOST: prod.all.mailhub.dc01
AIRFLOW__SMTP__SMTP_STARTTLS: 'false'
AIRFLOW__SMTP__SMTP_SSL: 'false'
AIRFLOW__API__INSTANCE_NAME: Airflow INFRA PROD DC01
AIRFLOW__API__EXPOSE_CONFIG: 'True'
AIRFLOW__API__SSL_CERT: "/opt/airflow/api.pem"
AIRFLOW__API__SSL_KEY: "/opt/airflow/api.key"
AIRFLOW__CELERY__BROKER_URL: amqp://app_inf_airflow01_prod:blablabla@prod.inf-airflowbroker-mq-amqp.service.consul:5672/inf-airflow3-broker01-prod
AIRFLOW__CELERY__BROKER_USE_SSL: 'False'
AIRFLOW__CELERY__FLOWER_URL_PREFIX: "/flower"
AIRFLOW__CELERY__BROKER_CONNECTION_MAX_RETRIES: 10
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: '20'
AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL: '300'
AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD: '300'
AIRFLOW__DAG_PROCESSOR__DAG_FILE_PROCESSOR_TIMEOUT: '300'
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: '300'
AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: 'False'
TZ: Europe/Paris
HTTP_TIMEOUT: 60
REQUESTS_CA_BUNDLE: "/etc/ssl/certs/ca-certificates.crt"
PYTHONPATH: "/opt/airflow/dags/live/dags"
AIRFLOW__METRICS__STATSD_ON: 'True'
AIRFLOW__METRICS__STATSD_PREFIX: inf_airflow02.prod
AIRFLOW__METRICS__STATSD_HOST: 10.25.30.35
AIRFLOW__METRICS__STATSD_PORT: '9125'
AIRFLOW__METRICS__STATSD_INFLUXDB_ENABLED: 'True'
AIRFLOW__KERBEROS__KINIT_PATH: "/usr/bin/kinit.heimdal"
AIRFLOW__KERBEROS__REINIT_FREQUENCY: '3600'
AIRFLOW__KERBEROS__KEYTABS_PATH: "/opt/airflow/security"
AIRFLOW__KERBEROS__CCACHE_PATH: "/opt/airflow/kerberos_cache"
AIRFLOW__FAB__ENABLE_PROXY_FIX: 'True'
AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
AIRFLOW__LOGGING__REMOTE_LOGGING: 'True'
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://prod-airflow01-logs/logs
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: s3_infra
AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME: '86400'
AIRFLOW__WORKERS__SOCKET_CLEANUP_TIMEOUT: '120'
AIRFLOW__WORKERS__EXECUTION_API_RETRY_WAIT_MIN: '10'
AIRFLOW__WORKERS__MAX_FAILED_HEARTBEATS: '5'
AIRFLOW__WORKERS__EXECUTION_API_TIMEOUT: '10'
AIRFLOW__CORE__XCOM_BACKEND: airflow.providers.common.io.xcom.backend.XComObjectStorageBackend
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: s3://s3_infra@prod-airflow01-logs/xcom
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: '0'
AIRFLOW__API__BASE_URL: https://prod-scheduler-inf.int.test.net
AIRFLOW__CORE__EXECUTION_API_SERVER_URL: https://prodv3.inf-airflow-webserver-http.service.consul:8080/execution/
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Jeoffreybauvin commentedon Mar 25, 2026
I think I found the problem :
Jeoffreybauvin commentedon Mar 25, 2026
Something is wrong with the memory consumption of the webserver, but I can't figure what :
3aa05c64dee6 inf-airflow02-p02.dc01-airflow-webserver 41.68% 9.157GiB / 11.63GiB 78.71% 13.9GB / 191MB 295MB / 422kB 8wjddn279 commentedon Mar 26, 2026
Doesn't the DAG version in Airflow keep incrementing? Like this...
Jeoffreybauvin commentedon Mar 26, 2026
Indeed @wjddn279 , I have the same thing :
wjddn279 commentedon Mar 26, 2026
@Jeoffreybauvin
The dag_version keeps increasing, causing it to consume a lot of memory on the API server.
What does the dag file for vmware_inventory look like?
(please remove your sensitive information in code)
Jeoffreybauvin commentedon Mar 26, 2026
And vmware_inventory.yaml is not beeing modified at all, the dag and its conf are the same since a few months...
wjddn279 commentedon Mar 26, 2026
what is full contents of
clusters_txt = Variable.get('VMWARE_INVENTORY')?I also want to know value of
clustersJeoffreybauvin commentedon Mar 26, 2026
Here it is :
VMWARE_INVENTORY: "[ 'dc01', 'dc02', 'fo_audit_01_dc05', 'fo_audit_01_dc06', 'fo_datalake_01_dc05', 'fo_datalake_01_dc06', 'fo_default_01_dc05', 'fo_default_01_dc06', 'fo_hadoop_01_dc05', 'fo_hadoop_02_dc05', 'fo_hadoop_01_dc06', 'fo_hadoop_02_dc06', 'fo_observability_01_dc05', 'fo_observability_01_dc06', 'fo_observability_02_dc05', 'fo_observability_02_dc06', 'fo_pfm_01_dc05', 'fo_pfm_01_dc06', 'fo_pfm_02_dc05', 'fo_pfm_02_dc06' ]"anavrotski commentedon Mar 26, 2026
#61684
wjddn279 commentedon Mar 26, 2026
@Jeoffreybauvin
It's difficult to identify the cause of the dag_version continuously increasing just by looking at the dag file. Can you check the
datacolumn inserialized_dagtable in the metaDB to see what values are changing?To explain further, the data values are hashed and compared. We need to find the differences in the data values between versions to determine what is causing the hash values to differ. If it's difficult to find, please share two of those data values with us.
Jeoffreybauvin commentedon Mar 26, 2026
Ok, I got it, @wjddn279 ... My airflow cluster is on 2 different datacenters, and I can see the differences between the first DC and the second. The tld is different, and that's what we want. Not sure how I can fix this...
wjddn279 commentedon Mar 27, 2026
What you said means dag-processors are seperated with two DC (machine) right? You can check the environment variable used in dag file is same between two dag-processor. If there are different, it could make dag version keep growing
Jeoffreybauvin commentedon Mar 27, 2026
Yes indeed. After setting my PROXY on the same "value" for the 2 DC, dag version stop growing :).
My Airflow cluster seems stable. I still have this, randomly :
But i'm trying to determine if it's related to this issue :).
wjddn279 commentedon Apr 1, 2026
@Jeoffreybauvin
Although the Dag version stop growing, the memory still increase? There is no improvement after fixing Dag version inflation?
Jeoffreybauvin commentedon Apr 1, 2026
It's better indeed :).
wjddn279 commentedon Apr 1, 2026
good news
Andrushika commentedon Jul 10, 2026
The reporter’s original trigger (API server OOM) was already mitigated earlier in this thread. But from the traceback, it looks like several secondary failures can stack on top of the original timeout, making the final visible exception misleading.
(Some of them seem to have already been addressed by other PRs)
I’d like to share my observations below, and please correct me if this reading is off!
A role-based view of the chain looks roughly like this:
flowchart LR subgraph R["Task runtime"] R1["1. Task calls xcom_pull()<br/>and sends a request via SUPERVISOR_COMMS"] R2["4. Task waits for response frame<br/>in _read_frame() / socket.recv(4)"] R3["5. execution_timeout fires<br/>SIGALRM interrupts the blocked recv"] R4["6. handle_timeout() runs<br/>and tries to log first"] R5["8. AirflowTaskTimeout is masked<br/>by the logging failure"] R6["9. Generic error path / cleanup runs"] end subgraph S["Supervisor"] S1["2. Supervisor receives the task request"] S2["3. Supervisor handles the request<br/>and calls the Execution API / webserver"] S3["3b. Transport/non-HTTP failure<br/>may prevent a normal response"] end subgraph A["Execution API / webserver"] A1["3a. API is slow/unavailable<br/>ReadTimeout / retries / possible OOM context"] end subgraph L["Logging channel"] L1["7. Timeout handler writes a log record<br/>through the task logging channel"] L2["7b. Logging channel is unavailable<br/>BrokenPipeError / OSError"] end subgraph C["Cleanup"] C1["10. Cleanup may hit unset state<br/>and raise UnboundLocalError"] end R1 --> S1 S1 --> S2 S2 --> A1 A1 -. timeout / transport failure .-> S3 S3 -. no or late response .-> R2 S3 -. supervisor/request failure may leave<br/>logging channel unavailable .-> L2 R2 --> R3 R3 --> R4 R4 --> L1 L1 --> L2 L2 --> R5 R5 --> R6 R6 --> C1The related PRs seem to cut different failure or masking points in this chain:
Don't crash supervisor IPC loop on transient network errors #66572 → steps 3 / 3b
If supervisor request handling hits a transport or non-HTTP exception, it now returns an
ErrorResponseand keeps the request loop alive. This cuts one path where the task could otherwise keep waiting without a usable response.fix: guard the to.finish stats emission #67955 → step 10
If
statewas never assigned because an earlier error path failed, cleanup should not raiseUnboundLocalErrorwhile emittingti.finishstats.Preserve AirflowTaskTimeout when timeout logging fails #69713 → steps 6 / 7 / 8 (opened as a follow-up, not merged yet)
handle_timeout()should treat timeout logging as best-effort. If the log write fails withOSError, it should still raiseAirflowTaskTimeoutinstead of letting the logging failure mask the timeout.(I may still be missing details around the exact supervisor/log socket lifecycle, since I have not reproduced this locally.)