Skip to content

fix: keep the impit client from timing out before the crawler's deadline - #4215

Merged
barjin merged 7 commits into
apify:masterfrom
cruspy2004:fix/impit-timeout
Oct 23, 2026
Merged

barjin merged 7 commits into
apify:masterfrom
cruspy2004:fix/impit-timeout

Conversation

@cruspy2004

Copy link
Copy Markdown
Contributor

Closes #4181

ImpitHttpClient hands impit only the caller's abort signal, so impit's own 30 s default timeout still applies underneath it. With navigationTimeoutSecs: 60, a page that answers after 35 s fails at 30 s with "Request timed out after 60 seconds.", and extendTimeout() can't get past it either.

When an abort signal is passed, the client now gives impit a timeout of about 24 days, so the signal alone decides when the request ends. impit has no way to disable its timeout (0 and Infinity both time out immediately), hence the large value. Without a signal, impit's default is unchanged.

Against a local server that answers after 35 s, the page is now handled at ~36 s with navigationTimeoutSecs: 60, and still fails at 5 s with "Navigation timed out after 5 seconds." when set to 5.

`ImpitHttpClient` passed impit only the caller's abort signal, so impit's
own 30 s default timeout still applied underneath it. Any
`navigationTimeoutSecs` above 30 was ignored, `extendTimeout()` could not
push past it, and the error still named the configured value.

When the caller provides an abort signal, give impit a timeout far beyond
any navigation window so the signal alone decides. Without a signal,
impit's default stays.

Closes apify#4181
@nikitachapovskii-dev

nikitachapovskii-dev commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Thanks! Tested locally and the issue sample works now

A few things though:
FileDownload still gets cut at 30 s. It doesn’t pass a signal, so impit keeps its default.
If the navigation window runs out while the body is still downloading, the download now just keeps going. Before, impit’s 30 s stopped it.
sendRequest with a signal but no deadline can now hang forever. Before, it failed at 30 s.

After the response headers, an aborted signal only stopped impit's body
reads and left the download running, which impit's own timeout used to
cut off. The body is now piped through the signal, so aborting it also
cancels the download.
@cruspy2004

Copy link
Copy Markdown
Contributor Author

Thanks for testing!

  • Download after the window runs out: right. On abort impit stops the body read but keeps the connection downloading. 7e8ad92 pipes the body through the signal, so an abort now cancels the download as well. Against a slow-streaming server, the connection now closes right after the abort.
  • FileDownload: its request runs in the context pipeline outside any timeout frame, so there is no task signal to pass yet. Giving downloads a deadline means choosing one, which felt out of scope here, but I'm happy to follow up if you have a preference.
  • A signal without a deadline: that matches FetchHttpClient, since fetch has no default timeout either. Callers who want a limit can pass timeoutMillis, which gets combined into the signal.

# Conflicts:
#	test/core/impit-http-client.test.ts

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you @cruspy2004! I pushed two commits on top: the switch to impit's new timeout: null, and a fix for HttpCrawler, where the body download kept going after a failed request (the request's signal never aborted once the headers were in). Hope you don't mind 🙏 FileDownload as a follow-up is fine by me.

@barjin
barjin merged commit 6655629 into apify:master Oct 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ImpitHttpClient aborts every request after 30 s, ignoring navigationTimeoutSecs and extendTimeout()

3 participants