Repository navigation
fix: keep the impit client from timing out before the crawler's deadline - #4215
Conversation
`ImpitHttpClient` passed impit only the caller's abort signal, so impit's own 30 s default timeout still applied underneath it. Any `navigationTimeoutSecs` above 30 was ignored, `extendTimeout()` could not push past it, and the error still named the configured value. When the caller provides an abort signal, give impit a timeout far beyond any navigation window so the signal alone decides. Without a signal, impit's default stays. Closes apify#4181
|
Thanks! Tested locally and the issue sample works now A few things though: |
After the response headers, an aborted signal only stopped impit's body reads and left the download running, which impit's own timeout used to cut off. The body is now piped through the signal, so aborting it also cancels the download.
|
Thanks for testing!
|
# Conflicts: # test/core/impit-http-client.test.ts
barjin
left a comment
There was a problem hiding this comment.
lgtm, thank you @cruspy2004! I pushed two commits on top: the switch to impit's new timeout: null, and a fix for HttpCrawler, where the body download kept going after a failed request (the request's signal never aborted once the headers were in). Hope you don't mind 🙏 FileDownload as a follow-up is fine by me.
Closes #4181
ImpitHttpClienthands impit only the caller's abort signal, so impit's own 30 s default timeout still applies underneath it. WithnavigationTimeoutSecs: 60, a page that answers after 35 s fails at 30 s with "Request timed out after 60 seconds.", andextendTimeout()can't get past it either.When an abort signal is passed, the client now gives impit a timeout of about 24 days, so the signal alone decides when the request ends. impit has no way to disable its timeout (
0andInfinityboth time out immediately), hence the large value. Without a signal, impit's default is unchanged.Against a local server that answers after 35 s, the page is now handled at ~36 s with
navigationTimeoutSecs: 60, and still fails at 5 s with "Navigation timed out after 5 seconds." when set to 5.