Reduce Open Library's reliance on Internet Archive and fail gracefully when it's slow or down

@RayBB

Feature Request

Problem / Opportunity

When archive.org is slow or down, Open Library slows down or goes down with it. web.py workers are synchronous, so a page that waits on archive.org holds a worker for the whole timeout, and enough of those take the site down (Oct 14, 2026; see #13956).

#13956 caps concurrent IA availability requests from web.py and falls back to Solr's ebook_access on the work and author pages. Many other pages still call archive.org while rendering, some with long timeouts or none at all. This issue collects them so we can reduce our reliance on IA and fail gracefully everywhere.

This gets better on its own as pages move to FastAPI: an async worker keeps serving other requests while one waits on archive.org. When choosing what to migrate next, we should prioritize the pages below that call IA.

Success: when archive.org is slow or down, every page still loads quickly. IA-dependent parts show as unknown or fall back to data we have, instead of blocking.

Proposal

web.py pages that call IA while rendering (not covered by #13956)

Page	IA call	Timeout	Risk
`/search/inside` (`plugins/inside/code.py`)	Full-text search API (`fulltext_search_async` through `async_bridge`)	30s	High. Each slow search holds a worker for up to 30s.
Your own My Books page, `/people/X/books` (`plugins/upstream/mybooks.py`)	`get_loans_of_user` (IA loan API) and `get_loan_history_data` (S3 loan API), not cached	10s each	High. Every logged-in user visiting their own My Books page.
`/account/loans` (`plugins/upstream/account.py`, `templates/account/loans.html`)	Loans, loan history, the user's waiting list, plus `get_waitinglist_size()` once per book in the template	10s each	Medium. Several calls in a row, and more for users with many loans.
`/account/validate` (signup form check)	`account_validation.ia_username_exists`: `requests.get`, no timeout	none	Medium. A hung IA holds the worker until gunicorn's 300s limit.
`/account/create`	`InternetArchiveAccount.xauth("create")`: `requests.post`, no timeout	none	Medium, same reason.
Homepage carousels (`core/carousels.py`)	IA advanced search (`lending.get_available`)	10s	Low to medium. The homepage is cached for 5 minutes, but a failed IA call caches an empty list and immediately retries synchronously, so each homepage cache miss during an outage makes two slow calls.
Loan status for logged-in users on search results, carousels and book pages (`macros/LoanStatus.html`, `lending.get_lending_state`)	`find_loans` and the waiting-list query	10s	Low. Cached for 5–10 minutes and refreshed in the background, so IA is only called synchronously on a cold cache.
`/books/ia:<id>` (`plugins/openlibrary/connection.py`), `/show-records` (`views/showmarc.py`)	IA metadata, MARC fetch	3s	Low. Rare pages, short timeouts.

Not a problem: covers only build URLs and never call IA, the search page's "Search Inside" band is lazy-loaded from a FastAPI partial, and FastAPI's borrow flow is fully async.

Beyond HTML pages

FastAPI login blocks its whole worker. async def login in openlibrary/fastapi/account.py calls audit_accounts() directly, which makes a blocking requests call to IA (xauth / s3auth) with no timeout, on the event loop. While IA hangs, every request on that FastAPI worker stalls, not just the login. Fix: add a timeout and move the call off the event loop (e.g. run_in_threadpool, or an async httpx client).
Borrowing on web.py and the Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956 cap. web.py still has its own borrow route (plugins/upstream/borrow.py). If production routes /books/.../borrow to web.py rather than FastAPI, the availability check inside the borrow goes through the cap, and when the cap is full it would see "error" and the borrow could fail. The routing lives in olsystem's haproxy config, so this needs checking there.

Suggested order

Add timeouts to the two no-timeout calls, and stop the FastAPI login from blocking the event loop. Small and low-risk.
Put /search/inside and the My Books / loans calls behind the same limit as Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956, under their own metric names (fts, loans), or migrate those pages to FastAPI.
Stop the homepage carousels from caching and retrying an empty IA result.
Longer term: put availability in Solr so pages don't need live IA calls at all (blocked on hardware for the reindex).

Keeping up Open Library during IA outage #6976 Keeping up Open Library during IA outage (2022): the original issue for this; this one updates it with today's call sites
Book Availability in Solr #7450 Book Availability in Solr: the long-term fix, so pages don't need live availability calls
Create replicas of key systems to enable automatic failover during downtime #8043 Create replicas of key systems to enable automatic failover during downtime
[Reliability/Security] Many outbound requests calls have no timeout=, blocking workers indefinitely #12704 Many outbound requests calls have no timeout= (closed, but the xauth/s3auth and /account/validate calls above still have none)
Internal Booklending_utils blocking OL on IA downtime #13023 Internal Booklending_utils blocking OL on IA downtime (closed)
Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956 / Disable IA availability request from get_sorted_editions #13344: the concurrency limit and the hotfix it builds on

Breakdown

Implementation Details (for maintainers)

Related files

Refer to this map of common Endpoints:

openlibrary/core/ia_bulkhead.py: the concurrency limit from Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956
openlibrary/core/lending.py: get_loans_of_user, get_loan_history_data, s3_loan_api_async, get_available_async, IA_Lending_API
openlibrary/core/fulltext.py, openlibrary/plugins/inside/code.py: search inside
openlibrary/plugins/upstream/mybooks.py, openlibrary/plugins/upstream/account.py, openlibrary/templates/account/loans.html
openlibrary/accounts/model.py: InternetArchiveAccount.xauth, s3auth, audit_accounts
openlibrary/fastapi/account.py: FastAPI login
openlibrary/core/carousels.py: homepage carousels

Requirements Checklist

Checklist of requirements that need to be satisfied in order for this issue to be closed:

Every outbound IA call made during a web.py request has a timeout
FastAPI login doesn't block the event loop
Confirm whether production routes borrow to web.py or FastAPI
Each high/medium-risk page above is capped, degrades gracefully, or has moved to FastAPI
Homepage carousels don't retry IA synchronously after a failure

Stakeholders

Instructions for Contributors

Before creating a new branch or pushing up changes to a PR, please first run these commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

@RayBB

Context (research findings)

openlibrary/core/lending.py already has 5/10-minute memoized wrappers sitting right next to the two "not cached" calls this issue flags: get_cached_loans_of_user and get_cached_user_waiting_loans. User.get_loan_for() in core/models.py already supports calling either the cached or raw version via a use_cache flag:

openlibrary/openlibrary/core/models.py

Lines 1108 to 1116 in 920c561

    
               def get_loan_for(self, ocaid, use_cache=False): 
        
                   """Returns the loan object for given ocaid. 
        
                   Returns None if this user hasn't borrowed the given book. 
        
                   """ 
        
                   from ..plugins.upstream import borrow  # noqa: F401 side effects may be needed 
        
                   loans = lending.get_cached_loans_of_user(self.key) if use_cache else lending.get_loans_of_user(self.key) 
        
                   for loan in loans:

. mybooks.py calls the raw, uncached get_loans_of_user directly:

openlibrary/openlibrary/plugins/upstream/mybooks.py

Line 78 in 920c561

myloans = get_loans_of_user(mb.me.key)

. This makes the My Books fix smaller than a cap/fallback: switching to the existing cached wrapper (or calling through the model with use_cache=True) removes a synchronous, uncached IA call on every logged-in visit without new infrastructure.

The "concurrency limit from Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956" referenced in this issue's Breakdown doesn't exist in the codebase yet (no ia_bulkhead.py or equivalent on master). Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956 is itself still open, and its implementation is in an open PR: Cap concurrent IA availability requests from web.py, with a Solr fallback #13971 ("Cap concurrent IA availability requests from web.py, with a Solr fallback", opened by @RayBB the same day as this issue). Suggested order step 2 ("put /search/inside and the My Books / loans calls behind the same limit as Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956") is therefore dependent on Cap concurrent IA availability requests from web.py, with a Solr fallback #13971 landing first, not on existing infrastructure.
Disable IA availability request from get_sorted_editions #13344 ("Disable IA availability request from get_sorted_editions", @cdrini) — the hotfix Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956 says it supersedes — is also still open/unmerged.
Confirmed via code: InternetArchiveAccount.xauth()'s requests.post (L882) and s3auth()'s requests.get (L908) both have no timeout= argument, matching the issue's claim for /account/validate and /account/create.

Confirmed via code: FastAPI's async def login calls audit_accounts() directly and synchronously on the event loop, with no run_in_threadpool or async offload:

openlibrary/openlibrary/fastapi/account.py

Lines 402 to 436 in 920c561

    
           async def login( 
        
               request: Request, 
        
               form_data: Annotated[LoginForm, Form()], 
        
           ) -> Response: 
        
               """ 
        
               Login endpoint - works identically to web.py version. 
        
               This endpoint: 
        
               1. Validates email/password against Internet Archive 
        
               2. Creates/links OpenLibrary account if needed 
        
               3. Sets session cookie and other cookies 
        
               4. Redirects to target page 
        
               This reuses all existing authentication logic from the legacy system. 
        
               """ 
        
               # A JSON body follows web.py's account_login_json: S3 keys (access/secret) 
        
               # are required for password-less login. 
        
               if (request.headers.get("content-type") or "").lower().startswith("application/json"): 
        
                   return await _login_json(request) 
        
               # web.py's web.input() merges query string params with the posted form 
        
               # (the public openlibrary/api.py client logs in via query string). 
        
               q = request.query_params 
        
               username = form_data.username or q.get("username", "") 
        
               password = form_data.password or q.get("password", "") 
        
               remember = form_data.remember or q.get("remember", "").lower() in ("yes", "true", "on", "1") 
        
               redirect = form_data.redirect or q.get("redirect", "") 
        
               action = form_data.action or q.get("action", "") 
        
               access = form_data.access or q.get("access") or request.headers.get("x-s3-access") 
        
               secret = form_data.secret or q.get("secret") or request.headers.get("x-s3-secret") 
        
               test = form_data.test or q.get("test", "").lower() in ("yes", "true", "on", "1") 
        
               # Call the EXACT same audit function that web.py uses 
        
               audit = audit_accounts(

.

The /show-records (show_marc) 3s-timeout item in this issue's table overlaps with MARC show-records timeouts #13874 ("MARC show-records timeouts", open, Priority: 1, Lead: @hornc), which is actively investigating live timeout failures on the exact same code path (openlibrary/views/showmarc.py). Worth linking so the two don't get fixed independently/inconsistently.

Issue Assessment

Open questions

Is /account/loans's get_waitinglist_size() really called once per book in the template with no batching? (Answer: Yes — openlibrary/templates/account/loans.html calls book.get_waitinglist_size() inline per row, twice in the template (lines 108 and 174, for the two loan tables), each a separate synchronous IA-backed lookup.)
Does production's olsystem haproxy config route /books/.../borrow to web.py or FastAPI? This determines whether web.py's own borrow route (and the Cap concurrent Internet Archive availability requests globally so a slow archive.org can't take down Open Library #13956/Cap concurrent IA availability requests from web.py, with a Solr fallback #13971 availability cap) is actually reachable in production. olsystem is a private/internal repo, so this can't be confirmed from openlibrary alone — needs someone with infra access.

🔔 Stakeholders

@cdrini — author of #13344 (the hotfix this issue's premise references) and owner of the #13956/#13971 concurrency-cap work this issue's Suggested-order step 2 depends on; also the listed search/frontend-performance lead relevant to the /search/inside item.

Full triage checklist

Note

This comment was automatically generated by Richy, Open Library's issue enrichment assistant, on behalf of @mekarpeles. This is NOT a sign-off. This issue needs lead review before work begins.

@RayBB

@RayBB, I would like to work on this issue.

@RsbhThakur

@RsbhThakur this one is still in the proposal phase with more to flesh out. If there's a particular part you think is easy and clear that we should do you can open a PR for it but ootherwise please take a simpler issue!

@RayBB

Hi @RayBB,

I'd like to help with this issue. After reading #13970, #13956/#13971 and #14289, here's what I found:

Cap concurrent IA availability requests from web.py, with a Solr fallback #13971 limits how many IA availability calls run at once, but each call can still wait the full timeout, and loans, accounts, search-inside and carousels aren't covered.
Some account calls to IA (ia_username_exists, xauth, s3auth) have no timeout.
The FastAPI login makes a blocking IA call inside async def, which stalls the whole worker.
My Books uses the uncached get_loans_of_user, although a cached wrapper already exists.
/account/loans asks IA once per book, and the carousels cache an empty result when IA fails.

I'd like to start with timeouts on the account calls. Adding them raises a few edge cases I'd like your view on:

If /account/validate times out, should signup show an error, or let the user through and rely on /account/create to reject duplicates?
/account/create could time out after IA has actually created the account, so a retry might say "already exists". Should that case be handled specially?

A timeout should help a lot, but it only shortens how long each worker is stuck, so the cap in #13971 is still needed for full protection.

Could you let me know which behavior you prefer, and whether anyone is already working on this?After that, I’d like to move on to the FastAPI login fix, then the My Books cache, the carousel caching, and /account/loans, if those are still free.

Thanks!

Navigation Menu

Repository navigation

Reduce Open Library's reliance on Internet Archive and fail gracefully when it's slow or down #13970

Feature Request

Problem / Opportunity

Proposal

web.py pages that call IA while rendering (not covered by #13956)

Beyond HTML pages

Suggested order

Related

Breakdown

Related files

Requirements Checklist

Stakeholders

Instructions for Contributors

🔔 Stakeholders

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def get_loan_for(self, ocaid, use_cache=False):
	"""Returns the loan object for given ocaid.

	Returns None if this user hasn't borrowed the given book.
	"""
	from ..plugins.upstream import borrow # noqa: F401 side effects may be needed

	loans = lending.get_cached_loans_of_user(self.key) if use_cache else lending.get_loans_of_user(self.key)
	for loan in loans:

	async def login(
	request: Request,
	form_data: Annotated[LoginForm, Form()],
	) -> Response:
	"""
	Login endpoint - works identically to web.py version.

	This endpoint:
	1. Validates email/password against Internet Archive
	2. Creates/links OpenLibrary account if needed
	3. Sets session cookie and other cookies
	4. Redirects to target page

	This reuses all existing authentication logic from the legacy system.
	"""

	# A JSON body follows web.py's account_login_json: S3 keys (access/secret)
	# are required for password-less login.
	if (request.headers.get("content-type") or "").lower().startswith("application/json"):
	return await _login_json(request)

	# web.py's web.input() merges query string params with the posted form
	# (the public openlibrary/api.py client logs in via query string).
	q = request.query_params
	username = form_data.username or q.get("username", "")
	password = form_data.password or q.get("password", "")
	remember = form_data.remember or q.get("remember", "").lower() in ("yes", "true", "on", "1")
	redirect = form_data.redirect or q.get("redirect", "")
	action = form_data.action or q.get("action", "")
	access = form_data.access or q.get("access") or request.headers.get("x-s3-access")
	secret = form_data.secret or q.get("secret") or request.headers.get("x-s3-secret")
	test = form_data.test or q.get("test", "").lower() in ("yes", "true", "on", "1")

	# Call the EXACT same audit function that web.py uses
	audit = audit_accounts(

Reduce Open Library's reliance on Internet Archive and fail gracefully when it's slow or down #13970

Description

Feature Request

Problem / Opportunity

Proposal

web.py pages that call IA while rendering (not covered by #13956)

Beyond HTML pages

Suggested order

Related

Breakdown

Related files

Requirements Checklist

Stakeholders

Instructions for Contributors

Activity

openlibrary-bot commented on Oct 16, 2026

🔔 Stakeholders

RsbhThakur commented on Oct 16, 2026

RayBB commented on Oct 17, 2026

Ved-sketch commented on Nov 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions