Skip to content

Rotate through all given proxies before letting yagoogle handle the waiting#70

Open
sporetec wants to merge 4 commits into
opsdisk:masterfrom
sporetec:master
Open

Rotate through all given proxies before letting yagoogle handle the waiting#70
sporetec wants to merge 4 commits into
opsdisk:masterfrom
sporetec:master

Conversation

@sporetec

Copy link
Copy Markdown

This rework lets pagodo:

  • Have a new flag and argument to divide the wanted wait time by the amount of unblocked proxies and recalculate on a proxy being blocked. Default behavior stays unchanged
  • Rotate through all given proxies before letting yagoogle handle the 429 to reduce execution time

@opsdisk

opsdisk commented Aug 17, 2022

Copy link
Copy Markdown
Owner

Thanks for submitting a PR @sporetec! Just a heads up, it may be a week or two until I can take a good look at it.

@opsdisk

opsdisk commented Sep 16, 2022

Copy link
Copy Markdown
Owner

Hey @sporetec - haven't forgotten about this! Hoping to dig into the PR soon.

@opsdisk

opsdisk commented Sep 29, 2022

Copy link
Copy Markdown
Owner

@sporetec - Would it be possible to tackle this feature first?

"Rotate through all given proxies before letting yagoogle handle the 429 to reduce execution time"

To be clear, you don't want the script to automatically go to sleep if an HTTP 429 is detected, especially if there are other proxies that can be tried?

As it is currently written, since yagooglesearch_manages_http_429s is not set during the instantiation of a yagooglesearch.SearchClient object (like is done here https://github.com/opsdisk/pagodo/blob/master/pagodo.py#L200), it defaults to True https://github.com/opsdisk/yagooglesearch/blob/master/yagooglesearch/__init__.py#L82

@senf666

senf666 commented Apr 8, 2024

Copy link
Copy Markdown

would be good to get this into master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants