Skip to content

Raise RatelimitException for HTTP 429 responses - #479

Open
KetanHegde wants to merge 1 commit into
deedy5:mainfrom
KetanHegde:fix/raise-ratelimit-exception
Open

KetanHegde wants to merge 1 commit into
deedy5:mainfrom
KetanHegde:fix/raise-ratelimit-exception

Conversation

@KetanHegde

Copy link
Copy Markdown

Summary

Closes #478.

This PR raises RatelimitException when a search engine request receives an HTTP 429 response, allowing callers to distinguish rate limiting from other DDGSException errors.

Additionally, _search_sync() now preserves RatelimitException by re-raising it instead of wrapping it as a generic DDGSException.

Changes

  • Raise RatelimitException for HTTP 429 responses in BaseSearchEngine.request().
  • Preserve RatelimitException in DDGS._search_sync() by re-raising it.

Testing

  • Ran ruff check .
  • Ran pytest

All tests passed.

@roli-lpci

Copy link
Copy Markdown

The HTTP 429 classification is useful, but the new immediate re-raise in _search_sync() also aborts a metasearch that already has a healthy result. I reproduced this offline at head ef3299599107dd2202491b7f8fe4a31b8f750a42 using two engine stubs (no requests to search providers):

from threading import Event
from ddgs import DDGS
from ddgs.exceptions import RatelimitException
from ddgs.results import TextResult

ready = Event()
class Healthy:
    name = provider = 'healthy'
    def search(self, *args, **kwargs):
        ready.set()
        return [TextResult(title='Python', href='https://example.org/python', body='Python programming')]
class Limited:
    name = provider = 'limited'
    def search(self, *args, **kwargs):
        assert ready.wait(2)
        raise RatelimitException('HTTP 429')
client = DDGS(timeout=2)
client._get_engines = lambda *args: [Healthy(), Limited()]
try:
    print(client.text('Python', max_results=10))
except Exception as exc:
    print(type(exc).__name__ + ': ' + str(exc))

PR head prints RatelimitException: HTTP 429; current main returns the healthy Python result with the same script. The issue is the unconditional except RatelimitException: raise, regardless of accumulated results or other backends.

Could the collector retain the rate-limit exception, continue collecting successful results, and raise that typed exception only if the search finishes without results? An offline mixed-success/rate-limit regression plus an all-rate-limited regression would cover both the fallback contract and #478.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RatelimitException is defined but never raised — users cannot catch rate limits distinctly

2 participants