Handle unexpected HTTP 2XX responses#96
Open
edsu wants to merge 2 commits intodigital-science:masterfrom
Open
Conversation
When HTTP 202 responses are returned from the API that can cause query() to return None, which causes problems in query_iterative() which doesn't handle that situation. It seems like we would want to surface unexpected HTTP responses as an exception. This change to query() will treat any unhandled HTTP response as an exception. Fixes digital-science#95 It also can be useful to retry requests when doing iterative queries. If the HTTP 202 responses really are transient they will get retried, like other errors.
I saw this error go by when using query_iterative(): ``` TypeError: '>' not supported between instances of 'NoneType' and 'int' File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112 in run File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1523 in _execute_task File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 252 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 228 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 251 in execute_callable File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/callback_runner.py", line 82 in run File "/opt/airflow/rialto_airflow/dags/harvest.py", line 82 in dimensions_harvest File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 42 in harvest File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 108 in publications_from_orcid File "/opt/airflow/rialto_airflow/harvest/dimensions.py", line 217 in query_with_retry File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 342 in query_iterative File "/home/airflow/.local/lib/python3.12/site-packages/dimcli/core/api.py", line 325 in query_iterative ``` It seems like ensuring `tot` is not `None` here will help matters?
edsu
commented
Apr 14, 2026
|
|
||
| new_skip = skip+limit | ||
| if tot > 0 and new_skip > tot: | ||
| if tot is not None and tot > 0 and new_skip > tot: |
Author
There was a problem hiding this comment.
We ran into a situation where (somehow) tot = None which caused an exception here.
Author
|
@lambdamusic I would be curious to hear what you make of this change. We hadn't been seeing 202 responses for over a year, but suddenly started seeing them in the last week or so. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When HTTP 202 responses are returned from the API that can cause
query()to return None, which causes problems inquery_iterative()which isn't expecting a return value ofNone.It seems like we would want to surface unexpected HTTP responses as an exception. This change to
query()will treat any unhandled HTTP response as an exception.Fixes #95
It also can be useful to retry requests when doing iterative queries. If the HTTP 202 responses really are transient they will get retried, like other responses that aren't explicitly handled.