Skip to content

fix: retry errored repos per collection hook (#502) - #576

Open
eelasyot-spec wants to merge 1 commit into
chaoss:mainfrom
eelasyot-spec:fix/502-retry-errored-repos-reschedule
Open

eelasyot-spec wants to merge 1 commit into
chaoss:mainfrom
eelasyot-spec:fix/502-retry-errored-repos-reschedule

Conversation

@eelasyot-spec

Copy link
Copy Markdown

A backlog of new repos in one stream no longer blocks Error retries in others.

Description
retry_errored_repos used a single global check: if any collection hook had a new-repo backlog, Error retries were skipped for every hook. That left errored secondary/facade/ml work stuck whenever core (or another stream) still had pending new repos.

This PR evaluates the new-repo backlog per hook. Idle hooks reset their Error repos (never-collected → Pending; previously collected → Success) while busy hooks still skip retries so they are not overwhelmed. Helpers live in collection_util.py; unit tests cover the ready-hook selection and per-hook SQL.

This PR fixes #502

Notes for Reviewers

  • Same Pending/Success reset rules as before; only the gate changed from global to per-hook.
  • Logger for this task now uses retry_errored_repos (it previously logged under create_collection_status_records).
  • ML is only considered outside Docker, matching the previous counting behavior (that module is not imported in Docker).

Signed commits

  • Yes, I signed my commits.

Generative AI disclosure

Please select one option:

  • This contribution was NOT assisted or created by Generative AI tools.
  • This contribution was assisted or created by Generative AI tools.

If AI tools were used, please provide details below:
- What tools were used? Cursor (Composer)
- How were these tools used? Implementation of the per-hook retry logic, helper extraction, and unit tests; also drafting this PR description.
- Did you review these outputs before submitting this PR? Yes — reviewed the behavior against #502, confirmed unit tests pass, and verified the SQL/status rules match the previous all-hooks path.

Made with Cursor

A backlog of new repos in one stream no longer blocks Error retries in others.
success = CollectionState.SUCCESS.value

return (
f"UPDATE collection_status SET {status_col} = '{pending}'"
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.84%. Comparing base (d5a8686) to head (04ed1dd).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
collectoss/tasks/start_tasks.py 0.00% 18 Missing ⚠️
collectoss/tasks/util/collection_util.py 0.00% 11 Missing ⚠️
Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@MoralCode

MoralCode commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

ML is only considered outside Docker, matching the previous counting behavior (that module is not imported in Docker).

Docker is currently the only supported method of installing/running CollectOSS

Also please signoff your commits



@pytest.mark.unit
def test_build_error_retry_sql_core_resets_pending_and_success():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What problem would be detected if this test fails?

Can we document this for each test? As we add more tests itll start to get really busy and we should make sure to document what each test is intended to indicate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repositories that have error collection state are not getting rescheduled

3 participants