:seedling: Migrate cron/batch scanning pipeline to `ossf/scorecard-infra` by justaugustus · Pull Request #5210 · ossf/scorecard

justaugustus · 2026-09-01T22:54:14Z

What kind of change does this PR introduce?

Infra/Other: completes the repository split of the batch scanning pipeline
that migrate-batch-pipeline (an OpenSpec change in ossf/scorecard-infra)
started. cron/ was already imported into ossf/scorecard-infra with its
full commit history; this PR removes the now-duplicated copy here.

PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

cron/ (the PubSub controller, batch/CII workers, BigQuery transfer,
release webhook, GitHub token-pool server, and the projects.csv/
gitlab-projects.csv scan inventories) lives in this repository, alongside
its Makefile targets, CI jobs, and Dependabot paths. The scan engine
(checker, checks, probes, pkg/scorecard, etc.) does not import any
of it. GCP cron itself stopped running 2026-08-31 when the openssf GCP
project was turned down.

What is the new behavior (if this is a feature change)?

cron/ and clients/githubrepo/roundtripper/tokens/server/ are removed.
cron/README.md becomes a tombstone pointing to
ossf/scorecard-infra's cron/ — retained indefinitely rather than on a
removal date, since the scan inventories are a high-traffic community
contribution surface linked from external docs and old bookmarks.
Makefile targets, CI jobs (docker_matrix, build-matrix's cron entries,
add-projects/validate-projects, build-proto), Dependabot paths, the
.codecov.yml ignore, and .gitignore entries that existed solely to
build/validate the removed code are stripped. All of it already has a
live equivalent in ossf/scorecard-infra's CI.
cloudbuild/ is also removed: both configs there built and pushed to
gcr.io/openssf/scorecard, the same openssf GCP project that stopped
running cron — nothing references them from GitHub Actions.
CONTRIBUTING.md, README.md, and the Pinned-Dependencies check's
Dockerfile example now point at ossf/scorecard-infra.
A new issue-template contact link redirects "add my repo to the weekly
scan" requests there too.
The Makefile's ##@ TODO(#744) help section is renamed to ##@ Images
and its remaining targets (ko-images, scorecard-ko) get help text,
per review feedback.
Tests for the changes have been added (for bug fixes/features)
N/A — this is a deletion. Verified instead via a full build/test/lint
pass (see below).

Which issue(s) this PR fixes

Refs #5208

Special notes for your reviewer

No cron runtime behavior, scan cadence, or output schema changes — this is
a repository split, not a rewrite. No Scorecard check, probe, score, or
output format changes.
cron/data and cron/config are the only public (non-internal)
packages under the removed tree; migrate-batch-pipeline's discovery
phase found zero external Go importers via pkg.go.dev and GitHub code
search, so this is a clean delete rather than one needing a deprecation
window.
clients/githubrepo/roundtripper/tokens/ (the parent package —
roundtripper.go/transport.go's dependency) is untouched; only its
server/ subdirectory (the standalone token-pool RPC server) moved.
Open question for maintainers: [IMPORTANT] Scorecard Infrastructure Changes #5208 is the community notice this PR
relies on for sign-off on the projects.csv contribution path moving. Flag
if the Steering Committee wants a dedicated notice for this specific
removal before it merges.
Rebased onto current main (2026-09-28) and re-verified on this branch:
go build ./..., go vet ./..., SKIP_GINKGO=1 go test ./..., and
make generate-docs (no-op) all pass. golangci-lint run shows the same
two pre-existing findings (policy/policy.go, policy/policy_test.go)
present on plain main — unrelated to this diff. actionlint/zizmor
on the edited workflows show no new findings (fewer, since whole jobs
were deleted).

Does this PR introduce a user-facing change?

The weekly batch-scan pipeline and its projects.csv/gitlab-projects.csv
inventories have moved to ossf/scorecard-infra. ACTION REQUIRED: add new
repositories to the weekly scan there instead of here.

codecov · 2026-09-01T23:01:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.59%. Comparing base (353ed60) to head (ddcbec5).
⚠️ Report is 380 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5210      +/-   ##
==========================================
+ Coverage   66.80%   75.59%   +8.78%     
==========================================
  Files         230      251      +21     
  Lines       16602    13958    -2644     
==========================================
- Hits        11091    10551     -540     
+ Misses       4808     3406    -1402     
+ Partials      703        1     -702

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-09-12T01:38:02Z

This pull request has been marked stale because it has been open for 10 days with no activity

The batch scanning pipeline (PubSub controller, workers, CII worker, BigQuery transfer, release webhook, GitHub token-pool server, and the projects.csv/gitlab-projects.csv scan inventories) has been imported into ossf/scorecard-infra with full commit history. GCP cron stopped running 2026-08-31 when the openssf GCP project was turned down, so this deletion removes no working function. Leave a tombstone at cron/README.md rather than a bare 404: the scan inventories are a high-traffic community contribution surface linked from external docs and old bookmarks. Announced in ossf#5208. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

Remove the Makefile targets, CI jobs, Dependabot paths, and coverage ignore that existed solely to build and validate the now-deleted batch pipeline: six docker/ko image targets, the add-projects/ validate-projects jobs, the build-proto protobuf codegen path (owned by scorecard-infra's cron/data now), and the GitHub token-pool server's build rule. All of this already has a live equivalent in ossf/scorecard-infra's CI (docker_matrix in build-images.yml, add-projects/validate-projects in presubmits.yml, publish-cron-images.yml) -- nothing here drops coverage, it just stops duplicating it upstream. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

Point CONTRIBUTING.md, README.md, and the Pinned-Dependencies check's Dockerfile example at ossf/scorecard-infra, where cron/ (including the projects.csv/gitlab-projects.csv scan inventories) now lives. Also drop CONTRIBUTING.md's two already-dead TOC entries for a dailyscore-cronjob section that no longer has a matching heading, and update cloudbuild/README.md now that only the scorecard image builds here. docs/checks.md is regenerated via `make generate-docs` from the checks.yaml edit, not hand-edited. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

Add an issue-template contact link so someone opening an issue to ask for their repository to be added to the weekly scan lands on ossf/scorecard-infra's contribution path instead of filing here. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

##@ TODO(ossf#744) rendered literally as a `make help` section header. Now that this PR has trimmed the docker/ko image targets down to just ko-images and scorecard-ko, give the section its real name and add descriptions so both targets actually show up under it. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

Both scorecard.yaml and scorecard-tag.yaml built and pushed to gcr.io/openssf/scorecard -- the same openssf GCP project that stopped running the batch scanning pipeline on 2026-08-31. Nothing in .github/workflows references these Cloud Build configs; they were only ever driven by GCP-side triggers, which no longer exist. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>

Copilot

Copilot review overview

🟡 Changes recommended

Address the unresolved documentation, inventory-path, and Go module cleanup findings before approval.

Review effort: Lite
Findings: 1 · 1

Open (2)

Preserve direct URLs to deleted project inventories · New
Remove or replace obsolete Public Data instructions · New

What changed in this PR

Migrates the batch-scanning pipeline and token server to ossf/scorecard-infra, removing duplicated infrastructure from this repository.

Changes:

Removes cron source, tests, fixtures, manifests, and build configurations.
Removes obsolete CI, Dependabot, Codecov, and Cloud Build integration.
Updates documentation and migration links.
Follow-ups remain for stale documentation links, inventory redirects, and unused Go dependencies.

File	Summary
`README.md`	Updates inventory link; public data instructions need follow-up.
`docs/checks/internal/checks.yaml`	Updates Dockerfile reference.
`docs/checks.md`	Updates Dockerfile reference.
`cron/worker/worker.go`	Removes migrated worker.
`cron/worker/worker_test.go`	Removes worker tests.
`cron/README.md`	Adds migration tombstone; direct inventory paths need preservation.
`cron/monitoring/printer.go`	Removes monitoring code.
`cron/monitoring/exporter.go`	Removes monitoring exporter.
`cron/k8s/worker.yaml`	Removes worker deployment.
`cron/k8s/worker.release.yaml`	Removes release worker deployment.
`cron/k8s/webhook.release.yaml`	Removes webhook deployment.
`cron/k8s/transfer.yaml`	Removes transfer job.
`cron/k8s/transfer.release.yaml`	Removes release transfer job.
`cron/k8s/transfer.release-raw.yaml`	Removes raw release transfer job.
`cron/k8s/transfer-raw.yaml`	Removes raw transfer job.
`cron/k8s/README.md`	Removes obsolete deployment documentation.
`cron/k8s/controller.yaml`	Removes controller deployment.
`cron/k8s/controller.release.yaml`	Removes release controller deployment.
`cron/k8s/cii.yaml`	Removes CII job.
`cron/k8s/auth.yaml`	Removes authentication deployment.
`cron/internal/worker/Dockerfile`	Removes worker image definition.
`cron/internal/webhook/main.go`	Removes migrated webhook.
`cron/internal/webhook/Dockerfile`	Removes webhook image definition.
`cron/internal/shuffle/main.go`	Removes migrated shuffler.
`cron/internal/pubsub/subscriber.go`	Removes Pub/Sub subscriber.
`cron/internal/pubsub/subscriber_gocloud.go`	Removes GoCloud subscriber.
`cron/internal/pubsub/subscriber_gocloud_test.go`	Removes subscriber tests.
`cron/internal/pubsub/subscriber_gcs.go`	Removes GCS subscriber.
`cron/internal/pubsub/publisher.go`	Removes Pub/Sub publisher.
`cron/internal/pubsub/publisher_test.go`	Removes publisher tests.
`cron/internal/format/testdata/valid.schema`	Removes schema fixture.
`cron/internal/format/testdata/check6.json`	Removes result fixture.
`cron/internal/format/testdata/check5.json`	Removes result fixture.
`cron/internal/format/testdata/check4.json`	Removes result fixture.
`cron/internal/format/testdata/check3.json`	Removes result fixture.
`cron/internal/format/testdata/check2.json`	Removes result fixture.
`cron/internal/format/testdata/check1.json`	Removes result fixture.
`cron/internal/format/testdata/bq-valid.schema`	Removes schema fixture.
`cron/internal/format/schema_gen.go`	Removes schema generator.
`cron/internal/format/schema_gen_test.go`	Removes schema tests.
`cron/internal/format/mock_doc.go`	Removes format test mock.
`cron/internal/format/json.v2.schema`	Removes JSON schema.
`cron/internal/format/json.raw.schema`	Removes raw JSON schema.
`cron/internal/format/json.go`	Removes JSON formatter.
`cron/internal/format/json_raw_results.go`	Removes raw result formatter.
`cron/internal/format/json_raw_results_test.go`	Removes formatter tests.
`cron/internal/format/bq.raw.schema`	Removes BigQuery schema.
`cron/internal/emulator/README.md`	Removes emulator documentation.
`cron/internal/emulator/projects.csv`	Removes emulator inventory.
`cron/internal/emulator/fakegcs/ossf-scorecard-rawdata/.gitignore`	Removes emulator fixture.
`cron/internal/emulator/fakegcs/ossf-scorecard-data2/.gitignore`	Removes emulator fixture.
`cron/internal/emulator/fakegcs/ossf-scorecard-cron-results/.gitignore`	Removes emulator fixture.
`cron/internal/emulator/fakegcs/ossf-scorecard-cii-data/.gitignore`	Removes emulator fixture.
`cron/internal/emulator/config.yaml`	Removes emulator configuration.
`cron/internal/data/validate/main.go`	Removes inventory validator.
`cron/internal/data/add/testdata/skip_latest.csv`	Removes test fixture.
`cron/internal/data/add/testdata/skip_empty.csv`	Removes test fixture.
`cron/internal/data/add/testdata/skip_empty_2.csv`	Removes test fixture.
`cron/internal/data/add/testdata/skip_duplicates.csv`	Removes test fixture.
`cron/internal/data/add/testdata/no_change.csv`	Removes test fixture.
`cron/internal/data/add/testdata/add_metadata.csv`	Removes test fixture.
`cron/internal/data/add/main.go`	Removes inventory tool.
`cron/internal/data/add/main_test.go`	Removes inventory tool tests.
`cron/internal/controller/testdata/getPrefix/marker`	Removes controller fixture.
`cron/internal/controller/main.go`	Removes batch controller.
`cron/internal/controller/Dockerfile`	Removes controller image definition.
`cron/internal/controller/bucket.go`	Removes bucket input logic.
`cron/internal/controller/bucket_test.go`	Removes controller tests.
`cron/internal/cii/main.go`	Removes CII worker.
`cron/internal/cii/Dockerfile`	Removes CII image definition.
`cron/internal/cdn/client.go`	Removes CDN client.
`cron/internal/cdn/client_test.go`	Removes CDN tests.
`cron/internal/bq/transfer.go`	Removes transfer logic.
`cron/internal/bq/main.go`	Removes transfer executable.
`cron/internal/bq/Dockerfile`	Removes transfer image definition.
`cron/data/writer.go`	Removes inventory writer.
`cron/data/writer_test.go`	Removes writer tests.
`cron/data/testdata/summary_test/invalid/unknown_file`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.26/020003/shard-1234567`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.26/020003/shard-0000001`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.26/020003/shard-0000000`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.26/020003/.shard_metadata`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.19/020001/shard-0000001`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.19/020001/shard-0000000`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.19/020001/.transfer_complete`	Removes summary fixture.
`cron/data/testdata/summary_test/basic/2022.09.19/020001/.shard_metadata`	Removes summary fixture.
`cron/data/testdata/split_file.csv`	Removes iterator fixture.
`cron/data/testdata/split_file_empty.csv`	Removes iterator fixture.
`cron/data/testdata/only_header.csv`	Removes iterator fixture.
`cron/data/testdata/no_header.csv`	Removes iterator fixture.
`cron/data/testdata/ignore_header.csv`	Removes iterator fixture.
`cron/data/testdata/failing_urls.csv`	Removes iterator fixture.
`cron/data/testdata/extra_column.csv`	Removes iterator fixture.
`cron/data/testdata/empty_row.csv`	Removes iterator fixture.
`cron/data/testdata/comment.csv`	Removes iterator fixture.
`cron/data/testdata/blob_test/subdir/nested/key5.txt`	Removes blob fixture.
`cron/data/testdata/blob_test/subdir/key4.txt`	Removes blob fixture.
`cron/data/testdata/blob_test/key3.txt`	Removes blob fixture.
`cron/data/testdata/blob_test/key2.txt`	Removes blob fixture.
`cron/data/testdata/blob_test/key1.txt`	Removes blob fixture.
`cron/data/testdata/basic.csv`	Removes iterator fixture.
`cron/data/testdata/basic-with-gitlab.csv`	Removes iterator fixture.
`cron/data/testdata/basic-gitlab-only.csv`	Removes iterator fixture.
`cron/data/summary.go`	Removes bucket summary logic.
`cron/data/summary_test.go`	Removes summary tests.
`cron/data/request.proto`	Removes batch request protobuf.
`cron/data/README.md`	Removes protobuf documentation.
`cron/data/metadata.proto`	Removes metadata protobuf.
`cron/data/metadata.pb.go`	Removes generated protobuf code.
`cron/data/iterator.go`	Removes inventory iterator.
`cron/data/iterator_test.go`	Removes iterator tests.
`cron/data/format.go`	Removes CSV formatting types; dependency cleanup remains needed.
`cron/data/format_test.go`	Removes formatting tests.
`cron/data/blob.go`	Removes blob utilities.
`cron/data/blob_test.go`	Removes blob tests.
`cron/config/testdata/optional_maps.yaml`	Removes configuration fixture.
`cron/config/testdata/missing_field.yaml`	Removes configuration fixture.
`cron/config/testdata/basic.yaml`	Removes configuration fixture.
`cron/config/config.yaml`	Removes cron configuration.
`cron/cloudbuild/worker.yaml`	Removes migrated build config.
`cron/cloudbuild/webhook.release.yaml`	Removes migrated build config.
`cron/cloudbuild/transfer.yaml`	Removes migrated build config.
`cron/cloudbuild/controller.yaml`	Removes migrated build config.
`cron/cloudbuild/cii.yaml`	Removes migrated build config.
`CONTRIBUTING.md`	Redirects scan contributions; TOC anchor needs updating.
`cloudbuild/scorecard.yaml`	Removes obsolete Cloud Build config.
`cloudbuild/scorecard-tag.yaml`	Removes obsolete tagged build config.
`cloudbuild/README.md`	Removes obsolete Cloud Build documentation.
`clients/githubrepo/roundtripper/tokens/server/main.go`	Removes migrated token server.
`clients/githubrepo/roundtripper/tokens/server/Dockerfile`	Removes token server image definition.
`clients/githubrepo/roundtripper/tokens/server/cloudbuild.yaml`	Removes token server build config.
`.gitignore`	Removes obsolete cron entries.
`.github/workflows/main.yml`	Removes cron and protobuf CI jobs.
`.github/workflows/docker.yml`	Removes cron image jobs.
`.github/ISSUE_TEMPLATE/config.yml`	Adds infrastructure contact link.
`.github/dependabot.yml`	Removes obsolete paths.
`.codecov.yml`	Removes cron coverage exclusion.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+scheduled removal date: `cron/internal/data/projects.csv` and `gitlab-projects.csv`
+are high-traffic community contribution surfaces linked from external docs and old
+bookmarks, and a 404 there costs more than one retained file.


+The list of projects that are checked is available in
+[`cron/internal/data/projects.csv`](https://github.com/ossf/scorecard-infra/blob/main/cron/internal/data/projects.csv)
+in [`ossf/scorecard-infra`](https://github.com/ossf/scorecard-infra), where the batch
+scanning pipeline now lives. If you would like us to track more, please feel free to


github-project-automation Bot added this to OpenSSF Scorecard Sep 1, 2026

justaugustus temporarily deployed to integration-test September 1, 2026 22:54 — with GitHub Actions Inactive

justaugustus temporarily deployed to gitlab September 1, 2026 22:54 — with GitHub Actions Inactive

This was referenced Sep 8, 2026

🌱 Add Posnic POS to weekly Scorecard projects #5212

Closed

🌱 Add intel edge-aosp-bsp public Github repos #4980

Closed

github-actions Bot added the Stale label Sep 12, 2026

Walentalien reviewed Sep 25, 2026

View reviewed changes

Comment thread Makefile Outdated

justaugustus added 6 commits September 28, 2026 01:19

justaugustus force-pushed the remove-cron-batch-pipeline branch from a211d3f to ddcbec5 Compare September 28, 2026 05:28

justaugustus deployed to gitlab September 28, 2026 05:28 — with GitHub Actions Active

justaugustus deployed to integration-test September 28, 2026 05:28 — with GitHub Actions Active

justaugustus removed the Stale label Sep 28, 2026

justaugustus changed the title ~~🌱 Remove cron/ batch pipeline, now hosted at ossf/scorecard-infra~~ 🌱 Migrate cron/batch scanning pipeline to ossf/scorecard-infra Sep 28, 2026

justaugustus marked this pull request as ready for review September 28, 2026 17:52

justaugustus requested a review from a team as a code owner September 28, 2026 17:52

justaugustus requested review from spencerschrock and a lite review from Copilot and removed request for a team September 28, 2026 17:52

Copilot started reviewing on behalf of justaugustus September 28, 2026 18:41 View session

Copilot AI reviewed Sep 28, 2026

View reviewed changes

Navigation Menu

🌱 Migrate cron/batch scanning pipeline to `ossf/scorecard-infra` - #5210

🌱 Migrate cron/batch scanning pipeline to `ossf/scorecard-infra`#5210
justaugustus wants to merge 6 commits into
ossf:mainfrom
justaugustus:remove-cron-batch-pipeline

justaugustus commented Sep 1, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 12, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

justaugustus commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior (if this is a feature change)?

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

Uh oh!

codecov Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 12, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

justaugustus commented Sep 1, 2026 •

edited

Loading

codecov Bot commented Sep 1, 2026 •

edited

Loading