Repository navigation
Replace lifetime pull-count ranking with precomputed download-rate data - #60
Conversation
…ad-rate # Conflicts: # src/lib/rest-client/rest-client.test.ts
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID:
Comment |
|
|
||
| const response = await fetch(url); | ||
| if (!response.ok) { | ||
| throw new Error(`HTTP error at offset ${offset}: ${response.status}`); |
There was a problem hiding this comment.
do the response contains actual error. If so shall we include it in the Error object as well
There was a problem hiding this comment.
Same fix as above both now use the shared readErrorBody() helper.
| "maturityMinAgeDays": 15 | ||
| }, | ||
| "packages": { | ||
| "ballerina/http": { |
There was a problem hiding this comment.
shall we order this in the alphabetical order. so it is easier to find the diff in the next run.
There was a problem hiding this comment.
Done . sorted by "org/name" key before writing the file. Regenerated ranking-data.json so the committed version reflects it.
| @@ -0,0 +1,5853 @@ | |||
| { | |||
There was a problem hiding this comment.
is it the right place to add it? it should inside public/ directory. @sm1990
There was a problem hiding this comment.
+1. Anyone else needing the same sorting can refer the json then.
There was a problem hiding this comment.
Done. Moved ranking-data.json to public/ and switched to runtime fetching with in-memory caching, only for “Most Popular”. Verified in the browser with no extra requests on other sorts.
Also fixed the workflow add-paths to use the new public/ location.
3 participants
Description
Replaces "Most Popular" sorting (previously raw lifetime pullCount) with a download-rate metric computed from each connector's latest mature version. This is build-time/scheduled, not live — no new backend, no runtime calls to Central from the Store itself.
A version is mature once it is at least 15 days old. An immature version falls back to the most recent mature version's rate; a package with no mature version ever gets a null rate and sorts to the bottom.
Type of Change
Related Issue(s)
Relates to the connector ranking/prioritization discussion with the team.
Changes Made
fetchPageDatatwice on every initial page loadscripts/generate-ranking-data.js— fetches from Central, computes each package's download rate from its latest mature version.github/workflows/update-ranking-data.yml— runs the generator monthly (or on manual dispatch), opens a PR only when the data changesrest-client.tsto read from the precomputed data, log-transformed before comparing (the real spread across the catalog is several orders of magnitude)Testing Performed
Test Environment
Test Cases
npm test) — 101/101, 6 suitesAlso dry-run tested the scheduled workflow end-to-end on a personal fork: confirmed it correctly opens a PR when the generated data changes, and correctly skips when it doesn't.
Screenshots
Before
(raw lifetime-pullCount order — Twilio, Google Sheets, etc. at the top)
After
(download-rate order — attach the screenshot showing the live Store with
ballerina-org stdlib modules now ranking first)Code Quality Checklist
npm run lintpasses)npm run format:checkpasses)npx tsc --noEmit)Documentation
Performance Impact
Details: "Most Popular" (the default sort) now always takes the full-fetch path instead of a fast, server-sorted single page — it needs the complete catalog in memory to rank by the precomputed data correctly. This makes the default homepage load noticeably slower (several seconds for the full fetch, vs. near-instant before). This tradeoff was discussed and accepted as part of the design; the real fix, if this cost proves unacceptable, would be server-side scoring or pagination-aware computation rather than more client-side optimization.
Breaking Changes
Details: Not a breaking API change, but a real behavior change: the default "Most Popular" order changes significantly. Standard-library (
ballerina-org) modules, which release far more often than third-party connectors, now rank above high-lifetime-usage connectors like Twilio — e.g.ballerina/timeat ~780 downloads/day vs.ballerinax/twilioat ~1/day despite Twilio's far higher total usage. This was demonstrated and discussed with the team and is expected/accepted behavior for this version, not a bug — flagging it here so reviewers aren't surprised.Additional Notes
Still open, not blocking this PR but worth tracking: