Skip to content

Run example-app browser tests against the current SDK - #1500

Merged
brianr merged 1 commit into
masterfrom
fix/example-tests-use-current-sdk
Sep 28, 2026
Merged

brianr merged 1 commit into
masterfrom
fix/example-tests-use-current-sdk

Conversation

@brianr

@brianr brianr commented Sep 28, 2026

Copy link
Copy Markdown
Member

What

Stacked on #1487. The base is deps/examples-drop-webpack-dev-server because both PRs edit examples/react-16's README and .gitignore. Merge #1487 first, then retarget this PR to master.

test/examples/react.test.ts and test/examples/webpack.test.ts load committed bundles that were last rebuilt in 2019. They contain rollbar.js 2.7.1 and 2.13.0, so these tests passed or failed regardless of the SDK in the repo. This PR removes those bundles and builds both apps against the current SDK before the browser tests.

How the builds get made

  • New npm run build:test-examples. It runs npm run pack (tarball of the SDK's current dist/), then validate-examples.js -p react-16 webpack. That's the same install-and-build CI already does for every example, limited to these two. validate-examples.js now accepts example names; with none it validates all examples, as before.
  • CI runs it right before "Run browser tests", against the SDK the Build step just produced.
  • npm test runs it first, so the documented command works with no extra setup (about 9s with warm node_modules).
  • npm run test:wtr on its own doesn't build anything. If a build is missing, the example tests fail with, for example, Failed to load examples/react-16/dist/index.html: HTTP 404. Run npm run build:test-examples to build the example apps. Before, a missing bundle failed later with a confusing assertion error.
  • examples/react-16/dist/ and examples/webpack/dist/ are removed and gitignored in each example's .gitignore.

Stale-tarball fix in validate-examples.js

The tarball keeps version 3.1.0 across SDK rebuilds, and an example's package-lock.json from an earlier install pins the old tarball's integrity. npm then reinstalls the old copy from its cache, even after node_modules/rollbar is deleted. I reproduced this locally: after repacking, the example still installed the old SDK. Local runs would have silently tested a stale SDK. The script now removes the example's lockfile and installed rollbar before npm install. Example lockfiles are gitignored and none are committed. CI always installs fresh, so it wasn't affected. This also fixes the same stale install for local runs of validate:examples.

Test updates for the 3.x SDK

Against fresh builds, all 6 tests failed for two reasons, both intended SDK behavior:

  • Requests are sent a tick later. api.postItem schedules the request with setTimeout(0), so the tests now wait setTimeoutAsync(1) before server.respond(). test/examples/universalBrowser.test.ts already does the same.
  • The access token moved to a header. It's sent in X-Rollbar-Access-Token instead of the body, so the tests check the request header. buildJsonPayload no longer embeds it, so the stored-payload test now asserts the token is absent from the stored JSON. The header is added when sendJsonPayload sends it.

I also removed the webpack test's Travis Chrome 62 workaround from 2019, which parsed the user agent to pick an expected message.

Docs

The example READMEs' "commit a new bundle" steps are replaced with how the builds are made now. CONTRIBUTING.md, CLAUDE.md and AGENTS.md mention build:test-examples.

Validation (Node 22, npm 10)

  • Baseline on Drop vulnerable dev servers from old examples (clears 13 Dependabot alerts) #1487's branch: the two tests pass (6/6) against the committed 2019 bundles.
  • With the bundles removed and no build, both files fail with the hint message above.
  • npm run build:test-examples builds both apps in about 9s. The resulting bundles contain 3.1.0, and git status stays clean.
  • npm test, starting with no example builds: both apps built, then 52/52 browser test files and 621 tests passed, then the server suite passed. Exit 0, 27s.
  • npm run lint (--max-warnings 0), npm run format:check and npm run typecheck:tests all pass.
  • validate-examples.js -n -p still lists all 16 examples, -p 4 react-16 webpack still sets 4 jobs, and an unknown name fails with the list of valid names.
  • Merges cleanly with Stop committing the react-replay-webpack4 build #1499 (git merge-tree).

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T16:50:54.006750Z 63ad672 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@brianr
brianr added this pull request to stack #1501 September 28, 2026 17:09
@rollbar-circleci-machine

Copy link
Copy Markdown

AI Agent Review LGTM (openai, openai-astra)

LGTM. No blocking findings were found.

Review: run example-app browser tests against the current SDK

LGTM. I found nothing to fix in the changed lines.

What I checked in the checkout

  • scripts/validate-examples.js: -p react-16 webpack runs in parallel and builds only those two examples (lines 16-27, 96-108). validate:examples (-p alone) still builds all of them. Deleting package-lock.json is safe because examples/.gitignore:1 ignores lockfiles in every example. The rm calls are skipped on dry runs.
  • The new await setTimeoutAsync(1) waits are enough: from Queue.addItem to api.postItem everything runs synchronously (src/queue.js:117-200), and postItem sends inside a setTimeout(0) (src/api.js:83-90). The stored-payload test reads server.requests[0] right after the sendJson click with no wait. That is correct, because sendJsonPayload → postJsonPayload has no async step (src/rollbar.js:142-144, src/api.js:140-151, src/browser/transport.js:95-118).
  • Token assertions match the SDK: payloads no longer carry access_token (src/apiUtility.js:15-17). The token goes in the X-Rollbar-Access-Token header (src/browser/transport/xhr.js:88), and nise keeps the header name as given.
  • The react-16 page resolves its bundle correctly: the production build uses publicPath: '/examples/react-16/dist/' (examples/react-16/webpack.config.js:12).
  • The SDK tarball stays clean: .npmignore:8 excludes examples/, so the new dist/ builds can't leak into rollbar.tgz.

Outside the diff (not findings)

  • examples/webpack4-replay/README.md:23 and examples/react-replay-webpack4/README.md:27 still say to commit up-to-date bundles into examples/webpack/dist/. That is now wrong.
  • The comment in examples/webpack/src/index.html:9 says tests need a different bundle.js path. It is stale: both script tags use the same path.
  • On the Node 22 CI leg, react-16 and webpack are built twice: once by the new step and again by Validate examples. This costs time but causes no failures.
  • npm test now runs build:test-examples first. A local npm test therefore needs network access for the example installs, and an install failure stops test:server from running.

I could not run anything, so I have not confirmed that any test or CI step passes.

@brianr
brianr force-pushed the fix/example-tests-use-current-sdk branch from 63ad672 to 17285ac Compare September 28, 2026 18:25
Base automatically changed from deps/examples-drop-webpack-dev-server to master September 28, 2026 18:34
test/examples/react.test.ts and webpack.test.ts loaded committed bundles
of rollbar.js 2.7.1 and 2.13.0 from 2019, so they passed or failed
regardless of the SDK in the repo. Build those apps fresh instead.

- Add `npm run build:test-examples`: packs the SDK's current dist/ and
  installs and builds react-16 and webpack via validate-examples.js,
  which now accepts example names.
- Run it in CI before the browser tests, and at the start of `npm test`.
  `test:wtr` on its own fails with a hint to run it if a build is missing.
- Remove and gitignore both apps' dist/.
- validate-examples.js now removes an example's gitignored lockfile and
  installed rollbar before installing. The tarball keeps its version
  across SDK rebuilds and the lockfile pins its old integrity, so npm
  otherwise reinstalls the cached copy and local runs test a stale SDK.
- Update the tests for 3.x: requests are sent a tick later, and the
  access token is sent in the X-Rollbar-Access-Token header rather than
  the body (or a stored JSON payload). Drop a 2019 Travis Chrome 62
  workaround.
- Update the example READMEs, CONTRIBUTING.md, CLAUDE.md and AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brianr
brianr force-pushed the fix/example-tests-use-current-sdk branch from 17285ac to 39f0cbd Compare September 28, 2026 18:34
@brianr brianr self-assigned this Sep 28, 2026
@brianr
brianr merged commit 0a339c7 into master Sep 28, 2026
6 checks passed
@brianr
brianr deleted the fix/example-tests-use-current-sdk branch September 28, 2026 19:02
devtools-agent Bot pushed a commit that referenced this pull request Sep 28, 2026
Resolve the conflict in scripts/validate-examples.js between master's
named-example filter (#1500) and this branch's engines.node skip: an
example skipped for the running Node version is no longer reported as
unknown when named, and its skip line is only printed when it was
requested (or when no names are given).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devtools-agent Bot pushed a commit that referenced this pull request Sep 28, 2026
Picks up #1500, whose CI step `npm run build:test-examples` failed here:
lint-action checks out the PR head before later steps run, so the job
used master's ci.yml with this branch's older package.json, which lacks
that script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants