Self-hosted: static SEO files, canonical policy and responsive media - #1463
Merged
Conversation
Per-tenant robots.txt (with its Sitemap line), sitemap.xml and rss.xml are written by the existing sync pass into the served configs volume, built from one bounded bridge page (latest 100 posts), regenerated on a thirty-minute freshness window (files touched after unchanged writes so writeIfChanged cannot defeat the mtime check) and cleaned up by the same delete path and reconcile sweep as every other served file. nginx serves them per tenant in both server blocks with the generic robots as fallback. Managed instances now link their OWN feed (crawler-visible from the SSI snippet, and getRssFeedUrl follows for the visible link) instead of pointing at ecency.com. Canonical policy: a tenant with a verified custom domain canonicalizes to itself, a subdomain tenant to the ecency.com SSR page (home and per-post alike); og:url stays the served URL. Media: both card layouts and the Journal thumbnail emit srcset and sizes from render-helper's builders, so a phone stops downloading the desktop cut, and the per-post og:image goes through render-helper's own cover extraction and modern proxy path instead of a local regex duplicate and the legacy redirecting sized route. render-helper loads through its CJS build in the API: the package's node ESM entry crashes on a directory import, tracked in #1461. escapeHtml and the excerpt move to their own utils, since config-service, post-meta and seo-files now consume each other.
Code Review by Qodo
1.
|
PR Summary by QodoSelf-hosted: static SEO files, canonical policy, and responsive media srcset
AI Description
Diagram
High-Level Assessment
Files changed (17)
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 620b83920d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const fresh = await TenantService.getByUsername(tenant.username); | ||
| if (!fresh || !isPublishableTenant(fresh)) return; | ||
| await this.writeConfigFile(fresh); | ||
| await this.writeSeoFilesIfStale(fresh); |
There was a problem hiding this comment.
When the Hive RPC is slow or unavailable, every stale tenant can block here for the full 10-second SEO timeout while syncAllConfigs processes tenants sequentially. Tenants later in the list therefore wait up to tenant count × 10s for config publication and stale-file cleanup, and the single-flight guard in src/index.ts skips all five-minute retries while that pass remains active. Run SEO refreshes separately or with bounded concurrency so an external feed outage cannot stall the existing config reconciliation path.
Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment.
Confirmed and fixed in d9317ff: the SEO refresh runs on its own five-minute loop with its own single flight and a four-worker pool, fully decoupled from the serial config pass, so an RPC outage can no longer delay config publication and cleanup.
| ` <link>${link}</link>`, | ||
| ` <guid isPermaLink="true">${link}</guid>`, | ||
| ...(pubDate ? [` <pubDate>${pubDate}</pubDate>`] : []), | ||
| ` <description>${escapeHtml(excerptOf(post.body))}</description>`, |
There was a problem hiding this comment.
For a post whose chain-authored body contains an XML 1.0-disallowed control character such as \u000b, escapeHtml leaves that character intact because it only replaces markup metacharacters. The resulting rss.xml is not well-formed, so one malicious or malformed recent post can make feed readers reject the entire tenant or community feed; strip or replace XML-invalid code points from titles, descriptions, and excerpts before interpolation.
Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment.
Confirmed and fixed in d9317ff: escapeHtml removes XML 1.0-invalid code points (and lone surrogates) before entity escaping, covering every interpolated field in the feed, the sitemap and the meta snippets. Tests pin U+0000, U+000B and non-characters while keeping tab and newline.
Contributor
|
Warning Review limit reached
Next review available in: 43 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughThe change adds static tenant SEO files, tenant-aware canonical metadata, managed RSS routing, responsive post images, and shared escaping and excerpt utilities. ChangesSelf-hosted discovery and media
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Browser
participant Nginx
participant SeoSyncWorker
participant ConfigService
participant HiveRPC
Browser->>Nginx: request tenant /rss.xml or /sitemap.xml
SeoSyncWorker->>ConfigService: synchronize stale tenant artifacts
ConfigService->>HiveRPC: fetch bounded tenant posts
HiveRPC-->>ConfigService: return post records
ConfigService-->>Nginx: publish static SEO artifact
Nginx-->>Browser: return tenant SEO file
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/self-hosted/hosting/api/src/services/seo-files.ts`:
- Around line 115-118: Update the type guard in the raw-record filter to require
created to be a string and, when present, updated to also be a string before
returning TenantPost; otherwise normalize those fields to valid values before
return. Add coverage for malformed date fields mixed into an otherwise valid RPC
response, ensuring SEO sync does not pass non-strings to lastmodOf or pubDateOf.
- Around line 77-113: Update bounded and fetchTenantPosts so each SEO RPC uses
an AbortController created before the call. Pass RPC_TIMEOUT_MS as the third
callRPC argument and the controller.signal as the fifth argument, then abort the
controller when the bounded timeout expires while preserving the existing
timeout rejection behavior.
In `@apps/self-hosted/hosting/api/src/utils/escape-html.ts`:
- Around line 2-8: Update escapeHtml to remove XML 1.0-invalid code points,
including control characters and lone surrogates, before applying the existing
entity replacements. Preserve valid Unicode characters and the current escaping
behavior for &, <, >, double quotes, and apostrophes.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a1fcd3c0-709b-4edd-aec5-f0465862c52f
⛔ Files ignored due to path filters (1)
apps/self-hosted/hosting/api/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (17)
apps/self-hosted/hosting/api/package.jsonapps/self-hosted/hosting/api/src/services/config-publish-lock.test.tsapps/self-hosted/hosting/api/src/services/config-service.tsapps/self-hosted/hosting/api/src/services/hivesigner-registry.test.tsapps/self-hosted/hosting/api/src/services/post-meta.test.tsapps/self-hosted/hosting/api/src/services/post-meta.tsapps/self-hosted/hosting/api/src/services/seo-files.test.tsapps/self-hosted/hosting/api/src/services/seo-files.tsapps/self-hosted/hosting/api/src/utils/escape-html.tsapps/self-hosted/hosting/api/src/utils/excerpt.tsapps/self-hosted/hosting/nginx-multi-tenant.confapps/self-hosted/src/features/blog/components/blog-post-item.tsxapps/self-hosted/src/features/blog/layout/blog-navigation.tsxapps/self-hosted/src/index.tsxapps/self-hosted/src/themes/journal/journal-post-card.tsxapps/self-hosted/src/utils/rss-feed-url.test.tsapps/self-hosted/src/utils/rss-feed-url.ts
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
…dent path The SEO refresh runs on its own five-minute loop with its own single flight and a small worker pool, since spending the chain timeout inside the serial config pass let an RPC outage delay config publication and cleanup by tenant count times ten seconds. The chain call is aborted at its deadline (the SDK timeout plus an AbortController), a malformed answer throws so yesterday's files are kept rather than blanked and marked fresh, every record field the builders touch is type-checked and mtime freshness is fingerprinted by the RPC-free robots content so a custom-domain change regenerates immediately instead of advertising the old address for the window. escapeHtml removes XML 1.0-invalid code points and lone surrogates before escaping, so one chain-authored control character can no longer unparse a tenant's whole feed. The independent path ships: a generate-seo CLI in the hosting-api image runs the same builders from an instance's config.json (its own URL acts as a verified custom domain, so the canonical policy lands on self), DEPLOYMENT.md documents the cron and compose mounts and a general.rssFeedUrl config override points the app's RSS link at the owner's own feed, with junk overrides ignored.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/self-hosted/hosting/api/scripts/generate-seo.ts`:
- Around line 45-67: Replace the regex URL validation with a single URL parse
near the initial argument checks, and validate that the URL uses HTTPS, has no
username or password, query, or fragment, and has pathname “/” so trailing-root
URLs remain valid. Reuse the parsed URL when deriving host in the existing host
initialization instead of constructing a second URL instance, while preserving
the current usage error and exit behavior for invalid input.
In `@apps/self-hosted/hosting/api/src/services/post-meta.ts`:
- Around line 69-73: Update proxyBaseOf in
apps/self-hosted/hosting/api/src/services/post-meta.ts (lines 69-73) to parse
imageProxy with URL, accepting only HTTP(S) URLs with a hostname before removing
trailing slashes; otherwise return null. Apply the same URL parsing and
protocol/hostname validation to rssFeedUrl in
apps/self-hosted/src/utils/rss-feed-url.ts (lines 16-17). Add malformed HTTP(S)
cases in apps/self-hosted/hosting/api/src/services/post-meta.test.ts (lines
114-132) and verify metadata falls back to the default proxy.
In `@apps/self-hosted/hosting/api/src/utils/escape-html.ts`:
- Around line 9-13: Update INVALID_XML_CHARS in
apps/self-hosted/hosting/api/src/utils/escape-html.ts:9-13 to remove the
C1-control ranges and U+FDD0-U+FDEF, preserving those XML 1.0 Fifth Edition
characters while retaining genuinely invalid ranges. In
apps/self-hosted/hosting/api/src/utils/escape-html.test.ts:11-16, update the
test description and add coverage confirming these characters are preserved.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a63eaccd-3c15-4b8d-b7c6-f656ba0d4ed8
📒 Files selected for processing (19)
apps/self-hosted/DEPLOYMENT.mdapps/self-hosted/hosting/api/Dockerfileapps/self-hosted/hosting/api/package.jsonapps/self-hosted/hosting/api/scripts/generate-seo.tsapps/self-hosted/hosting/api/src/index.tsapps/self-hosted/hosting/api/src/services/config-service.test.tsapps/self-hosted/hosting/api/src/services/config-service.tsapps/self-hosted/hosting/api/src/services/hivesigner-registry.test.tsapps/self-hosted/hosting/api/src/services/post-meta.test.tsapps/self-hosted/hosting/api/src/services/post-meta.tsapps/self-hosted/hosting/api/src/services/seo-files.test.tsapps/self-hosted/hosting/api/src/services/seo-files.tsapps/self-hosted/hosting/api/src/utils/escape-html.test.tsapps/self-hosted/hosting/api/src/utils/escape-html.tsapps/self-hosted/src/core/configuration-loader.tsapps/self-hosted/src/features/blog/layout/blog-navigation.tsxapps/self-hosted/src/index.tsxapps/self-hosted/src/utils/rss-feed-url.test.tsapps/self-hosted/src/utils/rss-feed-url.ts
🚧 Files skipped from review as they are similar to previous changes (5)
- apps/self-hosted/src/index.tsx
- apps/self-hosted/hosting/api/package.json
- apps/self-hosted/src/features/blog/layout/blog-navigation.tsx
- apps/self-hosted/src/utils/rss-feed-url.test.ts
- apps/self-hosted/hosting/api/src/services/seo-files.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1451
Closes #1452
Implements the approved proposals from both research issues; static-only, no new request-time compute.
Static SEO files (#1451)
Per-tenant
robots.txt(with its Sitemap line),sitemap.xml(home, about and the latest posts with lastmod) andrss.xml(RSS 2.0 with atom self-link, escaped chain text, RFC822 dates) are written by the existing sync pass into the served configs volume: one bounded bridge page per tenant, a thirty-minute freshness window (files are touched after unchanged writes so write-if-changed cannot defeat the mtime check), the same per-tenant locks, delete path and reconcile sweep as every other served file. nginx serves them per tenant in both server blocks, generic robots as fallback. Managed instances now advertise their OWN feed — crawler-visible from the SSI head snippet — instead of pointing at ecency.com from client-injected markup crawlers never ran.Canonical policy (#1451, as decided)
A tenant with a verified custom domain canonicalizes to itself; a subdomain tenant canonicalizes to the ecency.com SSR page, home and per-post alike. Derived from the tenant row, no config option. og:url stays the served URL.
Responsive media (#1452)
Both card layouts and the Journal thumbnail emit srcset/sizes from render-helper's builders, so phones stop downloading desktop cuts (the CSS height token already pins the boxes, so CLS was bounded). The per-post og:image goes through render-helper's own cover extraction and the modern hashed proxy path instead of a local regex duplicate and the legacy redirecting sized route.
render-helper loads through its CJS build in the hosting API: the package's Node ESM entry crashes at module load on a directory import — that packaging fix plus a body-image dimensions option are filed as #1461, and the missing editor image upload as #1462.
Tests
Summary by CodeRabbit
New Features
robots.txt,sitemap.xml, and RSS feeds.Bug Fixes