Skip to content

j2kenton/jester

Repository files navigation

My Daily Dad Joke

mydailyjoke.dad

A production ML system that learns individual humor preferences from behavioral signal and uses those learned representations to personalize daily joke delivery — entirely through WhatsApp, with no app install required.

What It Does

Users receive one dad joke per day via WhatsApp. They tap 😂 or 🙄. That single reaction trains two complementary preference models in the background. No app. No account. No context switch.

Preference learning

The recommendation engine is a hybrid of a neural model and a statistical filter, each compensating for the other's weaknesses at different points in the user's journey:

Taste vector (active from first like). Every joke is embedded as a 1,024-dimensional vector by Amazon Titan Embed v2. A user's taste is the weighted mean centroid of their liked joke embeddings — a position in semantic space that automatically captures nuance no hand-labelled axis could: subject matter, register, the type of incongruity used. Recommendations are ranked by cosine distance to this centroid. Updates live on every reaction with no batch retraining required. Referral interactions are weighted 10× to bootstrap cold-start signal when a new user arrives via a friend's share link.

IQR box filter (activates at 10+ likes). A 2D interquartile-range fence over two hand-labelled axes — structure (wordplay → situational) and tone (wholesome → cynical) — that narrows the candidate pool to the user's statistically proven taste region before the embedding layer re-ranks within it. Deliberately delayed to 10 likes to avoid overfitting on sparse early data. A dislike centroid computed from the same axes acts as an explicit repulsion zone, pushing candidates away from patterns the user has consistently rejected.

The two models run in tandem: the IQR box narrows the candidate pool, the taste vector re-ranks within it. New users fall back to top-scored unseen jokes until enough signal accumulates.

Constrained mutation experiments

The system sidesteps the hardest problem in computational humor — generating funny material from scratch — by treating it as constrained contextual adaptation. This mirrors how professional comedians actually work: they rarely invent material in isolation but instead gather baseline concepts, adjust semantic elements, and adapt existing motifs into a personalized repertoire.

Claude Haiku generates single-axis variants of source jokes — setup reframe, punchline rephrase, tone shift, structure shift — one mutation type per variant, producing differences that are measurable and attributable. Variants are delivered in a separate experiment corpus isolated from the preference-learning corpus, so experimental content cannot corrupt the learned taste models. Only generation-0 interactions train the preference models.

Architecture

WhatsApp Cloud API
    │
    ├── Inbound messages  →  jester-webhook Lambda (HMAC verify + enqueue)
    │                               │
    │                          SQS Queue (jester-webhook-queue)
    │                               │
    │                        jester-processor Lambda  →  preference model update
    │                               │
    └── Outbound delivery  ←  jester-cron Lambda  ←  EventBridge hourly trigger
                                    │
                                    └── Aurora Serverless v2 (PostgreSQL + pgvector)
                                              ├── jokes (embeddings, axes, lineage)
                                              ├── interactions (behavioral log)
                                              ├── userProfiles (taste vector, IQR box)
                                              └── deliveryQueue (pre-fetched weekly schedule)

mydailyjoke.dad  →  CloudFront  →  S3 (Next.js static export)
    └── /r/[token]  →  Lambda@Edge  →  referral click logged  →  301 to WhatsApp

Webhook flow: The webhook Lambda verifies the HMAC signature and immediately enqueues the message to SQS, returning 200 to Meta in ~100ms. The processor Lambda consumes the queue asynchronously, handling all business logic and DB writes. This decouples Meta's webhook timeout from Aurora's cold-start latency and eliminates duplicate messages from Meta retries.

  • Delivery scheduling: Day-of-week sharding via MOD(ABS(HASHTEXT(userId)), 7) spreads load across 168 hourly buckets. Each user's week of jokes is pre-fetched once and queued; daily sends are a single-row atomic pop.
  • LLM: Claude Haiku via Amazon Bedrock for joke classification and variant generation.
  • Embeddings: Amazon Titan Embed v2 via Bedrock; stored as vector(1024) with HNSW indexing.
  • Infrastructure: AWS CDK (TypeScript), Aurora Serverless v2, Lambda Function URLs, SQS, CloudFront + S3 for the landing page.

Prerequisites

  • Node.js 20+
  • AWS CLI configured with credentials that have Lambda, SQS, Secrets Manager, and Bedrock access
  • An Aurora PostgreSQL cluster (deploy via npm run infra:deploy)
  • A Meta developer account with a WhatsApp Business app and a permanent US phone number

Setup

1. Install dependencies

npm install

2. Configure environment

cp apps/server/.env.example apps/server/.env.local
# Fill in all values — see apps/server/.env.example for descriptions

3. Create secrets in Secrets Manager

# WhatsApp credentials
aws secretsmanager create-secret \
  --name jester/whatsapp/credentials \
  --secret-string '{"apiToken":"...","phoneNumberId":"...","appSecret":"...","verifyToken":"..."}'

# Phone hash secret (random hex string for hashing phone numbers)
aws secretsmanager create-secret \
  --name jester/phone/hash-secret \
  --secret-string "$(openssl rand -hex 32)"

4. Request and validate the ACM certificate

aws acm request-certificate \
  --domain-name mydailyjoke.dad \
  --subject-alternative-names www.mydailyjoke.dad \
  --validation-method DNS --region us-east-1
# Add the returned CNAME records to DNS, then wait for status ISSUED

5. Deploy infrastructure

npm run infra:deploy

infra:deploy resolves the issued ACM certificate for mydailyjoke.dad and www.mydailyjoke.dad from ACM in us-east-1, then passes the ARN to Amplify at deploy time. The ARN is not committed to the public repo, and the deploy fails if the landing CloudFront distribution would fall back to the default *.cloudfront.net certificate. On first bootstrap, Amplify may choose its default sandbox identifier; set AMPLIFY_SANDBOX_IDENTIFIER in the deploy environment to pin it. For a throwaway sandbox only, opt out with LANDING_ENABLE_CUSTOM_DOMAIN=false LANDING_ALLOW_DEFAULT_CLOUDFRONT_CERT=true.

6. Create SQS queues

# Create DLQ
aws sqs create-queue --queue-name jester-webhook-dlq --region us-east-1

# Create main queue (set redrive policy to DLQ after 3 failures)
aws sqs create-queue --queue-name jester-webhook-queue \
  --attributes VisibilityTimeout=300 --region us-east-1

7. Deploy application code

npm run deploy

8. Apply database migrations

cd apps/server && npx drizzle-kit migrate

9. Seed jokes

npm run db:seed:jokes -w server        # insert seed jokes + embed them
npm run db:seed:variations -w server   # generate LLM variants per joke

10. Register WhatsApp webhook

In the Meta developer dashboard, set your webhook callback URL to:

https://<lambda-function-url>/api/whatsapp/webhook

Use the verifyToken value from your Secrets Manager secret. Subscribe to the messages field.

Running Locally

npm run server     # start Hono server on http://localhost:3000

Deployment

npm run deploy                # deploy application code (webhook + processor + rolling cron + landing)
npm run deploy:code           # same as npm run deploy
npm run infra:deploy          # deploy Amplify-managed infrastructure/resources
npm run deploy:all            # deploy infrastructure, then application code
npm run webhook:deploy        # webhook Lambda only
npm run processor:deploy      # processor Lambda only
npm run cron:deploy           # rolling-wave cron Lambda only
npm run landing:deploy        # Next.js static site → S3 + CloudFront invalidation

Use npm run infra:deploy when changing Amplify resources, IAM, schedules, environment variables, or Amplify-managed catch-all functions. Use npm run deploy for faster code-only updates to the webhook, processor, rolling-wave cron, and landing site.

Useful Scripts

Script What it does
npm run deploy Deploy application code surfaces
npm run deploy:code Same as npm run deploy
npm run infra:deploy Deploy Amplify-managed infrastructure/resources, resolving the landing ACM cert automatically
npm run deploy:all Deploy infrastructure through infra:deploy, then application code
npm run webhook:deploy Deploy webhook Lambda
npm run processor:deploy Deploy processor Lambda
npm run cron:deploy Deploy rolling-wave cron Lambda
npm run landing:deploy Build + sync landing page to S3
npm run whatsapp:send -- --user-id <hash> --message "Thanks." Dry-run a manual WhatsApp reply to a user
npm run db:seed:jokes -w server Seed generation-0 jokes with embeddings
npm run db:seed:variations -w server Generate LLM variants for all gen-0 jokes
npm run db:clear:jokes -w server Clear all jokes and interactions
npm run db:clear:interactions -w server Clear interactions and delivery queue
npm run cron:test -w server Force-invoke the delivery Lambda

Manual WhatsApp replies are dry-run by default. Add --send to deliver:

npm run whatsapp:send -- --user-id <user_id_hash> --message "Thanks for the feedback." --send

The command refuses free-form text sends unless the user has a recent catch-all feedback row, because WhatsApp may reject non-template messages outside the service window. Use an approved template for cold replies. Paused users can still receive manual replies; paused only suppresses daily joke delivery.

Daily Digest

The catch-all classifier runs hourly at :05 UTC and classifies only unclassified rows, so hourly runs do not reprocess old feedback.

An hourly actionable feedback alert runs at :10 UTC. It checks the previous completed UTC hour for classified unresolved feedback and sends an email only when at least one actionable item exists. The hourly alert is metadata-only: it includes counts and category/subcategory/sentiment breakdowns, but no message text, phone numbers, user IDs, or user hashes.

The daily catch-all email also includes a compact growth and engagement section when CATCHALL_DIGEST_GROWTH_ENABLED is unset or true.

Metrics use UTC day boundaries. Same-day metrics report the digest targetDay; closed cohort metrics report targetDay - 1 as the cohort day with a 24-hour observation window:

  • New users from user_profiles.created_at
  • Likes and dislikes from latest-state interactions timestamps, excluding referral interactions
  • Active users as distinct users with any interaction timestamp
  • Paused users from explicit subscription_state_events pause transitions
  • Delivery response rate from the prior-day sent cohort
  • Referral conversion rate from the prior-day click cohort

The digest Lambda accepts { "targetDay": "YYYY-MM-DD" } for deterministic daily replay. Use CATCHALL_DIGEST_TARGET_DAY=YYYY-MM-DD npm run catchall:digest or pass the day as the script argument. Set CATCHALL_DIGEST_GROWTH_ENABLED=false to suppress the growth section while keeping the catch-all digest unchanged. For hourly alert replay, use CATCHALL_DIGEST_MODE=hourly CATCHALL_ALERT_TARGET_HOUR=YYYY-MM-DDTHH:00:00.000Z npm run catchall:digest.

Project Structure

amplify/
  functions/rollingWaveCron/   # Delivery cron Lambda handler
  database/                    # Aurora CDK construct
  webhook/                     # Webhook Lambda + Function URL CDK construct
  landing/                     # CloudFront + S3 + Lambda@Edge CDK construct
  backend.ts                   # CDK stack root

apps/server/
  src/
    lambda.ts                  # Webhook Lambda entry — HMAC verify + SQS enqueue
    processor.ts               # Processor Lambda entry — SQS consumer, all business logic
    index.ts                   # Hono app — webhook + referral routes
  lib/
    db.ts                      # Drizzle client + queries incl. getMatchingJokes
    schema.ts                  # Full Drizzle schema
    iqr.ts                     # IQR box + dislike centroid computation
    embed.ts                   # Titan Embed v2 helper + meanVector
    delivery.ts                # sendJokeToUser
    whatsapp-copy.ts           # All WhatsApp message copy
    validation.ts              # IQR box sanity checks
  drizzle/                     # SQL migration files

apps/web/
  app/                         # Next.js app router (static export)
    page.tsx                   # Landing page
    privacy/                   # Privacy policy
    terms/                     # Terms of service
    data-deletion/             # Data deletion instructions
  components/                  # Shared React components
  lib/whatsapp.ts              # WhatsApp join link helpers

scripts/
  deploy-webhook.mjs           # Bundle + deploy webhook Lambda
  deploy-processor.mjs         # Bundle + deploy processor Lambda
  deploy-cron.mjs              # Bundle + deploy cron Lambda
  deploy-landing.mjs           # Build + S3 sync + CloudFront invalidation

artifacts/
  Project_Interview_Pitch.md   # Interviewer-facing project overview
  ML_Upgrade_Roadmap.md        # Roadmap for deepening the ML layer

About

ML-powered joke personalization engine — learns humor preferences from WhatsApp reactions using neural embeddings and delivers a daily personalised joke, zero app install required.

Topics

Resources

Stars

Watchers

Forks

Contributors