Skip to content

Foundation: Flyway migrations, profiles, externalized config, hygiene - #8

Merged
ShrishRajGupta merged 3 commits into
ShrishRajGupta:mainfrom
shrish-browserstack:refactor/foundation-hygiene
Sep 13, 2026
Merged

ShrishRajGupta merged 3 commits into
ShrishRajGupta:mainfrom
shrish-browserstack:refactor/foundation-hygiene

Conversation

@shrish-browserstack

Copy link
Copy Markdown
Collaborator

Stacks on #7 — merge that first

This branch is built on fix/build-and-ci (PR #7). Until #7 merges, the diff below also shows #7's two commits; once it does, this PR collapses to just the foundation work. Part 2 of the overhaul plan; the trade math is untouched here.

What

Schema is now owned by Flyway

  • flyway-core + flyway-mysql; Hibernate switched to ddl-auto: validate.
  • db/migration/V1__baseline.sql reproduces exactly the schema Hibernate had been generating (captured from a live ddl-auto database).
  • Existing databases are adopted automatically (baseline-on-migrate, baseline-version: 1) — no manual step for anyone already running it.

Profiles & externalized config

  • application.yaml → application.yml + application-dev.yml + application-prod.yml. dev is the default locally (SQL echo on, seed endpoints enabled); deployments set SPRING_PROFILES_ACTIVE=prod.
  • Everything comes from the environment with sane defaults: MYSQL_HOST/PORT/DATABASE|DB/USER/PASSWORD, KAFKA_BOOTSTRAP_SERVERS (the compose/k8s name KAFKA_ADVERTISED_LISTENERS is still honored), STOCK_SHEET_URL, STOCK_PRICE_CRON.
  • The hardcoded LAN IP 172.16.97.42 is gone from application.yaml, Dockerfile, docker-compose.yaml, and KafkaConfig.
  • open-in-view: false, Hikari pool sizing, Kafka bootstrap gets a default instead of failing on a missing env var.
  • The not-yet-wired trade consumer is gated behind app.kafka.consumer-enabled (default false) via @KafkaListener(autoStartup=…). ⚠️ Gotcha worth knowing: spring.kafka.listener.auto-startup does not exist in Spring Boot 3.0.2 (added in 3.1) and is silently ignored — I hit this during verification; the per-listener attribute is the switch that actually works.

Stock ingestion

  • StockServiceImpl: a single parse path (commons-csv, which was a declared-but-unused dependency) shared by the upload endpoint and the scheduled download. Rows upsert by stock name instead of inserting duplicates on every import; header / short / blank rows are skipped rather than aborting; one batched saveAll; HTTP client with connect + read timeouts; failures are logged with context instead of printStackTrace. The near-duplicate updateStocksFromCsv is removed.
  • The @Scheduled price refresh moves out of PortfolioController into StockPriceScheduler.

HTTP semantics

  • PopulateDatabaseController: the three DB-mutating GETs are now POSTs, and the whole controller is @Profile("dev") — seeders can't reach production.
  • PortfolioController: removed the redundant @Controller, the unreachable duplicate null-check, unused imports.

Cleanup

  • Deleted dead Demo.java. Removed jacoco declared as a runtime dependency (the plugin stays), and unused deps okhttp, quartz, commons-io 1.3.2 (2007), java-dotenv. mysql-connector-java → com.mysql:mysql-connector-j. Pinned spring-boot-maven-plugin (was unversioned). -parameters enabled, which silences the repository LocalVariableTable fallback warnings at boot.
  • Dockerfile carries only app settings (dropped the broker env vars and the baked MYSQL_ROOT_PASSWORD). README: populate endpoints corrected, k8s manifest path fixed, new Configuration & Schema section.

Verification (local MySQL 8)

Check Result
mvn verify green — 28 tests, 0 failures (was 27; the consolidated CSV path gained header/short-row and upsert tests)
Existing ddl-auto database, first boot Flyway created flyway_schema_history and recorded V1 as BASELINE; Hibernate validate passed; app started in 2.5 s
Fresh empty database, prod profile V1 applied as SQL, all tables created, validate passed
prod gating POST /api/populate/users → 404; SQL echo lines in log: 0
dev default POST /api/populate/users → 200; old GET verb → 405
Trade flow regression POST /api/trade → 200, GET /api/portfolio/1 → 200
CSV upsert (POST /api/stocks/update) Stock1 updated in place (same id, close_price 999), NewCo inserted — no duplicate rows
Boot noise Kafka consumer connection warnings: 0 (were 10 per boot)

Known and deliberately left for the next PRs: the empty-portfolio "totalPLPercentage":"NaN" (fixed with the real aggregation algorithm in the portfolio PR), Double money (domain PR), Kafka wiring (outbox PR). Also left alone: the explicit jackson-databind 2.15.2 override and the deprecated openjdk:17-jdk-slim base image.

🤖 Generated with Claude Code

shrish-browserstack and others added 2 commits September 13, 2026 02:47
main has not compiled since PR ShrishRajGupta#4, which left an unreachable
System.out.println("tanya") after the return in PortfolioServiceImpl
(and a stray "tanya" line at the end of README.md). Remove both.

Also make the build reproducible on modern JDK 17 toolchains:
- Configure maven-compiler-plugin explicitly with Lombok on the
  annotationProcessorPaths so @DaTa setters are generated reliably
  (classpath processor discovery was failing, so TradeResponseDTO's
  setters went missing); mark Lombok provided and exclude it from the
  Spring Boot fat jar.
- Bump Lombok 1.18.34 -> 1.18.38 and Jacoco 0.8.12 -> 0.8.15 so
  annotation processing and coverage instrumentation work on current
  JDK 17 patch releases.

Replace the release-only, JDK-11, settings.xml-dependent
maven-publish workflow with a CI workflow that runs `mvn verify` on
every push and PR against main (JDK 17) and uploads the Jacoco report.

Verified locally: mvn verify is green, all 27 tests pass, coverage
report generates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Platform work the analytics/positions/outbox PRs build on. No change to
trade math yet.

Schema
- Introduce Flyway (flyway-core + flyway-mysql). Schema is now owned by
  versioned SQL under db/migration; Hibernate runs in ddl-auto=validate.
  V1__baseline.sql reproduces the schema Hibernate generated so far.
  baseline-on-migrate adopts databases created earlier by ddl-auto=update.

Config
- Split application.yaml into application.yml + application-dev.yml +
  application-prod.yml. dev is the default profile locally (SQL echo,
  seed endpoints); deployments set SPRING_PROFILES_ACTIVE=prod.
- Externalize everything: MYSQL_HOST/PORT/DATABASE(|DB)/USER/PASSWORD,
  KAFKA_BOOTSTRAP_SERVERS (KAFKA_ADVERTISED_LISTENERS still honored),
  STOCK_SHEET_URL, STOCK_PRICE_CRON. Remove the hardcoded LAN IP from
  application.yaml, Dockerfile, docker-compose.yaml and KafkaConfig.
- open-in-view=false; Hikari pool sizing; Kafka bootstrap has a default.
- Gate the not-yet-wired trade consumer behind
  app.kafka.consumer-enabled (default false) via @KafkaListener
  autoStartup. Note: spring.kafka.listener.auto-startup does not exist
  in Boot 3.0.2, so the per-listener attribute is the working switch.

Ingestion
- StockServiceImpl: one parse path (commons-csv, actually used now) for
  uploads and the scheduled download; rows upsert by stock name instead
  of inserting duplicates; header/short/blank rows skipped; batch
  saveAll; HTTP client with connect/read timeouts; errors logged, not
  printStackTrace. Drop the duplicate updateStocksFromCsv.
- Move the @scheduled price refresh out of the controller into
  StockPriceScheduler.

HTTP semantics
- PopulateDatabaseController: DB-mutating GETs become POSTs; the whole
  controller is @Profile("dev").
- PortfolioController: drop redundant @controller, dead duplicate
  null-check, unused imports.

Cleanup
- Delete dead Demo.java. Remove jacoco declared as a runtime dependency
  and unused deps okhttp, quartz, commons-io 1.3.2, java-dotenv. Switch
  to com.mysql:mysql-connector-j. Pin spring-boot-maven-plugin. Enable
  -parameters (silences the LocalVariableTable fallback warnings).
- Dockerfile: only app settings (drop broker env vars and the baked root
  password). README: populate endpoints, k8s manifest path, new
  Configuration & Schema section.

Verified locally (MySQL 8): mvn verify green, 28 tests. Existing
ddl-auto database adopted as Flyway BASELINE; fresh database migrated
V1 as SQL with Hibernate validate passing; prod profile hides the seed
endpoints (404) and SQL echo; boot is quiet (no Kafka consumer
connection warnings); trade/portfolio flow and CSV upsert exercised.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ShrishRajGupta
ShrishRajGupta merged commit 5be27ca into ShrishRajGupta:main Sep 13, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants