From 8057b8e3ec079d8402eeaef4b7e399d2a16ff85d Mon Sep 17 00:00:00 2001 From: Surendar Chandra Date: Mon, 14 Sep 2026 18:45:24 +0000 Subject: [PATCH 1/2] MTR: preserve a pre-existing component manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mysql-test-run.pl treats /bin/mysqld.my as scratch space it owns. create_manifest_file truncates whatever is there (using a two-arg open in write mode) and writes an MTR redirect manifest {"read_local_manifest": true}; remove_manifest_file then unlinks the file. There is no backup and no restore, so a component manifest that was present before the run is silently destroyed. The global manifest is the only mechanism that loads components before InnoDB starts (mysql.component is InnoDB-resident). A distribution or administrator who installs a manifest — for example to load a keyring component for tablespace encryption at rest — loses it the first time MTR runs against that basedir. Design ------ Move the manifest aside (.mtr_saved) before the first server start; a same-directory rename preserves mode and ownership and needs only directory write permission, so a read-only manifest is handled without changing its permissions. Restore it from the single END hook inside its parent-only guard, which is armed before the manifest is touched. The restore is three-state idempotent: * pending — backup exists, rename it back; clear state only on success. * done — no-op (double-restore safe). * (unset) — nothing was saved, nothing to do. Additional safeguards: * A run holds an exclusive flock on .mtr_lock from the backup until the restore, so a second run sharing the basedir dies before touching anything instead of renaming its stub over the backup. The lock file is created shared-mode (0666), as mtr_unique.pm does for build-thread ids, so sequential runs by different users of a group-writable directory can reuse it. * The restore is skipped only when --start-and-exit hands off after the worker reports that the servers are running (and every worker exited 0): the detached server keeps running and needs MTR's manifest in place, and the backup stays adjacent for the next MTR run to adopt and restore. A --start-and-exit run whose setup or server start fails, or whose tests are all skipped, is restored like any other run. * An existing backup from a crashed prior run is adopted only when the file at the manifest path is MTR's own stub; if both hold different content, MTR dies before truncating anything rather than guessing which is current. * When MTR creates the stub itself it also drops a marker (.mtr_stub). At the next start, a file beside a marker is MTR-owned: a leftover stub is unlinked, and anything else (e.g. a manifest written by an interrupted test) is moved to .mtr_leftover with a warning rather than being restored into the installation. * die() is called before truncating if the rename fails. * Every move is a same-directory rename(), never File::Copy::move: it is atomic and has no copy fallback, so a failed restore leaves the stub and the backup untouched for the END-block retry or for adoption by the next run. * Restore state is cleared only after the files are actually gone, so the END-block safety net can retry after a failed unlink or rename. * On exit the stub is unlinked only if it still holds MTR's own text; a stub that changed during the run (e.g. a keyring test that failed before its teardown) is quarantined to .mtr_leftover with a warning, so the next run never adopts it as a user manifest. If that rename fails the marker is kept and the next run quarantines it. * Restore never dies: an unreadable stub is reported and kept for a later retry. * A manifest that cannot be read is reported as such, never as a content conflict. Abort handling: when MTR is interrupted or dies, its workers may still be running a test that rewrites the manifest (keyring tests rename their own backup over it at teardown). The END hook therefore stops and reaps the workers (stop_workers(), shared with the normal exit path) before it restores the manifest. On Windows mtr_error() leaves via POSIX::_exit, which skips END blocks, so the parent installs a pre-exit hook in mtr_report that stops the workers and restores the manifest first. The lifecycle logic is extracted into My::Manifest so it can be tested independently of the MTR harness. Test coverage ------------- mysql-test/lib/t/manifest.t exercises the module with Test::More, following the existing lib/t convention; manifest.t runs under main.mtr_unit_tests, with test descriptions and skip reasons normalized so the recorded result does not depend on the platform or user. Cases: * crashed-prior-run backup adoption * conflicting backup and newly installed manifest → refuse loudly * double-restore no-op * rename failure at create (die before truncate) * whitespace-path open * failed restore leaves stub and backup intact; retry restores, and a next run adopts the backup without a conflict * failed unlink keeps state for retry * leftover-stub (marker) detection * stub-text manifest without marker is preserved * content written after the stub was created is quarantined, not restored * interrupted keyring-style test leaves no permanent manifest * mode/inode preservation across backup and restore * manifest replaced during the run is quarantined at cleanup and not adopted by the next run * stub-write failure after the backup is restored by the END path * unreadable manifest yields an accurate error * unreadable stub at restore is kept and retried * a concurrent run is refused while the first holds the lock; a lock left by a dead run does not block the next one; the lock file is mode 0666 regardless of umask and an existing one is reused * --start-and-exit: a failed setup is restored; a hand-off leaves the stub and backup for the next run, which restores the original * root-unreliable chmod injection (SKIPped when running as root) --- mysql-test/lib/My/Manifest.pm | 335 +++++++++ mysql-test/lib/mtr_report.pm | 4 + mysql-test/lib/t/manifest.t | 1056 ++++++++++++++++++++++++++++ mysql-test/mysql-test-run.pl | 114 ++- mysql-test/r/mtr_unit_tests.result | 186 +++++ mysql-test/t/mtr_unit_tests.test | 2 + 6 files changed, 1669 insertions(+), 28 deletions(-) create mode 100644 mysql-test/lib/My/Manifest.pm create mode 100644 mysql-test/lib/t/manifest.t diff --git a/mysql-test/lib/My/Manifest.pm b/mysql-test/lib/My/Manifest.pm new file mode 100644 index 000000000000..81eea9b89ba5 --- /dev/null +++ b/mysql-test/lib/My/Manifest.pm @@ -0,0 +1,335 @@ +# -*- cperl -*- + +# Copyright (c) 2026, Amazon.com, Inc. or its affiliates. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License, version 2.0, +# as published by the Free Software Foundation. +# +# This program is designed to work with certain software (including +# but not limited to OpenSSL) that is licensed under separate terms, +# as designated in a particular file or component or in included license +# documentation. The authors of MySQL hereby grant you an additional +# permission to link the program and your derivative works with the +# separately licensed software that they have either included with +# the program or referenced in the documentation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License, version 2.0, for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +# +# Manifest lifecycle: save a pre-existing manifest, write an MTR stub, +# and restore the original on exit. +# +# The marker file (.mtr_stub) records that MTR owned the path +# when the current (or crashed-prior) run started. At the next start +# a file beside a marker is MTR-owned: a leftover stub is unlinked, +# and anything else (e.g. a manifest written by an interrupted test) +# is quarantined to .mtr_leftover with a warning rather than +# being restored into the installation. The residual case — someone +# who deliberately overwrote MTR's leftover stub — finds their file +# in .mtr_leftover, not lost. +# +# On exit the stub is unlinked only if it still holds MTR's own text; +# a stub that changed during the run (e.g. a test that failed before +# its teardown) is quarantined to .mtr_leftover at cleanup. +# +# Moving (renaming) the manifest to .mtr_saved in the same +# directory is atomic, preserves mode and ownership, and needs only +# directory write permission, so a read-only manifest is handled +# without changing its permissions. +# +# A run holds an exclusive flock on .mtr_lock from manifest_create +# until the original is restored, so a concurrent run sharing the same +# basedir is refused instead of clobbering the backup (flock is also +# available in Win32 Perl; mtr_unique.pm relies on it too). The lock +# file is created mode 0666, like mtr_unique.pm's build-thread id files. +# +# Extracted from mysql-test-run.pl so the three-state restore logic +# can be unit-tested. +# + +package My::Manifest; + +use strict; +use warnings; +use Carp; +use Fcntl qw(:flock O_RDWR O_CREAT); + +use base qw(Exporter); +our @EXPORT = qw(manifest_create manifest_restore manifest_reset + manifest_hand_off); + +# ---- module-level state (one manifest per MTR process) ---------------- + +# Path to the backed-up pre-existing manifest, if one was preserved. +# Cleared after restore to make the operation idempotent. +my $preserved_manifest; + +# True when MTR itself created the manifest stub (no pre-existing file). +# Only when this flag is set is it safe to unlink on cleanup. +my $manifest_stub_created; + +# Handle holding the exclusive flock on .mtr_lock. The kernel +# drops the lock when the process dies, so a stale lock file is +# harmless. The file itself is never unlinked: a new run could lock +# the old inode while another run creates and locks a new one. +my $lock_fh; + +# True once servers started by --start-and-exit have been handed the +# manifest; manifest_restore then leaves everything for the next run. +my $handed_off = 0; + +# ---- private helpers -------------------------------------------------- + +# _release_lock() — drop the ownership lock, if held. +sub _release_lock { + return unless $lock_fh; + close $lock_fh; + undef $lock_fh; +} + +# _acquire_lock($manifest_file_path) — take the ownership lock or die +# naming the run that holds it. Records this pid in the lock file. +sub _acquire_lock { + my ($manifest_file_path) = @_; + return if $lock_fh; + my $lock_path = $manifest_file_path . ".mtr_lock"; + # Shared-mode lock file, as mtr_unique.pm does for build-thread ids, so + # sequential runs by different users in a group-writable directory can + # reuse it. The chmod is best-effort (only the owner can change it). + my $old_umask = umask(0); + my $opened = sysopen(my $fh, $lock_path, O_RDWR | O_CREAT, 0666); + umask($old_umask); + $opened or die "Could not open manifest lock $lock_path: $!"; + chmod 0666, $lock_path; + if (!flock($fh, LOCK_EX | LOCK_NB)) { + my $owner = '?'; + if (open(my $rh, '<', $lock_path)) { + my $line = <$rh>; + close $rh; + $owner = $1 if defined $line && $line =~ /^(\d+)/; + } + close $fh; + die "Another MTR run (pid $owner) is using $manifest_file_path; " + . "run MTR runs that share a basedir one after another.\n"; + } + # The pid is informational (named in the error above); best-effort. + truncate($fh, 0); + my $old = select($fh); $| = 1; select($old); + print $fh "$$\n"; + $lock_fh = $fh; +} + +# _content_matches($path, $content) — slurp the file and byte-compare. +# Dies on open failure: every caller is about to truncate or unlink +# the path, so an unreadable manifest must fail loudly. +sub _content_matches { + my ($path, $content) = @_; + open my $fh, '<', $path or die "Could not read manifest $path: $!"; + local $/; + my $data = <$fh>; + close $fh; + return defined $data && $data eq $content; +} + +# _write_file($path, $content) — create/truncate $path and write $content. +# Dies on any open/print/close failure. +sub _write_file { + my ($path, $content) = @_; + open(my $fh, '>', $path) or + die "Could not create manifest file $path: $!"; + print $fh $content or + die "Could not write manifest file $path: $!"; + close($fh) or die "Could not write manifest file $path: $!"; +} + +# ---- public API ------------------------------------------------------- + +# manifest_create($manifest_file_path, $config_content) +# +# Save a pre-existing manifest (if any) to .mtr_saved, adopting +# an existing backup rather than clobbering it, then write the +# caller-supplied stub content. A marker file (.mtr_stub) is +# created alongside the stub so that a leftover stub from a killed run +# can be recognised without comparing file content. +sub manifest_create { + my ($manifest_file_path, $config_content) = @_; + croak "usage: manifest_create(, )" + unless defined $manifest_file_path && defined $config_content; + + # Serialize ownership before any -e check: two runs that both saw + # no backup would otherwise rename one's stub over the other's backup. + _acquire_lock($manifest_file_path); + + my $backup = $manifest_file_path . ".mtr_saved"; + my $marker = $manifest_file_path . ".mtr_stub"; + + if (-e $backup) { + # A backup from a prior run that died before restoring already + # holds the real manifest. Adopt it only when the file at + # is absent or is MTR's own stub (left by the crashed run); never + # silently prefer either candidate when both hold real content. + if (-e $manifest_file_path && + !_content_matches($manifest_file_path, $config_content)) { + die "Both $manifest_file_path and its backup $backup exist with " + . "different content (a previous run was interrupted and a new " + . "manifest was installed since). Refusing to guess which one " + . "is current; reconcile them and remove $backup before " + . "re-running.\n"; + } + $preserved_manifest = $backup; + # Remove a stale marker if present. + unlink $marker if -e $marker; + } elsif (-e $manifest_file_path && -e $marker) { + # Marker present ⇒ MTR owned this path when the interrupted run + # started. Whatever content is here now was written after MTR took + # over (a leftover stub, an interrupted test, or someone who + # overwrote the leftover). + if (_content_matches($manifest_file_path, $config_content)) { + # Leftover MTR stub — just re-use the path; unlink on restore. + $manifest_stub_created = 1; + } else { + # Content differs from the stub (e.g. an interrupted keyring test + # wrote a custom manifest). Quarantine it so it is never restored + # into the installation, but also never silently destroyed. + # An existing .mtr_leftover is overwritten by rename (acceptable: + # the previous leftover was equally MTR-era content). + my $leftover = $manifest_file_path . ".mtr_leftover"; + rename($manifest_file_path, $leftover) + or die "Could not move manifest $manifest_file_path " + . "to $leftover: $!"; + warn "Manifest $manifest_file_path was written after MTR created " + . "its stub (marker $marker present, e.g. by an interrupted " + . "test run); moved to $leftover and not restored\n"; + $manifest_stub_created = 1; + } + } elsif (-e $manifest_file_path) { + # No marker, no backup — a manifest installed while no MTR run was + # active. This is a user manifest; move it aside for restore. + # A same-directory rename is atomic, preserves mode and ownership, + # and needs only directory write permission. + rename($manifest_file_path, $backup) + or die "Could not move manifest $manifest_file_path to $backup: $!"; + $preserved_manifest = $backup; + } else { + $manifest_stub_created = 1; + } + + # Drop the marker BEFORE writing the stub so that a crash between + # the two leaves marker-without-stub (harmless: next run sees no + # manifest and overwrites the marker). + if ($manifest_stub_created) { + open(my $mh, '>', $marker) + or die "Could not create marker $marker: $!"; + close($mh) + or die "Could not write marker $marker: $!"; + } + + _write_file($manifest_file_path, $config_content); +} + +# manifest_restore($manifest_file_path, $config_content) +# +# Idempotent restore helper; never dies so the END-block safety net +# always runs to completion. Three states: +# 1. $preserved_manifest set & file exists -> rename backup over manifest; +# clear state only on success so the END-block can retry. +# 2. $manifest_stub_created true -> unlink the MTR-written stub +# only if it still holds MTR's +# own text; a manifest changed +# during the run is quarantined +# to .mtr_leftover. +# An unreadable stub is reported +# and kept for a later retry. +# The marker is removed only +# when the stub is handled. +# Clear state only when +# everything succeeded. +# 3. Neither -> true no-op (second call). +sub manifest_restore { + my ($manifest_file_path, $config_content) = @_; + croak "usage: manifest_restore(, )" + unless defined $manifest_file_path && defined $config_content; + # Servers detached by --start-and-exit keep reading the manifest. + return if $handed_off; + + if (defined $preserved_manifest && -e $preserved_manifest) { + # Same-directory rename: atomic, no copy fallback, so a failure leaves + # the stub and backup intact for retry or next-run adoption. + if (rename($preserved_manifest, $manifest_file_path)) { + $preserved_manifest = undef; + } else { + warn "Could not restore manifest from $preserved_manifest: $!"; + # Leave $preserved_manifest set so the END-block safety net can retry. + } + } elsif ($manifest_stub_created) { + # Stub-only: unlink both the stub and the marker; clear state only + # when both are actually gone so the END-block safety net can retry. + my $marker = $manifest_file_path . ".mtr_stub"; + my $ok = 1; + if (-e $manifest_file_path) { + my $matches = eval { _content_matches($manifest_file_path, + $config_content) }; + if ($@) { + warn "Could not read stub $manifest_file_path; " + . "leaving it in place: $@"; + $ok = 0; + } elsif ($matches) { + unlink $manifest_file_path + or do { warn "Could not unlink stub $manifest_file_path: $!"; + $ok = 0 }; + } else { + # Changed during the run (e.g. a test that failed before its + # teardown). Quarantine it now so the next run never adopts it + # as a user manifest; keep the marker if that fails so the next + # run's manifest_create quarantines it instead. + my $leftover = $manifest_file_path . ".mtr_leftover"; + if (rename($manifest_file_path, $leftover)) { + warn "Manifest $manifest_file_path changed during the run; " + . "moved to $leftover\n"; + } else { + warn "Manifest $manifest_file_path changed during the run and " + . "could not be moved to $leftover: $!"; + $ok = 0; + } + } + } + if ($ok && -e $marker) { + unlink $marker + or do { warn "Could not unlink marker $marker: $!"; + $ok = 0 }; + } + $manifest_stub_created = undef if $ok; + } + # Otherwise: nothing MTR owns to touch -- true no-op. + + # Hold the lock until the original is fully back in place. + _release_lock() + unless defined $preserved_manifest || $manifest_stub_created; +} + +# manifest_hand_off() +# +# Record that servers left running by --start-and-exit now own the +# manifest: manifest_restore becomes a no-op and the backup stays +# adjacent for the next run to adopt. +sub manifest_hand_off { $handed_off = 1 } + +# manifest_reset() +# +# Reset module state. Intended for unit tests only. +sub manifest_reset { + $preserved_manifest = undef; + $manifest_stub_created = undef; + $handed_off = 0; + _release_lock(); +} + +1; diff --git a/mysql-test/lib/mtr_report.pm b/mysql-test/lib/mtr_report.pm index 1b77c6b4161b..155556e13d89 100644 --- a/mysql-test/lib/mtr_report.pm +++ b/mysql-test/lib/mtr_report.pm @@ -76,6 +76,9 @@ our $summary_report_file; our $verbose; our $xml_report_file; +# Called by mtr_error before POSIX::_exit (which skips END blocks). +our $pre_exit_hook; + our $disk_usage = 0; our $prev_report_length = 0; our $timediff = 0; @@ -959,6 +962,7 @@ sub mtr_error (@) { print STDERR _name() . _timestamp() . "mysql-test-run: *** ERROR: " . join(" ", @_) . "\n"; if (IS_WINDOWS) { + eval { $pre_exit_hook->() } if $pre_exit_hook; POSIX::_exit(1); } else { exit(1); diff --git a/mysql-test/lib/t/manifest.t b/mysql-test/lib/t/manifest.t new file mode 100644 index 000000000000..1f1d1b40470b --- /dev/null +++ b/mysql-test/lib/t/manifest.t @@ -0,0 +1,1056 @@ +#!/usr/bin/perl +# -*- cperl -*- + +# Copyright (c) 2026, Amazon.com, Inc. or its affiliates. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License, version 2.0, +# as published by the Free Software Foundation. +# +# This program is designed to work with certain software (including +# but not limited to OpenSSL) that is licensed under separate terms, +# as designated in a particular file or component or in included license +# documentation. The authors of MySQL hereby grant you an additional +# permission to link the program and your derivative works with the +# separately licensed software that they have either included with +# the program or referenced in the documentation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License, version 2.0, for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +use strict; +use warnings 'FATAL'; +use lib "lib"; + +use Fcntl (); +use File::Temp qw(tempdir); +use POSIX (); +use Test::More tests => 185; + +BEGIN { use_ok("My::Manifest"); } + +my $stub_content = '{ "read_local_manifest": true }'; + +# ---- helpers ---------------------------------------------------------- + +# Read the whole file and return its content. +sub slurp { + my ($path) = @_; + open my $fh, '<', $path or die "Cannot read $path: $!"; + local $/; + my $data = <$fh>; + close $fh; + return $data; +} + +# Write arbitrary content to a file. +sub spew { + my ($path, $data) = @_; + open my $fh, '>', $path or die "Cannot write $path: $!"; + print $fh $data; + close $fh; +} + +# ====================================================================== +# TEST 1: No pre-existing manifest -> stub created, removed on restore; +# marker exists during the run and is removed after restore +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + ok(!-e $mf, "T1: manifest does not exist before create"); + manifest_create($mf, $stub_content); + ok(-e $mf, "T1: stub created"); + is(slurp($mf), $stub_content, "T1: stub content matches"); + ok(!-e "$mf.mtr_saved", "T1: no backup created (nothing to back up)"); + ok(-e "$mf.mtr_stub", "T1: marker exists during the run"); + + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T1: stub removed on restore"); + ok(!-e "$mf.mtr_stub", "T1: marker removed on restore"); +} + +# ====================================================================== +# TEST 2: Pre-existing manifest -> moved aside, stub in place, original +# byte-identical after restore +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "original-manifest-content-42\n"; + manifest_reset(); + + spew($mf, $original); + manifest_create($mf, $stub_content); + + ok(-e "$mf.mtr_saved", "T2: backup created"); + is(slurp("$mf.mtr_saved"), $original, "T2: backup is original content"); + is(slurp($mf), $stub_content, "T2: stub overwrote manifest"); + ok(!-e "$mf.mtr_stub", "T2: no marker (user manifest, not stub)"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T2: manifest restored"); + is(slurp($mf), $original, "T2: restored content byte-identical"); + ok(!-e "$mf.mtr_saved", "T2: backup removed after restore"); +} + +# ====================================================================== +# TEST 3: Pre-existing manifest AND stale .mtr_saved from crashed prior +# run -> backup NOT clobbered, REAL manifest is what gets +# restored (path holds MTR's own stub from the crashed run) +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $real_orig = "real-original-from-first-run\n"; + manifest_reset(); + + # Simulate a crashed prior run: .mtr_saved holds the real original, + # the manifest file itself is MTR's stub from that crashed run. + spew("$mf.mtr_saved", $real_orig); + spew($mf, $stub_content); + + manifest_create($mf, $stub_content); + + # The existing backup must NOT have been overwritten. + is(slurp("$mf.mtr_saved"), $real_orig, + "T3: stale backup adopted, not clobbered"); + is(slurp($mf), $stub_content, "T3: new stub written"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T3: manifest restored"); + is(slurp($mf), $real_orig, + "T3: real original restored (not the stale stub)"); + ok(!-e "$mf.mtr_saved", "T3: backup removed after restore"); +} + +# ====================================================================== +# TEST 3b: Stale marker cleaned up when backup exists (path is stub) +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Simulate: backup exists AND a stale marker from the same crash; + # the file at is MTR's own stub. + spew("$mf.mtr_saved", "real-original\n"); + spew($mf, $stub_content); + spew("$mf.mtr_stub", ""); + + manifest_create($mf, $stub_content); + ok(!-e "$mf.mtr_stub", "T3b: stale marker removed when backup exists"); + + manifest_restore($mf, $stub_content); + is(slurp($mf), "real-original\n", "T3b: real original restored"); +} + +# ====================================================================== +# TEST 4: restore called twice -> second call is a no-op and does NOT +# delete the restored manifest (double-unlink defect) +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "precious-manifest-data\n"; + manifest_reset(); + + spew($mf, $original); + manifest_create($mf, $stub_content); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T4: manifest present after first restore"); + is(slurp($mf), $original, "T4: first restore correct"); + + # Second restore must be a true no-op. + manifest_restore($mf, $stub_content); + ok(-e $mf, "T4: manifest still present after second restore"); + is(slurp($mf), $original, + "T4: content unchanged after second restore (no double-unlink)"); +} + +# ====================================================================== +# TEST 4b: restore called twice for stub-only case (no pre-existing) -> +# second call is a no-op +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + manifest_create($mf, $stub_content); + ok(-e $mf, "T4b: stub created"); + ok(-e "$mf.mtr_stub", "T4b: marker created"); + + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T4b: stub removed after first restore"); + ok(!-e "$mf.mtr_stub", "T4b: marker removed after first restore"); + + # Write something new to simulate server dropping a manifest + spew($mf, "server-manifest\n"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T4b: second restore is no-op, file untouched"); + is(slurp($mf), "server-manifest\n", + "T4b: content unchanged after second restore"); +} + +# ====================================================================== +# TEST 5: move failure -> dies BEFORE the original is lost +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 4 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "must-not-lose-this\n"; + manifest_reset(); + + spew($mf, $original); + # A prior run leaves the lock file behind; with it present the lock + # needs no directory write, so the rename is what fails. + spew("$mf.mtr_lock", ""); + + # Make the directory read-only so rename (move) fails. + chmod 0555, $dir; + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + + # Restore write permission for cleanup. + chmod 0755, $dir; + + ok($err, "T5: manifest_create died on move failure"); + like($err // '', qr/Could not move manifest/, + "T5: died with correct message"); + is(slurp($mf), $original, + "T5: original not lost after move failure"); + ok(!-e "$mf.mtr_saved", + "T5: no partial backup left behind"); +} + +# ====================================================================== +# TEST 6: Leftover MTR stub from a crashed prior run (stub + marker +# present, no backup) -> recognised by marker, unlinked on +# restore, NOT backed up +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Simulate: stub and marker left behind, no backup. + spew($mf, $stub_content); + spew("$mf.mtr_stub", ""); + + manifest_create($mf, $stub_content); + ok(!-e "$mf.mtr_saved", + "T6: no backup created for leftover stub"); + is(slurp($mf), $stub_content, + "T6: stub content unchanged after create"); + + manifest_restore($mf, $stub_content); + ok(!-e $mf, + "T6: leftover stub unlinked on restore"); + ok(!-e "$mf.mtr_stub", + "T6: marker removed on restore"); +} + +# ====================================================================== +# TEST 6b: File with DIFFERENT content (no backup, no marker) -> +# still backed up (negative check for marker detection) +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "real-manifest-not-a-stub\n"; + manifest_reset(); + + spew($mf, $original); + + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_saved", + "T6b: non-stub file is backed up"); + is(slurp("$mf.mtr_saved"), $original, + "T6b: backup holds original content"); + is(slurp($mf), $stub_content, + "T6b: stub overwrote manifest"); + + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, + "T6b: original restored from backup"); + ok(!-e "$mf.mtr_saved", + "T6b: backup removed after restore"); +} + +# ====================================================================== +# TEST 6c: File whose content EQUALS the stub text but with NO marker +# -> backed up (.mtr_saved exists with that content) and +# restored byte-identical. This is the AutoSDE case: a user +# manifest that happens to look like the stub must not be lost. +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Write a user manifest whose content is identical to the stub text. + spew($mf, $stub_content); + # No marker present — this is a user's file, not an MTR leftover. + + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_saved", + "T6c: stub-text user manifest is backed up (no marker)"); + is(slurp("$mf.mtr_saved"), $stub_content, + "T6c: backup holds stub-text content"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, + "T6c: manifest restored"); + is(slurp($mf), $stub_content, + "T6c: restored content byte-identical to stub text"); + ok(!-e "$mf.mtr_saved", + "T6c: backup removed after restore"); + ok(!-e "$mf.mtr_stub", + "T6c: no marker left behind"); +} + +# ====================================================================== +# TEST 6d': Marker present + file with DIFFERENT content (no backup) +# -> after create: .mtr_leftover holds that content, +# no .mtr_saved, stub written, marker present; after restore: +# path and marker gone, .mtr_leftover still present with the +# content. The marker proves MTR owned the path so the +# content is quarantined, not restored. +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $test_cfg = '{ "components": "file://component_keyring_file" }'; + manifest_reset(); + + # Simulate: marker left from a prior killed run, path holds content + # written after MTR took over (e.g. by an interrupted test). + spew($mf, $test_cfg); + spew("$mf.mtr_stub", ""); + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_leftover", + "T6d': leftover file created"); + is(slurp("$mf.mtr_leftover"), $test_cfg, + "T6d': leftover holds quarantined content"); + ok(!-e "$mf.mtr_saved", + "T6d': no .mtr_saved (content is MTR-owned, not user)"); + is(slurp($mf), $stub_content, + "T6d': stub written"); + ok(-e "$mf.mtr_stub", + "T6d': marker present"); + ok(scalar @warnings > 0 && grep { /moved to/ } @warnings, + "T6d': quarantine warning emitted"); + + @warnings = (); + manifest_restore($mf, $stub_content); + ok(!-e $mf, + "T6d': path gone after restore"); + ok(!-e "$mf.mtr_stub", + "T6d': marker gone after restore"); + ok(-e "$mf.mtr_leftover", + "T6d': .mtr_leftover still present after restore"); + is(slurp("$mf.mtr_leftover"), $test_cfg, + "T6d': .mtr_leftover content unchanged after restore"); +} + +# ====================================================================== +# TEST 6e: Interrupted-keyring simulation — marker + path with custom +# content + .backup holding the stub (as the keyring +# helper leaves it: rename stub -> .backup, write custom, +# then interrupted). Same quarantine outcome, and the .backup +# file is untouched by us. +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $test_cfg = '{ "components": "file://component_keyring_file" }'; + manifest_reset(); + + # Simulate: the keyring test helper renamed the stub to .backup and + # wrote custom content to the path, then MTR was interrupted. + spew($mf, $test_cfg); + spew("$mf.mtr_stub", ""); + spew("$mf.backup", $stub_content); + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_leftover", + "T6e: leftover file created"); + is(slurp("$mf.mtr_leftover"), $test_cfg, + "T6e: leftover holds quarantined content"); + is(slurp($mf), $stub_content, + "T6e: stub written"); + ok(-e "$mf.mtr_stub", + "T6e: marker present"); + is(slurp("$mf.backup"), $stub_content, + "T6e: .backup file untouched by manifest_create"); + + @warnings = (); + manifest_restore($mf, $stub_content); + ok(!-e $mf, + "T6e: path gone after restore"); + ok(!-e "$mf.mtr_stub", + "T6e: marker gone after restore"); + ok(-e "$mf.mtr_leftover", + "T6e: .mtr_leftover still present after restore"); + is(slurp("$mf.mtr_leftover"), $test_cfg, + "T6e: .mtr_leftover content unchanged after restore"); + is(slurp("$mf.backup"), $stub_content, + "T6e: .backup file untouched by manifest_restore"); +} + +# ====================================================================== +# TEST 7b: Mode preservation — a read-only manifest (0444) keeps its +# mode across backup and restore; inode preserved by rename +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "mode-preserved-content\n"; + manifest_reset(); + + spew($mf, $original); + chmod 0444, $mf; + my $orig_ino = (stat($mf))[1]; + + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_saved", "T7b: backup created"); + ok(-e $mf, "T7b: stub written"); + is((stat("$mf.mtr_saved"))[2] & 07777, 0444, + "T7b: backup carries original mode (0444)"); + # Inode should follow the rename. + is((stat("$mf.mtr_saved"))[1], $orig_ino, + "T7b: backup inode equals original inode (rename, not copy)"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T7b: manifest restored"); + is((stat($mf))[2] & 07777, 0444, + "T7b: restored manifest mode is 0444"); + is((stat($mf))[1], $orig_ino, + "T7b: restored inode equals original inode"); + is(slurp($mf), $original, + "T7b: restored content byte-identical"); + ok(!-e "$mf.mtr_saved", + "T7b: backup removed after restore"); +} + +# ====================================================================== +# TEST 8: Filename with whitespace — the three-arg open form handles +# it correctly; the old two-arg form ("> $path") would +# silently misinterpret the leading space. +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + # Create a subdirectory whose name starts with a space. + my $spacedir = "$dir/ spaced"; + mkdir $spacedir or die "mkdir: $!"; + my $mf = "$spacedir/mysqld.my"; + manifest_reset(); + + manifest_create($mf, $stub_content); + ok(-e $mf, + "T8: manifest created at whitespace-containing path"); + is(slurp($mf), $stub_content, + "T8: stub content written correctly to whitespace path"); + + manifest_restore($mf, $stub_content); + ok(!-e $mf, + "T8: manifest removed on restore (stub-only path)"); +} + +# ====================================================================== +# TEST 8a: backup exists + == stub text → adopted, restore +# yields backup content (covers the content==stub guard) +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $real_orig = "original-from-crashed-run\n"; + manifest_reset(); + + # Path holds MTR's own stub (from a crashed run); backup holds the + # real manifest. No marker present. + spew("$mf.mtr_saved", $real_orig); + spew($mf, $stub_content); + + manifest_create($mf, $stub_content); + is(slurp("$mf.mtr_saved"), $real_orig, + "T8a: backup adopted (not clobbered)"); + is(slurp($mf), $stub_content, + "T8a: stub written over previous stub"); + + manifest_restore($mf, $stub_content); + is(slurp($mf), $real_orig, + "T8a: real original restored from backup"); + ok(!-e "$mf.mtr_saved", + "T8a: backup removed after restore"); +} + +# ====================================================================== +# TEST 8b: backup exists + with DIFFERENT content → die with +# message matching /different content/; both files untouched +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $real_orig = "original-manifest-from-old-run\n"; + my $new_user = '{ "components": "file://new_keyring" }'; + manifest_reset(); + + # Simulate: backup from a prior crash AND a newly installed manifest + # with different content. + spew("$mf.mtr_saved", $real_orig); + spew($mf, $new_user); + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + + ok($err, "T8b: manifest_create dies on conflicting backup"); + like($err // '', qr/different content/, + "T8b: error message mentions 'different content'"); + # Both files must be byte-identical to what they were before. + is(slurp($mf), $new_user, + "T8b: manifest at path untouched after die"); + is(slurp("$mf.mtr_saved"), $real_orig, + "T8b: backup untouched after die"); + ok(!-e "$mf.mtr_stub", + "T8b: no stub marker written"); +} + +# ====================================================================== +# TEST 8c: restore failure retry — unlink fails, state not cleared, +# second restore after restoring dir perms removes both +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 8 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Create a stub normally (no pre-existing file). + manifest_create($mf, $stub_content); + ok(-e $mf, "T8c: stub created"); + ok(-e "$mf.mtr_stub", "T8c: marker created"); + + # Make the directory read-only so unlink fails. + chmod 0555, $dir; + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + + # Stub and marker must still exist (unlink failed). + ok(-e $mf, "T8c: stub still present after failed unlink"); + ok(-e "$mf.mtr_stub", "T8c: marker still present after failed unlink"); + ok(scalar @warnings > 0, "T8c: warnings issued on unlink failure"); + + # Restore write permission. + chmod 0755, $dir; + + # Second restore (simulating END-block retry) must succeed now. + @warnings = (); + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T8c: stub removed on retry"); + ok(!-e "$mf.mtr_stub", "T8c: marker removed on retry"); + ok(scalar @warnings == 0, "T8c: no warnings on successful retry"); +} + +# ====================================================================== +# TEST 9: restore failure keeps state — rename fails, the stub and +# backup are left untouched, and the reference is NOT cleared +# so the END-block safety net can retry the restore +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 6 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "move-retry-test-content\n"; + manifest_reset(); + + spew($mf, $original); + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_saved", "T9: backup created"); + + # Make directory read-only so the same-directory rename fails. + chmod 0555, $dir; + + # First restore: rename fails, should warn but NOT clear the reference. + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + chmod 0755, $dir; + + ok(scalar @warnings > 0, "T9: warn issued on rename failure"); + + # rename has no copy fallback: neither file was touched. + is(slurp($mf), $stub_content, + "T9: manifest still holds the stub after failed rename"); + is(slurp("$mf.mtr_saved"), $original, + "T9: backup still holds the original after failed rename"); + + # Second restore (simulating END-block retry) must succeed now. + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, + "T9: retry after rename failure restores the original"); + ok(!-e "$mf.mtr_saved", + "T9: backup removed after successful retry"); +} + +# ====================================================================== +# TEST 9': restore failure and MTR exits before any retry — the next +# run adopts the untouched backup (stub at path) without a +# conflict die and restores the original +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 8 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "next-run-adoption-content\n"; + manifest_reset(); + + spew($mf, $original); + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_saved", "T9': backup created"); + + chmod 0555, $dir; + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + ok(scalar @warnings > 0, "T9': warn issued on rename failure"); + + # Simulate the next MTR run: fresh state, permissions back. + manifest_reset(); + chmod 0755, $dir; + is(slurp($mf), $stub_content, + "T9': stub left at path by the failed run"); + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err, "T9': next run's manifest_create does not die on conflict"); + is(slurp($mf), $stub_content, "T9': next run wrote the stub"); + + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, + "T9': next run restores the original"); + ok(!-e "$mf.mtr_saved", "T9': backup removed after restore"); + ok(!-e "$mf.mtr_stub", "T9': no marker left behind"); +} + +# ====================================================================== +# TEST 9a: Manifest replaced during the run (e.g. a keyring test that +# failed before its teardown) — quarantined to .mtr_leftover +# at cleanup, marker removed, nothing left at the path +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $new_cfg = '{ "components": "file://component_keyring_file" }'; + manifest_reset(); + + # Create the stub normally (no pre-existing file). + manifest_create($mf, $stub_content); + ok(-e $mf, "T9a: stub created"); + ok(-e "$mf.mtr_stub", "T9a: marker created"); + + # Simulate a mid-run install: overwrite the stub with different content. + spew($mf, $new_cfg); + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + + ok(scalar @warnings > 0 && grep { /changed during the run/ } @warnings, + "T9a: warning mentions 'changed during the run'"); + ok(grep({ /moved to \Q$mf.mtr_leftover\E/ } @warnings), + "T9a: warning names the .mtr_leftover destination"); + ok(!-e $mf, "T9a: manifest path gone after restore"); + is(slurp("$mf.mtr_leftover"), $new_cfg, + "T9a: .mtr_leftover holds the mid-run content"); + ok(!-e "$mf.mtr_stub", "T9a: marker removed"); + + # Second restore must be a no-op (state cleared). + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T9a: second restore is a no-op, path still absent"); + is(slurp("$mf.mtr_leftover"), $new_cfg, + "T9a: second restore leaves .mtr_leftover unchanged"); +} + +# ====================================================================== +# TEST 9a': Same as 9a, then another MTR run — the quarantined content +# is never adopted as a user manifest: no .mtr_saved, stub +# written, and the stub is unlinked again at cleanup +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $new_cfg = '{ "components": "file://component_keyring_file" }'; + manifest_reset(); + + manifest_create($mf, $stub_content); + ok(-e "$mf.mtr_stub", "T9a': marker created"); + spew($mf, $new_cfg); + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + is(slurp("$mf.mtr_leftover"), $new_cfg, + "T9a': changed stub quarantined at cleanup"); + + # Next MTR run. + manifest_reset(); + @warnings = (); + manifest_create($mf, $stub_content); + ok(!-e "$mf.mtr_saved", + "T9a': next run creates no .mtr_saved (nothing adopted)"); + is(slurp($mf), $stub_content, "T9a': next run wrote the stub"); + ok(-e "$mf.mtr_stub", "T9a': next run dropped a marker"); + + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T9a': next run's stub unlinked at cleanup"); + ok(!-e "$mf.mtr_stub", "T9a': next run's marker removed"); + is(slurp("$mf.mtr_leftover"), $new_cfg, + "T9a': .mtr_leftover untouched by the next run"); +} + +# ====================================================================== +# TEST 9b: Unreadable manifest in the conflict guard — accurate error +# message (not misreported as 'different content') +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 2 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Set up: backup exists and manifest exists but is unreadable. + spew("$mf.mtr_saved", "real-original\n"); + spew($mf, "some-content\n"); + chmod 0000, $mf; + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + + # Restore permissions for cleanup. + chmod 0644, $mf; + + like($err // '', qr/Could not read manifest/, + "T9b: error mentions 'Could not read manifest'"); + unlike($err // '', qr/different content/, + "T9b: error does NOT mention 'different content'"); +} + +# ====================================================================== +# TEST 9c: Unreadable stub at restore — reported and kept for retry; +# a second restore after restoring permissions removes it +# ====================================================================== +SKIP: { + skip "chmod-based failure injection is unreliable as root", 9 + if $> == 0; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + # Create a stub normally (no pre-existing file). + manifest_create($mf, $stub_content); + ok(-e $mf, "T9c: stub created"); + ok(-e "$mf.mtr_stub", "T9c: marker created"); + + # Make the stub unreadable. + chmod 0000, $mf; + + my @warnings; + local $SIG{__WARN__} = sub { push @warnings, $_[0] }; + manifest_restore($mf, $stub_content); + + ok(-e $mf, "T9c: stub still present (unreadable, kept)"); + ok(-e "$mf.mtr_stub", "T9c: marker still present (state kept for retry)"); + ok(scalar @warnings > 0 && grep { /Could not read stub/ } @warnings, + "T9c: warning mentions 'Could not read stub'"); + + # Restore permissions so the retry can read and remove the stub. + chmod 0644, $mf; + + @warnings = (); + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T9c: stub removed on retry after chmod"); + ok(!-e "$mf.mtr_stub", "T9c: marker removed on retry"); + ok(scalar @warnings == 0, "T9c: no warnings on successful retry"); + + # Third restore is a true no-op. + manifest_restore($mf, $stub_content); + ok(!-e $mf, "T9c: third restore is a no-op"); +} + +# ====================================================================== +# TEST 10: Stub write fails after the user manifest was moved aside — +# manifest_create dies, and manifest_restore (the END-block +# path) brings the original back byte-identical; a later run +# starts cleanly with no conflict +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "stub-write-failure-test-content\n"; + manifest_reset(); + + spew($mf, $original); + + my $err = do { + no warnings 'redefine'; + local *My::Manifest::_write_file = sub { die "injected\n" }; + eval { manifest_create($mf, $stub_content) }; + $@; + }; + is($err, "injected\n", "T10: manifest_create dies on stub-write failure"); + is(slurp("$mf.mtr_saved"), $original, + "T10: .mtr_saved holds the original after the failure"); + + # END-block safety net. + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, + "T10: restore brings the original back byte-identical"); + ok(!-e "$mf.mtr_saved", "T10: no .mtr_saved left after restore"); + ok(!-e "$mf.mtr_stub", "T10: no .mtr_stub left after restore"); + + # A fresh run starts cleanly. + manifest_reset(); + my $err2 = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err2, "T10: fresh manifest_create succeeds with no conflict"); + is(slurp($mf), $stub_content, "T10: fresh run wrote the stub"); + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T10: fresh run restores the original"); + ok(!-e "$mf.mtr_saved", "T10: fresh run leaves no .mtr_saved"); +} + +# True when no other open file description holds the ownership lock. +sub lock_is_free { + my ($mf) = @_; + open my $fh, '>>', "$mf.mtr_lock" or die "Cannot open $mf.mtr_lock: $!"; + my $free = flock($fh, Fcntl::LOCK_EX() | Fcntl::LOCK_NB()); + close $fh; + return $free; +} + +# ====================================================================== +# TEST 11: Two concurrent runs sharing a manifest path — the second +# dies before touching anything; the first restores the +# original byte-identical +# ====================================================================== +SKIP: { + skip "fork-based concurrency test is not run on Windows", 7 + if $^O eq 'MSWin32'; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "concurrent-run-test-content\n"; + manifest_reset(); + + spew($mf, $original); + + pipe(my $ready_r, my $ready_w) or die "pipe: $!"; + pipe(my $go_r, my $go_w) or die "pipe: $!"; + my $pid = fork(); + die "fork: $!" unless defined $pid; + if ($pid == 0) { + close $ready_r; + close $go_w; + my $rc = eval { manifest_create($mf, $stub_content); 1 } ? 0 : 1; + syswrite($ready_w, $rc ? "fail\n" : "ok\n"); + <$go_r>; # block until the parent has tried to take over + manifest_restore($mf, $stub_content); + POSIX::_exit($rc); + } + close $ready_w; + close $go_r; + my $line = <$ready_r>; + is($line, "ok\n", "T11: first run created its stub"); + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + like($err // '', qr/Another MTR run \(pid \d+\)/, + "T11: second run dies naming the owner"); + is(slurp("$mf.mtr_saved"), $original, + "T11: .mtr_saved still holds the original"); + is(slurp($mf), $stub_content, "T11: path still holds the first run's stub"); + + close $go_w; # let the first run finish + waitpid($pid, 0); + is($?, 0, "T11: first run exited cleanly"); + is(slurp($mf), $original, "T11: first run restored the original"); + ok(!-e "$mf.mtr_saved", "T11: no .mtr_saved left"); + manifest_reset(); +} + +# ====================================================================== +# TEST 11b: Stale lock — the owning run died without restoring; the +# kernel released its lock, so the next run adopts the backup +# ====================================================================== +SKIP: { + skip "fork-based concurrency test is not run on Windows", 5 + if $^O eq 'MSWin32'; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "stale-lock-test-content\n"; + manifest_reset(); + + spew($mf, $original); + + my $pid = fork(); + die "fork: $!" unless defined $pid; + if ($pid == 0) { + my $rc = eval { manifest_create($mf, $stub_content); 1 } ? 0 : 1; + POSIX::_exit($rc); # die without restoring + } + waitpid($pid, 0); + is($?, 0, "T11b: crashed run had created its stub"); + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err, "T11b: next run takes over the stale lock"); + is(slurp($mf), $stub_content, "T11b: next run wrote the stub"); + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T11b: adopted backup restored"); + ok(!-e "$mf.mtr_saved", "T11b: no .mtr_saved left"); +} + +# ====================================================================== +# TEST 12a: --start-and-exit setup fails before the hand-off (stub +# write dies) — the restore still runs and releases the lock +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "failed-start-and-exit-content\n"; + manifest_reset(); + + spew($mf, $original); + + my $err = do { + no warnings 'redefine'; + local *My::Manifest::_write_file = sub { die "injected\n" }; + eval { manifest_create($mf, $stub_content) }; + $@; + }; + is($err, "injected\n", "T12a: manifest_create dies on stub-write failure"); + + # No manifest_hand_off(): nothing was detached. + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T12a: original restored"); + ok(!-e "$mf.mtr_saved", "T12a: no .mtr_saved left"); + ok(lock_is_free($mf), "T12a: ownership lock released"); +} + +# ====================================================================== +# TEST 12b: Successful --start-and-exit hand-off — restore leaves the +# stub and backup for the detached servers; the next run +# adopts the backup and restores the original +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "handed-off-content\n"; + manifest_reset(); + + spew($mf, $original); + + manifest_create($mf, $stub_content); + manifest_hand_off(); + manifest_restore($mf, $stub_content); + is(slurp($mf), $stub_content, "T12b: stub left for the detached servers"); + is(slurp("$mf.mtr_saved"), $original, "T12b: .mtr_saved untouched"); + ok(!lock_is_free($mf), "T12b: lock held until the process exits"); + + # The next run. + manifest_reset(); + ok(lock_is_free($mf), "T12b: lock free once the run is gone"); + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err, "T12b: next run adopts the backup"); + is(slurp($mf), $stub_content, "T12b: next run wrote the stub"); + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T12b: next run restores the original"); + ok(!-e "$mf.mtr_saved", "T12b: no .mtr_saved left"); + ok(!-e "$mf.mtr_stub", "T12b: no .mtr_stub left"); +} + +# ====================================================================== +# TEST 13: The lock file is created mode 0666 regardless of the umask +# (as mtr_unique.pm does), so a later run by another user with +# write access to the directory can open it +# ====================================================================== +SKIP: { + skip "POSIX file modes are not checked on Windows", 3 + if $^O eq 'MSWin32'; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + + my $saved_umask = umask(022); + manifest_create($mf, $stub_content); + is(umask(), 022, "T13: caller's umask restored after manifest_create"); + umask($saved_umask); + is((stat("$mf.mtr_lock"))[2] & 0777, 0666, "T13: lock file is mode 0666"); + manifest_restore($mf, $stub_content); + is((stat("$mf.mtr_lock"))[2] & 0777, 0666, + "T13: lock file kept, still 0666, after restore"); + manifest_reset(); +} + +# ====================================================================== +# TEST 13b: A lock file left by an earlier run is reused. Opening it +# needs only read-write permission on the file, so a 0666 +# lock owned by another user is equally openable; here the +# earlier run is our own (a test cannot switch users) +# ====================================================================== +SKIP: { + skip "POSIX file modes are not checked on Windows", 6 + if $^O eq 'MSWin32'; + + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = "reused-lock-content\n"; + manifest_reset(); + + spew($mf, $original); + spew("$mf.mtr_lock", "99999\n"); + chmod 0666, "$mf.mtr_lock"; + + my $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err, "T13b: manifest_create reuses an existing 0666 lock"); + is(slurp("$mf.mtr_lock"), "$$\n", "T13b: lock now names this run"); + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T13b: original restored"); + manifest_reset(); + + # A lock left 0644 by an older MTR is widened by its owner on reuse. + chmod 0644, "$mf.mtr_lock"; + $err = do { eval { manifest_create($mf, $stub_content) }; $@ }; + ok(!$err, "T13b: manifest_create reuses an existing 0644 lock"); + is((stat("$mf.mtr_lock"))[2] & 0777, 0666, "T13b: owner widened it to 0666"); + manifest_restore($mf, $stub_content); + is(slurp($mf), $original, "T13b: original restored again"); + manifest_reset(); +} diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index 6da6ef49e8d3..ac73742c56c5 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -59,6 +59,7 @@ use My::CoreDump; use My::File::Path; # Patched version of File::Path use My::Find; +use My::Manifest; use My::Options; use My::Platform; use My::RouterConfigFactory; @@ -99,6 +100,9 @@ # Local variables my $parent_pid; +my %children; # Worker pids, so the END hook can stop them on an abort +my $start_exit_servers_started = 0; # A worker sent START_EXIT_OK +my $worker_server_sock; # Worker's socket to run_test_server my $opt_boot_dbx; my $opt_boot_ddd; my $opt_boot_gdb; @@ -199,6 +203,8 @@ my $shutdown_report = 0; my $valgrind_reports = 0; +my $manifest_stub_content = "{ \"read_local_manifest\": true }"; + my @valgrind_args; # Storage for changed environment variables @@ -448,6 +454,19 @@ END if ($parent_pid && $current_id == $parent_pid) { remove_redundant_thread_id_file_locations(); clean_unique_id_dir(); + # Safety net: restore the pre-existing manifest on die/SIGINT. + # Idempotent — no-op if the normal exit path already restored it. + # The restore is skipped only after a successful --start-and-exit + # hand-off (manifest_hand_off): that is the only case in which MTR + # exits while leaving servers running, and they need the manifest + # (original or MTR-written stub) in place. A --start-and-exit run + # that fails before the hand-off is restored like any other run. + # --start and --start-and-test keep MTR alive until the servers + # exit, so the restore runs after them. + # Workers may still be running on an abort; stop them before + # restoring the manifest so a test cannot overwrite it. + stop_workers(); + remove_manifest_file(); } if (defined $opt_tmpdir_pid and $opt_tmpdir_pid == $$) { if (!$opt_start_exit) { @@ -851,12 +870,18 @@ sub main { mtr_report("ports_per_thread:".$ports_per_thread); + # Arm the parent-only END cleanup before touching the manifest. + $parent_pid = $$; + # mtr_error uses POSIX::_exit on Windows, which skips END; restore the + # manifest from the parent before exiting. + $mtr_report::pre_exit_hook = sub { + return unless $$ == $parent_pid; + stop_workers(); + remove_manifest_file(); + }; create_manifest_file(); # Create child processes - my %children; - - $parent_pid = $$; for my $child_num (1 .. $opt_parallel) { my $child_pid = My::SafeProcess::Base::_safe_fork(); if ($child_pid == 0) { @@ -884,23 +909,17 @@ sub main { my $completed = run_test_server($server, $tests, $opt_parallel); - exit(0) if $opt_start_exit; - - # Send Ctrl-C to any children still running - kill("INT", keys(%children)); - - if (!IS_WINDOWS) { - # Wait for children to exit - foreach my $pid (keys %children) { - my $ret_pid = waitpid($pid, 0); - if ($ret_pid != $pid) { - mtr_report("Unknown process $ret_pid exited"); - } else { - delete $children{$ret_pid}; - } - } + # Hand off only if a worker reported START_EXIT_OK (servers running) and + # exited 0; a failed or all-skipped run restores the manifest. + if ($opt_start_exit && $start_exit_servers_started && + start_exit_workers_ok()) { + # Servers are running detached; leave them the manifest. + manifest_hand_off(); + exit(0); } + stop_workers(); + # Remove config files for components read_plugin_defs("include/plugin.defs", 1); for my $plugin_def (@plugin_defs) { @@ -1027,7 +1046,7 @@ sub main { # connections, the bulk of the loop is handling the different messages. # # The message starts with a codeword, which can be 'TESTRESULT', -# 'START', 'SPENT' or 'VALGREP'. +# 'START', 'SPENT', 'VALGREP' or 'START_EXIT_OK'. # # After 'TESTRESULT' or 'START', the master thread finds the next test # to run by this worker. It also contains the logic to find a more @@ -1256,6 +1275,11 @@ ($$$) } elsif ($line eq 'SRV_CRASH') { # Mysqld detected crash during shutdown $shutdown_report = 1; + } elsif ($line eq 'START_EXIT_OK') { + # --start-and-exit: the worker started its servers and is exiting; + # schedule nothing more for it. + $start_exit_servers_started = 1; + next; } else { # Unknown message from worker mtr_error("Unknown response: '$line' from client"); @@ -1413,6 +1437,7 @@ ($) Proto => 'tcp'); mtr_error("Could not connect to server at port $server_port: $!") unless $server; + $worker_server_sock = $server; # Set worker name report_option('name', "worker[$thread_num]"); @@ -2508,23 +2533,54 @@ () } } -# Create global manifest file +# Create global manifest file. Delegates to My::Manifest. sub create_manifest_file { use strict; use File::Basename; - my $config_content = "{ \"read_local_manifest\": true }"; my $manifest_file_ext = ".my"; my $exe_mysqld = find_mysqld($basedir); my ($exename, $path, $suffix) = fileparse($exe_mysqld, qr/\.[^.]*/); my $manifest_file_path = $path.$exename.$manifest_file_ext; - open(my $mh, "> $manifest_file_path") or - die "Could not create manifest file $manifest_file_path"; - print $mh $config_content or - die "Could not write manifest file $manifest_file_path"; - close($mh); + + manifest_create($manifest_file_path, $manifest_stub_content); +} + +# Reap the --start-and-exit workers, which exit on their own once the +# servers are started; true only if every worker exited with status 0. +sub start_exit_workers_ok { + return 1 if IS_WINDOWS; + local $?; + for my $pid (keys %children) { + my $r = waitpid($pid, 0); + return 0 if $r != $pid || $? != 0; + delete $children{$pid}; + } + return 1; +} + +# Send Ctrl-C to any workers still running and reap them. Every pid is +# forgotten afterwards, so a second call (from the END hook) is a no-op. +sub stop_workers { + return unless %children; + local $?; # waitpid sets $?; keep MTR's exit status when called from END + kill("INT", keys(%children)); + + if (!IS_WINDOWS) { + # Wait for children to exit + foreach my $pid (keys %children) { + my $ret_pid = waitpid($pid, 0); + if ($ret_pid != $pid) { + mtr_report("Unknown process $ret_pid exited"); + } + delete $children{$pid}; + } + } else { + %children = (); + } } -# Delete global manifest file +# Delete global manifest file, restoring a pre-existing backup if one +# was preserved by create_manifest_file(). Delegates to My::Manifest. sub remove_manifest_file { use strict; use File::Basename; @@ -2532,7 +2588,8 @@ sub remove_manifest_file { my $exe_mysqld = find_mysqld($basedir); my ($exename, $path, $suffix) = fileparse($exe_mysqld, qr/\.[^.]*/); my $manifest_file_path = $path.$exename.$manifest_file_ext; - unlink $manifest_file_path; + + manifest_restore($manifest_file_path, $manifest_stub_content); } # Create config for one component @@ -5308,6 +5365,7 @@ ($) if ($start_only) { if ($opt_start_exit) { mtr_print("Server(s) started, not waiting for them to finish"); + print $worker_server_sock "START_EXIT_OK\n" if $worker_server_sock; if (IS_WINDOWS) { POSIX::_exit(0); # exit hangs here in ActiveState Perl } else { diff --git a/mysql-test/r/mtr_unit_tests.result b/mysql-test/r/mtr_unit_tests.result index 66c109ded50c..6a6e9ac41684 100644 --- a/mysql-test/r/mtr_unit_tests.result +++ b/mysql-test/r/mtr_unit_tests.result @@ -250,3 +250,189 @@ ok 10 - 'character-sets-dir' generated ok 11 - 'MASTER_MY_PORT' generated ok 12 - group like 'mysqld' exists 1..12 +1..185 +ok 1 +ok 2 +ok 3 +ok 4 +ok 5 +ok 6 +ok 7 +ok 8 +ok 9 +ok 10 +ok 11 +ok 12 +ok 13 +ok 14 +ok 15 +ok 16 +ok 17 +ok 18 +ok 19 +ok 20 +ok 21 +ok 22 +ok 23 +ok 24 +ok 25 +ok 26 +ok 27 +ok 28 +ok 29 +ok 30 +ok 31 +ok 32 +ok 33 +ok 34 +ok 35 +ok 36 +ok 37 +ok 38 +ok 39 +ok 40 +ok 41 +ok 42 +ok 43 +ok 44 +ok 45 +ok 46 +ok 47 +ok 48 +ok 49 +ok 50 +ok 51 +ok 52 +ok 53 +ok 54 +ok 55 +ok 56 +ok 57 +ok 58 +ok 59 +ok 60 +ok 61 +ok 62 +ok 63 +ok 64 +ok 65 +ok 66 +ok 67 +ok 68 +ok 69 +ok 70 +ok 71 +ok 72 +ok 73 +ok 74 +ok 75 +ok 76 +ok 77 +ok 78 +ok 79 +ok 80 +ok 81 +ok 82 +ok 83 +ok 84 +ok 85 +ok 86 +ok 87 +ok 88 +ok 89 +ok 90 +ok 91 +ok 92 +ok 93 +ok 94 +ok 95 +ok 96 +ok 97 +ok 98 +ok 99 +ok 100 +ok 101 +ok 102 +ok 103 +ok 104 +ok 105 +ok 106 +ok 107 +ok 108 +ok 109 +ok 110 +ok 111 +ok 112 +ok 113 +ok 114 +ok 115 +ok 116 +ok 117 +ok 118 +ok 119 +ok 120 +ok 121 +ok 122 +ok 123 +ok 124 +ok 125 +ok 126 +ok 127 +ok 128 +ok 129 +ok 130 +ok 131 +ok 132 +ok 133 +ok 134 +ok 135 +ok 136 +ok 137 +ok 138 +ok 139 +ok 140 +ok 141 +ok 142 +ok 143 +ok 144 +ok 145 +ok 146 +ok 147 +ok 148 +ok 149 +ok 150 +ok 151 +ok 152 +ok 153 +ok 154 +ok 155 +ok 156 +ok 157 +ok 158 +ok 159 +ok 160 +ok 161 +ok 162 +ok 163 +ok 164 +ok 165 +ok 166 +ok 167 +ok 168 +ok 169 +ok 170 +ok 171 +ok 172 +ok 173 +ok 174 +ok 175 +ok 176 +ok 177 +ok 178 +ok 179 +ok 180 +ok 181 +ok 182 +ok 183 +ok 184 +ok 185 diff --git a/mysql-test/t/mtr_unit_tests.test b/mysql-test/t/mtr_unit_tests.test index dd809c068311..58a1ef03f4e7 100644 --- a/mysql-test/t/mtr_unit_tests.test +++ b/mysql-test/t/mtr_unit_tests.test @@ -10,3 +10,5 @@ --exec perl $MYSQL_TEST_DIR/lib/t/SafeProcess.t 2>&1 --exec perl $MYSQL_TEST_DIR/lib/t/testMyConfig.t 2>&1 --exec perl $MYSQL_TEST_DIR/lib/t/testMyConfigFactory.t 2>&1 +--replace_regex /ok ([0-9]+)( - [^\n]*| # skip [^\n]*)/ok \1/ +--exec perl $MYSQL_TEST_DIR/lib/t/manifest.t 2>&1 From f194c20fb73ad572f6b0c66a48a0d65746c8fa09 Mon Sep 17 00:00:00 2001 From: Surendar Chandra Date: Fri, 4 Sep 2026 23:10:35 +0000 Subject: [PATCH 2/2] Components: support merge_local_manifest in the global manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit read_local_manifest discards the global manifest reader before the "components" key is parsed, so code that needs both the global and local component lists has no way to obtain them from a single reader — they are mutually exclusive. A new boolean key, merge_local_manifest, makes the global reader append the local manifest's components after the global ones. Existing single-key manifests are unaffected: the key defaults to false and the reader's behaviour is unchanged when it is absent. The component separator is single-sourced as a constexpr consumed by both the splitter (get_next_component) and an inline merge helper (merge_component_lists), so there is exactly one representation of the delimiter in the codebase. Merged lists are de-duplicated and a local manifest resolving to the global file is not merged into itself. A local manifest that cannot be opened is treated as absent, as Manifest_reader does elsewhere; one that opens but has no readable "components" is a startup error. The merge step (merge_local_components) always passes the global list through the de-duplicating merge, so a repeated global URN is loaded once even when no local manifest exists or it is zero bytes. The MTR consumer (My::Manifest) preserves the deployment's components for the duration of the run, ensuring restarts see the same list that was present at boot. It decodes the saved manifest with JSON::PP and takes only the top-level "components" string, as the server's reader does. A saved manifest whose read_local_manifest is true keeps the plain redirect stub, because the server applies that key before reading "components". A saved manifest that repeats "components" or read_local_manifest at the top level is refused: JSON::PP keeps the last occurrence while the server's RapidJSON lookup selects the first. GUnit coverage (manifest-t.cc): * composition of global + local component lists * empty inputs (global-only, local-only, both empty) * no trailing separator in merged output * duplicate URN across global and local is kept once (global position) * local entirely contained in global yields global unchanged * duplicates within one side are also collapsed * empty tokens (',,') do not produce empty URNs * new-key default-false (merge_local_manifest absent) * read_local_manifest behaviour unchanged * merge step with an absent, zero-byte, populated or self-referencing local manifest de-duplicates the global list; a local manifest without "components" fails * POSIX fixture (tempfile cleanup) guarded for non-Windows Tests (manifest.t): * crash recovery when the merge stub is on disk * nested "components" ignored; escaped key recognised; value re-encoded; non-string value and malformed JSON rejected * repeated top-level "components" / read_local_manifest rejected; nested or unrelated repeated keys accepted --- include/manifest.h | 112 +++++++++- mysql-test/lib/My/Manifest.pm | 138 ++++++++++++- mysql-test/lib/t/manifest.t | 243 +++++++++++++++++++++- mysql-test/r/mtr_unit_tests.result | 49 ++++- sql/sql_component.cc | 35 +++- unittest/gunit/CMakeLists.txt | 1 + unittest/gunit/manifest-t.cc | 315 +++++++++++++++++++++++++++++ 7 files changed, 884 insertions(+), 9 deletions(-) create mode 100644 unittest/gunit/manifest-t.cc diff --git a/include/manifest.h b/include/manifest.h index ffca84b1c7b2..58e09b23f9a9 100644 --- a/include/manifest.h +++ b/include/manifest.h @@ -24,9 +24,11 @@ #ifndef MANIFEST_INCLUDED #define MANIFEST_INCLUDED +#include #include /* std::ifstream */ #include #include +#include #include "scope_guard.h" @@ -37,7 +39,14 @@ namespace manifest { -std::string manifest_version_1_0 = +/** + Separator between component URNs in a manifest "components" string. + Used when joining and splitting — a single source of truth so + the merge side and the consumer always agree. +*/ +inline constexpr const char *kComponentSeparator = ","; + +inline std::string manifest_version_1_0 = "{" " \"title\": \"Manifest validator version 1.0\"," " \"description\": \"Expected schema for version 1.0\"," @@ -48,6 +57,11 @@ std::string manifest_version_1_0 = "data directory\"," " \"type\": \"boolean\"" " }," + " \"merge_local_manifest\": {" + " \"description\": \"Flag to indicate that components from the local " + "(instance-path) manifest should be merged with the global list\"," + " \"type\": \"boolean\"" + " }," " \"components\": {" " \"description\": \"The list of components to be loaded at " "bootstrap\"," @@ -140,6 +154,14 @@ class Manifest_reader final { return read_local_manifest; } + bool merge_local_manifest() const { + bool merge_local_manifest = false; + if (get_element("merge_local_manifest", merge_local_manifest) == + false) + return false; + return merge_local_manifest; + } + bool components(std::string &components_string) const { return get_element("components", components_string); } @@ -182,6 +204,94 @@ class Manifest_reader final { bool ro_; }; +/** + Merge a global and a local component list using kComponentSeparator. + Either side may be empty; no trailing separator is produced. + A URN present in both lists is kept once (global position); loading + the same component twice would fail. + + Empty tokens produced by consecutive separators (',,') are tolerated + and skipped, exactly as Deployed_components::get_next_component does. + + @param global Component URN list from the global manifest + @param local Component URN list from the instance-path manifest + + @returns The de-duplicated merged string, or whichever side is + non-empty, or empty if both are empty. +*/ +inline std::string merge_component_lists(const std::string &global, + const std::string &local) { + /* Split a separator-delimited string, skipping empty tokens. */ + auto split = [](const std::string &s) -> std::vector { + std::vector tokens; + const std::string sep(kComponentSeparator); + std::string::size_type start = 0; + std::string::size_type pos; + while ((pos = s.find(sep, start)) != std::string::npos) { + if (pos != start) tokens.push_back(s.substr(start, pos - start)); + start = pos + sep.size(); + } + if (start < s.size()) tokens.push_back(s.substr(start)); + return tokens; + }; + + std::vector result; + + /* Add global tokens, skipping duplicates within global itself. */ + for (const auto &tok : split(global)) { + if (std::find(result.begin(), result.end(), tok) == result.end()) + result.push_back(tok); + } + + /* Append local tokens that are not already present (exact match). */ + for (const auto &tok : split(local)) { + if (std::find(result.begin(), result.end(), tok) == result.end()) + result.push_back(tok); + } + + /* Join with the separator — no trailing separator. */ + std::string merged; + for (size_t i = 0; i < result.size(); ++i) { + if (i > 0) merged += kComponentSeparator; + merged += result[i]; + } + return merged; +} + +/** + Apply merge_local_manifest: fold the instance-path manifest's + components into the list read from the global manifest. + + The result always passes through merge_component_lists, so the global + list is de-duplicated even when there is nothing to merge: no local + manifest, a zero-byte one, or an instance path that resolves to the + global manifest itself (which is not merged into itself). + + A local manifest that cannot be opened is treated as absent, as + Manifest_reader does elsewhere; a local manifest that opens but has + no readable "components" is an error. + + @param global_reader Reader for the global manifest + @param local_reader Reader for the instance-path manifest + @param [in,out] components Global component list; the merged list + on success + + @retval true components holds the merged list + @retval false local manifest opened but its "components" unreadable; + components is unchanged +*/ +inline bool merge_local_components(const Manifest_reader &global_reader, + const Manifest_reader &local_reader, + std::string &components) { + std::string local_components; + if (local_reader.manifest_file() != global_reader.manifest_file() && + !local_reader.empty() && + local_reader.components(local_components) == false) + return false; + components = merge_component_lists(components, local_components); + return true; +} + } // namespace manifest #endif // !MANIFEST_INCLUDED diff --git a/mysql-test/lib/My/Manifest.pm b/mysql-test/lib/My/Manifest.pm index 81eea9b89ba5..bc02a094b7d7 100644 --- a/mysql-test/lib/My/Manifest.pm +++ b/mysql-test/lib/My/Manifest.pm @@ -61,6 +61,8 @@ use strict; use warnings; use Carp; use Fcntl qw(:flock O_RDWR O_CREAT); +use B (); +use JSON::PP (); use base qw(Exporter); our @EXPORT = qw(manifest_create manifest_restore manifest_reset @@ -150,6 +152,105 @@ sub _write_file { close($fh) or die "Could not write manifest file $path: $!"; } +# _top_level_keys($text) — the member names of the top-level object in +# $text, in document order with repeats kept, each still JSON-encoded +# (quotes and escapes included). $text must already have decoded as a +# JSON object; strings are skipped whole, so brackets and colons inside +# them do not count, and only names at depth 1 are collected. +sub _top_level_keys { + my ($text) = @_; + my @keys; + my $depth = 0; + while ($text =~ /("(?:[^"\\]++|\\.)*+")|([{\[])|([}\]])/gs) { + if (defined $1) { + my $string = $1; + push @keys, $string if $depth == 1 && $text =~ /\G\s*:/gc; + } elsif (defined $2) { + $depth++; + } else { + $depth--; + } + } + return @keys; +} + +# _merge_stub_for($backup_path, $plain_stub) — derive the stub that +# manifest_create writes for the preserved manifest at $backup_path: +# the merge stub when the backup's top-level object has a "components" +# member, otherwise $plain_stub unchanged. +# +# A backup whose top-level "read_local_manifest" is true always gets +# $plain_stub: the server applies read_local_manifest before reading +# "components", so that deployment's global components were never +# loaded and must not be merged (nor validated) here. +# +# The backup is decoded with JSON::PP (core Perl) so that only the +# top-level member the server's Manifest_reader uses is selected: a +# "components" key nested inside another object is ignored, and an +# escaped key such as "\u0063omponents" is recognised. The value is +# re-encoded when the merge stub is written. +# +# Dies when a top-level "components" member is not a string (the +# server's manifest schema requires a string), and when the backup is +# not valid JSON but mentions "components" — a component list is never +# silently dropped. A backup that is not valid JSON and does not +# mention "components" has nothing the server could load, so the plain +# stub is used and the backup is preserved byte-for-byte as before. +# +# Also dies when the top-level object repeats "components" or +# "read_local_manifest": JSON::PP keeps the last occurrence while the +# server's RapidJSON lookup selects the first, so either choice here +# could differ from what the server loads. +sub _merge_stub_for { + my ($backup_path, $plain_stub) = @_; + my $saved_content = do { + open(my $fh, '<', $backup_path) + or die "Could not read preserved manifest $backup_path: $!"; + binmode $fh; + local $/; + <$fh>; + }; + my $unparseable = + "Preserved manifest $backup_path contains a " + . "\"components\" key whose value could not be parsed; " + . "refusing to silently drop it.\n"; + + my $json = JSON::PP->new->utf8; + my $doc = eval { $json->decode($saved_content) }; + if (!defined $doc || ref($doc) ne 'HASH') { + die $unparseable if $saved_content =~ /"components"/; + return $plain_stub; # not a JSON object; nothing to merge + } + # Refuse repeated keys the stub depends on (compared once decoded, + # so an escaped spelling of a key counts as the same key). + my $key_json = JSON::PP->new->utf8->allow_nonref; + my %seen; + for my $key (map { $key_json->decode($_) } _top_level_keys($saved_content)) { + next unless $key eq 'components' || $key eq 'read_local_manifest'; + die "Manifest $backup_path has a duplicate key \"$key\"; " + . "refusing to derive a stub.\n" + if $seen{$key}++; + } + # The server honours read_local_manifest first and ignores global + # "components"; keep the redirect stub in that case. + return $plain_stub + if exists $doc->{read_local_manifest} + && JSON::PP::is_bool($doc->{read_local_manifest}) + && $doc->{read_local_manifest}; + return $plain_stub unless exists $doc->{components}; + + my $value = $doc->{components}; + # A JSON string decodes to a plain scalar with a string value + # (POK); numbers, booleans, null, arrays and objects are rejected. + die $unparseable + unless defined $value && !ref($value) + && (B::svref_2object(\$value)->FLAGS & B::SVf_POK()); + + my $components_value = $json->allow_nonref->encode($value); + return + "{ \"components\": $components_value, \"merge_local_manifest\": true }"; +} + # ---- public API ------------------------------------------------------- # manifest_create($manifest_file_path, $config_content) @@ -176,13 +277,23 @@ sub manifest_create { # holds the real manifest. Adopt it only when the file at # is absent or is MTR's own stub (left by the crashed run); never # silently prefer either candidate when both hold real content. + # After a crash the file may hold the plain stub OR the merge stub + # that manifest_create derived from the backup's components, so + # both forms count as MTR-owned. if (-e $manifest_file_path && !_content_matches($manifest_file_path, $config_content)) { - die "Both $manifest_file_path and its backup $backup exist with " - . "different content (a previous run was interrupted and a new " - . "manifest was installed since). Refusing to guess which one " - . "is current; reconcile them and remove $backup before " - . "re-running.\n"; + # Not the plain stub — check the merge stub derived from the backup. + my $merge = eval { _merge_stub_for($backup, $config_content) }; + # A backup that cannot be read or parsed is its own error, not a + # content conflict; report it as such. + die $@ if $@; + if (!_content_matches($manifest_file_path, $merge)) { + die "Both $manifest_file_path and its backup $backup exist with " + . "different content (a previous run was interrupted and a new " + . "manifest was installed since). Refusing to guess which one " + . "is current; reconcile them and remove $backup before " + . "re-running.\n"; + } } $preserved_manifest = $backup; # Remove a stale marker if present. @@ -192,6 +303,9 @@ sub manifest_create { # started. Whatever content is here now was written after MTR took # over (a leftover stub, an interrupted test, or someone who # overwrote the leftover). + # Note: a merge stub can never appear here — merge stubs are only + # written when a backup exists, and in this branch there is no + # backup. Only the plain stub can be a leftover stub. if (_content_matches($manifest_file_path, $config_content)) { # Leftover MTR stub — just re-use the path; unlink on restore. $manifest_stub_created = 1; @@ -222,6 +336,15 @@ sub manifest_create { $manifest_stub_created = 1; } + # Build a merge manifest that keeps the pre-existing components + # loaded alongside MTR's local-manifest components. + # + # Skip merge when the file is a leftover MTR stub (no backup): + # $manifest_stub_created is set, $preserved_manifest is undef. + if (defined $preserved_manifest) { + $config_content = _merge_stub_for($preserved_manifest, $config_content); + } + # Drop the marker BEFORE writing the stub so that a crash between # the two leaves marker-without-stub (harmless: next run sees no # manifest and overwrites the marker). @@ -241,6 +364,8 @@ sub manifest_create { # always runs to completion. Three states: # 1. $preserved_manifest set & file exists -> rename backup over manifest; # clear state only on success so the END-block can retry. +# (The rename overwrites the path regardless, so neither the plain +# stub nor the merge stub requires special handling here.) # 2. $manifest_stub_created true -> unlink the MTR-written stub # only if it still holds MTR's # own text; a manifest changed @@ -252,6 +377,9 @@ sub manifest_create { # when the stub is handled. # Clear state only when # everything succeeded. +# (A merge stub can never exist +# without a backup, so this +# branch sees only plain stubs.) # 3. Neither -> true no-op (second call). sub manifest_restore { my ($manifest_file_path, $config_content) = @_; diff --git a/mysql-test/lib/t/manifest.t b/mysql-test/lib/t/manifest.t index 1f1d1b40470b..e67c937a2dcc 100644 --- a/mysql-test/lib/t/manifest.t +++ b/mysql-test/lib/t/manifest.t @@ -31,7 +31,7 @@ use lib "lib"; use Fcntl (); use File::Temp qw(tempdir); use POSIX (); -use Test::More tests => 185; +use Test::More tests => 232; BEGIN { use_ok("My::Manifest"); } @@ -417,6 +417,213 @@ SKIP: { "T6e: .backup file untouched by manifest_restore"); } +# ====================================================================== +# TEST 6f: Pre-existing manifest with NO "components" mention -> silent +# fall-through to caller-supplied default, restore works +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = '{ "read_local_manifest": true, "version": 1 }'; + manifest_reset(); + + spew($mf, $original); + manifest_create($mf, $stub_content); + + # The stub should be the caller-supplied default (no merge manifest), + # because the pre-existing file had no components to preserve. + is(slurp($mf), $stub_content, + "T6f: no-components manifest falls through to default stub"); + ok(-e "$mf.mtr_saved", "T6f: backup still created"); + is(slurp("$mf.mtr_saved"), $original, + "T6f: backup holds original content"); + + manifest_restore($mf, $stub_content); + ok(-e $mf, "T6f: manifest restored"); + is(slurp($mf), $original, + "T6f: original no-components manifest byte-identical after restore"); +} + +# ====================================================================== +# TEST 7: Pre-existing manifest mentions "components" but value is an +# array (the server's schema requires a string) -> dies loudly, +# original manifest intact +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $original = '{ "components": ["file://x"] }'; + manifest_reset(); + + spew($mf, $original); + + my $died = 0; + eval { + manifest_create($mf, $stub_content); + }; + if ($@) { + $died = 1; + like($@, qr/could not be parsed/, + "T7: dies with 'could not be parsed' message"); + like($@, qr/\Q.mtr_saved\E/, + "T7: error message names the preserved file"); + } + + ok($died, "T7: manifest_create dies on unparseable components"); + + # The original was moved to .mtr_saved before the die fired, so + # the manifest at the original path is gone but the backup is safe. + ok(!-e $mf, + "T7: original moved away (not truncated with stub)"); + ok(-e "$mf.mtr_saved", "T7: backup exists (created before guard)"); + is(slurp("$mf.mtr_saved"), $original, + "T7: backup holds the original content"); +} + +# ====================================================================== +# TEST 7a: The saved manifest is decoded as JSON; only the top-level +# "components" member is merged and the value is re-encoded. +# ====================================================================== +{ + # $merge_case->($original) — install $original, run + # manifest_create, and return (stub or undef, error or '', the + # content left at the path or in the backup). + # Restores so each case leaves the directory clean. + my $merge_case = sub { + my ($original) = @_; + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + manifest_reset(); + spew($mf, $original); + my $stub = eval { manifest_create($mf, $stub_content); slurp($mf) }; + my $err = $@; + manifest_restore($mf, $stub_content) unless $err; + my $restored = -e $mf ? slurp($mf) : slurp("$mf.mtr_saved"); + manifest_reset(); + return ($stub, $err, $restored); + }; + my $merge_of = sub { + '{ "components": "' . $_[0] . '", "merge_local_manifest": true }'; + }; + + # Nested "components" inside another object is ignored; the + # top-level member is the one merged. + my $nested = '{ "metadata": { "components": "file://other" }, ' + . '"components": "file://component_keyring_file" }'; + my ($stub, $err, $restored) = $merge_case->($nested); + is($err, '', "T7a: nested + top-level components accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: top-level components merged, nested one ignored"); + is($restored, $nested, "T7a: original restored byte-identical"); + + # Nested-only: no top-level member, so the plain stub is used. + ($stub, $err) = + $merge_case->('{ "metadata": { "components": "file://other" } }'); + is($err, '', "T7a: nested-only components accepted"); + is($stub, $stub_content, "T7a: nested-only components not merged"); + + # An escaped key is the same member once decoded. + ($stub, $err) = + $merge_case->('{ "\u0063omponents": "file://component_keyring_file" }'); + is($err, '', "T7a: escaped key accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: escaped \"components\" key recognised"); + + # The value is re-encoded: escapes are normalised, quotes kept escaped. + ($stub, $err) = $merge_case->('{ "components": "file:\/\/a\"b" }'); + is($stub, $merge_of->('file://a\"b'), + "T7a: components value re-encoded as a JSON string"); + + # A non-string value is rejected (the schema requires a string). + ($stub, $err) = $merge_case->('{ "components": 42 }'); + like($err, qr/could not be parsed/, "T7a: numeric components dies"); + + # Malformed JSON that mentions "components" dies rather than + # dropping a component list; the backup is kept. + my $bad = '{ "components": "file://x", }'; + ($stub, $err, $restored) = $merge_case->($bad); + like($err, qr/could not be parsed/, + "T7a: malformed JSON mentioning components dies"); + is($restored, $bad, "T7a: malformed backup kept byte-identical"); + + # Malformed JSON with no "components" mention: plain stub. + ($stub, $err) = $merge_case->('{ "read_local_manifest": true, '); + is($err, '', "T7a: malformed JSON without components accepted"); + is($stub, $stub_content, "T7a: malformed JSON without components " + . "uses the plain stub"); + + # read_local_manifest precedence: the server applies it before + # reading "components", so a true value keeps the plain stub. + my $both = '{ "components": "file://component_keyring_file", ' + . '"read_local_manifest": true }'; + ($stub, $err, $restored) = $merge_case->($both); + is($err, '', "T7a: components + read_local true accepted"); + is($stub, $stub_content, + "T7a: components + read_local true uses the plain stub"); + is($restored, $both, + "T7a: components + read_local true restored byte-identical"); + + ($stub, $err) = + $merge_case->('{ "components": "file://component_keyring_file" }'); + is($err, '', "T7a: components only accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: components only uses the merge stub"); + + ($stub, $err) = $merge_case->('{ "read_local_manifest": true }'); + is($err, '', "T7a: read_local only accepted"); + is($stub, $stub_content, "T7a: read_local only uses the plain stub"); + + ($stub, $err) = $merge_case->( + '{ "components": "file://component_keyring_file", ' + . '"read_local_manifest": false }'); + is($err, '', "T7a: components + read_local false accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: components + read_local false uses the merge stub"); + + ($stub, $err) = $merge_case->( + '{ "components": ["file://x"], "read_local_manifest": true }'); + is($err, '', "T7a: non-string components + read_local true no die"); + is($stub, $stub_content, + "T7a: non-string components + read_local true uses the plain stub"); + + # Repeated keys the stub depends on: the server (RapidJSON) selects + # the first occurrence, JSON::PP the last, so refuse rather than + # guess. The backup is kept. + my $dup = '{ "components": "file://component_keyring_file", ' + . '"components": "" }'; + ($stub, $err, $restored) = $merge_case->($dup); + like($err, qr/duplicate key "components"; refusing to derive a stub/, + "T7a: duplicate components dies"); + is($restored, $dup, "T7a: duplicate-key backup kept byte-identical"); + + ($stub, $err) = $merge_case->( + '{ "read_local_manifest": false, "components": "file://x", ' + . '"read_local_manifest": true }'); + like($err, qr/duplicate key "read_local_manifest"/, + "T7a: duplicate read_local_manifest dies"); + + ($stub, $err) = $merge_case->( + '{ "components": "file://x", "\u0063omponents": "file://y" }'); + like($err, qr/duplicate key "components"/, + "T7a: duplicate via escaped key spelling dies"); + + # Repeats nested in another object, or of keys the stub does not + # read, do not affect the derivation and are accepted. + ($stub, $err) = $merge_case->( + '{ "metadata": { "components": "a", "components": "b" }, ' + . '"components": "file://component_keyring_file" }'); + is($err, '', "T7a: nested duplicate accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: nested duplicate ignored, top-level components merged"); + + ($stub, $err) = $merge_case->( + '{ "note": "[{\"x\": 1}]", "note": 2, ' + . '"components": "file://component_keyring_file" }'); + is($err, '', "T7a: duplicate unrelated key accepted"); + is($stub, $merge_of->('file://component_keyring_file'), + "T7a: duplicate unrelated key uses the merge stub"); +} + # ====================================================================== # TEST 7b: Mode preservation — a read-only manifest (0444) keeps its # mode across backup and restore; inode preserved by rename @@ -534,6 +741,40 @@ SKIP: { "T8b: no stub marker written"); } +# ====================================================================== +# TEST 8b2: crash recovery when the merge stub is on disk — backup +# holds a components manifest and holds the MERGE stub +# that manifest_create derived before the crash. +# manifest_create must adopt (no die), rewrite the merge stub; +# restore yields the original backup content. +# ====================================================================== +{ + my $dir = tempdir(CLEANUP => 1); + my $mf = "$dir/mysqld.my"; + my $comp_orig = '{ "components": "file://component_keyring_file" }'; + my $merge_stub = + '{ "components": "file://component_keyring_file", "merge_local_manifest": true }'; + manifest_reset(); + + # Simulate a crash: backup holds the original components manifest, + # holds the merge stub that the previous manifest_create wrote. + spew("$mf.mtr_saved", $comp_orig); + spew($mf, $merge_stub); + + # Must NOT die with 'different content'. + manifest_create($mf, $stub_content); + is(slurp("$mf.mtr_saved"), $comp_orig, + "T8b2: backup adopted (not clobbered)"); + is(slurp($mf), $merge_stub, + "T8b2: merge stub rewritten"); + + manifest_restore($mf, $stub_content); + is(slurp($mf), $comp_orig, + "T8b2: original components manifest restored from backup"); + ok(!-e "$mf.mtr_saved", + "T8b2: backup removed after restore"); +} + # ====================================================================== # TEST 8c: restore failure retry — unlink fails, state not cleared, # second restore after restoring dir perms removes both diff --git a/mysql-test/r/mtr_unit_tests.result b/mysql-test/r/mtr_unit_tests.result index 6a6e9ac41684..820914b5e4bf 100644 --- a/mysql-test/r/mtr_unit_tests.result +++ b/mysql-test/r/mtr_unit_tests.result @@ -250,7 +250,7 @@ ok 10 - 'character-sets-dir' generated ok 11 - 'MASTER_MY_PORT' generated ok 12 - group like 'mysqld' exists 1..12 -1..185 +1..232 ok 1 ok 2 ok 3 @@ -436,3 +436,50 @@ ok 182 ok 183 ok 184 ok 185 +ok 186 +ok 187 +ok 188 +ok 189 +ok 190 +ok 191 +ok 192 +ok 193 +ok 194 +ok 195 +ok 196 +ok 197 +ok 198 +ok 199 +ok 200 +ok 201 +ok 202 +ok 203 +ok 204 +ok 205 +ok 206 +ok 207 +ok 208 +ok 209 +ok 210 +ok 211 +ok 212 +ok 213 +ok 214 +ok 215 +ok 216 +ok 217 +ok 218 +ok 219 +ok 220 +ok 221 +ok 222 +ok 223 +ok 224 +ok 225 +ok 226 +ok 227 +ok 228 +ok 229 +ok 230 +ok 231 +ok 232 diff --git a/sql/sql_component.cc b/sql/sql_component.cc index 069010a6781d..01dbfec49430 100644 --- a/sql/sql_component.cc +++ b/sql/sql_component.cc @@ -239,7 +239,7 @@ Deployed_components::~Deployed_components() { void Deployed_components::get_next_component(std::string &components_list, std::string &one_component) { - std::string component_separator(","); + const std::string component_separator(manifest::kComponentSeparator); one_component.clear(); if (components_list.find(component_separator) != std::string::npos) { one_component = component_separator; @@ -320,6 +320,39 @@ bool Deployed_components::load() { return false; } + /* + merge_local_manifest: the global manifest carries both a "components" + list and the flag "merge_local_manifest": true. We have already + collected the global components above; now open the instance-path + manifest and append its components so that both sets are loaded. + read_local_manifest (checked earlier) discards the global reader + entirely, so the two keys are mutually exclusive in practice — + but no in-tree config combines them, and the silent-discard has + no test coverage. + */ + if (current_reader->merge_local_manifest() == true) { + std::unique_ptr local_reader = + std::make_unique(program_name_, instance_path_); + + if (local_reader->manifest_file() != current_reader->manifest_file() && + !local_reader->empty() && local_reader->ro() == false) { + LogErr(WARNING_LEVEL, ER_WARN_COMPONENTS_INFRASTRUCTURE_MANIFEST_NOT_RO, + local_reader->manifest_file().c_str()); + } + + /* + Also de-duplicates the global list when no local manifest is + merged, so a repeated global URN never reaches the loader. + */ + if (manifest::merge_local_components(*current_reader, *local_reader, + components_) == false) { + last_error_.assign( + "Could not parse 'components' attribute from local manifest " + "file."); + return false; + } + } + std::vector urns; auto free_memory = [&urns]() { for (auto element : urns) diff --git a/unittest/gunit/CMakeLists.txt b/unittest/gunit/CMakeLists.txt index aa127a0a53b7..ca3454c1da27 100644 --- a/unittest/gunit/CMakeLists.txt +++ b/unittest/gunit/CMakeLists.txt @@ -127,6 +127,7 @@ SET(TESTS key like_range m_string + manifest mdl mem_root_array mem_root_deque diff --git a/unittest/gunit/manifest-t.cc b/unittest/gunit/manifest-t.cc new file mode 100644 index 000000000000..9a61060d4d38 --- /dev/null +++ b/unittest/gunit/manifest-t.cc @@ -0,0 +1,315 @@ +/***************************************************************************** + +Copyright (c) 2026 Amazon.com, Inc. All rights reserved. + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License, version 2.0, +as published by the Free Software Foundation. + +This program is designed to work with certain software (including +but not limited to OpenSSL) that is licensed under separate terms, +as designated in a particular file or component or in included license +documentation. The authors of MySQL hereby grant you an additional +permission to link the program and your derivative works with the +separately licensed software that they have either included with +the program or referenced in the documentation. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License, version 2.0, for more details. + +You should have received a copy of the GNU General Public License +along with this program; if not, write to the Free Software +Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +*****************************************************************************/ + +#include +#include +#include +#include +#include +#include +#include +#ifndef _WIN32 +#include +#include +#endif + +#include "manifest.h" + +namespace manifest_unittest { + +using manifest::kComponentSeparator; +using manifest::Manifest_reader; +using manifest::merge_component_lists; + +/* ------------------------------------------------------------------ */ +/* Helper: split a component string the way the consumer does — */ +/* always on comma, matching the historical get_next_component(). */ +/* When the separator constant is wrong (e.g. ";"), the split will */ +/* not find the delimiter and the test fails. */ +/* ------------------------------------------------------------------ */ +static std::vector consumer_split(const std::string &s) { + std::vector result; + if (s.empty()) return result; + const std::string sep(","); /* hardcoded — this is the contract */ + std::string::size_type start = 0; + std::string::size_type pos; + while ((pos = s.find(sep, start)) != std::string::npos) { + result.push_back(s.substr(start, pos - start)); + start = pos + sep.size(); + } + result.push_back(s.substr(start)); + return result; +} + +/* ================================================================== */ +/* (a) merge_component_lists round-trips through the separator */ +/* ================================================================== */ + +TEST(ManifestMerge, MergeJoinsWithConsumerSeparator) { + const std::string global = "file://component_a"; + const std::string local = "file://component_b"; + const std::string merged = merge_component_lists(global, local); + + /* The merged string must split back into exactly the two inputs + when the consumer splits on comma — this catches the ";" bug. */ + auto parts = consumer_split(merged); + ASSERT_EQ(2u, parts.size()); + EXPECT_EQ(global, parts[0]); + EXPECT_EQ(local, parts[1]); +} + +/* ================================================================== */ +/* (b) edge cases */ +/* ================================================================== */ + +TEST(ManifestMerge, EmptyGlobalReturnsLocal) { + EXPECT_EQ("file://local", merge_component_lists("", "file://local")); +} + +TEST(ManifestMerge, EmptyLocalReturnsGlobal) { + EXPECT_EQ("file://global", merge_component_lists("file://global", "")); +} + +TEST(ManifestMerge, BothEmptyReturnsEmpty) { + EXPECT_EQ("", merge_component_lists("", "")); +} + +TEST(ManifestMerge, NoTrailingSeparator) { + const std::string merged = merge_component_lists("a", "b"); + const std::string sep(kComponentSeparator); + /* Must not end with the separator. */ + EXPECT_NE(sep, merged.substr(merged.size() - sep.size())); + /* Must not start with the separator. */ + EXPECT_NE(sep, merged.substr(0, sep.size())); +} + +/* ================================================================== */ +/* (d) de-duplication */ +/* ================================================================== */ + +TEST(ManifestMerge, DuplicateAcrossGlobalLocalKeptOnceGlobalPosition) { + /* 'shared' appears in both; it should appear once, in the global + position, and the local-only URN appended after. */ + const std::string merged = + merge_component_lists("file://a,file://shared", "file://shared,file://b"); + auto parts = consumer_split(merged); + ASSERT_EQ(3u, parts.size()); + EXPECT_EQ("file://a", parts[0]); + EXPECT_EQ("file://shared", parts[1]); + EXPECT_EQ("file://b", parts[2]); +} + +TEST(ManifestMerge, LocalEntirelyContainedInGlobalYieldsGlobalUnchanged) { + const std::string global = "file://a,file://b,file://c"; + const std::string merged = merge_component_lists(global, "file://b,file://a"); + EXPECT_EQ(global, merged); +} + +TEST(ManifestMerge, DuplicatesWithinOneSideCollapsed) { + /* Duplicates within global should collapse; likewise within local. */ + const std::string merged = + merge_component_lists("file://x,file://x", "file://y,file://y"); + auto parts = consumer_split(merged); + ASSERT_EQ(2u, parts.size()); + EXPECT_EQ("file://x", parts[0]); + EXPECT_EQ("file://y", parts[1]); +} + +TEST(ManifestMerge, EmptyTokensFromDoubleCommaDoNotProduceEmptyURNs) { + /* Consecutive separators (',,') must not yield empty strings. */ + const std::string merged = + merge_component_lists("file://a,,file://b", ",,file://c,,"); + auto parts = consumer_split(merged); + ASSERT_EQ(3u, parts.size()); + EXPECT_EQ("file://a", parts[0]); + EXPECT_EQ("file://b", parts[1]); + EXPECT_EQ("file://c", parts[2]); +} + +/* ================================================================== */ +/* (c) Manifest_reader via temp manifest files */ +/* ================================================================== */ + +#ifndef _WIN32 +/* mkdtemp()/rmdir() are POSIX-only; the "/tmp" path does not exist */ +/* on Windows. Guard the entire fixture so it compiles everywhere. */ + +/* Helper: write a manifest file for a fake executable and return */ +/* the executable path that Manifest_reader expects. */ +class ManifestReaderTest : public ::testing::Test { + protected: + std::string tmpdir_; + + void SetUp() override { + /* Create a unique temporary directory. */ + char tmpl[] = "/tmp/manifest_test_XXXXXX"; + char *d = mkdtemp(tmpl); + ASSERT_NE(nullptr, d); + tmpdir_ = d; + } + + void TearDown() override { + /* Clean up temp files. */ + std::string manifest = tmpdir_ + "/mysqld.my"; + std::remove(manifest.c_str()); + std::string local_manifest = local_dir() + "mysqld.my"; + std::remove(local_manifest.c_str()); + rmdir(local_dir().c_str()); + rmdir(tmpdir_.c_str()); + } + + /* Instance path for local manifests (trailing separator, as the */ + /* server passes it). */ + std::string local_dir() const { return tmpdir_ + "/instance/"; } + + /* Write content to local_dir()/mysqld.my. */ + void write_local_manifest(const std::string &content) { + mkdir(local_dir().c_str(), 0700); + std::ofstream ofs(local_dir() + "mysqld.my"); + ofs << content; + ofs.close(); + } + + /* Write content to tmpdir_/mysqld.my and return the fake exe path. */ + std::string write_manifest(const std::string &content) { + std::string manifest_path = tmpdir_ + "/mysqld.my"; + std::ofstream ofs(manifest_path); + ofs << content; + ofs.close(); + return tmpdir_ + "/mysqld"; + } +}; + +TEST_F(ManifestReaderTest, MergeLocalManifestDefaultsFalse) { + /* A manifest with only read_local_manifest — merge_local_manifest + must default to false. */ + std::string exe = write_manifest(R"({ "read_local_manifest": true })"); + + Manifest_reader reader(exe, ""); + EXPECT_TRUE(reader.file_present()); + EXPECT_TRUE(reader.read_local_manifest()); + EXPECT_FALSE(reader.merge_local_manifest()); +} + +TEST_F(ManifestReaderTest, MergeLocalManifestTrueWhenSet) { + std::string exe = write_manifest( + R"({ "components": "file://comp_a", "merge_local_manifest": true })"); + + Manifest_reader reader(exe, ""); + EXPECT_TRUE(reader.file_present()); + EXPECT_TRUE(reader.merge_local_manifest()); + + std::string components; + EXPECT_TRUE(reader.components(components)); + EXPECT_EQ("file://comp_a", components); +} + +TEST_F(ManifestReaderTest, ReadLocalManifestParsesAsExpected) { + std::string exe = write_manifest(R"({ "read_local_manifest": true })"); + + Manifest_reader reader(exe, ""); + EXPECT_TRUE(reader.read_local_manifest()); + EXPECT_FALSE(reader.merge_local_manifest()); +} + +/* ------------------------------------------------------------------ */ +/* merge_local_components: the merge_local_manifest step of */ +/* Deployed_components::load(), driven by real manifest files. */ +/* ------------------------------------------------------------------ */ + +TEST_F(ManifestReaderTest, MergeNoLocalManifestDedupsGlobal) { + std::string exe = write_manifest( + R"({ "components": "file://a,file://a", "merge_local_manifest": true })"); + Manifest_reader global(exe, ""); + Manifest_reader local(exe, local_dir()); /* absent */ + EXPECT_FALSE(local.file_present()); + + std::string components; + ASSERT_TRUE(global.components(components)); + EXPECT_TRUE(manifest::merge_local_components(global, local, components)); + EXPECT_EQ("file://a", components); +} + +TEST_F(ManifestReaderTest, MergeZeroByteLocalManifestDedupsGlobal) { + std::string exe = write_manifest( + R"({ "components": "file://a,file://a", "merge_local_manifest": true })"); + write_local_manifest(""); + Manifest_reader global(exe, ""); + Manifest_reader local(exe, local_dir()); + EXPECT_TRUE(local.file_present()); + EXPECT_TRUE(local.empty()); + + std::string components; + ASSERT_TRUE(global.components(components)); + EXPECT_TRUE(manifest::merge_local_components(global, local, components)); + EXPECT_EQ("file://a", components); +} + +TEST_F(ManifestReaderTest, MergeLocalManifestAppendedAndDeduped) { + std::string exe = write_manifest( + R"({ "components": "file://a,file://a", "merge_local_manifest": true })"); + write_local_manifest(R"({ "components": "file://a,file://b" })"); + Manifest_reader global(exe, ""); + Manifest_reader local(exe, local_dir()); + + std::string components; + ASSERT_TRUE(global.components(components)); + EXPECT_TRUE(manifest::merge_local_components(global, local, components)); + EXPECT_EQ("file://a,file://b", components); +} + +TEST_F(ManifestReaderTest, MergeLocalIsGlobalFileDedupsGlobal) { + /* Instance path resolving to the global manifest: not merged into */ + /* itself, but the global list is still de-duplicated. */ + std::string exe = write_manifest( + R"({ "components": "file://a,file://a", "merge_local_manifest": true })"); + Manifest_reader global(exe, ""); + Manifest_reader local(exe, tmpdir_ + "/"); + ASSERT_EQ(global.manifest_file(), local.manifest_file()); + + std::string components; + ASSERT_TRUE(global.components(components)); + EXPECT_TRUE(manifest::merge_local_components(global, local, components)); + EXPECT_EQ("file://a", components); +} + +TEST_F(ManifestReaderTest, MergeLocalWithoutComponentsFails) { + std::string exe = write_manifest( + R"({ "components": "file://a", "merge_local_manifest": true })"); + write_local_manifest(R"({ "read_local_manifest": false })"); + Manifest_reader global(exe, ""); + Manifest_reader local(exe, local_dir()); + + std::string components; + ASSERT_TRUE(global.components(components)); + EXPECT_FALSE(manifest::merge_local_components(global, local, components)); + EXPECT_EQ("file://a", components); /* unchanged on failure */ +} + +#endif /* !_WIN32 */ + +} // namespace manifest_unittest