Support running the Aspire dashboard from tests - #19135
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ffeff87e-f284-434d-87d3-843e21a7aebb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ffeff87e-f284-434d-87d3-843e21a7aebb
Reuse the canonical dashboard URL path, enforce authenticated testing defaults, and preserve cancellation and disposal semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ffeff87e-f284-434d-87d3-843e21a7aebb
Lifecycle correctness: - BuildAsync no longer blocks on the released AppHost when the caller's token fires. It reclaims the late application on a background continuation and rethrows immediately, so cancellation is prompt even when the AppHost never finishes building. - DistributedApplicationFactory.OnBuiltCoreAsync disposes an application that arrives after the factory was disposed. Previously TrySetResult simply returned false and the built DistributedApplication, its service provider, and its orchestrator processes leaked for the lifetime of the test process. - DisposeAsync claims disposal with Interlocked.Exchange. The previous IsCancellationRequested read was not atomic with OnDisposed(), so concurrent disposers could both run teardown and race on the same application. - ObjectDisposedException now consistently reports IDistributedApplicationTestingBuilder rather than leaking the internal factory type name. Behavior: - DistributedApplicationOptions.DisableDashboard = false, the pre-existing spelling of "run the dashboard", now receives the same hardened testing defaults as the new EnableDashboard option instead of only one of them being hardened. - Dashboard endpoints are configured with an empty URL, which is how the product asks for a dynamically assigned port. The previous "http://127.0.0.1:0" parses to a literal fixed port 0 and was only dynamic while DcpPublisher:RandomizePorts stayed true. - A fresh browser token is generated per application and supplied through the command line, which has the highest precedence. Disabling anonymous access only closes the door if a credential exists, and an ambient ASPIRE_DASHBOARD_FRONTEND_BROWSERTOKEN would otherwise share one known token across every application on a CI agent. - DistributedApplicationTestingBuilderOptions.DefaultWaitBehavior lets a debugging session keep a stuck resource alive to inspect it, rather than always tearing the application down. Conventions: - Exception messages moved to Properties/Resources.resx with regenerated xlf, matching the rest of the file. - Commented the Aspire.Hosting InternalsVisibleTo grant, including why the shared file links had to be dropped. - Replaced vacuous Assert.NotNull-on-a-lambda assertions with assertions that exercise the overloads, and added coverage for per-application token isolation, DefaultWaitBehavior, publish-mode rejection through configureBuilder, and disposal of a late-arriving application. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f3f649b5-52c4-4eb5-8d74-c4744e79faf4
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19135Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19135" |
There was a problem hiding this comment.
Pull request overview
Adds first-class support for authenticated Aspire dashboards in integration tests and hardens testing-builder lifecycle behavior.
Changes:
- Adds dashboard testing options and login URL retrieval.
- Enforces secure, dynamic dashboard defaults.
- Fixes cancellation/disposal races and adds lifecycle/dashboard tests.
Show a summary per file
| File | Description |
|---|---|
tests/TestingAppHost1/TestingAppHost1.AppHost/TestingAppHostBuildProbe.cs |
Adds deterministic build lifecycle probe. |
tests/TestingAppHost1/TestingAppHost1.AppHost/Program.cs |
Supports dashboard and build test scenarios. |
tests/Aspire.Hosting.Testing.Tests/DashboardTestingBuilderTests.cs |
Tests dashboard configuration and lifecycle behavior. |
tests/Aspire.Hosting.Testing.Tests/DashboardLoginUrlTests.cs |
Tests dashboard URL and authentication behavior. |
src/Aspire.Hosting/Backchannel/DashboardUrlsHelper.cs |
Escapes tokens and preserves dashboard failures. |
src/Aspire.Hosting/Aspire.Hosting.csproj |
Grants testing package internal access. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.cs.xlf |
Updates Czech resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.de.xlf |
Updates German resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.es.xlf |
Updates Spanish resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.fr.xlf |
Updates French resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.it.xlf |
Updates Italian resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.ja.xlf |
Updates Japanese resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.ko.xlf |
Updates Korean resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.pl.xlf |
Updates Polish resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.pt-BR.xlf |
Updates Brazilian Portuguese resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.ru.xlf |
Updates Russian resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.tr.xlf |
Updates Turkish resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.zh-Hans.xlf |
Updates Simplified Chinese resources. |
src/Aspire.Hosting.Testing/Properties/xlf/Resources.zh-Hant.xlf |
Updates Traditional Chinese resources. |
src/Aspire.Hosting.Testing/Properties/Resources.resx |
Adds dashboard error messages. |
src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs |
Exposes generated resource accessors. |
src/Aspire.Hosting.Testing/DistributedApplicationTestingBuilderOptions.cs |
Adds public dashboard testing options. |
src/Aspire.Hosting.Testing/DistributedApplicationTestingBuilder.cs |
Configures dashboards and fixes build lifecycle handling. |
src/Aspire.Hosting.Testing/DistributedApplicationHostingTestingExtensions.cs |
Adds dashboard login URL API. |
src/Aspire.Hosting.Testing/DistributedApplicationFactory.cs |
Fixes late-application and concurrent disposal handling. |
src/Aspire.Hosting.Testing/Aspire.Hosting.Testing.csproj |
Removes duplicated shared source links. |
Review details
Files not reviewed (1)
- src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs: Generated file
Suppressed comments (1)
src/Aspire.Hosting.Testing/DistributedApplicationTestingBuilder.cs:253
- This existing public overload can now reject publish mode when
configureBuilderenables the dashboard, but the newInvalidOperationExceptionis absent from its XML documentation. Please document the condition as part of this behavior change.
public static IDistributedApplicationTestingBuilder Create(string[] args, Action<DistributedApplicationOptions, HostApplicationBuilderSettings> configureBuilder)
=> CreateCore(args, testingOptions: null, configureBuilder);
- Files reviewed: 25/26 changed files
- Comments generated: 2
- Review effort level: Balanced
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
This overlaps slightly with #18746 |
Restrict the hardened dashboard testing defaults to the explicit DistributedApplicationTestingBuilderOptions.EnableDashboard opt-in. Treating DistributedApplicationOptions.DisableDashboard = false as an equivalent spelling changed behavior for every caller already using it: - The args rebuild read hostBuilderOptions.Args, which the factory has already populated, so arguments a configureBuilder callback assigned to applicationOptions.Args were dropped. DashboardIsNotAddedInPublishMode lost "--publisher manifest", ran in run mode, and saw a dashboard resource it asserted was absent. - The appended dashboard settings overrode the caller's own configuration. GetDashboardUrlsAsync_ReturnsBaseUrl_WhenDashboardAllowsAnonymousAccess asked for anonymous access and got a browser token instead. Also merge caller args instead of choosing one array, so the new opt-in appends to whatever the callback left in place. Build Aspire.Dashboard from Aspire.Hosting.Testing.Tests. The tests that start a real dashboard resolve it from the repo-wide AspireDashboardDir, which was empty in the Hosting.Testing CI leg, failing all eight of them. Add ConcurrentDisposeAsyncRunsTeardownOnce, covering the interlocked disposal claim in DistributedApplicationFactory.DisposeAsync. With the claim reverted to a cancellation-token read the test fails consistently; with the claim in place it passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs: Generated file
Suppressed comments (2)
src/Aspire.Hosting.Testing/DistributedApplicationTestingBuilder.cs:290
- The PR description still says
DistributedApplicationOptions.DisableDashboard = falseis equivalent to the new option and receives the same hardening, but this code now explicitly preserves the older unhardened behavior (and the new tests assert that behavior). Please update the description and security notes so reviewers and users are not promised hardening on that existing path.
// Only the explicit option turns on the hardened testing defaults below. Setting
// DistributedApplicationOptions.DisableDashboard = false through the configureBuilder callback is the older,
// already-shipped spelling of "run a dashboard", and it has to keep the behavior it shipped with: callers use
// it to exercise dashboard behavior against configuration they chose themselves (fixed URLs, anonymous
// access, an ambient browser token), and in publish mode it is simply ignored because no dashboard resource
// is ever added. Treating it as equivalent to the option silently rewrote that configuration and turned
// publish-mode callers into an InvalidOperationException.
src/Aspire.Hosting.Testing/DistributedApplicationFactory.cs:510
- A concurrent caller that loses this exchange returns from
DisposeAsyncimmediately, potentially before the winner callsOnDisposed()and before any teardown completes. Consequently, awaitingDisposeAsynccan report completion while the application is still running, and public factory operations can still pass the_disposingCtsdisposed check during that window. Store/share the winning disposal task (or completion source) so every concurrent caller awaits the same teardown rather than returning early.
if (Interlocked.Exchange(ref _disposeClaimed, 1) == 1)
{
// Dispose already called.
return;
- Files reviewed: 26/27 changed files
- Comments generated: 1
- Review effort level: Balanced
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs: Generated file
Suppressed comments (3)
src/Aspire.Hosting.Testing/DistributedApplicationFactory.cs:507
- The atomic disposal claim still leaves a publication race.
DisposeAsynccan observe_appTcsas incomplete, thenOnBuiltCoreAsynccan successfully publish the application beforeTrySetCanceledruns. The disposer then returns,OnBuiltCoreAsyncdoes not reclaim the successfully published application, and every later disposer exits because this exchange already claimed disposal. Recheck the result ofTrySetCanceledand fall through to teardown when publication won the race.
if (Interlocked.Exchange(ref _disposeClaimed, 1) == 1)
src/Aspire.Hosting.Testing/DistributedApplicationTestingBuilder.cs:290
- The PR description still says
DistributedApplicationOptions.DisableDashboard = falseis equivalent to the new option and receives the same hardening, but this code deliberately preserves its older, unhardened behavior. Update the description and security summary so reviewers and users are not told this existing opt-in gets protections that the implementation intentionally does not apply.
// Only the explicit option turns on the hardened testing defaults below. Setting
// DistributedApplicationOptions.DisableDashboard = false through the configureBuilder callback is the older,
// already-shipped spelling of "run a dashboard", and it has to keep the behavior it shipped with: callers use
// it to exercise dashboard behavior against configuration they chose themselves (fixed URLs, anonymous
// access, an ambient browser token), and in publish mode it is simply ignored because no dashboard resource
// is ever added. Treating it as equivalent to the option silently rewrote that configuration and turned
// publish-mode callers into an InvalidOperationException.
tests/Aspire.Hosting.Testing.Tests/DashboardTestingBuilderTests.cs:391
- The PR description promises that AppHost configuration cannot downgrade the authenticated dashboard defaults, while this test intentionally codifies that the AppHost can clear
AppHost:BrowserToken, causing the dashboard to use unsecured frontend authentication. Since the escape hatch is intentional, update the description and security guarantee to document this exception.
public async Task DashboardTestingDoesNotPinTheBrowserTokenAgainstTheAppHost(CreationSurface creationSurface)
{
// Anonymous access is settled while the builder is constructed and cannot be taken back, but the browser
// token is only read out of AppHost:BrowserToken when DashboardOptions is first resolved, so an AppHost
// that clears the key still gets a dashboard with no credential. That asymmetry is deliberate rather than
- Files reviewed: 26/29 changed files
- Comments generated: 1
- Review effort level: Balanced
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Review asked for the generated browser token to be restored after the AppHost configures itself, so an AppHost clearing AppHost:BrowserToken could not downgrade the dashboard to Unsecured authentication. Pinning it onto DashboardOptions works, but it removes the only way to reach the anonymous dashboard path once EnableDashboard has appended its own arguments. Two tests in this PR rely on that escape hatch through the returned builder, and both fail with the token pinned: GetDashboardLoginUrlAsyncThrowsWhenDashboardAllowsAnonymousAccess and CanonicalDashboardLoginUrlEscapesBrowserToken. The state is also not silent. DashboardUrlsHelper reports HasBrowserToken false and GetDashboardLoginUrlAsync throws rather than returning an unauthenticated URL. Cover the boundary instead, so neither side of it can drift unnoticed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
f2acafc to
10b6337
Compare
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Reaching EndpointHostHelpers.GetUrlWithTargetHostAsync from Aspire.Hosting.Testing needed access that the package did not have. The first attempt granted Aspire.Hosting.Testing an InternalsVisibleTo in Aspire.Hosting.csproj, which is the exact pattern microsoft#18918 removed from Aspire.Hosting.Dotnet and 453da9c removed from Aspire.Hosting.Rust: two independently restorable packages with a default >= dependency range, so a version mismatch fails at runtime rather than at compile time, and none of the coupling is visible to API review. The grant also forced removing five $(SharedDir) Compile items from Aspire.Hosting.Testing, because source-shared internals and IVT-exposed internals are the same type twice (CS0436) - churn in the opposite direction from what the Rust change established. Every dependency the helper actually needs is already public: EndpointReference.EndpointAnnotation, EndpointReference.GetValueAsync, EndpointAnnotation.TargetHost, and EndpointHostHelpers.IsLocalhostTld. The internal modifier on this one method was incidental, and the class around it is already public with eight documented public methods. Make the method public, drop the IVT, and restore the shared Compile items. Aspire.Hosting.csproj now has no diff at all against the PR base. Cover the substitution rule in EndpointHostHelpersTests, which is deterministic and needs neither a container runtime nor a real socket, unlike the GetDashboardUrlAsync integration test that pins the call site. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 522a7395-2ea1-4d1a-91b3-40050853b9f4
|
Pushed a follow-up to The previous commit reached Everything the helper needs is already public — Also added |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Making GetUrlWithTargetHostAsync public widened its contract to every EndpointReference, including one carrying a non-local ContextNetworkID. EndpointReference resolves those against the container network, so the same endpoint yields container1.dev.internal:10005 rather than a loopback address - and the method then replaced that host with the .localhost target host. A *.localhost name always resolves to the caller's own loopback, so the rewritten URL pointed a container at itself. Substitute only when the resolved address is itself loopback, which is exactly the case the method documents: DCP allocates "localhost" because that is what the service binds to, and the TLD is the name the user expects to see. All three production callers - the dashboard startup summary, the CLI backchannel API URL, and GetDashboardUrlAsync - resolve host-facing loopback endpoints, so their behavior is unchanged. Verified by a test that pins the container-network address; it reproduces the rewrite before the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 522a7395-2ea1-4d1a-91b3-40050853b9f4
|
One more in
Now it only substitutes when the resolved address is itself loopback, which is what the doc comment already described (DCP allocates |
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs: Generated file
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
tests/Aspire.Hosting.Testing.Tests/DashboardUrlTests.cs:17
- The validation command in the PR description filters for
*.DashboardLoginUrlTests, but the new class is namedDashboardUrlTestsand noDashboardLoginUrlTestsclass exists. As written, the command does not exercise these dashboard URL/authentication tests, so update the filter and rerun the stated validation.
public class DashboardUrlTests
- Files reviewed: 29/30 changed files
- Comments generated: 0 new
- Review effort level: Balanced
GetUrlWithTargetHostAsync became public in an earlier commit on this branch, so `endpoint` is now an external API input. It was dereferenced without validation, which surfaces a NullReferenceException instead of the repository-standard ArgumentNullException when a null reaches it from a language without nullable reference type enforcement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 522a7395-2ea1-4d1a-91b3-40050853b9f4
EndpointHostHelpers is a shipped public type whose summary described it as validating localhost addresses. GetUrlWithTargetHostAsync became public on this branch, so the type now also resolves endpoint URLs against a configured target host and the summary no longer matched what the type exposes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 522a7395-2ea1-4d1a-91b3-40050853b9f4
Ella Hathaway (ellahathaway)
left a comment
There was a problem hiding this comment.
I reviewed the testing-builder creation paths, dashboard startup and authentication flow, URL resolution, lifecycle and disposal behavior, public API shape, and regression coverage.
I found no concrete correctness or security defects. I left two nonblocking notes about security documentation and test-host wait semantics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- src/Aspire.Hosting.Testing/Properties/Resources.Designer.cs: Generated file
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
tests/Aspire.Hosting.Testing.Tests/DashboardUrlTests.cs:17
- The Validation command still filters
*.DashboardLoginUrlTests, but this class is now namedDashboardUrlTests. That filter selects no tests, so the documented 39-test validation is not reproducible; update the PR command to filter*.DashboardUrlTests.
public class DashboardUrlTests
- Files reviewed: 29/30 changed files
- Comments generated: 0 new
- Review effort level: Balanced
f841c47
into
microsoft:main
Summary
Adds first-class support for running the Aspire dashboard from an application test.
Changes
DistributedApplicationTestingBuilderOptions.EnableDashboard.GetDashboardUrlAsync()for a healthy, started dashboard and keeps the live browser token out of AppHost/test logs.DisableDashboard = falsebehavior; only the explicit testing option applies the hardened defaults.The independent testing-builder disposal state machine and dashboard URL-encoding fix are intentionally not part of this PR.
Validation
Checklist