Skip to content

Consume FSAC test hierarchy and enable MTP tests - #2180

Open
shayanhabibi wants to merge 9 commits into
ionide:mainfrom
shayanhabibi:feature/mtp-server-mode
Open

shayanhabibi wants to merge 9 commits into
ionide:mainfrom
shayanhabibi:feature/mtp-server-mode

Conversation

@shayanhabibi

@shayanhabibi shayanhabibi commented Sep 23, 2026 •

Copy link
Copy Markdown

Problem

Ionide's test explorer infers grouping nodes by splitting test names, and selects tests to run with VSTest filter expressions. Neither works for Microsoft.Testing.Platform (MTP) projects: MTP test names are display names that need not be dotted paths, and MTP has no VSTest filter.

Change

Depends on ionide/FsAutoComplete#1552. Tracks #2069.

  • Setting: adds FSharp.enableTestingPlatform, off by default, and passes it to FSAC. With it off, behaviour is unchanged.
  • Tree from FSAC: explorer nodes are built from the id, parent id and leaf marker FSAC reports for each test, instead of from name splitting. The TRX and AST paths still infer from names.
  • Node labels: a node is labelled by what its name adds to its parent's name. A name that contains separators but is one test stays one node.
  • Running by id: a run sends the ids FSAC issued, for VSTest and MTP projects alike. Selecting a group sends the ids of its leaf tests. A request carries either ids or a VSTest filter, never both.
  • Stable identity: a test keeps its FSAC id when its code is edited, and tests FSAC tells apart stay separate in the explorer even when their names match.
  • Refresh: refreshing builds MTP test projects, as it already did for VSTest projects.

How MTP tests are grouped is decided in FSAC: by the parent MTP reports, otherwise by the type MTP names for the test.

Validation

On the branch after merging current main:

  • dotnet fsi build.fsx (format check, Fable build, VSIX package) passes.
  • dotnet fsi build.fsx -p Test (VS Code smoke test) passes.
  • The paired FSAC branch, on net10.0: 176 test-explorer tests pass, LSP TestExplorerTests 74 pass, LSP test-detection tests 12 pass. The rest of the FSAC LSP suite was not run.

On an earlier revision of this branch:

  • A development-host run against the test-explorer regression workspace discovered and ran the MTP xUnit project, with all four tests passing.
  • Debugging a selected MTP test attached CoreCLR and hit the breakpoint.

Not yet checked:

  • The node-label change and the type-based MTP grouping have not been exercised in VS Code. This repository has no unit tests for the explorer tree.
  • GitHub reports no checks for this fork PR.

Known limits

  • Needs an FSAC release that contains Add Microsoft.Testing.Platform server-mode test support FsAutoComplete#1552.
  • MTP filter expressions are not supported; MTP tests run by id only.
  • A framework that reports neither a parent nor a type for its tests shows them flat under the project.
  • TUnit parameterised cases are not grouped under their method.
  • NUnit reports the wrong type for test names that contain separators, so those tests are grouped by that type.

FsAutoComplete now reports grouping nodes and parent ids, and normalises
parameterised test names itself. Rebuilding the tree here by splitting
fully-qualified names would disagree with the server as soon as a project
runs under Microsoft.Testing Platform, whose nodes carry parent ids the
names do not encode.

TestItemDTO carries Id, ParentId and IsLeaf, and discovery builds its
hierarchy by following those links. getFullname_withNestedParamTests is
gone: the server's FullName already distinguishes theory cases, so applying
the same rule here appended the case parameters twice.

Name inference stays for the TRX and AST paths, which report names alone.
Exposes the FSAC flag that routes projects setting
IsTestingPlatformApplication through the Microsoft.Testing.Platform server
protocol. Off by default.
Microsoft.Testing.Platform runs a test named by the opaque uid it was
discovered under, rather than by a filter expression over its name, so a test
carries that uid from discovery through to the run.

A grouping node holds no uid of its own, so selecting one is read as the
runnable tests beneath it. A test discovered through VSTest carries no uid and
is still named by the filter expression, which is sent alongside.

@shayanhabibi shayanhabibi left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question: Should we discriminate between DTOs targeting VSTest/MTP instead of optional fields

@shayanhabibi
shayanhabibi marked this pull request as ready for review September 24, 2026 05:46
@shayanhabibi

Copy link
Copy Markdown
Author

@farlee2121

See the other PR for the heart of the implementation.

I do want to pursue some cleanup of Ionide settings (ie grouping them) and other facets, but this is the foundational work.

Open Question: Should I continue adding these other facets, and we cherry pick them into separate PRs later/just merge them all later?

@farlee2121

Copy link
Copy Markdown
Contributor

Unique ids

I haven't gotten to look closely at this, but I see one item that concerns me

Should we discriminate between DTOs targeting VSTest/MTP instead of optional fields

Preserve MTP test UIDs through discovery and send selected leaf UIDs on run requests, including when a grouping node is selected. Continue sending VSTest filters for VSTest projects.

This sounds like VSTest and MTP behave in different and incompatible ways through the same api, causing a leaky abstraction that requires consumers to understand how certain calls will behave based on implicit context.

Is there a way we can bridge the gap? Have unique identifiers for both MTP and VSTest?

I notice VsTestConsoleWrapper.RunTests can run a sequence of TestCases, and a TestCase can be constructed with FullyQualifiedName, ExecutorUri, and Source. To my understanding, that combination should be unique. Perhaps we can package these into a unique identifier. Then both VSTest and MTP could run specific tests by an "id". The id is provided by the server and interpreted by the server and treated as a black box by the client.

Consumers could get themselves into trouble by trying to parse out id components, but that'd be unsupported shenanigans requiring some reverse engineering.

Unique Ids and Test Filters

Related. What happens if both test ids and a filter are specified in the run request?
Those seem like mutually exclusive parameters.

Union types don't properly exist over process boundaries. We could throw an error if both are specified. In any case, it's a behavior we should think about.

@shayanhabibi

Copy link
Copy Markdown
Author

Things get really finicky as we start to treat one field as another. Strictly speaking, display names are not unique ids. The default filter works on display names, not unique ids. There is a separate filter argument in MTP for specifying unique ids.

For us in FSharp, where our tests are constructed as values instead of associated with types/members, we essentially treat them the same.

So I don't disagree with you at all - yeah I'll backport VSTest API to fill fields that are required in MTP so we can remove fields that are optional only because one API uses it - however, we should be careful when communicating how filters work (as you point out).

I don't actually know if MTP allows you to specify uid filters with normal name filters. Something I'll look into. But - we should be able to request both.

I'll report back after review

@farlee2121

farlee2121 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

I think I didn't correctly communicate my full idea.

I didn't mean to suggest stuffing unique ids into the filter or filters into unique ids. I would expect filter and id-based runs to be distinct.

I'm suggesting that both filters and id-based runs are available for both MTP and VSTest, and work consistently between the two. All tests could be returned from FSAC with a unique id. FSAC could service run requests by these ids using any underlying platform.

  • For MTP, they already have proper dedicated ids so we use those ids and underlying apis
  • For VSTest, VSTest exposes two overloads of RunTests. The one we currently use runs the whole project. The other accepts a list of specific TestCase objects to run. These TestCases have no Id, but the combination of members is unique (or vstest throws an error). So, we could possibly package these members into a string in a standard format to act as a unique identifier. Then, when the id is passed back to the server, the server can unpack the values back into a TestCase to run the specific tests. The client is indifferent to the format of the id, and the FSAC consumer experience is uniform between MTP and VSTest

UPDATE: I looked at the VSTest code for TestCase, and they're actually constructing a Guid based on the three TestCase members for internal use as a unique id. So I think we'd be pretty safe assuming the combo of FullyQualifiedName, ExecutorUri, and Source can act as an id.

// Part of TestCase
	private Guid GetTestId()
	{
		string text = Source;
		try {
			text = Path.GetFileName(text);
		} catch {
		}
		string text2 = ExecutorUri?.ToString() + text;
		text2 += GetFullyQualifiedName();
		return EqtHash.GuidFromString(text2);
	}

FsAutoComplete now gives every test an opaque id that routes it to its
project, target framework and platform, and runs a selection named by those
ids. A test explorer item keeps the id FSAC reported for it, and a run sends
the ids of the tests under the selection and nothing else: FSAC rejects ids
sent together with a filter expression, and works out the projects to run
from the ids themselves. The filter expression stays with the dotnet CLI path.

A selected test FSAC did not report, such as one found only in code, has no
id. Discovery runs once to find it, and a test still without one is marked
errored instead of widening the run.

Results and started notifications are matched to the explorer by that id,
so tests that share a name within a project, such as theory rows run on
Microsoft.Testing.Platform, no longer collide. The name-based id is only a
fallback for items and results that carry no id.

PlatformUid is gone from TestItemDTO, and TestUids is now TestIds.
An explorer item is still named by its project and full name, and FSAC
does not keep full names unique. A grouping can share its name with a
test, and on Microsoft.Testing.Platform, where the full name is the
display name, several tests can share one. Siblings that share a name got
one explorer id, so the item added last replaced the others: a grouping
hid the test of its name, and only one of several same-named tests stayed
in the tree. The previous commit matched results by FSAC's id, but the
tree still collapsed these tests, so they did not stay apart as it said.

A grouping is now shown as one node with the test of its name, and that
test runs along with the tests under it. Each further test of a name gets
a numbered name. Selections and missing results are told apart by FSAC's
id where there is one. Before, tests of one name under different parents
were deduplicated to one, so the others were never sent to FSAC.
@shayanhabibi

Copy link
Copy Markdown
Author

@farlee2121 Gotcha, that makes sense - I've reworked it so every test FSAC hands out gets an opaque id, and id runs and filter runs are separate paths that behave the same on VSTest and MTP.

The VSTest triple

I tried the FullyQualifiedName + ExecutorUri + Source approach first, but it doesn't hold up by itself:

  • Not unique for parameterised tests. Every xUnit [<Theory>] row and MSTest DataRow shares one FQN, so all rows collapse to the same triple (and the same GetTestId hash).
  • A TestCase rebuilt from the triple doesn't run everywhere. Ran discovery then RunTests(IEnumerable<TestCase>) against real projects:
Adapter Rebuilt TestCase runs? Needs
NUnit (adapter 5.1) ✅ -
Expecto (YoloDev 0.14) ✅ -
xUnit v2 (runner 2.8) ❌ XunitTestCase serialized property (~2.6 KB)
xUnit v2 on runner 3.x / xUnit v3 ❌ XunitTestCaseUniqueID + XunitTestCaseSerialization (~4 KB)
MSTest 3.6 ❌ MSTestDiscoverer.TestClassName, plus DynamicData per row

xUnit and MSTest set their own TestCase.Id (unique per row) rather than the triple hash. When an adapter doesn't set one, TestCase.Id falls back to the GetTestId hash you found - so NUnit/Expecto end up with your triple anyway.

What I've got now

  • Id: t1|<kind>|<project>|<tfm>|<key>, escaped. kind is vs, mtp or grp; the key is the adapter's TestCase.Id for VSTest, and the node uid for MTP. Clients treat it as a black box - server only uses it to route back to the right project.
  • VSTest id runs: rediscover just the sources the ids point at, pick out the matching TestCases, and RunTests(IEnumerable<TestCase>) (or RunTestsWithCustomTestHost when debugging). Adapters get back exactly what they produced, and there's no server-side cache to go stale. If the extra discovery pass hurts we can cache off the binary timestamp later.
  • MTP id runs: each app only gets its own uids; apps with nothing selected aren't launched at all (previously every MTP app got every uid).
  • Request: TestIds replaces TestUids, and PlatformUid is gone from the item.
    • TestIds + TestCaseFilter together -> rejected with InvalidParams. Same for malformed ids, group ids, unknown projects, and ids outside LimitToProjects - never quietly widened into a run-all.
    • Empty TestIds runs nothing.
    • Ionide expands a selected group into its leaves.
  • Filters: still VSTest syntax, and still rejected when MTP projects are in scope. MTP filtering can be its own PR imo.

Side effect: two existing bugs fell out - duplicate display names were being merged (dropping tests), and a leaf with the same name as a group was dropping the group.

Got this locally across both repos; I'll push once I've smoke tested the explorer in VS Code.

Editing a test file merged the new code locations into the explorer by
adding every test whose line had moved. Adding replaces the item of the
same id, and a test found in code carries no FSAC id and none of the
children FSAC reported under it, such as theory rows. An edited test so
lost its id and could no longer be run: it was "not found by test
discovery".

A test from code that is already in the tree now has its range updated in
place, and its children are merged the same way. Tests new to the tree are
still added and tests gone from the code are still removed.
Refresh builds only the projects it takes for test projects, and it knew
them by the VSTest packages alone. A project that opts into
Microsoft.Testing.Platform, such as one on xunit.v3 or MSTest.Sdk, need
not reference those packages, so it was never built and FSAC found no
tests in it.

FSAC does not pass on IsTestingPlatformApplication, so such a project is
now told by the Microsoft.Testing.Platform package its runner resolves,
or by the IsTestProject property FSAC does report.
@shayanhabibi

shayanhabibi commented Sep 25, 2026 •

Copy link
Copy Markdown
Author

@farlee2121 Pushed - smoke tested the explorer in VS Code against a mixed VSTest + MTP workspace and it all behaves. Plain-terms rundown of where it's at:

The id rework (what you asked for)

  • Every test gets an id from FSAC. Ionide doesn't look inside it - it just hands it back when you hit run. Same deal whether the test lives in a VSTest or an MTP project.
  • Running by id and running by filter are separate. Send ids or a filter, not both. Anything off (both at once, a garbled id, a group id, an id for a project that isn't loaded or is outside LimitToProjects, an id for the wrong platform, a filter with MTP projects in scope) gets rejected with InvalidParams rather than quietly turning into a run-all.
  • Empty id list runs nothing. Each MTP app only gets launched with its own tests; apps with nothing selected aren't started at all.
  • Tests that don't come back get flagged. If you ask for an id and no result shows up, you get a warning naming it. If the run itself falls over, you get InternalError.
  • Ionide side: picking a group runs its leaves by id. Two tests with the same name (e.g. Tests.My test and Other.My test) now show up and report separately instead of one swallowing the other.

Fixes since the last push

Did an adversarial review pass over both PRs and fixed the bigger things it turned up:

  • Editing a test file no longer breaks running it. Before, an edit swapped the explorer item for a fresh one without its FSAC id, so running a theory right after an edit failed with "not found by test discovery". Items are now updated in place and keep their id and rows.
  • Refresh now builds MTP projects. It only recognised test projects by the VSTest packages, so an xunit.v3 / MSTest-runner project never got built. It now also counts projects that pull in Microsoft.Testing.Platform. (FSAC doesn't send IsTestingPlatformApplication to the extension yet - that'd be the cleaner signal, happy to add it if you'd rather.)
  • FSAC side (details on Add Microsoft.Testing.Platform server-mode test support FsAutoComplete#1552): one broken or unbuilt MTP project no longer takes down discovery/runs for everything else, and MTP tests from frameworks other than xUnit now get a proper Namespace.Class.Method tree instead of piling up at the project root.

Still open / heads up

  • Needs an FSAC release with update FSAC, purge old mono-referencing options, update options texts #1552 before this can merge - the extension relies on the new ids.
  • MTP filtering is still finicky - own PR imo, don't want the PR to blow out (I can make a PR that is off this branch though).
  • xunit.v3 only hands console output over when the test assembly opts in with [<assembly: CaptureConsole>], so "no output" on an MTP test without that is expected.

Would appreciate another look when you get a chance 🙏

@farlee2121

Copy link
Copy Markdown
Contributor

Looking good. Just one rename suggestion and a few spots I'll want to run the regression tests on.
I haven't gotten through all of the FSAC changes yet, but looking pretty clean thus far

@shayanhabibi

Copy link
Copy Markdown
Author

Looking good. Just one rename suggestion and a few spots I'll want to run the regression tests on. I haven't gotten through all of the FSAC changes yet, but looking pretty clean thus far

Let me know; I did run against your regression library earlier in the dev which brought edge cases up. Maybe we should include them as a suite in FSAC?

@farlee2121

Copy link
Copy Markdown
Contributor

Manual Testing Results

I found a few unexpected behaviors while testing

  • MTP projects aren't getting live test explorer updates as the code changes (i.e. renaming a test)
  • NUnit with MTP wasn't getting location-based features like "go to test". I confirmed the same project has code locations in Visual Studio. I added a new MTPv2.NUnit.Tests project to the regression test repo.
  • TUnit parameterized tests aren't grouped
  • Across mtp-based test frameworks, tests with + or . in the name are being split. Confirmed that visual studio is now handling these correctly (except for NUnit), so I'd guess they're coming back from the framework with the expected parent ids.

Heads up that MTPv2.MSTest.Tests has some odd behavior. Visual Studio won't run it if <EnableMSTestRunner> is set but dotnet test won't run it if that same property isn't set.

MTP filters

Regarding filters for MTP. I think it's fine if we delay that feature.

I found this suggestion that MSTest or NUnit might have a filter converter

VSTestBridge has some very basic support for filter translation. But really only for Uid-based test filtering

The NUnit filter conversion is to their own filter system.

@farlee2121

Copy link
Copy Markdown
Contributor

@shayanhabibi Anything questions or things you want to collaborate on?

@shayanhabibi

Copy link
Copy Markdown
Author

@shayanhabibi Anything questions or things you want to collaborate on?

Hey! Just amongst some other projects, I was planning on circling back to this on Thursday if that's okay! Otherwise I can push some through tomorrow?

@farlee2121

Copy link
Copy Markdown
Contributor

No rush. Just wanted to check in and make sure you had what you needed.

@shayanhabibi

shayanhabibi commented Oct 12, 2026 •

Copy link
Copy Markdown
Author
  • MTP projects aren't getting live test explorer updates as the code changes (i.e. renaming a test)

This isn't supported by MTP. The suite has to be rebuilt. We can look at automatic rebuilds as part of opt-in strategies?

  • NUnit with MTP wasn't getting location-based features like "go to test". I confirmed the same project has code locations in Visual Studio. I added a new MTPv2.NUnit.Tests project to the regression test repo.

I couldn't figure this out, but there are two parts to this supposedly (aka according to clankers):

MTP's VSTest bridge defaults CollectSourceInformation to false. If you explicitly enable it, then we get source locations metadata. So this is a switch we need to flip, easy fix afaict.

The other is apparently an upstream issue.

  • TUnit parameterized tests aren't grouped

We don't normalize test names with brackets yet. See the next point. I can add this if you are happy with with the next point too.

  • Across mtp-based test frameworks, tests with + or . in the name are being split. Confirmed that visual studio is now handling these correctly (except for NUnit), so I'd guess they're coming back from the framework with the expected parent ids.

I might have misunderstood, I thought this is what we wanted? We're doing this as a backwards compatibility to support expecto as it is. We are manually creating a hierarchy by splitting test names.

Correct behavior would be the frameworks for MTP supply correct parent/child node relationships.

Heads up that MTPv2.MSTest.Tests has some odd behavior. Visual Studio won't run it if <EnableMSTestRunner> is set but dotnet test won't run it if that same property isn't set.

The problem was <OutputType>Exe</OutputType> was not set for the MSTest project.

MTP filters

Regarding filters for MTP. I think it's fine if we delay that feature.

I agree for this; there's other priorities in FSAC/Ionide I'd rather hit first before looking at that (such as supporting <Content> and <None> property items)


Outstanding

  • Should we infer and group TUnits parameterised tests?

@farlee2121

Copy link
Copy Markdown
Contributor
  • MTP projects aren't getting live test explorer updates as the code changes (i.e. renaming a test)

This isn't supported by MTP. The suite has to be rebuilt. We can look at automatic rebuilds as part of opt-in strategies?

Ionide's live updates are achieved through code analysis. FSAC watches for changes and sends a notification to Ionide which merges the updates. I don't think code analysis should be effected by the test runner

Ionide side

Notifications.testDetected.Invoke(fun testsForFile ->
        codeTestsDiscoveredMailbox.Post(testsForFile)
        None)
    |> unbox
    |> context.subscriptions.Add

FSAC side is AdaptiveServerState.fs > detectTests

I suspect the problem is how we detect test libraries in that code

let res = 
        if proj.OtherOptions |> Seq.exists (fun o -> o.Contains "Expecto.dll") then
          TestAdapter.getExpectoTests parseResults.ParseTree
        elif proj.OtherOptions |> Seq.exists (fun o -> o.Contains "nunit.framework.dll") then
          TestAdapter.getNUnitTest parseResults.ParseTree
        elif proj.OtherOptions |> Seq.exists (fun o -> o.Contains "xunit.assert.dll") then
          TestAdapter.getXUnitTest parseResults.ParseTree
        else
          []

We can look at automatic rebuilds as part of opt-in strategies?

The build as triggered from ionide is very slow because of ordering challenges. We can't always guarantee a solution file and there was no fable equivalent to nosln.

I've been thinking about doing something similar to Visual Studio, where they watch for build artifact updates and then re-discover tests. This would also work better with custom build processes.

NUnit with MTP wasn't getting location-based features like "go to test". I confirmed the same project has code locations in Visual Studio. I added a new MTPv2.NUnit.Tests project to the regression test repo.

I'd be ok with leaving this to a later work scope.

  • Across mtp-based test frameworks, tests with + or . in the name are being split. Confirmed that visual studio is now handling these correctly (except for NUnit), so I'd guess they're coming back from the framework with the expected parent ids.

I might have misunderstood, I thought this is what we wanted? We're doing this as a backwards compatibility to support expecto as it is. We are manually creating a hierarchy by splitting test names.

Correct behavior would be the frameworks for MTP supply correct parent/child node relationships.

Hmm. I think I can see where I miscommunicated.
My intent is that FSAC should provide the parent ids for both VSTest and MTP. MTP already provides parent ids so we can use their ids to benefit from the more reliable hierarchy. VSTest doesn't have this so we back fill it from the name inference. This way the FSAC caller doesn't have to handle requests to the same api differently based on test framework.

set but dotnet test won't run it if that same property isn't set.

The problem was <OutputType>Exe</OutputType> was not set for the MSTest project.

Good catch

@shayanhabibi

Copy link
Copy Markdown
Author
  • MTP projects aren't getting live test explorer updates as the code changes (i.e. renaming a test)
    This isn't supported by MTP. The suite has to be rebuilt. We can look at automatic rebuilds as part of opt-in strategies?

Ionide's live updates are achieved through code analysis. FSAC watches for changes and sends a notification to Ionide which merges the updates. I don't think code analysis should be effected by the test runner

MTP isn't supported by reflection (no attributes, method reflection or other mechanisms), and the test runner is the server in this case. The server has to be running for us to request the test list, it's not really something that would work without rebuilding it on demand.

If we did decide to use reflection/code analysis (which I wouldn't recommend, as it seems antithetical to the design of MTP), I would think it should be added as an enhancement afterwards.

Hmm. I think I can see where I miscommunicated. My intent is that FSAC should provide the parent ids for both VSTest and MTP. MTP already provides parent ids so we can use their ids to benefit from the more reliable hierarchy. VSTest doesn't have this so we back fill it from the name inference. This way the FSAC caller doesn't have to handle requests to the same api differently based on test framework.

So just confirming, the idea that is that we perform this hierarchical inference only for VSTest?

@farlee2121

Copy link
Copy Markdown
Contributor

MTP & Code Analysis

I should probably just fork your branch and see if I can show you what I mean / figure out the issue.

The way ionide updates the test tree while files are edited doesn't depend on a build artifact or the test runner. It depends on the F# compiler services and updates in the AST. It walks the syntax tree looking for framework-specific test syntax.

Hieararchy

Yes. We infer the hierarchy only for for VSTest.

@shayanhabibi

shayanhabibi commented Oct 13, 2026 •

Copy link
Copy Markdown
Author

Ah yeah that's fair; I would be happy to implement that as a separate PR mostly because I think we should have a standard/protocol/SDK for FSharp frameworks to use so it doesn't have to be a case by case basis. If we already have that (and I'm just oblivious 🤣) point me in the direction and I'll get it done!

As for the 3-4 test frameworks we have no control over, I get that we'll need to do it on a case by case basis.

So I can start by doing the 3-4 non FSharp ones (mstest, xunit etc), but I'd want to leave expecto and establish a contract so that it's less maintenance work on Ionides side

@shayanhabibi

Copy link
Copy Markdown
Author

Actually there's already a contract, the MTP machinery itself 😅

I might need some more time to figure out how to elegantly do this

@farlee2121

farlee2121 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

The benefit of Ionide's code analysis approach is that it can update immediately as the code is edited.
The downsides include

  • That immediacy excludes options like depending on MTP or VSTest
  • Therefore, it can get out of sync with the test framework's understanding of the tests

But, we're already merging the code analysis view with the test platform view.

My thought is that we could have reactive updates from both sources, just at different speeds.
The code analysis is immediate and already in place.
The test platforms will only be able to discover changes when the code is built. So, we could watch for updates to the build artifacts and reactively rediscover tests. This should be fairly fast since there's no need to rebuild. This approach also has the benefit that it works uniformly across all libraries with vstest support and similarly for all libraries with MTP support.

Thoughts?

@shayanhabibi

shayanhabibi commented Oct 13, 2026 •

Copy link
Copy Markdown
Author

It's a bit edgier with MTP.

  • The tests are created dynamically, non-trivially, requiring side effects, or some other mechanism
    • IE: consumes a directory to create test groups based on expected vs actual compiled outputs for something like fable
    • The 'server' is built once and does not need to be rebuilt to provide a different result on request for test discovery
  • Theoretically, the test suite can be provided by an executable independently which constructs the suite from dynamic input data at time of execution. Very edge case. Different to the above in that there is no project file to begin with.

I know there is a lot of overlap here; however, I think, with MTP, that there is a much higher chance (at least with how I envision using MTP) that the actual hierarchy we receive from a built test suite is different to the predicted test suite from code analysis.

In the case this occurs: We should cease providing code analysis updates for that MTP project/exe.
This would be cached in a .ionide file. We can provide it a life-time.

When I think of the advantages of MTP, I think of areas that code analysis and reflection are not possible to give me the correct test hierarchy. So long as ionide doesn't fight me on this point, I don't have any problem.

In this sense, I am happy to provide support for code-analysis with MTP, but only to a minimal working degree. Mostly because my excitement for MTP comes from the areas that code-analysis won't be able to help me (without a lot of assumptions).

With MTP, there's better support for doing something like: creating tests by consuming .fsx files with fsi (with fsi as a long running process on the 'server').

Edit: I think it's hard for me to communicate the nuance in my mind regarding hesitancy for trying to maintain code-analysis at this stage for MTP. It might be easier to discuss this over a discord voice call for me. Let me know if you want to chat about it on the fsharp server, perhaps other people can give input too.

@farlee2121

Copy link
Copy Markdown
Contributor

I'd be open chatting on the F# slack

@shayanhabibi

Copy link
Copy Markdown
Author

I'd be open chatting on the F# slack

I've had trouble previously getting authorisation to join the F# slack, but I'll give it another try

@farlee2121

Copy link
Copy Markdown
Contributor

@shayanhabibi Any hang ups I can help with?

@farlee2121

Copy link
Copy Markdown
Contributor

@shayanhabibi Do you still intend to work on this?
It's very close to ready

@shayanhabibi

Copy link
Copy Markdown
Author

@shayanhabibi Do you still intend to work on this?

It's very close to ready

Hey, regarding the test names responding to edits, it IS possible; I made an incremental declaration based FSAC https://github.com/ranvier-org/ranvier-fsharp which tracks changes without having to rebuild dlls.

On top of this, I built an incremental hot module reloader. It might be possible to use something like this to keep MTP projects hot.

But those will need FCS changes before we can do PRs on FSAC.

Otherwise, I still think code analysis isn't the right move.

So should we keep that for later?

@farlee2121

Copy link
Copy Markdown
Contributor

No further action is needed on the code analysis front. The PR I made fixed the issue xUnit was having. MSTest didn't have code analysis updates to start with.

The only outstanding issue is building the MTP test hierarchy using the ids from MTP rather than inferring the hierarchy like we need to for VSTest

A node of the tree FSAC reports was labelled by the last `.` or `+`
segment of its full name. FSAC no longer splits the name of a testing
platform test, so one holding a separator was shown by its tail alone.
@shayanhabibi

Copy link
Copy Markdown
Author

@farlee2121 Pushed to both PRs. The MTP tree is now built from what MTP reports, and name inference is VSTest-only.

What MTP actually sends

I dumped discovery for xUnit v3, NUnit and TUnit from the regression repo. None of them report a parent uid; every node comes back with no parent. What they do report is the type and method as separate fields (location.type, location.method). So in practice "MTP's ids" means:

  • a node with a reported parent uid hangs off that parent
  • otherwise it hangs off the type the platform names for it
  • a node with neither sits at the project root

The test's own name is never split. Contains.separator+characters is one test under XUnitTests, and a theory row like theory(x: 1.5) stays one test.

The type name is still broken on . and + to give the namespace and nested-type groups, so nested modules nest as before.

Changes

Tested

  • FSAC test-explorer tests: 176 passed, including a new one that discovers a real xUnit v3 MTP app with a test named Contains.separator+characters.
  • FSAC LSP TestExplorerTests: 74 passed on net10.0 only.
  • Ionide compiles; I have not re-run the regression workspace in VS Code after this change, so a pass over it on your side would be welcome.

Still open

  • NUnit reports the wrong type for names containing separators (MTPv2NUnitTests.Contains seperators + and as the type), so that one still nests wrongly. That matches what you saw in Visual Studio.
  • TUnit parameterised cases hang off their type and are not grouped under their method. I left that out of this PR.
  • A framework that reports neither parents nor a type would now show flat rather than being grouped by name.

…mode

# Conflicts:
#	src/Components/TestExplorer.fs
#	src/Core/LanguageService.fs
@farlee2121

farlee2121 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

I test it out and it's looking great overall!

Two easy items to address

  1. I made a PR with some readability refactors
  2. I added question comment on labelUnder

One less clear item to address:

I made a branch of the regression repo to test out experimental expecto MTP support so we could get a good example of hierarchies made using explicit parent ids. It almost works as expected, but it complains about missing results if you run any grouping of tests.

@farlee2121

Copy link
Copy Markdown
Contributor

I suspect the missing test results warning error is on the ionide side of the fence

@farlee2121

farlee2121 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

I dug into the missing results problem. Turns out it was a misunderstanding in Expecto.MTP. If you set DiscoveredTestNodeStateProperty, MTP will consider the node an action not a group. No change is needed in FSAC/ionide for this issue.

I did notice a two more problem though.

  • In the MTPv2.Expecto.Tests, "Contains+separator.characters" is not displaying correctly. It's nested in the right spot, but it only shows "characters" as the test name
  • Live updates are creating new test items and leaving the old names (expecto-only)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants