Repository navigation
Consume FSAC test hierarchy and enable MTP tests - #2180
shayanhabibi wants to merge 9 commits into
Conversation
FsAutoComplete now reports grouping nodes and parent ids, and normalises parameterised test names itself. Rebuilding the tree here by splitting fully-qualified names would disagree with the server as soon as a project runs under Microsoft.Testing Platform, whose nodes carry parent ids the names do not encode. TestItemDTO carries Id, ParentId and IsLeaf, and discovery builds its hierarchy by following those links. getFullname_withNestedParamTests is gone: the server's FullName already distinguishes theory cases, so applying the same rule here appended the case parameters twice. Name inference stays for the TRX and AST paths, which report names alone.
Exposes the FSAC flag that routes projects setting IsTestingPlatformApplication through the Microsoft.Testing.Platform server protocol. Off by default.
Microsoft.Testing.Platform runs a test named by the opaque uid it was discovered under, rather than by a filter expression over its name, so a test carries that uid from discovery through to the run. A grouping node holds no uid of its own, so selecting one is read as the runnable tests beneath it. A test discovered through VSTest carries no uid and is still named by the filter expression, which is sent alongside.
shayanhabibi
left a comment
There was a problem hiding this comment.
Open question: Should we discriminate between DTOs targeting VSTest/MTP instead of optional fields
|
See the other PR for the heart of the implementation. I do want to pursue some cleanup of Ionide settings (ie grouping them) and other facets, but this is the foundational work. Open Question: Should I continue adding these other facets, and we cherry pick them into separate PRs later/just merge them all later? |
Unique idsI haven't gotten to look closely at this, but I see one item that concerns me
This sounds like VSTest and MTP behave in different and incompatible ways through the same api, causing a leaky abstraction that requires consumers to understand how certain calls will behave based on implicit context. Is there a way we can bridge the gap? Have unique identifiers for both MTP and VSTest? I notice VsTestConsoleWrapper.RunTests can run a sequence of TestCases, and a TestCase can be constructed with FullyQualifiedName, ExecutorUri, and Source. To my understanding, that combination should be unique. Perhaps we can package these into a unique identifier. Then both VSTest and MTP could run specific tests by an "id". The id is provided by the server and interpreted by the server and treated as a black box by the client. Consumers could get themselves into trouble by trying to parse out id components, but that'd be unsupported shenanigans requiring some reverse engineering. Unique Ids and Test FiltersRelated. What happens if both test ids and a filter are specified in the run request? Union types don't properly exist over process boundaries. We could throw an error if both are specified. In any case, it's a behavior we should think about. |
|
Things get really finicky as we start to treat one field as another. Strictly speaking, display names are not unique ids. The default filter works on display names, not unique ids. There is a separate filter argument in MTP for specifying unique ids. For us in FSharp, where our tests are constructed as values instead of associated with types/members, we essentially treat them the same. So I don't disagree with you at all - yeah I'll backport VSTest API to fill fields that are required in MTP so we can remove fields that are optional only because one API uses it - however, we should be careful when communicating how filters work (as you point out). I don't actually know if MTP allows you to specify uid filters with normal name filters. Something I'll look into. But - we should be able to request both. I'll report back after review |
|
I think I didn't correctly communicate my full idea. I didn't mean to suggest stuffing unique ids into the filter or filters into unique ids. I would expect filter and id-based runs to be distinct. I'm suggesting that both filters and id-based runs are available for both MTP and VSTest, and work consistently between the two. All tests could be returned from FSAC with a unique id. FSAC could service run requests by these ids using any underlying platform.
UPDATE: I looked at the VSTest code for TestCase, and they're actually constructing a Guid based on the three TestCase members for internal use as a unique id. So I think we'd be pretty safe assuming the combo of FullyQualifiedName, ExecutorUri, and Source can act as an id. // Part of TestCase
private Guid GetTestId()
{
string text = Source;
try {
text = Path.GetFileName(text);
} catch {
}
string text2 = ExecutorUri?.ToString() + text;
text2 += GetFullyQualifiedName();
return EqtHash.GuidFromString(text2);
} |
FsAutoComplete now gives every test an opaque id that routes it to its project, target framework and platform, and runs a selection named by those ids. A test explorer item keeps the id FSAC reported for it, and a run sends the ids of the tests under the selection and nothing else: FSAC rejects ids sent together with a filter expression, and works out the projects to run from the ids themselves. The filter expression stays with the dotnet CLI path. A selected test FSAC did not report, such as one found only in code, has no id. Discovery runs once to find it, and a test still without one is marked errored instead of widening the run. Results and started notifications are matched to the explorer by that id, so tests that share a name within a project, such as theory rows run on Microsoft.Testing.Platform, no longer collide. The name-based id is only a fallback for items and results that carry no id. PlatformUid is gone from TestItemDTO, and TestUids is now TestIds.
An explorer item is still named by its project and full name, and FSAC does not keep full names unique. A grouping can share its name with a test, and on Microsoft.Testing.Platform, where the full name is the display name, several tests can share one. Siblings that share a name got one explorer id, so the item added last replaced the others: a grouping hid the test of its name, and only one of several same-named tests stayed in the tree. The previous commit matched results by FSAC's id, but the tree still collapsed these tests, so they did not stay apart as it said. A grouping is now shown as one node with the test of its name, and that test runs along with the tests under it. Each further test of a name gets a numbered name. Selections and missing results are told apart by FSAC's id where there is one. Before, tests of one name under different parents were deduplicated to one, so the others were never sent to FSAC.
|
@farlee2121 Gotcha, that makes sense - I've reworked it so every test FSAC hands out gets an opaque id, and id runs and filter runs are separate paths that behave the same on VSTest and MTP. The VSTest tripleI tried the
xUnit and MSTest set their own What I've got now
Side effect: two existing bugs fell out - duplicate display names were being merged (dropping tests), and a leaf with the same name as a group was dropping the group. Got this locally across both repos; I'll push once I've smoke tested the explorer in VS Code. |
Editing a test file merged the new code locations into the explorer by adding every test whose line had moved. Adding replaces the item of the same id, and a test found in code carries no FSAC id and none of the children FSAC reported under it, such as theory rows. An edited test so lost its id and could no longer be run: it was "not found by test discovery". A test from code that is already in the tree now has its range updated in place, and its children are merged the same way. Tests new to the tree are still added and tests gone from the code are still removed.
Refresh builds only the projects it takes for test projects, and it knew them by the VSTest packages alone. A project that opts into Microsoft.Testing.Platform, such as one on xunit.v3 or MSTest.Sdk, need not reference those packages, so it was never built and FSAC found no tests in it. FSAC does not pass on IsTestingPlatformApplication, so such a project is now told by the Microsoft.Testing.Platform package its runner resolves, or by the IsTestProject property FSAC does report.
|
@farlee2121 Pushed - smoke tested the explorer in VS Code against a mixed VSTest + MTP workspace and it all behaves. Plain-terms rundown of where it's at: The id rework (what you asked for)
Fixes since the last pushDid an adversarial review pass over both PRs and fixed the bigger things it turned up:
Still open / heads up
Would appreciate another look when you get a chance 🙏 |
|
Looking good. Just one rename suggestion and a few spots I'll want to run the regression tests on. |
Let me know; I did run against your regression library earlier in the dev which brought edge cases up. Maybe we should include them as a suite in FSAC? |
Manual Testing ResultsI found a few unexpected behaviors while testing
Heads up that MTPv2.MSTest.Tests has some odd behavior. Visual Studio won't run it if MTP filtersRegarding filters for MTP. I think it's fine if we delay that feature. I found this suggestion that MSTest or NUnit might have a filter converter VSTestBridge has some very basic support for filter translation. But really only for Uid-based test filtering The NUnit filter conversion is to their own filter system. |
|
@shayanhabibi Anything questions or things you want to collaborate on? |
Hey! Just amongst some other projects, I was planning on circling back to this on Thursday if that's okay! Otherwise I can push some through tomorrow? |
|
No rush. Just wanted to check in and make sure you had what you needed. |
This isn't supported by MTP. The suite has to be rebuilt. We can look at automatic rebuilds as part of opt-in strategies?
I couldn't figure this out, but there are two parts to this supposedly (aka according to clankers): MTP's VSTest bridge defaults The other is apparently an upstream issue.
We don't normalize test names with brackets yet. See the next point. I can add this if you are happy with with the next point too.
I might have misunderstood, I thought this is what we wanted? We're doing this as a backwards compatibility to support expecto as it is. We are manually creating a hierarchy by splitting test names. Correct behavior would be the frameworks for MTP supply correct parent/child node relationships.
The problem was
I agree for this; there's other priorities in FSAC/Ionide I'd rather hit first before looking at that (such as supporting Outstanding
|
Ionide's live updates are achieved through code analysis. FSAC watches for changes and sends a notification to Ionide which merges the updates. I don't think code analysis should be effected by the test runner Ionide side Notifications.testDetected.Invoke(fun testsForFile ->
codeTestsDiscoveredMailbox.Post(testsForFile)
None)
|> unbox
|> context.subscriptions.AddFSAC side is AdaptiveServerState.fs > detectTests I suspect the problem is how we detect test libraries in that code let res =
if proj.OtherOptions |> Seq.exists (fun o -> o.Contains "Expecto.dll") then
TestAdapter.getExpectoTests parseResults.ParseTree
elif proj.OtherOptions |> Seq.exists (fun o -> o.Contains "nunit.framework.dll") then
TestAdapter.getNUnitTest parseResults.ParseTree
elif proj.OtherOptions |> Seq.exists (fun o -> o.Contains "xunit.assert.dll") then
TestAdapter.getXUnitTest parseResults.ParseTree
else
[]
The build as triggered from ionide is very slow because of ordering challenges. We can't always guarantee a solution file and there was no fable equivalent to nosln. I've been thinking about doing something similar to Visual Studio, where they watch for build artifact updates and then re-discover tests. This would also work better with custom build processes.
I'd be ok with leaving this to a later work scope.
Hmm. I think I can see where I miscommunicated.
Good catch |
MTP isn't supported by reflection (no attributes, method reflection or other mechanisms), and the test runner is the server in this case. The server has to be running for us to request the test list, it's not really something that would work without rebuilding it on demand. If we did decide to use reflection/code analysis (which I wouldn't recommend, as it seems antithetical to the design of MTP), I would think it should be added as an enhancement afterwards.
So just confirming, the idea that is that we perform this hierarchical inference only for VSTest? |
MTP & Code AnalysisI should probably just fork your branch and see if I can show you what I mean / figure out the issue. The way ionide updates the test tree while files are edited doesn't depend on a build artifact or the test runner. It depends on the F# compiler services and updates in the AST. It walks the syntax tree looking for framework-specific test syntax. HieararchyYes. We infer the hierarchy only for for VSTest. |
|
Ah yeah that's fair; I would be happy to implement that as a separate PR mostly because I think we should have a standard/protocol/SDK for FSharp frameworks to use so it doesn't have to be a case by case basis. If we already have that (and I'm just oblivious 🤣) point me in the direction and I'll get it done! As for the 3-4 test frameworks we have no control over, I get that we'll need to do it on a case by case basis. So I can start by doing the 3-4 non FSharp ones (mstest, xunit etc), but I'd want to leave expecto and establish a contract so that it's less maintenance work on Ionides side |
|
Actually there's already a contract, the MTP machinery itself 😅 I might need some more time to figure out how to elegantly do this |
|
The benefit of Ionide's code analysis approach is that it can update immediately as the code is edited.
But, we're already merging the code analysis view with the test platform view. My thought is that we could have reactive updates from both sources, just at different speeds. Thoughts? |
|
It's a bit edgier with MTP.
I know there is a lot of overlap here; however, I think, with MTP, that there is a much higher chance (at least with how I envision using MTP) that the actual hierarchy we receive from a built test suite is different to the predicted test suite from code analysis. In the case this occurs: We should cease providing code analysis updates for that MTP project/exe. When I think of the advantages of MTP, I think of areas that code analysis and reflection are not possible to give me the correct test hierarchy. So long as ionide doesn't fight me on this point, I don't have any problem. In this sense, I am happy to provide support for code-analysis with MTP, but only to a minimal working degree. Mostly because my excitement for MTP comes from the areas that code-analysis won't be able to help me (without a lot of assumptions). With MTP, there's better support for doing something like: creating tests by consuming .fsx files with fsi (with fsi as a long running process on the 'server'). Edit: I think it's hard for me to communicate the nuance in my mind regarding hesitancy for trying to maintain code-analysis at this stage for MTP. It might be easier to discuss this over a discord voice call for me. Let me know if you want to chat about it on the fsharp server, perhaps other people can give input too. |
|
I'd be open chatting on the F# slack |
I've had trouble previously getting authorisation to join the F# slack, but I'll give it another try |
|
@shayanhabibi Any hang ups I can help with? |
|
@shayanhabibi Do you still intend to work on this? |
Hey, regarding the test names responding to edits, it IS possible; I made an incremental declaration based FSAC https://github.com/ranvier-org/ranvier-fsharp which tracks changes without having to rebuild dlls. On top of this, I built an incremental hot module reloader. It might be possible to use something like this to keep MTP projects hot. But those will need FCS changes before we can do PRs on FSAC. Otherwise, I still think code analysis isn't the right move. So should we keep that for later? |
|
No further action is needed on the code analysis front. The PR I made fixed the issue xUnit was having. MSTest didn't have code analysis updates to start with. The only outstanding issue is building the MTP test hierarchy using the ids from MTP rather than inferring the hierarchy like we need to for VSTest |
A node of the tree FSAC reports was labelled by the last `.` or `+` segment of its full name. FSAC no longer splits the name of a testing platform test, so one holding a separator was shown by its tail alone.
|
@farlee2121 Pushed to both PRs. The MTP tree is now built from what MTP reports, and name inference is VSTest-only. What MTP actually sends I dumped discovery for xUnit v3, NUnit and TUnit from the regression repo. None of them report a parent uid; every node comes back with no parent. What they do report is the type and method as separate fields (
The test's own name is never split. The type name is still broken on Changes
Tested
Still open
|
…mode # Conflicts: # src/Components/TestExplorer.fs # src/Core/LanguageService.fs
|
I test it out and it's looking great overall! Two easy items to address One less clear item to address: I made a branch of the regression repo to test out experimental expecto MTP support so we could get a good example of hierarchies made using explicit parent ids. It almost works as expected, but it complains about missing results if you run any grouping of tests. |
|
I suspect the missing test results warning error is on the ionide side of the fence |
|
I dug into the missing results problem. Turns out it was a misunderstanding in Expecto.MTP. If you set DiscoveredTestNodeStateProperty, MTP will consider the node an action not a group. No change is needed in FSAC/ionide for this issue. I did notice a two more problem though.
|
Problem
Ionide's test explorer infers grouping nodes by splitting test names, and selects tests to run with VSTest filter expressions. Neither works for Microsoft.Testing.Platform (MTP) projects: MTP test names are display names that need not be dotted paths, and MTP has no VSTest filter.
Change
Depends on ionide/FsAutoComplete#1552. Tracks #2069.
FSharp.enableTestingPlatform, off by default, and passes it to FSAC. With it off, behaviour is unchanged.How MTP tests are grouped is decided in FSAC: by the parent MTP reports, otherwise by the type MTP names for the test.
Validation
On the branch after merging current
main:dotnet fsi build.fsx(format check, Fable build, VSIX package) passes.dotnet fsi build.fsx -p Test(VS Code smoke test) passes.TestExplorerTests74 pass, LSP test-detection tests 12 pass. The rest of the FSAC LSP suite was not run.On an earlier revision of this branch:
Not yet checked:
Known limits