Faster CC

[nohwnd]

• Move CC setup after discovery so we don't slow it down
• Look for a faster way to do CC, that still does not modify the scripts
• Investigate providing a custom debugger to use the tracing for CC
• Investigate more light weight breakpoints that have single hit
• revisit the loops

[Glober777]

Hey @nohwnd, I think that one of the possible ways to make Code Coverage run faster is to be able to configure which ones of my tests do not contribute to the Code Coverage at all and can be processed quicker (by bypassing all the Code Coverage related overhead).

For example, let's say I have a project folder for some custom PowerShell Module, which apart from the module code also has a bunch of Pester tests (which is what any properly developed PS Module should always have :)). While there could be tests for the module code, there might also be tests for things like: Script Analyzer rules, help articles (about_MyModuleName and public cmdlets), whether the module folder contains other important content, etc. Clearly, there is no point in running these tests through the Code Coverage routine regardless of how optimized it is.

If I could tell Pester which tests could bypass code coverage, I could save a good portion of my build time. The way I see it is by settings some tag on Describe, Context and/or It blocks, and configure it in Pester through, say, PesterConfiguration.CodeCoverage.ExcludeTag (or something of that sort). It might also be handy to be able to exclude the whole .Tests.ps1 file/path from Code Coverage (i.e. through PesterConfiguration.CodeCoverage.ExcludePath).

[nohwnd]

@Glober777 I think we are excluding Tests.ps1 all files from the analysis by default, unless you provide a path to a Tests.ps1 file directly, then it will be calculated even for that file.

I quickly looked at CC and the biggest time is spent in setting up the breakpoints which seems to be a PowerShell limitation that takes way more time to add a single breakpoint when there already are many breakpionts.

But most of the time is spent in the analysis after the code has run, there I did not look yet to see if I can identify some basic perf oversights.

[Glober777]

@nohwnd, I think I may have confused you a bit when I mentioned the ability to exclude the whole .Tests.ps1 file... I don't need code coverage for any of my .Tests.ps1 files themselves. What I actually meant was to exclude all the tests that are defined in certain .Tests.ps1 files from running through the CC subroutine. So that Pester wouldn't have to bother setting up a bunch of breakpoints on my code, when I'm just running Script Analyzer or testing whether the help is there or not.

While I could run Pester twice - once with the code coverage just for the functionality tests and second time for the other stuff (that doesn't need affect code coverage), but that will result in having two separate test results, which is what I'd like to avoid.

[nohwnd]

@Glober777 Okay now I understand. But I don't know how to do that. How the CC works is that it sets up breakpoints in all files that should be measured by CC and then it runs the tests as normal. There is nothing special done in this test running part apart from running against code that has a ton of breakpoints. There is some impact on the speed of execution, but it is fairly minimal, at least compared to the rest of the CC process. Then when all running is done, CC will look through the list of breakpoints and check which of them were hit and will calculate coverage from that.

The setup and the coverage calculation are the very slow parts. The run itself is not, and there is not much we can do in order to exclude some tests, apart from maybe disabling all the breakpoints when you run a certain file, but I expect the time saving there would be outweighted by the time it takes to disable and enable the breakpoints, and the effort that it takes to exclude some of the test files from the run. But I might try it at some point anyway. Thanks for the tip :)

[nohwnd]

Did all of the stuff above. PR for improved debugger performace is pending here PowerShell/PowerShell#14953.

There is another one for the profiler which was still not merged, but if the debugger PR is merged there is little benefit to the profiler, at least if I don't want to rewrite how code coverage is done. Lightweight breakpoints are implemented. And the rest of the code takes not so much time. There are some improvements possible, but nothing that would have such huge impact as the debugger fix.