ng test: Collect total code coverage in multi project repository

[dherges]

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request

Area

- [x] devkit
- [ ] schematics

Versions

Angular CLI: 6.0.8
Node: 8.9.1
OS: win32 x64
Angular: 5.2.11
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.6.8
@angular-devkit/schematics        0.6.8
@angular/cdk                      5.2.5
@angular/cli                      6.0.8
@ngtools/json-schema              1.1.0
@ngtools/webpack                  6.0.8
@schematics/angular               0.6.1
@schematics/update                0.6.8
ng-packagr                        3.0.2
rxjs                              6.2.1
typescript                        2.6.2
webpack                           4.8.3

Repro steps

Multi project workspace, run ng test to test all projects.

The log given by the failure

Produced coverage report in coverage/lcov.info only contains results for the last run code coverage.

HTML reports are stored per project in coverage/<project>.

Desired functionality

Write indivudual lcovs per project, e.g, coverage/<project>/lcov.info, allowing users to get a total code coverage for the whole project.

Alternative would be functionality similar to istanbul-combine. Write coverage.json for each project and write a combined coverage report and end of test runs.

Mention any other details that might be useful

Should eht changes be made in the builder or in the schematic?

https://github.com/angular/angular-cli/blob/master/packages/schematics/angular/library/files/__projectRoot__/karma.conf.js#L19

https://github.com/angular/angular-cli/blob/master/packages/schematics/angular/application/files/root/karma.conf.js#L19

[dtychshenko]

I've ran into a similar issue today. Our Angular workspace has one app and four libraries, each producing their own coverage results. I wanted a combined HTML report after all tests for all projects have run. I'm not sure if there's a better way, but so far I ended up with the following setup that seems to be working so far:

1. karma.conf.js file in each project is configured to report to its own project folder and produce a json report:

    coverageIstanbulReporter: {
      dir: require('path').join(__dirname, '../../coverage/libraries/<project>'),
      reports: ['json'],
      fixWebpackSourcePaths: true
    },

2. Each project has an NPM script configured to run tests with code coverage:

    "cover:library:<project>": "ng test <project> --code-coverage --watch=false",

3. There is a high level NPM script to run all tests for all projects in parallel using npm-run-all library (we're on Windows here):

    "cover": "run-p cover:**",

4. There is a post NPM script to combine the results of all test runs into a single JSON report and a small script to produce the actual reports that I want from the combined JSON:

    "postcover": "istanbul report json && node combine-coverage.js",

5. The combine-coverage.js file looks like this:

const createReporter = require('istanbul-api').createReporter;
const istanbulCoverage = require('istanbul-lib-coverage');
const coverage = require('./coverage/coverage-final.json');

const map = istanbulCoverage.createCoverageMap();
Object.keys(coverage).forEach(filename => map.addFileCoverage(coverage[filename]));

const reporter = createReporter();
reporter.addAll(['html', 'lcovonly', 'text-summary']);
reporter.write(map);

This way, when I execute npm run cover, I have tests running for all projects in parallel and get a unified coverage report.

The last step was necessary in order to produce a newer-style report with correct source paths and styles. At first, I was just running istanbul report html as a post command, but that is using an older Istanbul CLI and uses an old built-in HTML reporter which was producing a report with absolute source paths and broken styles. Going through the istanbul-api in the script makes use of the latest istanbul-reports that produce the correct HTML output.

Hope this helps

[Annie-Huang]

What is content in the coverage-final.json file? @dtychshenko

[dtychshenko]

What is content in the coverage-final.json file? @dtychshenko

There are multiple coverage-final.json files. Here's how it works.

coverage-final.json file is produced by Istanbul and simply contains code coverage details in JSON format. In the above configuration, a coverage-final.json file will be generated for each individual project because we've requested reports in JSON format (reports: ['json']) in the Karma config of step 1.

In step 4, the first command istanbul report json will read each of the individual coverage-final.json files for each of the projects and will produce a single combined coverage-final.json report file.

The script in step 5 reads the last combined JSON report to produce 'html', 'lcovonly', 'text-summary' reports out of it.

Note that your mileage may vary with different library versions, but this setup currently works for us with the following:

+-- @angular-devkit/build-angular@0.11.0
| `-- istanbul@0.4.5
+-- @angular/cli@7.1.4
+-- istanbul-api@1.3.1
| +-- istanbul-lib-coverage@1.2.1
| +-- istanbul-lib-instrument@1.10.2
| | `-- istanbul-lib-coverage@1.2.1  deduped
| +-- istanbul-lib-report@1.1.5
| | `-- istanbul-lib-coverage@1.2.1  deduped
| `-- istanbul-lib-source-maps@1.2.6
|   `-- istanbul-lib-coverage@1.2.1  deduped
+-- karma@2.0.5
`-- karma-coverage-istanbul-reporter@1.4.3
  `-- istanbul-api@1.3.1  deduped

I see that @angular-devkit/build-angular in its latest versions has removed the deprecated istanbul@0.4.5 dependency, so the istanbul report json command might no longer work for you, but it's not a big problem.

All you have to do in your combine-coverage.js script is instead of requiring the single combined JSON report file ( const coverage = require('./coverage/coverage-final.json');), read each of the individual coverage-final.json files in each of your packages.

[charliemc]

Thank you @dtychshenko for sharing your approach, it really helped here!

Regarding the removal of the istanbul@0.4.5 dependency in @angular-devkit/build-angular I wanted to share in case somebody else finds it useful that I was able to generate a full report straightaway using istanbul-combine. All single reports generated previously will be combined and merged by istanbul-combine into a single one.

So, in this case, steps 4 and 5 could be replaced by a script of this kind:

"postcover": "istanbul-combine -d coverage/reports -r lcov -r html /coverage/libraries/*.json"

[matses]

I was able to generate a full report straightaway using istanbul-combine. All single reports generated previously will be combined and merged by istanbul-combine into a single one.

@charliemc Could you post your package.json & karma.conf.js ?

i get some known issue with istanbul-combine :

gotwarlost/istanbul#817

Update : fixed with --coverage --source-map=true

[wkit23]

For those who still needs this, I manage to get the report to work with the following script. I used karma-coverage to return json format which then the script will merge and combine into other reporting format.

Would be great if we can do this directly from the cli.

var glob = require('glob');
var libCoverage = require('istanbul-lib-coverage');
var libReport = require('istanbul-lib-report');
var reports = require('istanbul-reports');

var map = libCoverage.createCoverageMap();

// Gets all the coverage files and merge them
var files = glob.sync('./tests/coverage/**/coverage-final.json');
files.forEach(f => map.merge(require(f)));

// Creates a context for report generation
var context = libReport.createContext({
    dir: './tests/coverage',
    coverageMap: map
});

var reportTypes = ['html', 'lcovonly', 'text-summary', 'cobertura'];
reportTypes.forEach(t => {
    var report = reports.create(t);
    report.execute(context);
});