TS: do more work in parallel #479
Conversation
|
@xiemaisi it will most likely conflict with the parallel extraction PR. I'm happy to withhold this until that lands, as long as it gets in before feature freeze. |
asger-semmle
force-pushed
the
typescript-extractor-perf1
branch
from
a0021f3 to
b5d3dd5
Compare
| extractorState.getTypeScriptParser().prepareFiles(files); | ||
| for (File f : files) { | ||
| Path path = f.toPath(); | ||
| if (extractedFiles.add(path)) { |
There was a problem hiding this comment.
If we skip a file here (because it has already been extracted), won't the TypeScript parser throw an error on the next file we request an AST for?
There was a problem hiding this comment.
Good point. It can't happen, but it's not very obvious at the moment.
The list of files given here has been built to only contain unextracted files, so the return value of add really doesn't need to be checked. I'll try to make this a bit more clear.
There was a problem hiding this comment.
Oh wait, there was a huge bug in the last commit here.
There was a problem hiding this comment.
Turns out the attempt to get AutoBuildTest working actually introduced a bug. The extracted file was added twice, and thereby skipped in the extraction step.
We'll obviously need some better testing for this file.
There was a problem hiding this comment.
I've put up an internal PR that fixes the test that should catch this.
I've confirmed that the test fails for the previous commit, and passes after 84c1ba0.
…-notes Add missing change notes
This PR speeds up TypeScript extraction by making the Java process and Node.js process do more work in parallel.
When the Node.js process has sent an AST, it immediately starts to work on the next AST while the Java process is processing the first one. In order for the Node.js process knows which file to work on next, a list of files to work on is sent in advance using the new
prepare-filescommand.In principle the Node.js process could prepare more than one file in advance, but then we risk being busy when the Java process is ready to receive the next file, delaying overall extraction. So it only looks one file ahead.
Extraction-time benchmarks:
The full-mode extraction for TypeScript itself is a bit of an outlier, but I'll look into that separately.