Missing end-of-file line feed duplicates paragraph

[gorango]

Changing line 147 in test/index.js to "minify" the incoming html with the following line:

var file = vfile(fs.readFileSync(input, 'utf8').replace(/\n*^\s*/gm, ''))

This particular change results in 14 failures (I recognize that a proper minification might avoid a few of these failures).

I wonder if this issue should be better addressed in the rehype-parse package?

[ChristianMurphy]

@gorango can you provide an example of minified HTML that is causing an issue?

[wooorm]

The regex /\n*^\s*/gm replaces zero or more line feeds, followed by a line feed (important), followed by zero or more white space, with an empty string.
That means that any line feed is removed.

HTML collapses whitespace. That means that a line feed is seen as a single space character.
There’s a difference between nothing and a space.

Sounds like you‘re falling for an XY problem. As Christian mentioned. We need more info.

[gorango]

Agreed 🤦‍♂️. I also didn't take into account the position attributes and WhiteSpaceNodes that are affected by the minification. But just before returning to close this issue, I did spot a small anomaly...

Running rm test/fixtures/**/*.json && node test to generate new output files, I noticed the following results changed unexpectedly:

• test/fixtures/overlap-explicit
• test/fixtures/source

Both outputs contained clones of the last sentence in each.

Adding any space at the end of the vfile.contents prevents the duplicates.

[gorango]

To replicate this, you can replace test/fixtures/source/index.html, with:

<p>This is <span data-nlcst="source">marked</span>.</p><p data-nlcst="source">Completely marked.</p>

You should get similar results if you minify the overlap-explicit/index.html file.

[wooorm]

@gorango I tested these two out and:

• rm test/fixtures/**/*.json && node test did not produce different output.json files for me.
• Replacing test/fixtures/source/index.html with that document, removing its output.json file, and running node test, only changes positional information for me.

Removing the EOF EOL (the last \n before the end of the file) in the source fixture did indeed duplicate the last paragraph.