Free the npm package from third party dependencies

And support Linux ARM and macOS ARM.

Author: lydell

installers/npm/.gitignore

@@ -1 +1,3 @@
 node_modules/
+packages/*/elm
+packages/*/elm.exe

installers/npm/PUBLISHING.md

@@ -2,22 +2,95 @@
 
 Here's how to update the `npm` installer.
 
+## 0. Overview
+
+- There is one _main npm package_ called `elm`.
+- Then there is one _binary npm package_ for each platform, called for example `@evancz/elm_darwin_arm64`.
+
+The binary packages declare which OS and CPU they are compatible with. For example:
+
+```json
+  "os": [ "darwin" ],
+  "cpu": [ "arm64" ]
+```
+
+The main npm package depend on the binary packages via [optional dependencies](https://docs.npmjs.com/cli/v9/configuring-npm/package-json#optionaldependencies):
+
+```json
+    "@evancz/elm_darwin_arm64": "0.19.1-0",
+    "@evancz/elm_darwin_x64": "0.19.1-0",
+    "@evancz/elm_linux_arm64": "0.19.1-0",
+    ...
+```
+
+When installing, `npm` fetches the metadata for all the optional dependencies and only installs the one with a matching OS and CPU. If none of them match, `npm` still considers the install successful. However, the main npm package contains an install script that gives a helpful error.
+
 
 ## 1. GitHub Release
 
 Create a [GitHub Release](https://github.com/elm/compiler/releases) with the following files:
 
 1. `binary-for-mac-64-bit.gz`
-2. `binary-for-windows-64-bit.gz`
+2. `binary-for-mac-arm-64-bit.gz`
 3. `binary-for-linux-64-bit.gz`
+4. `binary-for-linux-arm-64-bit.gz`
+5. `binary-for-windows-64-bit.gz`
 
 Create each of these by running the `elm` executable for each platform through `gzip elm`.
 
 
-## 2. Try a beta release
+## 2. Put the binaries in place
+
+Put the above files at:
+
+1. `packages/elm_darwin_arm64/elm`
+2. `packages/elm_darwin_x64/elm`
+3. `packages/elm_linux_x64/elm`
+4. `packages/elm_linux_arm64/elm`
+5. `packages/elm_win32_x64/elm.exe` (Note the `.exe` file extension!)
+
+(They are ignored by git.)
+
+
+## 3. Publish the binary packages
+
+Repeat this for all the packages mentioned in the previous section. This uses `packages/elm_darwin_arm64` as an example.
+
+1. Go to the folder: `cd packages/elm_darwin_arm64`
+2. Double-check that you put the right binary in the right package: `file elm`
+3. Double-check that the file is executable: `ls -l elm`
+4. In `package.json` of the binary package, bump the version for example to `"0.19.1-2"`.
+5. In `package.json` of the main npm package, update `"optionalDependencies"` to point to the bumped version. For example: `"@evancz/elm_darwin_arm64": "0.19.1-2"`
+6. Publish the package: `npm publish --access=public`
+
+   `--access=public` is needed because scoped packages are private by default.
+
+<details>
+<summary>Notes about the versions of the binary packages</summary>
+
+- End users never have to think about them. They only need to think about the version of the main npm package.
+
+- The binary packages can have different versions. One can have `"0.19.1-0"` while another is at `"0.19.1-1"`. This is useful if you mess up publishing one platform: Then you can bump just that one and re-release, instead of having to re-release _all_ platforms.
+
+- The version of the main npm package is not related to the versions of the binary packages – they’re all independent. So the main npm package can be at `"0.19.1-6"` while the binary packages have suffixes like `-0`, `-1` and `-9`. (They all share the `0.19.1` prefix though to make things more understandable!)
+
+- The main npm package pins the versions of the binary packages _exactly_ – no version ranges.
+  - This means that installing `[email protected]` installs the exact same bytes in two years as today.
+  - The `package.json` of each binary package says which OS and CPU it works for. `binary.js` in the main npm package has code that deals with OS and CPU too, so the main npm package needs to install binary packages with known OS and CPU declarations.
+
+- There is no need to use `beta` suffixes for the binary packages. Just bump the number suffix and point to it in a beta release of the main npm package. As mentioned above:
+  - Already published versions of the main npm package depend on exact versions of the binary packages, so they won’t accidentally start downloading beta versions.
+  - End users only see the version of the main npm package.
+
+</details>
+
+
+## 4. Try a beta release
 
 In `package.json`, bump the version to `"0.19.2-beta"`.
 
+Double-check that `"optionalDependencies"` is in sync with the binary packages.
+
 ```bash
 npm publish --tag beta
 ```
@@ -34,7 +107,7 @@ The `latest` tag should not be changed, and there should be an additional `beta`
 Try this on Windows, Linux, and Mac.
 
 
-## 3. Publish final release
+## 5. Publish final release
 
 Remove the `-beta` suffix from the version in `package.json`. Then run:
 
@@ -43,7 +116,7 @@ npm publish
 ```
 
 
-## 4. Tag the `latest-0.19.1` version
+## 6. Tag the `latest-0.19.1` version
 
 Many compiler releases have needed multiple `npm` publications. Maybe something does not work on Windows or some dependency becomes insecure. Normal `npm` problems.
 

installers/npm/bin/elm

@@ -1,54 +1,28 @@
 #!/usr/bin/env node
 
 var child_process = require('child_process');
-var path = require('path');
-var fs = require('fs');
 
 
 // Some npm users enable --ignore-scripts (a good security measure) so
 // they do not run the post-install hook and install.js does not run.
 // Instead they will run this script.
 //
-// On Mac and Linux, we download the elm executable into the exact same
+// On Mac and Linux, we hard link the elm executable into the exact same
 // location as this file. Since npm uses symlinks on these platforms,
 // that means that the first run will invoke this file and subsequent
 // runs will call the elm binary directly.
 //
-// On Windows, we must download a file named elm.exe for it to run properly.
+// On Windows, our binary file must be named elm.exe for it to run properly.
 // Instead of symlinks, npm creates two files:
 //
 //   - node_modules/.bin/elm (a bash file)
 //   - node_modules/.bin/elm.cmd (a batch file)
 //
 // Both files specifically invoke `node` to run the file listed at package.bin,
-// so there is no way around instantiating node for no reason on Windows. So
-// the existsSync check is needed so that it is not downloaded more than once.
+// so there is no way around instantiating node for no reason on Windows.
 
 
-// figure out where to put the binary (calls path.resolve() to get path separators right on Windows)
-//
-var binaryPath = path.resolve(__dirname, 'elm') + (process.platform === 'win32' ? '.exe' : '');
-
-// Run the command directly if possible, otherwise download and then run.
-// This check is important for Windows where this file will be run all the time.
-//
-if (process.platform === 'win32')
-{
-	fs.existsSync(binaryPath)
-		? runCommand()
-		: require('../download.js')(runCommand);
-}
-else
-{
-	require('../download.js')(runCommand);
-}
-
-
-function runCommand()
-{
-	// Need double quotes and { shell: true } when there are spaces in the path on windows:
-	// https://github.com/nodejs/node/issues/7367#issuecomment-229721296
-	child_process
-		.spawn('"' + binaryPath + '"', process.argv.slice(2), { stdio: 'inherit', shell: true })
-		.on('exit', process.exit);
-}
+var binaryPath = require('../binary.js')();
+child_process
+	.spawn(binaryPath, process.argv.slice(2), { stdio: 'inherit' })
+	.on('exit', process.exit);

installers/npm/binary.js

@@ -0,0 +1,122 @@
+var fs = require('fs');
+var package = require('./package.json');
+var path = require('path');
+
+
+
+// MAIN
+//
+// This function is used by install.js and by the bin/elm backup that gets
+// called when --ignore-scripts is enabled.
+
+
+module.exports = function()
+{
+	// figure out package of binary
+	var version = package.version.replace(/^(\d+\.\d+\.\d+).*$/, '$1'); // turn '1.2.3-alpha' into '1.2.3'
+	var subPackageName = '@evancz/elm_' + process.platform + '_' + process.arch;
+
+	verifyPlatform(version, subPackageName);
+
+	var fileName = process.platform === 'win32' ? 'elm.exe' : 'elm';
+
+	try
+	{
+		var subBinaryPath = require.resolve(subPackageName + '/' + fileName);
+	}
+	catch (error)
+	{
+		if (error && error.code === 'MODULE_NOT_FOUND')
+		{
+			exitFailure(version, missingSubPackageHelp(subPackageName));
+		}
+		else
+		{
+			exitFailure(version, 'I had trouble requiring the binary package for your platform (' + subPackageName + '):\n\n' + error);
+		}
+	}
+
+	// as mentioned in bin/elm we cannot do any optimizations on Windows
+	if (process.platform === 'win32')
+	{
+		return subBinaryPath;
+	}
+
+	// figure out where to put the binary
+	var binaryPath = path.resolve(__dirname, package.bin.elm);
+	var tmpPath = binaryPath + '.tmp';
+
+	// optimize by replacing the JS bin/elm with the native binary directly
+	try
+	{
+		// atomically replace the file with a hard link to the binary
+		fs.linkSync(subBinaryPath, tmpPath);
+		fs.renameSync(tmpPath, binaryPath);
+	}
+	catch (error)
+	{
+		exitFailure(version, 'I had some trouble writing file to disk. It is saying:\n\n' + error);
+	}
+
+	return binaryPath;
+}
+
+
+
+// VERIFY PLATFORM
+
+
+function verifyPlatform(version, subPackageName)
+{
+	if (subPackageName in package.optionalDependencies) return;
+
+	var situation = process.platform + '_' + process.arch;
+	console.error(
+		'-- ERROR -----------------------------------------------------------------------\n\n'
+		+ 'I am detecting that your computer (' + situation + ') may not be compatible with any\n'
+		+ 'of the official pre-built binaries.\n\n'
+		+ 'I recommend against using the npm installer for your situation. Check out the\n'
+		+ 'alternative installers at https://github.com/elm/compiler/releases/tag/' + version + '\n'
+		+ 'to see if there is something that will work better for you.\n\n'
+		+ 'From there I recommend asking for guidance on Slack or Discourse to find someone\n'
+		+ 'who can help with your specific situation.\n\n'
+		+ '--------------------------------------------------------------------------------\n'
+	);
+	process.exit(1);
+}
+
+
+
+// EXIT FAILURE
+
+
+function exitFailure(version, message)
+{
+	console.error(
+		'-- ERROR -----------------------------------------------------------------------\n\n'
+		+ message
+		+ '\n\nNOTE: You can avoid npm entirely by downloading directly from:\n'
+		+ 'https://github.com/elm/compiler/releases/tag/' + version + '\n'
+		+ 'All this package does is distributing a file from there.\n\n'
+		+ '--------------------------------------------------------------------------------\n'
+	);
+	process.exit(1);
+}
+
+
+
+// MISSING SUB PACKAGE HELP
+
+
+function missingSubPackageHelp(subPackageName)
+{
+	return (
+		'I support your platform, but I could not find the binary package (' + subPackageName + ') for it!\n\n'
+		+ 'This can happen if you use the "--omit=optional" (or "--no-optional") npm flag.\n'
+		+ 'The "optionalDependencies" package.json feature is used by Elm to install the correct\n'
+		+ 'binary executable for your current platform. Remove that flag to use Elm.\n\n'
+		+ 'This can also happen if the "node_modules" folder was copied between two operating systems\n'
+		+ 'that need different binaries - including "virtual" operating systems like Docker and WSL.\n'
+		+ 'If so, try installing with npm rather than copying "node_modules".'
+	);
+}