Crappy error message when smoketest before-hook crashes

[aquamoth]

My Ubuntu 15.04 automatically upgraded Firefox to v38.0, which breaks the selenium smoke tests.
The selenium-bug is already reported to Selenium, see SeleniumHQ/selenium#437.

This issue isn't with selenium though, but with the crappy error messages in the smoke tests, and I know how you feel about crappy error messages :).

First time I run the smoke tests I get:

mattias@maas-linux-1:~/git/lets_code_javascript$ ./jake.sh loose=true smoketest
Collating client files: .
Bundling client files with Browserify: ..
Smoke testing app:


  ․․

  1 passing (2s)
  1 failing

  1) Smoke test "before all" hook:
     Uncaught Error: Failed to install profile; firefox terminated with Result(code=null, signal=SIGSEGV)
      at Error (native)
      at Array.forEach (native)
  From: Task: WebDriver.createSession()
      at src/_smoke_test.js:26:14
      at Socket.<anonymous> (src/_run_server.js:18:55)
      at readableAddChunk (_stream_readable.js:163:16)
      at Socket.Readable.push (_stream_readable.js:126:10)
      at Pipe.onread (net.js:538:20)

This message isn't the worst I've seen. At least it mentions both firefox and WebDriver. If I re-run the command though (which I often tend to do when I'm unfocused), the server on port 5000 hasn't shutdown correctly and I instead get:

mattias@maas-linux-1:~/git/lets_code_javascript$ ./jake.sh loose=true smoketest
Collating client files: .
Bundling client files with Browserify: ..
Smoke testing app:


  ․․

  1 passing (11s)
  1 failing

  1) Smoke test "before all" hook:
     Error: timeout of 10000ms exceeded. Ensure the done() callback is being called in this test.

Now it takes 10 seconds to time out and the error message is really vague. It wasn't obvious to me I suddenly had an abandoned server running on port 5000.

[jamesshore]

Urg. Not again. This is the second time we've had a Firefox upgrade break the Selenium tests.

I added a browser version check and error message the last time this happened, but of course the version check depends on Selenium initializing correctly. :(

Thanks for bringing this up; I'll add it to the to-do list as of episode 310.

[jamesshore]

Okay, I've made a note to fix three problems:

• Selenium 2.45.1 doesn't work with Firefox 38 — upgraded to 2.46.0
• The smoke tests don't report the expected browser version when Selenium fails to initialize — can't be fixed because problem occurs while Selenium is loading browser
• When the smoke tests' setup fails, the server doesn't shut down, leading to confusing error messages on subsequent test runs — not fixed, see comment below

[aquamoth]

If it's ok with you, I'm sending you a pull-request in a little while, which I hope alleviates the crappy error message. You are of course free to do whatever you like with it, but it's fun to have an actual task to fix in node. :)

[jamesshore]

Sure, knock yourself out. :-) I'll include it in the discussion when we fix this issue on camera.

[jamesshore]

@aquamoth's pull request is in #8.

[jamesshore]

I've taken a closer look at this and decided not to implement it at this time. It's a good change, but big enough that I'd want to do it on camera, and I (and others ;-) ) am impatient to move on to other things.

The error's not too bad, in that it says "EADDRINUSE" (meaning that the server can't be launched due to something already running at that address) before timing out. It could definitely be better, and your fix is a good one. I imagine it will come up again, and we can fix it then.