ex_unit: Add :capture_io tag

[wojtekmach]

Possible future work:

1. Allow ExUnit.start(capture_io: true) to mimic capture_log: true?

2. Add api to programmatically append to the StringIO input buffer, right now it's only possible on creation:

   {:ok, pid} = StringIO.open("3")
   "3" = IO.gets(pid, "1 + 2")

   The idea is something like this:

   {:ok, pid} = StringIO.open("")
   
   StringIO.buffer("1 + 2")
   "1 + 2" = IO.gets(pid, "> ")
   
   :eof = IO.gets(pid, "> ")
   
   StringIO.buffer("2 + 3")
   "2 + 3" = IO.gets(pid, "> ")

   And so we could replace the usage of:

   capture_io(prompt, fn -> ... end)

   too.

   Maybe instead of calling it StringIO.buffer it could be called StringIO.input_write and StringIO.input_puts to mimic IO.write and IO.puts. Not sure if we'd need .input_binwrite too.

3. @tag capture_io: :standard_error?

   I think we could do this as long as the test is running in async: false which we'd know and then we can raise otherwise. This is as opposed to capture_io(:standard_error, ...) where ExUnit does not know if it's sync or async.

   If we do this, do we want to capture both? I guess we could accept a list of devices and set it in context:

   @tag capture_io: [:stdio, :standard_error]
   test "foo", %{capture_io: [stdio, stderr]}

   Not sure if :stdio is even semantically correct, my understanding is it's not stdio per se but the group leader. So yeah, it's probably a pass, and a reason to use capture_io in this case, but I thought I'd mention this for completeness.

4. Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

[sabiwara]

StringIO.buffer("1 + 2")

StringIO.buffer(pid, "1 + 2") right?

Make the StringIO available to @tag :capture_log tests in context as %{capture_log: io}. This could be considered a breaking change since today the context is usually set as %{capture_log: true} or %{capture_log: options} but I can't think of a reason to read them back in a test.

This one seems solvable by picking a slightly different name, captured_log: / captured_io:?

[sabiwara]

This is a really neat addition 🤩 💜

[josevalim]

@tag capture_io: :standard_error?

Right, this doesn't play well with StringIO.flush, because then you are flushing data between tests. I think it is fine to keep this specific to stdin/stdout. It is potentially the same reason why adding a similar API to capture_log would be confusing, you cannot use flush and that would lead to more imprecise tests. So I believe this is good to go as is.

[josevalim]

Maybe we should call it stdout for consistency with logs?

[wojtekmach]

Good call, test.stdout sounds good. Instead of context.capture_io should we have context.stdout too? I think that'd be nicer. And then same for logs, instead of context.capture_log (or per @sabiwara context.capture_logs) have context.logs?

[josevalim]

I think the issue is that we don't want to expose %{logs: device} because those devices are by definition shared. You will most likely read and flush data from other tests, so I think a tiny capture window works better.

In other words, we need to decide if we are happy shipping this feature considering it will only be available for stdout. The renaming to stdout in the struct should be a separate discussion.

[wojtekmach]

Are they shared though? I thought each test gets its own StringIO for capturing logs.

[josevalim]

Yes, you are right.