std: `/dev/ptmx` in MSYS2 is not recognized as a tty in `std::io::IsTerminal`

[petrochenkov]

How to reproduce:

./x t tests/run-make --test-args emit-to-stdout

That test (tests\run-make\emit-to-stdout\Makefile) checks that rustc's behavior with options like --emit obj is different depending on whether stdout is tty or not.
That test uses /dev/ptmx as stdout and fails when run under MSYS2.

cc @ChrisDenton

[petrochenkov]

we@WIN-GQ6AS7OLS34 MINGW64 ~
$ rustc --emit obj=- hello.rs 1>/dev/ptmx

we@WIN-GQ6AS7OLS34 MINGW64 ~
$ rustc --emit obj=- hello.rs
error: option `-o` or `--emit` is used to write binary output type `obj` to stdout, but stdout is a tty

error: aborting due to 1 previous error

Cannot check the behavior on Linux right now, but if nobody complains about failing run-make tests, then /dev/ptmx is probably considered a tty there.
UPD: Yep, /dev/ptmx is a tty on Linux and rustc refuses to --emit obj into it.

[ChrisDenton]

Does it work with MSYS=enable_pcon set in msys2.ini?

For msys2 without pseudoconsole support there's no way to really know if something is a tty or not so we use heuristics based on the pipe name, these may need updated. If msys2 is using the Windows pseudoconsole then it should just work as it should be detected as a normal Windows console handle (unless /dev/ptmx is special in some other way).

[petrochenkov]

Does it work with MSYS=enable_pcon set in msys2.ini?

No, it doesn't help.

For msys2 without pseudoconsole support there's no way to really know if something is a tty or not so we use heuristics based on the pipe name, these may need updated

I did assume it was some missing heuristic in fn msys_tty_on.

[ChrisDenton]

Ah, actually the problem is this check:

rust/library/std/src/sys/windows/io.rs

Lines 100 to 112 in efb3f11

	// At this point, we *could* have a false negative. We can determine that this is a true
	// negative if we can detect the presence of a console on any of the standard I/O streams. If
	// another stream has a console, then we know we're in a Windows console and can therefore
	// trust the negative.
	for std_handle in [c::STD_INPUT_HANDLE, c::STD_OUTPUT_HANDLE, c::STD_ERROR_HANDLE] {
	let std_handle = c::GetStdHandle(std_handle);
	if !std_handle.is_null()
	&& std_handle != handle
	&& c::GetConsoleMode(std_handle, &mut out) != 0
	{
	return false;
	}
	}

The trouble is even if enable_pcon is set (which it turns out it is by default now) /dev/ptmx is still a pipe instead of a pseduoconsole handle. So unless all std handles are redirected, it will return negative.

[ChrisDenton]

I created a fix in #119664 by simply removing the above code. This decreases the chances of a false negative at the cost of increasing the chances of a false positive. But I think simply inspecting the pipe name is overall the most reliable way to detect an msys2 tty.

[lygstate]

Why /dev/ptmx is a pipe, can that be checked by environment variable?

[ChrisDenton]

You'd have to ask msys2 why that is. Maybe it's a bug or maybe there's some reason for it. I don't know.

can that be checked by environment variable

I don't think so.

[petrochenkov]

You'd have to ask msys2 why that is. Maybe it's a bug or maybe there's some reason for it. I don't know.

@mati865, maybe you know?

[mati865]

Unfortunately I don't know much about Cygwin's internals.