Need to re-fix Windows manifest embedding.

[vadimcn]

The previous attempt (#10878) to fix #10512 has a couple of problems:

1. Sometimes UpdateResource corrupts executables produced by binutils. (The executable still runs, however tools like dumpbin report corrupted PE header, and gdb crashes when trying to read debug info).
2. More importantly, this approach does not work for cross-compilation, because, well, you can't call Windows API if the host system is not Windows.

[alexcrichton]

So that's why I haven't been able to use gdb recently! That also explains why a recent attempt at objdump didn't succeed.

Sounds like we may need to revert the manifest embedding for now?

[vadimcn]

@alexcrichton: Yes, probably.

[vadimcn]

Regarding fixing this, I see a few approaches:

• Have rustc write out a .rc file (resource source file), then use windres tool to compile it to an object file. Pass resulting object file to the linker.
  • Pros: Flexible. Manifest can be modified per-executable (e.g. to change the requested invocation privilege level).
  • Cons: Requires invocation of an external tool.
• Include a .rc file containing the manifest in rust source tree, compile it using windres during librustc build, include produced object file into librustc via include_bin!(...). When compiling executable targeting Windows, write binary blob into .o file, and pass it to the linker.
  • Pros: Don't need to invoke windres tool on every compilation.
  • Cons: Manifest can't be altered without rebuilding rust compiler.
• Create .o file containing manifest directly.
  • Pros: Flexible. Can alter manifest as needed.
  • Cons: Complexity. Will need to implement a COFF file writer.
• Link executable without resources, then insert resource section into the output .exe file (without using Windows' UpdateResource API).
  • Pros: Flexible. Can alter manifest as needed.
  • Cons: Complexity. Need to parse/alter/emit COFF headers. Though probably less complex than approach No3.

Currently we have no need to alter .exe manifests, and if really needed this can always be done afterwards using windres tool, so I am leaning towards option 2.

[alexcrichton]

Do you know the format of the COFF file? LLVM certainly has the ability to write a COFF file, and we might be able to just use their bindings to emit our own COFF file ourselves.

They seem to have lots and lots of header files and defines related to COFF object files, so seems promising? Of the options, using LLVM to generate an object and then passing that to the linker sounds the most appealing to me.

[vadimcn]

COFF format is well-documented, but yes, it's complicated.

Regarding LLVM: as far as I know, it is not possible to create PE COFF resource section even in native assembly, much less in LL code. I actually investigated this approach, but ran into inability to emit RVA offsets. Apparently this can only be done by writing .obj file directly.

[yuriks]

As an additional option, could a combination of options 1 and 2 work? Ship a default compiled resource file with rustc, but allow it to be overriden by the user if he wants to. (In which case he must pay the "cost" of requiring windres for that to work.)

[vadimcn]

FWIW, UAP installer detection heuristics may be suppsessed by setting this environment variable: __COMPAT_LAYER=RunAsInvoker.