Make incremental compilation smarter about copying when reusing data

[nikomatsakis]

In the PR I'm about to open :), the code will copy object files into the incremental compilation directory upon completion, and then copy them out if they can be re-used. But it does more copying than needed in the case of reuse, since it will then copy them back in to the incr. comp. directory at the very end, even though the files are precisely the same.

(In all these cases, when I say copy, I really mean "hard link if possible".)

This is a FIXME-sort of issue. Putting it up for mentorship since fixing it seems fairly straightfoward.

[nagisa]

What you really want here is a reflink when possible, and hadlink only then.

[OlliV]

Uh, what's the point of copying .o files? Doesn't that ruin the whole performance gain? I don't think I have ever seen a build system that would copy .o files, usually out-dated files are just overwritten. Maybe I'm missing something?

[michaelwoerister]

@OlliV Normally we would not copy .o files at all, just create a new hard-link to the data. If data would have to be copied, that would indeed not be good for performance (although it probably still faster than generating the code from scratch and letting LLVM optimize it).

[michaelwoerister]

@nagisa What would be the advantage of using a reflink over using a hard-link? Ref-links are really interesting but they only seem to be supported by very few file systems.

[nagisa]

You would not need to reimplement CoW by yourself on those filesystems. The
filesystems are also better at doing CoW themselves than whatever you can
do from an application. E.g. it can handle CoW on both reflink and
reflinkee without any extra effort.

On Aug 16, 2016 10:51 PM, "Michael Woerister" notifications@github.com
wrote:

@nagisa https://github.com/nagisa What would be the advantage of using
a reflink over using a hard-link? Ref-links are really interesting but they
only seem to be supported by very few file systems.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#34955 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AApc0hR6Nqc2pUWkkQrVuBmDCVgCtwyRks5qghSzgaJpZM4JR6Ut
.

[michaelwoerister]

You would not need to reimplement CoW by yourself on those filesystems.

OK, but since we have to support the hard-link case anyway there would not be much of a difference in practice, right? Especially since we always either completely reuse a file or completely rewrite it (so far, at least).

[Mark-Simulacrum]

@nikomatsakis Is there a chance you could give some initial guidance on what needs to be done here to make this issue more accessible to potential mentees?

[nikomatsakis]

@Mark-Simulacrum a reasonable request, but I'm not quite sure what needs to be done, to be honest. I'll have to review the current status of the code. I can't remember where we left this in the end. It may even have been done already.

[Mark-Simulacrum]

Sure. Just trying to make these more accessible to incoming contributors; don't worry too much about it if it's hard, though it would be nice to assess this at some point so we don't have an indeterminate issue hanging around :).

[nikomatsakis]

Personally I think we should just close this issue.

[nikomatsakis]

(Until we see this as an actual perf problem in real life, and then we can re-open)