different sketch MD5 if built from different locations

[marcobrianza]

I am using ESP.getSketchMD5() to manage software releases.

I have noticed that if I use the same Arduino IDE, but in a different place in the filesystem the sketch MD5 is different.

I have used a portable version of the IDE to be sure that the same files are used.
For example loading a sketch that prints MD5 shows this behaviour.

Is this a known thing?

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println( ESP.getSketchMD5());
}
void loop() {
}

[marcobrianza]

using the strings command on a compiled binary shows some occurrence of the source path of the core, it would be nice if this could be avoided.

[d-a-v]

If it's about __FILE__ gcc macro related, then I'd like to have to solution too.

[igrr]

There's a GCC patch from Debian which can be used for this purpose: https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20170814/009124.html

…

On Thu, Mar 15, 2018, 20:03 david gauchard ***@***.***> wrote: If it's about __FILE__ gcc macro related, then I'd like to have to solution too. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4520 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJcei-jH7x29EYYgMGHi63B5UGJ0Ol2ks5telilgaJpZM4Sr5bq> .

[Eszartek]

The use of ESP.getSketchMD5() is interesting to me as well, so after running strings on my binaries I saw the long absolute paths being stored. Since I compile my projects on Linux using makeEspArduino.mk makefile instead of the Arduino IDE, it was trivial to edit my config.mk from:

ESP_ROOT=/home/user/Arduino/hardware/esp8266com/esp8266
LIBS=~/Arduino/hardware/esp8266com/esp8266/libraries/ ~/Arduino/libraries/

to be:

ESP_ROOT=../hardware/esp8266com/esp8266
LIBS=../hardware/esp8266com/esp8266/libraries/ ../libraries/

Then, I compiled my current project on two different machines where the working directory is different, and the resulting md5sums matched. That's cool!

However, the more interesting result for me was that my binary size was now smaller:
from:
Ram: 61220 bytes
Flash: 375172 bytes
to:
Ram: 61096 bytes
Flash: 374200 bytes

I'm always working to save a byte here and there, so this was a nice gift for very little effort.

Edit: And yes, running strings now shows all the paths stored are relative.

[d-a-v]

@igrr thanks for the interesting reading !
While it should be feasible to backport it to (xtensa-)gcc-4.8.2 (at which cost?), @Eszartek's solution with makeEspArduino.mk seems very good at little cost.

[earlephilhower]

Not to threadjack, but if we do rebuild gcc please add in the "#define JUMP_TABLES_IN_TEXT 1" change, too, to save extra heap at the same time.

Where is the actual patch referenced in the email thread? I can't seem to find the changes themselves.

[d-a-v]

In my comment just above yours I put a link to the changes

[earlephilhower]

@d-a-v The links themselves go to the mailing list archive, which talks about the patch. But where are the actual patch files they proposed? I'm not seeing any link to it.

[d-a-v]

@earlephilhower in the bottom of the link in my comment, reproduced here:
https://gcc.gnu.org/ml/gcc-patches/2016-11/msg00182.html
this is the link to the 3rd patch containing the __FILE__ management gcc patch.

[earlephilhower]

Thanks, @d-a-v . I saw the links but for some reason didn't see the actual changes. I'll see if they can apply to the GCC toolchain I think we're using now: https://github.com/jcmvbkbc/gcc-xtensa .

[d-a-v]

There are several branches on this repo.
I guess there is a good reason, still I don't know at what cost we could have -flto.

[igrr]

@d-a-v -flto, unfortunately, is not only a matter of rebuilding the toolchain. For simple programs it works pretty well, but for larger programs where you assign certain function to specific sections (like, put some stuff into IRAM and some into flash), LTO has problems tracking the correct sections to use.

[d-a-v]

I can't quantify flto benefit. I guess it could be huge especially for us.
Do you mean that the section-mixing flto issue is always true ?
Or is there a way to selectively use this option at the cost of splitting source files by hand to isolate and group objects from the same section together ?

[earlephilhower]

FWIW when testing the JUMP_TABLES_IN_ROM setting I was able to build and replace all the gcc executables dir using branch call0-4.8.2. Everything "just worked."

Whether that is, or is not, the branch being distributed I can't really say.

However, if you wanted to do FLTO tests, @d-a-v , it seems like it would be a safe starting point.

[earlephilhower]

@d-a-v It looks like the GCC patch could be applied by hand, with probably only minor mods, to the GCC-4.x in the repo. The meat of the change is in libcpp/macro.c which seems to not have undergone much editing from gcc4 to gcc7 (case BT_FILE:). Edit the local const char *name somehow (I think environment variables are not a good choice as it's hidden and not something easily used under both Linux/Mac and Windows) to remove the /.../Arduino/hardware/esp8266/xxx/ bit and you're golden.

Whether this is worth the grief, considering you'd need to rebuild the toolchain(this is a libcpp.so file) and use it on all supported platforms, is a question above my pay grade.

--edit--
How cool is it to see this at the top of a file you're looking at editing?

   Adapted to ANSI C, Richard Stallman, Jan 1987