Unnecessary rebuild all for any core lib file changed

[PaulStoffregen]

When any file within the core library changes, "Build options changed, rebuilding all" prints and all files are unnecessarily recompiled.

I tested just now with arduino-PR-beta1.9-BUILD-22 on Linux 64 bit (Ubuntu 14.04), with Arduino/Genuino Zero Programming Port selected. I have preferences set to show verbose info while compiling. The problem seems to happen with any code, but this is the program I used just now:

void setup() {
  Serial.begin(9600);
}
void loop() {
  static int count=0;
  Serial.print("test ");
  Serial.print(count++);
  delay(100);
}

To produce the problem, I first clicked Verify. Then I run this in a terminal window:

touch ~/.arduino15/packages/arduino/hardware/samd/1.6.16/cores/arduino/Print.cpp

When I click Verify again after this command, the result is "Build options changed, rebuilding all", and all files in the core library are recompiled, even though only Print.cpp has changed.

[facchinm]

We know it and do on purpose to solve a broader problem: with normal caching infrastructure, it you change anything in platform.txt or boards.txt the change is not applied unless you change the target board. Do you suggest we could only apply the this approach to non-source files, while continue trusting .d files for source dependencies?

[PaulStoffregen]

Yes. Definitely everything has to be recompiled if platform.txt or boards.txt change. If any of the settings that make up the fully qualified board name change, of course everything needs to be recompiled.

But for changes only in the source code, whether or not to recompile a particular source file is the whole purpose of those .d files. I believe they are very trustworthy, if you can verify the .d file and .o file are newer than the timestamps on all the files listed within the .d file. Arduino 1.6.5 (the last before switching to arduino-builder) did this quite well, even though it didn't have many of the features arduino-builder has now.

[pedzed]

I'm facing this issue as well. Compiling on macOS with VSCode's Arduino extension takes very long this way. Not sure if that has anything to do with it?

[facchinm]

@pedzed I'm not sure that VSCode extension uses arduino-builder under the hood (I'm pretty sure it doesn't) so you'd better report your issue to their repo (https://github.com/Microsoft/vscode-arduino)

[pedzed]

I'm not sure that VSCode extension uses arduino-builder under the hood (I'm pretty sure it doesn't)

I actually do think it does (based on the terminal with Arduino that opens up in the background).

Anyway, it seems to be solved in the newer Arduino version (1.8.5). I was using 1.8.3. I should have tried this a long time ago, but I remember not seeing it in the change log which was the reason I did not bother updating.

Really? I am getting it again. I am not even touching the code, just reuploading.

[facchinm]

@pedzed Arduino IDE != arduino-builder (although the IDE uses the builder internally) . If you launch multiple instances of the IDE you'll likely lose any cache produced by the previous run (if you could paste here the commandline that gets executed I can tell you what is going on).

@PaulStoffregen 9b0a81a should target this issue, since it wipes only if platform.txt or boards.txt has changed. Beta 1.9 already contains the fix if you want to try it out 😉

[stevstrong]

I am having the same problem.
V1.8.5, I am using an external code editor (Metapad).
Each time I just press the save button in Metapad to save the INO file, the IDE does rebuild all core files:

Build options changed, rebuilding all
Detecting libraries used...

EDIT: only the INO file is opened in Metapad.
Is there any way to solve this issue within this version? I don't want to change to a new (beta) version.

[artua]

I have the same issues running esp8266 board. It is always recompile all, but no boards or platforms file changed. 1.9 beta working fine. Any chance to have the same fix for 1.8.6?

[pedzed]

I switched from Arduino Builder to PlatformIO (in combination with the lovely VSCode editor). Definitely recommended. It solved all issues for me.

[PaulStoffregen]

fwiw, I've been using 1.8.5 with the arduino-builder from 1.9, and the tools-builder stuff it needs. I did have to add another rule into platform.txt for the new preprocessor. With platform.txt updated, it works very well. The parallel build is especially nice when regularly switching between boards.

With 1.9 so far on the horizon, I too hope a stable 1.8.6 release can be made with the updated arduino-builder. There is a huge community of Arduino users who aren't going to use beta software, and are needlessly suffering slow compile times. I know the dev effort is going into 1.9, but can't a little work go into giving so many ordinary non-beta users these very stable improvements while the rest of the new stuff remains in 1.9?

[guino]

@PaulStoffregen care to share what you changed in platform.txt ? I am dealing with full recompile every time on 1.8.5 and have no solution right now.

[PaulStoffregen]

@guino - First, try the nightly build, or copy arduino-builder from the nightly into 1.8.5. The latest nightly has these changes (since this commit: arduino/Arduino@ee1967c ), but (probably) doesn't need the platform.txt change.

But to answer your question, with the caveat that this is for Teensy, here is what I added to platform.txt:

## New Preprocessor for Arduino 1.9
tools.arduino-preprocessor.path={runtime.tools.arduino-preprocessor.path}
tools.arduino-preprocessor.cmd.path={path}/arduino-preprocessor
tools.arduino-preprocessor.pattern="{cmd.path}" "{source_file}" "{codecomplete}" -- -std=gnu++14

This is almost certainly not correct for other non-Teensy boards, since we use C++14 and Teensyduino has a significantly different boards.txt and platform.txt. But you asked what I added, so there it is.

[guino]

@PaulStoffregen I tried the latest nightly compile and it worked to my surprise! Now the compiles are way faster without that full rebuild every time. I am just crossing fingers it doesn't break something else. Thanks for the suggestion and information!