preprocessor fails on #include "ESP8266wifi.h" (for external ESP8266 addon)

[ladyada]

there's something about this header file that crashes the preprocessor, probably indicates some more general purpose bug. reproduce by:

• add http://arduino.esp8266.com/stable/package_esp8266com_index.json to board manager prefs
• use board manager to install ESP8266 support
• try to compile this sketch

#include "ESP8266WiFi.h"
void setup() {}
void loop() { hello(); }
void hello(void) {}

• will fail due to not having auto-generated prototype for hello()
• now put #include "ESP8266WiFi.h" on last line, recompile
• works now because while it does crash on the #include, its after the prototypes are made (?)

Also mentioned @ esp8266/Arduino#1066

[igrr]

The #include which triggers the issue is #include <memory> in WiFiClient.h.
I have created a library with a single .h file (Dummy.h):

#include <memory>

Including this library in the sketch seems to be enough to trigger the issue.

Edit: actually we don't even need a library to trigger this. Compiling this sketch leads to the same error:

#include <memory>
void setup() {
  test();
}
void loop() {}
void test() {}

Edit 2: here's the output of arduino-builder with --debug=10: http://hastebin.com/matajofoti.md

Edit 3: interestingly, the list of prototypes is generated correctly by ctags:

$ "/Users/igrokhotkov/projects/esp8266/arduino.cc/master/build/macosx/work/Arduino.app/Contents/Java/tools-builder/ctags/5.8-arduino4/ctags" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns "/var/folders/15/ybtbgzpj7636vv4wp92l2c700000gn/T/buildad5feee0973bd3788deb217687c83661.tmp/preproc/ctags_target.cpp"
setup   /var/folders/15/ybtbgzpj7636vv4wp92l2c700000gn/T/buildad5feee0973bd3788deb217687c83661.tmp/preproc/ctags_target.cpp /^void setup() {$/;"    kind:function   line:4  signature:()    returntype:void
loop    /var/folders/15/ybtbgzpj7636vv4wp92l2c700000gn/T/buildad5feee0973bd3788deb217687c83661.tmp/preproc/ctags_target.cpp /^void loop() {$/;" kind:function   line:9  signature:()    returntype:void
test    /var/folders/15/ybtbgzpj7636vv4wp92l2c700000gn/T/buildad5feee0973bd3788deb217687c83661.tmp/preproc/ctags_target.cpp /^void test() {}$/;"    kind:function   line:14 signature:()    returntype:void

But the .ino.cpp file generated in the next step doesn't contain void test() prototype:

#include <Arduino.h>
#line 1
#line 1 "/var/folders/15/ybtbgzpj7636vv4wp92l2c700000gn/T/arduino_ad5feee0973bd3788deb217687c83661/sketch_nov21a.ino"
#include <memory>
void setup() {
  test();
}
void loop() {}
void test() {}

[igrr]

I pulled the latest git version of Arduino and rebuilt it. Also renamed test to myfunc to make it easier to grep.

Here's arduino-builder output: http://hastebin.com/mupuhalehe.cs
ctags_target_for_gcc_minus_e.cpp: https://gist.github.com/igrr/3095758a66bc28f7db23 (too large for hastebin:)
ctags output: https://gist.github.com/igrr/805dc5ef96cf9c0a0892
Note that there's really no myfunc in ctags output.

I had a thought that ctags gets confused by some of the c++11 constructs. Tried universal-ctags, and output looks much better (i.e. noexcept is no longer tagged as a function), but still it doesn't have myfunc.

[tigoe]

Haven't got a solution but I can reproduce all of the errors down to the rebuild reliably, and it definitely seems to be what caused #4137.

[ffissore]

At a first look, it looks like ctags is interrupting. It's not getting beyond that tuple_element. I'm investigating why

[igrr]

For the record, here's what i get from universal-ctags: https://gist.github.com/igrr/b82b708af81e7baa02fc
Unlike ctags shipped with Arduino, it is able to parse newer C++ keywords like noexcept. Still doesn't return full output though.

[ffissore]

Indeed, even universal-ctags misses myfunc. Rebasing our ctags fork on top of universal-ctags is a nice-to-have (and tracked at arduino/ctags#2) but currently not in my todo list

[ffissore]

I'm now using this

#include "ESP8266WiFi.h"
void setup() {
}
void loop() {
  myfunc();
}
void myfunc() {
}

When running ctags with -V (verbose) at the top of its output it prints
/tmp/arduino_12f75e84123cf24fa7ce3d06ac6342b3/sketch_nov23a.ino: failed to find match for '(' at line 36991
36991 is the last line of ctags_target_for_gcc_minus_e.cpp. A similar error message is displayed even after manually adding the prototype at the top of the sketch or removing all functions and leaving the sketch with just the first #include.

[ffissore]

Same does universal-ctags