[BUG] ESP8266 Cannot open certain TCP connections

[PedroD]

Basic Info

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: NodeMCU V0.9 (ESP-12E Module) (Compatible)
• Core Version: [2.4.1]
• Development Env: [Arduino IDE 1.8.5]
• Operating System: [Windows, OSX]

Settings in IDE

• Board: NodeMCU V0.9 (ESP-12E Module)
• Flash Size: 4MB (1M SPIFFS)
• Debug Port: Serial
• Debug Level: None
• lwIP Variant: v2 Lower Memory
• CPU Frequency: 80Mhz
• Upload Speed: 115200
• Erase Flash: Only Sketch
• Programmer: AVRISP mkII

The Problem

I have this server: 185.205.210.197
(Check that it works: http://185.205.210.197/)

And I have this code in my ESP8266:

#include <ESP8266WiFi.h>

#define HOST "185.205.210.197"
#define PORT 80
#define WIFI_SSID "SSID"
#define WIFI_PSW "PASSWORD"

const char* ssid = WIFI_SSID;
const char* password = WIFI_PSW;

void setup () {
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(5000);
    Serial.println("Connecting to Wifi...");
  }

  while(!pingServer()) {
    delay(3000); //Send a request every 3 seconds
  }
}


void loop() {}


bool pingServer() {
  WiFiClient client;

  if (!client.connect(HOST, PORT)) {
    Serial.println("connection failed");
    return false;
  }

  Serial.println("connection success!");
  return true;
}

The expected result: Connection success, but got connection failed.

If I try connecting with my PC, it works, only with ESP8266 it doesn't.

Wireshark

Tapping with Wireshark I suspect that there is a problem with ESP8266 TCP packets, here I leave them for you to analyze:

Success TCP Packet

PC TCP connection request (which works):

000af5f4e90c9801a7ad45b708004500004057b6400040066a7dc0a82b49b9cdd2c5dc6d1b3932f96fd800000000b002ffffc71e0000020405b4010303050101080a4ab5122c0000000004020000

(captured using basic ethernet)

Screenshot of an example of a TCP packet that is able to get a server's response.

ESP8266 TCP Packet (Not Success)

ESP8266 TCP connection request (which receives no response from server):

000019006f080000794c5e9b00000000126c9e098004d4a10008013c00000af5f4e90c2c3ae80f137f000af5f4e90c9002aaaa0300000008004500002c00270000ff0642f9c0a82b70b9cdd2c5c00d1b3900001c77000000006002086022f9000002040218a9636c75

(captured using monitor mode in Wireshark - has radiotap headers)

Screenshot example of an ESP8266 packet.

What I've also checked:

• The ESP connects fine to other servers.

• There is no firewall blocking the connection between ESP and the server.

• I've tried this in different networks and conditions, the error persists.

• Other computers can connect to the server, so ESP should be able as well.

• Tried with different ESP8266 boards to rule out hardware malfunctioning.

How to replicate:

0. Using the latest Arduino IDE and ESP8266 library.

1. Check that the test server works at http://185.205.210.197:80/ (just open this link basically and see that is running nginx).

2. Copy and upload the code above to your ESP8266 (NODEMCU or similar).

3. Open the Arduino IDE console/monitor and confirm that no connection is established to port 80.

4. Rage in admiration for such a strange bug.

Possible Workaround

@Pablo2048 Suggested setting the Arduino IDE board option lwIP Variant from v2 Lower Memory to v2 Higher Bandwidth. This increases the TCP MSS (http://lwip.wikia.com/wiki/Tuning_TCP) that goes from around 500 bytes to around 1400 bytes, which is accepted by this server.

Therefore by increasing the MSS, the server starts responding and a TCP connection is established, whereas with a low MSS the server wouldn't even answer back with an error packet (complete silence).

[PedroD]

PS. I learned from a comment in Freenode that telnet also appears not to work with that server, with the same error as ESP8266:

telnet 185.205.201.197 80

[KarolisL]

Telnet works, ip is mistyped in your command: 201 instead of 210.

[PedroD]

Right... Ok, so ESP8266 is the only one who cannot access the server. This must indeed be a library bug.

Has anyone been able to reproduce this with the steps I wrote?

[5chufti]

with no information about the core and lwip version you are using (no, the questionair is not to tease you) there will be low response ... if you are not willing to spend some time, why should others.

and no, this is not sufficient info:

Using the latest Arduino IDE and ESP8266 library.

[PedroD]

@5chufti done.

[Pablo2048]

Ok, so please try to:

1. turn on debugging messages to see whats SDK/libraries are doing inside
2. try to pass IPAddress parameter into .connect method instead of char * to eliminate gethostbyname error

[PedroD]

@Pablo2048 ok, I just did that, and I think we can confirm that he's parsing the hostname properly.

I attach below:

1. the sketch code with the modification to IPAddress you proposed
2. a screenshot of the wireshark tapping of all packets ESP8266 tried to send to the server while running that sketch (in this screenshot you see 6 packets, 3 per connection attempt)
3. the wireshark cap file
4. the console prints of said sketch

Sketch

#include <ESP8266WiFi.h>

#define HOST IPAddress(185,205,210,197)
#define PORT 80
#define WIFI_SSID "redecomfios"
#define WIFI_PSW ""

const char* ssid = WIFI_SSID;
const char* password = WIFI_PSW;

void setup () {
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(5000);
    Serial.println("Connecting to Wifi...");
  }

  while(!pingServer()) {
    delay(3000); //Send a request every 3 seconds
  }
}


void loop() {}


bool pingServer() {
  WiFiClient client;

  if (!client.connect(HOST, PORT)) {
    Serial.println("connection failed");
    return false;
  }

  Serial.println("connection success!");
  return true;
}

Console Prints

scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 5

connected with redecomfios, channel 11
dhcp client start...
cnt 
wifi evt: 0
ip:192.168.43.112,mask:255.255.255.0,gw:192.168.43.1
wifi evt: 3
Connecting to Wifi...
:ref 1
:ctmo
:abort
:ur 1
:del
connection failed
pm open,type:2 0
:ref 1
:ctmo
:abort
:ur 1
:del
connection failed
:ref 1
:ctmo
:abort
:ur 1
:del
connection failed
:ref 1
:ctmo
:abort
:ur 1
:del
connection failed
:ref 1
:ctmo
:abort
:ur 1
:del
connection failed

[Eszartek]

I was curious so I tested your server and I got the same results. I even tested from my app (a forth interp with network commands) and It also couldn't connect to your server. Interestingly, I run many Nginx servers and I've never encountered this problem before, so my first thought is there may be some packet mangling going on in a router that's in front of your server. It's not clear if you tested on the same LAN with local addresses. If you did, that rules out the router being the issue. Does the Nginx server OS have any firewall rules enabled?

I'll look at the packet capture.

[PedroD]

Hi @Eszartek thanks for testing.

What I have validated so far (99% sure):

• It is not nginx fault, I have tried with netcat and my own TCP server written in Java running in the server, ESP8266 still does not connect, I putted nginx there for you guys to try out this issue, but nginx is not the cause.

• I have tested this using my home router, my work router, my smartphone's AP (3G), and it happens no matter what. If there is something blocking the connection, it is not from my side.

Are the ESP8266 TCP packets being correctly generated? Everything else is being able to connect to the server.

This is creepy as hell. This server is the only server having issues with ESP8266. However, ESP8266 is the only client having issues connecting to this server as well. It is like a romantic relationship that is never going to happen ahah.

[Pablo2048]

According to your Wireshark screenshot it seems like ngnix doesn't respond with SYN ACK. I have no idea why :-( ...

[PedroD]

@Pablo2048 yeah, nothing will respond. Every server I put there (nginx, netcat, my custom server) will not respond.

But this only happens with ESP8266, because if you try with your browser (or other tcp client), nginx (and the other servers) responds just fine.

It makes me wonder if ESP8266 packets are properly built.

[PedroD]

@Pablo2048 here you can see a TCP packet directed the exact same way as ESP8266 one (to that server, to port 80 - nginx) and it is able to get an SYN ACK answer: http://prntscr.com/iz9h3a

@igrr @mmiscool This is a weird one, someone with deep network knowledge should take a look at it.

[Pablo2048]

Ok, there is one difference in MSS - can you try LWIP v2 but not the lower memory variant?

[PedroD]

@Pablo2048 OMG... it connected.

 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v614f7c32
~ld
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 2
cnt 

connected with redecomfios, channel 1
dhcp client start...
wifi evt: 0
ip:192.168.69.109,mask:255.255.255.0,gw:192.168.69.1
wifi evt: 3
Connecting to Wifi...
:ref 1
connection success!
:ur 1
:close
:del
pm open,type:2 0

I feel like crying.

Why was MSS limiting the connection? All I know about it comes from this article: http://lwip.wikia.com/wiki/Tuning_TCP

[Pablo2048]

Well it was just blind shot from me ;-) ... It seems like nginx need MSS to be set to bigger value (or there is something like router in the way which needs Don't fragment flag to be set, but lwip lower memory does not set this flag)...
Edit: wait - you wrote that you have tested more servers but no one work so it seems like something in TCP stack configuration on the target machine...