Regression in ssl module between 3.13.5 and 3.13.6: reading from a TLS-encrypted connection blocks

[aaugustin]

Bug report

Bug description:

The script below works with 3.13.5 and fails with 3.13.6.

It's a straightforward socket server and client with TLS enabled. Under 3.13.5, it runs successfully. Under 3.13.6, when the server calls recv(), it blocks and never receives what the client sent with sendall().

This is a minimal reproduction version of python-websockets/websockets#1648. I performed the reproduction on macOS while the person reporting the bug was on Linux so I think it's platform-independent.

To trigger the bug, the client must read from the connection in a separate thread. If you remove that thread, the bug doesn't happen. (For context, I do this because websockets is architecture with a Sans-I/O layer so I need a background thread to pump bytes received from the network into the Sans-I/O parser.)

Before you run the script, you must download https://github.com/python-websockets/websockets/blob/main/tests/test_localhost.pem and store it next to the file where you saved the Python script.

import os
import socket
import ssl
import threading


TLS_HANDSHAKE_TIMEOUT = 1

print("If Python locks hard:")
print("kill -TERM", os.getpid())
print()

# Create TLS contexts with a self-signed certificate. Download it here:
# https://github.com/python-websockets/websockets/blob/main/tests/test_localhost.pem

server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
server_context.load_cert_chain(b"test_localhost.pem")
# Work around https://github.com/openssl/openssl/issues/7967
server_context.num_tickets = 0

client_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client_context.load_verify_locations(b"test_localhost.pem")


# Start a socket server. Nothing fancy here. In a realistic server, we would
# have `serve_forever` with a `while True:` loop. For a minimal reproduction,
# `serve_one` is enough, as the bug occurs on the first request.

server_sock = socket.create_server(("localhost", 0))
server_port = server_sock.getsockname()[1]

server_sock = server_context.wrap_socket(
    server_sock,
    server_side=True,
    # Delay TLS handshake until after we set a timeout on the socket.
    do_handshake_on_connect=False,
)

def conn_handler(sock, addr) -> None:
    print("server accepted connection from", addr)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True)

    sock.settimeout(TLS_HANDSHAKE_TIMEOUT)
    assert isinstance(sock, ssl.SSLSocket)
    sock.do_handshake()
    sock.settimeout(None)

    handshake = sock.recv(4096)
    print("server rcvd:")
    print(handshake.decode())
    print()

def serve_one():
    sock, addr = server_sock.accept()
    handler_thread = threading.Thread(target=conn_handler, args=(sock, addr))
    handler_thread.start()

print("server listening on port", server_port)
server_thread = threading.Thread(target=serve_one)
server_thread.start()


# Connect a client to the server. Again, nothing fancy.

client_sock = socket.create_connection(("localhost", server_port))
client_sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True)

client_sock.settimeout(TLS_HANDSHAKE_TIMEOUT)
client_sock = client_context.wrap_socket(
    client_sock,
    server_hostname="localhost",
)
client_sock.settimeout(None)

### The bug happens only when we're reading from the client socket too! ###

def recv_one_event():
    msg = client_sock.recv(4096)
    print("client rcvd:")
    print(msg.decode())
    print()

client_background_thread = threading.Thread(target=recv_one_event)
client_background_thread.start()

### If you remove client_background_thread.start(), it doesn't happen. ###

handshake = (
    b"GET / HTTP/1.1\r\n"
    b"Host: 127.0.0.1:51970\r\n"
    b"Upgrade: websocket\r\n"
    b"Connection: Upgrade\r\n"
    b"Sec-WebSocket-Key: jjSVQ7XPjx2GIXKfQ49QDQ==\r\n"
    b"Sec-WebSocket-Version: 13\r\n"
    b"Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n"
    b"User-Agent: Python/3.13 websockets/15.0.1\r\n"
    b"\r\n"
)

print("client send:")
print(handshake.decode())
print()
client_sock.sendall(handshake)

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• gh-137583: Only lock the SSL context, not the SSL socket #137588
• [3.13] gh-137583: Only lock the SSL context, not the SSL socket (GH-137588) #137613

[picnixz]

Can you bisect the bad commit please?

[aaugustin]

I know how to bisect but I've never built Python before so it may take me a bit of time to do that. I'll give an update here if I manage.

[picnixz]

Thanks. Python 3.13.5 has been released in June, so we should see what changed since then.

[picnixz]

Ok we have only few commits in SSL.c:

• 5a20e79 (use of SSL_sendfile when available, commit not present in 3.13)
• e047a35 (hold a lock when the thread state is detached in SSL)
• 377b787 (setting SSL groups, shouldn't affect the existing code)

[picnixz]

While it appears to be OS-independent, both failures were seen on Unix-derived systems. If it also fails on Windows, this would give us an indication of whether it's truly independent or not. In the meantime, could you try the following patch: in your script put this after importing socket:

def _no(*args, **kwargs):
	raise socket._GiveupOnSendfile
socket.socket._sendfile_use_sendfile = _no

IOW, replace the _sendfile_use_sendfile method of socket objects like that to force using send instead of sendfile.

[picnixz]

Scratch that, my patch won't work as sendfile is only being used in main.

[aaugustin]

e047a35 was also my initial guess based on reading the changelog (python-websockets/websockets#1648 (comment)).

[picnixz]

cc @ZeroIntensity

[aaugustin]

I could build Python and confirm that the problem does not occur in cb93b6f and occurs e047a35, confirming that the regression happens in e047a35.