Long running PHP PubSub process that eats all cpu exactly 1h into processing

[udAL]

This is more of a plea for help than a bug report..

• OS: Dockerized Ubuntu 16.04.6 (Modified gcr.io/google-appengine/php72)
• PHP version: 7.2.22
• Package name and version:
  google/cloud-pubsub 1.11.0
  google/cloud-core 1.27.0
  (outdated because of compatibility reasons)

We have been using PubSub for a while for managing background tasks. We have a GCE vm that has supervisord with two long running php processes that check for new messages and process whatever. It looks something like this:

use Google\Cloud\PubSub\PubSubClient;
$pubSub = new PubSubClient([
    'keyFilePath' => 'includes/secret.json'
]);
$subscription = $pubSub->subscription('workers');
while(true)
{
    $message = false;
    try
    {
        foreach ($subscription->pull([
            'maxMessages' => 1
        ]) as $pullMessage)
        {
            $message = $pullMessage;
        }
    }
    catch( Exception $e)
    {
        // Nothing
    }
    if($message)
    {
        // Do stuff
        $subscription->acknowledge($message);
    }
}

As discussed on #1986, since we started we saw this errors popping up every hour:

[19-Sep-2019 16:30:30 Europe/Madrid] PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\ServiceException: cURL error 35: gnutls_handshake()$
Stack trace:
#0 /app/vendor/google/cloud-core/src/RequestWrapper.php(189): Google\Cloud\Core\RequestWrapper->convertToGoogleException(Object(GuzzleHttp\Exce$
#1 /app/vendor/google/cloud-core/src/RestTrait.php(95): Google\Cloud\Core\RequestWrapper->send(Object(GuzzleHttp\Psr7\Request), Array)
#2 /app/vendor/google/cloud-pubsub/src/Connection/Rest.php(193): Google\Cloud\PubSub\Connection\Rest->send('subscriptions', 'pull', Array)
#3 /app/vendor/google/cloud-pubsub/src/Subscription.php(409): Google\Cloud\PubSub\Connection\Rest->pull(Array)
#4 /app/worker.php(77): Google\Cloud\PubSub\Subscription->pull(Array)
#5 {main}
  thrown in /app/vendor/google/cloud-core/src/RequestWrapper.php on line 336

It wasn't a big deal for us since we had our supervisord to keep the processes open. It wasn't till a few days ago when we updated our docker (which we do frequently). Suddenly our worker vm's were constantly at 100% cpu, even though there wasn't anything to process on PubSub and no task processing. We could restart the processes and they would work fine for exactly one hour, then they'll pop to 100% cpu without apparent reason. We're stuck with an essential server that needs restarting every 60 min.

We have done a lot of debugging and the only difference we found between the old and new machines it's that ubuntu package libcurl3-gnutls was updated from <7.47.0-1ubuntu2.12> to <7.47.0-1ubuntu2.14>. We haven't found a way to downgrade.

We have tried to force Guzzle to use TLS 1.1. PubSub allows to pass parameters to it like this:

$subscription->pull([
    'maxMessages' => 1,
    'restOptions' => [
        'curl' => [
            CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_0
        ]
    ]
])

But it hasn't work... Not with CURL_SSLVERSION_TLSv1 neither with CURL_SSLVERSION_TLSv1_0 or CURL_SSLVERSION_TLSv1_2
I know this is a complex error, probably has nothing to do with google/cloud-pubsub, and this is an outdated version. Maybe it's a TLS incompatibility with google pubsub service. Maybe an outdated library in an updated os. I really don't know, but we'll take any suggestion you can give us... Any ideas?

Tomorrow i'll try with a clean project with updated packages and will post.

[dwsupplee]

@udAL that sounds really frustrating, 🙁. We'll definitely take a look to see where/how we can help. Do you have the protobuf/gRPC extensions enabled? If so, would you be able to share versions of each?

As an aside, would you be able to share the comparability issues you're experiencing? Maybe there's something we can do there to help as well.

[udAL]

Hi @dwsupplee
I understand you're referring to google/protobuf and grpc/grpc composer packages. We did have them installed:

• grpc/grpc 1.19.0
• google/protobuf v3.7.1

Our compatibility issues come mainly from requirements of package kreait/firebase-php. If I uninstall it and update I can get:

• google/cloud-core v1.33.0 (latest)
• google/cloud-pubsub v1.17.0 (latest)
• google/protobuf v3.9.1 (latest)
• grpc/grpc 1.22.0 (latest)

I'll do some tests to see if this was part of the problem.
Thanks

[udAL]

With last versions of the packages the result it's the same, 100% cpu after 1h into the script.
I'm sure this is related to libcurl and cURL error 35: gnutls_handshake() since this error used to pop up at 1h and has been replaced with this 100% spikes at 1h, so I'll get into that. Following the cURL error 35 I found suggestions to force cURL to use TLS1.0, do you know if my code it's correct when passing parameters to Guzzle? Any ideas on what else I could try?

[udAL]

To the best of my knowledge the connection to PubSub freezes 1h into it. Don't know how or why, but since PubSub it's usually used on long running processes, isn't that something that should be taken into account?

To hotfix the problem, we had installed cron and restarted the process every 50 min but that has it's own problems since a task could be cough in the middle and, not being acknowledged, would be reprocessed. I have put a timer in the script to destroy and create the PubSub object to reconnect and this should hotfix the problem for now.

[udAL]

This occurred also on google-cloud-python around Dec 2017:
googleapis/google-cloud-python#4600
It was fixed by requiring python package grpcio >= 1.8.2:
googleapis/google-cloud-python#4642

Does that help?

[udAL]

Is there anything being done to solve this?
Otherwise, how should I proceed?
Can you help?

[illambo]

About cURL error 35: gnutls_handshake() we experience a similar issue in one app hosted on flexible engine php (see screenshot), and we use a supervisord in order to maintain stable the worker.

Another thing, for this application we have different envs each in a different project, currently we find this problem only in one of the three projects, maybe try to redistribute in a new project, maybe the problem is for specific network zone.

• google/cloud-pubsub: v1.17.0
• google/cloud-core: v1.33.0
• google/protobuf: v3.9.1
• grpc/grpc: 1.22.0

[dwsupplee]

I understand you're referring to google/protobuf and grpc/grpc composer packages.

Actually, I was wondering about the extensions available on PECL (protobuf/grpc) :).

I have some time set aside next week to help look in to this, my apologies for the delay. We've had a team summit this week.

[udAL]

I did not have protobuf nor grpc extensions installed.
I have added those and it seems this solves the issue. I'll keep testing to be sure.

• ext-protobuf: 3.7.0
• ext-grpc: 1.19.0

Shouldn't this extensions be required on the google/cloud-core?

EDIT: False 'all is good' alarm. It still pops up to 100% CPU

[dwsupplee]

Shouldn't this extensions be required on the google/cloud-core?

Not all our libraries which depend on core require the extensions, and often times users may find themselves on something like shared hosting where the extensions aren't even an option. As a result we've worked to make sure they aren't hard requirements.

False 'all is good' alarm. It still pops up to 100% CPU

Gah, sorry to hear that. I'm setting up an environment to do some testing here soon.

[udAL]

Any news?

[dwsupplee]

@udAL I've been working on reproducing. Would you be able to share the modified dockerfile you're using?

[jdpedrie]

@udAL are you able to share this additional info? :)