Fail to run k3s using mysql with SSL (... it doesn't contain any IP SANs)

[up9cloud]

k3s-io/k3s#1763

There are two issues here:

1. kine does no provide ServerName for tlsConfig, so can't verify by name...
2. kine does not set InsecureSkipVerify to true when dsn is with param ?tls=skip-verify

Relative refs:

https://github.com/rancher/kine/blob/3faf3a7028014a5baf96454b2b3fe04984ebc69f/pkg/drivers/mysql/mysql.go#L156
https://github.com/rancher/kine/blob/3faf3a7028014a5baf96454b2b3fe04984ebc69f/pkg/tls/config.go#L15
https://github.com/etcd-io/etcd/blob/6e800b9b0161ef874784fc6c679325acd67e2452/pkg/transport/listener.go#L72
https://github.com/rancher/k3s/blob/fcb864a5e20f69a4d5b19feb7f265abb9086b749/pkg/cli/server/server.go#L101

[brandond]

This may be related to go-sql-driver/mysql#564

[up9cloud]

@brandond It's not related, that because if you want to connect to GCP mysql, you have to use dsn with ip (not with domain) and you have to manually pass the server name to tls config to let it verify by server name.

[brandond]

Wait, so you're connecting to it with an IP address in the datastore URI, but want to validate the certificate against a hostname, because the IP address isn't in the SAN list on the server's cert?

Can you provide an example?

[up9cloud]

@brandond yes it is.

See this commit for another project:

https://github.com/ajvb/kala/pull/211/files#diff-cdd396ae86656919264e05e2f781ed517f85aa8dd79c0cfe64ee1954bc98734eR209

I use config something like this:

{
  "jobDBAddress": "(123.123.123.123)/kala?tls=custom",
  "jobDBUsername": "user",
  "jobDBPassword": "the secret",
  "jobDBTlsServerName": "api-0000000000000-00000:blabla"
}

and it works perfectly

[up9cloud]

that's how GCP mysql with TLS works 😂

[up9cloud]

See this also:

https://stackoverflow.com/questions/53752383/how-do-you-create-a-tls-connection-to-a-cloud-sql-database-using-go

[brandond]

So they basically require you to disable TLS verification because "the mysql client libraries by default have hostname verification disabled" and they seem to think that using unique self-signed certs makes MITM impossible somehow? This all seems quite broken.

[up9cloud]

Well, to be honestly, I'm not familiar with MITM, and don't know why they made like that. I just want to use k3s with TLS mysql on GCP. All I know is there are 2 ways to solve this:

1. Let ?tls=skip-verify works (this is the way that we connect to server but skip that verification, like you said)
2. Pass ServerName to tls config to verify by server name (this is that we connect to it and verify the cert by the server name they request)

[fredleger]

Also hitting this.
I think the best option is to honor the tls=skip-verify flag since this is a classic and it might help in other reason than Google Cloud (if not the ssl lib won't have included it)

[fredleger]

For those that are affected by this one just want to share that using postgresql has not this side effect since pq does honor sslmode=verify-ca

[maradwan]

I am trying to use Vcluster k3s with MySQL and TLS but it seems it didn’t work
the values.yaml looks like , I am using Azure MySQL

vcluster:
env:
- name: K3S_DATASTORE_ENDPOINT
value: mysql://username:password@tcp(hostname:3306)/DB_NAME
- name: K3S_DATASTORE_CAFILE
valueFrom:
secretKeyRef:
name: mysql-ssl-certificates
key: DigiCertGlobalRootCA.crt.pem

Do you know what is the issue?