Connecting to Azure Redis Cluster

[hsadoyan]

Hey folks,

We have a rails app we're trying to connect to azure redis. I have a test cluster provisioned, and can connect as a single instance, but trying to connect in clustered mode gives an error: Redis Client could not connect to any cluster nodes

Here's a minimal config:

host = 'hostname'
key = 'password'
port = 6380
database = 0

connection_url = "rediss://:#{CGI.escape(key)}@#{host}:#{port}/#{database}"

client = Redis.new(cluster: [connection_url])

The documentation stated that I can pass in one instance like this and it'll discover the rest of the nodes through the CLUSTER NODES command.

If I switch from cluster to url I can connect to a redis node, but of course half the writes fail with a MOVED message.

I was able to find one SO post of somebody having a similar issue in 2019 that was never resolved, but otherwise no documentation that's redis-rb and Azure Redis specific

[supercaracal]

The above way of connecting Redis cluster looks like legitimate. I will try to enhance the error messages. It might be caused by ACL.

• Configuration for cluster using ACL #1054
• Redis commands not supported in Azure Cache for Redis
• What features aren't supported with Redis 6?

[hsadoyan]

I'm trying to connect to a Redis6 cluster, but it doesn't support ACL. Is there another way around this?

[hsadoyan]

Thanks for the extra logging! Very helpful. The actual error I'm getting is WRONGPASS invalid username-password pair (Redis::Cluster::InitialSetupError)

I'm not actually passing a username, only a password@hostname. Interesting thing is: that configuration worked for non-clustered redis, or if I'm trying to call it as a single instance with url: connection_string. Is the client doing something different when connecting as a cluster? Or is it likely to be some config in Azure redis itself?

[supercaracal]

It might be a bug in the client of cluster mode. I'll look into the differences of implementation and behavior between stand-alone mode and cluster mode.

redis-rb/lib/redis/client.rb

Lines 460 to 477 in 13c7a8e

	uri = URI(url)
	case uri.scheme
	when "unix"
	defaults[:path] = uri.path
	when "redis", "rediss"
	defaults[:scheme] = uri.scheme
	defaults[:host] = uri.host if uri.host
	defaults[:port] = uri.port if uri.port
	defaults[:username] = CGI.unescape(uri.user) if uri.user && !uri.user.empty?
	defaults[:password] = CGI.unescape(uri.password) if uri.password && !uri.password.empty?
	defaults[:db] = uri.path[1..-1].to_i if uri.path
	defaults[:role] = :master
	else
	raise ArgumentError, "invalid uri scheme '#{uri.scheme}'"
	end
	defaults[:ssl] = true if uri.scheme == "rediss"

redis-rb/lib/redis/cluster/option.rb

Lines 63 to 69 in 13c7a8e

	uri = URI(addr)
	raise InvalidClientOptionError, "Invalid uri scheme #{addr}" unless VALID_SCHEMES.include?(uri.scheme)
	db = uri.path.split('/')[1]&.to_i
	{ scheme: uri.scheme, username: uri.user, password: uri.password, host: uri.host, port: uri.port, db: db }
	.reject { |_, v| v.nil? || v == '' }

redis-rb/test/cluster_client_options_test.rb

Lines 9 to 51 in 13c7a8e

	def test_option_class
	option = Redis::Cluster::Option.new(cluster: %w[redis://127.0.0.1:7000], replica: true)
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	assert_equal true, option.use_replica?
	option = Redis::Cluster::Option.new(cluster: %w[redis://127.0.0.1:7000], replica: false)
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	assert_equal false, option.use_replica?
	option = Redis::Cluster::Option.new(cluster: %w[redis://127.0.0.1:7000])
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	assert_equal false, option.use_replica?
	option = Redis::Cluster::Option.new(cluster: %w[rediss://johndoe:foobar@127.0.0.1:7000/1/namespace])
	assert_equal({ '127.0.0.1:7000' => { scheme: 'rediss', username: 'johndoe', password: 'foobar', host: '127.0.0.1', port: 7000, db: 1 } }, option.per_node_key)
	option = Redis::Cluster::Option.new(cluster: %w[rediss://127.0.0.1:7000], scheme: 'redis')
	assert_equal({ '127.0.0.1:7000' => { scheme: 'rediss', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	option = Redis::Cluster::Option.new(cluster: %w[redis://bazzap:@127.0.0.1:7000], username: 'foobar')
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', username: 'bazzap', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	option = Redis::Cluster::Option.new(cluster: %w[redis://:bazzap@127.0.0.1:7000], password: 'foobar')
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', password: 'bazzap', host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	option = Redis::Cluster::Option.new(cluster: %w[redis://127.0.0.1:7000/0], db: 1)
	assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', host: '127.0.0.1', port: 7000, db: 0 } }, option.per_node_key)
	option = Redis::Cluster::Option.new(cluster: [{ host: '127.0.0.1', port: 7000 }])
	assert_equal({ '127.0.0.1:7000' => { host: '127.0.0.1', port: 7000 } }, option.per_node_key)
	assert_raises(Redis::InvalidClientOptionError) do
	Redis::Cluster::Option.new(cluster: nil)
	end
	assert_raises(Redis::InvalidClientOptionError) do
	Redis::Cluster::Option.new(cluster: %w[invalid_uri])
	end
	assert_raises(Redis::InvalidClientOptionError) do
	Redis::Cluster::Option.new(cluster: [{ host: '127.0.0.1' }])
	end
	end

[hsadoyan]

I was looking at the cluster_client_options_test.rb file on my own.

option = Redis::Cluster::Option.new(cluster: %w[redis://bazzap:@127.0.0.1:7000], username: 'foobar') 
   assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', username: 'bazzap', host: '127.0.0.1', port: 7000 } }, option.per_node_key) 
  
   option = Redis::Cluster::Option.new(cluster: %w[redis://:bazzap@127.0.0.1:7000], password: 'foobar') 
   assert_equal({ '127.0.0.1:7000' => { scheme: 'redis', password: 'bazzap', host: '127.0.0.1', port: 7000 } }, option.per_node_key) 

These are the two types that make sense. The former (without ':') returns `Redis client could not fetch cluster information: NOAUTH Authentication required. (Redis::Cluster::InitialSetupError)'

The latter returns WRONGPASS invalid username-password pair (Redis::Cluster::InitialSetupError)

I'm not aware of a username I'd even attempt to pair with the password. Someone having a similar issue with the JS client suggested that default worked for them, but it didn't work for me.

[hsadoyan]

I haven't been able to get the gem to build locally so I can play around with some of the tests myself, but I'll keep trying that

[hsadoyan]

 option = Redis::Cluster::Option.new(cluster: [{ host: '127.0.0.1', port: 7000 }]) 
   assert_equal({ '127.0.0.1:7000' => { host: '127.0.0.1', port: 7000 } }, option.per_node_key) 

Trying to pass in a hash instead of a format string just results in a Connection timed out. (Trying to pass the same hash as a single instance does connect fine)

connection_url = { host: host, password: key, port: port, ssl: true }
client = Redis.new(cluster: [connection_url])

[supercaracal]

I reproduced it in my local machine. It seems that special characters are doubly escaped. It is a client bug for cluster mode. I'll fix it later.

success in stand-alone mode with a url option:

irb(main):010:0> r = Redis.new url: "redis://:#{CGI.escape('!&<123-abc>')}@127.0.0.1:7000/0"
=> #<Redis client v4.6.0 for redis://127.0.0.1:7000/0>

irb(main):011:0> r.ping
=> "PONG"

success in cluster mode with a password option as plain text:

irb(main):018:0> r = Redis.new cluster: [{password: '!&<123-abc>', host: '127.0.0.1', port: 7000}]
=> #<Redis client v4.6.0 for redis://127.0.0.1:7000/0 redis://127.0.0.1:7001/0 redis://127.0.0.1:7002/0>

irb(main):019:0> r.ping
=> "PONG"

failure in cluster mode with a URI string:

irb(main):012:0> r = Redis.new cluster: %W[redis://:#{CGI.escape('!&<123-abc>')}@127.0.0.1:7000/0]
/path/to/redis-rb/lib/redis/cluster/slot_loader.rb:21:in `load': Redis client could not fetch cluster information: WRONGPASS invalid username-password pair or user is disabled. (Redis::Cluster::InitialSetupError)
        from /path/to/redis-rb/lib/redis/cluster.rb:116:in `fetch_cluster_info!'
        from /path/to/redis-rb/lib/redis/cluster.rb:26:in `initialize'
        from /path/to/redis-rb/lib/redis.rb:84:in `new'
        from /path/to/redis-rb/lib/redis.rb:84:in `initialize'
        from (irb):12:in `new'
        from (irb):12:in `<main>'
        from bin/console:8:in `<main>'

server configuration:

$ diff -u makefile /tmp/redis-rb-makefile
--- makefile    2022-02-20 19:02:23.660431802 +0900
+++ /tmp/redis-rb-makefile      2022-02-20 18:16:35.986257109 +0900
@@ -18,6 +18,7 @@
 CLUSTER_PID_PATHS  := $(addprefix ${TMP}/redis,$(addsuffix .pid,${CLUSTER_PORTS}))
 CLUSTER_CONF_PATHS := $(addprefix ${TMP}/nodes,$(addsuffix .conf,${CLUSTER_PORTS}))
 CLUSTER_ADDRS      := $(addprefix 127.0.0.1:,${CLUSTER_PORTS})
+PASSWORD           := !&<123-abc>

 define kill-redis
   (ls $1 > /dev/null 2>&1 && kill $$(cat $1) && rm -f $1) || true
@@ -43,21 +44,24 @@

 start: ${BINARY}
        @${BINARY}\
-               --daemonize  yes\
-               --pidfile    ${PID_PATH}\
-               --port       ${PORT}\
-               --unixsocket ${SOCKET_PATH}
+               --daemonize   yes\
+               --pidfile     ${PID_PATH}\
+               --port        ${PORT}\
+               --unixsocket  ${SOCKET_PATH}\
+               --requirepass '${PASSWORD}'

 stop_slave:
        @$(call kill-redis,${SLAVE_PID_PATH})

 start_slave: ${BINARY}
        @${BINARY}\
-               --daemonize  yes\
-               --pidfile    ${SLAVE_PID_PATH}\
-               --port       ${SLAVE_PORT}\
-               --unixsocket ${SLAVE_SOCKET_PATH}\
-               --slaveof    127.0.0.1 ${PORT}
+               --daemonize   yes\
+               --pidfile     ${SLAVE_PID_PATH}\
+               --port        ${SLAVE_PORT}\
+               --unixsocket  ${SLAVE_SOCKET_PATH}\
+               --slaveof     127.0.0.1 ${PORT}\
+               --requirepass '${PASSWORD}'\
+               --masterauth  '${PASSWORD}'

 stop_sentinel:
        @$(call kill-redis,${SENTINEL_PID_PATHS})
@@ -72,6 +76,7 @@
                echo 'sentinel down-after-milliseconds ${HA_GROUP_NAME} 5000'                >> $$conf;\
                echo 'sentinel failover-timeout        ${HA_GROUP_NAME} 30000'               >> $$conf;\
                echo 'sentinel parallel-syncs          ${HA_GROUP_NAME} 1'                   >> $$conf;\
+               echo 'sentinel auth-pass               ${HA_GROUP_NAME} ${PASSWORD}'         >> $$conf;\
                ${BINARY} $$conf\
                        --daemonize yes\
                        --pidfile   ${TMP}/redis$$port.pid\
@@ -105,7 +110,9 @@
                        --cluster-node-timeout 5000\
                        --pidfile              ${TMP}/redis$$port.pid\
                        --port                 $$port\
-                       --unixsocket           ${TMP}/redis$$port.sock;\
+                       --unixsocket           ${TMP}/redis$$port.sock\
+                       --requirepass          '${PASSWORD}'\
+                       --masterauth           '${PASSWORD}';\
        done

 create_cluster:

$ diff -u test/support/cluster/orchestrator.rb /tmp/redis-rb-cluster-helper.rb
--- test/support/cluster/orchestrator.rb        2022-02-20 19:02:23.660431802 +0900
+++ /tmp/redis-rb-cluster-helper.rb     2022-02-20 18:16:55.590696731 +0900
@@ -11,6 +11,7 @@
     @clients = node_addrs.map do |addr|
       Redis.new(url: addr,
                 timeout: timeout,
+                password: '!&<123-abc>',
                 reconnect_attempts: 10,
                 reconnect_delay: 1.5,
                 reconnect_delay_max: 10.0)

[hsadoyan]

Thanks for the quick fix. After merging lastest master I no longer get the WRONGPASS, but I'm getting a new error now.

If I try with the format string: "rediss://:#{CGI.escape(key)}@#{host}:#{port}" I get:

redis-rb-a7c2a5bfacf9/lib/redis/connection/ruby.rb:264:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (unspecified certificate verification error) (OpenSSL::SSL::SSLError)

And if I try with [{ host: host, password: key, port: port, ssl: true }] I get:

redis-rb-a7c2a5bfacf9/lib/redis/connection/ruby.rb:58:in `block in _read_from_socket': Connection timed out (Redis::TimeoutError)

Is it possible there's a different issue underneath? I can connect with the same credentials in non-clustered mode

[supercaracal]

I'll look into the issue later. There might be a another bug.

[supercaracal]

I've tried to reproduce SSL/TLS connection error with cluster in local machine using mutual self signed certs. However, it couldn't.

The former error is still under investigation. The latter error is a bug in cluster client. I'll fix it later.

irb(main):001:0> r = Redis.new cluster: %w[rediss://127.0.0.1:7000], ssl_params: {ca_file: File.join('test', 'support', 'ssl', 'trusted-ca.crt'),cert: OpenSSL::X509::Certificate.new(File.read(File.join('test', 'support', 'ssl', 'trusted-cert.crt'))),key: OpenSSL::PKey::RSA.new(File.read(File.join('test', 'support', 'ssl', 'trusted-cert.key')))}
=> #<Redis client v4.6.0 for redis://127.0.0.1:7000/0 redis://127.0.0.1:7001/0 redis://127.0.0.1:7002/0>

irb(main):002:0> r.ping
=> "PONG"

irb(main):003:0> r.cluster 'nodes'
=>
[{"node_id"=>"1014379650c31231d046b8163fcc5d64c1eab738", "ip_port"=>"127.0.0.1:7001@17001", "flags"=>["master"], "master_node_id"=>"-", "ping_sent"=>"0", "pong_recv"=>"1645606584000", "config_epoch"=>"2", "link_state"=>"connected", "slots"=>"5461".."10922"},
 {"node_id"=>"a3d1a4e4eb688ab795a0ddff4ec465d77b43e83c", "ip_port"=>"127.0.0.1:7005@17005", "flags"=>["slave"], "master_node_id"=>"9675ecc0378329e6d0f9903609a617f2da2a370c", "ping_sent"=>"0", "pong_recv"=>"1645606584793", "config_epoch"=>"3", "link_state"=>"connected", "slots"=>nil},
 {"node_id"=>"7ddabe0e224bf8d0846c998447a9e4108b57993a", "ip_port"=>"127.0.0.1:7004@17004", "flags"=>["slave"], "master_node_id"=>"1014379650c31231d046b8163fcc5d64c1eab738", "ping_sent"=>"0", "pong_recv"=>"1645606585597", "config_epoch"=>"2", "link_state"=>"connected", "slots"=>nil},
 {"node_id"=>"fb1e267e1e62d9b8d7d1370eada00705cf606c15", "ip_port"=>"127.0.0.1:7000@17000", "flags"=>["myself", "master"], "master_node_id"=>"-", "ping_sent"=>"0", "pong_recv"=>"1645606585000", "config_epoch"=>"1", "link_state"=>"connected", "slots"=>"0".."5460"},
 {"node_id"=>"dfe3cc76119460d0721d96f37c7d124f5ad148f7", "ip_port"=>"127.0.0.1:7003@17003", "flags"=>["slave"], "master_node_id"=>"fb1e267e1e62d9b8d7d1370eada00705cf606c15", "ping_sent"=>"0", "pong_recv"=>"1645606585000", "config_epoch"=>"1", "link_state"=>"connected", "slots"=>nil},
 {"node_id"=>"9675ecc0378329e6d0f9903609a617f2da2a370c", "ip_port"=>"127.0.0.1:7002@17002", "flags"=>["master"], "master_node_id"=>"-", "ping_sent"=>"0", "pong_recv"=>"1645606585798", "config_epoch"=>"3", "link_state"=>"connected", "slots"=>"10923".."16383"}]

irb(main):004:0> Redis.new(cluster: %w[rediss://127.0.0.1:7000]).ping
/path/to/redis-rb/lib/redis/connection/ruby.rb:264:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

irb(main):005:0> Redis.new(cluster: %w[redis://127.0.0.1:7000]).ping
/path/to/redis-rb/lib/redis/cluster/slot_loader.rb:21:in `load': Redis client could not fetch cluster information: Connection lost (ECONNRESET) (Redis::Cluster::InitialSetupError)

documents:

• TLS Support - Redis
• SSL/TLS Support - redis-rb

cert files:
https://github.com/redis/redis-rb/tree/master/test/support/ssl

server configuration:

$ diff -u /tmp/redis-rb-makefile.bk makefile
--- /tmp/redis-rb-makefile.bk   2022-02-23 16:24:08.536745615 +0900
+++ makefile    2022-02-23 17:23:22.028594220 +0900
@@ -104,7 +104,13 @@
                        --cluster-config-file  ${TMP}/nodes$$port.conf\
                        --cluster-node-timeout 5000\
                        --pidfile              ${TMP}/redis$$port.pid\
-                       --port                 $$port\
+                       --port                 0\
+                       --tls-port             $$port\
+                       --tls-cert-file        $(CURDIR)/test/support/ssl/trusted-cert.crt\
+                       --tls-key-file         $(CURDIR)/test/support/ssl/trusted-cert.key\
+                       --tls-ca-cert-file     $(CURDIR)/test/support/ssl/trusted-ca.crt\
+                       --tls-cluster          yes\
+                       --logfile              /tmp/redis.log\
                        --unixsocket           ${TMP}/redis$$port.sock;\
        done

$ diff -u /tmp/redis-rb-cluster-helper.rb.bk test/support/cluster/orchestrator.rb
--- /tmp/redis-rb-cluster-helper.rb.bk  2022-02-23 18:03:21.288383625 +0900
+++ test/support/cluster/orchestrator.rb        2022-02-23 18:03:23.656437261 +0900
@@ -1,6 +1,7 @@
 # frozen_string_literal: true

 require 'redis'
+require 'openssl'

 class ClusterOrchestrator
   SLOT_SIZE = 16_384
@@ -11,6 +12,12 @@
     @clients = node_addrs.map do |addr|
       Redis.new(url: addr,
                 timeout: timeout,
+                ssl: true,
+                ssl_params: {
+                  ca_file: File.join(__dir__, '..', 'ssl', 'trusted-ca.crt'),
+                  cert: OpenSSL::X509::Certificate.new(File.read(File.join(__dir__, '..', 'ssl', 'trusted-cert.crt'))),
+                  key: OpenSSL::PKey::RSA.new(File.read(File.join(__dir__, '..', 'ssl', 'trusted-cert.key'))),
+                },
                 reconnect_attempts: 10,
                 reconnect_delay: 1.5,
                 reconnect_delay_max: 10.0)