4.x: BasicLoadBalancingPolicy.getReplicas() is always empty with Tablets enabled

[scrat98]

Problem
In Cassandra to calculate a Token we only need keyspace and partition key. TokenFactory (or Partitioner in Scylla) is an internal API and does not exposed https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/BasicLoadBalancingPolicy.java#L313

After introducing the Tablets #379, now we also need a table, but what's most important we need Partitioner, because we can obtain Tablet only having Token and the TabletMap does not provide similar to the TokenMap method where we can pass only partition key without partitioner. https://github.com/scylladb/java-driver/blob/scylla-4.x/core/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/BasicLoadBalancingPolicy.java#L353

For that reason current implementation relies that Request will have Partitioner, that's not the case, because default implementation returns null for all Statements except of DefaultBoundStatement that delegates to the PreparedStatement.getPartitioner(). But if we'll have a look at DefaultPreparedStatement creation we'll find that Partitioner not null only for scylla CDC tables.
https://github.com/scylladb/java-driver/blob/scylla-4.x/core/src/main/java/com/datastax/oss/driver/internal/core/cql/Conversions.java#L390
https://github.com/scylladb/java-driver/blob/scylla-4.x/core/src/main/java/com/datastax/oss/driver/internal/core/metadata/PartitionerFactory.java#L45

Therefore, partitioner is always null that leads to the empty replica list and breaks load balancing at all.

Solution
According to the doc of Request.getPartitioner() method

The partitioner to use for token-aware routing. If null, the cluster-wide partitioner will be used.

Therefore, it's necessary to obtain TokenFactory also from the DefaultTokenMap, like it does here https://github.com/scylladb/java-driver/blob/scylla-4.x/core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java#L291

Steps to repo

• Setup multi node scylla cluster.
• Create keyspace with RF = 1
• Create simple table (p, c, v)
• Create prepared statement like "SELECT * FROM table WHERE p = ?"
• Bind this prepared statement N times with the same partition key
• Use session.context.loadBalancingPolicy.newQueryPlan(statement)
• You'll find that queryPlan every time just a round robin of the cluster node, but it must prioritized the same single node each time, because we're using ShardAwarePolicy and the same partition key should point to the same node in our case

[dkropachev]

@scrat98 , thanks for reporting, I need some code example of what exactly you are doing.
partitioner if not provided with the query going to be picked up from tokenMap.

[dkropachev]

I have confirmed that there are problems in the driver in regards of the tablets update, working on the fix.
Meanwhile, you can do the following:

1. Ensure that session either built with withKeyspace or you run USE ks on it.
2. Use prepared statement with bound primary key values.

Be aware that tablets information is getting filled while you are executing queries, it is empty on new sessions and getting filled with every request that hits wrong host+shard.

[scrat98]

@dkropachev I'm sorry for the late reply, but just would like to provide example of repro.

partitioner if not provided with the query going to be picked up from tokenMap.

Unfortunately after 4.18.0.2 version (after PR #405) it was changed. If tablets are enabled and Partitioner is null (which is always true for all statements) then empty replicas will be returned.

It's not possible to subscribe on TabletMap updates, but you can also put a breakpoint to the AddTabletRefresh to see that it will be executed multiple times, because we're reaching the wrong node.

Note: to see the problem you need a Scylla cluster, not a single node.

Example:

  val session = CqlSessionBuilder()
    .addContactPoints(listOf(InetSocketAddress("172.17.0.2", 9042)))
    .withLocalDatacenter("datacenter1")
    .build()
  session.execute("DROP KEYSPACE IF EXISTS my_keyspace")
  session.execute(
    """
      CREATE KEYSPACE IF NOT EXISTS my_keyspace
      WITH replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 1 }
      AND tablets = { 'enabled': true }
      AND durable_writes = true;
    """
  )
  session.execute(
    """
      CREATE TABLE IF NOT EXISTS my_keyspace.my_table
      (
        p      text,
        c      text,
        v      text,
        PRIMARY KEY (p, c)
      ) WITH CLUSTERING ORDER BY (c ASC)
        AND compaction = { 'class' : 'SizeTieredCompactionStrategy' };
    """
  )
  val writeStatement = session.prepare("INSERT INTO my_keyspace.my_table (p, c, v) VALUES (?, ?, ?)")
  val readStatement = session.prepare("SELECT p, c, v FROM my_keyspace.my_table WHERE p = ?")
  val loadBalancingPolicy = session.context.getLoadBalancingPolicy(DriverExecutionProfile.DEFAULT_NAME)
  // just write some random data
  repeat(10_000) {
    val toWrite = writeStatement.boundStatementBuilder()
      .setString(0, UUID.randomUUID().toString())
      .setString(1, UUID.randomUUID().toString())
      .setString(2, UUID.randomUUID().toString())
      .build()
    session.execute(toWrite)
  }
  repeat(100) {
    // with RF = 1, the same key should be routed always to the same node
    val toRead = readStatement.boundStatementBuilder()
      .setString(0, "theSameKey")
      .build()
    // Partitioner is always null for all statements
    println("Partitioner: ${toRead.partitioner}")
    // after a while TabletMap must be refreshed
    println("TabletMap: ${session.metadata.tabletMap.mapping}")
    // to trigger the Tablets update
    session.execute(toRead)
    // using tablets result is always round-robin of the cluster nodes, because getReplicas returns empty list
    val nodesWithTabletMap = loadBalancingPolicy.newQueryPlan(toRead, session)
    println("Nodes with tablets: $nodesWithTabletMap")
    // using TokenMap everything is okay, and we'll get the same node over and over again
    val nodesWithTokenMap = session.metadata.tokenMap.get().getReplicas(toRead.routingKeyspace!!, toRead.routingKey!!)
    println("Nodes with tokenMap: $nodesWithTokenMap")

    println(System.lineSeparator())
  }

Example output:

Partitioner: null
TabletMap: {}
Nodes with tablets: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06), Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a), Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]


Partitioner: null
TabletMap: {}
Nodes with tablets: [Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab), Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06), Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]


Partitioner: null
TabletMap: {KeyspaceTableNamePair{keyspace='my_keyspace', tableName='my_table'}=[]}
Nodes with tablets: [Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a), Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab), Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]


Partitioner: null
TabletMap: {KeyspaceTableNamePair{keyspace='my_keyspace', tableName='my_table'}=[DefaultTablet{firstToken=8646911284551352319, lastToken=8935141660703064063, replicaNodes=[Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)], replicaShards={Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)=6}}]}
Nodes with tablets: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06), Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a), Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]


Partitioner: null
TabletMap: {KeyspaceTableNamePair{keyspace='my_keyspace', tableName='my_table'}=[DefaultTablet{firstToken=8646911284551352319, lastToken=8935141660703064063, replicaNodes=[Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)], replicaShards={Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)=6}}]}
Nodes with tablets: [Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab), Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06), Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]


Partitioner: null
TabletMap: {KeyspaceTableNamePair{keyspace='my_keyspace', tableName='my_table'}=[DefaultTablet{firstToken=8646911284551352319, lastToken=8935141660703064063, replicaNodes=[Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)], replicaShards={Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab)=6}}]}
Nodes with tablets: [Node(endPoint=/172.17.0.3:9042, hostId=1f4a3c72-42ad-49ba-b63b-2aba8cf55044, hashCode=7d47976a), Node(endPoint=/172.17.0.4:9042, hostId=829c6f29-3d4d-45d4-812a-78b03db897c5, hashCode=7030f4ab), Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]
Nodes with tokenMap: [Node(endPoint=/172.17.0.2:9042, hostId=cf13c6ec-9467-45e9-bc51-59e6ee53a17a, hashCode=74035d06)]

[dkropachev]

@scrat98 , thanks for reporting it, as result we have found three bugs in this area.

[dkropachev]

4.19.0.1 is released, be available in 3 hours on maven