Skip to content

Remove protocol versions >5 from MAX_SUPPORTED #492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 12, 2025

Conversation

mykaul
Copy link

@mykaul mykaul commented Jun 19, 2025

Without removing associated code yet, this patch just ensure we begin negotiating protocol version 5, then fallback to 4, reducing the amount of protocol errors that the Scylla server sends when it sees the unsupported versions.

Refs: #244

Perhaps we should add documentation to this?

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • I added relevant tests for new features and bug fixes.
  • All commits compile, pass static checks and pass test.
  • PR description sums up the changes and reasons why they should be introduced.
  • I have provided docstrings for the public items that I want to introduce.
  • I have adjusted the documentation in ./docs/source/.
  • I added appropriate Fixes: annotations to PR description.

@mykaul mykaul requested a review from Copilot June 19, 2025 12:56
Copilot

This comment was marked as outdated.

@mykaul mykaul force-pushed the max_supported_v5 branch 2 times, most recently from 11a07bc to 7042125 Compare June 19, 2025 14:29
@mykaul
Copy link
Author

mykaul commented Jun 19, 2025

Need to handle this failure:

=================================== FAILURES ===================================
  ________________ ConnectionTest.test_use_requested_compression _________________
  
  self = <tests.unit.test_connection.ConnectionTest testMethod=test_use_requested_compression>
  args = (), c = <Connection(139701745826928) 1.2.3.4:9042>
  options_buf = <_io.BytesIO object at 0x7f0ed8eb3c70>
  options = b'\x00\x02\x00\x0bCQL_VERSION\x00\x01\x00\x053.0.3\x00\x0bCOMPRESSION\x00\x02\x00\x06snappy\x00\x03lz4'
  
      def test_use_requested_compression(self, *args):
          c = self.make_connection()
          c._requests = {0: (c._handle_options_response, ProtocolHandler.decode_message, [])}
          c.defunct = Mock()
          # request snappy compression
          c.compression = "snappy"
      
          locally_supported_compressions.pop('lz4', None)
          locally_supported_compressions.pop('snappy', None)
          locally_supported_compressions['lz4'] = ('lz4compress', 'lz4decompress')
          locally_supported_compressions['snappy'] = ('snappycompress', 'snappydecompress')
      
          # the server only supports snappy
          options_buf = BytesIO()
          write_stringmultimap(options_buf, {
              'CQL_VERSION': ['3.0.3'],
              'COMPRESSION': ['snappy', 'lz4']
          })
          options = options_buf.getvalue()
      
          c.process_msg(_Frame(version=4, flags=0, stream=0, opcode=SupportedMessage.opcode, body_offset=9, end_pos=9 + len(options)), options)
      
  >       self.assertEqual(c.decompressor, locally_supported_compressions['snappy'][1])
  E       AssertionError: None != 'snappydecompress'
  
  /project/tests/unit/test_connection.py:216: AssertionError
  ------------------------------ Captured log call -------------------------------
  2025-06-19 14:32:29.361 DEBUG [protocol_features:62]: Parsing sharding info from message options {'COMPRESSION': ['snappy', 'lz4']}
  2025-06-19 14:32:29.361 DEBUG [connection:1352]: Received options response on new connection (139701745826928) from 1.2.3.4:9042
  2025-06-19 14:32:29.362 DEBUG [connection:1403]: Snappy compression is not supported with protocol version 5 and checksumming. Consider installing lz4. Disabling compression.
  2025-06-19 14:32:29.362 DEBUG [connection:1417]: Sending StartupMessage on <Connection(139701745826928) 1.2.3.4:9042>

@mykaul
Copy link
Author

mykaul commented Jul 13, 2025

In #493 I've suggested (and implemented) removing protocols v1, v2. Here we need to decide if we wish to remove the higher versions.

@Lorak-mmk
Copy link

In #493 I've suggested (and implemented) removing protocols v1, v2. Here we need to decide if we wish to remove the higher versions.

I'm fine with removing DSE protocols, and all DSE functionality in general.

@mykaul mykaul marked this pull request as draft July 14, 2025 12:17
@mykaul mykaul force-pushed the max_supported_v5 branch 2 times, most recently from e1ef8db to ececd69 Compare July 16, 2025 13:57
@mykaul mykaul requested a review from Copilot July 16, 2025 15:14
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR restricts the driver to native protocol versions 3–5, makes V5 the default, and cleans up all DSE- and beta-protocol-specific paging logic and tests.

  • Limit supported protocol versions to V3–V5 and set default to V5.
  • Remove DSE/backpressured continuous-paging “next_pages” and queue-size behavior.
  • Update and prune unit/integration tests to drop references to removed protocols and adapt compression negotiation.

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
cassandra/init.py Trim SUPPORTED_VERSIONS to (V5, V4, V3) and update docstring
cassandra/cluster.py Change protocol_version default to V5; remove DSE‐paging support
cassandra/protocol.py Drop writing of queue size and backpressure support
tests/unit/test_protocol.py Simplify continuous-paging tests to only check unsupported cases
tests/unit/test_connection.py Adapt compression negotiation to protocol V5 behavior
tests/integration/init.py Update default/supported/unsupported protocol lists
tests/integration/* Prune DSE‐ and protocol-6 tests and related decorators
Comments suppressed due to low confidence (2)

cassandra/protocol.py:617

  • Since the code for writing max_queue_size has been removed, consider adding a unit test to verify that under protocol V5 only max_pages and max_pages_per_second are written and no queue-size field is emitted.
        write_int(f, paging_options.max_pages_per_second)

cassandra/init.py:175

  • [nitpick] The docstring is now misleading; it should be updated to accurately describe the current supported versions (V3–V5) rather than referring to “future v5 support.”
    A tuple of all supported protocol versions for ScyllaDB, including future v5 support.

@mykaul mykaul force-pushed the max_supported_v5 branch from ececd69 to fa48f97 Compare July 16, 2025 19:23
@mykaul
Copy link
Author

mykaul commented Jul 16, 2025

Ugh. Wrong rebase.

@mykaul mykaul force-pushed the max_supported_v5 branch from fa48f97 to ce73513 Compare July 16, 2025 20:04
@mykaul
Copy link
Author

mykaul commented Jul 16, 2025

TODO: need to remove this continuous paging support.

@mykaul
Copy link
Author

mykaul commented Jul 17, 2025

TODO: need to remove this continuous paging support.

Done, to some extent. I think it's reasonable.

@mykaul
Copy link
Author

mykaul commented Jul 17, 2025

Flakiness?

 ================================== FAILURES ===================================
  ________________ TestTwistedTimer.test_multi_timer_validation _________________
  
  self = <tests.unit.io.test_twistedreactor.TestTwistedTimer testMethod=test_multi_timer_validation>
  
      def test_multi_timer_validation(self):
          """
          Verify that timer timeouts are honored appropriately
          """
          # Tests timers submitted in order at various timeouts
          submit_and_wait_for_completion(self, self.create_timer, 0, 100, 1, 100)
          # Tests timers submitted in reverse order at various timeouts
  >       submit_and_wait_for_completion(self, self.create_timer, 100, 0, -1, 100)
  
  D:\a\python-driver\python-driver\tests\unit\io\utils.py:166: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  D:\a\python-driver\python-driver\tests\unit\io\utils.py:131: in submit_and_wait_for_completion
      unit_test.assertAlmostEqual(callback.expected_wait, callback.get_wait_time(), delta=.15)
  E   AssertionError: 0.09 != 0.2443854808807373 within 0.15 delta (0.1543854808807373 difference)

@mykaul mykaul marked this pull request as ready for review July 17, 2025 16:36
@mykaul mykaul force-pushed the max_supported_v5 branch from a94306b to b1fb07d Compare July 20, 2025 08:37
@mykaul
Copy link
Author

mykaul commented Jul 20, 2025

Rebased, passed CI. I think it's OK now.

@mykaul
Copy link
Author

mykaul commented Jul 27, 2025

If someone can review this, that'd be great. Not urgent.

@mykaul
Copy link
Author

mykaul commented Aug 4, 2025

If someone can review this, that'd be great. Not urgent.

Same request, still not urgent.
However, I do see we haven't released a new Python driver for ages and we've accumulated tens of patches since last release (3.29.3-scylla...master ) - it would be good to get this in before the/a release.

Copy link

@nyh nyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's some problem with how these two patches are split. The first patch claims it doesn't remove code, and yet it does remove a bunch of "continuous paging" and "graph paging" (whatever the heck that is) from the code - which later seems to be the focus of the second patch. Can you perhaps re-split the patches to better fit the descriptions?

Why did you want to remove the continuous paging stuff? What have we got to lose by having in the driver some DSE-only code? I know we don't recommend anyone using it, but if in the future we'll ever want to merge back with the upstream driver it will be easier if we don't change random pieces or the driver.
Or have we already given up the hope of merging from or to the upstream?

@mykaul mykaul force-pushed the max_supported_v5 branch from b1fb07d to 6ecf818 Compare August 4, 2025 15:39
Copy link

@nyh nyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with this PR, but it should be reviewed (and merged) by the drivers developers.

@mykaul
Copy link
Author

mykaul commented Aug 6, 2025

@scylladb/python-driver-maint - please review (not urgent).

@mykaul
Copy link
Author

mykaul commented Aug 10, 2025

I'm tempted to remove some more code, such as making monitor_reporting_enabled False, and some more... Please merge before I start pushing more code changes.

Copy link
Collaborator

@dkropachev dkropachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, one question though

mykaul added 2 commits August 12, 2025 12:38
The main goal of this patch is to have the driver start negotiating protocol versions at 5, and not begin by attempting DSE-specific protocols or protocol version 6,
which Scylla doesn't support and cause protocol errors which only wastes time in the beginning of each connection.
But then, since the driver won't even try DSE-specific protocols, we don't need to support them and can begin to remove DSE-specific code.
This patch only removes some obvious things, but there's still a bunch of DSE-specific code that hasn't been removed yet.

Refs: scylladb#244

Signed-off-by: Yaniv Kaul <[email protected]>
I'm not removing it entirely from the code, but ignoring it where it matters,
so it's essentially useless.

I think it's a reasonable balance between breaking the API of some classes
and removal of the feature.

Signed-off-by: Yaniv Kaul <[email protected]>
Per review comments, reverted the changes to test, but changed it to explicitly use v4 of the protocol.
@dkropachev dkropachev merged commit 8fec502 into scylladb:master Aug 12, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants