Skip to content

Commit 0b1b829

Browse files
authored
Docsp 31130 - batch partitions (#166)
1 parent 099521a commit 0b1b829

File tree

1 file changed

+16
-6
lines changed

1 file changed

+16
-6
lines changed

source/configuration/read.txt

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -295,9 +295,7 @@ Change Streams
295295
.. note::
296296

297297
If you use ``SparkConf`` to set the connector's change stream
298-
configurations, prefix ``spark.mongodb.change.stream.`` to each
299-
300-
property.
298+
configurations, prefix ``spark.mongodb.`` to each property.
301299

302300
.. list-table::
303301
:header-rows: 1
@@ -318,14 +316,14 @@ Change Streams
318316
original document and updated document as well as a copy of the
319317
entire updated document.
320318

319+
**Default:** "default"
320+
321321
.. tip::
322322

323323
For more information on how this change stream option works,
324324
see the MongoDB server manual guide
325325
:manual:`Lookup Full Document for Update Operation </changeStreams/#lookup-full-document-for-update-operations>`.
326326

327-
**Default:** "default"
328-
329327
* - ``change.stream.publish.full.document.only``
330328
- | Specifies whether to publish the changed document or the full
331329
change stream document.
@@ -338,14 +336,26 @@ Change Streams
338336
- If you don't specify a schema, the connector infers the schema
339337
from the change stream document rather than from the underlying collection.
340338

339+
**Default**: ``false``
340+
341341
.. note::
342342

343343
This setting overrides the ``change.stream.lookup.full.document``
344344
setting.
345345

346+
* - ``change.stream.micro.batch.max.partition.count``
347+
- | The maximum number of partitions the {+connector-short+} divides each
348+
micro-batch into. Spark workers can process these partitions in parallel.
349+
|
350+
| This setting applies only when using micro-batch streams.
346351
|
347-
| **Default**: ``false``
352+
| **Default**: ``1``
353+
354+
.. warning:: Event Order
348355

356+
Specifying a value larger than ``1`` can alter the order in which
357+
the {+connector-short+} processes change events. Avoid this setting
358+
if out-of-order processing could create data inconsistencies downstream.
349359

350360
.. _configure-input-uri:
351361

0 commit comments

Comments
 (0)