Skip to content

Commit 9014153

Browse files
committed
DOCSP-31215 - startup mode (#170)
(cherry picked from commit 086edd3)
1 parent c0c26cf commit 9014153

File tree

1 file changed

+37
-17
lines changed

1 file changed

+37
-17
lines changed

source/configuration/read.txt

Lines changed: 37 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -289,9 +289,6 @@ You must specify this partitioner using the full classname:
289289

290290
This partitioner creates a single partition.
291291

292-
293-
294-
295292
.. _spark-change-stream-conf:
296293

297294
Change Streams
@@ -329,6 +326,20 @@ Change Streams
329326
see the MongoDB server manual guide
330327
:manual:`Lookup Full Document for Update Operation </changeStreams/#lookup-full-document-for-update-operations>`.
331328

329+
* - ``change.stream.micro.batch.max.partition.count``
330+
- | The maximum number of partitions the {+connector-short+} divides each
331+
micro-batch into. Spark workers can process these partitions in parallel.
332+
|
333+
| This setting applies only when using micro-batch streams.
334+
|
335+
| **Default**: ``1``
336+
337+
.. warning:: Event Order
338+
339+
Specifying a value larger than ``1`` can alter the order in which
340+
the {+connector-short+} processes change events. Avoid this setting
341+
if out-of-order processing could create data inconsistencies downstream.
342+
332343
* - ``change.stream.publish.full.document.only``
333344
- | Specifies whether to publish the changed document or the full
334345
change stream document.
@@ -348,20 +359,29 @@ Change Streams
348359
This setting overrides the ``change.stream.lookup.full.document``
349360
setting.
350361

351-
* - ``change.stream.micro.batch.max.partition.count``
352-
- | The maximum number of partitions the {+connector-short+} divides each
353-
micro-batch into. Spark workers can process these partitions in parallel.
354-
|
355-
| This setting applies only when using micro-batch streams.
356-
|
357-
| **Default**: ``1``
358-
359-
.. warning:: Event Order
360-
361-
Specifying a value larger than ``1`` can alter the order in which
362-
the {+connector-short+} processes change events. Avoid this setting
363-
if out-of-order processing could create data inconsistencies downstream.
364-
362+
* - ``change.stream.startup.mode``
363+
- | Specifies how the connector starts up when no offset is available.
364+
365+
| This setting accepts the following values:
366+
367+
- ``latest``: The connector begins processing
368+
change events starting with the most recent event.
369+
It will not process any earlier unprocessed events.
370+
- ``timestamp``: The connector begins processing change events at a specified time.
371+
372+
To use the ``timestamp`` option, you must specify a time by using the
373+
``change.stream.startup.mode.timestamp.start.at.operation.time`` setting.
374+
This setting accepts timestamps in the following formats:
375+
376+
- An integer representing the number of seconds since the
377+
:wikipedia:`Unix epoch <Unix_time>`
378+
- A date and time in
379+
`ISO-8601 <https://www.iso.org/iso-8601-date-and-time-format.html>`__
380+
format with one-second precision
381+
- An extended JSON ``BsonTimestamp``
382+
383+
**Default**: ``latest``
384+
365385
.. _configure-input-uri:
366386

367387
``connection.uri`` Configuration Setting

0 commit comments

Comments
 (0)