Skip to content

Commit

Permalink
[FLINK-12212][docs] Clarify that operator state is checkpointed async…
Browse files Browse the repository at this point in the history
…hronously

This closes apache#8185.
  • Loading branch information
knaufk authored and StefanRRichter committed Apr 17, 2019
1 parent a505838 commit 046d752
Showing 1 changed file with 8 additions and 12 deletions.
20 changes: 8 additions & 12 deletions docs/ops/state/large_state_tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,22 +100,18 @@ number of network buffers used per outgoing/incoming channel is limited and thus
may be configured without affecting checkpoint times
(see [network buffer configuration](../config.html#configuring-the-network-buffers)).

## Make state checkpointing Asynchronous where possible
## Asynchronous Checkpointing

When state is *asynchronously* snapshotted, the checkpoints scale better than when the state is *synchronously* snapshotted.
Especially in more complex streaming applications with multiple joins, Co-functions, or windows, this may have a profound
Especially in more complex streaming applications with multiple joins, co-functions, or windows, this may have a profound
impact.

To get state to be snapshotted asynchronously, applications have to do two things:
For state to be snapshotted asynchronsously, you need to use a state backend which supports asynchronous snapshotting.
Starting from Flink 1.3, both RocksDB-based as well as heap-based state backends (`filesystem`) support asynchronous
snapshotting and use it by default. This applies to to both managed operator state as well as managed keyed state (incl. timers state).

1. Use state that is [managed by Flink](../../dev/stream/state/state.html): Managed state means that Flink provides the data
structure in which the state is stored. Currently, this is true for *keyed state*, which is abstracted behind the
interfaces like `ValueState`, `ListState`, `ReducingState`, ...

2. Use a state backend that supports asynchronous snapshots. In Flink 1.2, only the RocksDB state backend uses
fully asynchronous snapshots. Starting from Flink 1.3, heap-based state backends also support asynchronous snapshots.

The above two points imply that large state should generally be kept as keyed state, not as operator state.
<span class="label label-info">Note</span> *The combination RocksDB state backend with heap-based timers currently does NOT support asynchronous snapshots for the timers state.
Other state like keyed state is still snapshotted asynchronously. Please note that this is not a regression from previous versions and will be resolved with `FLINK-10026`.*

## Tuning RocksDB

Expand Down Expand Up @@ -150,7 +146,7 @@ timers, while storing timers inside RocksDB offers higher scalability as the num
When using RockDB as state backend, the type of timer storage can be selected through Flink's configuration via option key `state.backend.rocksdb.timer-service.factory`.
Possible choices are `heap` (to store timers on the heap, default) and `rocksdb` (to store timers in RocksDB).

<span class="label label-info">Note</span> *The combination RocksDB state backend / with incremental checkpoint / with heap-based timers currently does NOT support asynchronous snapshots for the timers state.
<span class="label label-info">Note</span> *The combination RocksDB state backend with heap-based timers currently does NOT support asynchronous snapshots for the timers state.
Other state like keyed state is still snapshotted asynchronously. Please note that this is not a regression from previous versions and will be resolved with `FLINK-10026`.*

**Predefined Options**
Expand Down

0 comments on commit 046d752

Please sign in to comment.