Skip to content

Commit

Permalink
[FLINK-11542][config][docs] Extend memory configuration descriptions
Browse files Browse the repository at this point in the history
  • Loading branch information
@zentol
zentol committed Mar 7, 2019
1 parent fd04309 commit 76bf54e
Show file tree
Hide file tree
Showing 4 changed files with 76 additions and 37 deletions.
25 changes: 0 additions & 25 deletions docs/_includes/generated/task_manager_configuration.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,31 +67,6 @@
<td style="word-wrap: break-word;">false</td>
<td>Whether to kill the TaskManager when the task thread throws an OutOfMemoryError.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.fraction</h5></td>
<td style="word-wrap: break-word;">0.7</td>
<td>The relative amount of memory (after subtracting the amount of memory used by network buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results. For example, a value of `0.8` means that a task manager reserves 80% of its memory for internal data buffers, leaving 20% of free memory for the task manager's heap for objects created by user-defined functions. This parameter is only evaluated, if taskmanager.memory.size is not set.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.off-heap</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Memory allocation method (JVM heap or off-heap), used for managed memory of the TaskManager as well as the network buffers.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.preallocate</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.segment-size</h5></td>
<td style="word-wrap: break-word;">"32kb"</td>
<td>Size of memory buffers used by the network stack and the memory manager.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.size</h5></td>
<td style="word-wrap: break-word;">"0"</td>
<td>Amount of memory to be allocated by the task manager's memory manager. If not set, a relative fraction will be allocated.</td>
</tr>
<tr>
<td><h5>taskmanager.network.bind-policy</h5></td>
<td style="word-wrap: break-word;">"ip"</td>
Expand Down
36 changes: 36 additions & 0 deletions docs/_includes/generated/task_manager_memory_configuration.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 20%">Key</th>
<th class="text-left" style="width: 15%">Default</th>
<th class="text-left" style="width: 65%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>taskmanager.memory.fraction</h5></td>
<td style="word-wrap: break-word;">0.7</td>
<td>The relative amount of memory (after subtracting the amount of memory used by network buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results. For example, a value of <span markdown="span">`0.8`</span> means that a task manager reserves 80% of its memory (on-heap or off-heap depending on taskmanager.memory.off-heap) for internal data buffers, leaving 20% of free memory for the task manager's heap for objects created by user-defined functions. This parameter is only evaluated, if taskmanager.memory.size is not set.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.off-heap</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Memory allocation method (JVM heap or off-heap), used for managed memory of the TaskManager. For setups with larger quantities of memory, this can improve the efficiency of the operations performed on the memory.<br />When set to true, then it is advised that <span markdown="span">`taskmanager.memory.preallocate`</span> is also set to true.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.preallocate</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting. When <span markdown="span">`taskmanager.memory.off-heap`</span> is set to true, then it is advised that this configuration is also set to true. If this configuration is set to false cleaning up of the allocated off-heap memory happens only when the configured JVM parameter MaxDirectMemorySize is reached by triggering a full GC. For streaming setups is is highly recommended to set this value to false as the core state backends currently do not use the managed memory.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.segment-size</h5></td>
<td style="word-wrap: break-word;">"32kb"</td>
<td>Size of memory buffers used by the network stack and the memory manager.</td>
</tr>
<tr>
<td><h5>taskmanager.memory.size</h5></td>
<td style="word-wrap: break-word;">"0"</td>
<td>The amount of memory (in megabytes) that the task manager reserves on-heap or off-heap (depending on taskmanager.memory.off-heap) for sorting, hash tables, and caching of intermediate results. If unspecified, the memory manager will take a fixed ratio with respect to the size of the task manager JVM as specified by taskmanager.memory.fraction.</td>
</tr>
</tbody>
</table>
6 changes: 6 additions & 0 deletions docs/ops/config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

{% include generated/task_manager_configuration.html %}

For *batch* jobs (or if `taskmanager.memoy.preallocate` is enabled) Flink allocates a fraction of 0.7 of the free memory (total memory configured via taskmanager.heap.mb minus memory used for network buffers) for its managed memory. Managed memory helps Flink to run the batch operators efficiently. It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations. If Flink runs out of managed memory, it utilizes disk space. Using managed memory, some operations can be performed directly on the raw data without having to deserialize the data to convert it into Java objects. All in all, managed memory improves the robustness and speed of the system.

The default fraction for managed memory can be adjusted using the taskmanager.memory.fraction parameter. An absolute value may be set using taskmanager.memory.size (overrides the fraction parameter). If desired, the managed memory may be allocated outside the JVM heap. This may improve performance in setups with large memory sizes.

{% include generated/task_manager_memory_configuration.html %}

### Distributed Coordination

{% include generated/cluster_configuration.html %}
Expand Down
46 changes: 34 additions & 12 deletions flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,24 @@
package org.apache.flink.configuration;

import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.annotation.docs.ConfigGroup;
import org.apache.flink.annotation.docs.ConfigGroups;
import org.apache.flink.annotation.docs.Documentation;
import org.apache.flink.configuration.description.Description;

import static org.apache.flink.configuration.ConfigOptions.key;
import static org.apache.flink.configuration.description.TextElement.code;
import static org.apache.flink.configuration.description.TextElement.text;

/**
* The set of configuration options relating to TaskManager and Task settings.
*/
@PublicEvolving
@ConfigGroups(groups = @ConfigGroup(name = "TaskManagerMemory", keyPrefix = "taskmanager.memory"))
public class TaskManagerOptions {

private static final String MANAGED_MEMORY_PRE_ALLOCATE_KEY = "taskmanager.memory.preallocate";

// ------------------------------------------------------------------------
// General TaskManager Options
// ------------------------------------------------------------------------
Expand Down Expand Up @@ -206,8 +212,10 @@ public class TaskManagerOptions {
public static final ConfigOption<String> MANAGED_MEMORY_SIZE =
key("taskmanager.memory.size")
.defaultValue("0")
.withDescription("Amount of memory to be allocated by the task manager's memory manager." +
" If not set, a relative fraction will be allocated.");
.withDescription("The amount of memory (in megabytes) that the task manager reserves on-heap or off-heap" +
" (depending on taskmanager.memory.off-heap) for sorting, hash tables, and caching of intermediate" +
" results. If unspecified, the memory manager will take a fixed ratio with respect to the size of" +
" the task manager JVM as specified by taskmanager.memory.fraction.");

/**
* Fraction of free memory allocated by the memory manager if {@link #MANAGED_MEMORY_SIZE} is
Expand All @@ -216,12 +224,15 @@ public class TaskManagerOptions {
public static final ConfigOption<Float> MANAGED_MEMORY_FRACTION =
key("taskmanager.memory.fraction")
.defaultValue(0.7f)
.withDescription("The relative amount of memory (after subtracting the amount of memory used by network" +
" buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results." +
" For example, a value of `0.8` means that a task manager reserves 80% of its memory" +
" for internal data buffers, leaving 20% of free memory for the task manager's heap for objects" +
" created by user-defined functions. This parameter is only evaluated, if " + MANAGED_MEMORY_SIZE.key() +
" is not set.");
.withDescription(new Description.DescriptionBuilder()
.text("The relative amount of memory (after subtracting the amount of memory used by network" +
" buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results." +
" For example, a value of %s means that a task manager reserves 80% of its memory" +
" (on-heap or off-heap depending on taskmanager.memory.off-heap)" +
" for internal data buffers, leaving 20% of free memory for the task manager's heap for objects" +
" created by user-defined functions. This parameter is only evaluated, if " + MANAGED_MEMORY_SIZE.key() +
" is not set.", code("0.8"))
.build());

/**
* Memory allocation method (JVM heap or off-heap), used for managed memory of the TaskManager
Expand All @@ -230,16 +241,27 @@ public class TaskManagerOptions {
public static final ConfigOption<Boolean> MEMORY_OFF_HEAP =
key("taskmanager.memory.off-heap")
.defaultValue(false)
.withDescription("Memory allocation method (JVM heap or off-heap), used for managed memory of the" +
" TaskManager as well as the network buffers.");
.withDescription(Description.builder()
.text("Memory allocation method (JVM heap or off-heap), used for managed memory of the" +
" TaskManager. For setups with larger quantities of memory, this can" +
" improve the efficiency of the operations performed on the memory.")
.linebreak()
.text("When set to true, then it is advised that %s is also set to true.", code(MANAGED_MEMORY_PRE_ALLOCATE_KEY))
.build());

/**
* Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting.
*/
public static final ConfigOption<Boolean> MANAGED_MEMORY_PRE_ALLOCATE =
key("taskmanager.memory.preallocate")
key(MANAGED_MEMORY_PRE_ALLOCATE_KEY)
.defaultValue(false)
.withDescription("Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting.");
.withDescription(Description.builder()
.text("Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting." +
" When %s is set to true, then it is advised that this configuration is also" +
" set to true. If this configuration is set to false cleaning up of the allocated off-heap memory" +
" happens only when the configured JVM parameter MaxDirectMemorySize is reached by triggering a full" +
" GC. For streaming setups is is highly recommended to set this value to false as the core state" +
" backends currently do not use the managed memory.", code(MEMORY_OFF_HEAP.key())).build());

// ------------------------------------------------------------------------
// Network Options
Expand Down

0 comments on commit 76bf54e

Please sign in to comment.