feat: support direct load (#55)

oceanbase · Feb 4, 2024 · d1433b5 · d1433b5
1 parent 3bf0b03
commit d1433b5
Show file tree

Hide file tree

Showing 21 changed files with 781 additions and 258 deletions.
diff --git a/docs/sink/flink-connector-oceanbase.md b/docs/sink/flink-connector-oceanbase.md
@@ -158,26 +158,31 @@ Once executed, the records should have been written to OceanBase.
 
 ## Configuration
 
-| Option | Required by Table API | Required by DataStream | Default | Type | Description |
-|--------------------------|-----------------------|------------------------|--------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------|
-| url | Yes | Yes | | String | JDBC url. |
-| username | Yes | Yes | | String | The username. |
-| password | Yes | Yes | | String | The password. |
-| schema-name | Yes | Not supported | | String | The schema name or database name. |
-| table-name | Yes | Not supported | | String | The table name. |
-| compatible-mode | No | No | mysql | String | The compatible mode of OceanBase, can be 'mysql' or 'oracle'. |
-| driver-class-name | No | No | com.mysql.cj.jdbc.Driver | String | The driver class name, use 'com.mysql.cj.jdbc.Driver' by default. If other value is set, you need to introduce the driver manually. |
-| cluster-name | No | No | | String | The cluster name of OceanBase, required when 'partition.enabled' is 'true'. |
-| tenant-name | No | No | | String | The tenant name of OceanBase, required when 'partition.enabled' is 'true'. |
-| druid-properties | No | No | | String | Druid connection pool properties, multiple values are separated by semicolons. |
-| sync-write | No | No | false | Boolean | Whether to write data synchronously, will not use buffer if it's set to 'true'. |
-| buffer-flush.interval | No | No | 1s | Duration | Buffer flush interval. Set '0' to disable scheduled flushing. |
-| buffer-flush.buffer-size | No | No | 1000 | Integer | Buffer size. |
-| max-retries | No | No | 3 | Integer | Max retry times on failure. |
-| memstore-check.enabled | No | No | true | Boolean | Whether enable memstore check. |
-| memstore-check.threshold | No | No | 0.9 | Double | Memstore usage threshold ratio relative to the limit value. |
-| memstore-check.interval | No | No | 30s | Duration | Memstore check interval. |
-| partition.enabled | No | No | false | Boolean | Whether to enable partition calculation and flush records by partitions. |
+| Option | Required by Table API | Required by DataStream | Default | Type | Description |
+|-------------------------------|-----------------------|------------------------|--------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| url | Yes | Yes | | String | JDBC url. |
+| username | Yes | Yes | | String | The username. |
+| password | Yes | Yes | | String | The password. |
+| schema-name | Yes | Not supported | | String | The schema name or database name. |
+| table-name | Yes | Not supported | | String | The table name. |
+| driver-class-name | No | No | com.mysql.cj.jdbc.Driver | String | The driver class name, use 'com.mysql.cj.jdbc.Driver' by default. If other value is set, you need to introduce the driver manually. |
+| druid-properties | No | No | | String | Druid connection pool properties, multiple values are separated by semicolons. |
+| sync-write | No | No | false | Boolean | Whether to write data synchronously, will not use buffer if it's set to 'true'. |
+| buffer-flush.interval | No | No | 1s | Duration | Buffer flush interval. Set '0' to disable scheduled flushing. |
+| buffer-flush.buffer-size | No | No | 1000 | Integer | Buffer size. |
+| max-retries | No | No | 3 | Integer | Max retry times on failure. |
+| memstore-check.enabled | No | No | true | Boolean | Whether enable memstore check. |
+| memstore-check.threshold | No | No | 0.9 | Double | Memstore usage threshold ratio relative to the limit value. |
+| memstore-check.interval | No | No | 30s | Duration | Memstore check interval. |
+| partition.enabled | No | No | false | Boolean | Whether to enable partition calculation and flush records by partitions. Only works when 'sync-write' and 'direct-load.enabled' are 'false'. |
+| direct-load.enabled | No | No | false | Boolean | Whether to enable direct load. Note that direct load task requires the sink parallelism to be 1. |
+| direct-load.host | No | No | | String | The hostname or IP address used in direct load task. Required when 'direct-load.enabled' is true. |
+| direct-load.port | No | No | 2882 | Integer | The rpc port used in direct load task. Required when 'direct-load.enabled' is true. |
+| direct-load.parallel | No | No | 8 | Integer | Parallelism of direct load task. |
+| direct-load.max-error-rows | No | No | 0 | Long | Maximum tolerable number of error rows of direct load task. |
+| direct-load.dup-action | No | No | REPLACE | STRING | Action when there is duplicated record of direct load task. Can be 'STOP_ON_DUP', 'REPLACE' or 'IGNORE'. |
+| direct-load.timeout | No | No | 7d | Duration | Timeout for direct load task. |
+| direct-load.heartbeat-timeout | No | No | 30s | Duration | Client heartbeat timeout in direct load task. |
 
 ## References