This demo brings up Apache Flink with Apache Iceberg, together with Nessie as an Iceberg transactional catalog and MinIO as a storage backend.
You need to have Docker installed before running this demo.
Clone this repository, cd into flink-iceberg-demo directory, and start up the demo.
git clone [email protected]:vontikov/flink-iceberg-demo.git
cd flink-iceberg-demo
docker compose upOpen a new terminal and run Flink SQL client
docker exec -it flink-sql-client sql-clientNow you can create a new table via set of SQL queries
CREATE CATALOG nessie
WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.nessie.NessieCatalog',
'uri'='https://nessie:19120/api/v1',
'ref'='main',
'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
'warehouse' = 's3:https://warehouse',
's3.endpoint'='https://minio:9000',
's3.path-style-access' = 'true'
);
USE CATALOG nessie;
CREATE DATABASE my_db
WITH (
'foo'='bar'
);
USE my_db;
CREATE TABLE my_table (
id BIGINT,
name STRING,
age INT
) PARTITIONED BY (
age
) WITH (
'foo'='bar'
);Insert some data into the table
INSERT INTO my_table
VALUES
(1, 'Bob', 42),
(2, 'Alice', 24),
(3, 'James', 35),
(4, 'Carter', 57),
(5, 'Avery', 30);Once the table is populated you can get the results
SELECT * FROM my_table;Open MinIO UI and explore objects created by the queries in the bucket
warehouse
open https://localhost:9001You can see Nessie catalog here
open https://localhost:19120And here you can Flink completed tasks
open https://localhost:8081