Moonlink is an Iceberg-native ingestion engine bringing streaming inserts and upserts to your lakehouse.

Ingest Postgres CDC, event streams (Kafka), and OTEL into Iceberg without complex maintenance and compaction.

Moonlink buffers, caches, and indexes data so Iceberg tables stay read-optimized.

             ┌──────────moonlink───────────┐                         
             │  ┌───────────────────────┐  │  ┌───────Iceberg───────┐
             │  │                       │  │  │      obj. store     │
Postgres ───►│  │┌ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ┐│  │  │┌───────┐ ┌─────────┐│
             │  │                       │  │  ││       │ │         ││
Kafka    ───►│  ││  index  │ │  cache  ││  ├──►│ index │ │ parquet ││
             │  │                       │  │  ││       │ │         ││
Events   ───►│  │└ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ┘│  │  │└───────┘ └─────────┘│
             │  │                  nvme │  │  │                     │
             │  └───────────────────────┘  │  └─────────────────────┘
             └─────────────────────────────┘                         

Note: Moonlink is in preview. Expect changes. Join our Community to stay updated!

Traditional ingestion tools write data and metadata files per update into Iceberg. That's fine for slow-changing data, but on real-time streams it causes:

• Tiny data files — frequent commits create thousands of small Parquet files
• Metadata explosion — equality-deletes compound this problem

which leads to:

• Slow read performance — query planning overhead scales with file count
• Manual maintenance — periodic Spark jobs for compaction/cleanup

Moonlink minimizes write amplification and metadata churn by buffering incoming data, building indexes and caches on NVMe, and committing read-optimized files and deletion vectors to Iceberg.

Inserts are buffered and flushed as size-tuned Parquet

          ┌───moonlink───┐  ┌────iceberg───┐
          │┌─ ─ ─ ─ ─ ─ ┐│  │┌─ ─ ─ ─ ─ ─ ┐│
raw insert│              │  │              │
────────► ││   Arrow    │├─►││   Parquet  ││
          │              │  │              │
          │└─ ─ ─ ─ ─ ─ ┘│  │└─ ─ ─ ─ ─ ─ ┘│
          └──────────────┘  └──────────────┘

Deletes are mapped to deletion vectors using an index built on row positions

           ┌───moonlink───┐   ┌────iceberg───┐
           │┌─ ─ ─── ─ ─ ┐│   │┌─ ─ ─ ─ ─ ─ ┐│
raw deletes│              │   │              │
  ────────►││   index    │├──►││  deletion  ││
           │              │   │   vectors    │
           │└─ ─ ─── ─ ─ ┘│   │└─ ─ ─ ─ ─ ─ ┘│
           └──────────────┘   └──────────────┘

Moonlink supports multiple input sources for ingest:

1. PostgreSQL CDC — ingest via logical replication with millisecond-level latency
2. REST API — simple HTTP endpoint for direct event ingestion
3. Kafka — sink support coming soon
4. OTEL — sink support on the roadmap

Moonlink commits data as Iceberg v3 tables with deletion vectors. These tables can be queried from any Iceberg-compatible engine.

Engines

1. DuckDB
2. Apache Spark
3. Postgres with pg_duckdb or pg_mooncake

Catalogs

1. AWS Glue — coming soon
2. Unity Catalog — coming soon

For workloads requiring sub-second visibility into new data, Moonlink supports real-time querying:

1. DuckDB — with the duckb_mooncake extension.
2. Postgres — with the pg_mooncake extension.
3. DataFusion – with Moonlink Datafusion

Clone the repository and build the service binary:

git clone https://github.com/Mooncake-Labs/moonlink.git
cd moonlink
cargo build --release --bin moonlink_service

Start the Moonlink service, which will store data in the ./data directory:

./target/release/moonlink_service ./data

Check that the service is running properly:

curl http://localhost:3030/health

Create a table with a defined schema. Here's an example creating a users table:

curl -X POST http://localhost:3030/tables/users \
  -H "Content-Type: application/json" \
  -d '{
    "database": "my_database",
    "table": "users",
    "schema": [
      {"name": "id", "data_type": "int32", "nullable": false},
      {"name": "name", "data_type": "string", "nullable": false},
      {"name": "email", "data_type": "string", "nullable": true},
      {"name": "age", "data_type": "int32", "nullable": true},
      {"name": "created_at", "data_type": "date32", "nullable": true}
    ],
    "table_config": {}
  }'

Insert data into the created table:

curl -X POST http://localhost:3030/ingest/users \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "insert",
    "request_mode": "async",
    "data": {
      "id": 1,
      "name": "Alice Johnson",
      "email": "alice@example.com",
      "age": 30,
      "created_at": "2024-01-01"
    }
  }'

Roadmap (near‑term):

1. Kafka sink preview
2. Schema evolution from Postgres and Kafka
3. Catalog integrations (AWS Glue, Unity Catalog)
4. REST API stabilization (Insert, Upsert into Iceberg directly)

We’re grateful for our contributors. If you'd like to help improve Moonlink, join our community.

🥮