Apache Paimon is an open-source data lake storage framework designed to manage large-scale analytical datasets efficiently. It enables organizations to build reliable, real-time data lakes that support streaming and batch processing while ensuring transactional consistency. Unlike traditional file-based data lakes, Apache Paimon provides features such as primary key support, schema evolution, time travel, snapshots, and efficient data compaction, making it suitable for modern data engineering and analytics workloads. The Apache Paimon Java API allows developers to interact with Paimon tables directly from Java applications without relying solely on SQL engines like Apache Flink or Apache Spark. Using the Java API, developers can programmatically create warehouses, databases, and tables, insert and modify records, execute queries, and manage metadata. This level of control makes the API particularly useful for building custom ETL pipelines, data ingestion services, microservices, and enterprise applications that require direct access to data lake storage. Whether you’re developing a standalone Java application or integrating Paimon into an existing data platform, the Java API offers a flexible and type-safe way to perform common data management operations while leveraging all of Paimon’s transactional capabilities. Let us delve into this Java Apache Paimon guide to explore the fundamentals of the Apache Paimon Java API, understand its architecture, and learn how to build and manage transactional data lake applications using Java.
1. Understanding Apache Paimon Java API
The Apache Paimon Java API is a collection of classes and interfaces that provide programmatic access to Paimon’s storage engine, enabling developers to interact with data lake tables using Java instead of SQL statements. Through the API, developers can create catalogs, define schemas, write records, read data, update existing rows, and delete records programmatically. A typical workflow begins by configuring a Catalog, which serves as the entry point for managing databases and tables by connecting to a warehouse location where Paimon stores its metadata and data files. Once the catalog is configured, developers can create databases, define table schemas, and perform various data operations using familiar Java objects. The Java API organizes these capabilities into several core components that simplify the development of scalable and transactional data lake applications.
- Catalog – Manages databases, tables, and metadata.
- Schema – Defines table columns, data types, primary keys, and table properties.
- Table – Represents a Paimon table and provides access to read and write operations.
- Write API – Inserts, updates, and deletes records while maintaining ACID guarantees.
- Read API – Retrieves records from tables for reporting or analytical purposes.
- Snapshot Management – Maintains table versions for time travel and rollback.
- Options and Configuration – Configures warehouses, storage backends, file formats, and optimization settings.
One of the major advantages of the Apache Paimon Java API is its support for primary key tables, allowing developers to update records by simply writing a new record with the same primary key instead of executing SQL UPDATE statements. Apache Paimon automatically merges the latest data while maintaining transactional consistency, and delete operations can also be performed using the primary key, simplifying record management. Since Paimon stores data in immutable files, updates and deletes are internally managed through merge operations and snapshots rather than modifying files directly, improving scalability, reliability, concurrent access, and storage efficiency while preserving ACID guarantees. Additionally, the Java API is storage agnostic, enabling developers to use local file systems during development or cloud storage platforms such as Amazon S3, Azure Data Lake Storage (ADLS), and Google Cloud Storage (GCS) in production environments. This flexibility allows applications to seamlessly scale from local testing to enterprise-grade data lake deployments with minimal code changes. Overall, the Apache Paimon Java API provides a powerful, type-safe, and developer-friendly interface for building modern data lake applications while leveraging advanced features such as schema evolution, snapshots, streaming ingestion, and transactional consistency.
2. Code Example
Now that we understand the fundamentals of the Apache Paimon Java API, let’s implement a complete example. The following program demonstrates the complete lifecycle of working with a Paimon table, including creating a warehouse, creating a database and table, inserting records, querying data, updating existing records, and deleting records.
// ApachePaimonJavaApiExample.java
package com.example.paimon;
import org.apache.paimon.catalog.Catalog;
import org.apache.paimon.catalog.CatalogContext;
import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.catalog.FileSystemCatalog;
import org.apache.paimon.data.BinaryRow;
import org.apache.paimon.data.GenericRow;
import org.apache.paimon.data.InternalRow;
import org.apache.paimon.disk.IOManager;
import org.apache.paimon.fs.Path;
import org.apache.paimon.options.Options;
import org.apache.paimon.reader.RecordReader;
import org.apache.paimon.schema.Schema;
import org.apache.paimon.table.FileStoreTable;
import org.apache.paimon.table.Table;
import org.apache.paimon.table.sink.BatchTableWrite;
import org.apache.paimon.table.sink.BatchTableCommit;
import org.apache.paimon.table.source.ReadBuilder;
import org.apache.paimon.table.source.TableRead;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataTypes;
import java.util.Arrays;
import java.util.Collections;
/**
* Complete Apache Paimon Java API Example
*
* Demonstrates:
* 1. Create a warehouse
* 2. Create database
* 3. Create table
* 4. Insert records
* 5. Query records
* 6. Update a record
* 7. Delete a record
*/
public class ApachePaimonJavaApiExample {
public static void main(String[] args) throws Exception {
// ---------------------------------------------------------------------
// Configure Warehouse
// ---------------------------------------------------------------------
Options options = new Options();
options.set("warehouse", "file:///tmp/paimon-warehouse");
Catalog catalog = new FileSystemCatalog(
CatalogContext.create(options));
// ---------------------------------------------------------------------
// Create Database
// ---------------------------------------------------------------------
String database = "company";
if (!catalog.databaseExists(database)) {
catalog.createDatabase(database, true);
}
// ---------------------------------------------------------------------
// Create Table
// ---------------------------------------------------------------------
Identifier tableId =
Identifier.create(database, "employees");
if (!catalog.tableExists(tableId)) {
Schema schema = new Schema(
Arrays.asList(
new DataField(0, "id", DataTypes.INT()),
new DataField(1, "name", DataTypes.STRING()),
new DataField(2, "department", DataTypes.STRING()),
new DataField(3, "salary", DataTypes.DOUBLE())
),
Collections.singletonList("id"),
Collections.emptyList(),
Collections.emptyMap()
);
catalog.createTable(tableId, schema, false);
}
FileStoreTable table =
(FileStoreTable) catalog.getTable(tableId);
// ---------------------------------------------------------------------
// INSERT RECORDS
// ---------------------------------------------------------------------
BatchTableWrite write = table.newBatchWriteBuilder().newWrite();
write.write(GenericRow.of(
101,
"John",
"Engineering",
85000.0));
write.write(GenericRow.of(
102,
"Alice",
"HR",
65000.0));
write.write(GenericRow.of(
103,
"David",
"Finance",
72000.0));
BatchTableCommit commit =
table.newBatchWriteBuilder().newCommit();
commit.commit(write.prepareCommit());
System.out.println("========== Initial Data ==========");
printTable(table);
// ---------------------------------------------------------------------
// UPDATE RECORD
// ---------------------------------------------------------------------
// Paimon performs updates by writing another row with the same
// primary key.
BatchTableWrite updateWrite =
table.newBatchWriteBuilder().newWrite();
updateWrite.write(GenericRow.of(
101,
"John",
"Architecture",
95000.0));
BatchTableCommit updateCommit =
table.newBatchWriteBuilder().newCommit();
updateCommit.commit(updateWrite.prepareCommit());
System.out.println("\n========== After Update ==========");
printTable(table);
// ---------------------------------------------------------------------
// DELETE RECORD
// ---------------------------------------------------------------------
// Delete is performed using delete() with primary key.
BatchTableWrite deleteWrite =
table.newBatchWriteBuilder().newWrite();
deleteWrite.delete(GenericRow.of(102));
BatchTableCommit deleteCommit =
table.newBatchWriteBuilder().newCommit();
deleteCommit.commit(deleteWrite.prepareCommit());
System.out.println("\n========== After Delete ==========");
printTable(table);
System.out.println("\nCompleted Successfully.");
}
/**
* Reads all records from the table.
*/
private static void printTable(FileStoreTable table) throws Exception {
ReadBuilder builder = table.newReadBuilder();
TableRead read = builder.newRead();
RecordReader<InternalRow> reader =
read.createReader(builder.newScan().plan());
RecordReader.RecordIterator<InternalRow> iterator;
while ((iterator = reader.readBatch()) != null) {
InternalRow row;
while ((row = iterator.next()) != null) {
System.out.printf(
"ID=%d, Name=%s, Department=%s, Salary=%.2f%n",
row.getInt(0),
row.getString(1),
row.getString(2),
row.getDouble(3)
);
}
}
}
}
Having examined the implementation, let’s understand how each part of the program works and how the Apache Paimon Java API performs various data lake operations behind the scenes.
2.1 Code Explanation
The above example demonstrates the complete lifecycle of working with the Apache Paimon Java API using a file system-based warehouse. It begins by configuring the warehouse location through the Options object and creating a FileSystemCatalog, which acts as the entry point for managing databases and tables. The program then checks whether the company database exists and creates it if necessary. Next, it creates an employees table by defining a schema containing four columns: id, name, department, and salary, with id configured as the primary key. After retrieving the table instance, the code creates a batch writer and inserts three employee records before committing the transaction to make the data persistent. The printTable() method demonstrates how to query data by creating a table scan, reading batches of records using a RecordReader, and printing each row to the console. To update an employee, the program writes another record with the same primary key but modified values, allowing Apache Paimon to automatically merge the latest version while maintaining ACID guarantees. The delete operation is performed by invoking the delete() method with the primary key of the record to be removed, followed by another commit. Finally, the application queries the table after each operation to display the latest state of the data, illustrating how Apache Paimon supports creating data lakes, inserting records, querying data, updating existing records, and deleting records programmatically through its Java API.
2.2 Code Output
After executing the application, the console displays the result of each CRUD operation, making it easy to verify that the data has been inserted, updated, queried, and deleted successfully.
========== Initial Data ========== ID=101, Name=John, Department=Engineering, Salary=85000.00 ID=102, Name=Alice, Department=HR, Salary=65000.00 ID=103, Name=David, Department=Finance, Salary=72000.00 ========== After Update ========== ID=101, Name=John, Department=Architecture, Salary=95000.00 ID=102, Name=Alice, Department=HR, Salary=65000.00 ID=103, Name=David, Department=Finance, Salary=72000.00 ========== After Delete ========== ID=101, Name=John, Department=Architecture, Salary=95000.00 ID=103, Name=David, Department=Finance, Salary=72000.00 Completed Successfully.
The output demonstrates each stage of the CRUD operations performed on the Apache Paimon table. Initially, three employee records are successfully inserted into the employees table and displayed under the Initial Data section. Next, the record with the primary key 101 is updated by changing the employee’s department from Engineering to Architecture and increasing the salary from 85000.00 to 95000.00, while the remaining records remain unchanged. In the final stage, the employee with the primary key 102 is deleted from the table, leaving only the records for employees 101 and 103. The message “Completed Successfully.” confirms that all operations, including table creation, record insertion, querying, updating, deletion, and transaction commits, were executed successfully without any errors.
3. Conclusion
The Apache Paimon Java API provides a robust and flexible way to manage data lake tables directly from Java applications, enabling developers to create catalogs, define schemas, insert records, query data, update existing rows, and delete records using a clean object-oriented programming model. With built-in support for ACID transactions, primary key updates, schema evolution, snapshots, and scalable storage backends, Apache Paimon simplifies the development of modern data engineering solutions that combine both streaming and batch processing, making it an excellent choice for building enterprise-grade ETL pipelines, real-time analytics platforms, and custom data management services. As organizations continue to embrace lakehouse architectures, the Apache Paimon Java API serves as an essential toolkit for Java developers who want to build reliable, scalable, and high-performance applications while leveraging the full capabilities of Apache Paimon’s transactional data lake storage engine.
