Arrow is memory format for flat and hierarchical data. It is a popular format used by various big data tools, among them BigQuery. One of the benefits that Arrow brings is that the format of the data has the same byte representation on the languages supported. So apart from the benefits of a columnar memory format there are also the benefits of zero-copy without the serialization overhead.
Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead. more
Let’s import the libraries
01 02 03 04 05 06 07 08 09 10 | <dependency> <groupId>org.apache.arrow</groupId> <artifactId>arrow-memory-netty</artifactId> <version>${arrow.version}</version> </dependency> <dependency> <groupId>org.apache.arrow</groupId> <artifactId>arrow-vector</artifactId> <version>${arrow.version}</version> </dependency> |
Before starting it is essential to understand that for Read/Write operations on Arrow, byte buffers are used. Operations like reading and writing is continuous exchange of bytes. To make this efficient Arrow comes with a buffer allocator, which can have a certain size or have an automatic expansion.
The libraries backing the allocation management is arrow-memory-netty and arrow-memory-unsafe. We shall use the netty one.
Storing Data in arrow requires a schema. Schemas can be defined programatically
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | package com.gkatzioura.arrow;import java.io.IOException;import java.util.List;import org.apache.arrow.vector.types.pojo.ArrowType;import org.apache.arrow.vector.types.pojo.Field;import org.apache.arrow.vector.types.pojo.FieldType;import org.apache.arrow.vector.types.pojo.Schema;public class SchemaFactory { public static Schema DEFAULT_SCHEMA = createDefault(); public static Schema createDefault() { var strField = new Field("col1", FieldType.nullable(new ArrowType.Utf8()), null); var intField = new Field("col2", FieldType.nullable(new ArrowType.Int(32, true)), null); return new Schema(List.of(strField, intField)); } public static Schema schemaWithChildren() { var amount = new Field("amount", FieldType.nullable(new ArrowType.Decimal(19,4,128)), null); var currency = new Field("currency",FieldType.nullable(new ArrowType.Utf8()), null); var itemField = new Field("item", FieldType.nullable(new ArrowType.Utf8()), List.of(amount,currency)); return new Schema(List.of(itemField)); } public static Schema fromJson(String jsonString) { try { return Schema.fromJSON(jsonString); } catch (IOException e) { throw new ArrowExampleException(e); } }} |
Also they have a parseable json representation.
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 | { "fields" : [ { "name" : "col1", "nullable" : true, "type" : { "name" : "utf8" }, "children" : [ ] }, { "name" : "col2", "nullable" : true, "type" : { "name" : "int", "bitWidth" : 32, "isSigned" : true }, "children" : [ ] } ]} |
Plus just like Avro you can have complex schemas and embedded values on a field.
1 2 3 4 5 6 7 | public static Schema schemaWithChildren() { var amount = new Field("amount", FieldType.nullable(new ArrowType.Decimal(19,4,128)), null); var currency = new Field("currency",FieldType.nullable(new ArrowType.Utf8()), null); var itemField = new Field("item", FieldType.nullable(new ArrowType.Utf8()), List.of(amount,currency)); return new Schema(List.of(itemField)); } |
On the next blog, we shall use the Streaming API for Arrow
Published on Java Code Geeks with permission by Emmanouil Gkatziouras, partner at our JCG program. See the original article here: Apache Arrow on the JVM: Get Started and Schemas Opinions expressed by Java Code Geeks contributors are their own. |
