Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0449562
feat(go): add row format null bitmap and alignment utils
ayush00git Jul 30, 2026
efdb6e6
fix: refactor in-line comments
ayush00git Jul 30, 2026
86774ce
feat(go): define fixed-width and variable-width values
ayush00git Jul 30, 2026
0c70062
feat(go): add writer to write data byte blob to the buffer
ayush00git Jul 30, 2026
9680d57
feat(go): add row reader
ayush00git Jul 30, 2026
81ae126
feat(test): add array and map reader tests
ayush00git Jul 30, 2026
6005b88
feat(schema): add schema bytes methods to interop with Java
ayush00git Jul 30, 2026
cc91502
fix: reject untrusted data payloads
ayush00git Jul 30, 2026
c3a57d8
feat: add InferSchema validating struct against the schema
ayush00git Jul 30, 2026
f674b0b
feat: encode go structs to fory row format
ayush00git Jul 30, 2026
20b31f3
feat(test): add bytes <-> schema tests with Java interop
ayush00git Jul 30, 2026
56cfb7c
feat: add xlang tests to wrap with Java
ayush00git Jul 30, 2026
f4c064a
feat: add cross-language tests with java
ayush00git Jul 30, 2026
62321e9
feat: wire ci and update gitignore
ayush00git Jul 30, 2026
658b794
fix: trucation of large dt and add ignore fields
ayush00git Jul 30, 2026
91d448e
Merge branch 'main' into feat/go-fory-row-format
ayush00git Jul 31, 2026
64212ff
Merge branch 'main' into feat/go-fory-row-format
ayush00git Aug 22, 2026
4aae171
refactor(go): copy row schema fields and resolve duplicate names like…
ayush00git Aug 22, 2026
be4724b
fix(go): validate row schemas before encoding and harden schema parsing
ayush00git Aug 22, 2026
17114bc
fix(go): align row schema inference with Java and harden the codecs
ayush00git Aug 22, 2026
98ac90a
test(go): cover byte arrays in the Go row cross-language test and fai…
ayush00git Aug 22, 2026
cc57b03
docs(go): add the Go row format guide and update the support matrix
ayush00git Aug 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down Expand Up @@ -1692,6 +1692,8 @@
mvn -T16 --no-transfer-progress clean install -DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true
cd fory-core
mvn -T16 --no-transfer-progress test -Dtest=org.apache.fory.xlang.GoXlangTest
cd ../fory-format
mvn -T16 --no-transfer-progress test -Dtest=org.apache.fory.format.GoCrossLanguageTest
- name: Run Go IDL Tests
run: ./integration_tests/idl_tests/run_go_tests.sh

Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ go/fory/tests/xlang_test_main
go/fory/tests/xlang/xlang_test_main
go/fory/xlang_test_main
go/fory/xlang
go/fory/tests/row_xlang_bin
go/fory/tests/row_xlang_bin.exe

# Build directories for all languages
# Java
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -800,11 +800,32 @@ ArrayData scores = scoresField.get(row);
int secondScore = scores.getInt32(1);
```

**Go**

```go
import "github.com/apache/fory/go/fory/row"

type User struct {
Id int32
Name string
Scores []int32
}

encoder, _ := row.NewEncoder[User]()
binary, _ := encoder.ToRow(&User{Id: 1, Name: "Alice", Scores: []int32{98, 100, 95}})

schema := encoder.Schema()
r := row.NewRow(schema, binary)
name := r.String(schema.FieldIndex("name"))
secondScore := r.Array(schema.FieldIndex("scores")).Int32(1)
```

For Java imports, nested structs, arrays/maps, Arrow integration, and partial
deserialization, see the [Java row-format guide](docs/row-format/java.md),
[Python row-format guide](docs/row-format/python.md),
[C++ row-format guide](docs/row-format/cpp.md),
[Rust row-format guide](docs/row-format/rust.md), and
[Rust row-format guide](docs/row-format/rust.md),
[Go row-format guide](docs/row-format/go.md), and
[row-format specification](docs/specification/row_format_spec.md).

## Fory JSON
Expand Down
1 change: 1 addition & 0 deletions ci/tasks/go.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,5 @@ def run():
logging.info("Executing fory go tests")
common.cd_project_subdir("go/fory")
common.exec_cmd("go test -v")
common.exec_cmd("go test -v ./row/...")
logging.info("Executing fory go tests succeeds")
258 changes: 258 additions & 0 deletions docs/row-format/go.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
---
title: Go Standard Row Format
sidebar_position: 7
id: go
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---

Apache Fory™ Go implements the Standard Row Format used by Java, C++, Python, and Rust in the
`github.com/apache/fory/go/fory/row` package. It provides a reflection-based struct encoder,
zero-copy readers for random field access, and the cross-language schema encoding.

## Overview

Use Row Format when readers need selected fields or collection elements rather than a fully
reconstructed value. Readers are views over the encoded bytes: reading one field costs a bitmap
test and a slot lookup, and other fields are never touched.

Row Format is a trusted in-memory format. Decode only bytes produced by a Fory row writer for the
same schema, from a source you trust. Malformed bytes surface as errors from the encoder's decode
methods, but the format does not defend against hostile input the way object serialization does.

## When to Use Row Format

- Analytics workloads with selective field access
- Large datasets where only a subset of fields is needed
- Memory-mapped or shared data read by several languages
- High-throughput pipelines that exchange Standard Row bytes with Java, C++, Python, or Rust

## Basic Usage

```go
package main

import (
"fmt"

"github.com/apache/fory/go/fory/row"
)

type UserProfile struct {
Id int64
Username string
Email *string
Scores []int32
Preferences map[string]string
IsActive bool
}

func main() {
encoder, err := row.NewEncoder[UserProfile]()
if err != nil {
panic(err)
}

email := "alice@example.com"
profile := UserProfile{
Id: 12345,
Username: "alice",
Email: &email,
Scores: []int32{95, 87, 92, 88},
Preferences: map[string]string{"theme": "dark", "language": "en"},
IsActive: true,
}

rowBytes, err := encoder.ToRow(&profile)
if err != nil {
panic(err)
}

// Random access without decoding the whole struct.
schema := encoder.Schema()
r := row.NewRow(schema, rowBytes)
fmt.Println(r.String(schema.FieldIndex("username"))) // alice
fmt.Println(r.Array(schema.FieldIndex("scores")).Int32(1)) // 87
fmt.Println(r.IsNullAt(schema.FieldIndex("email"))) // false

// Full decode when the whole value is needed.
decoded, err := encoder.FromRow(rowBytes)
if err != nil {
panic(err)
}
fmt.Println(decoded.Preferences["theme"]) // dark
}
```

`NewEncoder[T]` infers the schema from the struct type once and compiles the conversion for it.
Creating an encoder is comparatively expensive; create one per struct type and reuse it.

## Rows and Framed Messages

`ToRow` and `FromRow` work with bare row bytes, which is what the other languages' `toRow` and
`BinaryRow.pointTo` exchange. `Encode` and `Decode` add the framing used by the Java and Python
row encoders: an 8-byte little-endian schema hash followed by the row.

```go
framed, err := encoder.Encode(&profile) // hash + row
decoded, err := encoder.Decode(framed) // verifies the hash, then decodes
```

The hash is a type-shape fingerprint: it folds the recursive field type ids and nothing else, so
`Decode` rejects a writer whose field types differ but cannot detect renamed fields, changed
nullability, or reordered fields of the same type. Share the schema bytes (see
[Schema Exchange](#schema-exchange)) when peers must agree on more than the type shape.

`Decode` and `FromRow` return errors for truncated or inconsistent bytes; they never panic. Decoded
values never alias the input, so the input buffer can be reused immediately.

## Zero-Copy Reading

`row.NewRow(schema, bytes)` creates a view over a row. `Struct`, `Array`, and `Map` return views
over the nested bytes, and `Binary` returns a sub-slice of the input. These views stay valid only
while the underlying bytes are alive and unmodified. `String` copies, because a Go string must not
observe later changes to the buffer.

```go
r := row.NewRow(schema, rowBytes)
tags := r.Array(schema.FieldIndex("tags"))
for i := 0; i < tags.NumElements(); i++ {
if !tags.IsNullAt(i) {
fmt.Println(tags.String(i))
}
}
attrs := r.Map(schema.FieldIndex("attrs"))
fmt.Println(attrs.Keys().String(0), attrs.Values().Int32(0))
```

Fixed-width getters return the zero value for null fields; use `IsNullAt` to distinguish null from
zero. Variable-width getters return `nil` (or `""`) for null. Out-of-range indexes panic, as does
malformed data read through a raw view; use the encoder's decode methods when an error is needed.

## Nullability

A Go pointer field is a nullable field: `nil` writes the null bit and decodes back to `nil`.
Slices, maps, and `[]byte` are nullable as well and distinguish `nil` from empty.

Strings, nested value structs, `fory.Date`, `time.Time`, and `time.Duration` are also nullable in
the schema because their Java carriers are objects, but the Go value cannot hold `nil`. Decoding a
null into one of these fields is an error. Use a pointer carrier (`*string`, `*time.Time`, `*Inner`)
when nulls must round-trip, for example when reading rows written by Java with `null` values.

Map values are always nullable and map keys never are, matching the other languages. A `[]*int32`
element or `map[string]*int32` value carries a null element; `[]int32` and `map[string]int32` do
not.

## Field Order and Names

Fields are sorted by their lowerCamel name and named by its snake_case form (`UserName` becomes
`user_name`), matching Java's schema inference so both languages derive the same schema from
equivalent struct definitions. Unexported fields are skipped. The `fory` struct tag uses the same
grammar as object serialization: `fory:"-"`, `fory:"ignore"`, and `fory:"ignore=true"` skip a
field; other keys are accepted and ignored by Row Format.

Changing a field name or type changes the schema. Coordinate such changes across all producers and
consumers.

## Supported Types

| Go type | Standard Row Format encoding | Nullable |
| --------------------------------------------- | -------------------------------- | -------- |
| `bool`, `int8`, `int16`, `int32`, `int64` | Fixed-width scalar | No |
| `int` | Fixed-width int64 | No |
| `float32`, `float64` | Fixed-width IEEE 754 scalar | No |
| `fory.Date` | Fixed-width date32 in epoch days | Yes |
| `time.Time` | Fixed-width epoch microseconds | Yes |
| `time.Duration` | Fixed-width microseconds | Yes |
| `string` | Variable-width UTF-8 | Yes |
| `[]byte` | Standard array of int8 | Yes |
| `[]T` for supported element types | Standard array | Yes |
| `map[K]V` | Standard map | Yes |
| Nested struct | Nested Standard Row | Yes |
| `*T` for any supported non-slice, non-map `T` | Same encoding as `T` | Yes |

`[]byte` matches Java's `byte[]`, which Java also infers as a list of int8. The row format's
binary type is available only to hand-built schemas through `RowWriter.WriteBytes` and
`Row.Binary`.

Strings must be valid UTF-8. Timestamps must fit in an int64 number of microseconds. Map keys must
be scalars, strings, or value structs whose exported, non-ignored fields consist of such types, so
that the encoded key determines Go equality; `time.Time` and pointers are not valid keys.

Unsupported: unsigned integers, fixed-size arrays, nested pointers, pointers to slices or maps,
interfaces, channels, functions, recursive types, `float16`, and `decimal`.

## Schema Exchange

`SchemaToBytes` and `SchemaFromBytes` implement the cross-language schema encoding shared with
Java's `SchemaEncoder` and Python's `Schema.to_bytes`/`from_bytes`. `ComputeSchemaHash` computes
the same type-shape hash used by `Encode`.

```go
schemaBytes, err := row.SchemaToBytes(encoder.Schema())
schema, err := row.SchemaFromBytes(schemaBytes)
fmt.Println(schema.Equal(encoder.Schema())) // true
```

A Java bean and a Go struct with equivalent fields produce identical schema bytes when their
nullability matches: use pointer fields for Java boxed types (`Integer`, `String` in lists) and
value fields for Java primitives.

## Writing Rows by Hand

`RowWriter`, `ArrayWriter`, and `MapWriter` write rows for a schema you construct yourself, without
a Go struct. They share one `fory.ByteBuffer`; a nested value is written at the buffer's current
position and then attached to its parent slot with `SetOffsetAndSize`.

```go
schema := row.NewSchema([]row.Field{
row.NewField("id", row.Int64Type{}, false),
row.NewField("tags", row.List(row.StringType{}), true),
})
w := row.NewRowWriter(schema)
w.Reset()
w.WriteInt64(0, 7)

tags := row.NewArrayWriter(row.List(row.StringType{}).Elem, w.Buffer())
start := w.Buffer().WriterIndex()
if err := tags.Reset(2); err != nil {
panic(err)
}
if err := tags.WriteString(0, "go"); err != nil {
panic(err)
}
tags.SetNullAt(1)
if err := w.SetOffsetAndSize(1, start, w.Buffer().WriterIndex()-start); err != nil {
panic(err)
}
rowBytes := w.ToBytes() // valid until the buffer is written to again
```

Call `Reset` before each row; to reuse a writer for a new top-level row, set the buffer's writer
index back to zero first.

## Thread Safety

An `Encoder` owns a reusable write buffer and is not safe for concurrent use; create one encoder
per goroutine or guard it with a mutex. Writers share the same rule. `Row`, `ArrayData`, and
`MapData` views only read, so one view can be shared by concurrent readers as long as the
underlying bytes are not modified.

## Related Topics

- [Basic Serialization](../object-serialization/go/basic-serialization.md) - Object graph serialization
- [Standard Row Format](index.md#standard-row) - Shared layout for Java, Python, C++, Rust, and Go
- [Row Format Specification](../specification/row_format_spec.md) - Protocol details
15 changes: 8 additions & 7 deletions docs/row-format/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,20 @@ reconstruction as its primary access pattern.

## Choose a Layout

| Layout | Language support | Compatibility |
| ------------ | ----------------------- | -------------------------------- |
| Standard Row | Java, Python, C++, Rust | Shared Standard Row layout |
| Compact Row | Java | Java-only, space-oriented layout |
| Layout | Language support | Compatibility |
| ------------ | --------------------------- | -------------------------------- |
| Standard Row | Java, Python, C++, Rust, Go | Shared Standard Row layout |
| Compact Row | Java | Java-only, space-oriented layout |

## Standard Row

Standard Row is the interoperable layout for Java, Python, C++, and Rust.
Standard Row is the interoperable layout for Java, Python, C++, Rust, and Go.

### Features

- **Zero-copy random access**: Read selected fields directly from encoded data.
- **Partial deserialization**: Reconstruct only the values an application needs.
- **Cross-language compatibility**: Share Standard Row bytes between Java, Python, C++, and Rust.
- **Cross-language compatibility**: Share Standard Row bytes between Java, Python, C++, Rust, and Go.
- **Apache Arrow integration**: Convert rows to Arrow data in Java and Python.

### Layout
Expand All @@ -61,6 +61,7 @@ layout, alignment rules, type table, and endianness are defined by the
| Python | Compatible | [Python](python.md) | PyArrow schema and table conversion |
| C++ | Compatible | [C++](cpp.md) | Native row readers and writers |
| Rust | Compatible | [Rust](rust.md) | Borrowed struct, array, and map views |
| Go | Compatible | [Go](go.md) | Reflection struct encoder; zero-copy row views |

Use the language guides for installation, schema construction, encoding, random access, partial
reads, and language-specific integrations.
Expand Down Expand Up @@ -93,7 +94,7 @@ Reuse the encoder within one thread. Create separate encoders for concurrent thr
- Fixed-size nested structs can be stored inline.

Choose Compact Row only when every reader is Java and the space reduction justifies the
Java-specific layout. Use Standard Row for Java, Python, C++, and Rust interchange.
Java-specific layout. Use Standard Row for Java, Python, C++, Rust, and Go interchange.

See the [Row Format specification](../specification/row_format_spec.md) for the exact Standard and
Compact binary layouts.
2 changes: 1 addition & 1 deletion docs/row-format/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ license: |

## A Standard Row peer cannot read Compact Row bytes

Compact Row is a Java-only row family. Use Standard Row on Java, Python, C++, and Rust for shared
Compact Row is a Java-only row family. Use Standard Row on Java, Python, C++, Rust, and Go for shared
bytes.

## A field lookup fails
Expand Down
10 changes: 5 additions & 5 deletions docs/specification/row_format_spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,14 @@ Apache Fory Row Format is a cache-friendly, random-access binary format designed
- **Random Field Access**: Read individual fields without deserializing the entire row
- **Zero-Copy Operations**: Direct memory access without data transformation
- **Cache-Friendly Layout**: Optimized memory layout for CPU cache efficiency
- **Cross-Language Support**: Consistent binary format across Java, C++, Python, and Rust
- **Cross-Language Support**: Consistent binary format across Java, C++, Python, Rust, and Go

Fory provides two row format variants:

| Format | Languages | Use Case |
| --------------- | ----------------------- | ------------------------------ |
| Standard Format | Java, C++, Python, Rust | Cross-language compatibility |
| Compact Format | Java only | Space efficiency, smaller rows |
| Format | Languages | Use Case |
| --------------- | --------------------------- | ------------------------------ |
| Standard Format | Java, C++, Python, Rust, Go | Cross-language compatibility |
| Compact Format | Java only | Space efficiency, smaller rows |

## Format Comparison

Expand Down
Loading
Loading