← Back to Libraries

FlatBuffers

SerializationZero CopyHigh PerformanceMemory EfficientBinarySchemaGame DevelopmentGoogle

Library

FlatBuffers

Overview

FlatBuffers is a high-performance cross-platform serialization library developed by Google. Its key feature is "zero-copy deserialization," eliminating the need for memory copying during deserialization. Originally developed by game developer Wouter van Oortmerssen and used in Cocos2d-x, Facebook Android, and numerous game development projects, it particularly excels in mobile apps, games, and real-time systems where memory efficiency and access speed are critical.

Details

FlatBuffers 2025 edition achieves performance that significantly surpasses traditional serialization formats through its flat binary format design that pursues maximum memory efficiency. The schema-driven structure guarantees type safety during field access, and compared to Protocol Buffers, it can ignore over 99% of data when accessing only specific fields, achieving dramatic improvements in processing speed and memory usage reduction. The code generator (flatc) provides unified APIs for over 15 languages, ensuring comprehensive multi-language support for enterprise development.

Key Features

  • Zero-Copy Deserialization: No memory copying required during deserialization
  • Ultra-Fast Access: Highest level access speed through flat memory layout
  • Schema-Based: Strict type safety and version management
  • Memory Efficient: No temporary object creation during access
  • Multi-Language Support: API unification across 15+ languages
  • Enterprise Track Record: Adoption by Google, Facebook, and gaming industry

Pros and Cons

Pros

  • Overwhelming access speed and memory efficiency (zero-copy deserialization)
  • High-speed performance during partial access (access only required fields)
  • High cache efficiency through flat memory layout
  • Strict type safety and backward compatibility guarantee through schema-based approach
  • Reduced battery consumption and acceleration on mobile devices
  • Rich track record in game development and real-time systems

Cons

  • Implementation code is more complex and voluminous than JSON/Protocol Buffers
  • Some mutation operations are impossible (read-only)
  • Schema evolution flexibility is more limited than Protocol Buffers
  • Large constraints on field renaming and deletion
  • Limited benefits for small-scale data
  • Low readability due to binary format during debugging

Reference Pages

Code Examples

Basic Setup

# Installing FlatBuffers compiler
# Ubuntu/Debian
sudo apt-get install flatbuffers-compiler

# macOS
brew install flatbuffers

# Windows
choco install flatbuffers

# Build from source
git clone https://github.com/google/flatbuffers.git
cd flatbuffers
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
make -j4

# Installing language bindings
# C++ (often included in system)

# Python
pip install flatbuffers

# Java/Kotlin
# Maven
# <dependency>
#   <groupId>com.google.flatbuffers</groupId>
#   <artifactId>flatbuffers-java</artifactId>
#   <version>24.3.25</version>
# </dependency>

# JavaScript/Node.js
npm install flatbuffers

# C#
dotnet add package Google.FlatBuffers

# Go
go get github.com/google/flatbuffers/go

Schema Definition and Code Generation

// monster.fbs - FlatBuffers schema file
namespace MyGame.Sample;

enum Color : byte { Red = 0, Green, Blue = 2 }

union Equipment { Weapon }

struct Vec3 {
  x:float;
  y:float;
  z:float;
}

table Monster {
  pos:Vec3;              // struct field
  mana:short = 150;      // field with default value
  hp:short = 100;
  name:string;           // string field
  friendly:bool = false (deprecated);  // deprecated field
  inventory:[ubyte];     // byte array
  color:Color = Blue;    // enum type
  weapons:[Weapon];      // table array
  equipped:Equipment;    // union type
  path:[Vec3];          // struct array
}

table Weapon {
  name:string;
  damage:short;
}

root_type Monster;      // root type specification

# Generate code from schema
flatc --cpp --python --java --csharp --go monster.fbs

# Specific language only
flatc --cpp monster.fbs        # C++
flatc --python monster.fbs     # Python
flatc --java monster.fbs       # Java
flatc --js monster.fbs         # JavaScript
flatc --ts monster.fbs         # TypeScript

# Specify output directory
flatc --cpp --python -o ./generated monster.fbs

# Convert JSON to binary format
flatc --binary monster.fbs monster.json

# Convert binary to JSON (for debugging)
flatc --json monster.fbs monster.bin

Basic Usage in C++

#include "flatbuffers/flatbuffers.h"
#include "monster_generated.h"  // generated header

using namespace MyGame::Sample;

void createMonster() {
    flatbuffers::FlatBufferBuilder builder(1024);
    
    // Create strings
    auto weapon_one_name = builder.CreateString("Sword");
    auto weapon_two_name = builder.CreateString("Axe");
    auto monster_name = builder.CreateString("Orc");
    
    // Create Weapon tables
    short weapon_one_damage = 3;
    short weapon_two_damage = 5;
    
    auto sword = CreateWeapon(builder, weapon_one_name, weapon_one_damage);
    auto axe = CreateWeapon(builder, weapon_two_name, weapon_two_damage);
    
    // Create weapons vector
    std::vector<flatbuffers::Offset<Weapon>> weapons_vector;
    weapons_vector.push_back(sword);
    weapons_vector.push_back(axe);
    auto weapons = builder.CreateVector(weapons_vector);
    
    // Create inventory vector
    unsigned char treasure[] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
    auto inventory = builder.CreateVector(treasure, 10);
    
    // Create path vector (struct array)
    Vec3 points[] = { Vec3(1.0f, 2.0f, 3.0f), Vec3(4.0f, 5.0f, 6.0f) };
    auto path = builder.CreateVectorOfStructs(points, 2);
    
    // Create Monster table
    Vec3 position(1.0f, 2.0f, 3.0f);
    auto orc = CreateMonster(builder, 
                           &position,     // pos
                           150,           // mana
                           80,            // hp
                           monster_name,  // name
                           inventory,     // inventory
                           Color_Red,     // color
                           weapons,       // weapons
                           Equipment_Weapon, // equipped_type
                           axe.Union(),   // equipped
                           path);         // path
    
    // Finish buffer
    builder.Finish(orc);
    
    // Get serialized data
    uint8_t *buffer_pointer = builder.GetBufferPointer();
    int buffer_size = builder.GetSize();
    
    printf("FlatBuffer created: %d bytes\n", buffer_size);
}

void readMonster(const uint8_t* buffer) {
    // Start zero-copy access
    auto monster = GetMonster(buffer);
    
    // Direct access to basic fields
    printf("Monster HP: %d\n", monster->hp());
    printf("Monster Mana: %d\n", monster->mana());
    printf("Monster Name: %s\n", monster->name()->c_str());
    printf("Monster Color: %d\n", static_cast<int>(monster->color()));
    
    // Access position info (struct)
    auto pos = monster->pos();
    if (pos) {
        printf("Position: (%.2f, %.2f, %.2f)\n", pos->x(), pos->y(), pos->z());
    }
    
    // Access inventory vector
    auto inventory = monster->inventory();
    if (inventory) {
        printf("Inventory items: ");
        for (unsigned int i = 0; i < inventory->size(); i++) {
            printf("%d ", inventory->Get(i));
        }
        printf("\n");
    }
    
    // Access weapons vector (table array)
    auto weapons = monster->weapons();
    if (weapons) {
        printf("Weapons count: %u\n", weapons->size());
        for (unsigned int i = 0; i < weapons->size(); i++) {
            auto weapon = weapons->Get(i);
            printf("Weapon %u: %s (damage: %d)\n", 
                   i, weapon->name()->c_str(), weapon->damage());
        }
    }
    
    // Access union type
    if (monster->equipped_type() == Equipment_Weapon) {
        auto equipped_weapon = monster->equipped_as_Weapon();
        printf("Equipped: %s (damage: %d)\n", 
               equipped_weapon->name()->c_str(), equipped_weapon->damage());
    }
    
    // Access path struct array
    auto path = monster->path();
    if (path) {
        printf("Path points: %u\n", path->size());
        for (unsigned int i = 0; i < path->size(); i++) {
            auto point = path->Get(i);
            printf("Point %u: (%.2f, %.2f, %.2f)\n", 
                   i, point->x(), point->y(), point->z());
        }
    }
}

int main() {
    // Monster creation and serialization
    flatbuffers::FlatBufferBuilder builder(1024);
    
    // Creation process... (content of createMonster() above)
    
    // Reading process
    readMonster(builder.GetBufferPointer());
    
    return 0;
}

Python Usage Example

import flatbuffers
import MyGame.Sample.Monster as Monster
import MyGame.Sample.Weapon as Weapon
import MyGame.Sample.Vec3 as Vec3
import MyGame.Sample.Color as Color
import MyGame.Sample.Equipment as Equipment

def create_monster():
    """Create and serialize Monster"""
    builder = flatbuffers.Builder(1024)
    
    # Create strings
    weapon_one_name = builder.CreateString("Sword")
    weapon_two_name = builder.CreateString("Axe")
    monster_name = builder.CreateString("Orc")
    
    # Create Weapon tables
    Weapon.WeaponStart(builder)
    Weapon.WeaponAddName(builder, weapon_one_name)
    Weapon.WeaponAddDamage(builder, 3)
    sword = Weapon.WeaponEnd(builder)
    
    Weapon.WeaponStart(builder)
    Weapon.WeaponAddName(builder, weapon_two_name)
    Weapon.WeaponAddDamage(builder, 5)
    axe = Weapon.WeaponEnd(builder)
    
    # Create weapons vector
    Monster.MonsterStartWeaponsVector(builder, 2)
    builder.PrependUOffsetTRelative(axe)    # Add in reverse order
    builder.PrependUOffsetTRelative(sword)
    weapons = builder.EndVector(2)
    
    # Create inventory vector
    Monster.MonsterStartInventoryVector(builder, 10)
    for i in reversed(range(10)):  # Add in reverse order
        builder.PrependByte(i)
    inventory = builder.EndVector(10)
    
    # Create path vector (struct array)
    Monster.MonsterStartPathVector(builder, 2)
    Vec3.CreateVec3(builder, 4.0, 5.0, 6.0)  # Add in reverse order
    Vec3.CreateVec3(builder, 1.0, 2.0, 3.0)
    path = builder.EndVector(2)
    
    # Create Monster table
    Monster.MonsterStart(builder)
    Monster.MonsterAddPos(builder, Vec3.CreateVec3(builder, 1.0, 2.0, 3.0))
    Monster.MonsterAddHp(builder, 80)
    Monster.MonsterAddMana(builder, 150)
    Monster.MonsterAddName(builder, monster_name)
    Monster.MonsterAddInventory(builder, inventory)
    Monster.MonsterAddColor(builder, Color.Color.Red)
    Monster.MonsterAddWeapons(builder, weapons)
    Monster.MonsterAddEquippedType(builder, Equipment.Equipment.Weapon)
    Monster.MonsterAddEquipped(builder, axe)
    Monster.MonsterAddPath(builder, path)
    orc = Monster.MonsterEnd(builder)
    
    # Finish buffer
    builder.Finish(orc)
    
    # Get serialized byte array
    buf = builder.Output()
    print(f"FlatBuffer created: {len(buf)} bytes")
    
    return buf

def read_monster(buf):
    """Read and access Monster"""
    # Start zero-copy access
    monster = Monster.Monster.GetRootAs(buf, 0)
    
    # Direct access to basic fields
    print(f"Monster HP: {monster.Hp()}")
    print(f"Monster Mana: {monster.Mana()}")
    print(f"Monster Name: {monster.Name().decode('utf-8')}")
    print(f"Monster Color: {monster.Color()}")
    
    # Access position info (struct)
    pos = monster.Pos()
    if pos:
        print(f"Position: ({pos.X():.2f}, {pos.Y():.2f}, {pos.Z():.2f})")
    
    # Access inventory vector
    print("Inventory items:", end=" ")
    for i in range(monster.InventoryLength()):
        print(monster.Inventory(i), end=" ")
    print()
    
    # Access weapons vector (table array)
    print(f"Weapons count: {monster.WeaponsLength()}")
    for i in range(monster.WeaponsLength()):
        weapon = monster.Weapons(i)
        weapon_name = weapon.Name().decode('utf-8') if weapon.Name() else "Unknown"
        print(f"Weapon {i}: {weapon_name} (damage: {weapon.Damage()})")
    
    # Access union type
    if monster.EquippedType() == Equipment.Equipment.Weapon:
        equipped_weapon = Weapon.Weapon()
        equipped_weapon.Init(monster.Equipped().Bytes, monster.Equipped().Pos)
        weapon_name = equipped_weapon.Name().decode('utf-8') if equipped_weapon.Name() else "Unknown"
        print(f"Equipped: {weapon_name} (damage: {equipped_weapon.Damage()})")
    
    # Access path struct array
    print(f"Path points: {monster.PathLength()}")
    for i in range(monster.PathLength()):
        point = monster.Path(i)
        print(f"Point {i}: ({point.X():.2f}, {point.Y():.2f}, {point.Z():.2f})")

# Usage example
if __name__ == "__main__":
    # Monster creation and serialization
    buffer = create_monster()
    
    # Reading process
    read_monster(buffer)
    
    # File save example
    with open("monster.bin", "wb") as f:
        f.write(buffer)
    
    # File load example
    with open("monster.bin", "rb") as f:
        loaded_buffer = f.read()
        print("\n=== Loaded from file ===")
        read_monster(loaded_buffer)

Performance Optimization and Practical Techniques

#include "flatbuffers/flatbuffers.h"
#include "monster_generated.h"
#include <chrono>
#include <vector>
#include <memory>

using namespace MyGame::Sample;

class FlatBufferPerformanceDemo {
private:
    std::unique_ptr<flatbuffers::FlatBufferBuilder> builder_;
    std::vector<uint8_t> buffer_pool_;
    
public:
    FlatBufferPerformanceDemo() : builder_(std::make_unique<flatbuffers::FlatBufferBuilder>(65536)) {}
    
    // High-speed batch processing
    void createMonstersInBatch(int count) {
        auto start = std::chrono::high_resolution_clock::now();
        
        // Clear and reuse buffer
        builder_->Clear();
        
        // Create monster array in batch
        std::vector<flatbuffers::Offset<Monster>> monsters;
        monsters.reserve(count);
        
        for (int i = 0; i < count; i++) {
            // String pooling (reuse same names)
            static auto cached_names = std::vector<flatbuffers::Offset<flatbuffers::String>>();
            if (cached_names.empty()) {
                cached_names.push_back(builder_->CreateString("Orc"));
                cached_names.push_back(builder_->CreateString("Goblin"));
                cached_names.push_back(builder_->CreateString("Troll"));
            }
            
            auto monster_name = cached_names[i % cached_names.size()];
            
            // Create monster with minimal fields
            Vec3 position(static_cast<float>(i), static_cast<float>(i * 2), 0.0f);
            auto monster = CreateMonster(*builder_,
                                       &position,
                                       static_cast<short>(100 + i),  // mana
                                       static_cast<short>(80 + i),   // hp
                                       monster_name,
                                       0,  // inventory (null)
                                       Color_Red,
                                       0,  // weapons (null)
                                       Equipment_NONE,
                                       0,  // equipped (null)
                                       0); // path (null)
            monsters.push_back(monster);
        }
        
        // Store monster array in FlatBuffer
        auto monsters_vector = builder_->CreateVector(monsters);
        
        // Save as root table
        builder_->Finish(monsters_vector);
        
        auto end = std::chrono::high_resolution_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::microseconds>(end - start);
        
        printf("Created %d monsters in %ld microseconds\n", count, duration.count());
        printf("Buffer size: %d bytes\n", builder_->GetSize());
        printf("Average bytes per monster: %.2f\n", 
               static_cast<double>(builder_->GetSize()) / count);
    }
    
    // Partial access performance test
    void benchmarkPartialAccess() {
        createMonstersInBatch(10000);
        
        const uint8_t* buffer = builder_->GetBufferPointer();
        auto monsters_vector = flatbuffers::GetRoot<flatbuffers::Vector<flatbuffers::Offset<Monster>>>(buffer);
        
        auto start = std::chrono::high_resolution_clock::now();
        
        // Access only HP of 1000 monsters (ignore 99% of data)
        int total_hp = 0;
        for (unsigned int i = 0; i < std::min(1000u, monsters_vector->size()); i++) {
            auto monster = monsters_vector->Get(i);
            total_hp += monster->hp();  // Access only HP field
        }
        
        auto end = std::chrono::high_resolution_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start);
        
        printf("Partial access (HP only) for 1000 monsters: %ld nanoseconds\n", duration.count());
        printf("Average time per access: %.2f nanoseconds\n", 
               static_cast<double>(duration.count()) / 1000);
        printf("Total HP: %d\n", total_hp);
    }
    
    // Memory usage analysis
    void analyzeMemoryUsage() {
        const int monster_count = 1000;
        createMonstersInBatch(monster_count);
        
        const uint8_t* buffer = builder_->GetBufferPointer();
        int buffer_size = builder_->GetSize();
        
        printf("\n=== Memory Usage Analysis ===\n");
        printf("Monster count: %d\n", monster_count);
        printf("Total buffer size: %d bytes\n", buffer_size);
        printf("Average size per monster: %.2f bytes\n", 
               static_cast<double>(buffer_size) / monster_count);
        
        // Memory usage during access (almost zero)
        printf("Additional memory for access: ~0 bytes (zero-copy)\n");
        
        // Comparison: with normal objects
        struct NormalMonster {
            Vec3 pos;
            short mana, hp;
            std::string name;
            Color color;
        };
        
        printf("Estimated memory for %d normal objects: %lu bytes\n", 
               monster_count, monster_count * sizeof(NormalMonster));
        printf("Memory efficiency ratio: %.2fx\n",
               static_cast<double>(monster_count * sizeof(NormalMonster)) / buffer_size);
    }
    
    // Real-time application optimization
    void simulateRealtimeUsage() {
        printf("\n=== Realtime Usage Simulation ===\n");
        
        const int frame_count = 1000;  // 1000 frames
        const int monsters_per_frame = 100;
        
        auto total_start = std::chrono::high_resolution_clock::now();
        
        for (int frame = 0; frame < frame_count; frame++) {
            auto frame_start = std::chrono::high_resolution_clock::now();
            
            // Create new monster data per frame
            builder_->Clear();  // Fast buffer clear
            
            std::vector<flatbuffers::Offset<Monster>> frame_monsters;
            frame_monsters.reserve(monsters_per_frame);
            
            for (int i = 0; i < monsters_per_frame; i++) {
                Vec3 position(
                    static_cast<float>(i + frame * 0.1f), 
                    static_cast<float>(i * 2 + frame * 0.2f), 
                    0.0f
                );
                
                auto monster_name = builder_->CreateString("FrameMonster");
                auto monster = CreateMonster(*builder_,
                                           &position,
                                           static_cast<short>(100 + i),
                                           static_cast<short>(80 + i),
                                           monster_name);
                frame_monsters.push_back(monster);
            }
            
            auto monsters_vector = builder_->CreateVector(frame_monsters);
            builder_->Finish(monsters_vector);
            
            // Data reading (simulate game loop processing)
            const uint8_t* buffer = builder_->GetBufferPointer();
            auto monsters = flatbuffers::GetRoot<flatbuffers::Vector<flatbuffers::Offset<Monster>>>(buffer);
            
            // Update all monster positions (reading process)
            float total_distance = 0.0f;
            for (unsigned int i = 0; i < monsters->size(); i++) {
                auto monster = monsters->Get(i);
                auto pos = monster->pos();
                if (pos) {
                    total_distance += std::sqrt(pos->x() * pos->x() + pos->y() * pos->y());
                }
            }
            
            auto frame_end = std::chrono::high_resolution_clock::now();
            auto frame_duration = std::chrono::duration_cast<std::chrono::microseconds>(
                frame_end - frame_start);
            
            // Check if under 16.67ms (60FPS)
            if (frame % 100 == 0) {
                printf("Frame %d: %ld μs (%.2f FPS equivalent)\n", 
                       frame, frame_duration.count(),
                       1000000.0 / frame_duration.count());
            }
        }
        
        auto total_end = std::chrono::high_resolution_clock::now();
        auto total_duration = std::chrono::duration_cast<std::chrono::milliseconds>(
            total_end - total_start);
        
        printf("Total time for %d frames: %ld ms\n", frame_count, total_duration.count());
        printf("Average FPS: %.2f\n", 
               static_cast<double>(frame_count * 1000) / total_duration.count());
    }
};

int main() {
    FlatBufferPerformanceDemo demo;
    
    // Run performance tests
    demo.createMonstersInBatch(10000);
    demo.benchmarkPartialAccess();
    demo.analyzeMemoryUsage();
    demo.simulateRealtimeUsage();
    
    return 0;
}

JSON and FlatBuffers Interconversion

# monster.json - Sample JSON data
cat > monster.json << 'EOF'
{
  "pos": {
    "x": 1.0,
    "y": 2.0,
    "z": 3.0
  },
  "hp": 80,
  "mana": 150,
  "name": "Orc",
  "color": "Red",
  "inventory": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
  "weapons": [
    {
      "name": "Sword",
      "damage": 3
    },
    {
      "name": "Axe", 
      "damage": 5
    }
  ],
  "equipped_type": "Weapon",
  "equipped": {
    "name": "Axe",
    "damage": 5
  },
  "path": [
    { "x": 1.0, "y": 2.0, "z": 3.0 },
    { "x": 4.0, "y": 5.0, "z": 6.0 }
  ]
}
EOF

# Convert JSON to FlatBuffer binary format
flatc --binary monster.fbs monster.json

# Convert binary to JSON (for debugging/verification)
flatc --json monster.fbs monster.bin

# Check binary file size
ls -la monster.bin

# JSON vs FlatBuffer size comparison
echo "JSON size: $(wc -c < monster.json) bytes"
echo "FlatBuffer size: $(wc -c < monster.bin) bytes"

# Hex dump (binary structure inspection)
hexdump -C monster.bin | head -10

Advanced Schema Evolution and Best Practices

// version1.fbs - Initial version
namespace MyGame.Sample;

table Monster {
  hp:short = 100;
  mana:short = 150;
  name:string;
}

root_type Monster;

// version2.fbs - Safe evolution example
namespace MyGame.Sample;

table Monster {
  hp:short = 100;
  mana:short = 150;
  name:string;
  // Add new fields at the end (safe)
  level:short = 1;
  experience:int = 0;
  // Mark deprecated fields
  old_field:int (deprecated);
}

root_type Monster;

// version3.fbs - Safe reordering using IDs
namespace MyGame.Sample;

table Monster {
  // Use explicit IDs for safe reordering
  level:short = 1 (id: 3);
  hp:short = 100 (id: 0);
  experience:int = 0 (id: 4);
  mana:short = 150 (id: 1);
  name:string (id: 2);
  // New features
  skills:[string] (id: 5);
  metadata:[ubyte] (id: 6);
}

root_type Monster;

// Cross-version compatibility check
#include "flatbuffers/flatbuffers.h"

void checkCompatibility(const uint8_t* buffer) {
    auto monster = GetMonster(buffer);
    
    // Old fields are always accessible
    printf("HP: %d\n", monster->hp());
    printf("Name: %s\n", monster->name() ? monster->name()->c_str() : "null");
    
    // Check existence of new fields
    printf("Level: %d\n", monster->level());  // Default value returned
    
    // Check field existence by comparing with default value
    if (monster->level() != 1) {  // Different from default value
        printf("Level field was explicitly set to: %d\n", monster->level());
    }
    
    // Safe access to collection fields
    auto skills = monster->skills();
    if (skills) {
        printf("Skills count: %u\n", skills->size());
        for (unsigned int i = 0; i < skills->size(); i++) {
            printf("Skill %u: %s\n", i, skills->Get(i)->c_str());
        }
    } else {
        printf("No skills field present in this version\n");
    }
}