Library

bitsery

Overview

bitsery is a header-only C++11 binary serialization library. It is designed around the networking requirements for real-time data delivery, especially for games. It achieves high speed and minimal binary footprint, with no external dependencies and bit-level fine control capabilities, making it safe to use even with untrusted network data.

Details

bitsery is designed to combine the ease of use of cereal, the forward/backward compatibility of flatbuffers, and achieve fast performance with a small binary footprint. The library itself doesn't do any memory allocations, so data reading/writing is as fast as memcpy to/from your buffer.

Key features:

• Bit-Level Control: Extreme data size reduction through bit packing
• Cross-Platform: All requirements enforced at compile time, no metadata needed
• No Code Generation: Use your types directly without IDL or metadata
• Configurable Error Checking: Runtime error checking for untrusted data (can be disabled)
• Flexible I/O: Supports streams (files, network) and buffers (vectors, C-arrays)
• Extensibility: Easily extensible for any type
• Syntax Options: Supports both verbose syntax (s.value4b()) and brief syntax (s())
• Forward/Backward Compatibility: Compatibility support through Growable extension
• Endianness Support: Configurable endianness conversion

Performance optimizations:

• ValueRange: Reduce bit count by specifying value ranges (e.g., int16_t with 0-1000 range to 10 bits)
• CompactValue: Efficiently encode frequently occurring small values
• Bit Packing: Store bool values and enums with minimal bit count

Pros and Cons

Pros

• Extremely Fast: Significantly outperforms other libraries in benchmarks
• Minimal Binary Size: Industry-smallest data size through bit-level optimization
• Memory Efficient: No memory allocations in the library itself
• Safety-Focused: Built-in protection against malicious data
• Header-Only: Easy integration, no additional dependencies
• Embedded-Friendly: Minimal memory footprint ideal for embedded systems
• Flexible Extensions: Easy implementation of custom types and serialization strategies
• No Macros: Clean codebase

Cons

• Binary Format Only: Doesn't support other formats like JSON or XML
• No Type Information: All type information must be written in code
• Learning Curve: Advanced features like bit-level operations require time to master
• C++11 Required: Some extensions require newer C++ standards
• Dynamic Size Limits: For safety, dynamic containers require maximum size specification

References

• GitHub Repository
• Tutorial
• Design Documentation
• Examples

Code Examples

Basic Serialization

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <bitsery/traits/vector.h>

enum class MyEnum : uint16_t { V1, V2, V3 };

struct MyStruct {
    uint32_t i;
    MyEnum e;
    std::vector<float> fs;
};

template <typename S>
void serialize(S& s, MyStruct& o) {
    s.value4b(o.i);              // 4-byte value
    s.value2b(o.e);              // 2-byte value
    s.container4b(o.fs, 10);     // float array with max 10 elements
}

using Buffer = std::vector<uint8_t>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;

int main() {
    MyStruct data{8941, MyEnum::V2, {15.0f, -8.5f, 0.045f}};
    MyStruct res{};
    Buffer buffer;
    
    // Serialization
    auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
    
    // Deserialization
    auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);
    
    assert(state.first == bitsery::ReaderError::NoError && state.second);
    assert(data.fs == res.fs && data.i == res.i && data.e == res.e);
    
    return 0;
}

Using Detailed Serializer and Deserializer

#include <bitsery/bitsery.h>
#include <bitsery/adapter/stream.h>
#include <bitsery/traits/string.h>
#include <fstream>

struct GameData {
    std::string player_name;
    uint32_t score;
    uint16_t level;
    bool is_online;
    
    template<typename S>
    void serialize(S& s) {
        s.text1b(player_name, 50);  // Max 50 characters
        s.value4b(score);
        s.value2b(level);
        s.value1b(is_online);
    }
};

int main() {
    GameData save_data{"Player1", 150000, 42, true};
    
    // Save to file
    {
        std::ofstream file("savegame.dat", std::ios::binary);
        bitsery::Serializer<bitsery::OutputStreamAdapter> ser{file};
        ser.object(save_data);
        
        // Get actual written size from adapter
        auto writtenSize = ser.adapter().writtenBytesCount();
        std::cout << "Written " << writtenSize << " bytes" << std::endl;
    }
    
    // Load from file
    GameData loaded_data{};
    {
        std::ifstream file("savegame.dat", std::ios::binary);
        
        // Get file size
        file.seekg(0, std::ios::end);
        size_t fileSize = file.tellg();
        file.seekg(0, std::ios::beg);
        
        bitsery::Deserializer<bitsery::InputStreamAdapter> des{file};
        des.object(loaded_data);
        
        auto state = std::make_pair(des.adapter().error(), 
                                   des.adapter().isCompletedSuccessfully());
        
        if (state.first == bitsery::ReaderError::NoError && state.second) {
            std::cout << "Successfully loaded: " << loaded_data.player_name << std::endl;
        }
    }
    
    return 0;
}

Bit Packing and Optimization

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <bitsery/ext/value_range.h>
#include <bitsery/ext/compact_value.h>

struct OptimizedData {
    uint16_t health;      // Range 0-100
    uint8_t ammo;        // Range 0-50
    bool flags[8];       // 8 boolean flags
    int32_t position_x;  // Range -1000 to 1000
    int32_t position_y;  // Range -1000 to 1000
};

template <typename S>
void serialize(S& s, OptimizedData& o) {
    // Use ValueRange extension to specify value ranges
    s.ext(o.health, bitsery::ext::ValueRange<uint16_t>{0, 100});  // Stored in 7 bits
    s.ext(o.ammo, bitsery::ext::ValueRange<uint8_t>{0, 50});      // Stored in 6 bits
    
    // Use bit packing to store 8 bools in 1 byte
    s.enableBitPacking([&o](typename S::BPEnabledType& sbp) {
        for (int i = 0; i < 8; ++i) {
            sbp.boolValue(o.flags[i]);
        }
    });
    
    // CompactValue extension for efficient storage of small frequent values
    s.ext(o.position_x, bitsery::ext::CompactValue{});
    s.ext(o.position_y, bitsery::ext::CompactValue{});
}

int main() {
    OptimizedData data{75, 30, {true, false, true, true, false, false, true, false}, 
                      -150, 200};
    
    std::vector<uint8_t> buffer;
    auto writtenSize = bitsery::quickSerialization(buffer, data);
    
    std::cout << "Optimized size: " << writtenSize << " bytes" << std::endl;
    // Data that would normally require 16+ bytes is significantly compressed
    
    return 0;
}

Polymorphism and Pointers

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <bitsery/ext/inheritance.h>
#include <bitsery/ext/std_smart_ptr.h>
#include <memory>

struct Base {
    virtual ~Base() = default;
    int base_value = 10;
    
    template<typename S>
    void serialize(S& s) {
        s.value4b(base_value);
    }
};

struct Derived1 : Base {
    float derived1_value = 3.14f;
    
    template<typename S>
    void serialize(S& s) {
        s.ext(*this, bitsery::ext::BaseClass<Base>{});
        s.value4b(derived1_value);
    }
};

struct Derived2 : Base {
    std::string derived2_value = "test";
    
    template<typename S>
    void serialize(S& s) {
        s.ext(*this, bitsery::ext::BaseClass<Base>{});
        s.text1b(derived2_value, 20);
    }
};

// Register polymorphic types
namespace bitsery {
namespace ext {
    template<typename TBase>
    struct PolymorphicBaseClass;
    
    template <>
    struct PolymorphicBaseClass<Base> {
        template<typename S>
        void serialize(S& s, Base& obj) {
            s.ext(obj, bitsery::ext::PolymorphicType{
                std::make_tuple(bitsery::ext::PolymorphicSubclass<Derived1>{},
                               bitsery::ext::PolymorphicSubclass<Derived2>{})
            });
        }
    };
}
}

int main() {
    std::unique_ptr<Base> ptr1 = std::make_unique<Derived1>();
    std::unique_ptr<Base> ptr2 = std::make_unique<Derived2>();
    
    std::vector<uint8_t> buffer;
    
    // Serialization
    auto writtenSize = bitsery::quickSerialization(buffer, ptr1, ptr2);
    
    // Deserialization
    std::unique_ptr<Base> loaded_ptr1;
    std::unique_ptr<Base> loaded_ptr2;
    
    auto state = bitsery::quickDeserialization({buffer.begin(), writtenSize}, 
                                              loaded_ptr1, loaded_ptr2);
    
    if (state.first == bitsery::ReaderError::NoError) {
        std::cout << "Successfully deserialized polymorphic objects" << std::endl;
    }
    
    return 0;
}

Forward/Backward Compatibility

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <bitsery/ext/growable.h>

// Version 1 data structure
struct ConfigV1 {
    std::string app_name;
    uint32_t version;
    
    template<typename S>
    void serialize(S& s) {
        s.text1b(app_name, 50);
        s.value4b(version);
    }
};

// Version 2 data structure (added new field)
struct ConfigV2 {
    std::string app_name;
    uint32_t version;
    std::string new_field = "default";  // New field
    
    template<typename S>
    void serialize(S& s) {
        // Use Growable extension to maintain compatibility
        s.ext(*this, bitsery::ext::Growable{}, [](S& s, ConfigV2& o) {
            // Required fields (needed in all versions)
            s.text1b(o.app_name, 50);
            s.value4b(o.version);
        }, [](S& s, ConfigV2& o) {
            // Optional fields (added in new versions)
            s.text1b(o.new_field, 100);
        });
    }
};

int main() {
    // Read old version data with new version
    ConfigV1 old_config{"MyApp", 1};
    std::vector<uint8_t> buffer;
    
    // Save as V1 format
    auto writtenSize = bitsery::quickSerialization(buffer, old_config);
    
    // Load as V2 format (compatible)
    ConfigV2 new_config;
    auto state = bitsery::quickDeserialization({buffer.begin(), writtenSize}, new_config);
    
    if (state.first == bitsery::ReaderError::NoError) {
        std::cout << "Successfully loaded old version data" << std::endl;
        std::cout << "New field has default value: " << new_config.new_field << std::endl;
    }
    
    return 0;
}

Custom Type Extension

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <chrono>

// Custom serialization for std::chrono::time_point
namespace bitsery {
namespace ext {
    template<typename Clock, typename Duration>
    class TimePointExt {
    public:
        template<typename Ser, typename T>
        void serialize(Ser& s, T& tp) {
            auto duration = tp.time_since_epoch();
            auto count = duration.count();
            s.value8b(count);
            if (s.adapter().isReading()) {
                tp = T(Duration(count));
            }
        }
    };
}
}

struct Event {
    std::string name;
    std::chrono::system_clock::time_point timestamp;
    
    template<typename S>
    void serialize(S& s) {
        s.text1b(name, 100);
        s.ext(timestamp, bitsery::ext::TimePointExt<
              std::chrono::system_clock, 
              std::chrono::system_clock::duration>{});
    }
};

int main() {
    Event event{"Important Event", std::chrono::system_clock::now()};
    
    std::vector<uint8_t> buffer;
    auto writtenSize = bitsery::quickSerialization(buffer, event);
    
    Event loaded_event;
    auto state = bitsery::quickDeserialization({buffer.begin(), writtenSize}, loaded_event);
    
    if (state.first == bitsery::ReaderError::NoError) {
        std::cout << "Event: " << loaded_event.name << std::endl;
    }
    
    return 0;
}