Apache Avro .NET

SerializationC#.NETAvroSchema EvolutionBinary FormatBig Data

Library

Apache Avro .NET

Overview

Apache Avro .NET is the .NET implementation of the Avro data serialization system. It provides a self-describing binary format with schema evolution support, widely used in big data systems and Kafka integration. It offers rich data structures, fast binary format, RPC support, and high compatibility with the Hadoop ecosystem.

Details

Apache Avro .NET 1.12.0 is the official .NET implementation of the Avro protocol. As of 2025, it is actively maintained and supports three serialization methods: GenericRecord, SpecificRecord, and Reflect API. Schemas are defined in JSON format and stored/transmitted with data, enabling schema evolution. Integration with Azure Schema Registry and Confluent Kafka Platform is possible, making it suitable for enterprise-level streaming systems.

Key Features

Schema Evolution: Supports forward and backward compatible schema changes
Three Serialization Modes: Generic, Specific, and Reflect modes
Compact Binary Format: Highly efficient data compression compared to JSON
Code Generation Tools: Class generation from schemas using Apache.Avro.Tools
RPC Support: Remote procedure call implementation
Kafka Integration: Standard usage in streaming data processing

Pros and Cons

Pros

Schema evolution improves long-term system maintainability
Self-describing format integrates metadata with data
Excellent compatibility with Hadoop ecosystem, Kafka, and Spark
High-efficiency data compression and performance through binary format
Supports both dynamic and static typing
Reliability through official Apache support

Cons

Insufficient .NET-specific documentation leads to high learning curve
Incomplete .NET support for logical types (dates, fixed decimals, etc.)
Partial .NET Standard support with limitations in some environments
Binary format is less human-readable compared to JSON
Schema management complexity increases operational overhead
Feature parity with Java version is incomplete

Reference Pages

Code Examples

Basic Setup

<PackageReference Include="Apache.Avro" Version="1.12.0" />
<PackageReference Include="Apache.Avro.Tools" Version="1.12.0" />

Schema Definition

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.avro",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null},
    {"name": "age", "type": "int"},
    {"name": "created", "type": "long"}
  ]
}

Generic Record Usage

using Avro;
using Avro.Generic;
using Avro.IO;
using Avro.File;

// Load schema
var schemaJson = File.ReadAllText("user.avsc");
var schema = Schema.Parse(schemaJson) as RecordSchema;

// Create GenericRecord
var genericRecord = new GenericRecord(schema);
genericRecord.Add("id", 1001L);
genericRecord.Add("name", "John Doe");
genericRecord.Add("email", "[email protected]");
genericRecord.Add("age", 30);
genericRecord.Add("created", DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());

// Write to file
using (var writer = DataFileWriter<GenericRecord>.OpenWriter(
    new GenericDatumWriter<GenericRecord>(schema), "users.avro"))
{
    writer.Append(genericRecord);
}

// Read from file
using (var reader = DataFileReader<GenericRecord>.OpenReader("users.avro"))
{
    foreach (var record in reader.NextEntries)
    {
        Console.WriteLine($"ID: {record["id"]}, Name: {record["name"]}");
    }
}

Specific Record (Code Generation) Usage

# Generate C# classes from schema
avrogen -s user.avsc .

// Use generated classes
var user = new com.example.avro.User
{
    id = 1002,
    name = "Jane Smith",
    email = "[email protected]",
    age = 25,
    created = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()
};

// Serialize
using (var stream = new MemoryStream())
{
    var writer = new SpecificDatumWriter<User>(user.Schema);
    var encoder = new BinaryEncoder(stream);
    writer.Write(user, encoder);
    
    // Get as byte array
    byte[] serializedData = stream.ToArray();
}

Reflect API Usage

using Avro.Reflect;

// POCO class definition
public class Person
{
    public long Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
    public int Age { get; set; }
    public DateTime Created { get; set; }
}

// Serialize with Reflect API
var person = new Person
{
    Id = 1003,
    Name = "Bob Johnson",
    Email = "[email protected]",
    Age = 35,
    Created = DateTime.UtcNow
};

var writer = new ReflectWriter<Person>(person.GetType());
using (var stream = new MemoryStream())
{
    var encoder = new BinaryEncoder(stream);
    writer.Write(person, encoder);
    byte[] data = stream.ToArray();
}

Schema Evolution Example

// Schema with new field added
var newSchemaJson = @"{
  ""type"": ""record"",
  ""name"": ""User"",
  ""fields"": [
    {""name"": ""id"", ""type"": ""long""},
    {""name"": ""name"", ""type"": ""string""},
    {""name"": ""email"", ""type"": [""null"", ""string""], ""default"": null},
    {""name"": ""age"", ""type"": ""int""},
    {""name"": ""created"", ""type"": ""long""},
    {""name"": ""department"", ""type"": ""string"", ""default"": ""Unknown""}
  ]
}";

// Read old data with new schema
var writerSchema = Schema.Parse(originalSchemaJson);
var readerSchema = Schema.Parse(newSchemaJson);
var resolver = new DefaultResolver(writerSchema, readerSchema);

Kafka Integration Example

using Confluent.Kafka;
using Confluent.SchemaRegistry;
using Confluent.SchemaRegistry.Serdes;

// Configure Schema Registry client
var schemaRegistryConfig = new SchemaRegistryConfig
{
    Url = "http://localhost:8081"
};

using var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);

// Configure Avro serializer
var avroSerializerConfig = new AvroSerializerConfig
{
    BufferBytes = 100
};

// Create producer
var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092"
};

using var producer = new ProducerBuilder<string, GenericRecord>(producerConfig)
    .SetValueSerializer(new AvroSerializer<GenericRecord>(schemaRegistry, avroSerializerConfig))
    .Build();