← Back to Libraries

MessagePack

SerializationBinaryMulti-languageHigh PerformanceCompactReal-time

Library

MessagePack

Overview

MessagePack is an efficient binary serialization format. While maintaining JSON-like ease of use, it achieves faster and more compact data representation. Implemented in over 50 languages with high portability, it's continuously chosen for use cases where data size reduction is important, such as IoT, mobile applications, and real-time communication. It plays a crucial role in modern applications requiring game development, microservices, and high-frequency data exchange.

Details

MessagePack 2025 version is a mature binary format that achieves 30-50% size reduction and over 10x processing speed improvement compared to JSON. It eliminates JSON format constraints, enabling direct storage of binary data, support for non-UTF-8 encoded strings, and arbitrary type map keys (including arrays and numbers). With proven adoption in large-scale services like Redis, Pinterest, and SignalR, its reliability in enterprise environments has been demonstrated.

Key Features

  • Compactness: 30-50% smaller data size than JSON
  • High Speed: Over 10x faster serialization/deserialization
  • Multi-language Support: Implementation in 50+ languages with cross-platform compatibility
  • Type Flexibility: Support for binary data, non-UTF-8 strings, and arbitrary type keys
  • Proven Track Record: Adoption in large-scale services like Redis and Pinterest
  • Standardization: Compatibility through clear specifications and multiple implementations

Pros and Cons

Pros

  • Significant data size reduction (30-50%) and high-speed processing (10x+) compared to JSON
  • High compatibility in multi-platform environments through implementation in 50+ languages
  • No data representation constraints with direct support for binary data and non-UTF-8 strings
  • Reduced battery consumption and data communication costs in IoT devices and mobile environments
  • Proven performance in gaming, financial trading, and real-time communication
  • Low learning cost for implementation due to simple specification

Cons

  • Cannot be directly read/written by humans due to binary format
  • Limited tool and editor support compared to JSON
  • Requires decoding process to check data content during debugging
  • Implementation-dependent limitations on array and integer sizes
  • Some libraries have differences in functionality and optimization levels
  • Limited compression effects for small data

Reference Pages

Code Examples

Basic Setup

# JavaScript/Node.js
npm install msgpack5
# or
npm install @msgpack/msgpack

# Python
pip install msgpack

# Java
# Maven
<dependency>
    <groupId>org.msgpack</groupId>
    <artifactId>msgpack-core</artifactId>
    <version>0.9.8</version>
</dependency>

# C# (.NET)
dotnet add package MessagePack

# Go
go get github.com/vmihailenco/msgpack/v5

# Rust
# Cargo.toml
[dependencies]
msgpack = "1.0"

Basic Usage in JavaScript/Node.js

// Using @msgpack/msgpack (recommended)
import { encode, decode } from '@msgpack/msgpack';

// Basic data structure
const userData = {
  id: 123,
  name: "John Doe",
  email: "[email protected]",
  tags: ["admin", "user"],
  metadata: {
    createdAt: new Date(),
    isActive: true,
    balance: 1234.56
  }
};

// Encode (serialization)
const encoded = encode(userData);
console.log('Encoded size:', encoded.length, 'bytes');
console.log('Encoded data:', encoded);

// Decode (deserialization)
const decoded = decode(encoded);
console.log('Decoded data:', decoded);

// Size comparison with JSON
const jsonString = JSON.stringify(userData);
const jsonBytes = new TextEncoder().encode(jsonString);

console.log('JSON size:', jsonBytes.length, 'bytes');
console.log('MessagePack size:', encoded.length, 'bytes');
console.log('Size reduction:', 
  ((jsonBytes.length - encoded.length) / jsonBytes.length * 100).toFixed(1), '%');

// Binary data processing
const binaryData = {
  filename: "document.pdf",
  content: new Uint8Array([0x25, 0x50, 0x44, 0x46]), // PDF header
  metadata: {
    size: 1024,
    type: "application/pdf"
  }
};

const encodedBinary = encode(binaryData);
const decodedBinary = decode(encodedBinary);
console.log('Binary data preserved:', decodedBinary);

// Array data processing
const timeSeriesData = [];
for (let i = 0; i < 1000; i++) {
  timeSeriesData.push({
    timestamp: Date.now() + i * 1000,
    value: Math.random() * 100,
    status: i % 10 === 0 ? 'alert' : 'normal'
  });
}

const encodedTimeSeries = encode(timeSeriesData);
console.log('Time series data encoded:', encodedTimeSeries.length, 'bytes');

Advanced Usage in Python

import msgpack
import datetime
import numpy as np
from typing import Any, Dict, List

# Custom object serialization
class CustomObject:
    def __init__(self, name: str, value: int):
        self.name = name
        self.value = value
        self.created_at = datetime.datetime.now()

# Custom encoder
def custom_encoder(obj):
    if isinstance(obj, CustomObject):
        return {
            '__type__': 'CustomObject',
            'name': obj.name,
            'value': obj.value,
            'created_at': obj.created_at.isoformat()
        }
    elif isinstance(obj, datetime.datetime):
        return {
            '__type__': 'datetime',
            'value': obj.isoformat()
        }
    elif isinstance(obj, np.ndarray):
        return {
            '__type__': 'numpy_array',
            'dtype': str(obj.dtype),
            'shape': obj.shape,
            'data': obj.tobytes()
        }
    raise TypeError(f"Object of type {type(obj)} is not MessagePack serializable")

# Custom decoder
def custom_decoder(obj):
    if isinstance(obj, dict) and '__type__' in obj:
        if obj['__type__'] == 'CustomObject':
            custom_obj = CustomObject(obj['name'], obj['value'])
            custom_obj.created_at = datetime.datetime.fromisoformat(obj['created_at'])
            return custom_obj
        elif obj['__type__'] == 'datetime':
            return datetime.datetime.fromisoformat(obj['value'])
        elif obj['__type__'] == 'numpy_array':
            return np.frombuffer(
                obj['data'], 
                dtype=obj['dtype']
            ).reshape(obj['shape'])
    return obj

# Complex data structure
complex_data = {
    'users': [
        CustomObject('John Doe', 100),
        CustomObject('Jane Smith', 200)
    ],
    'timestamp': datetime.datetime.now(),
    'matrix': np.array([[1, 2, 3], [4, 5, 6]]),
    'config': {
        'debug': True,
        'timeout': 30.5,
        'features': ['auth', 'logging', 'metrics']
    }
}

# Encode
packed_data = msgpack.packb(complex_data, default=custom_encoder)
print(f"Packed size: {len(packed_data)} bytes")

# Decode
unpacked_data = msgpack.unpackb(packed_data, object_hook=custom_decoder, raw=False)
print(f"Unpacked data: {unpacked_data}")

# Streaming processing
def stream_processing():
    # Streaming encode
    packer = msgpack.Packer(default=custom_encoder)
    
    # Streaming processing of large data
    stream_data = []
    for i in range(10000):
        item = {
            'id': i,
            'name': f'Item {i}',
            'timestamp': datetime.datetime.now(),
            'data': np.random.rand(10)
        }
        stream_data.append(packer.pack(item))
    
    # Concatenate into one stream
    combined_stream = b''.join(stream_data)
    print(f"Stream size: {len(combined_stream)} bytes")
    
    # Streaming decode
    unpacker = msgpack.Unpacker(object_hook=custom_decoder, raw=False)
    unpacker.feed(combined_stream)
    
    decoded_items = []
    for item in unpacker:
        decoded_items.append(item)
    
    print(f"Decoded {len(decoded_items)} items from stream")
    return decoded_items

stream_processing()

# Performance measurement
import time
import json

def performance_benchmark():
    # Generate test data
    test_data = []
    for i in range(10000):
        test_data.append({
            'id': i,
            'name': f'User {i}',
            'email': f'user{i}@example.com',
            'active': i % 2 == 0,
            'score': i * 1.5,
            'tags': [f'tag{j}' for j in range(i % 5)]
        })
    
    # MessagePack encode
    start_time = time.time()
    msgpack_data = msgpack.packb(test_data)
    msgpack_encode_time = time.time() - start_time
    
    # MessagePack decode
    start_time = time.time()
    msgpack_decoded = msgpack.unpackb(msgpack_data, raw=False)
    msgpack_decode_time = time.time() - start_time
    
    # JSON encode
    start_time = time.time()
    json_data = json.dumps(test_data).encode()
    json_encode_time = time.time() - start_time
    
    # JSON decode
    start_time = time.time()
    json_decoded = json.loads(json_data.decode())
    json_decode_time = time.time() - start_time
    
    # Output results
    print(f"MessagePack encode: {msgpack_encode_time:.4f}s")
    print(f"MessagePack decode: {msgpack_decode_time:.4f}s")
    print(f"MessagePack size: {len(msgpack_data)} bytes")
    
    print(f"JSON encode: {json_encode_time:.4f}s")
    print(f"JSON decode: {json_decode_time:.4f}s")
    print(f"JSON size: {len(json_data)} bytes")
    
    print(f"Encode speedup: {json_encode_time / msgpack_encode_time:.1f}x")
    print(f"Decode speedup: {json_decode_time / msgpack_decode_time:.1f}x")
    print(f"Size reduction: {(len(json_data) - len(msgpack_data)) / len(json_data) * 100:.1f}%")

performance_benchmark()

Java Usage

// Assuming Maven dependencies are added
import org.msgpack.core.MessageBufferPacker;
import org.msgpack.core.MessagePack;
import org.msgpack.core.MessageUnpacker;
import org.msgpack.value.Value;
import org.msgpack.value.Variable;

import java.io.IOException;
import java.util.*;

public class MessagePackExample {
    
    // Basic serialization
    public static void basicSerialization() throws IOException {
        // Prepare data
        Map<String, Object> userData = new HashMap<>();
        userData.put("id", 123);
        userData.put("name", "John Doe");
        userData.put("email", "[email protected]");
        userData.put("active", true);
        userData.put("balance", 1234.56);
        userData.put("tags", Arrays.asList("admin", "user"));
        
        Map<String, Object> metadata = new HashMap<>();
        metadata.put("createdAt", System.currentTimeMillis());
        metadata.put("version", "1.0");
        userData.put("metadata", metadata);
        
        // Encode
        MessageBufferPacker packer = MessagePack.newDefaultBufferPacker();
        packer.packValue(MessagePack.newDefaultPacker().packValue(userData).build());
        byte[] packed = packer.toByteArray();
        packer.close();
        
        System.out.println("Packed size: " + packed.length + " bytes");
        
        // Decode
        MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(packed);
        Value value = unpacker.unpackValue();
        unpacker.close();
        
        System.out.println("Unpacked value: " + value);
    }
    
    // High-performance streaming processing
    public static void streamingProcessing() throws IOException {
        // Streaming write of large data
        MessageBufferPacker packer = MessagePack.newDefaultBufferPacker();
        
        // Start array
        packer.packArrayHeader(10000);
        
        for (int i = 0; i < 10000; i++) {
            packer.packMapHeader(4);
            packer.packString("id").packInt(i);
            packer.packString("name").packString("User " + i);
            packer.packString("timestamp").packLong(System.currentTimeMillis());
            packer.packString("active").packBoolean(i % 2 == 0);
        }
        
        byte[] packed = packer.toByteArray();
        packer.close();
        
        System.out.println("Stream packed size: " + packed.length + " bytes");
        
        // Streaming read
        MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(packed);
        
        int arraySize = unpacker.unpackArrayHeader();
        System.out.println("Processing " + arraySize + " items");
        
        int processedCount = 0;
        while (unpacker.hasNext() && processedCount < arraySize) {
            int mapSize = unpacker.unpackMapHeader();
            Map<String, Object> item = new HashMap<>();
            
            for (int i = 0; i < mapSize; i++) {
                String key = unpacker.unpackString();
                Value value = unpacker.unpackValue();
                item.put(key, value);
            }
            
            processedCount++;
            
            // Show progress every 100 items
            if (processedCount % 100 == 0) {
                System.out.println("Processed: " + processedCount + " items");
            }
        }
        
        unpacker.close();
        System.out.println("Total processed: " + processedCount + " items");
    }
    
    // Binary data handling
    public static void binaryDataHandling() throws IOException {
        MessageBufferPacker packer = MessagePack.newDefaultBufferPacker();
        
        // Create binary data
        byte[] binaryData = new byte[1024];
        for (int i = 0; i < binaryData.length; i++) {
            binaryData[i] = (byte) (i % 256);
        }
        
        // Object containing binary data
        packer.packMapHeader(3);
        packer.packString("filename").packString("data.bin");
        packer.packString("size").packInt(binaryData.length);
        packer.packString("content").packBinaryHeader(binaryData.length);
        packer.addPayload(binaryData);
        
        byte[] packed = packer.toByteArray();
        packer.close();
        
        System.out.println("Binary data packed size: " + packed.length + " bytes");
        
        // Read binary data
        MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(packed);
        
        int mapSize = unpacker.unpackMapHeader();
        String filename = null;
        int size = 0;
        byte[] content = null;
        
        for (int i = 0; i < mapSize; i++) {
            String key = unpacker.unpackString();
            switch (key) {
                case "filename":
                    filename = unpacker.unpackString();
                    break;
                case "size":
                    size = unpacker.unpackInt();
                    break;
                case "content":
                    int binarySize = unpacker.unpackBinaryHeader();
                    content = new byte[binarySize];
                    unpacker.readPayload(content);
                    break;
            }
        }
        
        unpacker.close();
        
        System.out.println("Filename: " + filename);
        System.out.println("Size: " + size);
        System.out.println("Content length: " + (content != null ? content.length : 0));
        System.out.println("Binary data integrity: " + 
            (content != null && content.length == binaryData.length && 
             Arrays.equals(content, binaryData) ? "OK" : "FAILED"));
    }
    
    public static void main(String[] args) {
        try {
            System.out.println("=== Basic Serialization ===");
            basicSerialization();
            
            System.out.println("\n=== Streaming Processing ===");
            streamingProcessing();
            
            System.out.println("\n=== Binary Data Handling ===");
            binaryDataHandling();
            
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Web Application Integration and Real-time Communication

// Real-time communication with WebSocket + MessagePack
import { encode, decode } from '@msgpack/msgpack';

class MessagePackWebSocket {
    constructor(url) {
        this.ws = new WebSocket(url);
        this.ws.binaryType = 'arraybuffer';
        this.messageHandlers = new Map();
        
        this.ws.onmessage = (event) => {
            try {
                const data = decode(new Uint8Array(event.data));
                this.handleMessage(data);
            } catch (error) {
                console.error('Failed to decode MessagePack data:', error);
            }
        };
    }
    
    send(type, payload) {
        if (this.ws.readyState === WebSocket.OPEN) {
            const message = {
                type,
                payload,
                timestamp: Date.now()
            };
            
            const encoded = encode(message);
            this.ws.send(encoded);
        }
    }
    
    on(messageType, handler) {
        this.messageHandlers.set(messageType, handler);
    }
    
    handleMessage(data) {
        const handler = this.messageHandlers.get(data.type);
        if (handler) {
            handler(data.payload);
        }
    }
}

// High-speed game data synchronization
class GameStateSync {
    constructor(websocket) {
        this.ws = websocket;
        this.gameState = {
            players: new Map(),
            entities: new Map(),
            worldState: {}
        };
        
        // Handle player state updates
        this.ws.on('player_update', (data) => {
            this.updatePlayer(data);
        });
        
        // Handle entity state updates
        this.ws.on('entity_update', (data) => {
            this.updateEntity(data);
        });
        
        // Regular state synchronization (60FPS)
        setInterval(() => {
            this.syncState();
        }, 1000 / 60);
    }
    
    updatePlayer(playerData) {
        this.gameState.players.set(playerData.id, {
            position: { x: playerData.x, y: playerData.y, z: playerData.z },
            rotation: { x: playerData.rx, y: playerData.ry, z: playerData.rz },
            health: playerData.health,
            timestamp: playerData.timestamp
        });
    }
    
    updateEntity(entityData) {
        this.gameState.entities.set(entityData.id, {
            type: entityData.type,
            position: entityData.position,
            state: entityData.state,
            timestamp: entityData.timestamp
        });
    }
    
    syncState() {
        // Send only changed state (differential sync)
        const changedPlayers = [];
        const changedEntities = [];
        
        // Detect player changes
        this.gameState.players.forEach((player, id) => {
            if (player.timestamp > Date.now() - 100) { // Changes within 100ms
                changedPlayers.push({ id, ...player });
            }
        });
        
        // Detect entity changes
        this.gameState.entities.forEach((entity, id) => {
            if (entity.timestamp > Date.now() - 100) {
                changedEntities.push({ id, ...entity });
            }
        });
        
        if (changedPlayers.length > 0 || changedEntities.length > 0) {
            this.ws.send('state_sync', {
                players: changedPlayers,
                entities: changedEntities,
                timestamp: Date.now()
            });
        }
    }
}

// MessagePack support in REST API
class MessagePackAPI {
    constructor(baseUrl) {
        this.baseUrl = baseUrl;
    }
    
    async request(endpoint, options = {}) {
        const url = `${this.baseUrl}${endpoint}`;
        const headers = {
            'Content-Type': 'application/msgpack',
            'Accept': 'application/msgpack',
            ...options.headers
        };
        
        let body = options.body;
        if (body && typeof body === 'object') {
            body = encode(body);
        }
        
        const response = await fetch(url, {
            ...options,
            headers,
            body
        });
        
        if (response.headers.get('content-type')?.includes('application/msgpack')) {
            const arrayBuffer = await response.arrayBuffer();
            return decode(new Uint8Array(arrayBuffer));
        } else {
            return response.json();
        }
    }
    
    async get(endpoint) {
        return this.request(endpoint, { method: 'GET' });
    }
    
    async post(endpoint, data) {
        return this.request(endpoint, {
            method: 'POST',
            body: data
        });
    }
}

// Usage examples
const wsClient = new MessagePackWebSocket('wss://game.example.com/ws');
const gameSync = new GameStateSync(wsClient);
const api = new MessagePackAPI('https://api.example.com');

// API usage example
async function loadUserData() {
    try {
        const userData = await api.get('/users/123');
        console.log('User data:', userData);
        
        const updateResult = await api.post('/users/123', {
            name: 'John Doe',
            email: '[email protected]',
            preferences: {
                theme: 'dark',
                notifications: true
            }
        });
        
        console.log('Update result:', updateResult);
    } catch (error) {
        console.error('API error:', error);
    }
}

loadUserData();