← Back to Frameworks

Varnish Cache

High-performance HTTP accelerator. Reverse proxy cache that significantly reduces website response time. Flexible configuration with VCL (Varnish Configuration Language).

Cache ServerHTTP AcceleratorReverse ProxyVCLHigh Performance

Cache Server

Varnish Cache

Overview

Varnish Cache is a high-performance HTTP reverse proxy known as a web application accelerator. It sits in front of any server that speaks HTTP and caches the content, typically achieving 300-1000x speed improvements in delivery. With its unique domain-specific language VCL (Varnish Configuration Language), you can freely write request handling policies. It achieves 20 Gbps delivery performance on regular hardware and boasts processing capabilities of 300,000 requests per second. When configuration changes are made, VCL is compiled to C language and dynamically loaded as a shared object, enabling policy updates without restarting.

Details

Varnish Cache 2024 edition maintains its solid position as the definitive HTTP accelerator for high-traffic websites. With memory-based cache functionality and high-speed delivery through reverse proxy, it plays crucial roles in CDNs and media sites. VCL's powerful configuration capabilities enable extensive customization including cache rules, authentication processing, error handling, ESI (Edge Side Includes) processing, and WebSocket support. With multi-threading support, it efficiently utilizes modern multi-core processors, and its network-bound performance characteristics make network speed the practical constraint.

Key Features

  • Ultra-High Performance: 20 Gbps delivery possible on regular hardware
  • VCL Configuration Language: Flexible policy description and real-time updates
  • Reverse Proxy: High-speed HTTP request processing and caching
  • ESI Support: Partial caching through Edge Side Includes
  • WebSocket Support: Proper handling of upgrade headers
  • Dynamic Configuration Updates: VCL configuration changes without restart

Pros and Cons

Pros

  • Extremely high performance (300-1000x speed improvement)
  • Advanced customization and flexibility through VCL
  • Dynamic configuration change capability without restart
  • Rich health checking and failover functionality
  • Integrated features like ESI, gzip compression, SSL termination
  • Extensive enterprise adoption with proven track record at large-scale sites

Cons

  • High learning cost due to specialized knowledge required for VCL
  • Large memory usage requiring proper sizing
  • Difficult debugging and troubleshooting with complex configurations
  • Increased CPU load during SSL termination
  • Complex cache invalidation control
  • Risk of service impact from configuration mistakes

Reference Links

Code Examples

Installation and Basic Setup

# Ubuntu/Debian installation
sudo apt update
sudo apt install varnish

# CentOS/RHEL/Fedora installation
sudo yum install varnish

# macOS (Homebrew)
brew install varnish

# Source build
curl -s https://packagecloud.io/install/repositories/varnishcache/varnish70/script.deb.sh | sudo bash
sudo apt install varnish

# Start Varnish
sudo systemctl start varnish
sudo systemctl enable varnish

# Configuration file check
varnishd -f /etc/varnish/default.vcl -T localhost:6082 -a :6081 -s malloc,256m

# Operation verification
curl -I http://localhost:6081

Basic VCL Configuration (/etc/varnish/default.vcl)

vcl 4.0;

# Backend server definition
backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 60s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
}

# Additional backend examples
backend webserver1 {
    .host = "192.168.1.10";
    .port = "80";
    .probe = {
        .url = "/health";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

backend webserver2 {
    .host = "192.168.1.11";
    .port = "80";
    .probe = {
        .url = "/health";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

# Load balancer configuration
import directors;

sub vcl_init {
    new cluster = directors.round_robin();
    cluster.add_backend(webserver1);
    cluster.add_backend(webserver2);
}

# Request reception processing
sub vcl_recv {
    # Backend server selection
    if (req.url ~ "^/api/") {
        set req.backend_hint = cluster.backend();
    } else {
        set req.backend_hint = default;
    }
    
    # Static file cache targets
    if (req.url ~ "\.(jpg|jpeg|png|gif|ico|css|js|pdf|txt)$") {
        unset req.http.Cookie;
        return(hash);
    }
    
    # Don't cache POST/PUT/DELETE requests
    if (req.method != "GET" && req.method != "HEAD") {
        return(pass);
    }
    
    # Don't cache admin pages
    if (req.url ~ "^/admin") {
        return(pass);
    }
    
    # Don't cache pages requiring authentication
    if (req.http.Authorization) {
        return(pass);
    }
    
    # Process requests with cookies
    if (req.http.Cookie) {
        # Pass if session cookie exists
        if (req.http.Cookie ~ "session_id") {
            return(pass);
        }
        # Remove other cookies
        unset req.http.Cookie;
    }
    
    return(hash);
}

# Backend response processing
sub vcl_backend_response {
    # Cache time configuration
    if (bereq.url ~ "\.(jpg|jpeg|png|gif|ico)$") {
        set beresp.ttl = 1h;
        set beresp.grace = 1m;
    } elseif (bereq.url ~ "\.(css|js)$") {
        set beresp.ttl = 1h;
        set beresp.grace = 1m;
    } elseif (bereq.url ~ "^/api/") {
        set beresp.ttl = 5m;
        set beresp.grace = 30s;
    } else {
        set beresp.ttl = 10m;
        set beresp.grace = 1m;
    }
    
    # Enable ESI processing
    if (bereq.url == "/dynamic-page") {
        set beresp.do_esi = true;
        set beresp.ttl = 1h;
    }
    
    # Enable gzip compression
    if (beresp.http.content-type ~ "text|javascript|json|xml") {
        set beresp.do_gzip = true;
    }
    
    # Conditions not to cache
    if (beresp.status >= 400) {
        set beresp.ttl = 0s;
    }
    
    # Don't cache if Set-Cookie header exists
    if (beresp.http.Set-Cookie) {
        set beresp.ttl = 0s;
    }
    
    return(deliver);
}

# Pre-delivery processing
sub vcl_deliver {
    # Add cache hit/miss information
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }
    
    # Add debug information (remove in production)
    set resp.http.X-Served-By = "Varnish";
    
    return(deliver);
}

# Custom error page
sub vcl_backend_error {
    if (beresp.status == 503 && bereq.retries < 3) {
        return(retry);
    }
    
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    synthetic({"
    <html>
        <head><title>Service Temporarily Unavailable</title></head>
        <body>
            <h1>Service Temporarily Unavailable</h1>
            <p>The server is temporarily unable to service your request.</p>
        </body>
    </html>
    "});
    
    return(deliver);
}

VCL Management Operations (varnishadm)

# Connect with varnishadm
sudo varnishadm

# VCL configuration management
varnishadm> vcl.list
# Display loaded VCL list

varnishadm> vcl.load new_config /etc/varnish/new.vcl
# Load new VCL configuration

varnishadm> vcl.use new_config
# Activate new configuration

varnishadm> vcl.discard old_config
# Remove old configuration

# Cache management
varnishadm> ban req.url ~ "^/api/"
# Invalidate cache for specific URL pattern

varnishadm> ban req.http.host == "example.com"
# Invalidate cache for specific host

varnishadm> ban.list
# Display ban (invalidation) rule list

# Statistics information check
varnishadm> stats
# Display Varnish statistics

varnishadm> param.show
# Display parameter list

varnishadm> param.set default_ttl 300
# Set default TTL to 5 minutes

# Backend health check status
varnishadm> backend.list
# Check backend server status

# Log output control
varnishadm> param.set vsl_mask +VCL_call,-Hit
# Enable VCL call, disable hit information

Health Check and Failover Configuration

vcl 4.0;

# Detailed health probe configuration
probe healthcheck {
    .url = "/health";
    .request = 
        "GET /health HTTP/1.1"
        "Host: example.com"
        "Connection: close"
        "User-Agent: Varnish Health Probe";
    .timeout = 2s;
    .interval = 5s;
    .window = 5;
    .threshold = 3;
    .initial = 3;
}

# Primary backend
backend primary {
    .host = "192.168.1.10";
    .port = "80";
    .probe = healthcheck;
    .connect_timeout = 5s;
    .first_byte_timeout = 10s;
    .between_bytes_timeout = 2s;
}

# Secondary backend
backend secondary {
    .host = "192.168.1.11";
    .port = "80";
    .probe = healthcheck;
    .connect_timeout = 5s;
    .first_byte_timeout = 10s;
    .between_bytes_timeout = 2s;
}

# Failover director
import directors;

sub vcl_init {
    new failover = directors.fallback();
    failover.add_backend(primary);
    failover.add_backend(secondary);
}

sub vcl_recv {
    set req.backend_hint = failover.backend();
    return(hash);
}

# Retry processing on backend error
sub vcl_backend_error {
    if (beresp.status == 503 && bereq.retries < 2) {
        return(retry);
    }
    
    # Custom error page
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    set beresp.status = 503;
    synthetic({"
    <!DOCTYPE html>
    <html>
    <head>
        <title>Service Unavailable</title>
        <style>
            body { font-family: Arial, sans-serif; text-align: center; padding: 50px; }
            .error { color: #cc0000; }
        </style>
    </head>
    <body>
        <h1 class="error">Service Temporarily Unavailable</h1>
        <p>We apologize, but the service is temporarily unavailable.</p>
        <p>Please try again later.</p>
        <p><a href="javascript:history.back()">Go Back</a></p>
    </body>
    </html>
    "});
    
    return(deliver);
}

ESI (Edge Side Includes) Configuration Example

# ESI backend configuration
sub vcl_backend_response {
    # ESI processing target page configuration
    if (bereq.url ~ "^/dynamic/") {
        set beresp.do_esi = true;
        set beresp.ttl = 1h;  # Main content TTL
    }
    
    # ESI fragment configuration
    if (bereq.url ~ "^/fragments/") {
        set beresp.ttl = 5m;  # Fragment TTL
    }
    
    return(deliver);
}

HTML template ESI usage example:

<!DOCTYPE html>
<html>
<head>
    <title>Dynamic Page with ESI</title>
</head>
<body>
    <h1>Main Content</h1>
    <p>This page is cached for 1 hour.</p>
    
    <!-- User-specific information (5-minute cache) -->
    <esi:include src="/fragments/user-info?user=123" />
    
    <!-- Latest news (1-minute cache) -->
    <esi:include src="/fragments/latest-news" ttl="60" />
    
    <!-- Weather information (no cache) -->
    <esi:include src="/fragments/weather" ttl="0" />
    
    <!-- Fallback for errors -->
    <esi:include src="/fragments/ads">
        <esi:alt>
            <p>Failed to load advertisements.</p>
        </esi:alt>
    </esi:include>
</body>
</html>

WebSocket Support Configuration

# WebSocket connection processing
sub vcl_recv {
    # Transfer WebSocket upgrade requests via pipe
    if (req.http.upgrade ~ "(?i)websocket") {
        return(pipe);
    }
}

sub vcl_pipe {
    # Transfer upgrade headers
    if (req.http.upgrade) {
        set bereq.http.upgrade = req.http.upgrade;
        set bereq.http.connection = req.http.connection;
    }
}

SSL/TLS Termination and HTTPS Configuration

# Example combination with Nginx (SSL termination)
# /etc/nginx/sites-available/varnish-ssl

server {
    listen 443 ssl http2;
    server_name example.com;
    
    ssl_certificate /path/to/certificate.crt;
    ssl_certificate_key /path/to/private.key;
    
    location / {
        proxy_pass http://127.0.0.1:6081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

HTTPS processing in VCL:

sub vcl_recv {
    # HTTPS redirect processing
    if (req.http.X-Forwarded-Proto != "https") {
        return(synth(301, "https://" + req.http.host + req.url));
    }
    
    # Security header configuration
    if (req.http.host == "example.com") {
        set req.http.X-Forwarded-Proto = "https";
    }
}

sub vcl_deliver {
    # Add security headers
    set resp.http.Strict-Transport-Security = "max-age=31536000; includeSubDomains";
    set resp.http.X-Content-Type-Options = "nosniff";
    set resp.http.X-Frame-Options = "DENY";
    set resp.http.X-XSS-Protection = "1; mode=block";
    
    return(deliver);
}

Monitoring and Logging Configuration

# Access log with varnishlog
sudo varnishlog -w /var/log/varnish/access.log

# Filtering by specific conditions
sudo varnishlog -q "ReqURL ~ '^/api/'" -w /var/log/varnish/api.log

# Real-time monitoring
sudo varnishlog -q "RespStatus >= 400"

# Statistics information acquisition
varnishstat -1

# Prometheus metrics output
varnish_exporter --varnish.instance=""