Library

Typhoeus

Overview

Typhoeus is a high-performance HTTP client library for Ruby that "wraps libcurl to make fast and reliable requests." Leveraging the performance of libcurl implemented in C language, it supports both synchronous and asynchronous HTTP requests. Its greatest feature is the unique Hydra system for parallel request processing, enabling efficient concurrent execution of multiple HTTP requests. It's a library valued in the Ruby ecosystem for scenarios requiring high-performance HTTP communication, suitable for enterprise-level application development.

Details

Typhoeus 2025 edition maintains its solid position as a proven high-performance HTTP client built on the stable foundation of libcurl. It provides easy access to the speed and reliability of libcurl from Ruby, excelling particularly in efficiently processing large volumes of HTTP requests. The Hydra system's parallel processing functionality is a unique strength not found in other Ruby HTTP libraries, capable of simultaneously executing dozens to hundreds of HTTP requests to significantly reduce processing time. It can also be used as a Faraday adapter, making integration into existing applications straightforward.

Key Features

• Hydra Parallel Processing: Efficient concurrent execution of multiple HTTP requests
• libcurl Foundation: Fast and stable HTTP communication powered by C language libcurl
• Faraday Integration: Can be used as a Faraday adapter
• Rich Authentication Support: Basic authentication, proxy authentication, custom authentication support
• Streaming Processing: Efficient downloading support for large files
• Advanced Configuration: Detailed settings for SSL/TLS, cookies, compression, timeouts, etc.

Pros and Cons

Pros

• Outstanding performance and stability based on libcurl
• Efficient execution of bulk requests through Hydra parallel processing
• Comprehensive support for rich HTTP features (authentication, proxy, SSL, compression, etc.)
• Excellent integration with the Faraday ecosystem
• Enterprise-level reliability and proven track record
• Proven results in Ruby community for scraping and API integration

Cons

• Setup complexity due to native dependency on libcurl
• High learning cost, excessive for simple HTTP requests
• Potential challenges with installation on Windows environments
• Tends to use more memory than pure Ruby implementations
• Difficult detailed control at Ruby level during debugging
• Unsuitable for scenarios requiring lightweight HTTP clients

Reference Pages

• Typhoeus Official GitHub Repository
• Typhoeus RubyGems Page
• libcurl Official Documentation

Code Examples

Installation and Basic Setup

# Add to Gemfile
gem 'typhoeus'

# Install with Bundler
bundle add typhoeus

# Direct installation
gem install typhoeus

# Check libcurl version
curl --version

# Verification in Ruby
ruby -e "require 'typhoeus'; puts Typhoeus::VERSION"

Basic Requests (GET/POST/PUT/DELETE)

require 'typhoeus'

# Basic GET request
response = Typhoeus.get('https://api.example.com/users')
puts response.code       # 200
puts response.body       # Response body
puts response.headers    # Response headers
puts response.total_time # Execution time (seconds)

# GET request with parameters
response = Typhoeus.get(
  'https://api.example.com/users',
  params: { page: 1, limit: 10, sort: 'created_at' }
)
puts response.effective_url  # Actual request URL

# POST request (sending JSON)
user_data = {
  name: 'John Doe',
  email: '[email protected]',
  age: 30
}

response = Typhoeus.post(
  'https://api.example.com/users',
  body: user_data.to_json,
  headers: {
    'Content-Type' => 'application/json',
    'Authorization' => 'Bearer your-token',
    'Accept' => 'application/json'
  }
)

if response.success?
  created_user = JSON.parse(response.body)
  puts "User created: ID=#{created_user['id']}"
else
  puts "Error: #{response.code} - #{response.body}"
end

# POST request (sending form data)
response = Typhoeus.post(
  'https://api.example.com/login',
  body: { username: 'testuser', password: 'secret123' }
)

# PUT request (data update)
updated_data = { name: 'Jane Doe', email: '[email protected]' }
response = Typhoeus.put(
  'https://api.example.com/users/123',
  body: updated_data.to_json,
  headers: {
    'Content-Type' => 'application/json',
    'Authorization' => 'Bearer your-token'
  }
)

# DELETE request
response = Typhoeus.delete(
  'https://api.example.com/users/123',
  headers: { 'Authorization' => 'Bearer your-token' }
)

puts "Delete completed" if response.code == 204

# HEAD request (header information only)
response = Typhoeus.head('https://api.example.com/users/123')
puts "Content-Length: #{response.headers['Content-Length']}"
puts "Last-Modified: #{response.headers['Last-Modified']}"

# OPTIONS request (check supported methods)
response = Typhoeus.options('https://api.example.com/users')
puts "Allow: #{response.headers['Allow']}"

Advanced Configuration and Customization (Authentication, Proxy, SSL, etc.)

require 'typhoeus'

# Custom header configuration
headers = {
  'User-Agent' => 'MyApp/1.0 (Ruby Typhoeus)',
  'Accept' => 'application/json',
  'Accept-Language' => 'en-US,ja-JP',
  'X-API-Version' => 'v2',
  'X-Request-ID' => SecureRandom.uuid
}

response = Typhoeus.get('https://api.example.com/data', headers: headers)

# Basic authentication
response = Typhoeus.get(
  'https://api.example.com/private',
  userpwd: 'username:password'
)

# Bearer Token authentication
response = Typhoeus.get(
  'https://api.example.com/protected',
  headers: { 'Authorization' => 'Bearer your-jwt-token' }
)

# Timeout configuration
response = Typhoeus.get(
  'https://api.example.com/slow',
  timeout: 30,           # Overall timeout (seconds)
  connecttimeout: 10     # Connection timeout (seconds)
)

# Redirect configuration
response = Typhoeus.get(
  'https://api.example.com/redirect',
  followlocation: true,  # Automatic redirect following
  maxredirs: 5          # Maximum redirect count
)

# Proxy configuration
response = Typhoeus.get(
  'https://api.example.com/data',
  proxy: 'http://proxy.example.com:8080'
)

# Authenticated proxy
response = Typhoeus.get(
  'https://api.example.com/data',
  proxy: 'http://proxy.example.com:8080',
  proxyuserpwd: 'proxy_user:proxy_password'
)

# SSL/TLS configuration
response = Typhoeus.get(
  'https://secure-api.example.com/data',
  ssl_verifypeer: true,  # SSL certificate verification
  ssl_verifyhost: 2,     # Host name verification
  cainfo: '/path/to/ca-bundle.crt'  # CA certificate bundle
)

# Disable SSL verification (development only)
response = Typhoeus.get(
  'https://self-signed.example.com/',
  ssl_verifypeer: false,
  ssl_verifyhost: 0
)

# Cookie configuration
response = Typhoeus.get(
  'https://api.example.com/user-data',
  cookiefile: '/tmp/cookies.txt',
  cookiejar: '/tmp/cookies.txt'
)

# Enable compression
response = Typhoeus.get(
  'https://api.example.com/large-data',
  accept_encoding: 'gzip'
)

# Set custom User-Agent (global)
Typhoeus::Config.user_agent = 'MyApp/1.0 (Custom Agent)'

# Enable verbose output (for debugging)
Typhoeus::Config.verbose = true

Error Handling and Retry Functionality

require 'typhoeus'

# Comprehensive error handling
def safe_request(url, options = {})
  request = Typhoeus::Request.new(url, options)
  
  request.on_complete do |response|
    if response.success?
      puts "Success: #{response.code}"
      return response
    elsif response.timed_out?
      puts "Timeout error: Request timed out"
    elsif response.code == 0
      puts "Connection error: #{response.return_message}"
    else
      puts "HTTP error: #{response.code} - #{response.body}"
    end
  end
  
  response = request.run
  response
end

# Usage example
response = safe_request('https://api.example.com/data', timeout: 10)

# Request with retry functionality
def request_with_retry(url, options = {}, max_retries = 3)
  retries = 0
  
  begin
    response = Typhoeus.get(url, options)
    
    if response.success?
      return response
    elsif response.timed_out? || response.code >= 500
      raise StandardError, "Retryable error: #{response.code}"
    else
      raise StandardError, "Non-retryable error: #{response.code}"
    end
    
  rescue StandardError => e
    retries += 1
    
    if retries <= max_retries
      wait_time = 2 ** retries  # Exponential backoff
      puts "Retry #{retries}/#{max_retries}: Retrying in #{wait_time} seconds"
      sleep(wait_time)
      retry
    else
      puts "Maximum retries reached: #{e.message}"
      raise
    end
  end
end

# Usage example
begin
  response = request_with_retry(
    'https://api.example.com/unstable',
    { timeout: 10 },
    3
  )
  puts "Success: #{response.body}"
rescue StandardError => e
  puts "Finally failed: #{e.message}"
end

# Status code-specific handling
response = Typhoeus.get('https://api.example.com/status-check')

case response.code
when 200
  puts "Success: #{JSON.parse(response.body)}"
when 401
  puts "Authentication error: Please check your token"
when 403
  puts "Permission error: Access denied"
when 404
  puts "Not found: Resource does not exist"
when 429
  puts "Rate limit: Please wait before retrying"
when 500..599
  puts "Server error: #{response.code} - #{response.body}"
else
  puts "Unexpected status: #{response.code}"
end

# Timeout check
response = Typhoeus.get('https://api.example.com/slow', timeout: 5)
if response.timed_out?
  puts "Request timed out"
end

Hydra Parallel Processing and Batch Requests

require 'typhoeus'

# Basic parallel processing
hydra = Typhoeus::Hydra.new
urls = [
  'https://api.example.com/users',
  'https://api.example.com/posts',
  'https://api.example.com/comments',
  'https://api.example.com/categories'
]

# Add requests to queue
requests = urls.map do |url|
  request = Typhoeus::Request.new(url, followlocation: true)
  hydra.queue(request)
  request
end

# Execute in parallel
hydra.run

# Collect results
responses = requests.map do |request|
  {
    url: request.url,
    status: request.response.code,
    body: request.response.body,
    time: request.response.total_time
  }
end

responses.each do |response|
  puts "#{response[:url]}: #{response[:status]} (#{response[:time]}s)"
end

# Parallel processing with callbacks
hydra = Typhoeus::Hydra.new
10.times do |i|
  request = Typhoeus::Request.new("https://api.example.com/items/#{i}")
  
  request.on_complete do |response|
    if response.success?
      data = JSON.parse(response.body)
      puts "Item #{i}: #{data['name']}"
    else
      puts "Error Item #{i}: #{response.code}"
    end
  end
  
  hydra.queue(request)
end

hydra.run
puts "All requests completed"

# Parallel processing with concurrency limit
hydra = Typhoeus::Hydra.new(max_concurrency: 5)

50.times do |i|
  request = Typhoeus::Request.new("https://api.example.com/data/#{i}")
  
  request.on_complete do |response|
    puts "Completed: #{i} - Status: #{response.code}"
  end
  
  hydra.queue(request)
end

hydra.run

# Dynamic request addition
hydra = Typhoeus::Hydra.new

first_request = Typhoeus::Request.new('https://api.example.com/posts/1')
first_request.on_complete do |response|
  if response.success?
    data = JSON.parse(response.body)
    
    # Generate additional requests based on response
    data['related_urls'].each do |url|
      related_request = Typhoeus::Request.new(url)
      hydra.queue(related_request)
    end
  end
end

hydra.queue(first_request)
hydra.run

# Memoization setting (cache)
Typhoeus::Config.memoize = true

hydra = Typhoeus::Hydra.new
2.times do
  hydra.queue(Typhoeus::Request.new('https://api.example.com/cache-test'))
end

hydra.run  # Second request reuses first result

# Disable memoization
Typhoeus::Config.memoize = false

Streaming Processing and File Operations

require 'typhoeus'

# Streaming download of large files
downloaded_file = File.open('large_file.zip', 'wb')
request = Typhoeus::Request.new('https://api.example.com/files/large.zip')

request.on_headers do |response|
  if response.code != 200
    downloaded_file.close
    File.delete('large_file.zip')
    raise "Download failed: #{response.code}"
  end
  
  content_length = response.headers['Content-Length']
  puts "File size: #{content_length} bytes" if content_length
end

total_downloaded = 0
request.on_body do |chunk|
  downloaded_file.write(chunk)
  total_downloaded += chunk.bytesize
  
  # Progress display
  print "\rDownloading: #{total_downloaded} bytes"
  
  # Conditional download abort
  if total_downloaded > 100 * 1024 * 1024  # 100MB limit
    puts "\nDownload aborted due to file size limit"
    :abort
  end
end

request.on_complete do |response|
  downloaded_file.close
  puts "\nDownload completed" if response.success?
end

request.run

# File upload
file_path = '/path/to/upload.pdf'
response = Typhoeus.post(
  'https://api.example.com/upload',
  body: {
    title: 'Upload File',
    description: 'This is a test file',
    file: File.open(file_path, 'rb')
  },
  headers: {
    'Authorization' => 'Bearer your-token'
  }
)

if response.success?
  upload_result = JSON.parse(response.body)
  puts "Upload completed: #{upload_result['file_id']}"
else
  puts "Upload failed: #{response.code}"
end

# Multiple file upload in multipart format
response = Typhoeus.post(
  'https://api.example.com/upload-multiple',
  body: {
    document1: File.open('file1.pdf', 'rb'),
    document2: File.open('file2.docx', 'rb'),
    metadata: { category: 'documents', public: false }.to_json
  },
  headers: {
    'Authorization' => 'Bearer your-token'
  }
)

# Response streaming processing
buffer = StringIO.new
request = Typhoeus::Request.new('https://api.example.com/stream-data')

request.on_body do |chunk|
  buffer.write(chunk)
  
  # Process each chunk
  lines = buffer.string.split("\n")
  
  # Process only complete lines
  complete_lines = lines[0..-2]
  buffer.string = lines.last || ""
  
  complete_lines.each do |line|
    begin
      data = JSON.parse(line)
      puts "Received data: #{data['timestamp']} - #{data['message']}"
    rescue JSON::ParserError
      # Ignore non-JSON lines
    end
  end
end

request.run

Faraday Integration and Practical Usage Examples

require 'faraday'
require 'typhoeus'
require 'typhoeus/adapters/faraday'

# Faraday with Typhoeus adapter
conn = Faraday.new(url: 'https://api.example.com') do |builder|
  builder.request :url_encoded
  builder.response :json
  builder.adapter :typhoeus
end

# Single request
response = conn.get('/users/123')
puts response.body

# Parallel processing with Faraday
conn.in_parallel do
  @user_response = conn.get('/users/123')
  @posts_response = conn.get('/users/123/posts')
  @comments_response = conn.get('/users/123/comments')
end

# Responses available after parallel processing completion
puts "User: #{@user_response.body['name']}"
puts "Posts count: #{@posts_response.body.size}"
puts "Comments count: #{@comments_response.body.size}"

# API client class
class APIClient
  def initialize(base_url, token = nil)
    @conn = Faraday.new(url: base_url) do |builder|
      builder.request :json
      builder.response :json
      builder.adapter :typhoeus
    end
    
    @conn.headers['Authorization'] = "Bearer #{token}" if token
    @conn.headers['User-Agent'] = 'APIClient/1.0'
  end
  
  def get(path, params = {})
    @conn.get(path, params)
  end
  
  def post(path, data = {})
    @conn.post(path, data)
  end
  
  def batch_requests(&block)
    @conn.in_parallel(&block)
  end
end

# Usage example
client = APIClient.new('https://api.example.com', 'your-token')

# Batch requests
client.batch_requests do
  @users = client.get('/users')
  @categories = client.get('/categories')
  @settings = client.get('/settings')
end

# Web scraping with Typhoeus
class WebScraper
  def initialize
    @hydra = Typhoeus::Hydra.new(max_concurrency: 10)
  end
  
  def scrape_urls(urls)
    results = []
    
    urls.each do |url|
      request = Typhoeus::Request.new(url, 
        followlocation: true,
        timeout: 30,
        headers: {
          'User-Agent' => 'Mozilla/5.0 (compatible; WebScraper/1.0)'
        }
      )
      
      request.on_complete do |response|
        if response.success?
          results << {
            url: url,
            title: extract_title(response.body),
            content: extract_content(response.body),
            status: response.code
          }
        else
          puts "Scraping failed: #{url} - #{response.code}"
        end
      end
      
      @hydra.queue(request)
    end
    
    @hydra.run
    results
  end
  
  private
  
  def extract_title(html)
    html.match(/<title>(.*?)<\/title>/i)&.captures&.first&.strip
  end
  
  def extract_content(html)
    # Simple content extraction example
    html.gsub(/<[^>]*>/, '').strip[0..200]
  end
end

# Usage example
scraper = WebScraper.new
urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3'
]

results = scraper.scrape_urls(urls)
results.each do |result|
  puts "#{result[:title]} - #{result[:url]}"
end