Development Tool

cURL

Overview

cURL is the most fundamental command-line HTTP client. Pre-installed on virtually all systems and widely used for automation and scripting. It maintains an unwavering position as a foundational tool and continues to play an important role as a reference implementation even with new API tools emerging.

Details

cURL is a command-line tool and library for transferring data using URL syntax. It supports over 30 protocols including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, SMTP, LDAP, and MQTT, and has been continuously developed for over 30 years as a foundation of web technology. The latest stable version is 8.14.1 (released June 4, 2025). In the field of HTTP scripting, it supports important modern tasks such as automatic information extraction from the web, user simulation, and data posting/uploading to web servers. Authentication features support a wide range of methods including Basic, Digest, NTLM, Negotiate, and Kerberos, with the ability to automatically select the most secure method. With automation support through configuration files (.curlrc), detailed cookie management, proxy support, and TLS/SSL encryption, it can be safely used in enterprise environments. Its design optimized for batch processing and scripting makes it indispensable in CI/CD pipelines, monitoring systems, and automation scripts. With its simple yet powerful command system, it's a high-investment-return tool that can be used long-term once learned.

Advantages & Disadvantages

Advantages

• Universal Presence: Pre-installed on virtually all systems
• Rich Protocol Support: Supports 30+ protocols beyond HTTP/HTTPS
• Complete Authentication Support: Supports Basic, Digest, NTLM, Kerberos, etc.
• Optimal for Automation: Specialized for script and batch processing usage
• Lightweight & Fast: High performance with minimal resources
• Configuration File Support: Automation and customization via .curlrc
• Complete Proxy Support: HTTP, SOCKS4, SOCKS5 proxy support
• Free & Open Source: Long-term stable licensing
• Track Record & Reliability: 30+ years of development history and widespread adoption

Disadvantages

• No GUI: Command-line only without visual interface
• Learning Curve: Complex options due to extensive features
• Basic Output: Limited response visualization and formatting capabilities
• Configuration Management: Managing complex configurations can be cumbersome
• Debugging Difficulties: Troubleshooting can be challenging when errors occur

Reference Links

• cURL Official Site
• cURL Documentation
• cURL Manual
• cURL Tutorial
• GitHub - curl/curl

Usage Examples

Basic HTTP Requests

# Basic GET request
curl https://api.example.com/users

# GET request with headers
curl -H "Authorization: Bearer token123" \
     -H "Accept: application/json" \
     https://api.example.com/users

# POST request with JSON data
curl -X POST \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer token123" \
     -d '{"name":"John Doe","email":"[email protected]","age":30}' \
     https://api.example.com/users

# PUT request
curl -X PUT \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer token123" \
     -d '{"name":"Jane Doe","active":true}' \
     https://api.example.com/users/123

# DELETE request
curl -X DELETE \
     -H "Authorization: Bearer token123" \
     https://api.example.com/users/123

Authentication Methods

# Basic authentication
curl -u username:password https://api.example.com/protected

# Basic authentication (password prompt)
curl -u username https://api.example.com/protected

# Bearer Token authentication
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." \
     https://api.example.com/profile

# API key authentication (header)
curl -H "X-API-Key: your-api-key-here" \
     https://api.example.com/data

# API key authentication (query parameter)
curl "https://api.example.com/search?q=example&api_key=your-api-key"

# Digest authentication
curl --digest -u username:password \
     https://api.example.com/digest-protected

# Automatic authentication from .netrc file
# Set authentication info in ~/.netrc
cat ~/.netrc
# machine api.example.com
# login myusername  
# password mypassword

curl https://api.example.com/auto-auth

# Ignore .netrc
curl --netrc-optional https://api.example.com/protected

Data Transmission and File Operations

# Send data from JSON file
curl -X POST \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer token123" \
     -d @user-data.json \
     https://api.example.com/users

# File upload (multipart/form-data)
curl -X POST \
     -H "Authorization: Bearer token123" \
     -F "[email protected]" \
     -F "title=Important Document" \
     -F "category=documents" \
     https://api.example.com/upload

# Binary file transmission
curl -X POST \
     -H "Content-Type: application/octet-stream" \
     -H "Authorization: Bearer token123" \
     --data-binary @image.png \
     https://api.example.com/upload/binary

# Save response to file
curl -o response.json \
     -H "Authorization: Bearer token123" \
     https://api.example.com/data

# Save response to file (using remote filename)
curl -O \
     -H "Authorization: Bearer token123" \
     https://api.example.com/files/report.pdf

# Download large file (with progress bar)
curl -# -o large-file.zip \
     https://example.com/downloads/large-file.zip

Detailed Output and Debugging

# Display response headers only
curl -I https://api.example.com/health

# Display both headers and body
curl -i -H "Authorization: Bearer token123" \
     https://api.example.com/users

# Display detailed debug information
curl -v -H "Authorization: Bearer token123" \
     https://api.example.com/users

# Very detailed trace information
curl --trace-ascii trace.log \
     -H "Authorization: Bearer token123" \
     https://api.example.com/users

# Measure request time
curl -w "Total time: %{time_total}s\nConnect time: %{time_connect}s\nHTTP code: %{http_code}\n" \
     -o /dev/null -s \
     https://api.example.com/performance

# Format JSON response (combined with jq)
curl -s -H "Authorization: Bearer token123" \
     https://api.example.com/users | jq '.'

# Parallel processing of multiple URLs
curl -s https://api.example.com/endpoint1 & \
curl -s https://api.example.com/endpoint2 & \
curl -s https://api.example.com/endpoint3 & \
wait

Proxy and Network Configuration

# Use HTTP proxy
curl --proxy http://proxy.example.com:8080 \
     https://api.example.com/data

# Proxy with authentication
curl --proxy http://user:[email protected]:8080 \
     https://api.example.com/data

# SOCKS5 proxy
curl --socks5-hostname proxy.example.com:1080 \
     https://api.example.com/data

# Skip SSL certificate verification (for test environments)
curl -k https://self-signed.example.com/api

# Use custom CA certificate
curl --cacert /path/to/ca-bundle.crt \
     https://secure.example.com/api

# Client certificate authentication
curl --cert client.pem --key client.key \
     https://secure.example.com/api

# Specify DNS servers
curl --dns-servers 8.8.8.8,8.8.4.4 \
     https://api.example.com/data

# Follow redirects
curl -L https://api.example.com/redirect-endpoint

# Limit maximum redirects
curl -L --max-redirs 5 \
     https://api.example.com/redirect-endpoint

Cookie Management and Sessions

# Save cookies
curl -c cookies.txt \
     -d "username=user&password=pass" \
     https://example.com/login

# Use saved cookies
curl -b cookies.txt \
     https://example.com/protected-page

# API request with cookies
curl -H "Authorization: Bearer token123" \
     -b "session=abc123; preferences=dark_mode" \
     https://api.example.com/user/settings

# Session management (from login to API calls)
# Step 1: Login
curl -c session-cookies.txt \
     -d "username=testuser&password=testpass" \
     https://example.com/api/login

# Step 2: API access (using session cookies)
curl -b session-cookies.txt \
     https://example.com/api/profile

# Step 3: Logout
curl -b session-cookies.txt \
     -X POST https://example.com/api/logout

Automation and Script Integration

#!/bin/bash
# API monitoring script example

API_ENDPOINT="https://api.example.com/health"
API_KEY="your-api-key-here"
LOG_FILE="/var/log/api-monitor.log"

# Health check function
check_api_health() {
    local response_code
    local response_time
    
    response_code=$(curl -s -o /dev/null \
                         -w "%{http_code}" \
                         -H "X-API-Key: $API_KEY" \
                         --max-time 30 \
                         "$API_ENDPOINT")
    
    response_time=$(curl -s -o /dev/null \
                         -w "%{time_total}" \
                         -H "X-API-Key: $API_KEY" \
                         --max-time 30 \
                         "$API_ENDPOINT")
    
    echo "$(date): HTTP $response_code, Time: ${response_time}s" >> "$LOG_FILE"
    
    if [ "$response_code" != "200" ]; then
        echo "API Error: HTTP $response_code at $(date)" | \
        mail -s "API Alert" [email protected]
    fi
}

# Configuration file (.curlrc) example
cat > ~/.curlrc << 'EOF'
# Default settings
silent
show-error
location
max-time = 30
user-agent = "API-Monitor/1.0"
header = "Accept: application/json"
EOF

# JSON data processing and error handling
api_call_with_retry() {
    local url="$1"
    local data="$2"
    local max_retries=3
    local retry_count=0
    
    while [ $retry_count -lt $max_retries ]; do
        response=$(curl -s -w "\n%{http_code}" \
                       -X POST \
                       -H "Content-Type: application/json" \
                       -H "Authorization: Bearer $API_TOKEN" \
                       -d "$data" \
                       "$url")
        
        http_code=$(echo "$response" | tail -n1)
        response_body=$(echo "$response" | sed '$d')
        
        if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
            echo "$response_body" | jq '.'
            return 0
        else
            echo "Attempt $((retry_count + 1)) failed: HTTP $http_code"
            retry_count=$((retry_count + 1))
            sleep 2
        fi
    done
    
    echo "API call failed after $max_retries attempts"
    return 1
}

# Usage examples
check_api_health

api_call_with_retry "https://api.example.com/users" \
    '{"name":"Test User","email":"[email protected]"}'

Advanced Configuration and Performance

# Concurrent request limiting and rate limiting
for i in {1..10}; do
    curl -s -H "Authorization: Bearer token123" \
         "https://api.example.com/data?page=$i" \
         -o "page_$i.json" &
    
    # Limit to maximum 5 requests per second
    if [ $((i % 5)) -eq 0 ]; then
        sleep 1
    fi
done
wait

# Force HTTP/2 usage
curl --http2 -H "Authorization: Bearer token123" \
     https://api.example.com/data

# Enable Keep-Alive for multiple requests
curl --keepalive-time 60 \
     -H "Authorization: Bearer token123" \
     https://api.example.com/endpoint1 \
     https://api.example.com/endpoint2 \
     https://api.example.com/endpoint3

# Detailed timeout control
curl --connect-timeout 10 \
     --max-time 30 \
     --retry 3 \
     --retry-delay 5 \
     --retry-max-time 120 \
     -H "Authorization: Bearer token123" \
     https://api.example.com/slow-endpoint

# Explicitly specify IPv4/IPv6
curl -4 https://api.example.com/ipv4-only
curl -6 https://api.example.com/ipv6-capable

# Bandwidth limiting
curl --limit-rate 100k \
     -o large-download.zip \
     https://example.com/large-file.zip