Technology Catalog | Developer's Catalog

Database

OpenSearch

Overview

OpenSearch is an AWS-led distributed search and analytics engine. Forked from Elasticsearch as an open-source project, it's completely free to use under the Apache 2.0 license. It provides fast full-text search, real-time analytics, and machine learning capabilities, supporting a wide range of use cases including log analysis, metrics monitoring, and application search.

Details

OpenSearch was launched by AWS in 2021 in response to Elasticsearch's license changes. In late 2024, the OpenSearch Software Foundation (OSSF) was established under the Linux Foundation, enabling development under neutral governance. Based on Elasticsearch 7.10.2, it continues with unique feature enhancements and optimizations.

Key Features

High-Speed Search: Up to 6x performance improvement (compared to early versions)
Vector Search: High-performance semantic search with Facebook FAISS integration
Hybrid Search: Parallel execution of keyword and vector searches
OpenSearch Dashboards: Data visualization and management UI
Security: Fine-grained access control and multi-tenant support
Alerting: Real-time monitoring and alerting capabilities
Data Prepper: Enhanced data ingestion pipeline
Remote-backed storage: Direct indexing to cloud storage
Segment replication: 25% improvement in data ingestion performance
SIMD optimization: Hardware acceleration support

Architecture

Distributed System: Multi-node cluster support
RESTful API: Elasticsearch-compatible API
Plugin System: Rich extension capabilities
OpenSearch Dashboards: Kibana-compatible visualization tool
Multi-tenancy: Complete isolation between tenants
Cross-cluster: Search and replication across multiple clusters

Pros and Cons

Pros

Fully Open Source: Apache 2.0 license with no commercial restrictions
Elasticsearch Compatible: Easy migration from existing systems
High Performance: Significant performance improvements with latest optimizations
Rich Features: Integrated search, analytics, visualization, and ML capabilities
Strong Security: Enterprise-level security features
Active Development: Vibrant development with 1,400+ contributors
AWS Integration: Tight integration with AWS services
Cost Efficiency: Cost reduction through cloud storage integration

Cons

Relatively New: Ecosystem still developing compared to Elasticsearch
Migration Complexity: Some feature differences when migrating from Elasticsearch
Learning Curve: Need to learn new features and unique extensions
Plugin Limitations: Some Elasticsearch plugins not available
Enterprise Support: Commercial support primarily through AWS

Key Links

Code Examples

Installation & Setup

# Start with Docker (single node)
docker run -d \
  --name opensearch-node1 \
  -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=MyStrongPassword123!" \
  opensearchproject/opensearch:latest

# Cluster setup with Docker Compose
cat > docker-compose.yml << EOF
version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=MyStrongPassword123!"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:
EOF

docker-compose up -d

# Verify startup
curl -k -u admin:MyStrongPassword123! https://localhost:9200

Basic Operations (CRUD)

# Create index
curl -k -X PUT "https://localhost:9200/products" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "name": { "type": "text", "analyzer": "standard" },
        "brand": { "type": "keyword" },
        "category": { "type": "keyword" },
        "price": { "type": "double" },
        "description": { "type": "text" },
        "tags": { "type": "keyword" },
        "created_at": { "type": "date" }
      }
    }
  }'

# Add document
curl -k -X POST "https://localhost:9200/products/_doc/1" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "iPhone 15",
    "brand": "Apple",
    "category": "Smartphone",
    "price": 999.99,
    "description": "Latest iPhone with A17 Pro chip",
    "tags": ["smartphone", "apple", "mobile"],
    "created_at": "2024-01-15"
  }'

# Bulk insert
curl -k -X POST "https://localhost:9200/products/_bulk" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '
{"index": {"_id": "2"}}
{"name": "MacBook Pro", "brand": "Apple", "category": "Laptop", "price": 1999.99, "description": "High-performance laptop with M3 chip", "tags": ["laptop", "apple"], "created_at": "2024-01-10"}
{"index": {"_id": "3"}}
{"name": "Galaxy S24", "brand": "Samsung", "category": "Smartphone", "price": 799.99, "description": "High-performance Android smartphone", "tags": ["smartphone", "samsung", "android"], "created_at": "2024-01-20"}
'

# Get document
curl -k -X GET "https://localhost:9200/products/_doc/1" \
  -u admin:MyStrongPassword123!

# Update document
curl -k -X POST "https://localhost:9200/products/_update/1" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "doc": {
      "price": 949.99,
      "description": "Latest iPhone with A17 Pro chip (price updated)"
    }
  }'

# Delete document
curl -k -X DELETE "https://localhost:9200/products/_doc/3" \
  -u admin:MyStrongPassword123!

Search Queries

# Simple search
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "match": {
        "name": "iPhone"
      }
    }
  }'

# Complex search
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "description": "high-performance" } }
        ],
        "filter": [
          { "term": { "brand": "Apple" } },
          { "range": { "price": { "gte": 500, "lte": 2000 } } }
        ]
      }
    },
    "sort": [
      { "price": { "order": "asc" } }
    ],
    "size": 10,
    "from": 0
  }'

# Fuzzy search
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "fuzzy": {
        "name": {
          "value": "iPhon",
          "fuzziness": "AUTO"
        }
      }
    }
  }'

# Search with highlighting
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "multi_match": {
        "query": "Apple",
        "fields": ["name", "description"]
      }
    },
    "highlight": {
      "fields": {
        "name": {},
        "description": {}
      },
      "pre_tags": ["<mark>"],
      "post_tags": ["</mark>"]
    }
  }'

Aggregations

# Basic aggregations
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 0,
    "aggs": {
      "brands": {
        "terms": {
          "field": "brand",
          "size": 10
        }
      },
      "avg_price": {
        "avg": {
          "field": "price"
        }
      },
      "price_stats": {
        "stats": {
          "field": "price"
        }
      }
    }
  }'

# Nested aggregations
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 0,
    "aggs": {
      "brands": {
        "terms": {
          "field": "brand"
        },
        "aggs": {
          "avg_price_per_brand": {
            "avg": {
              "field": "price"
            }
          },
          "categories": {
            "terms": {
              "field": "category"
            }
          }
        }
      }
    }
  }'

# Range aggregations
curl -k -X GET "https://localhost:9200/products/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 0,
    "aggs": {
      "price_ranges": {
        "range": {
          "field": "price",
          "ranges": [
            { "to": 500 },
            { "from": 500, "to": 1000 },
            { "from": 1000, "to": 2000 },
            { "from": 2000 }
          ]
        }
      }
    }
  }'

JavaScript Client

// OpenSearch client setup
import { Client } from '@opensearch-project/opensearch'

const client = new Client({
  node: 'https://localhost:9200',
  auth: {
    username: 'admin',
    password: 'MyStrongPassword123!'
  },
  ssl: {
    rejectUnauthorized: false // Development only
  }
})

// Create index
await client.indices.create({
  index: 'products',
  body: {
    settings: {
      number_of_shards: 1,
      number_of_replicas: 1
    },
    mappings: {
      properties: {
        name: { type: 'text' },
        brand: { type: 'keyword' },
        category: { type: 'keyword' },
        price: { type: 'double' },
        description: { type: 'text' }
      }
    }
  }
})

// Add document
await client.index({
  index: 'products',
  id: '1',
  body: {
    name: 'iPhone 15',
    brand: 'Apple',
    category: 'Smartphone',
    price: 999.99,
    description: 'Latest iPhone with A17 Pro chip'
  }
})

// Bulk insert
const body = [
  { index: { _index: 'products', _id: '2' } },
  { name: 'MacBook Pro', brand: 'Apple', category: 'Laptop', price: 1999.99 },
  { index: { _index: 'products', _id: '3' } },
  { name: 'Galaxy S24', brand: 'Samsung', category: 'Smartphone', price: 799.99 }
]

await client.bulk({ body })

// Search
const searchResponse = await client.search({
  index: 'products',
  body: {
    query: {
      bool: {
        must: [
          { match: { description: 'high-performance' } }
        ],
        filter: [
          { term: { brand: 'Apple' } },
          { range: { price: { gte: 500 } } }
        ]
      }
    },
    sort: [
      { price: { order: 'asc' } }
    ],
    size: 10
  }
})

console.log('Search results:', searchResponse.body.hits.hits)

// Aggregations
const aggregationResponse = await client.search({
  index: 'products',
  body: {
    size: 0,
    aggs: {
      brands: {
        terms: { field: 'brand' }
      },
      avg_price: {
        avg: { field: 'price' }
      }
    }
  }
})

console.log('Aggregation results:', aggregationResponse.body.aggregations)

Vector Search & AI Features

# Create index for vector search
curl -k -X PUT "https://localhost:9200/documents" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "settings": {
      "index": {
        "knn": true,
        "knn.algo_param.ef_search": 100
      }
    },
    "mappings": {
      "properties": {
        "title": { "type": "text" },
        "content": { "type": "text" },
        "vector_field": {
          "type": "knn_vector",
          "dimension": 768,
          "method": {
            "name": "hnsw",
            "space_type": "l2",
            "engine": "faiss",
            "parameters": {
              "ef_construction": 128,
              "m": 24
            }
          }
        }
      }
    }
  }'

# Add document with vector
curl -k -X POST "https://localhost:9200/documents/_doc/1" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "title": "OpenSearch Usage Guide",
    "content": "OpenSearch is a powerful search engine",
    "vector_field": [0.1, 0.2, 0.3, /* ... 768-dimensional vector */]
  }'

# k-NN search
curl -k -X GET "https://localhost:9200/documents/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 5,
    "query": {
      "knn": {
        "vector_field": {
          "vector": [0.1, 0.2, 0.3, /* ... 768-dimensional query vector */],
          "k": 5
        }
      }
    }
  }'

# Hybrid search (text + vector)
curl -k -X GET "https://localhost:9200/documents/_search" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 10,
    "query": {
      "hybrid": {
        "queries": [
          {
            "match": {
              "content": "search engine"
            }
          },
          {
            "knn": {
              "vector_field": {
                "vector": [0.1, 0.2, 0.3, /* ... */],
                "k": 5
              }
            }
          }
        ]
      }
    }
  }'

Security Configuration

# Create user
curl -k -X PUT "https://localhost:9200/_plugins/_security/api/internalusers/developer" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "password": "DeveloperPassword123!",
    "opendistro_security_roles": ["readall"],
    "backend_roles": ["developer"],
    "attributes": {
      "department": "engineering"
    }
  }'

# Create role
curl -k -X PUT "https://localhost:9200/_plugins/_security/api/roles/products_reader" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "cluster_permissions": ["cluster_composite_ops"],
    "index_permissions": [
      {
        "index_patterns": ["products*"],
        "allowed_actions": ["read", "search"]
      }
    ]
  }'

# Create tenant
curl -k -X PUT "https://localhost:9200/_plugins/_security/api/tenants/development" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "description": "Development environment tenant"
  }'

# Role mapping
curl -k -X PUT "https://localhost:9200/_plugins/_security/api/rolesmapping/products_reader" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "backend_roles": ["developer"],
    "users": ["developer"]
  }'

Monitoring & Alerting

# Create monitor
curl -k -X POST "https://localhost:9200/_plugins/_alerting/monitors" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "monitor",
    "name": "High Price Products Monitor",
    "enabled": true,
    "schedule": {
      "period": {
        "interval": 5,
        "unit": "MINUTES"
      }
    },
    "inputs": [
      {
        "search": {
          "indices": ["products"],
          "query": {
            "query": {
              "range": {
                "price": {
                  "gte": 1500
                }
              }
            }
          }
        }
      }
    ],
    "triggers": [
      {
        "name": "High price trigger",
        "severity": "1",
        "condition": {
          "script": {
            "source": "ctx.results[0].hits.total.value > 5"
          }
        },
        "actions": [
          {
            "name": "Send email",
            "destination_id": "email_destination_id",
            "message_template": {
              "source": "High-priced products found: {{ctx.results.0.hits.total.value}} items"
            }
          }
        ]
      }
    ]
  }'

# Create destination (email)
curl -k -X POST "https://localhost:9200/_plugins/_alerting/destinations" \
  -u admin:MyStrongPassword123! \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "email-destination",
    "type": "email",
    "email": {
      "email_account_id": "default_email",
      "recipients": ["[email protected]"],
      "subject": "OpenSearch Alert",
      "message": "Alert triggered"
    }
  }'

Production Configuration

# opensearch.yml
cluster.name: production-cluster
node.name: opensearch-node-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["opensearch-node-1", "opensearch-node-2", "opensearch-node-3"]
cluster.initial_cluster_manager_nodes: ["opensearch-node-1", "opensearch-node-2", "opensearch-node-3"]

# Memory settings
bootstrap.memory_lock: true

# Security settings
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /path/to/cert.pem
plugins.security.ssl.http.pemkey_filepath: /path/to/key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /path/to/ca.pem

# Performance settings
indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb
thread_pool.search.queue_size: 10000
thread_pool.write.queue_size: 1000

# Logging settings
logger.org.opensearch: INFO
logger.org.opensearch.index.search.slowlog: WARN, index_search_slow_log_file
logger.org.opensearch.index.indexing.slowlog: WARN, index_indexing_slow_log_file