Elastic Security in a Lab

Introduction to Elastic Security

The heart of Elastic Security/SIEM is Elasticsearch, which is also one of the core components in the Elastic Stack. Elasticsearch is a Lucene-based distributed search and analytics engine that stores data as JSON documents and provides near real-time search capabilities.

Let me cover some key Elasticsearch concepts you need to understand:

Index: A collection of documents with similar characteristics. Think of it as a database in traditional RDBMS terms.

Shards: An Elasticsearch index is divided into shards for horizontal scaling. Each shard is a fully functional Lucene index. Elasticsearch automatically manages shard distribution, though you can configure the number of primary shards when creating an index.

Index Templates: These define settings, mappings, and index patterns that are automatically applied when new indices are created. Modern Elasticsearch uses composable index templates (introduced in 7.8) which are more flexible than legacy templates.

Field Data Types: Elasticsearch supports various data types including text (analyzed for full-text search), keyword (exact values), ip, long, float, date, and many others. The key difference between text and keyword is that text fields are analyzed and tokenized for search, while keyword fields store exact values and are used for filtering, sorting, and aggregations.

Ingest Pipelines: These are preprocessing pipelines that transform documents before indexing. Ingest processors can parse, enrich, and normalize data. While Logstash is still widely used for complex transformations, ingest pipelines handle many common use cases directly within Elasticsearch.

Installation and Setup

For a complete setup guide with Docker Compose configurations and Logstash pipelines, check out my Elastic Stack SOP on GitHub.

Prerequisites

• Docker and Docker Compose installed
• At least 4GB of available RAM
• Log files prepared in the specified formats

Quick Start

1. Environment Setup

# Navigate to the elastic-stack directory
cd docker-compose/elastic-stack

# Review and modify environment variables if needed
cp .env.example .env  # if .env doesn't exist

2. Start the Elastic Stack

# Start all services
docker-compose up -d

# Check service status
docker-compose ps

# View logs
docker-compose logs -f

3. Access Services

• Elasticsearch: https://localhost:9200
• Kibana: http://localhost:5601

Default Credentials:

• Username: elastic
• Password: ChangeMe

Log Processing with Logstash

Now let’s process firewall logs and other structured data with Logstash. While you can install Logstash directly on the host machine for better file system access, using Docker containers provides better isolation and easier management.

Here’s a basic Logstash configuration skeleton for processing JSON logs:

input {
  file {
    path => "/data/logs/cloudflarelogs.ndjson"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  json {
    source => "message"
  }

  # Add timestamp parsing if needed
  date {
    match => [ "timestamp", "ISO8601" ]
  }

  # Remove unwanted fields
  mutate {
    remove_field => [ "message", "host" ]
  }
}

output {
  stdout {
    codec => dots
  }

  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "cloudflare-logs-%{+YYYY.MM.dd}"
    ssl_verification_mode => "none"
    user => "elastic"
    password => "ChangeMe"
  }
}

Key Configuration Notes:

• Use time-based index patterns (e.g., logs-%{+YYYY.MM.dd}) for better performance
• The sincedb_path tracks file reading position
• Multiple filter processors can be chained: grok, csv, mutate, date
• Always include error handling in production configurations

Run the configuration:

logstash -f config.conf

**Elastic is offering free on-demand training for siem module @ https://www.elastic.co/training/elastic-security-for-siem **