Graylog Setup Guide for Multiple Postfix MTA Servers on AlmaLinux

0 CentOS, Linux, Operating Systems, Tutorials,

Architecture Overview

This guide sets up:

  • 1 Central Graylog Server (AlmaLinux) – for log collection and analysis
  • Multiple MTA Servers (AlmaLinux with Postfix) – forwarding logs to Graylog

Part 1: Install Graylog Server (Central Server)

System Requirements

  • AlmaLinux 8 or 9
  • Minimum 4GB RAM (8GB+ recommended)
  • 50GB+ disk space for logs
  • Static IP address

Step 1: Update System and Install Prerequisites

# Update system
sudo dnf update -y

# Install required tools
sudo dnf install -y wget curl java-17-openjdk-headless pwgen

Step 2: Install MongoDB

# Add MongoDB repository
cat <<EOF | sudo tee /etc/yum.repos.d/mongodb-org-6.0.repo

[mongodb-org-6.0]

name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/\$releasever/mongodb-org/6.0/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc EOF # Install MongoDB sudo dnf install -y mongodb-org # Enable and start MongoDB sudo systemctl daemon-reload sudo systemctl enable mongod sudo systemctl start mongod # Verify MongoDB is running sudo systemctl status mongod

Step 3: Install OpenSearch (Elasticsearch alternative)

# Add OpenSearch repository
curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo -o /tmp/opensearch-2.x.repo
sudo mv /tmp/opensearch-2.x.repo /etc/yum.repos.d/

# Install OpenSearch
sudo dnf install -y opensearch-2.11.1

# Configure OpenSearch for Graylog
sudo tee -a /etc/opensearch/opensearch.yml > /dev/null <<EOF

# Graylog configuration
cluster.name: graylog
node.name: \${HOSTNAME}
network.host: 0.0.0.0
discovery.type: single-node
action.auto_create_index: false
plugins.security.disabled: true
EOF

# Set JVM heap size (50% of available RAM, max 32GB)
sudo sed -i 's/^-Xms.*/-Xms2g/' /etc/opensearch/jvm.options
sudo sed -i 's/^-Xmx.*/-Xmx2g/' /etc/opensearch/jvm.options

# Enable and start OpenSearch
sudo systemctl daemon-reload
sudo systemctl enable opensearch
sudo systemctl start opensearch

# Wait for OpenSearch to start
sleep 30

# Verify OpenSearch is running
curl -X GET "localhost:9200"

Step 4: Install Graylog Server

# Download and install Graylog repository
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-5.2-repository_latest.rpm

# Install Graylog server
sudo dnf install -y graylog-server

# Generate password secret (save this!)
PASSWORD_SECRET=$(pwgen -N 1 -s 96)
echo "Password Secret: $PASSWORD_SECRET"

# Generate admin password hash (change 'YourAdminPassword' to your desired password)
ADMIN_PASSWORD="YourAdminPassword"
ADMIN_PASSWORD_SHA2=$(echo -n "$ADMIN_PASSWORD" | sha256sum | awk '{print $1}')
echo "Admin Password: $ADMIN_PASSWORD"
echo "Admin Password Hash: $ADMIN_PASSWORD_SHA2"

# Configure Graylog
sudo cp /etc/graylog/server/server.conf /etc/graylog/server/server.conf.backup

# Update configuration
sudo sed -i "s/^password_secret =.*/password_secret = $PASSWORD_SECRET/" /etc/graylog/server/server.conf
sudo sed -i "s/^root_password_sha2 =.*/root_password_sha2 = $ADMIN_PASSWORD_SHA2/" /etc/graylog/server/server.conf

# Set HTTP bind address (change to your server IP)
SERVER_IP=$(hostname -I | awk '{print $1}')
sudo sed -i "s|^#http_bind_address = .*|http_bind_address = $SERVER_IP:9000|" /etc/graylog/server/server.conf
sudo sed -i "s|^#http_publish_uri = .*|http_publish_uri = http://$SERVER_IP:9000/|" /etc/graylog/server/server.conf

# Set Elasticsearch hosts
sudo sed -i "s|^#elasticsearch_hosts = .*|elasticsearch_hosts = http://127.0.0.1:9200|" /etc/graylog/server/server.conf

# Enable and start Graylog
sudo systemctl daemon-reload
sudo systemctl enable graylog-server
sudo systemctl start graylog-server

# Check status
sudo systemctl status graylog-server

Step 5: Configure Firewall

# Open required ports
sudo firewall-cmd --permanent --add-port=9000/tcp  # Graylog web interface
sudo firewall-cmd --permanent --add-port=1514/tcp  # Syslog TCP input
sudo firewall-cmd --permanent --add-port=1514/udp  # Syslog UDP input
sudo firewall-cmd --permanent --add-port=5044/tcp  # Beats input (optional)
sudo firewall-cmd --reload

# Verify ports are open
sudo firewall-cmd --list-ports

Step 6: Access Graylog Web Interface

  1. Open browser: http://YOUR_SERVER_IP:9000
  2. Login with:
    • Username: admin
    • Password: (the password you set in ADMIN_PASSWORD variable)

Part 2: Configure Graylog to Receive Logs

Step 1: Create Syslog Input

  1. Log into Graylog web interface
  2. Navigate to System → Inputs
  3. Select Syslog TCP from dropdown
  4. Click Launch new input
  5. Configure:
    • Title: Postfix Syslog TCP
    • Bind address: 0.0.0.0
    • Port: 1514
    • Store full message: Check this
  6. Click Save

Step 2: Create Postfix Stream

  1. Navigate to Streams
  2. Click Create Stream
  3. Configure:
    • Title: Postfix Logs
    • Description: All Postfix mail server logs
  4. Click Save
  5. Click Manage Rules on the new stream
  6. Add rule:
    • Field: application_name
    • Type: match regular expression
    • Value: postfix.*
  7. Click Save
  8. Click Start stream

Part 3: Configure MTA Servers to Send Logs

Option A: Using Rsyslog (Simpler, Built-in)

On each MTA server, run:

# Backup rsyslog configuration
sudo cp /etc/rsyslog.conf /etc/rsyslog.conf.backup

# Add forwarding configuration
sudo tee /etc/rsyslog.d/10-graylog.conf > /dev/null <<'EOF'
# Queue configuration for reliability
$ActionQueueType LinkedList
$ActionQueueFileName graylog_queue
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueMaxDiskSpace 1g

# Forward all logs to Graylog
*.* @@GRAYLOG_SERVER_IP:1514;RSYSLOG_SyslogProtocol23Format
EOF

# Replace GRAYLOG_SERVER_IP with actual IP
GRAYLOG_IP="192.168.1.100"  # Change this to your Graylog server IP
sudo sed -i "s/GRAYLOG_SERVER_IP/$GRAYLOG_IP/" /etc/rsyslog.d/10-graylog.conf

# Restart rsyslog
sudo systemctl restart rsyslog

# Verify rsyslog is running
sudo systemctl status rsyslog

# Test by sending a test message
logger -t postfix/test "Test message from $(hostname) to Graylog"

Option B: Using Filebeat (More Reliable, Recommended for Production)

On each MTA server, run:

Install Filebeat

sudo rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat <<EOF | sudo tee /etc/yum.repos.d/elastic.repo
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo dnf install -y filebeat

Backup default configuration

sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.backup

Create new configuration

sudo tee /etc/filebeat/filebeat.yml > /dev/null <<‘EOF’
filebeat.inputs:

  • type: log enabled: true paths:
    • /var/log/maillog
    • /var/log/maillog-*
      fields:
      log_type: postfix
      server_role: mta
      hostname: ${HOSTNAME}
      fields_under_root: true

output.logstash:
hosts: [“GRAYLOG_SERVER_IP:5044”]

logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
EOF

Replace GRAYLOG_SERVER_IP

GRAYLOG_IP=”192.168.1.100″ # Change this
sudo sed -i “s/GRAYLOG_SERVER_IP/$GRAYLOG_IP/” /etc/filebeat/filebeat.yml

Replace HOSTNAME placeholder

sudo sed -i “s/\${HOSTNAME}/$(hostname)/” /etc/filebeat/filebeat.yml

Enable and start Filebeat

sudo systemctl enable filebeat
sudo systemctl start filebeat

Check status

bash

# Install Filebeat
sudo rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat <<EOF | sudo tee /etc/yum.repos.d/elastic.repo
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo dnf install -y filebeat

# Backup default configuration
sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.backup

# Create new configuration
sudo tee /etc/filebeat/filebeat.yml > /dev/null <<‘EOF’
filebeat.inputs:
– type: log
enabled: true
paths:
– /var/log/maillog
– /var/log/maillog-*
fields:
log_type: postfix
server_role: mta
hostname: ${HOSTNAME}
fields_under_root: true

output.logstash:
hosts: [“GRAYLOG_SERVER_IP:5044”]

logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
EOF

# Replace GRAYLOG_SERVER_IP
GRAYLOG_IP=”192.168.1.100″ # Change this
sudo sed -i “s/GRAYLOG_SERVER_IP/$GRAYLOG_IP/” /etc/filebeat/filebeat.yml

# Replace HOSTNAME placeholder
sudo sed -i “s/\${HOSTNAME}/$(hostname)/” /etc/filebeat/filebeat.yml

# Enable and start Filebeat
sudo systemctl enable filebeat
sudo systemctl start filebeat

# Check status
sudo systemctl status filebeat

If using Filebeat, also create Beats input in Graylog:

  1. System → Inputs
  2. Select Beats from dropdown
  3. Configure:
    • Title: Filebeat Input
    • Bind address: 0.0.0.0
    • Port: 5044
  4. Click Save

Part 4: Create Postfix Log Extractors

Extract Common Postfix Fields

  1. Navigate to System → Inputs
  2. Click Manage extractors on your Syslog input
  3. Create the following extractors:

Extractor 1: Message ID

  • Type: Regular expression
  • Field: message
  • Pattern: .*(?:postfix[/\w-]*\[\d+\]): ([A-F0-9]+):
  • Store as field: postfix_message_id

Extractor 2: Queue ID

  • Type: Grok pattern
  • Field: message
  • Pattern: %{POSTFIX_QUEUEID:postfix_queue_id}

Extractor 3: From Address

  • Type: Regular expression
  • Field: message
  • Pattern: from=<([^>]*)>
  • Store as field: mail_from

Extractor 4: To Address

  • Type: Regular expression
  • Field: message
  • Pattern: to=<([^>]*)>
  • Store as field: mail_to

Extractor 5: Status

  • Type: Regular expression
  • Field: message
  • Pattern: status=(\w+)
  • Store as field: delivery_status

Part 5: Create Dashboards and Reports

Create Basic Postfix Dashboard

  1. Navigate to Dashboards
  2. Click Create dashboard
  3. Name it: Postfix Mail Traffic
  4. Add widgets:

Widget 1: Message Count Over Time

  • Type: Area chart
  • Metric: Count
  • Time Range: Last 24 hours
  • Interval: 5 minutes

Widget 2: Delivery Status Distribution

  • Type: Pie chart
  • Field: delivery_status
  • Top values: 10

Widget 3: Top Senders

  • Type: Data table
  • Field: mail_from
  • Top values: 20

Widget 4: Top Recipients

  • Type: Data table
  • Field: mail_to
  • Top values: 20

Widget 5: Messages by MTA Server

  • Type: Bar chart
  • Field: source or hostname
  • Top values: 10

Schedule Daily Reports

  1. Navigate to Alerts & Events → Event Definitions
  2. Click Create Event Definition
  3. Configure:
    • Title: Daily Postfix Report
    • Priority: Normal
    • Condition Type: Filter & Aggregation
    • Search Query: application_name:postfix*
    • Aggregation: Count
    • Time Range: Last 24 hours
  4. Add Notification:
    • Type: Email
    • Recipients: Your email address
    • Subject: Daily Postfix Report - {event_timestamp}
  5. Set schedule to run daily at desired time

Part 6: Useful Searches and Queries

Common Search Queries

# All bounced emails
delivery_status:bounced

# Emails from specific sender
mail_from:"user@example.com"

# All emails to specific domain
mail_to:*@example.com

# Emails from specific MTA server
source:"mta1.example.com"

# Failed deliveries in last hour
delivery_status:(deferred OR bounced) AND timestamp:[now-1h TO now]

# Large emails (over 10MB)
size:>10485760

# Emails with specific queue ID
postfix_queue_id:"ABC123DEF456"

# All relay denials
message:*"Relay access denied"*

# Authentication failures
message:*"authentication failed"*

Create Saved Searches

  1. Run a search query
  2. Click Save search
  3. Name it appropriately
  4. Use for quick access or in dashboards

Part 7: Maintenance and Optimization

Index Rotation and Retention

  1. Navigate to System → Indices
  2. Click Index Set (default)
  3. Configure:
    • Index Rotation: Daily (or based on size)
    • Index Retention: 30 days (adjust based on needs)
    • Max number of indices: 30

Performance Tuning

# Adjust JVM heap for Graylog (50% of available RAM)
sudo nano /etc/sysconfig/graylog-server
# Add: GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g"

# Adjust OpenSearch heap
sudo nano /etc/opensearch/jvm.options
# Change -Xms and -Xmx to 50% of available RAM (max 32g)

# Restart services
sudo systemctl restart graylog-server
sudo systemctl restart opensearch

Backup Configuration

# Backup Graylog configuration
sudo cp /etc/graylog/server/server.conf /backup/graylog-server.conf.$(date +%Y%m%d)

# Backup MongoDB (Graylog metadata)
sudo mongodump --out=/backup/mongodb-$(date +%Y%m%d)

# Backup OpenSearch indices (optional, large)
curl -X PUT "localhost:9200/_snapshot/backup_repo" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/backup/opensearch"
  }
}'

Part 8: Troubleshooting

Check Graylog Logs

sudo tail -f /var/log/graylog-server/server.log

Check if Logs are Being Received

# On Graylog server
sudo tcpdump -i any -n port 1514

# On MTA server - test connectivity
telnet GRAYLOG_SERVER_IP 1514

Verify Services are Running

sudo systemctl status mongod
sudo systemctl status opensearch
sudo systemctl status graylog-server

Check OpenSearch Health

curl -X GET "localhost:9200/_cluster/health?pretty"

Common Issues

Issue: Graylog web interface not accessible

  • Solution: Check firewall rules, verify http_bind_address in config

Issue: No logs appearing in Graylog

  • Solution: Check input status, verify MTA can reach Graylog port, check rsyslog/filebeat logs

Issue: High memory usage

  • Solution: Reduce JVM heap sizes, configure index retention, add more RAM

Summary

You now have:

  • ✅ Central Graylog server collecting logs from multiple MTAs
  • ✅ Real-time log analysis and search capabilities
  • ✅ Automated daily reports
  • ✅ Custom dashboards for mail traffic visualization
  • ✅ Long-term log retention and archival

Next Steps:

  1. Fine-tune extractors for your specific Postfix configuration
  2. Create additional dashboards for different reporting needs
  3. Set up alerting for critical events (queue buildups, high bounce rates)
  4. Configure TLS encryption for log transmission (optional but recommended)
  5. Implement user access controls for team members

Documentation Resources:

  • Graylog Official Docs: https://docs.graylog.org/
  • Postfix Log Format: http://www.postfix.org/postconf.5.html
  • OpenSearch Docs: https://opensearch.org/docs/

Leave a Reply

Your email address will not be published. Required fields are marked *