Monitoring Stack¶

Deploy Prometheus, Grafana, and Alertmanager using Docker Compose.

Overview¶

This example demonstrates:

Docker Compose service management with systemd
Multi-container orchestration
Volume and directory setup
Configuration templating
Secret injection

Architecture¶

                    ┌─────────────────────────────────────────────┐
                    │            Monitoring Stack                  │
   ┌────────────────┼──────────────────────────────────────────────┤
   │                │                                              │
   │   :3000    ┌───┴───┐   :9090    ┌────────────┐   :9093       │
   │ ◄─────────▶│Grafana│◄──────────▶│ Prometheus │◄──────────────│
   │            └───────┘            └─────┬──────┘               │
   │                                       │                       │
   │                                       ▼                       │
   │                              ┌─────────────────┐              │
   │                              │  Alertmanager   │              │
   │                              └─────────────────┘              │
   │                                       ▲                       │
   │                                       │                       │
   │            ┌──────────────────────────┤                       │
   │            ▼                          ▼                       │
   │    ┌───────────────┐          ┌───────────────┐              │
   │    │ Node Exporter │          │ cAdvisor      │              │
   │    │    :9100      │          │   :8080       │              │
   │    └───────────────┘          └───────────────┘              │
   │                                                               │
   └───────────────────────────────────────────────────────────────┘

Files¶

examples/monitoring_stack/
├── deploy.py
├── templates/
│   ├── docker-compose.yml.j2
│   ├── prometheus.yml.j2
│   └── grafana.ini.j2
└── README.md

Configuration¶

# Stack configuration
STACK_NAME = "monitoring"
STACK_DIR = "/opt/monitoring"

# Ports
GRAFANA_PORT = 3000
PROMETHEUS_PORT = 9090
ALERTMANAGER_PORT = 9093

# Grafana
GRAFANA_ADMIN_PASSWORD = "secure_password"

# Retention
PROMETHEUS_RETENTION = "30d"

# Targets to monitor
SCRAPE_TARGETS = [
    {"job": "node", "targets": ["localhost:9100"]},
    {"job": "docker", "targets": ["localhost:8080"]},
]

Usage¶

# Preview changes
python examples/monitoring_stack/deploy.py --dry-run

# Deploy
sudo python examples/monitoring_stack/deploy.py

# Access Grafana at http://localhost:3000

What Gets Deployed¶

1. Directory Structure¶

/opt/monitoring/
├── docker-compose.yml
├── prometheus/
│   └── prometheus.yml
├── grafana/
│   ├── grafana.ini
│   └── provisioning/
│       ├── datasources/
│       └── dashboards/
├── alertmanager/
│   └── alertmanager.yml
└── data/
    ├── prometheus/
    └── grafana/

2. Docker Compose Configuration¶

/opt/monitoring/docker-compose.yml:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus:/etc/prometheus
      - ./data/prometheus:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./data/grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager:/etc/alertmanager

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro

3. Prometheus Configuration¶

/opt/monitoring/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'docker'
    static_configs:
      - targets: ['cadvisor:8080']

4. Grafana Configuration¶

/opt/monitoring/grafana/grafana.ini:

[server]
http_port = 3000

[security]
admin_user = admin

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false

5. Grafana Datasource Provisioning¶

/opt/monitoring/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

6. Systemd Service¶

from fscm.modules import systemd

def create_systemd_service():
    systemd.docker_compose_service(
        name="monitoring",
        compose_file=f"{STACK_DIR}/docker-compose.yml",
        description="Monitoring Stack (Prometheus + Grafana)"
    )

Key Code Sections¶

Setup Directories¶

def setup_directories():
    dirs = [
        f"{STACK_DIR}",
        f"{STACK_DIR}/prometheus",
        f"{STACK_DIR}/grafana",
        f"{STACK_DIR}/grafana/provisioning/datasources",
        f"{STACK_DIR}/grafana/provisioning/dashboards",
        f"{STACK_DIR}/alertmanager",
        f"{STACK_DIR}/data/prometheus",
        f"{STACK_DIR}/data/grafana",
    ]
    for d in dirs:
        mkdir(d, mode="0755")

    # Grafana data needs to be writable by container user (uid 472)
    chown(f"{STACK_DIR}/data/grafana", "472:472")

Deploy Configurations¶

def deploy_configs():
    # Docker Compose
    compose = template(
        "templates/docker-compose.yml.j2",
        prometheus_port=PROMETHEUS_PORT,
        grafana_port=GRAFANA_PORT,
        alertmanager_port=ALERTMANAGER_PORT,
        retention=PROMETHEUS_RETENTION
    )
    file(f"{STACK_DIR}/docker-compose.yml", compose)

    # Prometheus
    prom_config = template(
        "templates/prometheus.yml.j2",
        scrape_targets=SCRAPE_TARGETS
    )
    file(f"{STACK_DIR}/prometheus/prometheus.yml", prom_config)

    # Grafana
    grafana_config = template("templates/grafana.ini.j2")
    file(f"{STACK_DIR}/grafana/grafana.ini", grafana_config)

    # Grafana datasource
    datasource = """apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
"""
    file(f"{STACK_DIR}/grafana/provisioning/datasources/prometheus.yml", datasource)

Management¶

Start Stack¶

cd /opt/monitoring
docker-compose up -d

View Logs¶

docker-compose logs -f prometheus
docker-compose logs -f grafana

Restart After Config Change¶

docker-compose restart prometheus

Using Systemd¶

systemctl status monitoring
systemctl restart monitoring
journalctl -u monitoring -f

Accessing Services¶

Service	URL	Default Credentials
Grafana	http://localhost:3000	admin / (configured password)
Prometheus	http://localhost:9090	—
Alertmanager	http://localhost:9093	—

Troubleshooting¶

Check Container Status¶

docker-compose ps

Prometheus Targets¶

Visit http://localhost:9090/targets to see scrape status.

Grafana Issues¶

# Check logs
docker logs grafana

# Reset admin password
docker exec -it grafana grafana-cli admin reset-admin-password newpassword