Build a Log Aggregation Starter Lab with Loki, Grafana, and Service-Specific Dashboards
Set up a starter log stack with Loki and Grafana so service logs land in one place and can be checked during incidents.
Expected Outcome
A working log aggregation system that collects logs from various services, stores them in Loki, and visualizes them in Grafana with custom dashboards.
Assumptions
- Basic understanding of Docker and Docker Compose
- Familiarity with command line interface
- Access to a Linux-based system or WSL on Windows
Bill of Materials
- Docker installed on your machine
- Docker Compose installed
- Text editor for configuration files
Build Steps
- Set up Docker Compose
Create a Docker Compose file to define the services for Loki and Grafana.
Safe to run: read-only
mkdir log-aggregation-lab cd log-aggregation-lab touch docker-compose.yml
- Configure Docker Compose for Loki and Grafana
Add the necessary configuration for Loki and Grafana in the docker-compose.yml file.
Safe to run: read-only
echo 'version: "3.7"' >> docker-compose.yml echo 'services:' >> docker-compose.yml echo ' loki:' >> docker-compose.yml echo ' image: grafana/loki:latest' >> docker-compose.yml echo ' ports:' >> docker-compose.yml echo ' - "3100:3100"' >> docker-compose.yml echo ' command: -config.file=/etc/loki/loki.yaml' >> docker-compose.yml echo ' volumes:' >> docker-compose.yml echo ' - ./loki:/etc/loki' >> docker-compose.yml echo ' grafana:' >> docker-compose.yml echo ' image: grafana/grafana:latest' >> docker-compose.yml echo ' ports:' >> docker-compose.yml echo ' - "3000:3000"' >> docker-compose.yml echo ' depends_on:' >> docker-compose.yml echo ' - loki' >> docker-compose.yml echo ' environment:' >> docker-compose.yml echo ' - GF_SECURITY_ADMIN_PASSWORD=admin' >> docker-compose.yml
- Create Loki configuration file
Create a configuration file for Loki to define how logs are processed and stored.
Example pattern only. Adjust for your environment before running.
mkdir loki touch loki/loki.yaml echo 'auth_enabled: false' >> loki/loki.yaml echo 'server:' >> loki/loki.yaml echo ' http:' >> loki/loki.yaml echo ' port: 3100' >> loki/loki.yaml echo ' grpc_port: 9095' >> loki/loki.yaml echo 'storage_config:' >> loki/loki.yaml echo ' boltdb_shipper:' >> loki/loki.yaml echo ' active_index_directory: /loki/index' >> loki/loki.yaml echo ' shared_store: filesystem' >> loki/loki.yaml echo ' filesystem:' >> loki/loki.yaml echo ' directory: /loki/chunks' >> loki/loki.yaml echo 'schema_config:' >> loki/loki.yaml echo ' configs:' >> loki/loki.yaml echo ' - from: 2020-10-24' >> loki/loki.yaml echo ' store: boltdb_shipper' >> loki/loki.yaml echo ' schema: v11' >> loki/loki.yaml echo ' index: boltdb_shipper' >> loki/loki.yaml
- Start the services
Use Docker Compose to start Loki and Grafana services.
Changes system state: review before running
docker-compose up -d
- Access Grafana
Open your web browser and navigate to Grafana to set up your dashboards.
Example pattern only. Adjust for your environment before running.
Open http://localhost:3000 Login with username: admin and password: admin
- Add Loki as a data source in Grafana
Configure Grafana to use Loki as a data source for your dashboards. Manual action: In Grafana, go to Configuration > Data Sources; Click on 'Add data source'; Select 'Loki' from the list; Click 'Save & Test'.
Example pattern only. Adjust for your environment before running.
Set the URL to http://loki:3100
- Create service-specific dashboards
Create dashboards in Grafana to visualize logs from specific services. Manual action: In Grafana, go to Dashboard > New Dashboard; Add a new panel and select 'Loki' as the data source.
Example pattern only. Adjust for your environment before running.
Use LogQL queries to filter logs from specific services Save the dashboard with a meaningful name
Validation
- Verify that Loki is running by checking logs with 'docker-compose logs loki'
- Check Grafana dashboards to ensure logs are being displayed correctly
Troubleshooting
- Check service logs before changing the design.
- Confirm ports, paths, credentials, DNS names, and container names match the guide assumptions.
Cleanup or Rollback
- Stop test services you no longer need and keep a copy of working configuration before deleting volumes or data directories.
Next Improvements
- Explore advanced LogQL queries for better log filtering
- Integrate additional services to send logs to Loki
- Set up alerts in Grafana based on log patterns