1016 lines
30 KiB
Markdown
1016 lines
30 KiB
Markdown
# 🐰 RabbitMQ Management Analysis & Production Guide
|
||
|
||
> **Complete Guide to Managing RabbitMQ in Your FreeLeaps Production Environment**
|
||
> *From configuration to monitoring to troubleshooting*
|
||
|
||
---
|
||
|
||
## 📋 **Table of Contents**
|
||
|
||
1. [🎯 **Quick Start**](#-quick-start)
|
||
2. [🏗️ **Your Production Setup**](#️-your-production-setup)
|
||
3. [🔧 **Current Configuration Analysis**](#-current-configuration-analysis)
|
||
4. [📊 **Management UI Guide**](#-management-ui-guide)
|
||
5. [🔍 **Production Monitoring**](#-production-monitoring)
|
||
6. [🚨 **Troubleshooting Guide**](#-troubleshooting-guide)
|
||
7. [⚡ **Performance Optimization**](#-performance-optimization)
|
||
8. [🔒 **Security Best Practices**](#-security-best-practices)
|
||
9. [📈 **Scaling & High Availability**](#-scaling--high-availability)
|
||
10. [🛠️ **Maintenance Procedures**](#️-maintenance-procedures)
|
||
|
||
---
|
||
|
||
## 🎯 **Quick Start**
|
||
|
||
### **🚀 First Day Checklist**
|
||
- [ ] **Access RabbitMQ Management UI**: Port forward to `http://localhost:15672`
|
||
- [ ] **Check your queues**: Verify `freeleaps.devops.reconciler.*` queues exist
|
||
- [ ] **Monitor connections**: Check if reconciler is connected
|
||
- [ ] **Review metrics**: Check message rates and queue depths
|
||
- [ ] **Test connectivity**: Verify RabbitMQ is accessible from your apps
|
||
|
||
### **🔑 Essential Commands**
|
||
```bash
|
||
# Access your RabbitMQ cluster
|
||
kubectl get pods -n freeleaps-alpha | grep rabbitmq
|
||
|
||
# Port forward to management UI
|
||
kubectl port-forward svc/rabbitmq-headless -n freeleaps-alpha 15672:15672
|
||
|
||
# Check RabbitMQ logs
|
||
kubectl logs -f deployment/rabbitmq -n freeleaps-alpha
|
||
|
||
# Access RabbitMQ CLI
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues
|
||
```
|
||
|
||
---
|
||
|
||
## 🏗️ **Your Production Setup**
|
||
|
||
### **🌐 Production Architecture**
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ RABBITMQ PRODUCTION SETUP │
|
||
├─────────────────────────────────────────────────────────────┤
|
||
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
|
||
│ │ freeleaps- │ │ freeleaps- │ │ freeleaps- │ │
|
||
│ │ devops- │ │ apps │ │ monitoring │ │
|
||
│ │ reconciler │ │ (Your Apps) │ │ (Metrics) │ │
|
||
│ └─────────────────┘ └─────────────────┘ └──────────────┘ │
|
||
│ │ │ │ │
|
||
│ │ AMQP 5672 │ AMQP 5672 │ │
|
||
│ │ HTTP 15672 │ HTTP 15672 │ │
|
||
│ └────────────────────┼────────────────────┘ │
|
||
│ │ │
|
||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||
│ │ RABBITMQ CLUSTER │ │
|
||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||
│ │ │ Node 1 │ │ Node 2 │ │ Node 3 │ │ │
|
||
│ │ │ (Primary) │ │ (Replica) │ │ (Replica) │ │ │
|
||
│ │ │ Port: 5672 │ │ Port: 5672 │ │ Port: 5672 │ │ │
|
||
│ │ │ UI: 15672 │ │ UI: 15672 │ │ UI: 15672 │ │ │
|
||
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||
│ └─────────────────────────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### **📊 Production Namespaces**
|
||
|
||
| **Environment** | **Namespace** | **Purpose** | **Status** |
|
||
|-----------------|---------------|-------------|------------|
|
||
| **Alpha** | `freeleaps-alpha` | Development & Testing | ✅ Active |
|
||
| **Production** | `freeleaps-prod` | Live Production | ✅ Active |
|
||
|
||
### **🔧 Production Services**
|
||
|
||
```bash
|
||
# Your actual RabbitMQ services
|
||
kubectl get svc -n freeleaps-alpha | grep rabbitmq
|
||
kubectl get svc -n freeleaps-prod | grep rabbitmq
|
||
|
||
# Service details:
|
||
# - rabbitmq-headless: Internal cluster communication
|
||
# - rabbitmq: External access (if needed)
|
||
# - rabbitmq-management: Management UI access
|
||
```
|
||
|
||
---
|
||
|
||
## 🔧 **Current Configuration Analysis**
|
||
|
||
### **📋 Configuration Sources**
|
||
|
||
#### **1. Helm Chart Configuration**
|
||
```yaml
|
||
# Location: freeleaps-ops/freeleaps/helm-pkg/3rd/rabbitmq/
|
||
# Primary configuration files:
|
||
# - values.yaml (base configuration)
|
||
# - values.alpha.yaml (alpha environment overrides)
|
||
# - values.prod.yaml (production environment overrides)
|
||
```
|
||
|
||
#### **2. Reconciler Configuration**
|
||
```yaml
|
||
# Location: freeleaps-devops-reconciler/helm/freeleaps-devops-reconciler/values.yaml
|
||
rabbitmq:
|
||
host: "rabbitmq-headless.freeleaps-alpha.svc.cluster.local"
|
||
port: 5672
|
||
username: "user"
|
||
password: "NjlhHFvnDuC7K0ir"
|
||
vhost: "/"
|
||
```
|
||
|
||
#### **3. Python Configuration**
|
||
```python
|
||
# Location: freeleaps-devops-reconciler/reconciler/config/config.py
|
||
RABBITMQ_HOST = os.getenv('RABBITMQ_HOST', 'localhost')
|
||
RABBITMQ_PORT = int(os.getenv('RABBITMQ_PORT', '5672'))
|
||
RABBITMQ_USERNAME = os.getenv('RABBITMQ_USERNAME', 'guest')
|
||
RABBITMQ_PASSWORD = os.getenv('RABBITMQ_PASSWORD', 'guest')
|
||
```
|
||
|
||
### **🔍 Configuration Analysis**
|
||
|
||
#### **✅ What's Working Well**
|
||
1. **Helm-based deployment** - Consistent and repeatable
|
||
2. **Environment separation** - Alpha vs Production
|
||
3. **Clustering enabled** - High availability
|
||
4. **Management plugin** - Web UI available
|
||
5. **Resource limits** - Proper resource management
|
||
|
||
#### **⚠️ Issues Identified**
|
||
|
||
##### **1. Configuration Mismatch**
|
||
```yaml
|
||
# ❌ PROBLEM: Different image versions
|
||
# Helm chart: bitnami/rabbitmq:4.0.6-debian-12-r0
|
||
# Reconciler: rabbitmq:3.12-management-alpine
|
||
|
||
# ❌ PROBLEM: Different credentials
|
||
# Alpha: username: "user", password: "NjlhHFvnDuC7K0ir"
|
||
# Production: Different credentials (not shown in config)
|
||
```
|
||
|
||
##### **2. Security Concerns**
|
||
```yaml
|
||
# ❌ PROBLEM: Hardcoded passwords in values files
|
||
auth:
|
||
username: user
|
||
password: "NjlhHFvnDuC7K0ir" # Should be in Kubernetes secrets
|
||
```
|
||
|
||
##### **3. Network Configuration**
|
||
```yaml
|
||
# ❌ PROBLEM: Inconsistent hostnames
|
||
# Reconciler uses: rabbitmq-headless.freeleaps-alpha.svc.cluster.local
|
||
# But should use service discovery
|
||
```
|
||
|
||
### **🎯 Recommended Improvements**
|
||
|
||
#### **1. Centralized Configuration**
|
||
```yaml
|
||
# Create a centralized RabbitMQ configuration
|
||
# Location: freeleaps-ops/config/rabbitmq/
|
||
rabbitmq-config:
|
||
image:
|
||
repository: bitnami/rabbitmq
|
||
tag: "4.0.6-debian-12-r0"
|
||
auth:
|
||
username: ${RABBITMQ_USERNAME}
|
||
password: ${RABBITMQ_PASSWORD}
|
||
clustering:
|
||
enabled: true
|
||
name: "freeleaps-${ENVIRONMENT}"
|
||
```
|
||
|
||
#### **2. Secret Management**
|
||
```yaml
|
||
# Use Kubernetes secrets instead of hardcoded values
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: rabbitmq-credentials
|
||
namespace: freeleaps-alpha
|
||
type: Opaque
|
||
data:
|
||
username: dXNlcg== # base64 encoded
|
||
password: TmphbEhGdm5EdUM3SzBpcg== # base64 encoded
|
||
```
|
||
|
||
#### **3. Service Discovery**
|
||
```yaml
|
||
# Use consistent service discovery
|
||
# Instead of hardcoded hostnames, use:
|
||
RABBITMQ_HOST: "rabbitmq-headless.${NAMESPACE}.svc.cluster.local"
|
||
```
|
||
|
||
---
|
||
|
||
## 📊 **Management UI Guide**
|
||
|
||
### **🌐 Accessing the Management UI**
|
||
|
||
#### **Method 1: Port Forward (Recommended)**
|
||
```bash
|
||
# Port forward to RabbitMQ management UI
|
||
kubectl port-forward svc/rabbitmq-headless -n freeleaps-alpha 15672:15672
|
||
|
||
# Access: http://localhost:15672
|
||
# Username: user
|
||
# Password: NjlhHFvnDuC7K0ir
|
||
```
|
||
|
||
#### **Method 2: Ingress (If configured)**
|
||
```bash
|
||
# If you have ingress configured for RabbitMQ
|
||
# Access: https://rabbitmq.freeleaps.mathmast.com
|
||
```
|
||
|
||
### **📋 Management UI Features**
|
||
|
||
#### **1. Overview Dashboard**
|
||
- **Cluster status** and health indicators
|
||
- **Node information** and resource usage
|
||
- **Connection counts** and message rates
|
||
- **Queue depths** and performance metrics
|
||
|
||
#### **2. Queues Management**
|
||
```bash
|
||
# Your actual queues to monitor:
|
||
# - freeleaps.devops.reconciler.queue (heartbeat)
|
||
# - freeleaps.devops.reconciler.input (input messages)
|
||
# - freeleaps.devops.reconciler.output (output messages)
|
||
|
||
# Queue operations:
|
||
# - View queue details and metrics
|
||
# - Purge queues (remove all messages)
|
||
# - Delete queues (with safety confirmations)
|
||
# - Monitor message rates and consumer counts
|
||
```
|
||
|
||
#### **3. Exchanges Management**
|
||
```bash
|
||
# Your actual exchanges:
|
||
# - amq.default (default direct exchange)
|
||
# - amq.topic (topic exchange)
|
||
# - amq.fanout (fanout exchange)
|
||
|
||
# Exchange operations:
|
||
# - View exchange properties and bindings
|
||
# - Create new exchanges with custom types
|
||
# - Monitor message routing and performance
|
||
```
|
||
|
||
#### **4. Connections & Channels**
|
||
```bash
|
||
# Monitor your reconciler connections:
|
||
# - Check if reconciler is connected
|
||
# - Monitor connection health and performance
|
||
# - View channel details and limits
|
||
# - Force disconnect if needed
|
||
```
|
||
|
||
#### **5. Users & Permissions**
|
||
```bash
|
||
# Current user setup:
|
||
# - Username: user
|
||
# - Permissions: Full access to vhost "/"
|
||
# - Tags: management
|
||
|
||
# User management:
|
||
# - Create new users for different applications
|
||
# - Set up proper permissions and access control
|
||
# - Monitor user activity and connections
|
||
```
|
||
|
||
### **🔧 Practical UI Operations**
|
||
|
||
#### **Monitoring Your Reconciler**
|
||
```bash
|
||
# 1. Check if reconciler is connected
|
||
# Go to: Connections tab
|
||
# Look for: freeleaps-devops-reconciler connections
|
||
|
||
# 2. Monitor message flow
|
||
# Go to: Queues tab
|
||
# Check: freeleaps.devops.reconciler.* queues
|
||
# Monitor: Message rates and queue depths
|
||
|
||
# 3. Check cluster health
|
||
# Go to: Overview tab
|
||
# Monitor: Node status and resource usage
|
||
```
|
||
|
||
#### **Troubleshooting via UI**
|
||
```bash
|
||
# 1. Check for stuck messages
|
||
# Go to: Queues > freeleaps.devops.reconciler.input
|
||
# Look for: High message count or no consumers
|
||
|
||
# 2. Check connection issues
|
||
# Go to: Connections tab
|
||
# Look for: Disconnected or error states
|
||
|
||
# 3. Monitor resource usage
|
||
# Go to: Overview tab
|
||
# Check: Memory usage and disk space
|
||
```
|
||
|
||
---
|
||
|
||
## 🔍 **Production Monitoring**
|
||
|
||
### **📊 Key Metrics to Monitor**
|
||
|
||
#### **1. Cluster Health**
|
||
```bash
|
||
# Check cluster status
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl cluster_status
|
||
|
||
# Monitor node health
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_nodes
|
||
```
|
||
|
||
#### **2. Queue Metrics**
|
||
```bash
|
||
# Check queue depths
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues name messages consumers
|
||
|
||
# Monitor message rates
|
||
# Use Management UI: Queues tab > Queue details > Message rates
|
||
```
|
||
|
||
#### **3. Connection Metrics**
|
||
```bash
|
||
# Check active connections
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_connections
|
||
|
||
# Monitor connection health
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_channels
|
||
```
|
||
|
||
#### **4. Resource Usage**
|
||
```bash
|
||
# Check memory usage
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl status
|
||
|
||
# Monitor disk usage
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- df -h
|
||
```
|
||
|
||
### **🚨 Alerting Setup**
|
||
|
||
#### **1. Queue Depth Alerts**
|
||
```yaml
|
||
# Alert when queue depth exceeds threshold
|
||
# Queue: freeleaps.devops.reconciler.input
|
||
# Threshold: > 100 messages
|
||
# Action: Send Slack notification
|
||
```
|
||
|
||
#### **2. Connection Loss Alerts**
|
||
```yaml
|
||
# Alert when reconciler disconnects
|
||
# Monitor: freeleaps-devops-reconciler connections
|
||
# Threshold: Connection count = 0
|
||
# Action: Page on-call engineer
|
||
```
|
||
|
||
#### **3. Resource Usage Alerts**
|
||
```yaml
|
||
# Alert when memory usage is high
|
||
# Threshold: Memory usage > 80%
|
||
# Action: Scale up or investigate
|
||
```
|
||
|
||
### **📈 Monitoring Dashboard**
|
||
|
||
#### **Grafana Dashboard**
|
||
```yaml
|
||
# Your existing RabbitMQ dashboard
|
||
# Location: freeleaps-ops/cluster/manifests/freeleaps-monitoring-system/kube-prometheus-stack/dashboards/rabbitmq.yaml
|
||
# Access: https://grafana.mathmast.com
|
||
# Dashboard: RabbitMQ Management Overview
|
||
```
|
||
|
||
#### **Key Dashboard Panels**
|
||
1. **Queue Depth** - Monitor message accumulation
|
||
2. **Message Rates** - Track throughput
|
||
3. **Connection Count** - Monitor client connections
|
||
4. **Memory Usage** - Track resource consumption
|
||
5. **Error Rates** - Monitor failures
|
||
|
||
---
|
||
|
||
## 🚨 **Troubleshooting Guide**
|
||
|
||
### **🔍 Common Issues & Solutions**
|
||
|
||
#### **1. Reconciler Connection Issues**
|
||
|
||
##### **Problem**: Reconciler can't connect to RabbitMQ
|
||
```bash
|
||
# Symptoms:
|
||
# - Reconciler logs show connection errors
|
||
# - No connections in RabbitMQ UI
|
||
# - Pods restarting due to connection failures
|
||
|
||
# Diagnosis:
|
||
kubectl logs -f deployment/freeleaps-devops-reconciler -n freeleaps-devops-system
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_connections
|
||
|
||
# Solutions:
|
||
# 1. Check network connectivity
|
||
kubectl exec -it deployment/freeleaps-devops-reconciler -n freeleaps-devops-system -- ping rabbitmq-headless.freeleaps-alpha.svc.cluster.local
|
||
|
||
# 2. Verify credentials
|
||
kubectl get secret rabbitmq-credentials -n freeleaps-alpha -o yaml
|
||
|
||
# 3. Check RabbitMQ status
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl status
|
||
```
|
||
|
||
#### **2. Queue Message Accumulation**
|
||
|
||
##### **Problem**: Messages stuck in queues
|
||
```bash
|
||
# Symptoms:
|
||
# - High message count in queues
|
||
# - No consumers processing messages
|
||
# - Increasing queue depth
|
||
|
||
# Diagnosis:
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues name messages consumers
|
||
|
||
# Solutions:
|
||
# 1. Check consumer health
|
||
kubectl logs -f deployment/freeleaps-devops-reconciler -n freeleaps-devops-system
|
||
|
||
# 2. Restart consumers
|
||
kubectl rollout restart deployment/freeleaps-devops-reconciler -n freeleaps-devops-system
|
||
|
||
# 3. Purge stuck messages (if safe)
|
||
# Via Management UI: Queues > Queue > Purge
|
||
```
|
||
|
||
#### **3. Memory Pressure**
|
||
|
||
##### **Problem**: RabbitMQ running out of memory
|
||
```bash
|
||
# Symptoms:
|
||
# - High memory usage
|
||
# - Slow performance
|
||
# - Connection drops
|
||
|
||
# Diagnosis:
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl status
|
||
kubectl top pods -n freeleaps-alpha | grep rabbitmq
|
||
|
||
# Solutions:
|
||
# 1. Increase memory limits
|
||
kubectl patch deployment rabbitmq -n freeleaps-alpha -p '{"spec":{"template":{"spec":{"containers":[{"name":"rabbitmq","resources":{"limits":{"memory":"2Gi"}}}]}}}}'
|
||
|
||
# 2. Restart RabbitMQ
|
||
kubectl rollout restart deployment/rabbitmq -n freeleaps-alpha
|
||
|
||
# 3. Check for memory leaks
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues name memory
|
||
```
|
||
|
||
#### **4. Cluster Issues**
|
||
|
||
##### **Problem**: RabbitMQ cluster not healthy
|
||
```bash
|
||
# Symptoms:
|
||
# - Nodes not in sync
|
||
# - Replication lag
|
||
# - Split-brain scenarios
|
||
|
||
# Diagnosis:
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl cluster_status
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_nodes
|
||
|
||
# Solutions:
|
||
# 1. Check node connectivity
|
||
kubectl get pods -n freeleaps-alpha | grep rabbitmq
|
||
|
||
# 2. Restart problematic nodes
|
||
kubectl delete pod rabbitmq-0 -n freeleaps-alpha
|
||
|
||
# 3. Rejoin cluster if needed
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl join_cluster rabbit@rabbitmq-0
|
||
```
|
||
|
||
### **🛠️ Debugging Commands**
|
||
|
||
#### **Essential Debugging Commands**
|
||
```bash
|
||
# Check RabbitMQ status
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl status
|
||
|
||
# List all queues
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues
|
||
|
||
# List all exchanges
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_exchanges
|
||
|
||
# List all bindings
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_bindings
|
||
|
||
# List all connections
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_connections
|
||
|
||
# List all channels
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_channels
|
||
|
||
# Check user permissions
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_users
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_user_permissions user
|
||
```
|
||
|
||
#### **Advanced Debugging**
|
||
```bash
|
||
# Check RabbitMQ logs
|
||
kubectl logs -f deployment/rabbitmq -n freeleaps-alpha
|
||
|
||
# Check system logs
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- journalctl -u rabbitmq-server
|
||
|
||
# Check network connectivity
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- netstat -tlnp
|
||
|
||
# Check disk usage
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- df -h
|
||
|
||
# Check memory usage
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- free -h
|
||
```
|
||
|
||
---
|
||
|
||
## ⚡ **Performance Optimization**
|
||
|
||
### **🎯 Performance Tuning**
|
||
|
||
#### **1. Memory Optimization**
|
||
```yaml
|
||
# Optimize memory settings
|
||
# Location: values.alpha.yaml
|
||
configuration: |-
|
||
# Memory management
|
||
vm_memory_high_watermark.relative = 0.6
|
||
vm_memory_high_watermark_paging_ratio = 0.5
|
||
|
||
# Message store
|
||
msg_store_file_size_limit = 16777216
|
||
msg_store_credit_disc_bound = 4000
|
||
```
|
||
|
||
#### **2. Disk Optimization**
|
||
```yaml
|
||
# Optimize disk settings
|
||
configuration: |-
|
||
# Disk free space
|
||
disk_free_limit.relative = 2.0
|
||
|
||
# Queue master location
|
||
queue_master_locator = min-masters
|
||
|
||
# Message persistence
|
||
queue.default_consumer_prefetch = 50
|
||
```
|
||
|
||
#### **3. Network Optimization**
|
||
```yaml
|
||
# Optimize network settings
|
||
configuration: |-
|
||
# TCP settings
|
||
tcp_listen_options.backlog = 128
|
||
tcp_listen_options.nodelay = true
|
||
|
||
# Heartbeat
|
||
heartbeat = 60
|
||
|
||
# Connection limits
|
||
max_connections = 1000
|
||
max_connections_per_user = 100
|
||
```
|
||
|
||
### **📊 Performance Monitoring**
|
||
|
||
#### **Key Performance Indicators**
|
||
1. **Message Throughput** - Messages per second
|
||
2. **Latency** - Message processing time
|
||
3. **Queue Depth** - Messages waiting to be processed
|
||
4. **Memory Usage** - Heap and process memory
|
||
5. **Disk I/O** - Write and read operations
|
||
|
||
#### **Performance Benchmarks**
|
||
```bash
|
||
# Your expected performance:
|
||
# - Message rate: 1000+ messages/second
|
||
# - Latency: < 10ms for local messages
|
||
# - Queue depth: < 100 messages (normal operation)
|
||
# - Memory usage: < 80% of allocated memory
|
||
# - Disk usage: < 70% of allocated storage
|
||
```
|
||
|
||
---
|
||
|
||
## 🔒 **Security Best Practices**
|
||
|
||
### **🛡️ Current Security Analysis**
|
||
|
||
#### **✅ Security Strengths**
|
||
1. **Network isolation** - RabbitMQ runs in Kubernetes namespace
|
||
2. **Resource limits** - Memory and CPU limits set
|
||
3. **Non-root user** - Runs as non-root in container
|
||
4. **TLS support** - SSL/TLS configuration available
|
||
|
||
#### **⚠️ Security Weaknesses**
|
||
1. **Hardcoded passwords** - Passwords in YAML files
|
||
2. **Default permissions** - Overly permissive user access
|
||
3. **No audit logging** - Limited security event tracking
|
||
4. **No network policies** - No ingress/egress restrictions
|
||
|
||
### **🔧 Security Improvements**
|
||
|
||
#### **1. Secret Management**
|
||
```yaml
|
||
# Use Kubernetes secrets
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: rabbitmq-credentials
|
||
namespace: freeleaps-alpha
|
||
type: Opaque
|
||
data:
|
||
username: dXNlcg== # base64 encoded
|
||
password: <base64-encoded-password>
|
||
---
|
||
# Reference in Helm values
|
||
auth:
|
||
existingSecret: rabbitmq-credentials
|
||
existingSecretPasswordKey: password
|
||
existingSecretUsernameKey: username
|
||
```
|
||
|
||
#### **2. User Access Control**
|
||
```yaml
|
||
# Create application-specific users
|
||
# Instead of one user with full access:
|
||
# - freeleaps-reconciler (reconciler access only)
|
||
# - freeleaps-monitoring (read-only access)
|
||
# - freeleaps-admin (full access, limited to admins)
|
||
```
|
||
|
||
#### **3. Network Policies**
|
||
```yaml
|
||
# Restrict network access
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: NetworkPolicy
|
||
metadata:
|
||
name: rabbitmq-network-policy
|
||
namespace: freeleaps-alpha
|
||
spec:
|
||
podSelector:
|
||
matchLabels:
|
||
app: rabbitmq
|
||
policyTypes:
|
||
- Ingress
|
||
- Egress
|
||
ingress:
|
||
- from:
|
||
- namespaceSelector:
|
||
matchLabels:
|
||
name: freeleaps-devops-system
|
||
ports:
|
||
- protocol: TCP
|
||
port: 5672
|
||
- protocol: TCP
|
||
port: 15672
|
||
```
|
||
|
||
#### **4. Audit Logging**
|
||
```yaml
|
||
# Enable audit logging
|
||
configuration: |-
|
||
# Audit logging
|
||
log.file.level = info
|
||
log.file.rotation.date = $D0
|
||
log.file.rotation.size = 10485760
|
||
|
||
# Security events
|
||
log.security = true
|
||
```
|
||
|
||
---
|
||
|
||
## 📈 **Scaling & High Availability**
|
||
|
||
### **🏗️ Current HA Setup**
|
||
|
||
#### **Cluster Configuration**
|
||
```yaml
|
||
# Your current clustering setup
|
||
clustering:
|
||
enabled: true
|
||
name: "freeleaps-alpha"
|
||
addressType: hostname
|
||
rebalance: false
|
||
forceBoot: false
|
||
partitionHandling: autoheal
|
||
```
|
||
|
||
#### **Replication Strategy**
|
||
```yaml
|
||
# Queue replication
|
||
# - Queues are replicated across cluster nodes
|
||
# - Automatic failover if primary node fails
|
||
# - Data consistency maintained across cluster
|
||
```
|
||
|
||
### **🚀 Scaling Strategies**
|
||
|
||
#### **1. Horizontal Scaling**
|
||
```bash
|
||
# Scale RabbitMQ cluster
|
||
kubectl scale statefulset rabbitmq -n freeleaps-alpha --replicas=5
|
||
|
||
# Verify scaling
|
||
kubectl get pods -n freeleaps-alpha | grep rabbitmq
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl cluster_status
|
||
```
|
||
|
||
#### **2. Vertical Scaling**
|
||
```yaml
|
||
# Increase resource limits
|
||
resources:
|
||
requests:
|
||
cpu: 500m
|
||
memory: 1Gi
|
||
limits:
|
||
cpu: 2000m
|
||
memory: 4Gi
|
||
```
|
||
|
||
#### **3. Queue Partitioning**
|
||
```yaml
|
||
# Partition large queues across nodes
|
||
# Strategy: Hash-based partitioning
|
||
# Benefits: Better performance and fault tolerance
|
||
```
|
||
|
||
### **🔧 High Availability Best Practices**
|
||
|
||
#### **1. Node Distribution**
|
||
```yaml
|
||
# Ensure nodes are distributed across availability zones
|
||
# Use pod anti-affinity to prevent single points of failure
|
||
affinity:
|
||
podAntiAffinity:
|
||
requiredDuringSchedulingIgnoredDuringExecution:
|
||
- labelSelector:
|
||
matchExpressions:
|
||
- key: app
|
||
operator: In
|
||
values:
|
||
- rabbitmq
|
||
topologyKey: kubernetes.io/hostname
|
||
```
|
||
|
||
#### **2. Data Replication**
|
||
```yaml
|
||
# Configure proper replication
|
||
# - All queues should have at least 2 replicas
|
||
# - Use quorum queues for critical data
|
||
# - Monitor replication lag
|
||
```
|
||
|
||
#### **3. Backup Strategy**
|
||
```bash
|
||
# Backup RabbitMQ data
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl export_definitions /tmp/rabbitmq-definitions.json
|
||
|
||
# Restore from backup
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl import_definitions /tmp/rabbitmq-definitions.json
|
||
```
|
||
|
||
---
|
||
|
||
## 🛠️ **Maintenance Procedures**
|
||
|
||
### **📅 Regular Maintenance Tasks**
|
||
|
||
#### **Daily Tasks**
|
||
```bash
|
||
# 1. Check cluster health
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl cluster_status
|
||
|
||
# 2. Monitor queue depths
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_queues name messages
|
||
|
||
# 3. Check connection count
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_connections | wc -l
|
||
|
||
# 4. Review error logs
|
||
kubectl logs --tail=100 deployment/rabbitmq -n freeleaps-alpha | grep ERROR
|
||
```
|
||
|
||
#### **Weekly Tasks**
|
||
```bash
|
||
# 1. Review performance metrics
|
||
# Access Grafana dashboard: RabbitMQ Management Overview
|
||
|
||
# 2. Check disk usage
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- df -h
|
||
|
||
# 3. Review user permissions
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_users
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_user_permissions user
|
||
|
||
# 4. Backup configurations
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl export_definitions /tmp/weekly-backup-$(date +%Y%m%d).json
|
||
```
|
||
|
||
#### **Monthly Tasks**
|
||
```bash
|
||
# 1. Security audit
|
||
# Review user access and permissions
|
||
# Check for unused queues and exchanges
|
||
# Verify network policies
|
||
|
||
# 2. Performance review
|
||
# Analyze message rates and latency
|
||
# Review resource usage trends
|
||
# Optimize configurations
|
||
|
||
# 3. Capacity planning
|
||
# Project growth based on usage trends
|
||
# Plan for scaling if needed
|
||
# Review backup and disaster recovery procedures
|
||
```
|
||
|
||
### **🔧 Maintenance Scripts**
|
||
|
||
#### **Health Check Script**
|
||
```bash
|
||
#!/bin/bash
|
||
# scripts/rabbitmq-health-check.sh
|
||
|
||
NAMESPACE="freeleaps-alpha"
|
||
POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=rabbitmq -o jsonpath='{.items[0].metadata.name}')
|
||
|
||
echo "🐰 RabbitMQ Health Check - $(date)"
|
||
echo "=================================="
|
||
|
||
# Check cluster status
|
||
echo "📊 Cluster Status:"
|
||
kubectl exec -it $POD_NAME -n $NAMESPACE -- rabbitmqctl cluster_status
|
||
|
||
# Check queue depths
|
||
echo "📋 Queue Depths:"
|
||
kubectl exec -it $POD_NAME -n $NAMESPACE -- rabbitmqctl list_queues name messages consumers
|
||
|
||
# Check connections
|
||
echo "🔗 Active Connections:"
|
||
kubectl exec -it $POD_NAME -n $NAMESPACE -- rabbitmqctl list_connections | wc -l
|
||
|
||
# Check resource usage
|
||
echo "💾 Resource Usage:"
|
||
kubectl top pods -n $NAMESPACE | grep rabbitmq
|
||
```
|
||
|
||
#### **Backup Script**
|
||
```bash
|
||
#!/bin/bash
|
||
# scripts/rabbitmq-backup.sh
|
||
|
||
NAMESPACE="freeleaps-alpha"
|
||
BACKUP_DIR="/tmp/rabbitmq-backups"
|
||
DATE=$(date +%Y%m%d_%H%M%S)
|
||
|
||
mkdir -p $BACKUP_DIR
|
||
|
||
echo "📦 Creating RabbitMQ backup..."
|
||
|
||
# Export definitions
|
||
kubectl exec -it deployment/rabbitmq -n $NAMESPACE -- rabbitmqctl export_definitions /tmp/rabbitmq-definitions-$DATE.json
|
||
|
||
# Copy backup file
|
||
kubectl cp $NAMESPACE/deployment/rabbitmq:/tmp/rabbitmq-definitions-$DATE.json $BACKUP_DIR/
|
||
|
||
echo "✅ Backup created: $BACKUP_DIR/rabbitmq-definitions-$DATE.json"
|
||
```
|
||
|
||
### **🚨 Emergency Procedures**
|
||
|
||
#### **1. RabbitMQ Node Failure**
|
||
```bash
|
||
# If a RabbitMQ node fails:
|
||
# 1. Check node status
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl list_nodes
|
||
|
||
# 2. Restart failed node
|
||
kubectl delete pod rabbitmq-1 -n freeleaps-alpha
|
||
|
||
# 3. Verify cluster health
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl cluster_status
|
||
```
|
||
|
||
#### **2. Data Loss Recovery**
|
||
```bash
|
||
# If data is lost:
|
||
# 1. Stop all consumers
|
||
kubectl scale deployment freeleaps-devops-reconciler -n freeleaps-devops-system --replicas=0
|
||
|
||
# 2. Restore from backup
|
||
kubectl cp backup-file.json freeleaps-alpha/deployment/rabbitmq:/tmp/
|
||
kubectl exec -it deployment/rabbitmq -n freeleaps-alpha -- rabbitmqctl import_definitions /tmp/backup-file.json
|
||
|
||
# 3. Restart consumers
|
||
kubectl scale deployment freeleaps-devops-reconciler -n freeleaps-devops-system --replicas=1
|
||
```
|
||
|
||
#### **3. Performance Emergency**
|
||
```bash
|
||
# If performance is severely degraded:
|
||
# 1. Check resource usage
|
||
kubectl top pods -n freeleaps-alpha | grep rabbitmq
|
||
|
||
# 2. Scale up resources
|
||
kubectl patch deployment rabbitmq -n freeleaps-alpha -p '{"spec":{"template":{"spec":{"containers":[{"name":"rabbitmq","resources":{"limits":{"memory":"4Gi","cpu":"2000m"}}}]}}}}'
|
||
|
||
# 3. Restart RabbitMQ
|
||
kubectl rollout restart deployment/rabbitmq -n freeleaps-alpha
|
||
```
|
||
|
||
---
|
||
|
||
## 🎯 **Summary & Next Steps**
|
||
|
||
### **📊 Current State Assessment**
|
||
|
||
#### **✅ Strengths**
|
||
1. **Production-ready setup** - Clustering, monitoring, management UI
|
||
2. **Helm-based deployment** - Consistent and repeatable
|
||
3. **Environment separation** - Alpha vs Production
|
||
4. **Integration working** - Reconciler successfully using RabbitMQ
|
||
5. **Monitoring available** - Grafana dashboards and metrics
|
||
|
||
#### **⚠️ Areas for Improvement**
|
||
1. **Security hardening** - Remove hardcoded passwords, implement secrets
|
||
2. **Configuration standardization** - Centralize configuration management
|
||
3. **Performance optimization** - Tune settings for your workload
|
||
4. **Documentation** - Create runbooks for common operations
|
||
5. **Automation** - Implement automated health checks and alerts
|
||
|
||
### **🚀 Recommended Actions**
|
||
|
||
#### **Immediate (This Week)**
|
||
1. **Implement secret management** - Move passwords to Kubernetes secrets
|
||
2. **Standardize configuration** - Create centralized RabbitMQ config
|
||
3. **Set up monitoring alerts** - Configure alerts for critical metrics
|
||
4. **Document procedures** - Create runbooks for common operations
|
||
|
||
#### **Short Term (Next Month)**
|
||
1. **Security audit** - Review and improve security posture
|
||
2. **Performance tuning** - Optimize settings based on usage patterns
|
||
3. **Automation** - Implement automated health checks and backups
|
||
4. **Training** - Train team on RabbitMQ management and troubleshooting
|
||
|
||
#### **Long Term (Next Quarter)**
|
||
1. **High availability** - Implement multi-zone deployment
|
||
2. **Disaster recovery** - Set up automated backup and recovery procedures
|
||
3. **Advanced monitoring** - Implement predictive analytics and alerting
|
||
4. **Capacity planning** - Plan for growth and scaling
|
||
|
||
### **📚 Additional Resources**
|
||
|
||
#### **Official Documentation**
|
||
- **[RabbitMQ Documentation](https://www.rabbitmq.com/documentation.html)** - Official guides
|
||
- **[RabbitMQ Management UI](https://www.rabbitmq.com/management.html)** - UI documentation
|
||
- **[RabbitMQ Clustering](https://www.rabbitmq.com/clustering.html)** - Cluster setup
|
||
|
||
#### **Community Resources**
|
||
- **[RabbitMQ Slack](https://rabbitmq-slack.herokuapp.com/)** - Community support
|
||
- **[RabbitMQ GitHub](https://github.com/rabbitmq/rabbitmq-server)** - Source code
|
||
- **[RabbitMQ Blog](https://blog.rabbitmq.com/)** - Latest updates and tips
|
||
|
||
#### **Books & Courses**
|
||
- **"RabbitMQ in Depth"** by Gavin M. Roy
|
||
- **"RabbitMQ Essentials"** by Lovisa Johansson
|
||
- **RabbitMQ Tutorials** - Official tutorial series
|
||
|
||
---
|
||
|
||
**🎉 You now have a comprehensive understanding of your RabbitMQ production environment! Use this guide to maintain, monitor, and optimize your message broker infrastructure.**
|
||
|
||
---
|
||
|
||
*Last updated: $(date)*
|
||
*Maintained by: FreeLeaps DevOps Team*
|