76 lines
2.3 KiB
Markdown
76 lines
2.3 KiB
Markdown
|
# Kafka Configuration for Freeleaps Data Platform
|
|||
|
|
|
|||
|
|
This directory contains the Kafka configuration for the Freeleaps Data Platform.
|
|||
|
|
|
|||
|
|
## Files
|
|||
|
|
|
|||
|
|
- `values.yaml` - Original Bitnami Kafka Helm values (legacy)
|
|||
|
|
- `vpa.yaml` - Vertical Pod Autoscaler configuration
|
|||
|
|
- `strimzi-kafka-cluster.yaml` - Strimzi Kafka Operator cluster configuration
|
|||
|
|
- `README.md` - This file
|
|||
|
|
|
|||
|
|
## Migration from Bitnami to Strimzi
|
|||
|
|
|
|||
|
|
### Current State
|
|||
|
|
- **Existing**: Bitnami Kafka deployment (single node)
|
|||
|
|
- **Target**: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes)
|
|||
|
|
|
|||
|
|
### Migration Steps
|
|||
|
|
|
|||
|
|
1. **Install Strimzi Operator** (already done)
|
|||
|
|
```bash
|
|||
|
|
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
|
|||
|
|
--namespace freeleaps-data-platform
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
2. **Delete existing Kafka deployment**
|
|||
|
|
```bash
|
|||
|
|
kubectl delete statefulset kafka-controller -n freeleaps-data-platform
|
|||
|
|
kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform
|
|||
|
|
kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform
|
|||
|
|
kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
3. **Deploy new Strimzi Kafka cluster**
|
|||
|
|
```bash
|
|||
|
|
kubectl apply -f strimzi-kafka-cluster.yaml
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### New Cluster Features
|
|||
|
|
|
|||
|
|
- **High Availability**: 3 Kafka brokers + 3 Zookeeper nodes
|
|||
|
|
- **Authentication**: SCRAM-SHA-512
|
|||
|
|
- **Storage**: 20Gi per broker, 10Gi per ZK node
|
|||
|
|
- **Replication**: 3x replication factor
|
|||
|
|
- **Security**: TLS support for internal communication
|
|||
|
|
|
|||
|
|
### Monitoring
|
|||
|
|
|
|||
|
|
Check cluster status:
|
|||
|
|
```bash
|
|||
|
|
kubectl get kafka -n freeleaps-data-platform
|
|||
|
|
kubectl get kafkatopic -n freeleaps-data-platform
|
|||
|
|
kubectl get kafkauser -n freeleaps-data-platform
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Client Configuration
|
|||
|
|
|
|||
|
|
For applications connecting to the new Kafka cluster, use:
|
|||
|
|
- **Bootstrap servers**: `kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092`
|
|||
|
|
- **Authentication**: SCRAM-SHA-512
|
|||
|
|
- **Username**: `freeleaps-user`
|
|||
|
|
- **Password**: Generated by Strimzi (check the secret)
|
|||
|
|
|
|||
|
|
## Storage Requirements
|
|||
|
|
|
|||
|
|
- **Kafka brokers**: 3 × 20Gi = 60Gi
|
|||
|
|
- **Zookeeper**: 3 × 10Gi = 30Gi
|
|||
|
|
- **Total**: 90Gi
|
|||
|
|
|
|||
|
|
## Network Policy
|
|||
|
|
|
|||
|
|
The cluster includes network policies that allow:
|
|||
|
|
- Internal cluster communication
|
|||
|
|
- External client access (with proper authentication)
|
|||
|
|
- Metrics collection (if enabled)
|