icecheng/freeleaps-ops
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
master
freeleaps-ops/docs/Custom_Resources_And_Operators_Guide.md

574 lines
28 KiB
Markdown
Raw Permalink Normal View History

(feat): Basic docs on Freeleaps Infra
2025-09-03 23:59:04 +00:00
# Custom Resources & Operators Guide
## 🎯 **Overview**
This guide explains **Custom Resources (CRs)**, **Custom Resource Definitions (CRDs)**, **Kubernetes Operators**, and how your `freeleaps-devops-reconciler` works as an operator to manage your DevOps infrastructure.
---
## 📊 **What Are Custom Resources?**
### **🔄 CR vs CRD Relationship**
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ CRD vs CR RELATIONSHIP │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ CUSTOM RESOURCE DEFINITION (CRD) │ │ │
│ │ │ │ │
│ │ apiVersion: apiextensions.k8s.io/v1 │ │ │
│ │ kind: CustomResourceDefinition │ │ │
│ │ metadata: │ │ │
│ │ name: devopsprojects.freeleaps.com │ │ │
│ │ spec: │ │ │
│ │ group: freeleaps.com │ │ │
│ │ names: │ │ │
│ │ kind: DevOpsProject │ │ │
│ │ plural: devopsprojects │ │ │
│ │ scope: Namespaced │ │ │
│ │ versions: │ │ │
│ │ - name: v1alpha1 │ │ │
│ │ schema: │ │ │
│ │ # Schema definition... │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ CUSTOM RESOURCE (CR) │ │ │
│ │ │ │ │
│ │ apiVersion: freeleaps.com/v1alpha1 │ │ │
│ │ kind: DevOpsProject │ │ │
│ │ metadata: │ │ │
│ │ name: my-project │ │ │
│ │ namespace: freeleaps-devops-system │ │ │
│ │ spec: │ │ │
│ │ projectName: "My Awesome Project" │ │ │
│ │ projectId: "my-awesome-project" │ │ │
│ │ git: │ │ │
│ │ url: "https://github.com/myorg/myproject" │ │ │
│ │ branch: "main" │ │ │
│ │ registry: │ │ │
│ │ url: "https://harbor.example.com" │ │ │
│ │ project: "myproject" │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### **🎯 Why Custom Resources?**
```yaml
# Instead of managing multiple resources manually:
# - Namespace
# - ServiceAccount
# - Role/RoleBinding
# - ConfigMap
# - Secret
# - Deployment
# - Service
# - Ingress
# You can create ONE custom resource:
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: my-project
spec:
projectName: "My Project"
projectId: "my-project"
git:
url: "https://github.com/myorg/myproject"
branch: "main"
registry:
url: "https://harbor.example.com"
project: "myproject"
```
---
## 🏭 **Your DevOps Reconciler Architecture**
### **📊 Reconciler vs DevOps Repo Relationship**
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ RECONCILER + DEVOPS ARCHITECTURE │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ FRELEAPS.COM PLATFORM │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ User │ │ Project │ │ Git │ │ Registry │ │ │
│ │ │ Creates │ │ Manager │ │ Webhook │ │ Manager │ │ │
│ │ │ Project │ │ Creates │ │ Triggers │ │ Creates │ │ │
│ │ │ │ │ DevOps │ │ Event │ │ Repo │ │ │
│ │ │ │ │ Project │ │ │ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ RABBITMQ MESSAGE QUEUE │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │DevOpsInit │ │DevOpsReconcile│ │DevOpsDeploy │ │DevOpsDelete │ │ │
│ │ │Event │ │Event │ │Event │ │Event │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ FRELEAPS-DEVOPS-RECONCILER (OPERATOR) │ │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
│ │ │ CONTROLLERS │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
│ │ │ │Controller │ │Controller │ │Settings │ │ │ │ │ │ │
│ │ │ │ │ │ │ │Controller │ │ │ │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │
│ │ ▼ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
│ │ │ CUSTOM RESOURCES │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
│ │ │ │CR │ │CR │ │Settings │ │ │ │ │ │ │
│ │ │ │ │ │ │ │CR │ │ │ │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ KUBERNETES RESOURCES │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ArgoCD │ │Jenkins │ │Harbor │ │Namespaces │ │ │
│ │ │Applications │ │Pipelines │ │Repositories │ │Services │ │ │
│ │ │Projects │ │Jobs │ │Credentials │ │Deployments │ │ │
│ │ │ │ │ │ │ │ │Ingresses │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 🔧 **Your Custom Resources**
### **1. DevOpsProject CRD**
```yaml
# 🏭 ACTUAL CRD FROM YOUR CODEBASE
# freeleaps-devops-reconciler/deploy/crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: devopsprojects.freeleaps.com
spec:
group: freeleaps.com
scope: Namespaced
names:
kind: DevOpsProject
plural: devopsprojects
shortNames: [dop, dops]
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
required: ['spec']
properties:
spec:
type: object
required:
- projectName
- projectId
- git
- registry
- environments
properties:
projectName:
type: string
description: "Human readable project name"
projectId:
type: string
description: "Unique project identifier"
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
git:
type: object
required: [url, branch]
properties:
url:
type: string
description: "Git repository URL"
branch:
type: string
description: "Default git branch"
default: "main"
registry:
type: object
required: [url, project]
properties:
url:
type: string
description: "Container registry URL"
project:
type: string
description: "Registry project name"
```
### **2. DevOpsProject CR Example**
```yaml
# 🏭 ACTUAL CR EXAMPLE
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: magicleaps-frontend
namespace: freeleaps-devops-system
labels:
app.kubernetes.io/name: magicleaps-frontend
app.kubernetes.io/instance: magicleaps-frontend
spec:
projectName: "Magicleaps Frontend"
projectId: "magicleaps-frontend"
git:
url: "https://github.com/freeleaps/magicleaps-frontend"
branch: "main"
credentialsRef:
name: "github-credentials"
namespace: "freeleaps-devops-system"
registry:
url: "https://harbor.freeleaps.mathmast.com"
project: "magicleaps"
credentialsRef:
name: "harbor-credentials"
namespace: "freeleaps-devops-system"
environments:
- name: "production"
domain: "magicleaps.mathmast.com"
replicas: 3
- name: "alpha"
domain: "alpha.magicleaps.mathmast.com"
replicas: 1
```
### **3. Other Custom Resources**
```yaml
# 🏭 YOUR COMPLETE CRD SET
# From freeleaps-devops-reconciler/docs/design/one-click-deployment.md
# DevOpsProject - Main project configuration
# ArgoSettings - ArgoCD settings for the DevOpsProject
# JenkinsSettings - Jenkins settings and generated pipelines
# ContainerRegistry - Container registry information
# ContainerImage - Every image manufactured by Jenkins pipeline
# DeploymentRecord - Track deployment records
# GitCredential - Git repository credentials
# IngressResource - Ingress configuration
```
---
## 🤖 **How Your Operator Works**
### **🔄 Reconciliation Loop**
```python
# 🏭 ACTUAL CODE FROM YOUR RECONCILER
# freeleaps-devops-reconciler/reconciler/controllers/devops_projects/controller.py
@kopf.on.create(group=consts.GROUP, version=consts.VERSION, kind=consts.DEVOPS_PROJECT_KIND)
def on_devops_proj_created(name: str, namespace: Optional[str], body: Body, logger: Logger, **kwargs):
logger.info(f"Newly created DevOpsProject resource and named {name} in namespace {namespace}, start to reconciling...")
devops_proj = DevOpsProject(body)
try:
devops_proj.parse_spec()
devops_proj.get_spec().validate(logger)
except SpecError as e:
devops_proj.update_status({
'devopsProject': {
'status': DevOpsProjectDiagStatus.INVALID.value,
'synced': False,
'ready': False,
'lastProbeTime': isotime(),
}
})
devops_proj.error(action='CreateDevOpsProject',
reason='InvalidSpecArgument', msg=str(e))
raise kopf.TemporaryError(f"Error found in DevOpsProject spec: {e}")
# Create resource manager and handle the project
resource_manager = DevOpsProjectResourceManager(namespace, logger)
# ... implementation details
```
### **📊 Event Flow**
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ EVENT FLOW │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ User │ │ RabbitMQ │ │ Operator │ │ Kubernetes │ │
│ │ Action │ │ Message │ │ Controller│ │ Resources │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │ │
│ │ 1. Create │ │ │ │
│ │ Project │ │ │ │
│ │───────────────▶│ │ │ │
│ │ │ 2. DevOpsInit │ │ │
│ │ │ Event │ │ │
│ │ │───────────────▶│ │ │
│ │ │ │ 3. Create CR │ │
│ │ │ │───────────────▶│ │
│ │ │ │ │ 4. CR Created │
│ │ │ │ │◀───────────────│
│ │ │ │ 5. Reconcile │ │
│ │ │ │◀───────────────│ │
│ │ │ │ 6. Create │ │
│ │ │ │ Resources │ │
│ │ │ │───────────────▶│ │
│ │ │ │ │ 7. Resources │
│ │ │ │ │ Ready │
│ │ │ │ │◀───────────────│
│ │ │ │ 8. Update │ │
│ │ │ │ Status │ │
│ │ │ │◀───────────────│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 🎯 **Understanding the Relationship**
### **📊 Reconciler vs DevOps Repo**
| Component | Purpose | Location | Responsibility |
|-----------|---------|----------|----------------|
| **freeleaps-devops-reconciler** | **Kubernetes Operator** | `freeleaps-ops/freeleaps-devops-reconciler/` | Watches CRs, creates K8s resources |
| **freeleaps.com Platform** | **Business Logic** | `freeleaps-service-hub/` | User interface, project management |
| **RabbitMQ** | **Message Queue** | Infrastructure | Event communication |
| **ArgoCD** | **GitOps** | `freeleaps-ops/cluster/manifests/` | Application deployment |
| **Jenkins** | **CI/CD** | Infrastructure | Pipeline execution |
### **🔄 How They Work Together**
```yaml
# 1. User creates project on freeleaps.com
# 2. Platform sends DevOpsInit event to RabbitMQ
# 3. Reconciler receives event and creates DevOpsProject CR
# 4. Reconciler watches CR and creates:
# - ArgoCD Application
# - Jenkins Pipeline
# - Harbor Repository
# - Namespace and RBAC
# 5. ArgoCD deploys the application
# 6. Jenkins runs the pipeline
```
---
## 🔧 **Practical Examples**
### **1. Creating a DevOpsProject**
```bash
# Create a DevOpsProject CR
kubectl apply -f - <<EOF
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: my-new-project
namespace: freeleaps-devops-system
spec:
projectName: "My New Project"
projectId: "my-new-project"
git:
url: "https://github.com/myorg/myproject"
branch: "main"
registry:
url: "https://harbor.example.com"
project: "myproject"
environments:
- name: "production"
domain: "myproject.example.com"
replicas: 2
EOF
# Check the CR status
kubectl get devopsprojects -n freeleaps-devops-system
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
```
### **2. Monitoring the Operator**
```bash
# Check operator logs
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler
# Check CR status
kubectl get devopsprojects --all-namespaces
kubectl get argosettings --all-namespaces
kubectl get jenkinssettings --all-namespaces
# Check created resources
kubectl get applications -n freeleaps-devops-system
kubectl get namespaces | grep my-new-project
```
### **3. Troubleshooting**
```bash
# Check CRD installation
kubectl get crd | grep freeleaps.com
# Check operator events
kubectl get events -n freeleaps-devops-system --sort-by='.lastTimestamp'
# Check resource creation
kubectl get all -n my-new-project
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
```
---
## 🎯 **Best Practices**
### **1. CRD Design**
```yaml
# ✅ DO: Use clear, descriptive names
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject # Clear, descriptive
# ❌ DON'T: Use generic names
kind: Project # Too generic
```
### **2. Validation**
```yaml
# ✅ DO: Include validation in CRD
spec:
openAPIV3Schema:
type: object
required: ['spec']
properties:
spec:
type: object
required: ['projectName', 'projectId']
properties:
projectId:
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
```
### **3. Status Management**
```yaml
# ✅ DO: Include status in CR
status:
conditions:
- type: Ready
status: "True"
reason: "ReconciliationSucceeded"
message: "All resources created successfully"
- type: Synced
status: "True"
reason: "ReconciliationSucceeded"
message: "Spec has been reconciled"
```
### **4. Error Handling**
```python
# ✅ DO: Proper error handling
try:
# Create resources
resource_manager.create_resources()
except Exception as e:
# Update status with error
devops_proj.update_status({
'status': 'Error',
'message': str(e)
})
raise kopf.TemporaryError(f"Failed to create resources: {e}")
```
---
## 🔍 **Debugging Your Operator**
### **1. Check Operator Status**
```bash
# Check if operator is running
kubectl get pods -n freeleaps-devops-system -l app=freeleaps-devops-reconciler
# Check operator logs
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler -f
# Check CRD installation
kubectl get crd devopsprojects.freeleaps.com
```
### **2. Check CR Status**
```bash
# Check CR status
kubectl get devopsprojects --all-namespaces -o wide
# Check CR events
kubectl describe devopsproject <project-name> -n <namespace>
# Check CR YAML
kubectl get devopsproject <project-name> -n <namespace> -o yaml
```
### **3. Check Created Resources**
```bash
# Check what resources were created
kubectl get all -n <project-namespace>
# Check ArgoCD applications
kubectl get applications -n freeleaps-devops-system
# Check Jenkins pipelines
kubectl get jenkinssettings --all-namespaces
```
---
## 📚 **Next Steps**
### **1. Learn More About Operators**
- [Kubernetes Operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
- [Kopf Framework](https://kopf.readthedocs.io/) (what your reconciler uses)
- [Operator SDK](https://sdk.operatorframework.io/)
### **2. Understand Your Architecture**
- Study your `freeleaps-devops-reconciler` code
- Understand the event flow from RabbitMQ
- Learn how CRs trigger resource creation
### **3. Extend Your Operator**
- Add new custom resources
- Implement new controllers
- Add validation and error handling
---
**Last Updated**: September 3, 2025
**Version**: 1.0
**Maintainer**: Infrastructure Team
Reference in New Issue Copy Permalink