ArgoCD GitOps持续交付实践

前言

GitOps是一种以Git为唯一真相来源的基础设施和应用交付方法论。ArgoCD是Kubernetes生态中最流行的GitOps工具，被CNCF列为孵化项目。本文将深入讲解GitOps原则、ArgoCD架构和实际使用方法。

GitOps原则

graph TB
    subgraph Principles["GitOps四大原则"]
        P1["声明式<br>Declarative<br>系统状态用声明式描述"]
        P2["版本化<br>Versioned<br>期望状态存储在Git中"]
        P3["自动拉取<br>Pulled Automatically<br>自动将变更应用到系统"]
        P4["持续调谐<br>Continuously Reconciled<br>确保实际状态与期望状态一致"]
    end

Push vs Pull模型

graph LR
    subgraph Push["Push模型 (传统CI/CD)"]
        CI1["CI Pipeline"] --> |"kubectl apply"| Cluster1["K8s Cluster"]
    end

    subgraph Pull["Pull模型 (GitOps)"]
        Dev["Developer"] --> |"git push"| Git["Git Repository"]
        ArgoCD["ArgoCD"] --> |"watch"| Git
        ArgoCD --> |"sync"| Cluster2["K8s Cluster"]
    end

    style Push fill:#FFE0B2
    style Pull fill:#C8E6C9

Pull模型的优势： - CI Pipeline不需要集群访问权限 - Git提供完整的审计日志 - 自动检测和修复配置漂移 - 回滚等于git revert

ArgoCD架构

graph TB
    subgraph ArgoCD["ArgoCD Components"]
        API["API Server<br>(gRPC/REST)"]
        Repo["Repo Server<br>(Git Clone & Render)"]
        Controller["Application Controller<br>(Reconcile Loop)"]
        Redis["Redis<br>(Cache)"]
        Dex["Dex<br>(SSO)"]
        Notif["Notifications Controller"]
    end

    subgraph External["外部系统"]
        Git["Git Repository"]
        K8s["Kubernetes API"]
        SSO["OIDC/LDAP"]
        Slack["Slack/Teams"]
    end

    UI["Web UI"] --> API
    CLI["argocd CLI"] --> API
    API --> Redis
    API --> Repo
    API --> Dex
    Controller --> Repo
    Controller --> K8s
    Repo --> Git
    Dex --> SSO
    Notif --> Slack

核心组件

组件	职责
API Server	提供Web UI、CLI和CI/CD集成的API
Repo Server	从Git仓库拉取配置，渲染Helm/Kustomize
Application Controller	监控应用状态，执行同步操作
Redis	缓存Git仓库数据和应用状态
Dex	SSO认证代理
Notifications	发送同步状态通知

Application CRD

Application是ArgoCD最核心的CRD，定义了一个应用的部署配置：

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: production

  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/myapp/overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true           # 删除Git中已移除的资源
      selfHeal: true         # 自动修复配置漂移
      allowEmpty: false      # 禁止同步空资源
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas   # 忽略HPA管理的副本数

  info:
    - name: url
      value: https://myapp.example.com

Helm Chart应用

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-helm
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://charts.example.com
    chart: myapp
    targetRevision: 1.5.0
    helm:
      releaseName: myapp
      valueFiles:
        - values-production.yaml
      values: |
        replicaCount: 3
        image:
          tag: v2.1.0
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
      parameters:
        - name: ingress.enabled
          value: "true"
        - name: ingress.host
          value: myapp.example.com

  destination:
    server: https://kubernetes.default.svc
    namespace: production

Kustomize应用

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-kustomize
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/myapp/overlays/production
    kustomize:
      images:
        - myregistry.io/myapp:v2.1.0
      namePrefix: prod-
      commonLabels:
        env: production
      commonAnnotations:
        team: backend

  destination:
    server: https://kubernetes.default.svc
    namespace: production

同步策略

同步流程

sequenceDiagram
    participant Git as Git Repository
    participant Repo as Repo Server
    participant Ctrl as App Controller
    participant K8s as Kubernetes

    loop 每3分钟
        Ctrl->>Repo: 获取最新配置
        Repo->>Git: git pull
        Git-->>Repo: 最新manifests
        Repo-->>Ctrl: 渲染后的资源
        Ctrl->>K8s: 获取实际状态
        K8s-->>Ctrl: 当前资源
        Ctrl->>Ctrl: 对比期望 vs 实际
        alt 有差异
            Ctrl->>K8s: 自动同步 (如已开启)
        else 无差异
            Ctrl->>Ctrl: 状态: Synced
        end
    end

同步波次（Sync Waves）

通过sync-wave注解控制资源的部署顺序：

# Wave -1: 先创建命名空间
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

---
# Wave 0: 创建ConfigMap和Secret
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "0"
data:
  config.yaml: |
    server:
      port: 8080

---
# Wave 1: 部署应用
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: app
          image: myapp:v2.1.0

---
# Wave 2: 创建Service
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  selector:
    app: myapp
  ports:
    - port: 80
      targetPort: 8080

---
# Wave 3: 创建Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  ingressClassName: nginx
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 80

graph LR
    W_1["Wave -1<br>Namespace"] --> W0["Wave 0<br>ConfigMap/Secret"]
    W0 --> W1["Wave 1<br>Deployment"]
    W1 --> W2["Wave 2<br>Service"]
    W2 --> W3["Wave 3<br>Ingress"]

健康检查与Hook

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync       # 同步前执行
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp:v2.1.0
          command: ["./migrate", "up"]
      restartPolicy: Never

---
apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync      # 同步后执行
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: test
          image: curlimages/curl:latest
          command: ["curl", "-f", "http://myapp.production/healthz"]
      restartPolicy: Never

回滚

# 查看应用历史
argocd app history myapp-production

# 回滚到指定版本
argocd app rollback myapp-production 3

# 通过Git回滚（推荐方式）
git revert HEAD
git push origin main
# ArgoCD会自动检测并同步

graph TB
    subgraph Rollback["回滚方式"]
        Git["Git Revert<br>(推荐)"]
        ArgoCD["ArgoCD Rollback"]
    end

    Git --> |"git revert + push"| Repo["Git Repository"]
    Repo --> |"自动同步"| Cluster["Kubernetes"]

    ArgoCD --> |"直接回滚"| Cluster
    ArgoCD -.-> |"会产生漂移"| Warning["注意: 下次同步会覆盖"]

多集群管理

# 注册外部集群
# argocd cluster add <context-name>

# 多集群Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-us-east
  namespace: argocd
spec:
  project: multi-cluster
  source:
    repoURL: https://github.com/myorg/k8s-manifests.git
    targetRevision: main
    path: apps/myapp/overlays/us-east
  destination:
    server: https://us-east.k8s.example.com  # 远程集群
    namespace: production

AppProject（项目隔离）

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  description: "Production environment project"

  # 允许的Git源
  sourceRepos:
    - 'https://github.com/myorg/*'
    - 'https://charts.example.com'

  # 允许的目标集群和命名空间
  destinations:
    - server: https://kubernetes.default.svc
      namespace: production
    - server: https://us-east.k8s.example.com
      namespace: production

  # 允许的资源类型
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'networking.k8s.io'
      kind: Ingress

  # 角色和权限
  roles:
    - name: developer
      description: "Developer access"
      policies:
        - p, proj:production:developer, applications, get, production/*, allow
        - p, proj:production:developer, applications, sync, production/*, allow
      groups:
        - dev-team
    - name: admin
      description: "Admin access"
      policies:
        - p, proj:production:admin, applications, *, production/*, allow
      groups:
        - platform-team

ApplicationSet

ApplicationSet可以从模板自动生成多个Application，适用于多集群、多租户场景：

Git Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/k8s-manifests.git
        revision: main
        directories:
          - path: apps/*
            exclude: false
          - path: apps/deprecated-*
            exclude: true
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: '{{path}}/overlays/production'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Cluster Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring-stack
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: production
  template:
    metadata:
      name: 'monitoring-{{name}}'
    spec:
      project: infrastructure
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: infrastructure/monitoring
        helm:
          values: |
            cluster: {{name}}
            region: {{metadata.labels.region}}
      destination:
        server: '{{server}}'
        namespace: monitoring

Matrix Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: multi-cluster-apps
  namespace: argocd
spec:
  generators:
    # 矩阵组合: clusters x apps
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: production
          - git:
              repoURL: https://github.com/myorg/k8s-manifests.git
              revision: main
              directories:
                - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/k8s-manifests.git
        targetRevision: main
        path: '{{path}}'
      destination:
        server: '{{server}}'
        namespace: '{{path.basename}}'

graph TB
    subgraph AppSet["ApplicationSet"]
        Generator["Generator<br>(生成参数)"]
        Template["Template<br>(应用模板)"]
    end

    Generator --> |"cluster-1 + app-a"| App1["Application<br>app-a-cluster-1"]
    Generator --> |"cluster-1 + app-b"| App2["Application<br>app-b-cluster-1"]
    Generator --> |"cluster-2 + app-a"| App3["Application<br>app-a-cluster-2"]
    Generator --> |"cluster-2 + app-b"| App4["Application<br>app-b-cluster-2"]

Secrets管理

ArgoCD本身不处理密钥，需要借助外部工具：

graph LR
    subgraph Solutions["密钥管理方案"]
        SOPS["Mozilla SOPS<br>+ age/KMS"]
        ESO["External Secrets<br>Operator"]
        Sealed["Sealed Secrets"]
        Vault["Vault Agent<br>Sidecar"]
    end

    SOPS --> |"加密存储在Git"| Git["Git Repository"]
    ESO --> |"运行时获取"| AWS["AWS SM / Vault"]
    Sealed --> |"加密存储在Git"| Git
    Vault --> |"运行时注入"| Pod["Application Pod"]

# 使用External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
    - secretKey: db-password
      remoteRef:
        key: production/myapp
        property: db_password
    - secretKey: api-key
      remoteRef:
        key: production/myapp
        property: api_key

通知配置

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token
  template.app-sync-succeeded: |
    slack:
      attachments: |
        [{
          "color": "#18be52",
          "title": "{{.app.metadata.name}} synced successfully",
          "fields": [
            {"title": "Repository", "value": "{{.app.spec.source.repoURL}}", "short": true},
            {"title": "Revision", "value": "{{.app.status.sync.revision}}", "short": true}
          ]
        }]
  trigger.on-sync-succeeded: |
    - when: app.status.sync.status == 'Synced'
      send: [app-sync-succeeded]
  subscriptions: |
    - recipients: [slack:deployments]
      triggers: [on-sync-succeeded, on-sync-failed, on-health-degraded]

总结

ArgoCD GitOps实践的核心要点：

Git作为唯一真相来源：所有集群状态变更通过Git提交
自动同步+自愈：开启automated sync和selfHeal
渐进式发布：利用sync-wave控制部署顺序
多集群管理：ApplicationSet简化大规模集群管理
项目隔离：AppProject控制权限和资源范围
安全密钥管理：使用External Secrets Operator或Sealed Secrets
可观测性：配置通知和监控，及时发现同步异常