Taming the Wild West of Research Computing: How Policies Saved Us a Thousand Headaches

At IBM Research, one of our focus areas is the convergence of High-Performance Computing (HPC), Artificial Intelligence (AI), hybrid cloud and quantum computing — a field we call , we selected Kyverno as our policy engine for designing and enforcing fine-grained access controls and resource governance, Kueue for implementing job queuing and scheduling, and Argo CD for our GitOps-based configuration management and automation framework:

• 
  offers a robust Policy-as-Code (PaC) framework for Kubernetes and cloud-native environments, enabling the management of the entire policy lifecycle. Policies are defined as Kubernetes resources, utilizing a declarative YAML syntax that aligns with existing Kubernetes configuration files, eliminating the need to learn additional languages, like those required by other tools, such as Open Policy Agent’s Rego. Kyverno supports four policy types — validate, mutate, generate, and cleanup — which collectively enable us to achieve two of our objectives.


• 
  and . This setup also facilitated easy deployment and tracking of quota increases and policy exceptions, making it simple to respond to user requests and deadlines.
  
  
  
  
  
  
  

  CODE Kopieren

  apiVersion: argoproj.io/v1alpha1
  
  kind: ApplicationSet
  
  metadata:
  
    name: user-projects
  
  spec:
  
    goTemplate: true
  
    goTemplateOptions: ["missingkey=error"]
  
    syncPolicy:
  
        automated: {}
  
    generators:
  
    - git:
  
        repoURL: https://gitsource/org/repo.git
  
        revision: HEAD
  
        directories:
  
        - path: user-projects/*
  
    template:
  
      metadata:
  
        name: '{{.path.basename}}'
  
      spec:
  
        project: "default"
  
        source:
  
          repoURL: https://gitsource/org/repo.git
  
          targetRevision: HEAD
  
          path: '{{.path.path}}'
  
        destination:
  
          server: https://kubernetes.default.svc
  
          namespace: '{{.path.basename}}'
  
        syncPolicy:
  
          syncOptions:
  
          - CreateNamespace=true
  
          automated: {}
  
  

  
  
  
  
  
  
  
  

  
  
  
  Preventing GPU Resource Hogging with Kyverno Policies
  

  
  
  

  We then addressed the most significant obstacle to accessing GPUs on our cluster: the proliferation of interactive pods. Initially, our goal was to prevent users from running commands that would indefinitely occupy resources, such as sleep infinity or tail -f /dev/null. However, we realized that removing the ability to exec into these “sleeping” pods and use them interactively would disincentivize this type of behavior.

  
  
  

  To solve this, we created a Kyverno ClusterPolicy that restricts pod exec access to cluster administrators only. By enforcing this policy, we effectively encouraged researchers to adopt a more declarative and Kubernetes-native approach to pod creation, where pods are designed to execute a specific command and then complete, rather than persist indefinitely.
  
  

  
  
  
  
  

  CODE Kopieren

  apiVersion: kyverno.io/v1
  
  kind: ClusterPolicy
  
  metadata:
  
    name: exec-only-from-cluster-admins
  
    namespace: kyverno-admin
  
  spec:
  
    validationFailureAction: Enforce
  
    background: false
  
    rules:
  
      - name: exec-only-from-cluster-admins
  
        context:
  
          - name: exec-namespace-exceptions
  
            configMap:
  
              name: exec-namespace-exceptions
  
              namespace: kyverno-admin
  
        match:
  
          any:
  
            - resources:
  
                kinds:
  
                  - Pod/exec
  
        preconditions:
  
          all:
  
            - key: "{{ request.operation || 'BACKGROUND' }}"
  
              operator: Equals
  
              value: CONNECT
  
            - key: "{{ request.clusterRoles.contains(@, 'cluster-admin') }}"
  
              operator: NotEquals
  
              value: true
  
            - key: "{{ request.namespace }}"
  
              operator: AnyNotIn
  
              value:
  
                '{{ "exec-namespace-exceptions".data."exceptions" | parse_json(@) }}'
  
        validate:
  
          message:
  
            Executing a command in a container is forbidden for Pods running in
  
            this Namespace. To request an exception, reach out to the admins.
  
          deny: {}
  
  

  
  
  
  
  
  
  
  

  
  
  
  Reserving GPU Nodes for GPU Workloads automatically with Affinity Rules
  

  
  
  

  To further optimize resource utilization and prevent over-allocation, we implemented resource quotas to establish a baseline enforcement mechanism that limits pods and resources in use by a single namespace. This ensured that each namespace had a defined ceiling for resource consumption, preventing any one namespace from monopolizing cluster resources.

  
  
  

  However, we soon realized that this alone was not sufficient to address the issue of GPU-enabled nodes being starved of CPU and memory by non-GPU pods. To mitigate this, we created a Kyverno Policy that dynamically adds , providing users with the flexibility to access any available GPU without the need to specify a particular type.
  
  

  
  
  
  
  

  CODE Kopieren

  apiVersion: kueue.x-k8s.io/v1beta1
  
  kind: ResourceFlavor
  
  metadata:
  
    name: "nvidia-a100-80gb-pcie"
  
  spec:
  
    nodeLabels:
  
      nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe
  
  ---
  
  apiVersion: kueue.x-k8s.io/v1beta1
  
  kind: ResourceFlavor
  
  metadata:
  
    name: "nvidia-h100-80gb-pcie"
  
  spec:
  
    nodeLabels:
  
      nvidia.com/gpu.product: NVIDIA-H100-PCIe
  
  ---
  
  apiVersion: kueue.x-k8s.io/v1beta1
  
  kind: ClusterQueue
  
  metadata:
  
    name: "a100-cluster-queue"
  
  spec:
  
    cohort: gpu-cohort
  
    namespaceSelector:
  
      matchLabels:
  
        kueue-enable: gpu-cluster-queue
  
    resourceGroups:
  
      - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
  
        flavors:
  
          - name: "nvidia-a100-80gb-pcie"
  
            resources:
  
              - name: "cpu"
  
                nominalQuota: 394
  
              - name: "memory"
  
                nominalQuota: 800Gi
  
              - name: "nvidia.com/gpu"
  
                nominalQuota: 8
  
  ---
  
  apiVersion: kueue.x-k8s.io/v1beta1
  
  kind: ClusterQueue
  
  metadata:
  
    name: "h100-cluster-queue"
  
  spec:
  
    cohort: gpu-cohort
  
    namespaceSelector:
  
      matchLabels:
  
        kueue-enable: gpu-cluster-queue
  
    resourceGroups:
  
      - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
  
        flavors:
  
          - name: "nvidia-h100-80gb-pcie"
  
            resources:
  
              - name: "cpu"
  
                nominalQuota: 160
  
              - name: "memory"
  
                nominalQuota: 400Gi
  
              - name: "nvidia.com/gpu"
  
                nominalQuota: 4
  
  ---
  
  apiVersion: kueue.x-k8s.io/v1beta1
  
  kind: ClusterQueue
  
  metadata:
  
    name: "gpu-cluster-queue"
  
  spec:
  
    cohort: gpu-cohort
  
    namespaceSelector:
  
      matchLabels:
  
        kueue-enable: gpu-cluster-queue
  
    resourceGroups:
  
      - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
  
        flavors:
  
          - name: "nvidia-a100-80gb-pcie"
  
            resources:
  
              - name: "cpu"
  
                nominalQuota: 0
  
              - name: "memory"
  
                nominalQuota: 0Gi
  
              - name: "nvidia.com/gpu"
  
                nominalQuota: 0
  
          - name: "nvidia-h100-80gb-pcie"
  
            resources:
  
              - name: "cpu"
  
                nominalQuota: 0
  
              - name: "memory"
  
                nominalQuota: 0Gi
  
              - name: "nvidia.com/gpu"
  
                nominalQuota: 0
  
  

  
  
  
  
  
  
  
  

  
  
  
  Conclusion
  

  
  
  

  The adoption of Kyverno, Kueue, and GitOps has been instrumental in transforming our OpenShift® clusters into a more stable, efficient, and fair shared environment for our colleagues. By establishing and implementing clear policies and controls on resource utilization, we have significantly reduced the occurrence of issues related to GPU usage, such as resource contention and over-allocation. This, in turn, has improved the overall user experience, enabling researchers to focus on their work without interruptions or delays. Moreover, the automation and standardization provided by these practices have greatly reduced the administrative burden on our team, freeing us up to concentrate on our own research.

  
  
  

  
  
  
  TL;DR
  

  
  
  

  We enhanced the user experience for researchers using our clusters by leveraging open source technologies, resulting in:

  
  
  
  
  
  • 
    Streamlined cluster management: we leveraged ArgoCD to establish a fully automated GitOps setup, ensuring seamless configuration management across our cluster and projects.
  
  
  • 
    GPU resource optimization: we utilized Kyverno to prevent users from monopolizing GPU resources with interactive pods and dynamically added affinity rules to CPU-only pods, ensuring efficient node utilization.
  
  
  • 
    Fair-sharing and HPC-like experience: we deployed Kueue to provide a fair-sharing, HPC-like experience for our users, promoting efficient resource allocation and minimizing contention.
  
  
  
  
  

  These improvements not only reduced resource contention and over-allocation but also increased efficiency, fairness, and user satisfaction, while decreasing the administrative burden on our team.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen