Modernizing Legacy Workloads: KubeVirt on AKS with Azure Arc Identity

graph TB

    subgraph K8sNode["🖥️ Kubernetes Node (Azure VM with Nested Virt)"]

        subgraph VirtLauncher["virt-launcher Pod"]

            QEMU["QEMU/KVM Hypervisor"]

            Libvirt["libvirtd"]

            Guest["🖥️ Guest VM<br/>(Ubuntu, Windows, etc.)"]

        end



        subgraph System["System Components"]

            Kubelet["kubelet"]

            VirtHandler["virt-handler<br/>(DaemonSet)"]

            CNI["CNI Plugin<br/>(Network)"]

        end

    end



    subgraph ControlPlane["☸️ Control Plane"]

        API["API Server"]

        VirtController["virt-controller"]

        VirtAPI["virt-api"]

    end



    API --> VirtController

    VirtController --> VirtHandler

    VirtHandler --> VirtLauncher

    Kubelet --> VirtLauncher

    Libvirt --> QEMU

    QEMU --> Guest

    CNI -.-> VirtLauncher



    style Guest fill:#e3f2fd,stroke:#1976d2

    style VirtLauncher fill:#e8f5e9,stroke:#388e3c

apiVersion: kubevirt.io/v1

kind: VirtualMachine

metadata:

  name: my-ubuntu-vm

  namespace: student-labs

spec:

  running: true # Desired state: running

  template:

    spec:

      domain:

        cpu:

          cores: 2

        memory:

          guest: 4Gi

        devices:

          disks:

            - name: rootdisk

              disk:

                bus: virtio

      volumes:

        - name: rootdisk

          dataVolume:

            name: my-ubuntu-vm-rootdisk

flowchart TB

    subgraph Azure["☁️ Azure Cloud"]

        subgraph RG["Resource Group: rg-uni-kubevirt"]

            subgraph AKS["🚀 AKS Cluster: aks-uni-platform"]

                subgraph SystemPool["System Node Pool<br/>(Standard_D2s_v3)"]

                    CoreDNS["CoreDNS"]

                    VirtOp["virt-operator"]

                    CDIOp["cdi-operator"]

                    VirtAPI["virt-api"]

                end



                subgraph KVPool["KubeVirt Node Pool<br/>(Standard_D4s_v3 - Nested Virt)"]

                    direction TB

                    subgraph NS1["Namespace: dept-cs"]

                        FacultyVM1["🎓 research-vm-1"]

                        FacultyVM2["🎓 research-vm-2"]

                    end

                    subgraph NS2["Namespace: student-labs"]

                        StudentVM1["📚 lab-vm-alice"]

                        StudentVM2["📚 lab-vm-bob"]

                    end

                end

            end



            Storage["💾 Azure Managed Disks<br/>(Premium SSD, Retain Policy)"]

        end



        subgraph ImageRG["Resource Group: rg-uni-images"]

            Blob["📦 Azure Blob Storage<br/>(VM Images)"]

        end



        subgraph Identity["🔐 Identity Services"]

            Arc["Azure Arc"]

            AAD["Azure AD / Entra ID"]

        end

    end



    KVPool --> Storage

    KVPool -.->|"Arc Agent"| Arc

    Arc <--> AAD

    CDIOp -->|"Pull Images"| Blob



    style AKS fill:#e3f2fd,stroke:#1976d2

    style Identity fill:#fff3e0,stroke:#f57c00

graph TB

    subgraph AKS_Node ["🖥️ AKS Node VM - Standard_D4s_v3"]

        direction TB

        host_imds["✅ Host can reach IMDS"]

        subgraph virt_launcher ["📦 virt-launcher Pod"]

            direction TB

            subgraph guest_vm ["🖥️ KubeVirt Guest VM - Ubuntu"]

                curl["curl 169.254.169.254"]

                blocked["❌ UNREACHABLE - NAT'd network"]

            end

        end

    end



    imds["🔐 IMDS 169.254.169.254"]



    curl --> blocked

    host_imds <--> imds

    blocked -.->|"Blocked by NAT"| imds



    style AKS_Node fill:#e3f2fd,stroke:#0078d4,stroke-width:2px

    style virt_launcher fill:#fff3e0,stroke:#ff9800,stroke-width:1px

    style guest_vm fill:#ffebee,stroke:#f44336,stroke-width:1px

    style host_imds fill:#e8f5e9,stroke:#4caf50,stroke-width:1px

    style blocked fill:#ffcdd2,stroke:#f44336,stroke-width:1px

    style imds fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px

sequenceDiagram

    autonumber

    participant TF as Terraform

    participant VM as KubeVirt VM

    participant Arc as Azure Arc

    participant ARM as Azure Resource Manager

    participant AAD as Azure AD



    Note over TF: Deployment Phase

    TF->>ARM: Create Service Principal (sp-arc-kubevirt)

    TF->>ARM: Assign "Azure Connected Machine Onboarding" role

    TF->>VM: Inject cloud-init with SP credentials



    Note over VM: VM Boot Phase

    VM->>VM: Wait for network (KubeVirt NAT stabilization)

    VM->>VM: Download azcmagent

    VM->>Arc: azcmagent connect --service-principal-id ... --service-principal-secret ...

    Arc->>ARM: Create Azure Arc Machine resource

    ARM-->>Arc: Resource ID assigned

    Arc-->>VM: Identity certificate installed



    Note over TF: Post-Registration Phase

    TF->>Arc: Poll for "Connected" status

    TF->>Arc: Install AADSSHLoginForLinux extension

    Arc->>VM: Push extension to agent

    VM->>VM: Configure SSH for AAD auth



    Note over AAD: User Connection Phase

    AAD->>AAD: User runs: az ssh vm --name lab-vm

    AAD-->>AAD: Generate ephemeral SSH certificate

    AAD->>VM: SSH connection with certificate

    VM->>Arc: Validate certificate

    Arc-->>VM: Access granted (RBAC checked)

    VM-->>AAD: Shell session established ✅

flowchart TB

    subgraph Cluster["AKS Cluster"]

        subgraph NS1["Namespace: dept-cs"]

            Quota1["ResourceQuota:<br/>CPU: 16 cores<br/>Memory: 32Gi<br/>PVCs: 10"]

            NP1["NetworkPolicy:<br/>Ingress: Same namespace only<br/>Egress: Internet + DNS"]

            RB1["RoleBinding:<br/>cs-faculty → admin"]

        end



        subgraph NS2["Namespace: student-labs"]

            Quota2["ResourceQuota:<br/>CPU: 8 cores<br/>Memory: 16Gi<br/>PVCs: 5"]

            LR2["LimitRange:<br/>Max CPU/VM: 2<br/>Max Memory/VM: 4Gi"]

            NP2["NetworkPolicy:<br/>Ingress: Same namespace only<br/>Egress: Internet + DNS"]

            RB2["RoleBinding:<br/>students → view"]

        end

    end



    style NS1 fill:#e3f2fd,stroke:#1976d2

    style NS2 fill:#fff3e0,stroke:#f57c00

# ResourceQuota: Limits total resources in namespace

apiVersion: v1

kind: ResourceQuota

metadata:

  name: student-lab-quota

  namespace: student-labs

spec:

  hard:

    requests.cpu: "8"

    requests.memory: 16Gi

    limits.cpu: "16"

    limits.memory: 32Gi

    persistentvolumeclaims: "5"



---

# LimitRange: Limits per-VM resources

apiVersion: v1

kind: LimitRange

metadata:

  name: student-vm-limits

  namespace: student-labs

spec:

  limits:

    - type: Container

      max:

        cpu: "2"

        memory: 4Gi

      default:

        cpu: "1"

        memory: 2Gi

terraform/

├── main.tf              # AKS cluster and node pools

├── providers.tf         # Azure, Kubernetes, kubectl providers

├── variables.tf         # Input variables with validation

├── outputs.tf           # Useful outputs (connection strings, etc.)

├── identity.tf          # Azure AD groups, RBAC assignments

├── arc.tf               # Azure Arc SP, roles, extension installer

├── platform.tf          # KubeVirt and CDI operator deployment

├── tenancy.tf           # Namespace, quota, network policy per tenant

├── storage.tf           # StorageClass definitions

├── networking.tf        # Egress network policies for operators

├── images.tf            # VM image storage (Azure Blob)

├── virtualmachines.tf   # Demo VM definition

└── templates/

    ├── cloud-init-arc.tftpl   # Cloud-init for Arc-enabled VMs

    └── cloud-init-lab.tftpl   # Cloud-init for basic VMs

# 1. Network readiness check (with timeout)

wait_for_network() {

    for i in $(seq 1 60); do

        if curl -s --connect-timeout 5 https://management.azure.com > /dev/null 2>&1; then

            echo "[Arc] Network ready"

            return 0

        fi

        echo "[Arc] Waiting for network... ($i/60)"

        sleep 5

    done

    return 1

}



# 2. Agent installation

install_arc_agent() {

    curl -sSL https://aka.ms/azcmagent-download | bash

}



# 3. Registration with retry logic

register_with_arc() {

    local max_retries=5

    local retry_delay=30



    for i in $(seq 1 $max_retries); do

        if azcmagent connect \

            --service-principal-id "$SP_ID" \

            --service-principal-secret "$SP_SECRET" \

            --tenant-id "$TENANT_ID" \

            --subscription-id "$SUB_ID" \

            --resource-group "$RG_NAME" \

            --location "$LOCATION" \

            --resource-name "$(hostname)"; then

            echo "[Arc] Registration successful"

            return 0

        fi

        echo "[Arc] Registration failed, retrying in ${retry_delay}s... ($i/$max_retries)"

        sleep $retry_delay

        retry_delay=$((retry_delay * 2))  # Exponential backoff

    done

    return 1

}

# 1. Clone the repository

git clone https://github.com/ykbytes/aks-kubevirt-arc-unilab.git

cd aks-kubevirt-arc-unilab



# 2. Login to Azure

az login

az account set --subscription "Your Subscription Name"



# 3. Configure variables

cp terraform.tfvars.example terraform.tfvars

# Edit terraform.tfvars with your settings



# 4. Initialize and Deploy (~15-20 minutes)

terraform init

terraform plan

terraform apply



# 5. Get AKS credentials

az aks get-credentials --resource-group rg-uni-kubevirt --name aks-uni-platform



# 6. Verify KubeVirt

kubectl get kubevirt -n kubevirt

# Expected: phase = Deployed



# 7. Verify the VM

kubectl get vm -n student-labs

# Expected: lab-vm in Running state



# 8. Connect via Azure AD SSH

az ssh vm --name lab-vm --resource-group rg-uni-kubevirt

null_resource.arc_aad_ssh_extension[0] (local-exec): [Arc] Waiting for lab-vm to connect...

null_resource.arc_aad_ssh_extension[0] (local-exec): [Arc] Status:  (attempt 1/90)

null_resource.arc_aad_ssh_extension[0]: Still creating... [00m10s elapsed]

...

null_resource.arc_aad_ssh_extension[0] (local-exec): [Arc] Status:  (attempt 27/90)

null_resource.arc_aad_ssh_extension[0]: Still creating... [05m10s elapsed]

null_resource.arc_aad_ssh_extension[0] (local-exec): [Arc] Machine connected

null_resource.arc_aad_ssh_extension[0] (local-exec): [Arc] Starting extension installation (async)...

# Check Arc registration

az connectedmachine show --name lab-vm --resource-group rg-uni-kubevirt \

    --query "{Name:name, Status:status, AgentVersion:agentVersion}" -o table



# Check extension status

az connectedmachine extension list --machine-name lab-vm --resource-group rg-uni-kubevirt \

    --query "[].{Name:name, Status:provisioningState}" -o table



# View VM console (alternative access method)

kubectl virt console lab-vm -n student-labs

PS> az ssh vm --name lab-vm --resource-group rg-uni-kubevirt

Port 22 is not allowed for SSH connections in this resource. 

Would you like to update the current Service Configuration? (y/n): y

Finished[###############################################]  100.0000%



Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-1088-kvm x86_64)



═══════════════════════════════════════════════════════════════════════════════

 🎓 KubeVirt Lab VM - Azure Arc Enabled

═══════════════════════════════════════════════════════════════════════════════



 🔐 Azure AD Authentication:

    This VM is registered with Azure Arc. Use Azure AD to authenticate:

    az ssh vm --name lab-vm --resource-group rg-uni-kubevirt



 📋 Required RBAC Roles:

    • Virtual Machine Administrator Login - for sudo access

    • Virtual Machine User Login - for standard user access



═══════════════════════════════════════════════════════════════════════════════



[email protected]@lab-vm:~$ whoami

[email protected]