Optimize Resilience and Reduce Cross-Zone Expenses Using HAZL

Unplanned downtime, whether it’s caused by hardware failures, glitches, or cyberattacks, is every organization’s worst nightmare, no matter its size and sectors. Its can not only cause a lost revenue, but also drops in stock value, hit to customer satisfaction, trust and damage to the company’s reputation. According to a Oxford Economics survey, the downtime costs for Global 2000 companies is estimated around $400B annually, which means $200M per company per year, with an average of of $9,000 or $540,000 per hour.

Sounds perfect, right? Well, it’s close, but there’s a catch. Some of these Cloud Service Providers, like AWS, GCP (not Azure), have additional Data Transfer Costs for cross-Availability Zone communication.

What Are Data Transfer Costs for Cross-Availability Zone Communication?

As the name suggests, these costs come from data moving between resources in different Availability Zones, and are usually calculated per gigabyte ($/GB). While it might not seem like much at first glance, let’s look at some example of real-life traffic.

In October 2023, Coupang.com, one of the main e-commerce in South Korea, had around 127.6 million monthly visits. With the average page size at 4.97MB, and about 12 pages visited per session, the monthly traffic easily reach tens of petabytes of data. Even if only half of this traffic involves cross-zone communication, the cost for the transit of data between Availability Zone quickly reach hundreds of thousands of dollars in a single month.

Kubernetes’s native Topology Aware Routing (aka Topology Aware Hints)

Starting in version 1.21, Kubernetes introduced Topology Aware Hints to minimize cross-zone traffic within clusters. This routing strategy is built on EndpointSlices, which were first introduced in version 1.17 to improve the scalability of the traditional Endpoints resource. When a new Service is created, Kubernetes automatically generates EndpointSlices, breaking down the network endpoints into manageable chunks. This reduces the overhead on kube-proxy and overcome the size limitations of objects stored in etcd (max 1.5MB).

The related EndpointSlice manifest will be the following:

apiVersion: discovery.k8s.io/v1

kind: EndpointSlice

metadata:

  ...

  ownerReferences:

  - apiVersion: v1

    blockOwnerDeletion: true

    controller: true

    kind: Service

    name: tasks-vastaya-svc

addressType: IPv4

ports:

- name: http

  port: 80

  protocol: TCP

endpoints:

- addresses:

  - 10.244.3.74

  conditions:

    ready: true

    serving: true

    terminating: false

  nodeName: aks-hazlpoolha-33634351-vmss000000

  targetRef:

    kind: Pod

    name: tasks-vastaya-dplmt-68cd4dd76c-rblxz

    namespace: vastaya

    uid: 8fddbf95-dac8-420c-b0ab-d5076f9f27e9

  zone: koreacentral-1

- addresses:

  - 10.244.2.181

  conditions:

    ready: true

    serving: true

    terminating: false

  nodeName: aks-hazlpoolha-33634351-vmss000001

  targetRef:

    kind: Pod

    name: tasks-vastaya-dplmt-68cd4dd76c-cwshq

    namespace: vastaya

    uid: 8c82addd-1123-4810-ad21-0533e8cd15ee

  zone: koreacentral-2

- addresses:

  - 10.244.1.108

  - 10.244.1.110

  conditions:

    ready: true

    serving: true

    terminating: false

  nodeName: aks-hazlpoolha-33634351-vmss000002

  targetRef:

    kind: Pod

    name: tasks-vastaya-dplmt-68cd4dd76c-dwxg2

    namespace: vastaya

    uid: b5128ae8-6615-41e6-97ec-8db9b81b588e

  zone: koreacentral-3

However, while Topology-Aware Routing helps reduce inter-zone traffic, it has some inherent limitations. Endpoint allocation is relatively static, meaning it doesn’t adapt to real-time conditions like traffic load, network latency, or service health beyond basic readiness and liveness probes. This can lead to imbalanced resource utilization, especially in dynamic environments where local endpoints are overwhelmed while remote ones remain underutilized.

This is where High Availability Zone-aware Load Balancing (HAZL) comes into play.

What is HAZL?

High Availability Zone-aware Load Balancing (HAZL) is a load balancer that leverages Topology-aware Routing, as well as the HTTP and gRPC traffic intercepted by the sidecar proxy running in meshed pods to load balance each request independently, routing to the best available backend based on current conditions. It operates at the request-level, unlike traditional connection-level load balancing, where all requests in a connection are sent to the same backend.

It also monitor the number of in-flight requests (requests waiting for resources or connections) referred as “load.” to the services, and handle the traffic between zones on a per-request basis. If load or latency spikes — signs that the system is under stress or unhealthy — HAZL adds additional endpoints from other zones. On the other hand, when the load decreases, HAZL remove those extra endpoints.

This adaptive approach fill the gaps of Topology-aware Routing allowing to a more controlled and more dynamic management of the cross-zone traffic, providing a balance between reducing latency, ensuring service reliability, and optimizing resource utilization.

HAZL is currently available only for Buoyant Enterprise for Linkerd and not for Linkerd Open Source.

What is Buoyant Enterprise for Linkerd?

Linkerd began its journey in the open-source world in 2016 and has since improved immensely. However, as corporations like Microsoft, Adidas, and Geico started incorporating Linkerd into their architectures, it became necessary to provide enterprise-level services and support that go beyond what is possible with open-source alone. This includes everything from Tailored Proofs of Concept, Software Bills-of-Materials for all components, a dedicated support channels by private support ticketing allowing them to have a direct point of contact instead of relying on public forums, Service Level Agreements, and more.

However, the Buoyant’s commitment to the open-source community is reflected in its pricing model. Anyone can try the enterprise features for non-production traffic, and companies with fewer than 50 employees can use Buoyant Enterprise for Linkerd in production for free, at any scale. Beyond that, there are different pricing tiers depending on the number of meshed pods and the specific features required.

Enough with the theory — let’s get our hands dirty and see HAZL in action.

Demonstration

In this demonstration, I will deploy the following infrastructure from scratch on an AKS cluster using Terraform. Then, I will install Prometheus, Linkerd Enterprise, and use Grafana to collect metrics of the traffic before and after enabling HAZL.

as it will provide a license to use during the installation of the control plane.

Now, all the metrics scraped by the Linkerd-viz Prometheus instance are available in our Prometheus instance.

Install and configure Grafana

Next, let’s install Grafana with the default configuration:

helm upgrade --install grafana grafana/grafana \

  --create-namespace \

  --namespace monitoring

After logging in, we need to add a new data source pointing to the Prometheus server running in the cluster. To do this:

• Expand the Connections option from the side pane and click Add new connection.


• Click Prometheus and enter the internal DNS endpoint of the Kubernetes cluster (
  
  
  

  Next, let’s create a new dashboard that will contain the visualizations we will use to monitor the traffic to and from our nodes using the previously created data source targeting the Prometheus server. The visualizations we need are the following:

  
  
  
  
  
  • 
    CPU Usage per Kubernetes Node: This query will display the CPU usage percentage for each node. In this case, we expect that there will be a peak of CPU utilization in one of the nodes when we will trigger the jobs to simulate the traffic.
    
  
  
  
  
  
  
  

  CODE Kopieren

  100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100)
  
  

  
  
  
  
  
  
  
  
  
  
  • 
    TCP Read Bytes Total (Outbound): This query shows the total number of bytes read over TCP connections for outbound traffic in the vastaya namespace, grouped by namespace, pod, instance, destination zone, and source zone. This metrics are collected by the Linkerd proxy.
    
  
  
  
  
  
  
  

  CODE Kopieren

  sum by (namespace, pod, instance, dst_zone, src_zone) (
  
    tcp_read_bytes_total{direction="outbound", namespace="vastaya"}
  
  )
  
  

  
  
  
  
  
  
  
  

  
  
  
  Simulate the traffic with HAZL disabled
  

  
  
  

  With this setup in place, we can proceed to trigger a job that will create 5 replicas, each one increasing the number of requests to the service every 10 seconds. To ensure that the pods are provisioned in the node pool without the application, we will also set a node affinity.
  
  

  
  
  
  
  

  CODE Kopieren

  apiVersion: batch/v1
  
  kind: Job
  
  metadata:
  
    name: bot-get-project-report
  
    namespace: vastaya
  
  spec:
  
    completions: 5          
  
    parallelism: 5          
  
    template:
  
      spec:
  
        affinity:
  
          nodeAffinity:
  
            requiredDuringSchedulingIgnoredDuringExecution:
  
              nodeSelectorTerms:
  
              - matchExpressions:
  
                - key: agentpool
  
                  operator: In
  
                  values:
  
                  - default
  
        containers:
  
        - name: project-creator
  
          image: curlimages/curl:7.78.0 
  
          command: ["/bin/sh", "-c"] 
  
          args:
  
          - |
  
            API_URL="http://projects.vastaya.svc.cluster.local/1/report"
  
            get_report() {
  
              local num_requests=$1
  
              echo "Getting $num_requests tasks..."
  
              for i in $(seq 1 $num_requests); do
  
                (
  
                  echo "Getting task $i..."
  
                  GET_RESPONSE=$(curl -s -X GET "$API_URL")
  
                  echo $GET_RESPONSE
  
                ) &
  
              done
  
              wait
  
            }
  
            wait_time=10
  
            for num_requests in 5000 10000 15000; do
  
              echo "Running with $num_requests requests..."
  
              get_report $num_requests
  
              echo "Waiting for $wait_time seconds before increasing requests..."
  
              sleep $wait_time
  
            done
  
        restartPolicy: Never
  
    backoffLimit: 1
  
  

  
  
  
  
  
  
  
  

  Since we haven’t enabled HAZL yet, Kubernetes will start directing the requests to pods running in different zones, resulting in an increase of cross-zone traffic.

  
  
  
  
  

  
  
  
  References
  

  
  
  
  
  
  • AWS Data Transfer Cost:
    
  
  
  • HAZL Official Documentation:
    
  
  
  • Demo Source Code: https://github.com/GTRekter/Vastaya
    
  
  
  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen