This publish was co-authored by Qi Ke, Company Vice President, Azure Kubernetes Service.
At present, we’re thrilled to announce the final availability of Azure CNI Overlay. This can be a large step ahead in addressing networking efficiency and the scaling wants of our clients.
As cloud-native workloads proceed to develop, clients are consistently pushing the dimensions and efficiency boundaries of our present networking options in Azure Kubernetes Service (AKS). For Occasion, the normal Azure Container Networking Interface (CNI) approaches require planning IP addresses upfront, which may result in IP tackle exhaustion as demand grows. In response to this demand, now we have developed a brand new networking resolution known as “Azure CNI Overlay”.
On this weblog publish, we’ll talk about why we would have liked to create a brand new resolution, the dimensions it achieves, and the way its efficiency compares to the prevailing options in AKS.
Fixing for efficiency and scale
In AKS, clients have a number of community plugin choices to select from when making a cluster. Nevertheless, every of those choices have their very own challenges in relation to large-scale clusters.
The “kubenet” plugin, an present overlay community resolution, is constructed on Azure route tables and the bridge plugin. Since kubenet (or host IPAM) leverages route tables for cross node communication it was designed for, not more than 400 nodes or 200 nodes in twin stack clusters.
The Azure CNI VNET gives IPs from the digital community (VNET) tackle area. This can be tough to implement because it requires a big, distinctive, and consecutive Classless Inter-Area Routing (CIDR) area and clients could not have the obtainable IPs to assign to a cluster.
Deliver your Personal Container Community Interface (BYOCNI) brings numerous flexibility to AKS. Prospects can use encapsulation—like Digital Extensible Native Space Community (VXLAN)—to create an overlay community as nicely. Nevertheless, the extra encapsulation will increase latency and instability because the cluster measurement will increase.
To deal with these challenges, and to help clients who need to run massive clusters with many nodes and pods with no limitations on efficiency, scale, and IP exhaustion, now we have launched a brand new resolution: Azure CNI Overlay.
Azure CNI Overlay
Azure CNI Overlay assigns IP addresses from the user-defined overlay tackle area as a substitute of utilizing IP addresses from the VNET. It makes use of the routing of those non-public tackle areas as a local digital community characteristic. Which means cluster nodes don’t have to carry out any further encapsulation to make the overlay container community work. This additionally permits this overlay addressing area to be reused for various AKS clusters even when related by way of the identical VNET.
When a node joins the AKS cluster, we assign a /24 IP tackle block (256 IPs) from the Pod CIDR to it. Azure CNI assigns IPs to Pods on that node from the block, and beneath the hood, VNET maintains a mapping of the Pod CIDR block to the node. This fashion, when Pod visitors leaves the node, VNET platform is aware of the place to ship the visitors. This permits the Pod overlay community to attain the identical efficiency as native VNET visitors and paves the best way to help tens of millions of pods and throughout hundreds of nodes.
Datapath efficiency comparability
This part sneaks into a number of the datapath efficiency comparisons now we have been operating in opposition to Azure CNI Overlay.
Observe: We used the Kubernetes benchmarking instruments obtainable at kubernetes/perf-tests for this train. Comparability can fluctuate based mostly on a number of components akin to underlining {hardware} and Node proximity inside a datacenter amongst others. Precise outcomes would possibly fluctuate.
Azure CNI Overlay vs. VXLAN-based Overlay
As talked about earlier than, the one choices for big clusters with many Nodes and plenty of Pods are Azure CNI Overlay and BYO CNI. Right here we evaluate Azure CNI Overlay with VXLAN-based overlay implementation utilizing BYO CNI.
TCP Throughput – Increased is Higher (19% achieve in TCP Throughput)
Azure CNI Overlay confirmed a major efficiency enchancment over VXLAN-based overlay implementation. We discovered that the overhead of encapsulating CNIs was a major consider efficiency degradation, particularly because the cluster grows. In distinction, Azure CNI Overlay’s native Layer 3 implementation of overlay routing eradicated the double-encapsulation useful resource utilization and confirmed constant efficiency throughout numerous cluster sizes. In abstract, Azure CNI Overlay is a most viable resolution for operating manufacturing grade workloads in Kubernetes.
Azure CNI Overlay vs. Host Community
This part will cowl how pod networking performs in opposition to node networking and see how native L3 routing of pod networking helps Azure CNI Overlay implementation.
Azure CNI Overlay and Host Community have related throughput and CPU utilization outcomes, and this reinforces that the Azure CNI Overlay implementation for Pod routing throughout nodes utilizing the native VNET characteristic is as environment friendly as native VNET visitors.
TCP Throughput – Increased is Higher (Much like HostNetwork)
Azure CNI Overlay powered by Cilium: eBPF dataplane
Up thus far, we’ve solely taken a take a look at Azure CNI Overlay advantages alone. Nevertheless, by means of a partnership with Isovalent, the subsequent era of Azure CNI is powered by Cilium. Among the advantages of this method embody higher useful resource utilization by Cilium’s prolonged Berkeley Packet Filter (eBPF) dataplane, extra environment friendly intra cluster load balancing, Community Coverage enforcement by leveraging eBPF over iptables, and extra. To learn extra about Cilium’s efficiency good points by means of eBPF, see Isovalent’s weblog publish on the topic.
In Azure CNI Overlay Powered by Cilium, Azure CNI Overlay units up the IP-address administration (IPAM) and Pod routing, and Cilium provisions the Service routing and Community Coverage programming. In different phrases, Azure CNI Overlay Powered by Cilium permits us to have the identical overlay networking efficiency good points that we’ve seen to date on this weblog publish plus extra environment friendly Service routing and Community Coverage implementation.
It is nice to see that Azure CNI Overlay powered by Cilium is ready to present even higher efficiency than Azure CNI Overlay with out Cilium. The upper pod to service throughput achieved with the Cilium eBPF dataplane is a promising enchancment. The added advantages of elevated observability and extra environment friendly community coverage implementation are additionally vital for these trying to optimize their AKS clusters.
TCP Throughput – Increased is best
To wrap up, Azure CNI Overlay is now typically obtainable in Azure Kubernetes Service (AKS) and presents important enhancements over different networking choices in AKS, with efficiency corresponding to Host Community configurations and help for linearly scaling the cluster. And pairing Azure CNI Overlay with Cilium brings much more efficiency advantages to your clusters. We’re excited to ask you to strive Azure CNI Overlay and expertise the advantages in your AKS setting.
To get began at this time, go to the documentation obtainable.
