NOTE: Application Gateway for Containers has been released, which introduces numerous performance, resilience, and feature changes. Please consider leveraging Application Gateway for Containers for your next deployment.
When you are using Application Gateway with AKS for L7, you need to make sure that you have setup network connectivity correctly between the gateway and the cluster. Otherwise, you might receive 502s when reaching your site.
There are two major things to consider when setting up network connectivity between Application Gateway and AKS
- Virtual Network Configuration
- When AKS and Application Gateway in the same virtual network
- When AKS and Application Gateway in different virtual networks
- Network Plugin used with AKS
If you have deployed AKS and Application Gateway in the same virtual network with Azure CNI
for network plugin, then you don't have to do any changes and you are good to go. Application Gateway instances should be able to reach the PODs.
If you are using kubenet
network plugin, then jump to Kubenet to setup the route table.
AKS can be deployed in different virtual network from Application Gateway's virtual network, however, the two virtual networks must be peered together. When you create a virtual network peering between two virtual networks, a route is added by Azure for each address range within the address space of each virtual network a peering is created for.
aksClusterName="<aksClusterName>"
aksResourceGroup="<aksResourceGroup>"
appGatewayName="<appGatewayName>"
appGatewayResourceGroup="<appGatewayResourceGroup>"
# get aks vnet information
nodeResourceGroup=$(az aks show -n $aksClusterName -g $aksResourceGroup -o tsv --query "nodeResourceGroup")
aksVnetName=$(az network vnet list -g $nodeResourceGroup -o tsv --query "[0].name")
aksVnetId=$(az network vnet show -n $aksVnetName -g $nodeResourceGroup -o tsv --query "id")
# get gateway vnet information
appGatewaySubnetId=$(az network application-gateway show -n $appGatewayName -g $appGatewayResourceGroup -o tsv --query "gatewayIpConfigurations[0].subnet.id")
appGatewayVnetName=$(az network vnet show --ids $appGatewaySubnetId -o tsv --query "name")
appGatewayVnetId=$(az network vnet show --ids $appGatewaySubnetId -o tsv --query "id")
# set up bi-directional peering between aks and gateway vnet
az network vnet peering create -n gateway2aks \
-g $appGatewayResourceGroup --vnet-name $appGatewayVnetName \
--remote-vnet $aksVnetId \
--allow-vnet-access
az network vnet peering create -n aks2gateway \
-g $nodeResourceGroup --vnet-name $aksVnetName \
--remote-vnet $appGatewayVnetId \
--allow-vnet-access
If you are using Azure CNI
for network plugin with AKS, then you are good to go.
If you are using Kubenet
network plugin, then jump to Kubenet to setup the route table.
When using Azure CNI, Every pod is assigned a VNET route-able private IP from the subnet. So, Gateway should be able reach the pods directly.
When using Kubenet mode, Only nodes receive an IP address from subnet. Pod are assigned IP addresses from the PodIPCidr
and a route table is created by AKS. This route table helps the packets destined for a POD IP reach the node which is hosting the pod.
When packets leave Application Gateway instances, Application Gateway's subnet need to aware of these routes setup by the AKS in the route table.
A simple way to achieve this is by associating the same route table created by AKS to the Application Gateway's subnet. When AGIC starts up, it checks the AKS node resource group for the existence of the route table. If it exists, AGIC will try to assign the route table to the Application Gateway's subnet, given it doesn't already have a route table. If AGIC doesn't have permissions to any of the above resources, the operation will fail and an error will be logged in the AGIC pod logs.
This association can also be performed manually:
aksClusterName="<aksClusterName>"
aksResourceGroup="<aksResourceGroup>"
appGatewayName="<appGatewayName>"
appGatewayResourceGroup="<appGatewayResourceGroup>"
# find route table used by aks cluster
nodeResourceGroup=$(az aks show -n $aksClusterName -g $aksResourceGroup -o tsv --query "nodeResourceGroup")
routeTableId=$(az network route-table list -g $nodeResourceGroup --query "[].id | [0]" -o tsv)
# get the application gateway's subnet
appGatewaySubnetId=$(az network application-gateway show -n $appGatewayName -g $appGatewayResourceGroup -o tsv --query "gatewayIpConfigurations[0].subnet.id")
# associate the route table to Application Gateway's subnet
az network vnet subnet update \
--ids $appGatewaySubnetId
--route-table $routeTableId