Troubleshooting Azure Application Gateway: Fixing DNS Configuration to Resolve Internal Container App Connection Issues

Symptoms

Azure Application Gateway is unable to connect to an internal container app due to DNS resolution failures.

Environment

Azure Cloud, Azure Application Gateway, Internal Container Apps (e.g., Azure Kubernetes Service, Azure Container Instances)

Most Likely Causes

The DNS configuration for the Azure Application Gateway is not correctly set up to resolve the internal container app's hostname, leading to connection failures.

What to Check First

1. Confirm the backend target FQDN and whether it should resolve through private DNS, custom DNS, or public DNS.
2. Capture current Application Gateway backend health before changing DNS or NSG settings.
3. Confirm the Application Gateway subnet can reach the resolver and the internal container app subnet.

Fix Steps

1. Verify DNS Settings in Azure Application Gateway

   Check the DNS settings configured in the Azure Application Gateway to ensure they point to the correct internal DNS server.

   Example pattern only. Adjust for your environment before running.

   Navigate to the Azure Portal.
   Select 'Application Gateways' from the services list.
   Choose the specific Application Gateway instance.
   Under 'Settings', select 'Configuration'.
   Review the 'DNS settings' section to verify the DNS servers listed.

2. Update DNS Settings if Necessary

   If the DNS settings are incorrect, update them to point to the appropriate internal DNS server that can resolve the container app's hostname.

   Example pattern only. Adjust for your environment before running.

   In the Azure Portal, go to the 'Configuration' section of the Application Gateway.
   Click on 'Edit' in the DNS settings.
   Add or modify the DNS server IP addresses to point to the internal DNS server.
   Click 'Save' to apply the changes.

3. Check Network Security Group (NSG) Rules

   Ensure that the Network Security Group associated with the Application Gateway allows outbound traffic to the internal container app's subnet.

   Example pattern only. Adjust for your environment before running.

   In the Azure Portal, navigate to 'Network Security Groups'.
   Select the NSG associated with the Application Gateway.
   Under 'Settings', click on 'Outbound security rules'.
   Verify that there is a rule allowing outbound traffic to the subnet of the internal container app.

4. Test DNS Resolution from Application Gateway

   Use the Azure CLI to test DNS resolution from the Application Gateway to the internal container app's hostname.

   Safe to run: read-only

   Open Azure Cloud Shell or your local terminal with Azure CLI installed.
   Run the following command to test DNS resolution:
   az network application-gateway show --name <YourAppGatewayName> --resource-group <YourResourceGroup> --query 'frontendIpConfigurations[].ipAddress'
   Use the resolved IP address to ping the internal container app's hostname.

5. Review Application Gateway Logs

   Check the Application Gateway access logs for any error messages related to the connection attempts to the internal container app.

   Example pattern only. Adjust for your environment before running.

   In the Azure Portal, go to 'Application Insights' linked to the Application Gateway.
   Select 'Logs' under the Application Insights resource.
   Run a query to filter logs for connection attempts to the internal container app.
   Analyze the logs for any DNS resolution errors or connection timeouts.

Validation

• Application Gateway backend health changes from unhealthy/DNS failure to healthy for the expected backend pool.
• A lookup from the same virtual network path resolves the container app hostname to the expected private address.
• Access logs no longer show repeated backend connection failures for the same hostname.

Logs to Check

• Application Gateway access logs and performance logs.
• Backend health details in Application Gateway.
• NSG flow logs or firewall logs for resolver and backend traffic.
• Private DNS zone records and virtual network links.

Rollback and Escalation

• Restore the previous DNS server list or private DNS record if the new resolver path breaks other backends.
• Revert temporary NSG or route changes after backend health is confirmed.
• Keep a screenshot or exported copy of the original Application Gateway and DNS settings before edits.

Edge Cases

• If the internal container app is deployed in a different region, ensure that the Application Gateway is configured for cross-region access.
• If using a custom DNS server, verify that it is operational and correctly configured to resolve the internal container app's hostname.

Notes from the Field

• Backend health usually tells you whether this is DNS, TCP reachability, TLS/SNI, or application response. Check it before editing DNS.
• Do not fix a private-name problem by pointing everything at public DNS. Confirm where that hostname is supposed to resolve from the gateway subnet.