Increased DNS Requests After Enabling Istio Egress Gateways

[volkorny]

Hello!

We have the ndots:1 configuration enabled for our applications. Before migrating to Istio, our DNS resolutions of external domains were done with a single request to CoreDNS.

After migrating to Istio, everything worked fine for months. Three months ago, we decided to enable Egress Gateways to control egress traffic and set the outboundTrafficPolicy:mode:REGISTRY_ONLY mode. We noticed that the number of requests to CoreDNS increased by 4-5 times. In the CoreDNS logs, we see a series of requests instead of a single request:

  [INFO] 10.x.x.55:39786 - 26360 "A IN taskrouter.twilio.com.istio-ingress.svc.cluster.local. udp 71 false 512" NXDOMAIN qr,aa,rd 164 0.000116158s
  [INFO] 10.x.x.55:39786 - 7696 "A IN taskrouter.twilio.com.svc.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.000092738s
  [INFO] 10.x.x.55:39786 - 62817 "A IN taskrouter.twilio.com.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.0000795s
  [INFO] 10.x.x.55:39786 - 3610 "A IN taskrouter.twilio.com.us-west-2.compute.internal. udp 66 false 512" NXDOMAIN qr,rd,ra 179 0.000313721s
  [INFO] 10.x.x.55:39786 - 62146 "A IN taskrouter.twilio.com. udp 39 false 512" NOERROR qr,aa,rd,ra 397 0.000054419s

Our resolv.conf file still has ndots:1, but it seems like this option doesn't work with Istio.

/app $ cat /etc/resolv.conf
search staging-mesh.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
nameserver 172.20.0.10
options ndots:1
/app $

The increased load on CoreDNS is causing timeout errors in our environment.

Please advise if it is possible to decrease the number of requests to CoreDNS to the level it was before enabling the Egress Gateways.
Thanks!

[husnialhamdani]

Maybe you can enable Istio Smart DNS proxy in Istio https://istio.io/latest/blog/2020/dns-proxy/#reduced-load-on-your-dns-servers-w-faster-resolution, https://istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/.

[volkorny]

My configuration contains this and dns proxy looks like enabled:

meshConfig:
  defaultConfig:
    proxyMetadata:
      # Enable basic DNS proxying
      ISTIO_META_DNS_CAPTURE: "true"
      # Enable automatic address allocation, optional
      ISTIO_META_DNS_AUTO_ALLOCATE: "true"

[howardjohn]

I cannot reproduce. Are you sure the queries this is coming from has ndots:1? Note it could be the ServiceEntry you created is applying to every proxy in the cluster (or at least, many proxies), and not all have ndots=1 set. You can find who the 10.x.x.55:39786 IP is in the query and double check?

BTW, the DNS proxy does not help as you found; that only applies to your own application, not DNS requests send from Istio itself.

[howardjohn]

have you tried manually querying the DNS from the pod (with dig or kdig, should be in the proxy image)?That will show the ttl. Maybe it really is 30s

[volkorny]

I have created new DNS record with TTL 7200 secnds.
But I still have on logs:
DNS refresh rate reset for DOMAIN, refresh rate XXX (XXX are: 30000 ms, 9000 ms, 29000 ms, 24000 ms)
but never more than 30000 ms

[volkorny]

In CoreDNS logs I see every pod in our meah make requests to this 7200-seconds record 1-2 timers per minute...

[howardjohn]

the reason I asked to do a query with dig is it could be the ttl is modified by kube-dns

[volkorny]

yes, it really modified by kube-dns, thanks