Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AdGuard Home's DNS-over-QUIC server functionality fails to make itself available on LAN IPs #3465

Closed
3 tasks done
DandelionSprout opened this issue Aug 18, 2021 · 21 comments
Closed
3 tasks done
Labels

Comments

@DandelionSprout
Copy link
Member

DandelionSprout commented Aug 18, 2021

Have a question or an idea? Please search it on our forum to make sure it was not yet asked. If you cannot find what you had in mind, please submit it here.

Prerequisites

Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.

  • I am running the latest version
  • I checked the documentation and found no answer
  • I checked to make sure that this issue has not already been filed

Issue Details

  • Version of AdGuard Home server:
    • v0.107.0-a.136+550b1798
  • How did you install AdGuard Home:
    • GitHub releases
  • How did you setup DNS configuration:
    • System
  • If it's a router or IoT, please write device model:
    • Raspberry Pi 4 8GB
  • CPU architecture:
    • ARM64
  • Operating system and version:
    • Fedora Workstation 34

Expected Behavior

AGH's DNS-over-QUIC server can be accessed from LAN home devices.

Actual Behavior

Okay, so I had for some weeks thought that my connection problems to quic:https://dandelionsprout.asuscomm.com:48582 were the result of failing to scan IPv4 addresses on IPv6-able connections. That may or may not have been the case, but now the problem has either changed in nature, or I was incorrect to begin with; since further testing tonight, led me to the conclusion that it's a problem with LAN connections instead:

(Easiest to reproduce with AdGuard for Android, as steps 5-7 hinge on a phone connection.)

  1. Have an AdGuard Home server with DNS-over-QUIC functionality, and with port forwarding set up for it on the home's router.
  2. Connect your phone to a Wi-Fi network in the same home.
  3. Connect to your DNS-over-QUIC server at AdGuard Settings → DNS filtering → Choose DNS server → Add custom DNS server.
  4. See the "No connection" error.
  5. Disconnect the phone from the Wi-Fi network, and change to using a 4G connection.
  6. Repeat step 3.
  7. See that the DNS server is connected now.

Screenshots

Screenshot:

N/A at the time of writing.

Additional Information

Successor to AdguardTeam/AdguardForAndroid#3927 and AdguardTeam/AdguardForWindows#3878

AdGuard for Android log: adguard.log — Highlight sequence:

21:07:32.514 [AsyncTask #14] INFO  c.a.android.filtering.commons.d - Found IPv6 address for network name:wlan0 (wlan0)
21:07:37.664 [AsyncTask #14] WARN  com.adguard.android.filtering.dns.c - Failed to test quic:https://dandelionsprout.asuscomm.com:48582
java.lang.IllegalArgumentException: Request timed out
	at com.adguard.dnslibs.proxy.DnsProxy.testUpstream(DnsProxy.java:236)
	at com.adguard.android.filtering.dns.c.a(DnsProxyUtils.java:88)
	at com.adguard.android.filtering.dns.c.a(DnsProxyUtils.java:51)
	at com.adguard.android.ui.CustomDnsActivity$a.doInBackground(CustomDnsActivity.java:2314)
	at android.os.AsyncTask$3.call(AsyncTask.java:378)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:289)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:919)

AdGuard for Windows log: service_18-08-2021-20_36_13.647-2021-08-18.log — Highlight sequence:

INFO, AdguardSvc, 38, 18.08.2021 20:39:40.757, Start creating DNS servers https://dandelionsprout.asuscomm.com:2501/dns-query
quic:https://dandelionsprout.asuscomm.com:48582
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.757, Creating DNS servers has been completed successfully
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.758, Start testing DNS server DNS server: , (https://dandelionsprout.asuscomm.com:2501/dns-query, quic:https://dandelionsprout.asuscomm.com:48582; 0), type: Multi
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.812, Adguard.Dns.Api.DnsApi | Testing upstream
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.812, Adguard.Dns.Api.DnsApi | Start testing upstream Adguard.Dns.Api.DnsProxyServer.Configs.UpstreamOptions
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.977, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Destroying...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Stopping event loop...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Done
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Waiting all requests completed...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Done
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Destroyed
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Api.DnsApi | Testing upstream has been completed successfully
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Api.DnsApi | Testing upstream has been successfully completed
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Address https://dandelionsprout.asuscomm.com:2501/dns-query has been resolved with result True
INFO, AdguardSvc, 38, 18.08.2021 20:39:41.032, Adguard.Dns.Api.DnsApi | Testing upstream
INFO, AdguardSvc, 38, 18.08.2021 20:39:41.032, Adguard.Dns.Api.DnsApi | Start testing upstream Adguard.Dns.Api.DnsProxyServer.Configs.UpstreamOptions
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Adguard.Dns.Api.DnsApi | Testing upstream failed with an error Request failed (empty packet)
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Adguard.Dns.Api.DnsApi | Testing upstream has been successfully completed
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Address quic:https://dandelionsprout.asuscomm.com:48582 has been resolved with result False
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Testing DNS server has been completed successfully with the result False

Notably, the errors shown in those logs, differ from those that were in the predecessor issue reports.

@ameshkov
Copy link
Member

ameshkov commented Aug 19, 2021

@DandelionSprout just in case, are you on the latest nightly of AG for Android?

AGH has a newer QUIC version, higher than the current stable version of AG supports.

@ainar-g ainar-g added the waiting for data Waiting for users to provide more data. label Aug 19, 2021
@DandelionSprout
Copy link
Member Author

From what I can tell, yes: v4.0 nightly 22.

@ameshkov
Copy link
Member

Hm, that's strange then.

Could you please enable debug log in AG for Android/Win and see if it starts printing additional info on the error?

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 19, 2021

AdGuard for Android debug log (Looks similar to the non-debug log in my eyes): adguard.log

AdGuard for Windows 7.7 nightly 13 verbose service-log (Also looks similar to me, sadly; also includes info on an unrelated but severe SSL certificate installation error):
service_19-08-2021-13_27_23.367-2021-08-19.log

@ameshkov
Copy link
Member

Tbh, I think the only viable way to troubleshoot this now is to use Wireshark on your Windows PC AND tcpdump on the AGH server and take a look at actual UDP packets on both sides.

@DandelionSprout
Copy link
Member Author

I'm on the case.

! ———Wireshark (QUIC requests only, as I didn't know what else to look for.)———
! ——— 84.202.44.53 is my router's public IP. 192.168.1.168 is my DNS server's LAN IP. 192.168.1.188 is my PC with AdGuard for Windows. ———
11879	9.671864	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=8339928ebea2f44d2ae4584373307f4cf3f8, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 0, CRYPTO, PADDING
11882	9.674080	84.202.44.53	192.168.1.188	QUIC	194	Retry, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7
11883	9.674190	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 1, CRYPTO, PADDING
11886	9.676154	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
12522	10.176299	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 2, CRYPTO, PADDING
12524	10.179608	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
14218	11.189265	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 3, CRYPTO, PADDING
14226	11.191422	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
16538	13.187478	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 4, CRYPTO, PADDING
16543	13.190047	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
23276	19.684265	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=5886a70450424098de9ac6f95b4be3ce09cd, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 0, CRYPTO, PADDING
23284	19.686454	84.202.44.53	192.168.1.188	QUIC	194	Retry, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db
23289	19.686583	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 1, CRYPTO, PADDING
23292	19.688472	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
23906	20.186259	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 2, CRYPTO, PADDING
23907	20.188317	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
25125	21.186081	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 3, CRYPTO, PADDING
25140	21.188347	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
28210	23.184121	192.168.1.188	84.202.44.53	QUIC	1274	Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 4, CRYPTO, PADDING
28213	23.186139	192.168.1.168	192.168.1.188	QUIC	97	Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC

! ——— tcpdump verbose ———
[dandelionsprout@fedora ~]$ sudo tcpdump port 48582 -v
[sudo] passord for dandelionsprout: 
dropped privs to tcpdump
tcpdump: listening on enp1s0u2c2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:29:52.979807 IP (tos 0x0, ttl 54, id 17141, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0x0cc4 (correct), seq 3690508245:3690508788, ack 8803269, win 11975, length 543
14:29:52.979843 IP (tos 0x0, ttl 64, id 29399, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xccbd (correct), ack 543, win 24572, length 0
14:29:52.981066 IP (tos 0x0, ttl 64, id 29400, offset 0, flags [DF], proto TCP (6), length 583)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [P.], cksum 0x5214 (correct), seq 1:544, ack 543, win 24576, length 543
14:29:53.021146 IP (tos 0x0, ttl 54, id 17142, offset 0, flags [DF], proto TCP (6), length 40)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [.], cksum 0xfbd3 (correct), ack 544, win 11975, length 0
14:29:53.022594 IP (tos 0x0, ttl 127, id 33910, offset 0, flags [none], proto UDP (17), length 1260)
    _gateway.59243 > fedora.48582: UDP, length 1232
14:29:53.023335 IP (tos 0x0, ttl 64, id 56588, offset 0, flags [DF], proto UDP (17), length 180)
    fedora.48582 > _gateway.59243: UDP, length 152
14:29:53.026033 IP (tos 0x0, ttl 127, id 33911, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:53.027096 IP (tos 0x0, ttl 64, id 1030, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:53.156024 IP (tos 0x0, ttl 54, id 17143, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0xc441 (correct), seq 543:1086, ack 544, win 11975, length 543
14:29:53.156072 IP (tos 0x0, ttl 64, id 29401, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xc87f (correct), ack 1086, win 24572, length 0
14:29:53.525704 IP (tos 0x0, ttl 127, id 33912, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:53.527272 IP (tos 0x0, ttl 64, id 1077, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:54.525010 IP (tos 0x0, ttl 127, id 33913, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:54.526183 IP (tos 0x0, ttl 64, id 1119, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:56.523577 IP (tos 0x0, ttl 127, id 33919, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:56.524755 IP (tos 0x0, ttl 64, id 1165, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:58.365743 IP (tos 0x0, ttl 64, id 29402, offset 0, flags [DF], proto TCP (6), length 583)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [P.], cksum 0x42a7 (correct), seq 544:1087, ack 1086, win 24576, length 543
14:29:58.407858 IP (tos 0x0, ttl 54, id 17144, offset 0, flags [DF], proto TCP (6), length 40)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [.], cksum 0xf795 (correct), ack 1087, win 11975, length 0
14:29:59.773475 IP (tos 0x0, ttl 54, id 17145, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0x795f (correct), seq 1086:1629, ack 1087, win 11975, length 543
14:29:59.773535 IP (tos 0x0, ttl 64, id 29403, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xc441 (correct), ack 1629, win 24572, length 0
14:30:01.309965 IP (tos 0x0, ttl 127, id 33920, offset 0, flags [none], proto UDP (17), length 1260)
    _gateway.59252 > fedora.48582: UDP, length 1232
14:30:01.310800 IP (tos 0x0, ttl 64, id 56983, offset 0, flags [DF], proto UDP (17), length 180)
    fedora.48582 > _gateway.59252: UDP, length 152
14:30:01.313597 IP (tos 0x0, ttl 127, id 33921, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:01.314587 IP (tos 0x0, ttl 64, id 1627, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:01.813964 IP (tos 0x0, ttl 127, id 33922, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:01.815119 IP (tos 0x0, ttl 64, id 1648, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:02.827199 IP (tos 0x0, ttl 127, id 33923, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:02.827878 IP (tos 0x0, ttl 64, id 1745, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:04.829200 IP (tos 0x0, ttl 127, id 33924, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:04.829777 IP (tos 0x0, ttl 64, id 1892, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:26.593909 IP (tos 0x8, ttl 233, id 20612, offset 0, flags [DF], proto UDP (17), length 96)
    182.3.104.57.39222 > fedora.48582: UDP, length 68
14:30:26.619361 IP (tos 0x0, ttl 64, id 3549, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:26.673074 IP (tos 0x0, ttl 64, id 3553, offset 0, flags [DF], proto UDP (17), length 248)
    fedora.48582 > 182.3.104.57.39222: UDP, length 220
14:30:26.913352 IP (tos 0x8, ttl 233, id 53310, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:26.913728 IP (tos 0x0, ttl 64, id 3561, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:27.243192 IP (tos 0x8, ttl 233, id 22725, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
14:30:28.674641 IP (tos 0x8, ttl 233, id 31648, offset 0, flags [DF], proto UDP (17), length 92)
    182.3.104.57.39222 > fedora.48582: UDP, length 64
14:30:28.701332 IP (tos 0x0, ttl 64, id 3586, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:28.708183 IP (tos 0x0, ttl 64, id 3587, offset 0, flags [DF], proto UDP (17), length 161)
    fedora.48582 > 182.3.104.57.39222: UDP, length 133
14:30:29.010228 IP (tos 0x8, ttl 233, id 63225, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:29.010918 IP (tos 0x0, ttl 64, id 3616, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:29.314693 IP (tos 0x8, ttl 233, id 26775, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
14:30:43.554794 IP (tos 0x8, ttl 233, id 8685, offset 0, flags [DF], proto UDP (17), length 84)
    182.3.104.57.39222 > fedora.48582: UDP, length 56
14:30:43.581071 IP (tos 0x0, ttl 64, id 3882, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:43.658933 IP (tos 0x0, ttl 64, id 3886, offset 0, flags [DF], proto UDP (17), length 204)
    fedora.48582 > 182.3.104.57.39222: UDP, length 176
14:30:43.954540 IP (tos 0x8, ttl 233, id 49439, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:43.955168 IP (tos 0x0, ttl 64, id 3915, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:44.236843 IP (tos 0x8, ttl 233, id 15916, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
^C
48 packets captured
56 packets received by filter
0 packets dropped by kernel

Full Wireshark dump (Remove the .txt suffix before loading it; also note that it's 25 MB): Wireshark dump for AdGuard QUIC problems.pcapng.txt

@ngorskikh
Copy link
Member

ngorskikh commented Aug 19, 2021

The issue is that your particular routing situation interferes with QUIC's address validation mechanism.

Probably, what's happening is that the first packet from your client to AGH server goes through NAT and arrives at AGH server as if it came from the router. After that, your router starts redirecting packets to AGH server transparently, so AGH server sees packets as if they are coming from the client directly, not the router.

This breaks QUIC's address validation mechanism:

  1. AGH sees an Initial packet from _gateway (not 192.168.1.188, since it went through NAT).
  2. AGH replies with a retry token which is specific for _gateway, and MUST be included in all subsequent Initial packets from that address.
  3. Subsequent packets are routed differently, and now come from 192.168.1.188, so AGH rejects them since they carry a retry token for a different address.

It would be interesting to see a Wireshark capture from your AGH machine as well, to better understand what's going on.

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 19, 2021

Fedora claims to have an aarch64 version of Wireshark ready for download, so I'll see if I can try to install and run it sometime tonight.

@ngorskikh
Copy link
Member

ngorskikh commented Aug 19, 2021

You could also try tcpdump -w <filename.pcap>

@ghost
Copy link

ghost commented Aug 21, 2021

Yogadns Dnscrypt is simple.

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 21, 2021

Okay, I've now run Wireshark on my server. The output doesn't make all that much more sense to me (192.168.1.130 is my Android phone):

1965	3.319471919	192.168.1.1	192.168.1.168	QUIC	1274	Initial, DCID=46398300e3e5745f2e96bbb6a35fc805e0f4, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 0, CRYPTO, PADDING
1966	3.320196650	192.168.1.168	192.168.1.1	QUIC	194	Retry, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2
1968	3.324200114	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 1, CRYPTO, PADDING
1969	3.325237156	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
2241	3.846250186	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 2, CRYPTO, PADDING
2242	3.846849252	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
2741	4.860325014	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 3, CRYPTO, PADDING
2742	4.861510775	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
3782	6.870938961	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 4, CRYPTO, PADDING
3783	6.871880930	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
10042	9.583268791	192.168.1.1	192.168.1.168	QUIC	1274	Initial, DCID=6966218c65af6a2e177ba8ba643a674bf1e1, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 0, CRYPTO, PADDING
10045	9.584316296	192.168.1.168	192.168.1.1	QUIC	194	Retry, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e
10057	9.588243039	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 1, CRYPTO, PADDING
10060	9.589311969	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
11331	10.115823073	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 2, CRYPTO, PADDING
11335	10.117543402	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
13694	11.126729647	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 3, CRYPTO, PADDING
13698	11.128217830	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
18475	13.136758453	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 4, CRYPTO, PADDING
18480	13.137905197	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
24677	15.742278190	192.168.1.1	192.168.1.168	QUIC	1274	Initial, DCID=e60b4534ac6bccda91577d44a94ad8bc5c78, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 0, CRYPTO, PADDING
24680	15.742948940	192.168.1.168	192.168.1.1	QUIC	194	Retry, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81
24692	15.747355807	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 1, CRYPTO, PADDING
24695	15.748004409	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
25981	16.273349955	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 2, CRYPTO, PADDING
25984	16.273987872	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
28316	17.275650978	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 3, CRYPTO, PADDING
28318	17.276256340	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
29822	19.281977964	192.168.1.130	192.168.1.168	QUIC	1274	Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 4, CRYPTO, PADDING
29823	19.282910507	192.168.1.168	192.168.1.130	QUIC	97	Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC

Full log: Wireshark Raspberry Pi AGH server capture.zip (31 MB when it's unpacked)

I do however notice one aspect: There are no captured queries at all from my Windows PC (192.168.1.188).

@ngorskikh
Copy link
Member

This seems to confirm the above hypothesis: the first initial packet comes from 192.168.1.1 as if the router did some sort of IP masquerading on output to the internal interface, while all subsequent packets come from 192.168.1.130, as if the router did DNAT on input from the internal interface.

To me this looks like some seriously broken router configuration :)

I've tried to reproduce this on my equipment, and my router correctly forwards the packets from the internal interface to the external one, then does masquerading, and then finally DNAT. For example this is what my AGH server sees when someone from the internal network tries accessing it via 123.123.123.123, which is the external (WAN) address of the router:

112	2021-08-22 18:54:02,557570	123.123.123.123	192.168.1.11	QUIC	1274	Initial, DCID=11d6990148d47319bacef3a2e507d53a9fa5, SCID=e6d11f0d9a46cef75c4a50c3b7df9624f2, PKN: 0, CRYPTO, PADDING
113	2021-08-22 18:54:02,557814	192.168.1.11	123.123.123.123	QUIC	100	Version Negotiation, DCID=e6d11f0d9a46cef75c4a50c3b7df9624f2, SCID=11d6990148d47319bacef3a2e507d53a9fa5
114	2021-08-22 18:54:02,563188	123.123.123.123	192.168.1.11	QUIC	1274	Initial, DCID=17cef632a21bc5eb32f47265bddab83701b4, SCID=fd9f0680d5eee528e78009819c877dd1cc, PKN: 0, CRYPTO, PADDING
115	2021-08-22 18:54:02,563469	192.168.1.11	123.123.123.123	QUIC	194	Retry, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=9d300575
116	2021-08-22 18:54:02,567608	123.123.123.123	192.168.1.11	QUIC	1274	Initial, DCID=9d300575, SCID=fd9f0680d5eee528e78009819c877dd1cc, PKN: 1, CRYPTO, PADDING
117	2021-08-22 18:54:02,575669	192.168.1.11	123.123.123.123	QUIC	1294	Handshake, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=eb762aeb
118	2021-08-22 18:54:02,575781	192.168.1.11	123.123.123.123	QUIC	1187	Handshake, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=eb762aeb
119	2021-08-22 18:54:02,575803	192.168.1.11	123.123.123.123	QUIC	198	Protected Payload (KP0), DCID=fd9f0680d5eee528e78009819c877dd1cc

Here are some of the possible solutions:

  1. Try to fix the router configuration (depending on your level of geekitude, you could try SSHing into the router and seeing what the hell they did to the iptables; or just poking around in their web interface, especially around things that might be named something like "Port forwarding", "NAT" or "DMZ")
  2. Make it so that the domain name of your AGH server resolves to an internal address from the internal network. You do have some sort of local plain DNS server as a bootstrap, right? It's probably even the AGH itself, in which case you could just write a rule to resolve dandelionsprout.asuscomm.com to 192.168.1.168 for internal clients to avoid all this routing strangeness.

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 23, 2021

My router is an ASUS RT-AC68U. It's becoming pretty old (2013 model), although it has almost all of the functions of more recent ASUS routers. I've telnet-ed into it on one previous occasion, but it's difficult to maneuver in.

As for the two suggestions, it should be decently possible for me to look into those various re-routing methods within 48 hours, possibly also including the Linux server's /etc/hosts file.

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 23, 2021

After 1½ hours, I've found out the underlying reason for this problem, which in turn lacks a solution:

The AdGuard Home alpha version (v0.107.0-a.136+550b1798) is lying about listening to port 48582 on LAN:
image

This can be confirmed when I telnet that port, which gives a connection error.

@ameshkov
Copy link
Member

ameshkov commented Aug 23, 2021

@DandelionSprout QUIC works over UDP and telnet tries to connect using TCP.

netstat command on your screenshot does not print those that listen to UDP at all due to grep LISTEN probably.

@DandelionSprout
Copy link
Member Author

Ah, okay. Now I see it.
image

So now I'm in a bit of a problem. I genuinely can't figure out what the problem is, after trying some 60 different things, and I've had a fever for 4 days now on top of that.

@ngorskikh
Copy link
Member

ngorskikh commented Aug 24, 2021

What about just resolving your domain to a private address from the private network? That would solve your routing issues.
To be clear, here's how that would work:

  1. Configure AGH to listen for plain DNS (maybe only on private network)
  2. Configure DHCP on your router to advertise AGH as the DNS server
  3. Add a client tag in AGH for your private network (e.g. 192.168.0.0/16 or 192.168.1.0/24)
  4. Add a rule like your.domain.com$client=YourClientNetTag,dnsrewrite=192.168.1.168
    This should work assuming the clients use the DHCP DNS settings to bootstrap the QUIC resolver (which they should do, but in AdGuard apps the bootstrap is also configurable).

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 24, 2021

Trying steps 3 and 4 while skipping steps 1 and 2, through dandelionsprout.asuscomm.com^$dnsrewrite=192.168.1.168,client='Imres Moto One Zoom' seemed to work, but with the fairly major drawback that it causes DoH/DoT connection errors to my secondary server at 192.168.1.188.

I guess I'll have to try to follow steps 1 and 2 as well.

Update 19:37 UTC: Turns out that my ASUS router is unable to use a LAN AGH server as both of its IPv4 DNS servers, as it'll for some reason cut the internet connection.

Additionally, trying to use two $dnsrewrite entries doesn't work, since AdGuard for Android insist on getting a working connection from both IPs for all connections.

dandelionsprout.asuscomm.com$dnsrewrite=192.168.1.168,client='Imres Moto One Zoom',dnstype=A
dandelionsprout.asuscomm.com$dnsrewrite=192.168.1.188,client='Imres Moto One Zoom',dnstype=A

@DandelionSprout
Copy link
Member Author

DandelionSprout commented Aug 24, 2021

I was at least finally able to find the router's netstat table, as was requested:
image

@ngorskikh
Copy link
Member

Regarding

Update 19:37 UTC: Turns out that my ASUS router is unable to use a LAN AGH server as both of its IPv4 DNS servers, as it'll for some reason cut the internet connection.

, if your AGH server is configured to use a non-plain upstream, and it gets it's DNS settings from DHCP, too, this would lead to an infinite loop as AGH would try to bootstrap it's upstream using itself :) Just configure your AGH to use any public DNS of your choosing instead of those from DHCP.

@ameshkov ameshkov added question and removed waiting for data Waiting for users to provide more data. labels Aug 25, 2021
@ameshkov
Copy link
Member

Closing this since this is not a bug

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants