nginx proxy returns 403: Forbidden

[llamafilm]

Describe the issue you are experiencing

When I try to connect to the nginx HTTPS proxy I get 403: Forbidden. This was working fine in the past, and I'm not sure what changed. My SSL certificate is valid. I'm getting the same issue in Chrome, Safari, and curl.
Nginx error log doesn't show anything, and neither does HA core log. If I bypass the proxy and use HTTP on port 8123 it works fine.

Addon configuration is like this:

domain: home.llamafilm.com
hsts: max-age=31536000; includeSubDomains
certfile: fullchain.pem
keyfile: privkey.pem
cloudflare: false
customize:
  active: false
  default: nginx_proxy_default*.conf
  servers: nginx_proxy/*.conf

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

NGINX Home Assistant SSL proxy

What is the version of the add-on?

3.5.0

Steps to reproduce the issue

% curl -v https://home.llamafilm.com:43443 -L
*   Trying 47.158.165.69:43443...
* Connected to home.llamafilm.com (47.158.165.69) port 43443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=home.llamafilm.com
*  start date: Jul 16 11:00:45 2023 GMT
*  expire date: Oct 14 11:00:44 2023 GMT
*  subjectAltName: host "home.llamafilm.com" matched cert's "home.llamafilm.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: home.llamafilm.com:43443]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15480c600)
> GET / HTTP/2
> Host: home.llamafilm.com:43443
> user-agent: curl/7.84.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 403 
< server: nginx
< date: Sat, 22 Jul 2023 20:33:38 GMT
< content-type: text/plain; charset=utf-8
< content-length: 14
< strict-transport-security: max-age=31536000; includeSubDomains
< 
* Connection #0 to host home.llamafilm.com left intact
403: Forbidden%

System Health information

System Information

version	core-2023.7.2
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.11.4
os_name	Linux
os_version	6.1.21-v8
arch	aarch64
timezone	America/Los_Angeles
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	4999
Installed Version	1.31.0
Stage	running
Available Repositories	1284
Downloaded Repositories	8

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 10.3
update_channel	stable
supervisor_version	supervisor-2023.07.1
agent_version	1.5.1
docker_version	23.0.6
disk_total	57.8 GB
disk_used	51.2 GB
healthy	true
supported	true
board	rpi4-64
supervisor_api	ok
version_api	ok
installed_addons	Terminal & SSH (9.7.1), Studio Code Server (5.9.0), Let's Encrypt (4.12.9), NGINX Home Assistant SSL proxy (3.5.0), ArgonOne Active Cooling (29c), HassOS SSH port 22222 Configurator (0.9.1a), Samba share (10.0.2), UniFi Network Application (2.7.0), Ring-MQTT with Video Streaming (5.5.0), Mosquitto broker (6.2.1)

Dashboards

dashboards	4
resources	2
views	14
mode	storage

Recorder

oldest_recorder_run	July 17, 2023 at 2:08 AM
current_recorder_run	July 21, 2023 at 7:44 PM
estimated_db_size	188.90 MiB
database_engine	sqlite
database_version	3.41.2

Anything in the Supervisor logs that might be useful for us?

No response

Anything in the add-on logs that might be useful for us?

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service crond: starting
s6-rc: info: service crond successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[19:49:01] INFO: Running nginx...

Additional information

I tried reproducing the issue using a separate Docker container with nginx:latest and the same nginx.conf file, just modifying the proxy_pass URL. That works fine, no problems.

[llamafilm]

The problem persists after upgrading core to 2023.7.3.

[llamafilm]

I'm trying to understand why the error_log (set to stdout) doesn't log anything for these requests.
I tried setting nginx to debug level inside the http block by setting active=true and then adding to /share/nginx_proxy/custom.conf:

error_log logs/custom_error.log debug;

That directive just creates an empty custom_error.log file.

I also tried setting HA debug level and that doesn't log anything either.

logger:
  logs:
    homeassistant.core: debug
    http: debug
    homeassistant.components.http: debug

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.