Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: suggest target_session_attrs=read-write when using HA and psql #126

Closed
desimone opened this issue Oct 7, 2022 · 0 comments · Fixed by #140
Closed

docs: suggest target_session_attrs=read-write when using HA and psql #126

desimone opened this issue Oct 7, 2022 · 0 comments · Fixed by #140
Assignees
Labels

Comments

@desimone
Copy link
Collaborator

desimone commented Oct 7, 2022

    I deployed [git-3b2cc672](https://hub.docker.com/layers/pomerium/pomerium/git-3b2cc672/images/sha256-45f01a6e2777a902c78aba439185e0ffcb30d12ff0e058e4ddf1b38b58a2622d?context=explore) today.  Things looked good - all 3 PSQL nodes came online (using basically the config I posted), pomerium connected to the PSQL backend, and it served traffic.  I successfully took down the primary PSQL node and the databroker survived, switching over to the new primary.
## WORKING
# Use HA PSQL databroker backend
databroker_storage_type: postgres
databroker_storage_connection_string: postgres:https://pomerium:pomerium@pg-0,pg-1,pg-2/pomerium?sslmode=disable&target_session_attrs=read-write

===

However, a word of caution! target_session_attrs=read-write is absolutely critical. Without it libpq correctly switches to a second node, but has a chance to end up on a read-only replica. When I tested this, the constant write attempts eventually took down repmgr and even the entire PSQL container.

{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2022-10-04T13:08:04-05:00","message":"initial sync"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","time":"2022-10-04T13:08:04-05:00","message":"sync latest"}
{"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = ERROR: cannot execute CREATE SCHEMA in a read-only transaction (SQLSTATE 
## BROKEN
# Use HA PSQL databroker backend
databroker_storage_type: postgres
databroker_storage_connection_string: postgres:https://customuser:custompassword@pg-0,pg-1,pg-2/pomerium?sslmode=disable

I'm not sure how the databroker works under the hood; perhaps on large installs it can make use of read-only replicas for load balancing? However, attempting to write on a read-only connection makes no sense to me. Arguably this behavior should be changed, or at the least the docs can be clarified in the docs to stress using target_session_attrs=read-write when using HA.

Originally posted by @alexrudd2 in pomerium/pomerium#3634 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants