HTTP 500 Errors Due to Missing Table in Database

[miwurster]

Our system is deployed on a Kubernetes cluster, and we have been encountering a puzzling issue where certain requests to the backend are failing with HTTP 500 (Internal Server Error) responses. After investigating the logs, we found that the error is related to a missing table named "events_v2" in our database. Strangely, the table appears to be available and accessible when we directly inspect the database.

Here is the error from the logs:

analytics Request: GET /sites
analytics ** (exit) an exception was raised:
analytics     ** (Ch.Error) Code: 60. DB::Exception: Table plausible_events_db.events_v2 doesn't exist. (UNKNOWN_TABLE) (version 23.6.2.18 (official build))
analytics 
analytics         (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
analytics         (ecto_ch 0.1.9) lib/ecto/adapters/clickhouse.ex:269: Ecto.Adapters.ClickHouse.execute/5
analytics         (ecto 3.9.5) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
analytics         (ecto 3.9.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
analytics         (plausible 0.0.1) lib/plausible/stats/clickhouse.ex:202: Plausible.Stats.Clickhouse.last_24h_visitors/1
analytics         (plausible 0.0.1) lib/plausible_web/controllers/site_controller.ex:41: PlausibleWeb.SiteController.index/2
analytics         (plausible 0.0.1) lib/plausible_web/controllers/site_controller.ex:1: PlausibleWeb.SiteController.action/2
analytics         (plausible 0.0.1) lib/plausible_web/controllers/site_controller.ex:1: PlausibleWeb.SiteController.phoenix_controller_pipeline/2
analytics 18:46:27.696 [error] #PID<0.6140.0> running PlausibleWeb.Endpoint (connection #PID<0.5850.0>, stream id 20) terminated

The strange thing is, not every request (e.g., /site etc.) is failing; it just occurs occasionally. Here is also a screenshot from my browser's network tab:

I really appreciate any tip or help. Thanks!

[ruslandoga]

👋 @miwurster

Maybe you have multiple pods running Plausible (or ClickHouse) and some have different state than the others?

[miwurster]

Yes you are right, I used the Bitnami Clickhouse Helm chart and they spin up a cluster with two shards and three replicas (including zookeeper). I configured to run only one replica w/o sharding and zookeeper - which is fine for now to explore/assess Plausible as a possible solution. It works now.

[dustinmoris]

Did you work out why it didn't replicate correctly across your replicas/shards? I had a cluster upgrade and after that none of my shards has the events_v2 table anymore and now my Plausible installation is broken and I'm wondering if we are running into the same underlying issue here perhaps?

[miwurster]

No I haven't investigated it further.