Missing database column when using latest version

[spapas]

I'm using the latest version (from github) and getting the following error:

14:18:33.849 [error] GenServer Plausible.Event.WriteBuffer terminating
** (Ch.Error) Code: 16. DB::Exception: No such column revenue_reporting_amount in table plausible_events_db.events_v2. (NO_SUCH_COLUMN_IN_TABLE) (version 23.4.2.11 (official build))

    (ecto_sql 3.10.1) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_ch 0.1.11) lib/ecto/adapters/clickhouse/schema.ex:33: Ecto.Adapters.ClickHouse.Schema.insert_all/8
    (ecto 3.10.3) lib/ecto/repo/schema.ex:59: Ecto.Repo.Schema.do_insert_all/7
    (stdlib 4.0.1) gen_server.erl:1158: :gen_server.try_terminate/3
    (stdlib 4.0.1) gen_server.erl:1348: :gen_server.terminate/10
    (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :tick

I've followed the data migration instructions from #2981
and run normal with mix ecto.migrate .

The events_v2 is like this:

appproxy.hcg.gr :) describe table events_v2;

DESCRIBE TABLE events_v2

Query id: 31a723b3-e716-4fbd-aaee-c11553b6e060

┌─name─────────────────────┬─type───────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│ timestamp                │ DateTime               │              │                    │         │ Delta(4), LZ4    │                │
│ name                     │ LowCardinality(String) │              │                    │         │                  │                │
│ site_id                  │ UInt64                 │              │                    │         │                  │                │
│ user_id                  │ UInt64                 │              │                    │         │                  │                │
│ session_id               │ UInt64                 │              │                    │         │                  │                │
│ hostname                 │ String                 │              │                    │         │                  │                │
│ pathname                 │ String                 │              │                    │         │ ZSTD(3)          │                │
│ referrer                 │ String                 │              │                    │         │                  │                │
│ referrer_source          │ String                 │              │                    │         │                  │                │
│ country_code             │ FixedString(2)         │              │                    │         │                  │                │
│ screen_size              │ LowCardinality(String) │              │                    │         │                  │                │
│ operating_system         │ LowCardinality(String) │              │                    │         │                  │                │
│ browser                  │ LowCardinality(String) │              │                    │         │                  │                │
│ utm_medium               │ String                 │              │                    │         │                  │                │
│ utm_source               │ String                 │              │                    │         │                  │                │
│ utm_campaign             │ String                 │              │                    │         │                  │                │
│ meta.key                 │ Array(String)          │              │                    │         │                  │                │
│ meta.value               │ Array(String)          │              │                    │         │                  │                │
│ browser_version          │ LowCardinality(String) │              │                    │         │                  │                │
│ operating_system_version │ LowCardinality(String) │              │                    │         │                  │                │
│ subdivision1_code        │ LowCardinality(String) │              │                    │         │                  │                │
│ subdivision2_code        │ LowCardinality(String) │              │                    │         │                  │                │
│ city_geoname_id          │ UInt32                 │              │                    │         │                  │                │
│ utm_content              │ String                 │              │                    │         │                  │                │
│ utm_term                 │ String                 │              │                    │         │                  │                │
│ transferred_from         │ String                 │              │                    │         │                  │                │
└──────────────────────────┴────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

[ruslandoga]

👋 @spapas

What do you mean by the latest version from github?

[spapas]

This is the version:

commit eeca63ff4c66500cc0e666d4cedfcaf48cd35c9c (HEAD -> master, tag: r2023.08.07-163, origin/master, origin/HEAD)
Author: RobertJoonas <[email protected]>
Date:   Mon Aug 7 18:11:14 2023 +0100

    Some fixes to the Custom Props feature (#3241)

    * rename Properties to Custom Properties in Site Settings

    * fix docs link for custom props

    * add back the external links for special goal URLs and 404 paths

    * grandfathering note

[ruslandoga]

You'd need to run more migrations to make the current master work. It hasn't been released yet.

[ruslandoga]

The current latest release is v2.0.0 #3132

[spapas]

I'm running all migrations on master:

[serafeim@imp plausible]$ mix ecto.migrate

14:46:53.307 [info] Migrations already up

14:46:53.386 [info] Migrations already up

Migrations are applied to my clickhouse db however the tables are broken. Could this be because I first run the migrations and then I followd the upgrade instructions to run the data ugprade for 2.0 ?

[spapas]

I had to actually add the columns by hand:

alter table events_v2 add column revenue_reporting_amount Nullable(Decimal64(3));
alter table events_v2 add column revenue_reporting_currency FixedString(3);
alter table events_v2 add column revenue_source_amount Nullable(Decimal64(3));
alter table events_v2 add column revenue_source_currency FixedString(3);

As I said in my previous comment it seems that applying the migrations firsT and then trying to run the data migrations results in breaking stuff. Maybe you'll need to fix the data migrations so they create the proper tables ?

[ruslandoga]

Good point, we'll need to keep that in mind when releasing the next version. Thank you!