You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Basically, the ALTER TABLE statement is missing the array type if the new column is a list - it will create a new column of the base type, i.e.: if you add a column of type list of strings, then peerdb will create a column of type string in BigQuery. This leads to avro errors down the line which look like this:
[...]
failed to sync records:
failed to push to avro stage:
failed to write records to local Avro file:
failed to write records to temporary Avro file:
failed to write records to OCF writer:
failed to write record to OCF:
cannot translate datum to binary:
[...]
cannot encode binary record "<affected table name>" field "<affected column name>":
value does not match its schema:
cannot encode binary union:
no member schema types support datum:
allowed types: [null string]; received: map[string]interface {}
Note: don't be deceived by allowed types: [null string]; received: map[string]interface {} - this is a very misleading error message. The value (and not the type) is important here, which is map[array:[]] and this value is correct: we want to write an empty list. But the avro only accepts nullable strings here since the BigQuery schema is incorrect.
The text was updated successfully, but these errors were encountered:
The background is this code:
Basically, the
ALTER TABLE
statement is missing the array type if the new column is a list - it will create a new column of the base type, i.e.: if you add a column of type list of strings, then peerdb will create a column of type string in BigQuery. This leads to avro errors down the line which look like this:Note: don't be deceived by
allowed types: [null string]; received: map[string]interface {}
- this is a very misleading error message. The value (and not the type) is important here, which ismap[array:[]]
and this value is correct: we want to write an empty list. But the avro only accepts nullable strings here since the BigQuery schema is incorrect.The text was updated successfully, but these errors were encountered: