How can we test in continuous the semantic layer?

[mchl-schrdng]

Utilizing a semantic layer for large language model (LLM) usage is highly beneficial. However, what if there's an error in the database? What if the schema is not synchronized with the semantic definition? One straightforward solution could be to employ the semantic layer to execute a generic SQL statement with a limit of zero, thereby incurring no cost. This approach would help identify any errors in the layer. In other tools like Looker, this test is run during pull requests or on a daily basis to detect any mismatches between the semantic layer and the underlying database. It would be advantageous to have this feature integrated into the LLM system as well.

[wwwy3y3]

Hi @mchl-schrdng thanks for opening this discussion and sharing your opinions with us!

Utilizing a semantic layer for large language model (LLM) usage is highly beneficial. However, what if there's an error in the database? What if the schema is not synchronized with the semantic definition? One straightforward solution could be to employ the semantic layer to execute a generic SQL statement with a limit of zero, thereby incurring no cost. This approach would help identify any errors in the layer.

We're indeed working on a feature that would scan schema conflicts between semantic definitions and the data source. The scan could be triggered manually by user or by background process on daily basis. We're working on it in this PR: #339.

The idea is to check if any tables/columns used in semantic definitions got deleted or altered and show warnings on our UI.

In other tools like Looker, this test is run during pull requests or on a daily basis to detect any mismatches between the semantic layer and the underlying database.

Didn't know Looker does that. Thanks for sharing!

Feel free to share more with us.

[wwwy3y3]

Really detailed sharing! thanks @mchl-schrdng. We'll see if we could get some inspirations from Spectacles

[wwwy3y3]

For data warehouse like bigquery, even if you run limit 0 or 1, I think there's still cost behind that. https://stackoverflow.com/a/63174099/1191953

We're thinking of running statements like describe to validate if the statements are valid.

[mchl-schrdng]

With a limit 1, yes you are still scanning.
But with limit 0, you don't (cf screenshot).

[wwwy3y3]

@mchl-schrdng I see. Then, it might be a good way to validate the statement! thanks for the screenshot!

[wwwy3y3]

cc @cyyeh @goldmedal @onlyjackfrost