Adopt fireo transactional for upload safety

[evamaxfield]

Use Case

Please provide a use case to help us understand your request in context

Further ensure that no "bad" or "incomplete" data is uploaded to a CDP database.

Solution

Please describe your ideal solution

Rework the event gather pipeline to create the fireo models from the ingestion models but do not upload until the very end of the pipeline where we can upload all of them in a single fireo transaction.

Alternatives

Please describe any alternatives you've considered, even if you've dismissed them

[evamaxfield]

Fleshing this out a little more:

I don't think we can build a transaction throughout the flow for multiple reasons

1. I don't think a multi function transaction is possible with the fireo API
2. I don't know if fireo models are serializable and all prefect / parallel things need to be pickleable

So it would more so be: do all the processing that we need to do up front and then have one large upload task at the end that is itself a large transaction

[evamaxfield]

Closing for now. We can revisit at a later point but many of our database models rely on each other and it would be hard to upload an event. We would have to do it in multiple transcactions if we wanted to at.