Filestore Improvements

text/0108-filestore-new.md

@@ -0,0 +1,95 @@
+- Start Date: 2023-07-13
+- RFC Type: informational
+- RFC PR: https://github.com/getsentry/rfcs/pull/108
+- RFC Status: draft
+
+# Summary
+
+One of the systems that Sentry internally operates today is an abstract concept referred
+to as "file store". It consists of a postgres level infrastructure to refer to blobs and
+a go service also called "file store" which acts as a stateful proxy in front of GCS to
+deal with latency spikes, write throughput and caching.
+
+This RFC summarizes issues with the current approach, the changed requirements that go
+into this system and proposes a path forward.
+
+# Motivation
+
+Various issues have ocurred over the years with this system so that some decisions were
+made that over time have resulted in new requirements for filestore and alternative
+implementations. Replay for instance operates a seperate infrastructure that goes
+straight to GCS but is running into write throughput issues that file store the Go service
+solves. On the other hand race conditions and complex blob book-keeping in Sentry itself
+prevent expiring of debug files and source maps after a period of time.
+
+The motivation of this RFC is to summarize the current state of affairs, work streams that
+are currently planned are are in motion to come to a better conclusion about what should be
+done with the internal abstractions and how they should be used.
+
+# Background
+
+The primary internal abstraction in Sentry today is the `filestore` service which itself
+is built on top of Django's `files` system. At this level "files" have names and they
+are stored on a specific GCS bucket (or an alternative backend). On top of that the `files`
+models are built. There each file is created out of blobs where each blob is stored
+(deduplicated) just once in the backend of `filestore`.
+
+For this purpose each blob is given a unique filename (a UUID). Blobs are deduplicated
+by content hash and only stored once. This causes some challenge to the system as it
+means that the deletion of blobs has to be driven by the system as auto-expiration is
+thus no longer possible.
+
+# Supporting Data
+
+We currently store petabytes of file assets we would like to delete.
+
+# Possible Changes
+
+These are some plans about what can be done to improve the system:
+
+## Removal of Blob Deduplication
+
+Today it's not possible for us to use GCS side expiration. That's because without the
+knowledge of the usage of blobs from the database it's not save to delete blobs. This
+can be resolved by removing deduplication. Blobs thus would be written more than once.
+This works on the `filestore` level, but it does not work on the `FileBlob` level.
+However `FileBlob` itself is rather well abstracted away from most users. A new model
+could be added to replace the one one. One area where `FileBlob` leaks out is the
+data export system which would need to be considered.
+
+`FileBlobOwner` itself could be fully removed, same with `FileBlobIndex` as once
+deduplication is removed the need of the owner info no longer exists, and the index
+info itself can be stored on the blob itself.
+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)
+ path = TextField(null=True)
+ offset = BoundedPositiveIntegerField()
+ size = BoundedPositiveIntegerField()
+ checksum = CharField(max_length=40, unique=True)
+ timestamp = DateTimeField(default=timezone.now, db_index=True)
+```
+
+## TTL Awareness
+
+The abstractions in place today do have any support for storage classes. Once however
+blobs are deduplicated it would be possible to fully rely on GCS to clean up on it's own.
+Because certain operations are going via our filestore proxy service, it would be preferrable
+if the policies were encoded into the URL in one form or another.
+
+## Assemble Staging Area
+
+The chunk upload today depends on the ability to place blob by blob somewhere. Once blobs are
+stored regularly in GCS there is no significant reason to slice them up into small pieces as
+range requests are possible. This means that the assembly of the file needs to be reconsidered.
+
+The easiest solution here would be to allow chunks to be uploaded to a per-org staging area where
+they linger for up to two hours per blob. That gives plenty of time to use these blobs for
+assembly. A cleanup job (or TTL policy if placed in GCS) would then collect the leftovers
+automatically. This also detaches the coupling of external blob sizes from internal blob
+storage which gives us the ability to change blob sizes as we see fit.
+
+# Unresolved questions
+
+TBD