Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server) user-defined storage structure #1098

Merged
merged 41 commits into from
Dec 16, 2022

Conversation

alextran1502
Copy link
Contributor

@alextran1502 alextran1502 commented Dec 12, 2022

This PR implements a way to let the user define a storage structure for the original files uploaded to the Immich server under a template builder option on the web.

Builder interface

image

Resulting in storage location with auto handling duplicated filename
image

@alextran1502 alextran1502 linked an issue Dec 12, 2022 that may be closed by this pull request
@vercel
Copy link

vercel bot commented Dec 12, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated
immich ✅ Ready (Inspect) Visit Preview Dec 16, 2022 at 8:22PM (UTC)

@alextran1502 alextran1502 linked an issue Dec 12, 2022 that may be closed by this pull request
Copy link
Member

@bo0tzz bo0tzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff! Mainly high level from me right now, about interfaces and such.

I think the IImmichStorage implementations should take all their config (eg s3 credentials, path templates, etc) in their constructor. That way in the future we can easily have multiple instances of the storage implementations for things like separate thumbnail storage, storage migrations, etc.
To align with that, we probably want separate functions on the StorageService for originals vs. thumbnails, etc. Or maybe the functions can take a Type enum for the type of file:

enum Type {
  OriginalPhoto,
  OriginalVideo,
  Thumbnail,
  TranscodedVideo,
  Etc
}

@@ -0,0 +1,4 @@
export interface IImmichStorage {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will probably need a few more functions, like deleting, checking whether a file exists, and maybe getting metadata (eg size).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And all the functions in this interface should take some object with asset metadata that they can use to find the file in their backend.
Some potential fields:

  • assetId
  • userId
  • assetType
  • originalFileName
  • createdAt
  • checksum

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like we're using this interface in this PR, can we remove it and related artifacts?

@jrasm91
Copy link
Contributor

jrasm91 commented Dec 12, 2022

In order to test the Storage service are you going to have to mock every single storage backend? That doesn't seem right.

@alextran1502
Copy link
Contributor Author

@jrasm91 What would be the best way to encapsulate this? I am laying out the ground work here so we can dissect the structure

@bo0tzz
Copy link
Member

bo0tzz commented Dec 14, 2022

If we're just refactoring with the legacy storage approach for now, I think we can potentially leave all the config stuff for later?

@alextran1502
Copy link
Contributor Author

@bo0tzz Yes, I am not planning to integrate the full storage provider configuration in this PR. I might abstract some of the legacy's calls away but that will be it.

@alextran1502 alextran1502 marked this pull request as ready for review December 16, 2022 20:24
@alextran1502 alextran1502 merged commit c754c86 into main Dec 16, 2022
@alextran1502 alextran1502 deleted the feat/user-defined-storage-structure branch December 16, 2022 20:26
@klaus1k
Copy link

klaus1k commented Dec 19, 2022

This looks absolutely awesome, thanks a lot!

Is it possible to have the user directory named with a "human readable" user name instead of user id?
That would be the cherry on the cake! :)

@jrasm91
Copy link
Contributor

jrasm91 commented Dec 19, 2022

Maybe, but there are some security implications with that part of the path, as well as uniqueness checks. So we didn't try changing it in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Server: Storage layer refactor Support user defined storage location/structure
5 participants