-
-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: Resource management system #357
Comments
I like the design overall. Thanks for starting this! Questions
Thoughts
For things like this, I suspect it is mostly useful in the context of Mach only, and should be deeply integrated. So this can live in a library at I'll have more thoughts on other questions you posed once I learn a bit more from your responses to my questions I think. |
For the sake of explaining, lets just assume that Collection is a single archive. Actually having multiple collections only makes sense in case of archives and thats why its there. But it can be used with local directory based structures. Lets take an example, you have a game and its data is contained in a archive called Do note that you do not need to have multiple collections in your game. You can perfectly store all resources in one collection/archive (but it may not scale well for large application organization). A collection itself internally follow tree structure, so it can have directories inside.
Sounds right. I didnt had any opinion on this so decided its better to ask, just in case. |
The json collection example I initially created didn't take into consideration having multiple collections (in fact it used the word collection for just any folder). So here's an updated one: {
"collections": [
{
"name": "texture_collection",
"tree": [
{
"dir": "player",
"tree": [
"player_standing.png",
"player_running.png"
]
},
"tree.png",
"rock.png"
]
},
{
"name": "audio_collection",
"tree": [
"c.mp3",
{
"dir": "ambient",
"tree": [
"chirping.mp3",
"wind.mp3"
]
}
]
}
]
} Json is not good for manually creating trees. |
OK this makes a lot of sense.
|
|
Collections is the abstraction / part of this proposal I feel least confident about. I think it enforces a certain way of working with your game data that may not be very clear, and the benefits are not always obvious. Being very critical of it, most of what it solves can be resolved in other ways: Modding
I think the best-case scenario for modding would be "Here's my mod directory/archive, it wants to override very specific resources, oh and i might have a few others of those.. just use a.png from one of the mod folders if you find it there first, otherwise fallback to the game's a.png file" Collections don't seem like they would do this at all: they would only let us override specific resources, and only in aggregate. If you wanted to override a few audio files and texture files in a game, for example, you'd need to provide an entirely new ChunkingWe had discussed this benefit of collections:
But, actually I think it's not a big benefit necessarily. A single file can manage all assets in a reasonable way, so long as the file format of that file is reasonable. A good example of this is Guild Wars, where both the original game and newer version 2 game are distributed as a single exe file which downloads a single Excluding of certain files
We could support this with a |
Suggestion: update the proposal with a solidified set of goals and non-goals. We could start with this set: Goals
Non-goals (to be handled at a later date)
|
Suggestion: We could remove the idea of collections entirely, and instead add explicit support to the proposal for exclusion of assets and modding: In order to access a resource, you use a URI instead of a file path:
Depending on where the application is running, and in what mode (release/debug), behavior will differ by default (but you can choose):
Exclusion of assetsGenerally speaking, you put all game assets under a folder called When excluded, they will not end up in the final ModdingTo enable resource modding of Mach games/applications generally, the following will occur: If running natively (not supported in wasm for now), then a
When loading a file, say |
Problem: I think the scoping logic may not be sufficient, OR I don't exactly understand how it should work. I see a few use cases we should support with scopes:
How could the API support all 3? |
Answers:
I agree, not needed. Detecting if a file is binary (and what that actually means) is notoriously difficult/annoying.
We need a way to register We could require that such a function be provided to the I think file extension-based would be bad, because some resources with the same extension need to be interpreted differently (e.g.
This may be quite important to sort out.
Agreed. |
I think we've gotten all of the major discussion points out on the table, so we can do one of two things (whatever you're comfortable with):
I'm OK with either at this point, I don't want to place a burden of writing out more stuff here on you just for the sake of it. |
1 and 2 is solved by the function Do note that pushGroup and popGroup are slightly misleading names as mentioned earlier. 3 ) I dont know if i understand it correctly. Is it about being able to individually load any resource? Well then it will be covered with the low level API. But let's say if the resource is a registered one, maybe we can provide an additional function to directly load with the URI? |
Alright, I agree with everything here
|
The job to convert PNG bytes -> gpu.Texture and so one is performed automatically behind the scenes by the load() function which we provide in ResourceManager.init. What I think can be done here that let say if load() function returns a particular error like |
Yes, and sounds good. The point is just being able to load/free resources manually, without groups (think basically "I want to implement my own grouping logic on top, can I?") |
The problem with this is we can't handle resources in different ways. Let's say my application needs to do two things:
We need some sort of way to support "handle the same resource type in different ways" I think. The problem is more noticeable when you talk about e.g. loading |
Can we use the URI scheme for this, since we aren't using it for anything else right now? Like |
That sounds like a great solution! Then we don't have to rely on extension or In the future (not part of library), we could also allow for loader functions to be defined as part of |
I have some audio-centric questions that I didn't see answered.
|
It's very difficult to ensure stutter-free streaming, since you can't predict where the file comes from. |
Right, those are factors that can affect streaming. Obviously it would be impossible to guarantee stutter-free streaming in those situations. There are plenty of other places where it is possible to stream the audio - consoles, most modern computers, phones (I assume? I actually don't know how fast storage is on mobile), etc. I guess the question will then come down each individual game, and what your target audience is. Do you want to target older computers to increase your possible reach, or are you wanting to push the limits of modern hardware, or something in-between? I mainly wanted to know if that use case had been planned for or bring attention to the possibility if it hadn't. |
Description
General purpose resource management engine which is meant to be cross platform (works on desktop, wasm, android etc.)
The general idea is that in order to access resources, we will use URIs instead of path. All resources will be accessed from the provided API and std.fs.* stuffs should not be used (but the library may use it internally).
The URI will internally evaluate to a path (for desktops) or to an URL (for web). On desktops it will use std.fs APIs and on web it will stream resources which are served on your web server.
Mechanism
Let's say the projects needs a.png, b.png and c.mp3. Then the resources have to be arranged in your directory as such:
This arrangement can be done automatically by some code, which takes a config file (json/yaml/toml) as input and sort the files by their names. The file can look like as such (using json for example):
The resources arrangement/installation can be part of a build step for seamless building. This step may just install or optionally generate an archive file from all the resources.
The resources will be referred inside the application as such:
textures/a.png
,textures/b.png
,audio/c.mp3
. But why notdata/textures/a.png
? (Answered below in API)Remember that the names
textures
,audio
etc above are totally cosmetic and has no significance. It can be arbitrarily named anything. [1]API
The library will provide the following API:
Open Questions
[]const ResourceType
be taken as comptime parameter? as it is known what type of resources the application will use beforehand.Future TODOs
EDIT: Oversights
Note: In all instances where multi-threading is mentioned, its about loading resources in parallel and not about the thread safety of the API.
The text was updated successfully, but these errors were encountered: