Big thanks to HeyItzCarlos194 for figuring out how to make this actually work
This is a setup "template" for a Minecraft skin server hosted right here on Github.
It uses the OfflineSkins mod for Fabric, which by default grab skins from Crafatar or a local folder but can also be configured to use practically any HTTP server.
To use this, clone the repository to your own account, and upload your skins (and capes!) named by your username, both with the .png extension and without. Next change the URL username & repo in the offlineskins.json config file to your own fork, install the mod and put the config file into your .minecraft/config.
No setup on your offline Minecraft server is required whatsoever.