If you are not sure whether this project fits your need, then it does not. You are probably looking for OneDrive Online or sync and FUSE implementations like rclone.
This project may be helpful for 🐧 real nerds 🐧 who enjoy wacky block device stacking, intend to leverage block level encryption or their existing BTRFS backup infrastructure, or explore fresh new bugs in BTRFS zoned mode, with the cost of everything.
System requirements:
-
Linux >= 5.19 is required for io-uring with
IORING_SETUP_SQE128
support. -
Kernel driver
ublk_drv
and zoned block device support should be enabled. Most distributions like Arch Linux and NixOS unstable meet these requirements by default. You can check your system by:$ zgrep -E 'CONFIG_BLK_DEV_UBLK|CONFIG_BLK_DEV_ZONED' /proc/config.gz CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEV_UBLK=m
If you see the same result, your kernel is probably supported.
-
You may need to run
sudo modprobe ublk_drv
manually to load the driver first. This is not required for running orb in the shipped systemd service or via NixOS module, which does this automatically.
This project is packaged in Nix flake. Here's the simplified output graph:
├───nixosModules
│ ├───default: Alias to `orb`.
│ └───orb: The NixOS module.
└───packages
├───x86_64-linux
│ ├───default: Alias to `orb`.
│ ├───orb: The main program with systemd units.
│ ├───cryptsetup-format-zoned: workaround script for cryptsetup-luksFormat on zoned devices.
│ └───ublk-chown-unprivileged: The optional utility for unprivileged ublk.
[..more Linux platforms are supported..]
Example configurations
To use the orb service, add the flake input github:oxalica/orb
, and import
its NixOS modules.
# Example flake.nix for demostration. Please edit your own one to add changes.
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.orb.url = "github:oxalica/orb";
outputs = { nixpkgs, orb, ... }: {
nixosConfigurations.your-system = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = with nixosModules; [
orb.nixosModules.orb
./path/to/your/configuration.nix
];
};
};
}
Now you can use the module in your configuration.nix
:
{ ... }:
{
services.orb.instances = {
# The instance name. It coresponds to the systemd service
# `[email protected]`. By default it will not be automatically started.
"my-device".settings = {
# Required device id. It's recommended to start at 80.
# This creates block device `/dev/ublkb80`.
ublk.id = 80;
# Other settings and their defaults can be seen in
# ./contrib/config-onedrive.example.toml
device = {
dev_size = "1TiB";
zone_size = "256MiB";
min_chunk_size = "1MiB";
max_chunk_size = "256MiB";
};
backend.onedrive.remote_dir = "/orb";
};
};
# If you want to mount the block device, you can create systemd mounts.
# This is an example.
systemd.mounts = [
{
type = "btrfs";
# Fill in your filesystem UUID after mkfs.
what = "/dev/disk/by-uuid/11111111-2222-3333-4444-555555555555";
where = "/mnt/my-mount-point";
# Do not forget dependencies.
requires = [ "[email protected]" ];
after = [ "[email protected]" ];
# It's recommended to set `noatime` and `compress` to reduce write
# frequency and amplification.
options = "noatime,compress=zstd:7";
}
];
}
Note that the service can only work after login and setup first. See the following sections for details.
You need following dependencies to be installed with your package manager:
- Rust >= 1.76
- pkg-config
- openssl
Build command: cargo build --release
contrib/[email protected]
is the example template systemd service to install.
The instance configurations locate at /etc/orb/<name>.toml
, whose format is
documented in
./contrib/config-onedrive.example.toml
.
Once configured and logined (see the next section), run
systemctl start orb@<name>.service
to start the service.
The service configuration does not contain the login credential. It must be interactively setup for the first time, and then the service will rotate the credentials automatically unless the user revokes the permission, or after a long offline time (seems to be >1month, but is determined by Microsoft).
-
First, you need to know this project (orb) is an third party program which access your files on Microsoft OneDrive on behalf of you, to provide block device interface as a service. Your files and/or data on your Microsoft OneDrive may be lost due to program bugs or other reasons. We provide no warranties. By following the login steps below, you understood and want to use orb at your own risk.
-
We cannot provide an "official App/Client ID" without risking impersonated because this project is open sourced and free to distribute. So you need to register your own App on Microsoft Azure.
In the registration page,
- In "Supported account types" section, select "Personal Microsoft accounts only". Other accounts are currently unsupported.
- In "Redirect URI (optional)" section, select "Public client/native
(mobile & desktop)", and enter the following URI:
It must be this exactly (it's
https://localhost
http
nothttps
), or you may fail the next step.
Then click "Register", it will jump to the registered App information page if success. In "Essential" section, copy the UUID in the "Application (client) ID" field. This is the Client ID to be used in the next step. Note that one App can be used in multiple accounts, for multiple times. You do not need to register more than one App in almost any cases.
-
Login with this command with root permission with arguments filled:
# orb login --systemd <instance> --client-id <the-client-uuid-from-the-last-step>
<instance>
is the instance name of your systemd service (for example, you setup/etc/orb/foo.toml
, thenfoo
is the instance name) or in NixOS module settingservices.orb.instances.<instance>
.It will prompt a URL, and you need to open it in your browser and following the interactive login steps to login into your Microsoft account with OneDrive.
The credential will be saved under
/var/lib/orb/<instance>
, owned by root, and cannot be accessed by non-root users. It will be rotated by the service, and please never copy or save it outside the local machine. If you need to login to the same account on two machines, login twice.⚠️ You must not serve the same remote directory simultaneously in multiple instances (or machines), or it will cause data race and your data will be corrupted. orb will try its best to detect and prevent such racing serving. -
On success, the web page will redirect to a mostly empty page with only one line:
Successfully logined. This page can be closed.
The command should exit normally with credential saved. Now you are ready to start the orb service.
Once your logined and started the service successfully, you are ready to use it. Usually you need to create an filesystem on the emulated block device, and this is almost the same as the setup for your fresh hard disks, with a few exceptions:
-
The emulated device is under
/dev/ublkb<ID>
whereID
is specified in your configurationublk.id
. -
The device is a zoned device (aka. ZBC/ZBD/ZNS, host managed SMR disks) due to API restrictions and performance reasons. Only a few filesystems and/or device mappers support it, eg. dm-crypt, F2FS and BTRFS.
-
It has a high latency and low throughput depend on your network. Doing active works on it should be avoided. It can be used, for example, for backup purpose.
-
⚠️ Since the block device is emulated, you must ensure toumount
the filesystem on it before shutting down the backing device service (orb@<instance>.service
), or you will lose your last written data. This could be enforced by systemd mounts with aBindsTo=
dependency.
Due to the limitation of OneDrive API, permanently deletion cannot be done via API. You may need to regularily "Empty recycle bin" on OneDrive online to free the capacity occupied.
backend.onedrive.remote_dir
). Otherwise, it may break filesystem
consistency and your data may be lost.
Details
cryptsetup does not support formatting zoned devices, but dm-crypt supports it.
We need to format and place the LUKS2 header manually, and then it can be
opened and/or closed in the normal way. For convenience, there is a script
under
./contrib/cryptsetup-format-zoned.sh
to mimic cryptsetup luksFormat
as a workaround. Run:
# ./contrib/cryptsetup-format-zoned.sh /dev/ublkb<ID> # Use a a password.
OR
# ./contrib/cryptsetup-format-zoned.sh /dev/ublkb<ID> /path/to/key/file # Use a key file.
Alternatively, you can run the script via flake package:
$ nix shell github:oxalica/orb#cryptsetup-format-zoned -c sudo cryptsetup-format-zoned /dev/ublkb<ID>
Note that editing header, ie. adding or removing keys, also requires careful manual operations. You need do it yourself when needed.
After formatting the block device, you can open and/or close it in the normal way:
# cryptsetup luksOpen /dev/ublkb<ID> my-device-unencrypted
# cryptsetup close my-device-unencrypted
If you are using key files, you can also use systemd-cryptsetup services to
manage dm-crypt. This is useful when you want to specify dependencies to
orb@<instance>.service
and downstream services, eg. backup services.
{ ... }:
{
environment.etc."crypttab".text = ''
mydecrypteddev /dev/ublkb<ID> /path/to/key/file noauto
'';
systemd.services."systemd-cryptsetup@mydecrypteddev" = {
# Inform Nix that this is an overriding units for auto-generated ones.
overrideStrategy = "asDropin";
# Specify dependencies to the orb service.
bindsTo = [ "[email protected]" ];
after = [ "[email protected]" ];
};
}
Details
It is recommended to format BTRFS with block-group-tree
feature enabled, to
dramastically reduce mounting time (~50s to ~2s). You need btrfs-progs >= 6.8.1
with a relevant bug getting fixed.
# mkfs.btrfs /dev/ublkb<ID> -O block-group-tree
zoned
feature will be automatically detected and enabled without manual
specification.
Now you can mount it and do read/write operations. These are recommended mount options (disable atime, high level zstd compression enabled):
sudo mount -t btrfs -o noatime,compress=zstd:7 /dev/ublkb<ID> /mnt/my-mount-point
The sub-package orb-ublk
and ublk-chown-unprivileged
(directory
/orb-ublk
, /ublk-chown-unprivileged
and the whole sub-tree of them)
are licensed under either of Apache License, Version
2.0 or MIT license at
your option.
The main package (all other files in the repository except content of
/orb-ublk
and/or /ublk-chown-unprivileged
directory) is licensed under
GNU General Public License v3.0 or (at your option) later
versions.