ffsync: Update to syncstorage-rs

[mreid-tt]

Description

This PR migrates the existing Firefox Sync Server (syncserver) to Mozilla Sync Storage (syncstorage-rs) built with Rust. This build will replace the package and all earlier versions which do not currently work will be removed.

Fixes #5939
Closes #2093
Closes #2747
Closes #2748
Closes #2816
Closes #2828
Closes #2873
Closes #2885
Closes #2979
Closes #2985
Closes #3124
Supersedes #3849

Download note

When obtaining a build from the 'Checks' tab, be aware that it will encompass builds for two other packages impacted by this pull request (PR). This is standard behaviour for build dependencies, and you can safely disregard these supplementary files.

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[th0ma7]

I suggest you bring back discussion here when being PR specific (#5939 (comment))

I'll have to do a test run of my own to try to capture where the issue is. Something else must be missin.

[mreid-tt]

I suggest you bring back discussion here when being PR specific (#5939 (comment))

Thanks @th0ma7, I'll continue discussions here. I've been looking more into the current error being experienced which is:

error: failed to run custom build command for `pyo3 v0.14.5`

Caused by:
  process didn't exit successfully: `/github/workspace/spk/ffsync/work-[build-type]/syncstorage-rs-0.14.1/target/release/build/pyo3-0a9fa2667963a478/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=_PYTHON_SYSCONFIGDATA_NAME

  --- stderr
  error: Could not find either libpython.so or _sysconfigdata*.py in /github/workspace/spk/ffsync/work-[build-type]/install/var/packages/ffsync/target/lib/

Based on the above you mentioned that I may be missing:

# Mandatory for rustc wheel building
ENV += PYO3_CROSS_LIB_DIR=$(STAGING_INSTALL_PREFIX)/lib/
ENV += PYO3_CROSS_INCLUDE_DIR=$(STAGING_INSTALL_PREFIX)/include/

Adding this either to the spk or the cross folder Makefiles does not resolve the issue. Following this I searched for where else in the codebase may have this implemented. I found it in spk/homeassistant/Makefile and spk/borgbackup/src/requirements-crossenv.txt. However, upon examining the build logs for the most recent pull requests in the respective packages, I did not see any lines suggesting that pyo3 was actually compiled.

These lines are of course present in the cross/python311/Makefile file and other files for python311. Within the last PR's build log I saw evidence of successful compile actions for pyo3. Now python is a huge package so I am not sure what in the Makefile (or requirements files) allows the compilation to succeed. Let me know if this sparks any ideas based on your experience.

[hgy59]

@mreid-tt with fixed python311 dependencies I get a different error:

error: the configured Python interpreter version (3.1) is lower than PyO3's minimum supported version (3.6)

So it looks like PyO3 is too old and cuts off the latest digit of 3.11 => 3.1 (PyO3 supports python >= 3.6)

[hgy59]

So it looks like PyO3 is too old and cuts off the latest digit of 3.11 => 3.1 (PyO3 supports python >= 3.6)

indeed, PyO3 v0.14.5 was released one month before Python 3.10 (the first release with two digit minor version).

[mreid-tt]

@hgy59, I have created two issues in the upstream repository based on our experiences with this PR:

#1502: Compilation issues with version of PyO3
#1503: Revising version of cryptography in requirements

Do let me know if there are any other issues with the package before merging. Also in our wiki for the Makefile variables, I propose we add the following lines to the table:

Variable name	Explanation	Example
CARGO_BUILD_ARGS	When building a Rust binary, you can add additional installation options. Specify each option in a separate statement. This must not include the --path option (see RUST_SRC_DIR)	CARGO_BUILD_ARGS = --no-default-features CARGO_BUILD_ARGS += --locked
RUST_SRC_DIR	Configures a custom --path parameter for building a Rust binary. Optional, default is RUST_SRC_DIR = $(WORK_DIR)/$(PKG_DIR)	RUST_SRC_DIR = $(WORK_DIR)/$(PKG_DIR)/custom

I am not sure which section they should be added to (cross/ Makefiles?), so your guidance would be appreciated.

[hgy59]

I am not sure which section they should be added to (cross/ Makefiles?), so your guidance would be appreciated.

Yes, this goes to the cross/Makefile section.

[hgy59]

Finally the naming should be consistently "Mozilla Sync Storage" instead of "Firefox Sync Server" (package description, wizard pages, et. al. )

[mreid-tt]

Finally the naming should be consistently "Mozilla Sync Storage" instead of "Firefox Sync Server" (package description, wizard pages, et. al. )

I'm not sure this is a change we should make. Looking at the Mozilla Services Documentation, I see that there are a few services including:

• Mozilla Accounts Server
• Token Server
• Storage Service

Additionally, in the same documentation it describes the topic further:

The Sync server infrastructure exposes a secure HTTP interface for user management and node assignment as well as storage. The storage server is actually a generic service that isn’t Sync-specific.

So, my understanding is that sync server (in this rust implementation) is a combination of a token server and a storage service, so calling the whole the name of one part seems incorrect. Additionally, the main documentation page for setting one up describes it as a Sync-1.5 Server (as well as Firefox Sync server).

Based on the above, to me the most accurate name would be Mozilla (or Firefox) Sync-1.5 Server. As such, I would propose we keep the naming as-is.

[hgy59]

Ok name it "Mozilla Sync Server" and leave the 1.5.

[mreid-tt]

Ok name it "Mozilla Sync Server" and leave the 1.5.

@hgy59, I have revised the package naming and the wizards to "Mozilla Sync Server" and removed the 1.5 from the name. I also updated the package description to use the wording from the upstream website. I found that the description is a bit tricky (must escape special characters, but not enclose in double-quotes).

I believe we are ready for final approval.

[hgy59]

LGTM

[mreid-tt]

@hgy59, thanks for the green light. For publishing, based on the changes which affect other packages, can this be done via GitHub or do you have to publish it manually?

[th0ma7]

Yes it can, look in the wiki online all the howto is the well written by @publicarray

[th0ma7]

And forgot to mention, good job guys!

[mreid-tt]

Yes it can, look in the wiki online all the howto is the well written by @publicarray

OK, I've started a build action so we'll see what happens.

EDIT: I ran it and it was successful and even though the package generated contained both a build of ffsync and python311, only the ffsync seems to be on the server at https://synocommunity.com/package/ffsync. So I guess it works!