Could not mirror LVFS repo using Pulp.

[FrancoCorleone]

Hi everyone,
I wonder if there's something missing in my setup, or is the documentation outdated. I tried to setup an LVFS mirror using pulp (in multiprocess container). However, nothing works as expected. I created a pulp rpm remote (I tried urls of the manifest file, base url of the cdn, download url - no success), then rpm repo pointing to that remote,but when I try "sync", I get this error:

Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='NOT FOUND', url=URL('https://cdn.fwupd.org/downloads/repodata/repomd.xml'))
pulp [78d5fff864c14597a95b045cf3bf89cd]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='NOT FOUND', url=URL('https://cdn.fwupd.org/downloads/repodata/repomd.xml'))
pulp [78d5fff864c14597a95b045cf3bf89cd]: pulp_rpm.app.tasks.synchronizing:INFO: Attempting to resolve a true url from potential mirrolist url 'https://cdn.fwupd.org/downloads'
('pulp [78d5fff864c14597a95b045cf3bf89cd]: ::ffff:127.0.0.1 - admin [03/Jun/2024:08:29:48 +0000] "GET /pulp/api/v3/tasks/018fdd38-07c4-7735-bb1c-33ed56cc9fa0/ HTTP/1.0" 200 1049 "-" "Pulp-CLI/0.25.2"',)
pulp [78d5fff864c14597a95b045cf3bf89cd]: pulpcore.tasking.tasks:INFO: Task 018fdd38-07c4-7735-bb1c-33ed56cc9fa0 failed (An invalid remote URL was provided: https://cdn.fwupd.org/downloads)
pulp [78d5fff864c14597a95b045cf3bf89cd]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 70, in _execute_task
    result = func(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 473, in synchronize
    remote_url = fetch_remote_url(remote, url)

  File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 302, in fetch_remote_url
    raise ValueError(_("An invalid remote URL was provided: {}").format(url))

As I said, I tried various URLs and tried to Google the possible issues but nothing works for me.

I'd appreciate any tips or info on how to set it up correctly.

Greetings

[hughsie]

I'm not sure why pulp is trying to download https://cdn.fwupd.org/downloads/repodata/repomd.xml -- it's not listed in https://cdn.fwupd.org/downloads/PULP_MANIFEST -- is this a pulp regression perhaps?

[FrancoCorleone]

@hughsie thanks for a brief response. I honestly have no idea. Never used Pulp before so I followed the link from the lvfs documentation and then installation with docker: https://pulpproject.org/pulp-in-one-container/
Then I installed Pulp CLI and I was a bit confused about "configure repository to point to manifest file", because there was no option to do so. So I created a remote first (using manifest link) and then repository that points to that remote.
After hitting sync I get those kind of errors.
To me it looks like pulp was upgraded and now it doesn't support such manifests and instead expects a repomd.xml in /repodata. But as I said - I'm a pulp noobie, so I can't tell if it's even me fault or not.

[FrancoCorleone]

A little update: I assumed I new to use RPM plugin,but since there are no xml config files, I decided to give a "file" plugin a try. Which almost worked:

A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz.jcat failed validation due to checksum. Expected ‘b3c202d5b3d7ca078cbd282cbf53cf3c866c80d82967eda61d9e43c4d78a1300’, Actual ‘416db706ab6b92755456358c54b2531fd6122f22b6893f209b48220475427818’'

Any ideas?

[hughsie]

I decided to give a "file" plugin a try

That sounds a bit better -- can you give me a screenshot or some text I can use to fix the docs please?

A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz.jcat failed validation due to checksum.

That's the legacy metadata (i.e. it doesn't have the CDN-fixing revision name) -- maybe we should drop that from the PULP_MANIFEST? Either way you can ignore that one.

[FrancoCorleone]

Honestly the pulp file was a guess on my part... But here: https://docs.pulpproject.org/pulp_file/ they say it replaces some plugin from pulp 2 rpm. So that seems like a good place to start.

Hmm... let me see if I can make it ignore such errors. The problem is I point to the live cersion of the MANIFEST so I can't really "skip" entries

[hughsie]

Okay, lets do https://gitlab.com/fwupd/lvfs-website/-/commit/bc2a562a3305671bfe1a50434897921de88fd39b then! :) I'll deploy now, and it'll take ~4h to regenerate.

[FrancoCorleone]

Perfect :) Thank you