Add Void Linux repository support #400

ninewise · 2017-11-28T21:56:01Z

I could use some feedback on repos.d/voidlinux.yaml, mainly the sources. I used a lot of subrepos because each source might have a different version, but perhaps I should use flavours for this? I'm not sure what's the accepted practice.

cc @Vaelatern

AMDmi3 · 2017-11-29T14:07:59Z

This data is not suitable for repology. There are incompatible versions with -N suffix (revision?), and there are subpackages (-32bit, -devel-32bit, -doc, etc.) which cannot be compared to anything. Having a lot of similar packages for all architectures is also a problem. See also #158.

ninewise · 2017-11-29T14:51:39Z

The revisions can easily be scratched (I believe Package.origversion could be used to refer to them?) and it shouldn't be hard to remove subpackages (provided I can get access to both a clone of the repository and the repodata file in the same parser?).

Should I then move the architectures to a flavour, would it still be possible to include voidlinux?

This allows to parse multiple independent (but overlapping) subrepositories such as "i386" and "amd64", and append a flavor based on repository name, so if a package has different versions for different architectures, older versions would be considered outdated and not legacy. Related to repology#400

AMDmi3 · 2017-11-29T15:25:29Z

The revisions can easily be scratched (I believe Package.origversion could be used to refer to them?)

Yes. See repology/parsers/freebsd.py for SanitizeVersion pattern used for this purpose. Please stick to it, as it would make refactoring easier in future.

and it shouldn't be hard to remove subpackages (provided I can get access to both a clone of the repository and the repodata file in the same parser?).

You can't, and I don't understand how it's going to help you. However, as I see there's source-revisions field which may be useful: you could split it by : and set package.effname to the first part right away. That way, Void packages will be born assigned to correct metapackage, and information on package name will still be preserved.

Should I then move the architectures to a flavour

No, the more correct would be to add flavor based on repository name. I've just added code to allow that.

would it still be possible to include voidlinux?

As long as the data is of sufficient quality, but that can only be seen after proper parsing is implemented.

AMDmi3 · 2017-11-29T15:26:15Z

repology/parsers/voidlinux.py

@@ -0,0 +1,46 @@
+# Copyright (C) 2016-2017 Dmitry Marakasov <[email protected]>


Please set proper copyright with your name,

AMDmi3 · 2017-11-29T15:27:21Z

repology/parsers/voidlinux.py

+from repology.package import Package
+
+
+def parse_maintainer(maintainerstr):


There's already from repology.util import GetMaintainers tool.

AMDmi3 · 2017-11-29T15:28:31Z

repology/parsers/voidlinux.py

+ plist_index = plistlib.load(open(index_path, 'rb'),
+ fmt=plistlib.FMT_XML)
+
+ return [Package(name=pkgname,


Please fill fields one by one. The plans are to hide Package fields behind setters to catch incorrect data on assignment.

AMDmi3 · 2017-11-29T15:30:06Z

repology/parsers/voidlinux.py

+ fmt=plistlib.FMT_XML)
+
+ return [Package(name=pkgname,
+ version=props['pkgver'].split('-')[-1].replace('_', '-'),


Not sure if that's safe - - may be encountered in a version. I'd ensure that it starts with package name and strip pkgname- from the pkgver. Don't forget to strip revision.

- may never be in a version.

AMDmi3 · 2017-11-29T15:31:09Z

repos.d/voidlinux.yaml

+###########################################################################
+# Voidlinux
+###########################################################################
+- name: voidlinux


Is `linux' suffix necessary everywhere? I'd prefer to drop it for simplicity.

AMDmi3 · 2017-11-29T15:31:38Z

repos.d/voidlinux.yaml

+ packagelinks:
+ - desc: Git
+ url: 'https://github.com/voidlinux/void-packages/tree/master/srcpkgs/{name}'
+ tags: [ all, production ]


Please add newline

AMDmi3 · 2017-11-29T15:32:46Z

repos.d/voidlinux.yaml

+ - desc: Void Linux package repository
+ url: https://github.com/voidlinux/void-packages
+ packagelinks:
+ - desc: Git


May also add another link to template

AMDmi3 · 2017-11-29T15:33:18Z

repos.d/voidlinux.yaml

+ fetcher: WgetTar
+ parser: VoidLinux
+ url: https://repo.voidlinux.eu/current/{source}-repodata
+ subrepo: '{source}'


+ flavor: '{source}' as mentioned in the comment

AMDmi3 · 2017-11-29T15:39:37Z

I still don't like these lots of repos. This produces excessive and duplicate package information which both slows down processing and makes website views cumbersome. Since you, as I assume, are Void Linux user, can't you set up native tools to produce machine readable index of source packages out of void-packges repository? That would make parsing easier, as unmangled package name and version are available and avoid duplicate packages.

ninewise · 2017-11-29T16:12:21Z

I did something parsing void-packages in my first commit, db19fa8, but I assumed it'd be better to avoid calling the shell in a subprocess. I could revert to that method again if that's preferred.

AMDmi3 · 2017-11-29T16:45:47Z

I've meant preparing information on Void source packages outside repology. Probably somewhere on Void Linux infrastructure.

Vaelatern · 2017-11-30T00:37:26Z

The void-packages repository is simple: you list main packages by listing regular directories (not symlinks) and adding the python3 symlinks.

Packages do not exist on user systems in any way not merely from the repodata (including no subpkg vs. pkg distinction).

ninewise · 2017-11-30T13:37:14Z

Latest version combines the void-package repository and the repodata. This way we can be sure which are actual packages and which are subpackages, while at the same time avoiding heuristics for parsing the template files.

For this to work, I had to write my own fetcher, and I supplied some extra variables to the sources to avoid hardcoding urls and flavors. This version creates a lot less packages compared to the previous, as it uses flavors for the architectures.

AMDmi3 · 2017-11-30T13:51:40Z

Looks overcomplicated now. Why's can't source-revisions be used?

AMDmi3 · 2017-11-30T14:30:21Z

I've tried it. The data looks mostly good, however for some reason source-revisions is missing for a fraction of packages. Is there a reason for it?

AMDmi3 · 2017-11-30T14:35:41Z

I've also noticed some subpackages with broken comment (seem to be replaced with suffix which was intended to be appended):

Nov 30 17:13:58 voidlinux:     sanity warning: libgadu-devel: comment is not stripped: " - development files"
Nov 30 17:13:59 voidlinux:     sanity warning: libsoxr-devel: comment is not stripped: " - development files"
Nov 30 17:13:59 voidlinux:     sanity warning: libsoxr-doc: comment is not stripped: " - documentation and examples"
Nov 30 17:14:00 voidlinux:     sanity warning: luaposix51: comment is not stripped: " - Lua 5.1"
Nov 30 17:14:00 voidlinux:     sanity warning: luaposix52: comment is not stripped: " - Lua 5.2"

Could it be that some packages need metadata reparsing?

AMDmi3 · 2017-11-30T15:20:43Z

See my branch: https://github.com/repology/repology/tree/voidlinux
The only problem left with it are packages without source-revisions. If this can be fixed from Void side, I'm merging it. Otherwise, we may investigate parsing raw template files with bashlex module I've just stumbled upon.

AMDmi3 · 2017-11-30T15:54:02Z

@ebfe from #voidlinux has a proof of concept json dump (https://gist.github.com/ebfe/e6ff186811cd207b884a9d7cff9d4e8a), I'd prefer to use it if it can be set up for automated regular updates anytime soon.

Chocimier · 2018-01-06T21:47:21Z

Another option: output from xbps-src included in git repository is quite readable:

~/github/void-packages ./xbps-src show netpbm
pkgname:        netpbm
version:        10.81.0
revision:       1
distfiles:      https://github.com/chneukirchen/netpbm-mirror/archive/a5fb397b3ab3ac6b2c17c72598036bedbdca1816.tar.gz
distfiles:      https://github.com/chneukirchen/netpbm-mirror/archive/ad3acab4d1ef5acd4a232c2bec3590620cd0edea.tar.gz
checksum:       ba1a03998d920f33a42450f583cd9024c90bb986c8b6a4ab7244d660deef593c
checksum:       55678f15a2ade5ed4c75f988a05296c895042e58a8c24829929562b3344a5349
maintainer:     Leah Neukirchen <[email protected]>
Upstream URL:   http:https://netpbm.sourceforge.net/
License(s):     BSD,GPL-2,custom
short_desc:     Toolkit for manipulation of graphic images
subpackages:    libnetpbm-devel
subpackages:    libnetpbm

Package is a subpackage if it lists itself in subpackages field.

AMDmi3 · 2018-01-07T02:08:22Z

Repology cannot run any scripts as it's intended to be portable. We need a ready to use dump.

Chocimier · 2018-11-28T12:18:20Z

We are rebuilding packages to generate missing source-revisions, more than half is done. Note that for some packages there is trailing /template there.

We moved website to https://voidlinux.org , github to https://github.com/void-linux/void-packages and repository to https://alpha.de.repo.voidlinux.org/current/ .

Chocimier · 2018-12-05T21:06:33Z

@AMDmi3 Indexes are now regenerated for source-revisions. There are few leftovers, that can't be easily fixed, but I hope the whole state it is acceptable.

AMDmi3 · 2018-12-06T12:37:20Z

Lets recap this. The code is a year old rotten and needs to be rewritten from scratch. I no longer remember details so I have no idea what source-revisions thing is about. I can't even examine current repository contents as it acts like it knows better than me whether I can handle a file list. The latest version of code wanted to parse 21 repodatas which would presumably lead to a lot of duplicated packages, if that's still the case that is not acceptable. I also assume there's still no dedicated json or xbps-src output (which looks usable too) dump.

Chocimier · 2018-12-06T14:51:03Z

I no longer remember details so I have no idea what source-revisions thing is about.

Just useful metadata field missing then (details).

I can't even examine current repository contents

Bothers me too, but parser is based on repodata files, not directory listing.

The latest version of code wanted to parse 21 repodatas which would presumably lead to a lot of duplicated packages, if that's still the case that is not acceptable.

We fixed repo as, you were about to accept it.

a lot of duplicated packages

If it is a problem, I am sure there is a way to handle that: keeping entry non-duplicated by database, having a set of names of already-registed packages in parser or something other.

I also assume there's still no dedicated json

You are right, there is no.

AMDmi3 · 2018-12-06T16:12:30Z

Bothers me too, but parser is based on repodata files, not directory listing.

I cannot maintain it unless I see the repository contents at least. Transparency is an absolute requirement.

We fixed repo as, you were about to accept it.

As the time pass, and more repositories are supported, quality requirements raise, as each problematic repository creates more work for me and more noise for the users.

If it is a problem, I am sure there is a way to handle that: keeping entry non-duplicated by database, having a set of names of already-registed packages in parser or something other.

Not an option, as this introduces inconsistencies.

However, we could stick with e.g. x86_64 + nonfree/x86_64.

ninewise · 2018-12-06T16:57:05Z

The code I originally PR'd is indeed disapproved, and was no longer even discussed here.

Chocimier · 2018-12-06T17:44:39Z

However, we could stick with e.g. x86_64 + nonfree/x86_64.

Fine for me. Is listing directory only major issue (while I do not understand why is it an issue) then?

AMDmi3 · 2018-12-06T17:55:15Z

Fine for me. Is listing directory only major issue (while I do not understand why is it an issue) then?

For now, yes. I still need to check the data.

the-maldridge · 2018-12-06T18:16:46Z

Seems like there's some serious confusion here about how xbps works and how our mirror infrastructure works.

First, there are multiple repodata files because there are multiple repositories. If you don't parse them then you won't have the complete view of the universe. If you're okay having massive holes in the dataset, then just parse a handful of files.

Second, the current/ directory is for machines, not humans. The page is there to prevent you from loading a page with 10k+ links on it which your browser will try to prefetch for validity. If you believe you actually want a file list then get it from rsync which is designed to do that, or get yourself netblock banned from someone else's mirror. There's no transparency in that list anyway, its pretty trivial to filter for specific UAs and give back a different list there.

Vaelatern · 2018-12-06T18:36:25Z

If you need to check the data, you can compose a URL to any package in question yourself. This is straightforward, and the package is provided as a tarball (by a different extension, but a tarball).

AMDmi3 · 2018-12-06T19:00:14Z

First, there are multiple repodata files because there are multiple repositories. If you don't parse them then you won't have the complete view of the universe.

There is no confusion, I understand that fully. But, since there is no source packages (e.g. templates) metadata, and I'm not going to just dump a whole bunch of duplicates (archs + musl + multilib) for each package into repology, that's what we'll have to live with.

If you're okay having massive holes in the dataset, then just parse a handful of files.

The completeness of the dataset should bother you, as a primary service consumer, in the first place. And I see no other way to improve that than void publishing source metadata. Which was already demonstrated by different people, but never finished.

Second, the current/ directory is for machines, not humans. The page is there to prevent you from loading a page with 10k+ links on it which your browser will try to prefetch for validity. If you believe you actually want a file list then get it from rsync which is designed to do that, or get yourself netblock banned from someone else's mirror. There's no transparency in that list anyway, its pretty trivial to filter for specific UAs and give back a different list there.

I need to see what's I'm working with to keep it working and improve. Mirrors do not help as they all serve the stub instead of a listing. It would be OK if it worked with curl.

the-maldridge · 2018-12-06T19:04:10Z

The completeness of the dataset should bother you, as a primary service consumer, in the first place.

I am not a consumer, I'm the person on the other end who's dealing with people hammering mirrors rather than parsing the machine readable data.

I need to see what's I'm working with to keep it working and improve. Mirrors do not help as they all serve the stub instead of a listing. It would be OK if it worked with curl.

Then summon the listing via rsync as suggested.

Vaelatern · 2018-12-06T19:08:17Z

But, since there is no source packages (e.g. templates) metadata, and I'm not going to just dump a whole bunch of duplicates (archs + musl + multilib) for each package into repology, that's what we'll have to live with.

How would you prefer to get all the data from archs, musl, nonfree? Since any format will have duplicates that don't share every quality (same pkgname version, different arch)

AMDmi3 · 2018-12-06T19:10:33Z

Then summon the listing via rsync as suggested.

Example commandline please?

AMDmi3 · 2018-12-06T19:11:33Z

How would you prefer to get all the data from archs, musl, nonfree? Since any format will have duplicates that don't share every quality (same pkgname version, different arch)

I'd prefer not to deal with binary package data at all.

the-maldridge · 2018-12-06T19:11:49Z

rsync rsync:https://alpha.de.repo.voidlinux.org/voidmirror/current

I suggest piping that to less.

the-maldridge · 2018-12-06T19:12:29Z

Well, if you don't deal with the binary package data then you'll get bogus data. Not all packages sitting in the webroot are in the current generation of the tree.

AMDmi3 · 2018-12-06T19:42:25Z

rsync rsync:https://alpha.de.repo.voidlinux.org/voidmirror/current/

Thanks, it needs a trailing slash, but it's acceptable for my purpose.

Well, if you don't deal with the binary package data then you'll get bogus data. Not all packages sitting in the webroot are in the current generation of the tree.

It's not bogus, it's just the discrepancy between source and binary packages. Ideally, we need knowledge of both, so we can both tell maintainers which packages need work (tied to commits), and tell users which packages are already available for them (tied to package builds). Software authors are probably interested in both. However for now we focus on maintainers, who want outdated status of their packages to be cleared as soon as possible after they commit the update. The lag of actual package builds is expected and considered tolerable, no one complained on that for now.

the-maldridge · 2018-12-06T19:44:22Z

As long as you understand that the tree probably doesn't resolve at all times, I guess that's fine. I'm not sure how your system works then without referential integrity checks.

You should probably pick another mirror though to pull that rsync listing from, unless of course your machines are in Germany.

Vaelatern · 2018-12-06T19:46:46Z

@AMDmi3 there are often multiple versions of a package in the physical directory, though only one in the repo index. The only thing that matters to users or maintainers is what's in the index. Everything else is irrelevant.

Vaelatern · 2018-12-06T19:47:58Z

To be clear. The repo index has nothing to do with the theoretical packages we have. It has EVERYTHING to do with the packages that people can actually install at that very moment.

AMDmi3 · 2018-12-06T19:58:51Z

The only thing that matters to users or maintainers is what's in the index.

That's definitely not true for maintainers as they commit to the repository and the state of repository is what is relevant to them.

Vaelatern · 2018-12-06T20:02:38Z

@AMDmi3 As someone with close knowledge about Void operations, you should know that the only thing that matters is what's in the repo index, and that repo index is authoritative.

Chocimier · 2018-12-06T20:08:02Z

That's definitely not true for maintainers as they commit to the repository and the state of repository is what is relevant to them.

Nope. When template is updated and binary index is not updated for time longer than usual processing, it is clear signal that template is broken in some way causing new version of program not being available for users. Therefore it is not important what template contain. Binary index is.

AMDmi3 · 2018-12-06T20:26:10Z

Well I don't see a point in dissuading you. The fact is that without source metadata, void support will always be lacking. Meanwhile, I've updated the code an run a local update, and it still looks good. Will commit after adding some rules.

AMDmi3 · 2018-12-07T18:25:56Z

It's deployed, enjoy.

AMDmi3 · 2018-12-25T18:06:08Z

@Chocimier JFYI, there are some minor parsing problems which could be worth fixing, including misformatted URLs and couple of missing source-revisions.

https://repology.org/log/550490

Chocimier · 2018-12-28T19:57:26Z

Missing source-revisons come from packages troublesome to rebuild, as I mentioned before. Homepages are fixed now, thanks for info.

Felix Van der Jeugt added 4 commits November 28, 2017 21:00

attempt using xbps-src

db19fa8

proper solution using repodata

89efb76

add rules for voidlinux

16fa899

comply to flake8 and schemacheck

fe9be78

ninewise force-pushed the voidlinux branch from 42f00a2 to fe9be78 Compare November 28, 2017 22:16

AMDmi3 force-pushed the voidlinux branch from 013ddb7 to fe9be78 Compare November 29, 2017 12:18

AMDmi3 requested changes Nov 29, 2017

View reviewed changes

AMDmi3 mentioned this pull request Nov 30, 2017

Provide minimal test target jarun/ddgr#27

Merged

add fetcher to combine repository and repodata

b0f7931

ninewise force-pushed the voidlinux branch 3 times, most recently from ef25b5c to 11aea6e Compare November 30, 2017 13:24

pass schemacheck

f01394f

ninewise force-pushed the voidlinux branch from 11aea6e to f01394f Compare November 30, 2017 13:28

AMDmi3 removed the upstream cooperation required label Dec 6, 2018

AMDmi3 closed this in d3b58cd Dec 7, 2018

		@@ -0,0 +1,46 @@
		# Copyright (C) 2016-2017 Dmitry Marakasov <[email protected]>

		from repology.package import Package


		def parse_maintainer(maintainerstr):

Add Void Linux repository support #400

Add Void Linux repository support #400

Conversation

ninewise commented Nov 28, 2017

AMDmi3 commented Nov 29, 2017

ninewise commented Nov 29, 2017

AMDmi3 commented Nov 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AMDmi3 Nov 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AMDmi3 commented Nov 29, 2017

ninewise commented Nov 29, 2017

AMDmi3 commented Nov 29, 2017

Vaelatern commented Nov 30, 2017

ninewise commented Nov 30, 2017

AMDmi3 commented Nov 30, 2017

AMDmi3 commented Nov 30, 2017

AMDmi3 commented Nov 30, 2017

AMDmi3 commented Nov 30, 2017

AMDmi3 commented Nov 30, 2017

Chocimier commented Jan 6, 2018 • edited Loading

AMDmi3 commented Jan 7, 2018

Chocimier commented Nov 28, 2018

Chocimier commented Dec 5, 2018

AMDmi3 commented Dec 6, 2018 • edited Loading

Chocimier commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

ninewise commented Dec 6, 2018

Chocimier commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

the-maldridge commented Dec 6, 2018

Vaelatern commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

the-maldridge commented Dec 6, 2018

Vaelatern commented Dec 6, 2018 • edited Loading

AMDmi3 commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

the-maldridge commented Dec 6, 2018

the-maldridge commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

the-maldridge commented Dec 6, 2018

Vaelatern commented Dec 6, 2018

Vaelatern commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

Vaelatern commented Dec 6, 2018

Chocimier commented Dec 6, 2018

AMDmi3 commented Dec 6, 2018

AMDmi3 commented Dec 7, 2018

AMDmi3 commented Dec 25, 2018 • edited Loading

Chocimier commented Dec 28, 2018 via email

AMDmi3 Nov 29, 2017 •

edited

Loading

Chocimier commented Jan 6, 2018 •

edited

Loading

AMDmi3 commented Dec 6, 2018 •

edited

Loading

Vaelatern commented Dec 6, 2018 •

edited

Loading

AMDmi3 commented Dec 25, 2018 •

edited

Loading