Sci-libs/ni-visa: update, and introduce script to keep it up to date easily

[AndrewAmmerlaan]

Hello,

So first of all let me apologize for the insane size of this PR, but bear with me please because it is definitely worth it IMO.

This PR introduces a script ./update-ebuilds.sh which first fetches all the rpm files from a set url, in our case: https://download.ni.com/ni-linux-desktop/2020.07/rpm/ni/el8/, this is a repository of rpm files distributed by National Instruments for use on Red Hat Enterprise Linux and derivatives, there is also a repository for Debian and derivatives. This script will then read things like the name, version, description, dependencies and such from each rpm file, and create an ebuild with this information.

Why do it like this? Well because there are a lot of drivers in that NI repo and it is not doable to write an ebuild for each one manually (I tried, it was taking way too long). And secondly because NI releases these rpm files in bulk, which means that from one day to the next there will be a lot of version bumps to do. And thirdly, doing this manually makes it insanely difficult to get the dependencies right. The fact that lots of things are spread over several rpms, each with their own essential postinstall script, does not make things easier.

Some of the ebuilds in here are fetch restricted, most notably LabVIEW and related stuff, those ebuilds are generated with a second script ./update-fetch-restrict.sh, which is essentially the same thing but it does not fetch things (obviously). This requires access to the relevant restricted rpm files, either through a university or individually by buying a subscription for the LabVIEW software from National Instruments.

The included eclass helps avoid code repetition, and might also be useful for other rpm-only proprietary packages.

This has been tested and found to work perfectly to the extent that I have access to hardware to test it with.

Some of these ebuilds install to /usr/local (looking at you LabVIEW). Sadly moving everything to /usr or /opt is far from trivial, as several packages and scripts expect LabVIEW and friends to be in /usr/local. And IMO porting everything is not worth the effort, I explored a bit in this direction but too many things broke.

To reduce the size of this PR I could remove the generated ebuilds from this PR and leave only the scripts that generated them. However, that will likely make the repoman checks fail due to unresolvable dependencies.

So to summarize, this PR adds:

• sys-kernel/dkms, which is a new dependency of some things here, the ebuild is based on the ebuild that can be found in ::guru
• A new category (sci-ni) to keep this all isolated from the rest of the ebuilds.
• A new eclass which extends the functionality of the existing rpm eclass to include relevant src_compile, src_install, and pkg_* phases.
• 2 scripts that generate ebuilds from the NI rpm repository, and from a collection of fetch-restricted rpms in a local directory.
• The generated ebuilds.
• A version bump of related software: Lab-VISA, PyVISA, and PyVISA-py, and an unmask of the latter.

[epsilon-0]

holy hell...

[epsilon-0]

OK. So we can start with making things a bit more modular for better reviewing.
Could you make dkms into a separate PR. Then the version bumps for the py-visa* packages into another one?
Then all the 'generated' ebuilds into another one.
We can see more after that.

[AndrewAmmerlaan]

OK. So we can start with making things a bit more modular for better reviewing.
Could you make dkms into a separate PR. Then the version bumps for the py-visa* packages into another one?
Then all the 'generated' ebuilds into another one.
We can see more after that.

On it, here's a separate PR for sys-kernel/dkms: #994

The new py-visa* ebuilds and lab-visa depend on the new ni-visa ebuild, so we'll have to merge the generated ebuilds first. I'll drop the py-visa* and lab-visa version bumps from this PR for now and then we can get back to those later.

[AndrewAmmerlaan]

@epsilon-0 And here's a new PR that does everything but generate the new ebuilds, it adds the eclass, new catagory, and scripts but does not generate the ebuild. (This PR is of a much more reasonable size, with only 5 files changed): #995

[AndrewAmmerlaan]

And this PR, only adds the eclass: #996

[epsilon-0]

awesome, will start looking :)

[AndrewAmmerlaan]

@epsilon-0 @TheChymera I just opened the final PR which bumps dev-python/Py-VISA and related packages: #997

To add a bit of clarity in this web of inter-depending PRs

• This PR Sci-libs/ni-visa: update, and introduce script to keep it up to date easily #993 (adds everything, categories, eclasses scripts and ebuilds) I would suggest to start with the sub-PRs of this PR in the order listed here.
  • sys-kernel/dkms: dynamic kernel module support #994 (sub-PR of this PR, adds only sys-kernel/dkms)
  • eclass/rpm-extended: add eclass #996 (sub-PR of this PR, adds only the eclass)
  • Sci-ni: new catagory and scripts #995 (sub-PR of this PR, adds only categories and scripts but does not add the ebuilds)
• Sci-libs/linux-gpib: bump python targets and EAPI gentoo#18702 (bumps linux-gpib's python targets)
• eclass/rpm-extended: add eclass #996 (bumps python and perl VISA frontends, should be merged last)

[waebbl]

Wouldn't this be an example for an approach similar to the ones from R_overlay or g-ocatve packages, i.e. a small package for the update and/or the process to rebuild the ebuild files, which will then be stored in a separate overlay?

This approach looks volatile to me, alone for the huge amount of ebuilds which are going to get updated each time the script(s) are run and could easily make ::science overlay again a repo with lots of issues. I'm happy and thankful it has been stabilized so much during the last year.

[AndrewAmmerlaan]

This approach looks volatile to me, alone for the huge amount of ebuilds which are going to get updated each time the script(s) are run and could easily make ::science overlay again a repo with lots of issues. I'm happy and thankful it has been stabilized so much during the last year.

I disagree, this creates essentially an "ebuild-wrapper" around the rpm files in the NI repository. As long as the NI repository remains internally consistent, the packages in sci-ni/* will remain internally consistent (and lets give NI the benefit of the doubt that they will keep their own repositories internally consistent). These ebuilds are in a separate category specifically for the purpose of keeping them isolated from the rest, this way whenever NI updates their repository (which is about once or twice a year I think), one can just remove all ebuilds in the sci-ni category and regenerate them using the script. Therefore, it is ensured that the whole thing will continue to function, and that all dependencies are resolvable and will remain resolvable.

The current sci-libs/ni-visa ebuild is a) very out of date, and b) not functional at all, this on the other hand does work, and provides an easy way to keep it up to date and functional. I suppose I could put all this in a separate dedicated NI overlay, but IMO that is not very user-friendly as the science overlay does contain ebuilds which require ni-visa and this would create inter-overlay dependencies. And to be honest I don't see how a separate overlay would be any different from a dedicated isolated category, except that a separate overlay is less user-friendly.

[waebbl]

And to be honest I don't see how a separate overlay would be any different from a dedicated isolated category, except that a separate overlay is less user-friendly.

The user-friendlyness of having several overlays is debateable. The difference lies in handling more than 1k packages at once. You wrote the packages all work perfectly. Let's assume, on the next NI version bump, several packages don't work that perfectly and we got errors from repoman, pkgcheck or egencache or even issues with resolving dependencies. This has nothing to do with how NI is preparing their packages. Rpm is not portage and the handling might very well work with rpm but will need some additional attention with portage.

Some issues that might come up:

• While your eclass has basic error handling, using die after the sed statements, you don't check the resulting scripts any further. According to rpm documentation (https://rpm.org/user_doc/scriptlet_expansion.html) the scriptlets can have dynamic expansion terms with >=rpm-4.9 (always true on Gentoo), which only get evaluated at install time. This can include lua statements, but rpm is by default not build with lua support on Gentoo.
• AFAICS all rpm macros can be used as a dynamic macro, and thus, evaluated at install time. If you take a look at the /usr/libexec/rpm/macros file there are a lot of macros which could be used. Just at the top of the file are several macros for tools. Some of them are not installed by default on Gentoo, but only through a USE flag request or specific profiles. I have currently several listed programs, e.g. lzip, restorecon, bzr and semodule which are not installed on my machine, but can nevertheless be used in a scriptlet.

Don't get me wrong. I honor your request for adding additional packages. But I'm worried about the just risen quality of the ::science overlay might again go down. Seeing that NI has only 1-2 releases per year, this might in fact be an over-reacting on my side. But if you look at the R_overlay, which are also mostly science related ebuilds, maintaining it is an awful lot of work.

[epsilon-0]

Just to get an idea, how many packages does this add?

[waebbl]

My 1k statement just refers to the number of files listed above (1270). If you calculate an ebuild and metadata.xml for each package we have ~600 packages. If there are packages with several versions, it's even less.

[AndrewAmmerlaan]

Just to get an idea, how many packages does this add?

407

• While your eclass has basic error handling, using die after the sed statements, you don't check the resulting scripts any further. According to rpm documentation (https://rpm.org/user_doc/scriptlet_expansion.html) the scriptlets can have dynamic expansion terms with >=rpm-4.9 (always true on Gentoo), which only get evaluated at install time. This can include lua statements, but rpm is by default not build with lua support on Gentoo.

• AFAICS all rpm macros can be used as a dynamic macro, and thus, evaluated at install time. If you take a look at the /usr/libexec/rpm/macros file there are a lot of macros which could be used. Just at the top of the file are several macros for tools. Some of them are not installed by default on Gentoo, but only through a USE flag request or specific profiles. I have currently several listed programs, e.g. lzip, restorecon, bzr and semodule which are not installed on my machine, but can nevertheless be used in a scriptlet.

I only execute the script if it is marked as a script that should be run with /bin/sh, for this collection of rpm's all scripts are sh-scripts. If there would be some other collection of rpm files that happen to use e.g. python to run the scripts than a special case could be added to the eclass for this purpose that extracts only python scripts and runs them with EPYTHON. Perhaps we could even add a variable that controls which kind of scripts are extracted and executed, e.g. RPM_SCRIPT_RUNNER, which could then be set to e.g. sh, bash, python, lua or whatever. But as that is not necessary at the moment, I'd rather not go through the trouble of supporting anything other than a sh script for the time being. Perhaps in the future we could extend the eclass in such a way to support other script runners, it should not be that hard, all we would need to add is a variable, a case statement and some code specific to each case. In any case, by making this specific to sh scripts I think I have prevented the case where the ebuild would try to execute a e.g. lua script with bash.

And if you're worried that I'm going to leave this to rot when NI releases the new versions of their drivers and other software. I can only offer you my word that I will not. And if you're worried that in the next versions, NI will have changed everything and this script will not longer work, than I can only promise that I will do my very best to fix it. And if I were to not succeed in fixing it, than we can always remove the whole thing again, that would be as simple as rm -r sci-ni.

My 1k statement just refers to the number of files listed above (1270). If you calculate an ebuild and metadata.xml for each package we have ~600 packages. If there are packages with several versions, it's even less.

There are indeed some packages with multiple versions, the 407 I referenced above is the folder count in the sci-ni category of my localrepo.

[epsilon-0]

😮 😮 😮 😮
We only have 634 packages in all of science with only ~250 updated (recently in past 2-3 months)... I'm not sure if it is possible/desirable to do this...
I was (falsely) under the impression this was 25-30 packages >.<
I should have looked better, this mistake is on me.

[AndrewAmmerlaan]

We only have 634 packages in all of science with only ~250 updated (recently in past 2-3 months)... I'm not sure if it is possible/desirable to do this...

😢 why not? It is all updated in one go, so it should not be that much effort to maintain. Once I got the script setup it took me about 10 minutes to generate all the ebuilds, and it's all automatic. All I had to do manually was shorten some descriptions, because repoman said they were too long.

[epsilon-0]

Sorry to be a party pooper. 😢
I talked and asked for opinions on IRC and everyone who commented said that this would not be the best idea right of the bat, to add them to an official overlay.
All the devs who were not totally against it, said that it is better for this to be in a totally separated overlay of its own.
This has historically been true for all generated overlays, Roverlay, octave, heroxbd's new python overlay and also for non generated overlays but those which are very targeted, like haskell and ocaml overlays.
I feel this is another of those candidates, where it has enough complexity to stand independently in its own right.

It is possible to add a reference to the overlay so that it is easier to find and can be linked in the README (have actually been meaning to add references for above mentioned overlays).

Another thing to do is to make the packages that are dependent on this generated list to also be moved to the overlay (like py-visa and ni-visa). They are already rotting here.

(This is not at all to say that I don't appreciate your other efforts, they are really awesome :D just that this particular effort may not be the best fit here.)

[epsilon-0]

On a totally different note, there might actually be a more fundamental solution at the package manager level, as portage can generate RPM packages but it cannot install RPM packages... maybe the reverse directionality could be useful to have...

[AndrewAmmerlaan]

Sorry to be a party pooper. cry
I talked and asked for opinions on IRC and everyone who commented said that this would not be the best idea right of the bat, to add them to an official overlay.
All the devs who were not totally against it, said that it is better for this to be in a totally separated overlay of its own.
This has historically been true for all generated overlays, Roverlay, octave, heroxbd's new python overlay and also for non generated overlays but those which are very targeted, like haskell and ocaml overlays.
I feel this is another of those candidates, where it has enough complexity to stand independently in its own right.

It is possible to add a reference to the overlay so that it is easier to find and can be linked in the README (have actually been meaning to add references for above mentioned overlays).

Another thing to do is to make the packages that are dependent on this generated list to also be moved to the overlay (like py-visa and ni-visa). They are already rotting here.

(This is not at all to say that I don't appreciate your other efforts, they are really awesome :D just that this particular effort may not be the best fit here.)

Very well, I will make my own overlay tomorrow than. I'll also move stuff like PyVISA and LAB-VISA in there, and then I'll make a PR to clean those packages up here.

I do appreciate your comments on the eclass I wrote, they were very helpful. Thanks 😄

On a totally different note, there might actually be a more fundamental solution at the package manager level, as portage can generate RPM packages but it cannot install RPM packages... maybe the reverse directionality could be useful to have...

Portage has excellent support for installing the contents of a rpm. The problem starts when the rpm file has important stuff going on in the pre/post/install/rm scripts. Usually when packaging an rpm all one has to do is inherit the rpm eclass, and cp the needed stuff to ${ED}, however for this particular collection of rpms that is sadly not enough. What would be awesome is if we could have the rpm command handle the whole install process, while portage keeps track of the installed files and resolves dependencies. However, that would require implementing that into emerge itself, which sounds like a lot of work, and I suspect there will be many problems with some portage features. IDK if it is worth the effort, but perhaps this is something for the future.

[epsilon-0]

Yes, cleanup PRs are appreciated :D