-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sci-libs/ni-visa: update, and introduce script to keep it up to date easily #993
Conversation
holy hell... |
OK. So we can start with making things a bit more modular for better reviewing. |
On it, here's a separate PR for sys-kernel/dkms: #994 The new py-visa* ebuilds and lab-visa depend on the new ni-visa ebuild, so we'll have to merge the generated ebuilds first. I'll drop the py-visa* and lab-visa version bumps from this PR for now and then we can get back to those later. |
@epsilon-0 And here's a new PR that does everything but generate the new ebuilds, it adds the eclass, new catagory, and scripts but does not generate the ebuild. (This PR is of a much more reasonable size, with only 5 files changed): #995 |
And this PR, only adds the eclass: #996 |
awesome, will start looking :) |
@epsilon-0 @TheChymera I just opened the final PR which bumps dev-python/Py-VISA and related packages: #997 To add a bit of clarity in this web of inter-depending PRs
|
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Package-Manager: Portage-3.0.12, Repoman-3.0.2 Signed-off-by: Andrew Ammerlaan <[email protected]>
Wouldn't this be an example for an approach similar to the ones from R_overlay or g-ocatve packages, i.e. a small package for the update and/or the process to rebuild the ebuild files, which will then be stored in a separate overlay? This approach looks volatile to me, alone for the huge amount of ebuilds which are going to get updated each time the script(s) are run and could easily make ::science overlay again a repo with lots of issues. I'm happy and thankful it has been stabilized so much during the last year. |
I disagree, this creates essentially an "ebuild-wrapper" around the rpm files in the NI repository. As long as the NI repository remains internally consistent, the packages in sci-ni/* will remain internally consistent (and lets give NI the benefit of the doubt that they will keep their own repositories internally consistent). These ebuilds are in a separate category specifically for the purpose of keeping them isolated from the rest, this way whenever NI updates their repository (which is about once or twice a year I think), one can just remove all ebuilds in the sci-ni category and regenerate them using the script. Therefore, it is ensured that the whole thing will continue to function, and that all dependencies are resolvable and will remain resolvable. The current sci-libs/ni-visa ebuild is a) very out of date, and b) not functional at all, this on the other hand does work, and provides an easy way to keep it up to date and functional. I suppose I could put all this in a separate dedicated NI overlay, but IMO that is not very user-friendly as the science overlay does contain ebuilds which require ni-visa and this would create inter-overlay dependencies. And to be honest I don't see how a separate overlay would be any different from a dedicated isolated category, except that a separate overlay is less user-friendly. |
The user-friendlyness of having several overlays is debateable. The difference lies in handling more than 1k packages at once. You wrote the packages all work perfectly. Let's assume, on the next NI version bump, several packages don't work that perfectly and we got errors from repoman, pkgcheck or egencache or even issues with resolving dependencies. This has nothing to do with how NI is preparing their packages. Rpm is not portage and the handling might very well work with rpm but will need some additional attention with portage. Some issues that might come up:
Don't get me wrong. I honor your request for adding additional packages. But I'm worried about the just risen quality of the ::science overlay might again go down. Seeing that NI has only 1-2 releases per year, this might in fact be an over-reacting on my side. But if you look at the R_overlay, which are also mostly science related ebuilds, maintaining it is an awful lot of work. |
Just to get an idea, how many packages does this add? |
My 1k statement just refers to the number of files listed above (1270). If you calculate an ebuild and metadata.xml for each package we have ~600 packages. If there are packages with several versions, it's even less. |
407
I only execute the script if it is marked as a script that should be run with /bin/sh, for this collection of rpm's all scripts are sh-scripts. If there would be some other collection of rpm files that happen to use e.g. python to run the scripts than a special case could be added to the eclass for this purpose that extracts only python scripts and runs them with And if you're worried that I'm going to leave this to rot when NI releases the new versions of their drivers and other software. I can only offer you my word that I will not. And if you're worried that in the next versions, NI will have changed everything and this script will not longer work, than I can only promise that I will do my very best to fix it. And if I were to not succeed in fixing it, than we can always remove the whole thing again, that would be as simple as
There are indeed some packages with multiple versions, the 407 I referenced above is the folder count in the sci-ni category of my localrepo. |
😮 😮 😮 😮 |
😢 why not? It is all updated in one go, so it should not be that much effort to maintain. Once I got the script setup it took me about 10 minutes to generate all the ebuilds, and it's all automatic. All I had to do manually was shorten some descriptions, because repoman said they were too long. |
Sorry to be a party pooper. 😢 It is possible to add a reference to the overlay so that it is easier to find and can be linked in the README (have actually been meaning to add references for above mentioned overlays). Another thing to do is to make the packages that are dependent on this generated list to also be moved to the overlay (like py-visa and ni-visa). They are already rotting here. (This is not at all to say that I don't appreciate your other efforts, they are really awesome :D just that this particular effort may not be the best fit here.) |
On a totally different note, there might actually be a more fundamental solution at the package manager level, as portage can generate RPM packages but it cannot install RPM packages... maybe the reverse directionality could be useful to have... |
Very well, I will make my own overlay tomorrow than. I'll also move stuff like PyVISA and LAB-VISA in there, and then I'll make a PR to clean those packages up here. I do appreciate your comments on the eclass I wrote, they were very helpful. Thanks 😄
Portage has excellent support for installing the contents of a rpm. The problem starts when the rpm file has important stuff going on in the pre/post/install/rm scripts. Usually when packaging an rpm all one has to do is inherit the rpm eclass, and cp the needed stuff to |
Yes, cleanup PRs are appreciated :D |
Hello,
So first of all let me apologize for the insane size of this PR, but bear with me please because it is definitely worth it IMO.
This PR introduces a script ./update-ebuilds.sh which first fetches all the rpm files from a set url, in our case: https://download.ni.com/ni-linux-desktop/2020.07/rpm/ni/el8/, this is a repository of rpm files distributed by National Instruments for use on Red Hat Enterprise Linux and derivatives, there is also a repository for Debian and derivatives. This script will then read things like the name, version, description, dependencies and such from each rpm file, and create an ebuild with this information.
Why do it like this? Well because there are a lot of drivers in that NI repo and it is not doable to write an ebuild for each one manually (I tried, it was taking way too long). And secondly because NI releases these rpm files in bulk, which means that from one day to the next there will be a lot of version bumps to do. And thirdly, doing this manually makes it insanely difficult to get the dependencies right. The fact that lots of things are spread over several rpms, each with their own essential postinstall script, does not make things easier.
Some of the ebuilds in here are fetch restricted, most notably LabVIEW and related stuff, those ebuilds are generated with a second script ./update-fetch-restrict.sh, which is essentially the same thing but it does not fetch things (obviously). This requires access to the relevant restricted rpm files, either through a university or individually by buying a subscription for the LabVIEW software from National Instruments.
The included eclass helps avoid code repetition, and might also be useful for other rpm-only proprietary packages.
This has been tested and found to work perfectly to the extent that I have access to hardware to test it with.
Some of these ebuilds install to /usr/local (looking at you LabVIEW). Sadly moving everything to /usr or /opt is far from trivial, as several packages and scripts expect LabVIEW and friends to be in /usr/local. And IMO porting everything is not worth the effort, I explored a bit in this direction but too many things broke.
To reduce the size of this PR I could remove the generated ebuilds from this PR and leave only the scripts that generated them. However, that will likely make the repoman checks fail due to unresolvable dependencies.
So to summarize, this PR adds: