Docker image

[robeson1010]

Hi all,
I 'm a remote sensing guy from RADI, China. Could you provide a dockfile or image that we can try this system directly.

[lnicola]

Hi, and thanks for the interest. We know that the deployment story can be better and I'll try to work on this when I get the chance.

Can you tell us if you're planning to test the complete system (that automatically downloads the products and executes the processors on scheduled dates), or just manually running the processors?

[robeson1010]

I want to use the complete system and test it in China.

[chris-aeviator]

@lnicola what's the current status of docker?

[Marwe]

no docker yet and will be pretty difficult to achieve anyway I guess. A colleague works on an ansible setup (not yet published)

[chris-aeviator]

@Marwe thanks for your reply. Concerning ansible, I just found https://github.com/Open-Forecast/Sen2Agri-ansible today

[chris-aeviator]

@Marwe
It seems the recent installer is broken. With multiple clean OS installs I'm unable to get a working Sen2Agri installation.

Could you share with me the complete filetree of your /usr/share/sen2agri folder? After installation I had to manually set paths in the database to have the GUI config util running. From there I was seeing many band related textfiles missing which I downloaded from github. Still my sen2agri installation does not start processing, e.g. the bands in "new site" are completely empty in L8 & S2, even though the .cfg files with the bands have been copied manually.

[Baitcookie]

@chris-aeviator
I am currently working on sen2agri deployment. Seems that https://github.com/Open-Forecast/Sen2Agri-ansible is not the latest version (its really working). I will contact my colleague for an update.

[chris-aeviator]

@Baitcookie would love to connect - I'm extremely frustrated getting a workable deployment of sen2agri. The installer does just go on even though package installation might fail. Currently I'm stepping through the install scripts manually and fail to get a working gdal + libgeotiff installation, they seem to have incompatible requirements with Cent 7

[lnicola]

Hello everyone. This might get a little long-winded, so sorry for that, especially as it won't help much with your issues. But I know how frustrating it might be to talk to walls, and I'll try at least to provide some background. Note that even though I worked on the Sen2-Agri project, these are all my opinions, they don't necessarily represent an official project announcement or reflect the opinions of my employer or the Consortium.

• the gnarly install script: the system was developed mostly with a small number of non-technical users in mind. We have quite a few external dependencies -- to list some: SLURM, MUNGE (required by SLURM), Apache (hosting the website), MACCS (which we weren't allowed to distribute, so we didn't), MACCS configuration files (for which there wasn't a single source-of-truth), the SRTM 3 arc-second DEM, the SWBD dataset, MariaDB (used by SLURM), PostgreSQL (used by the rest of the system), not to mention a whole set of various dependencies. Out of these, SLURM, MUNGE, Apache, MariaDB and PostgreSQL are daemons, which makes creating a single Docker container pretty difficult. Of course, these daemons can be containerized just fine, so a Docker approach will also need docker-compose. With my developer hat on, it would have been easier, cleaner and simpler to ask the users to install all the software above. But given the non-technical nature of our users (most of which had no previous experience with Linux), it was decided to prepare an install script that does its best to put everything in the right place. In retrospect, we should have documented this somewhere.
• making containers: as I already mentioned, the software above can be containerized relatively easier. The harder part, however, is running the other parts of the system in containers:
  • we use DBus as an IPC mechanism
  • we need to know whether and how to run the processors in a container
  • some of the processors need to access the Postgres database to retrieve their inputs; doing that in Docker isn't trivial because you need to set things up to make it possible for one container to connect to another
  • Docker itself has some weird interactions with e.g. NFS shares configured with strict permissions (we've had reports of this), because two users need to access the files (dockerd running as root and our applications, which don't run as root because they don't need to)
  • there were concerns that some users might not want to use the Docker setup, and supporting both is pretty hard
• the current status: as you might have noticed, Sen2-Agri doesn't see much active development by itself; it was, however, extended in the course of the Sen4CAP project. In the context of that project, we were able to move our Postgres database to a container, which trickled into a Sen2-Agri release. Of course, this is only a small step, but it still helped a little with various ABI incompatibilities we've ran into (libgeotiff and geos being especially problematic). I also got other parts, including SLURM and MUNGE, running in containers, but that was only an experiment, and wasn't included in a release
  • in Sen4CAP we've recently migrated more components to containers: the L2A processors, MACCS/MAJA, and the other processors. Unfortunately, this is still not a complete Docker setup
• external services: the external services we use to download products (USGS, to a lesser extent SciHub) have made changes to their APIs, breaking our code. We've generally updated it, but the downloader itself (a Java service) is only available in the installation package, but not in this GitHub repository. It's possible for the latest installation package to contain a version that is no longer working. On the other hand, updating that component isn't always safe, because it tends to change a lot and might have compatibility issues.
• Ansible: it should be possible to replicate the "interesting" parts (which might differ from one user to another) in an Ansible script; I've personally never used it, so I don't have any specific advice
• binary packages: as you noticed, libgeotiff might still be problematic in some cases. There is a build script that uses Docker to produce binary packages in a clean CentOS installation. I'm not sure if the version in this repository is the latest one, though. It's also not used to produce the installation packages available for download.
• tossing source code over a fence: this might be less important now, but we've generally dumped code to this repo without playing well with the community. This was partly due to legal concerns about accepting outside contributions, and about making code public before the official project milestones. I understand that this is an awful thing to do when you're trying to build a community, but I never found a solution for it.
• maintenance: to be honest, while we offered support to a couple dozens of users who contacted us, we don't have the means to maintain the project as well as it might deserve. We've done some relatively good work here, but a project like this needs continuous maintenance to keep up with changing dependencies, reduce technical debt (of which we have a lot), or implement features requested by users. A lot of the developments done for Sen4CAP ended up in Sen2-Agri as well, but that project is also ending soon.

[chris-aeviator]

@lnicola amazing - thank you for you sharing some insights in the how's and why's - is there any chance that someone provides a VM image/ ISO of a fresh install as this would likely solve some headaches, looking also at entries in the s2a forum. Happy to discuss non-publicly also via [email protected] .

[lnicola]

No promises, but I'll ask around.

[chris-aeviator]

@lnicola it turned out that my sen2agri installation seems to work. I re-ran the installer again and tried a different shapefile and suddenly had sensors listed in my "create site" section. Due to other inconsistencies with the docs and my system, plus some weeks of messing with my environment, I had not realized that my (possible multipolygon) shapefile has caused my issues.

[lnicola]

I'm not sure what went wrong. Shapefiles with multiple geometries are supported and merged into a MULTIPOLYGON geometry which is stored in the database.