Use lindas production even for test.visualize-admin.ch

[ptbrowne]

These are my thoughts / questions on why we should not continue to use lindas-int for visualize-int.

Context

Users from BAFU primarily use int.lindas.admin.ch as their work tool as it is connected to lindas int.

Why having datasets on int feels like a bad idea

• Difficult to remember which datasets on which environment has which feature
• Int requires periodic clean up
  • Regression testing is done on int datasets because the datasets there have more features
  • If those datasets were on production, we would not have to worry about int database being cleaned up
• Int Lindas is noticeably slower
• Datasets from search on int are sometimes messy (title not OK, description not OK)
  • Even those that are not in draft
  • Do people not use the draft feature ? Is this because the drafts are not easy enough to use ?
  • Why don’t we use “drafts” on production ?
• Prevents data from landing on production
  This is an assumption
  When users have finished tweaking their dataset, they need to put it in production
  They have to export / import their data from the cubes to “publish” a dataset from int to production
  Then people have to recreate their charts inside the production environment
  Users have to remember to sync the datasets from int to production, this sounds hard
  The whole model would be simpler if charts could have a “visibility” setting.

Why do users use the int environment ?

Quotations are from @Rdataflow

• Because they want to play with the dataset, refine it, before it lands to prod.

  • schema development for all those cubes not already 100% clear how to structure the data – to allow internal review of the data in preparation (i.e. which granularity makes sense to be published, how will it look like, drive user tests & collect feedback on the data, discover needs for visualize features)

  • people are looking for non-official playground to learn and improve before going public

• Why don’t they publish in draft in production ?
  • Already too scary, I guess the dataset feels too public
• Not to pollute the production database (I guess from a technical point of view)

  • understanding PROD is meant for continuous data needed to be tidy – while INT will be tidied up from time to time, so there is less need to clean unused test data

• To test with hierarchy and shared dimensions
  • To my knowledge there is no solution for drafting a new version of a hierarchy or a shared dimension, so int is used as a playground for those.

What would be possible solutions ?

• If there was a login, how would it impact this ?

  • Draft datasets would not be searchable from the search
  • Users could reach draft datasets from their user account ?
    • Toggle in their settings to be able to see draft datasets ?

• How would user be able to use a custom endpoint to load their data ? Should they be allowed to ?

Personally I do not see the point on production.

• If this is necessary, and a chart is made from data on a custom endpoint, we should have a big red warning

[ptbrowne]

@PDumasBAR @l00mi What is your general opinion on this ?

[PDumasBAR]

Hi Patrick,
We need a INT environment for the same reasons you think it's a bad idea.
We need to offer our user an environment where :

• they cannot make problem to the published datasets (here published = in Production). A bad conversion can reduce the performance of the target or even destroy existing dataset (cube or else).
• they can test the conversion of a CSV to a cube. Not all conversion are correct at the first try. Sometimes the final cube is far away from the initial conversion. All the versions are visible in Lindas even those that can give false results to the end users.
• they can make non publishable (or Or a quick and dirty) cubes with less risks to be exploited by final users, in INT than in PROD. That includes, incomplete metadata, subset of the data, etc.
• as there will be always a lot of junk in the "conception" or "learning" environment, we need to clean this environment regularly. It's easier and safer to us to do the cleaning in INT than in PROD where again a bad or false cleaning action can have undesired impacts.
• our goal is to offer a high performance PROD. If all the conception works are made in the PROD, it will become as slow as the INT. The performances in INT impacts only a small amount of users and with contact with them so they know the reason. We hope some better performances in INT with the return in cluster mode.
• the work in INT is also very important to test each new version of Stardog before the production. The experience as shown that the tests in TEST are not sufficient to identify bugs and problems because we have not enough users and datasets in TEST and the applications (Visualize, Cube-Creator, etc.) are work in progress and can have different behaviors than in INT and PROD.
• the publication in production is not so complex. The export/import function is very simple. The advantages of the distinct environments are most valuable than a unique environment :

• Only experienced users are authorized in PROD (87 users in INT, 16 in PROD)
• Only well tested cubes are published in PROD (48 cubes, 119 versions in INT after a recent cleaning, 12 cubes, 62 versions in PROD)
• Only the last version(s) of cubes are published that means less data in PROD and better performances
  An experienced user can make 2 or 3 cubes in INT to learn how to use the tools and how to configure the conversion to achieve the attended results. He can then continue to create cubes directly in PROD. There is not constraint.

But I'm ok with you about the draft status. For me INT is draft, PROD is publication.
The draft status in PROD make no sense as the tests should be made in INT and only when the cube is ready it will be published in production. A draft cube is 100% visible in Lindas, it is hidden only in Visualize, so a draft cube in PROD is visible for the end users.

Login in Visualize will not impact the need for Lindas INT and PROD. But I agree, it can help, maybee.

[l00mi]

Not sure if I missunderstand, but int.visualize.admin.ch does point to PROD Lindas, as documented here: https://gitlab.ldbar.ch/bafu/umweltdatenkiosk-planning#umgebungen

So I assume you wanted to say test.visualize.admin.ch ?

Nevertheless, the only real solution to this ever upcoming discussion is, that there is a (hidden) possibility to change the end-point in the client. (There should obviously be a default one, and it could be always PROD). Then the other two endpoints should be best selectable (drop-down). And finally it would be very use full to also allow a complete new end-point.
That would also allow to make more tests outside the LINDAS cluster at all.. (E.g. WSL could just spin up their own end-point for tests.)

I know that this will need you to also save the endpoint to each visualization, which is a bit of work. But I am pretty sure, that is the most robust and future proof solution, to this never ending discussion.

[ptbrowne]

So I assume you wanted to say test.visualize.admin.ch ?

Yes my mistake 😊 Edited.

Yes the selectable endpoint is related. We have planned to do this in the sprint. But even having the selectable endpoint is not the end of the story.

There should obviously be a default one, and it could be always PROD

I do not think we had ever statuted on this one, and I am glad that we agree.

While we will make it possible to change endpoints, I think we should make it as unfrequent as possible. For me, the best would be that this is disabled in production. @Rdataflow @pierredumas Would it sound possible to you ?

WSL could just spin up their own end-point for tests.

Why can't they use the draft feature for this ?
Is this because the hierarchies / shared dimensions do not support a draft/WIP status ?

[Rdataflow]

@l00mi feel free to go and tell the PO of map.geo.admin.ch if you think it's no good https://s.geo.admin.ch/994bde62e9 to share 3rd party content (beware there is no federal content in this view). You could even show really bad stuff via your own service - though @davidoesch never had issues with that.

@ptbrowne let's not reinvent the square wheel, as we can build upon the fundaments of openness and trust built by the geodata community. Trust is equally important as is openness. What michael maybe missed is: Source attribution absolutely crucial (see red hints in map) - could of course also be more prominent...

... and taking into account the statement from pierre INT and TEST should not be shown prefilled in a selection but would need to be typed (or added via URL param) 👍

[l00mi]

@Rdataflow I think the problem is not very big if we have a portal per Canton https://map.geo.bs.ch/ or https://map.geo.fr.ch/ ... So I would not open up this flood gate to early still.

But the more important point, that is an application, a work tool. The goal is not to create embeddable widgets, which can be put on websites, used in news papers. (And yes I know about the iframes you can create). I would not feel strongly about visualising itself. But it should not be possible to create a graphic to share. Even if you have a very clear disclaimer of the source. It is simply not the same.

Also I will let @ptbrowne handle the questions and complaints based on faulty external data sets. (-;

[davidoesch]

Regarding the import feature of map.geo.admin.ch https://help.geo.admin.ch/?ids=62&lang=en

The legal base you can find in the GeoIG law of geo information: we provide a simple tool to visualize and connect open standardized data sources. data is not stored in the federal data infra.
There are several disclaimer regarding accountability.

The backbone of the internet philosophy is : links , urls etc and all what https://help.geo.admin.ch/?ids=62&lang=en does is linking....

[l00mi]

@davidoesch thank you for the pointers.
That is a good perspective, visualize.admin.ch in contrast does create secondary (refined) artefacts (the sharable graphics with own title, description and selection of data slices), which are hosted.

[Rdataflow]

I have difficulty seeing any contrast. @l00mi see this

https://s.geo.admin.ch/99515aff22 - admin.ch is afraid of anyones content? - obviously not. let's stick to the existing philosophy...

[ptbrowne]

Thanks to awesome work @bprusinowski , we will now be able to select production database from test.visualize.admin.ch. I think this closes the discussion 🎉

[PDumasBAR]

Simple and effective! 🥇
Thanks