Oasis operation broken after update to 1.3.3 #108
Comments
|
Hi @behnle! This seems like a hard problem to track down. I cannot reproduce anything like this if I e.g. run a local Oasis and use the central authentication, or in our current deployments which are based on 1.3.3 and 1.3.4 and use the central authentication. I think @blueraft or @markus1978 might know better if anything critical has changed in 1.3.3 that could cause this. Would you know which version of the |
|
I don't believe anything changed with regards to keycloak. @Sideboard was looking into updating the keycloak version but we are still using |
|
Thanks for Your replies @lauri-codes @blueraft . The last version that is remember to work was an 1.2.2 (?) image (with SHA256 sum 279c097945fe553be09e8f50d0502f20210836eff3d8b5c6b2213f8297b32724) docker image inspect[root@u-030-s007 nomad]# docker image inspect 279c097945fe
[
{
"Id": "sha256:279c097945fe553be09e8f50d0502f20210836eff3d8b5c6b2213f8297b32724",
"RepoTags": [],
"RepoDigests": [
"gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair@sha256:be4b78aa30969cd88b6ca23841a07282496649e0a98b7645b607b072ddf235a2"
],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-02-06T15:04:27.110797819+01:00",
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "nomad",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8000/tcp": {},
"9000/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
"PYTHON_VERSION=3.9.18",
"PYTHON_PIP_VERSION=23.0.1",
"PYTHON_SETUPTOOLS_VERSION=58.1.0",
"PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/049c52c665e8c5fd1751f942316e0a5c777d304f/public/get-pip.py",
"PYTHON_GET_PIP_SHA256=7cfd4bdc4d475ea971f1c0710a5953bcc704d171f83c797b9529d9974502fcc6",
"PYTHONPATH=/app/plugins"
],
"Cmd": [
"python3"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": {
"/app/.volumes/fs": {}
},
"WorkingDir": "/app",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 1883886525,
"GraphDriver": {
"Data": {
"LowerDir": "/dockerdata/volumes/overlay2/115e904ebcd76c1bccfcda5549ab1681b895babfbe1968d16afa6677daa3bf26/diff:/dockerdata/volumes/overlay2/dc501ca5cfceb3e0334bf4caa3bccd2f2113a06cc9cb94f259362a9f5726b663/diff:/dockerdata/volumes/overlay2/d676ebd69562462e8b2be0084c697feab1a0426a9446cd5f5cdc402210842722/diff:/dockerdata/volumes/overlay2/ad6d73441a8db834f40746356502198b73ec3debfabd0185ce6ed9ede70f056c/diff:/dockerdata/volumes/overlay2/7294a6e95fc74c2a72da9f485fcb67095896fdb103969faba9971e9dd25e4582/diff:/dockerdata/volumes/overlay2/9d8ec0b0c1ca04bfe0293af096cde23979b7f707892ce372f14635718686645c/diff:/dockerdata/volumes/overlay2/af761b214b08f5ba00d1d01db95ff92213a4557a97a40b10ea5832d240f3151e/diff:/dockerdata/volumes/overlay2/433c9a4a3676ffe5e142deb6b01dff36581174cacf86293054770853c6f03ce9/diff:/dockerdata/volumes/overlay2/3efe44a4d06aa62463c0e0cdb7aa697d80891f174c2320c716c6f0fdf7d4e08d/diff:/dockerdata/volumes/overlay2/f5eb910c7070ddb5cab7071b6c841c80a2aecb28a1f0aeed9a51ad48f08b7c17/diff:/dockerdata/volumes/overlay2/a3e9dabd0f9f4cc114303a898c0bb565e9fab76b5ca4af66a91148912df4ff8b/diff:/dockerdata/volumes/overlay2/005be47edaf1c594bfaaff029c48eaf1f246688f4e81f5f2c54e3003168f0af0/diff:/dockerdata/volumes/overlay2/bd771adf438b9ff7270519229801783985b783cd689d88ea11318296fa360deb/diff",
"MergedDir": "/dockerdata/volumes/overlay2/88f956e0ebbe0d92b456f097129f166b0deef067b767b2dd0ff65cef9d847b77/merged",
"UpperDir": "/dockerdata/volumes/overlay2/88f956e0ebbe0d92b456f097129f166b0deef067b767b2dd0ff65cef9d847b77/diff",
"WorkDir": "/dockerdata/volumes/overlay2/88f956e0ebbe0d92b456f097129f166b0deef067b767b2dd0ff65cef9d847b77/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:fb1bd2fc52827db4ce719cc1aafd4a035d68bc71183b3bc39014f23e9e5fa256",
"sha256:da5d55102092b80b04fcb9e6cce42b12f7c53ed72cb1568811576763c9d40786",
"sha256:c4e334227ccac6bda44f5768a5459ad5f8def8e9bb3df0e5323feffd89b9480b",
"sha256:087aa9f40b611f4de7ee0079dfd3600cc038b8be247f82c6abf3b99df7a5624d",
"sha256:18a1e69d7a2d521683b54e7deadc70dbd2b498b68ea2e05115e14c147a5497ff",
"sha256:a0254b855be6bf5ecad7b09b7b97f28be0b2676e9fbb99b04610318d02cbe279",
"sha256:e31f05acf7dd708f0f905abfc969cd41eb63afe3a61cc614a61d3be17aec75df",
"sha256:666aa383b6a2a9c2be8d2f5fa200c20e909ef78b0a9d5f1ce35cb20c9100b100",
"sha256:fd9f50d931645dab893e2da7c6e77d480117012892612785ffad21f3e57c9b04",
"sha256:de8b1961466c64fc0577d8a297b3a38a0ee9da8f4ca4dc416f3e2f7acc9ff7c0",
"sha256:2d93d7cba9761cdc66514bc06946c1ec8724d63ca1632fe15a579d2e679ca7bb",
"sha256:32867b0930e582286f6909b92d3cfca0acea0cdfa91b5c2aeb2eb66a80b59c1d",
"sha256:f2772f2d7f778831050d8d449ee9f0a8930cc9fccfcf3bf23766aa78922f6062",
"sha256:2fb27a2af30200bbeee57f32f116200433ffc2333254894109c642b44739a3ea"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
The last actions i made was first to update Keycloak from 24.0.5. to 25.0.2 (after which all clients were still working). |
|
It should be possible to increase the Python log level in My first thought would be that there is some incompatibility with Keycloak 25.0.2 and the docker image for NOMAD 1.3.3 (but maybe even older versions of NOMAD). To try and reproduce the problem locally, we could spin up a Keycloak service with version 25.0.2 alongside the other services in the default |
I wasn't able to find release cycle date info for keycloack. |
|
Unfortunately, only the latest version of Keycloak receives security fixes (https://github.com/keycloak/keycloak/security/policy#supported-versions), and even if you buy LTS from RedHat, the oldest version they provide backports for is now 22.x (https://access.redhat.com/articles/7033107). Keycloak has a terribly rapid release cycle, i wish they would spend more time on QA and less time on agile feature development. |
|
That's unfortunate, I'll check with Sascha about updating to v25 and let you know if we're able to fix the compatibility issue. |
|
While i still am unable to explain and solve the issue, i can at least provide you with a set of config files for an MWE that reproduces the issue. |
|
Thank you for the MWE. I am able to reproduce this. One thing I've noticed is that the With v25: 
With central NOMAD: 
|
|
Indeed i can confirm that the web storage does not contain an authorization cookie after authorization: |
|
Thank you for confirming, I'll take a look tomorrow with v1.2 to see if we are doing something differently there. |
|
I've used the same docker compose file and used nomad v1.2.1 and it doesn't work there either. Same issue with no Authorization cookie being sent back in response headers by Keycloack. Something probably changed on keycloak side then I'd imagine. Does |
|
If I understood correclty, @behnle already tested that with keycloak 24.0.5 everything worked fine in combination with nomad 1.3.3. So I would assume that something happened in the transition from 24.0.5 to 25.0.2. It might be worthwhile to check if v24.0.2 works, and then check the keycloak changelogs. Maybe 25.0.2 needs us to update our JS keycloak version ( |
|
@blueraft I didn't try NOMAD 1.2.1 yet, that's for sure worth a try. Just have to figure how to pull it from your registry. # keycloak user management
keycloak:
restart: unless-stopped
#image: quay.io/keycloak/keycloak:16.1.1
image: quay.io/keycloak/keycloak:24.0.5
container_name: nomad_oasis_keycloak
environment:
- TZ=Europe/Berlin
- PROXY_ADDRESS_FORWARDING=true
- KEYCLOAK_ADMIN=admin
- KEYCLOAK_ADMIN_PASSWORD=password
- KEYCLOAK_USER=admin
- KEYCLOAK_PASSWORD=password
# - KEYCLOAK_FRONTEND_URL=http:https://localhost/keycloak/auth
- KC_HOSTNAME_STRICT=false
- KC_HTTP_ENABLED=true
- KC_HTTP_PORT=8080
- KC_PROXY=edge
#- KC_LOG_LEVEL=DEBUG
# - KC_HOSTNAME=http:https://localhost/keycloak/
- KC_HOSTNAME_URL=http:https://localhost/keycloak/
#- KEYCLOAK_IMPORT=/opt/keycloak/data/import/nomad-realm.json -Dkeycloak.profile.feature.upload_scripts=enabled"
- KEYCLOAK_EXTRA_ARGS_PREPENDED="--proxy-headers xforwarded --hostname-debug=true --http-enabled true --health-enabled=true --verbose"
#- KEYCLOAK_EXTRA_ARGS="--import-realm --verbose"
command: start-dev --import-realm
#- "-Dkeycloak.import=/opt/keycloak/data/import -Dkeycloak.migration.strategy=IGNORE_EXISTING"
# start-dev --import-realm
volumes:
- keycloak:/opt/keycloak/data
- ./configs/keycloak-import/:/opt/keycloak/data/import:ro
# healthcheck:
# #test:
# test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9000;echo -e 'GET /health/ready HTTP/1.1\r\nhost: http:https://localhost\r\nConnection: close\r\n\r\n' >&3;if [ $? -eq 0 ]; then echo 'Healthcheck Successful';exit 0;else echo 'Healthcheck Failed';exit 1;fi;"]
# # - "CMD"
# # - "curl"
# # - "--fail"
# # - "--silent"
# # - "http:https://127.0.0.1:9990/health/live"
# # - "http:https://keycloak:9000/health/live"
#interval: 10s
#timeout: 10s
#retries: 30
#start_period: 30s
i.e. i am able to perform SSO login. |
In the docker compose file, this would be for the app and the worker: image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:v1.2.1
Good to know this works, probably just some breaking change from 24 to 25 then. |
|
Just checked the 1.3.4 image mentioned in #107 (comment), unfortunately, the problem still seems to persist with exactly the same symptoms. |
Dear NOMAD developers,
i operate a NOMAD Oasis with decentralized user management.
After an update of NOMAD to version 1.3.3 / the latest docker image, i am unable to log into my Oasis.
The setup is as follows:
Observations:
After entering the credentials, i am redirected to NOMAD, however the page looks the same as before. When clicking on "PUBLISH -> UPLOADS", NOMAD tells me "You have to login to use this functionality.", although i just logged in.
NOMAD stack:
images:
The (redacted) keycloak part of
nomad.yaml:There are no obvious errors in the docker-compose logs of Keycloak or NOMAD, there are no errors in the Keycloak GUI, there are no errors in the browser console, it just looks as if NOMAD does not set the session cookie.
Have there been any changes in NOMAD from 1.2 to 1.3 which would require a reconfiguration of the client settings in Keycloak?
What can i do to further track down the root cause of the issue?
The only maybe relevant warning is the following:
If it helps, i can also provide you with the client settings in Keycloak
The text was updated successfully, but these errors were encountered: