[BUG] - Jupyter scheduler doesn't run successfully unless users are in developer group.

[Adam-D-Lewis]

Describe the bug

https://www.nebari.dev/docs/tutorials/jupyter-scheduler/ states
Only users in the admin or developer groups will have access to create notebook jobs. For more information regarding users and groups, please visit [How to configure Keycloak](https://www.nebari.dev/docs/how-tos/configuring-keycloak)
but I had a user that was a member of /users, /analyst, and /admin groups, but Jupyter scheduler would fail for him. After I added him to developer group, things worked as expected for him.

Expected behavior

Update docs or fix bug

OS and architecture in which you are running Nebari

Linux x86-64

How to Reproduce the problem?

See above

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

v2024.4.1 of Nebari

[nkaretnikov]

@Adam-D-Lewis Hey Adam, I couldn't reproduce the behavior your describe (Nebari and images 2024.5.1). For me, it works with admin, without developer. I can think of two reasons as to why you ran into this:

• It was actually a papermill issue. It needs to be in the env for the job to succeed. This is fixed in Mention if papermill is not installed in failed status argo-jupyter-scheduler#10
• You didn't restart the JupyterLab server and didn't log out/in via Keycloak after making changes to groups. This caused an old token to be used, causing a permission issue. If that's the case, an error message is raised, which you can see in the job details. I did create a docs PR to further clarify how the groups should be configured and how to troubleshoot permission issues: Add more info on the scheduler group issues nebari-docs#467

Error message:

The code that sets the token is in src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py. See the argo_sa part that's set based on available groups:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py

Line 400 in 619684d

def profile_argo_token(groups):

Please let me know if there are any questions. Otherwise, I think this can be closed.

[Adam-D-Lewis]

Thanks for looking into it Nikita. I'm confident the issue wasn't with paper mill. I don't think it was a restart issue either, but if we can't reproduce I'm fine to have it closed for now.

[kcpevey]

@nkaretnikov can you confirm that admin is required? If so, I will likely open up a new issue to fix that. Or, rather, adjust it to a more fine grained permission model once the permissions revamp is further along.

[nkaretnikov]

@kcpevey I apologize for not being clear. It's either admin or developer as the current docs say.