Prefect with Jupyter in cubes

[cram0s]

Bug summary

Good afternoon! I am working with jupiter and prefect in cubes (Kubernetes). In this architecture there is a problem with the deploy method - although Docker is deployed in Kubernetes, Jupyter does not see it and the only solution to the problem is to deploy Docker in Docker, which is incorrect. The second problem is trying to deploy via prefect.yml: in the prefect-docker.build step the error please ensure docker is running occurs again, which also rejects this option. The only option left is to use CI/CD to deploy to prefect, but I would also like to be able to deploy directly from jupiter. Please advise how to proceed in this situation

Version info

prefect version 3.0.5

Additional context

No response

[zzstoatzz]

hi @cram0s - I've converted this to an issue because there isn't a specific problem expressed, and I think there might be answers to your questions in the docs

although Docker is deployed in Kubernetes, Jupyter does not see it and the only solution to the problem is to deploy Docker in Docker

I'm not sure what you mean by "Jupyter does not see it" but in general we do not require that you run Docker in Docker.

Generally speaking, you can follow this guide, where you:

• install a prefect worker in your cluster via helm
  • this worker listens for scheduled flow runs, and when it finds one, it submits the flow run for execution to your cluster as a pod.
• will need a third-party code storage location, like github, S3, or your docker images themselves. i.e. a Jupyter notebook itself will not work as the storage location, you'd need to push a .py file with your code to github, s3 or bake it into your image as linked above.

here are some examples and a primer video on workers and work pools if that's helpful

if you think there is a specific issue with prefect here, please feel free to let me know