-
Notifications
You must be signed in to change notification settings - Fork 369
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confusing documentation around cords or a bug #565
Comments
Also if a cord check fails with healthy (unhealthy) thingy, things seems to fully break as it just exits, so deploy lock is also not released.. which I am not sure makes any sense. |
Hi @olivermt! The cord should work automatically, so you shouldn't need to do anything yourself. It does two things to your docker container:
This allows us to delete the file to force the healthcheck to fail before we stop the container. It sounds maybe from your description that the healthcheck is not failing? But to help debug could you share the logs from where it fails (redacting anything private from them!)? Also do you have any custom configuration under the And could you extract the actual healthcheck from one of the containers? (via |
Hello!
Correct, it just keeps responding healthy. I don't quite understand what you mean with the So there is probably something when you run two services on one host (which is supported, but not recommended?). I am just gonna move this to another small host instead of forcing them onto the same one, but I can hold off if you want me to debug some more for you, if you want to make sure the two(or more)-on-same-host should work. |
@olivermt - the cord files are namespaced by the app and destination so two services shouldn't be an issue (but I've not tested that out). Re: the docker healthcheck - sorry I meant could you run:
It should output something like:
Maybe there's something in the healthcheck that means that it always returns a healthy result? |
I am trying to deploy a django app with kamal. I cannot healthcheck with cord. The container seems unhealthy. The command requested above gives the following:
I also added an
|
Facing the exact same issue with a NextJs App with Kamal + Traefik! |
Do you guys have any updates in this issue? I'm getting the same here, my "replaced" containers never become |
The cord file is a workaround to allow us to force the container to be unhealthy while it still can handle requests. Sending How it should work is: Before starting a new container Kamal creates a new directory on the host and adds an empty file
The when starting the new container, it will map the directory into it (by default to This will look like:
Then when that new container has started up and is healthy, Kamal deletes the cord file from the old container, which should cause the healthcheck to fail and the container to be marked as unhealthy. This will appear as:
Since the problem is that the container is not getting marked as unhealthy, that suggests the healthcheck is continuing to succeed, I guess either because the cord file is not deleted, or because we have a healthcheck that doesn't fail when it is deleted. The way to debug this is to find out exactly what the health check is and then see what happens when you run it manually in the container. So from the host run:
Then exec into the container and see why that command doesn't return an error code:
|
Hello,
I have been struggling a lot with getting kamal >1.0.0 working with cords.
I am not 100% sure if there is a bug or if I am doing something wrong.
Either way I think my questions / observations should either lead to a bugfix or to improved documentation text :)
Are cords automatic?
The docs leave me very unsure if we should be handling the cord and responding something non-200 OK in the healthcheck or if the healthchecker itself knows how to look for the cord. The release upgrade text reads like as if the healthcheck now simply uses the cord, but when we ran our second deploy on kamal 1.1.0 after upgrading from Mrsk it started failing with the very strange "failed healthy unhealthy" or something along those lines.
First of all, this error message could probably be a lot more informative like "cord is configured but container is still responding healthy". And again, there is nothing explicitly telling you to respond to this file to send different statuses to signal to Traefik whats up.
If cord is not automatic, initial deploy will always fail
Since the cord is only there on a rolling deploy, an initial deploy does not get a cord set up. I discovered this after doing a
kamal app remove
and doing a fresh deploy trying to handle the cord myself. If you are supposed to handle the cord yourself then this is currently broken for any new users and/or new server setups.I have no idea how to reliably detect if a deploy is an initial one so I can't really do any heuristics to say if a missing cord should result in a 404 or not.
Edit: There is a kamal-cord directory on the fresh install, as I guess its a default container mount now, but the build logs have zero trace of cords being created and the regular
touch ..cordstuff../cord
Edit2: cc @djmb as the cord PR author :)
The text was updated successfully, but these errors were encountered: