Container corruption can cause the supervisor to crash

[pipex]

Seen on a device where balena inspect <container> returns

Error response from daemon: readlink /var/lib/docker/overlay2/l/X4CB3KC5BWUW5JOBOFRZ2IQG24: no such file or directory

This causes the getAll call to throw here

balena-supervisor/src/compose/service-manager.ts

Line 82 in df02724

throw e;

Following the call upwards, this error is obtained when doing an applyTarget

balena-supervisor/src/device-state.ts

Line 869 in df02724

return applyTarget({ force, initial });

which calls getCurrentState here

balena-supervisor/src/device-state.ts

Line 697 in df02724

getCurrentState(),

which eventually calls getAll

None of the calls are wrapped in a try catch, causing the supervisor to crash as below.

Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [debug]   Handling of local mode switch is completed
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]   Uncaught exception: Error: (HTTP code 500) server error - readlink /var/lib/docker/overlay2/l/X4CB3KC5BWUW5JOBOFRZ2IQG24: no such file or directory
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]         at /usr/src/app/dist/app.js:2:643810
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at /usr/src/app/dist/app.js:2:643742
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at Modem.buildPayload (/usr/src/app/dist/app.js:2:643762)
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:2:643015)
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at IncomingMessage.emit (node:events:525:35)
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at endReadableNT (node:internal/streams/readable:1358:12)
Dec 13 19:36:37 1ed3b30 balena-supervisor[10470]: [error]       at processTicksAndRejections (node:internal/process/task_queues:83:21)
Dec 13 19:36:37 1ed3b30 systemd[1]: balena-supervisor.service: Main process exited, code=exited, status=1/FAILURE

The problem is that is hard to know what to do in this situation as this happens even before we can apply a target state, so even if the exception is caught, at best the supervisor (with the existing architecture) will have to keep looping until somebody can manually fix the device.

This is probably better though as at least the supervisor can report on the error and keep reporting the system state

[jellyfish-bot]

[pipex] This has attached https://jel.ly.fish/2b9105be-8bf3-4fad-a75e-25cc17816baf