Send notifications for dependency failures

[AlanCoding]

SUMMARY

I had a report that notifications were not firing for cases when an update-on-launch dependency failed. This is fairly easily verifiable. It's somewhat of a stupidly simple case where handle_work_error sent websockets, but not notifications.

But then you get into the weeds of how to fix it, and realize there are race conditions, consider how the callback receiver handles this:

        # Update host_status_counts while holding the row lock
        with transaction.atomic():
            uj = UnifiedJob.objects.select_for_update().get(pk=job_identifier)
            uj.host_status_counts = host_status_counts
            uj.save(update_fields=['host_status_counts'])

This solves a contention problem between the callback receiver and the dispatcher.

But wait - here, we have a job that was created, spawned a project update, and then is getting marked failed by the dispatcher running the project update. At no point did either the dispatcher or the callback receiver go through the run path for that job. There shouldn't be any contention problem!

With the errback/callback methods as they are, there is still at least the vague possibility of the job starting because the project update fails after updating its final status... but this is going too far beyond realistic concerns. The much more realistic concern is that we set "Previous task failed task:" twice so sending notifications here would introduce a new contention problem.

Systems-wise, why are we even dealing with the contention problems? What if, instead, we only triggered the failure logic in a reliable system that has a periodic fallback... this would be the task manager. I have already argued that the errback methods are not needed, so this is a good time to delete them.

Missing notification problem solved, contention problems are not a risk with this solution.

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

[fosterseth]

good code cleanup

[fosterseth]

💯