fix(executor): handle podlog in deadlineExceed termination. Fixes #7092 #7081

[tczhao]

Fixes #7092

At the moment, when deadlineexceed, the wait container will be stuck at the Wait step and won't proceed further to savelog and annotate output.
This is because when deadlineexceed, the main container.status.termianted is always nil, and k8sapi wait will only proceed when terminated becomes not nil

Fixes #7081

---

Don't bother creating a PR until you've done this:

• Run make pre-commit -B to fix codegen, lint, and commit message problems.

Create your PR as a draft.

• Your PR needs to pass the required checks before it can be approved. If the check is not required (e.g. E2E tests) it
  does not need to pass.
• Once required tests have passed, you can make it "Ready for review".
• Say how how you tested your changes. If you changed the UI, attach screenshots.

Tips:

• If changes were requested, and you've made them, then dismiss the review to get it looked at again.
• Add you organization to USERS.md if you like.
• You can ask for help!

[sarabala1979]

@tczhao please fix the conflict

[tczhao]

Thanks @sarabala1979 , fixed conflict

[alexec]

Needs master merge again.

[tczhao]

any further comments?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tczhao]

Hello @alexec , I've made some changes, could you have a review again?

Sorry was on holiday mode and took a while to get back.

[alexec]

Rate limiting is set by DEFAULT_REQUEUE_TIME. This defaults to 10s. When run make start it is set to 100ms to speed up tests. Try running make start DEFAULT_REQUEUE_TIME=10s?

[tczhao]

setting DEFAULT_REQUEUE_TIME=1s works

I'm wondering what's the best strategy here or if there are any tricks available
seems with the existing change, its failing e2e because DEFAULT_REQUEUE_TIME=100ms

[alexec]

Change e2e test set-up?

I'm wondering what's the best strategy here or if there are any tricks available
seems with the existing change, its failing e2e because DEFAULT_REQUEUE_TIME=100ms

Change test set-up. If tests run much longer, then we discus. If not, happy day.

[alexec]

@tczhao I plan to refactor assessNodeStatus. Perhaps you'd like to review my PR #7996 and see if that fixes your issue.

The k8sapi executer will be removed in v3.4 or v3.5. I don't know how much work we should do on it.

[tczhao]

Change e2e test set-up?

I'm wondering what's the best strategy here or if there are any tricks available
seems with the existing change, its failing e2e because DEFAULT_REQUEUE_TIME=100ms

Change test set-up. If tests run much longer, then we discus. If not, happy day.

Change DEFAULT_REQUEUE_TIME from 100ms to 1s results in about 12% increase in test time
test time is recorded from all e2e Run make test- scripts

[tczhao]

@tczhao I plan to refactor assessNodeStatus. Perhaps you'd like to review my PR #7996 and see if that fixes your issue.

The k8sapi executer will be removed in v3.4 or v3.5. I don't know how much work we should do on it.

• That Re-factor assessNodeStatus  #7996 PR solves the issue of not annotating artifact
  but we still need DEFAULT_REQUEUE_TIME to be at 1s to pass the ActiveDeadlineSeconds test

• I'm not concerned too much on the k8sapi part,
  maybe still a good patch on 3.3,
  I will leave it up to you to make a judgement on if we need that

[alexec]

Please resolve conflicts.