Investigate intermittent 503's

[cbron]

Darren/Dax reported seeing some 503's while clicking around rio services

[daxmc99]

no healthy upstream
Reproduction with Chrome:

• Have a service scaled to zero.
• Attempt to visit a service endpoint using Chrome with dev tools enabled and disabling the cache
• While waiting for the request to scale, make another request (refresh).
• This and subsequent requests will return no healthy upstream and 503 errors.

[rancher-max]

#584 (comment)

[daxmc99]

After working with @rancher-max and @StrongMonkey it looks like the root of the problem is that the service version endpoint $service-v0-on.rio.io is not being populated correctly.

The SSL_SYSCALL error manifests itself on services scaled to zero, when there is no virtual service for that version endpoint. Sometimes, it is dynamically created but the correct behavior should be for the virtual service to always exist even if a service is scaled to zero.

Proposed solution:
All services in Rio should have a version endpoint regardless of scale.

[rancher-max]

This has been fixed in latest master, commit: c0f480fe. The scenario this fixes is, as mentioned above:

1. Have a single service on an app, and have it with min scale of 0 and max scale of 1 or greater
   2 Wait for the service to be at observed scale of 0, then query the app endpoint (curl is fine)
2. While the pod is initializing, make more queries to the app endpoint

This was previously returning no healthy upstream on at least one of those subsequent requests, but now returns the correct result:

$ for x in {1..20} ; do curl https://other-as-default.7oign7.on-rio.io ; done
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6
Hi there, I am rio:production6