Merging of split VirtualServices using the mesh gateway (sidecars)

[erSitzt]

Hi,

we deploy lots of different version of the same services and i have to split up the deployments into multiple VirtualServices and DestinationRules using the same host but routing http traffic via the url prefix.

like so:

• svc-a.domain.com/v1.0
• svc-a.domain.com/v1.1
• svc-a.domain.com/v1.2

All services use an istio gateway and the "mesh" gateway, because the services are talking to each other a lot.
As merging does not work in the sidecars only one of the three entries above will work as long as i keep them attached to the "mesh"-gateway.

If i remove the "mesh"-gateway from the VirtualServices everything works as planned but all traffic goes through my ingress-gateway and the "Visibility" benefit of the Service Mesh especially in Kiali is gone because i cant see svc-a talking to svc-b directly

I didnt find any clues or get any answers why host merging works in gateways but not the sidecar, but i think this is a feature a lot of people will need/use
I read a comment somewhere here about preventing unpredictable behavior which can result from merging, but i think this should be up to the user. These are configs we used with haproxy for years, they are only unpredictable if you f##k up, but that goes for every config ;)

Any feedback on this is welcome

[rshriram]

Hi.. virtual service merging is not implemented for the mesh gateways as of now. So, unfortunately to get the behavior you desire, you need to have one large virtual service/destination rule. We can add this to the feature request list and see if we can knock this off in the next release. cc @nrjpoddar to track.

[erSitzt]

thanks for the info @rshriram this would be great.
Creating one large VirtualService/destinationRule in our deployments is really hard because we are running up to 10 versions of the same service. because we have to support older app version for a long time here.

[robert-blackman]

Hi.. virtual service merging is not implemented for the mesh gateways as of now. So, unfortunately to get the behavior you desire, you need to have one large virtual service/destination rule. We can add this to the feature request list and see if we can knock this off in the next release. cc @nrjpoddar to track.

Hey @rshriram any movement on this?

[erSitzt]

Is there any more info on why VirtualService merging was/is done differently in gateways and sidecars ? And why DestinationRules merging works in both ?
Just to understand the situation ;)

[luisdalves]

Hey Guys,

I'm stuck with the same issue. Is there any advance on this or maybe a workaround?

Thanks

[erSitzt]

Please dont close this issue.

If you feel this issue or pull request deserves attention, please reopen the issue
How ?

@rshriram

[Leo7654]

Do you have any plans on how to handle this?

[luisdalves]

Hi @rshriram / @nrjpoddar
Please, can you reopen this issue? I think is a good feature to discuss.

Thanks

[nrjpoddar]

@luisdalves We have added forward reference/delegate feature which kind of is in the similar vein. Will that help your use case?

[erSitzt]

@nrjpoddar the delegates do not solve the problem when single deployments run.
We tag a version v1.0.0 and a deployment runs that renders all templates needed to run this service in kubernetes

• deployment
• service
• virtualservice
• destinationrule
• configmaps

there is no other instance on top of this service where the virtualservice with the delegates could be rendered.
any arbitraty version of our service could be deployed at any time and im not sure there is a good way to handle this via delegates.

...and as virtualservice merging is already there, at least in gateways, and nobody really answered my question on why merging in sidecars is a problem, i urge you to please reopen this issue

[billyshambrook]

I added another user case to a similar issue in the smi-spec repo - servicemeshinterface/smi-spec#59 (comment)

One user scenario we have for this is to do with "branch" deployments whereby you want to create a unique deployment of a service for each feature branch that is open for engineers to test their changes before merging to "main" branch.

In a microservice environment it's tricky to deal with routing traffic through the services but wanting a particular request to hit a specific "branch" deployment for a given service.

One way we are solving this is by using header based routing. When we deploy the "branch" environment for Service B we add an additional routing logic to only receive requests with the header x-review:my-branch-name. When an engineer makes a request to Service A with that header, the request will go to the "main" branch deployment of Service A. If Service A then makes a request to Service B, Service A will passthrough the x-review:my-branch-name header to Service B. The service mesh would then see that header and route the request to the "branch" deployment of Service B.

We deploy each "branch" deployment as a separate Helm deployment so would be nice if we could have a separate TrafficSplit in each deployment which will reference the same service but introduce a new header backend route.

Our current workaround is to run a custom controller that is listening for virtualservice changes and will merge them together. This works but is quite a large workaround and are a number of prerequisites on how a virtualservice is constructed and deployed so that they can be safely merged.

This "preview environment routing via headers" concept seems to be an emerging practice that is becoming popular by other companies/vendors, for example https://github.com/microsoft/mindaro and https://github.com/alibaba/virtual-environment.

[nrjpoddar]

@howardjohn @ramaraochavali let's discuss this during the next WG meeting.

[nrjpoddar]

@erSitzt can you attend this week's Networking WG meeting to present your views on this? I have added it to the agenda. Meeting's at 10 AM MST Thursday 10/01/2020. Thanks!

[erSitzt]

@nrjpoddar I'll try to attend, but can't promise.
How is the meeting held ?

[erSitzt]

Btw. im not sure if the problem was limited to VirtualService or if DestinationRule was/is causing problems too...

This was one of my initial questions on the forums:
https://discuss.istio.io/t/multiple-destinationrules-for-the-same-host/5416/4

[erSitzt]

@nrjpoddar i dont where/how to attend this meeting..

[nrjpoddar]

@erSitzt https://meet.google.com/xjj-ujhi-qfk

[erSitzt]

@nrjpoddar
FYI
As suggested in the WG meeting i had a look at https://github.com/kubernetes-sigs/service-apis and asked around a bit
https://kubernetes.slack.com/archives/CR0H13KGA/p1601923906046100

I'm not sure if this helps as i dont know if there is something like the "mesh"-gateway in their concept ?

Thanks again for hearing me out :)

[okhaliavka]

@billyshambrook We have exactly the same use case. Is your controller publicly available by any chance?

[jdloft]

This is especially important given that HTTPRoute support doesn't have rewrite working yet as far as I can tell. The routes never get created when a URLRewrite filter is added.

I imagine we're far off from having complete gateway API stability and support so this shouldn't be reliant on that.

[hzxuzhonghu]

Splitting of VirtualService across namespaces and merging them based on host by mesh gateway is important for solving multi-tenant service-to-service routing use cases.

From my perspective, merging virtualservices across namespaces violates multi tenants requirement. We should make namespace as a soft boundary. However we have other way delegate virtualService which will allow to explictly reference another vs in different namespace.

If no explictly referenced, this could be very error-prone. There is a very similar issue we met recently for service entries with same hostname but resides across namespaces.

[farooqashraf1]

It is also error-prone to have one huge VirtualService object that must be patched with every service onboard offboard operation. Having separate VS for each service namespace and automatic merge by mesh provides a clean onboard offboard process with a "kubectl apply" or "kubectl delete".

[howardjohn]

Why do you need *mesh* routes with a shared hostname in different namespaces? Typically we see each hostname is owned by a single namespace.

…

On Fri, Sep 30, 2022 at 10:46 AM farooq ***@***.***> wrote: It is also error-prone to have one huge VirtualService object that must be patched with every service onboard offboard operation. Having separate VS for each service namespace and automatic merge by mesh provides a clean onboard offboard process with a "kubectl apply" or "kubectl delete". — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXI2O6LAHR7WMFGYMKDWA4RORANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[farooqashraf1]

Why do you need mesh routes with a shared hostname in different namespaces? Typically we see each hostname is owned by a single namespace.
…
On Fri, Sep 30, 2022 at 10:46 AM farooq @.> wrote: It is also error-prone to have one huge VirtualService object that must be patched with every service onboard offboard operation. Having separate VS for each service namespace and automatic merge by mesh provides a clean onboard offboard process with a "kubectl apply" or "kubectl delete". — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXI2O6LAHR7WMFGYMKDWA4RORANCNFSM4MJY2IBA . You are receiving this because you were mentioned.Message ID: @.>

The use case is for different tenant tiers of the same service, each deployed in a separate namespace for isolation and simplified provisioning. The shared hostname is important to allow seamless routing to the right tier based on http headers.

[amitde69]

its not only about merging across namespaces but rather decoupling all service resources per service helm chart for example.
assuming we are deploying new services everyday its a hustle to also update a central VirtualService and also error-prone.

[erSitzt]

@farooqashraf1 @howardjohn
Not quite sure if we are still on topic ? :)

Original issue is/was related to different behavior of VirtualService merging for the same hostname in the ingress gateway and the sidecars ( mesh gateway).

And because of this behavior there were only two options at the time ( to my knowlegde ) for mesh internal communication:

1. using unique hostnames when accessing other hosts via the mesh gateway, so merging would not overwrite the previous settings for the same hostname ( which works fine for me most of the time, only a bit confusing for devs, because they have to look up internal names sometimes )
2. manage a large single VirtualService outside of the scope of the helm/service deployments

and @amitde69, yes that was the gist of this issue in the first place.
Merging across namespaces was not, it sounds more like a feature request than related to this issue.

[erSitzt]

Just to clarify

These two VirtualServices would work using the istio-gateways/standard-gateway but not the mesh-gateway
for these urls

• artikel-neo.prod.mydomain.com/v1.0.0
• artikel-neo.prod.mydomain.com/v1.2.0

So mesh internal communication is using the unique hostnames, as their config is not lost during sidecar merging

• artikel-v1-0-0.neo.svc.cluster.local/v1.0.0
• artikel-v1-0-0.neo.svc.cluster.local/v1.2.0
  or
• artikel-v1-0-0-neo.prod.mydomain.com/v1.0.0
• artikel-v1-0-0-neo.prod.mydomain.com/v1.2.0

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: artikel-v1-0-0
  namespace: neo
spec:
  gateways:
  - istio-gateways/standard-gateway
  - mesh
  hosts:
  - artikel-neo.prod.mydomain.com
  - artikel-v1-0-0.neo.svc.cluster.local
  - artikel-v1-0-0-neo.prod.mydomain.com
  http:
  - match:
    - uri:
        prefix: /v1.0.0/
    - uri:
        prefix: /v1.0.0
    rewrite:
      uri: /
    route:
    - destination:
        host: artikel-v1-0-0
        port:
          number: 8888
        subset: v1-0-0

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: artikel-v1-2-0
  namespace: neo
spec:
  gateways:
  - istio-gateways/standard-gateway
  - mesh
  hosts:
  - artikel-neo.prod.mydomain.com
  - artikel-v1-2-0.neo.svc.cluster.local
  - artikel-v1-2-0-neo.prod.mydomain.com
  http:
  - match:
    - uri:
        prefix: /v1.2.0/
    - uri:
        prefix: /v1.2.0
    rewrite:
      uri: /
    route:
    - destination:
        host: artikel-v1-2-0
        port:
          number: 8888
        subset: v1-2-0

And btw. these two VirtualServices are coming from two completely separate deployments ( helm template ) with no dependencies except sharing the same service fqdn/hostname

[howardjohn]

I see. That is more typical in gateways. The approach that would be expected is to have one owner of the hostname which can then delegate routing to the other namespaces. Just having all namespaces blindly merge across hostname is not something we will likely do.

…

On Fri, Sep 30, 2022 at 12:06 PM farooq ***@***.***> wrote: Why do you need *mesh* routes with a shared hostname in different namespaces? Typically we see each hostname is owned by a single namespace. … <#m_-2061024698235847054_> On Fri, Sep 30, 2022 at 10:46 AM farooq *@*.*> wrote: It is also error-prone to have one huge VirtualService object that must be patched with every service onboard offboard operation. Having separate VS for each service namespace and automatic merge by mesh provides a clean onboard offboard process with a "kubectl apply" or "kubectl delete". — Reply to this email directly, view it on GitHub <#22997 (comment) <#22997 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXI2O6LAHR7WMFGYMKDWA4RORANCNFSM4MJY2IBA <https://github.com/notifications/unsubscribe-auth/AAEYGXI2O6LAHR7WMFGYMKDWA4RORANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: @.*> The use case is for different tenant tiers of the same service, each deployed in a separate namespace for isolation and simplified provisioning. The shared hostname is important to allow seamless routing to the right tier based on http headers. — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNGH3VEMSIBVQZQKPTWA422TANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[howardjohn]

Sunsets should already merge

…

On Wed, Apr 6, 2022, 12:57 PM Zach Bintliff ***@***.***> wrote: @fperearodriguez <https://github.com/fperearodriguez> , I'm also interested in know if they will merge destination rules. — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNELIK3667PSEF7KPDVDXGBNANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[farooqashraf1]

Definitely a feature similar to Gateway. In fact a Gateway-like object that can be deployed inside the mesh with a mechanism to allow VS objects to associate themselves would be ideal.

The use case probably doesn't fit into the subset model.

Also, k8s hierarchical namespaces may be another solution, if Istio supports it. In that case all VS objects could be under the same parent namespace.

[jcam]

Trying to understand where this feature is at... my use case is within a single namespace (so not the multi-namespace tangent above). I am currently using kong gateway as the ingress gateway, as it has the multiple different auth plugins that I need, with the kong pods in the mesh, but then i need to route to any number of branch-deployed-integration subsets based on a header value.
(i.e. if the branch header is set to staging route to the staging subset)
I can't route dynamically, the virtualservice destination object doesn't support variable lookup.
I would like to just have multiple virtualservices that get merged on the hostname.

On March 25, howardjohn said "in 1.14 this will be supported for HTTPRoutes"

I'm on 1.15, and it doesn't seem to happen. Is there a requirement to use the Gateway api for this somehow? The documentation seems to still be sparse, unless i'm just looking in the wrong place.

[gard-semvox]

@jcam, that's pretty much the same use case we try to solve. Instead of using the Kong gateway, we use the Istio ingress gateway. This gateway supports merging separated VirtualService definitions based on the hostname for incoming traffic. That is working fine.

But internal communication (mesh gateway) that should be routed the same way (staging header -> staging subset) is not working at all. If I understand it correctly, merging separated VirtualService definitions based on the hostname is not supported by the sidecar proxies if you use the Istio CRDs.

Your incoming traffic is handled by the Kong gateway and is forwarded internally. Hence, the merge is not support.

Please let me know, if you managed to solve this issue with the Gateway API stuff ;)

[jcam]

I just looked and actually the v1.16 documentation talks a lot more about using the gateway API for traffic routing here:
https://istio.io/latest/docs/tasks/traffic-management/request-routing/

I am not running 1.16, and it isn't clear whether this was just a change in the 1.16 docs or if there are a lot of under the hood changes in 1.16 to make it work. I think i'll give this a shot with one of my dev clusters on 1.15.3 first and then upgrade to 1.16 if it doesn't work at all. no indication what kubernetes version is required for this new gateway API stuff either. I'm hoping it will work on 1.15 as i have a number of workloads that my team unfortunately hardcoded to 1.13, so at the moment i'm running a 1.13 and 1.15 istiod... which is barely "supported", while 1.13+1.16 is not.

[jcam]

Ah, "Configuring internal mesh traffic is an experimental feature of the Gateway API"

So we're still likely months away from being able to use this

[lsibilla]

Ah, "Configuring internal mesh traffic is an experimental feature of the Gateway API"

So we're still likely months away from being able to use this

I believe this note refers to Kubernetes Gateway API. Not the "istio classic" method which uses VirtualServices.

[jcam]

Ah, "Configuring internal mesh traffic is an experimental feature of the Gateway API"
So we're still likely months away from being able to use this

I believe this note refers to Kubernetes Gateway API. Not the "istio classic" method which uses VirtualServices.

It does, yes. howardjohn wrote in March 2022 that merging could be accomplished using the Gateway API as of Istio 1.14. I also remember reading that someone stated that there is no intention to complete this issue, to update mesh internal VirtualServices to support merging, since the Gateway construct is the future and already supports it. But now I can't find that comment anywhere so maybe I imagined it, and Gateway API for mesh internal endpoints is still experimental, which I would consider pre-alpha...

[zbintliff]

But doesn't that only solve when traffic is coming into the mesh and going through the gateway? Is the gateway involved for direct service to service calls for two services on the mesh? This may be what you were trying to say is pre-alpha, but I don't think it would be desired for traffic on the mesh to also go back through the gateway...or am I misunderstanding something?

[jcam]

Gateway API is not Istio Gateway.
The Gateway API is a new kubernetes api for configuring things... it replaces Ingress and it also will replace VirtualService and a number of other things, falling under gateway.networking.k8s.io. You can read more about it here on the Istio docs and here on the gateway api docs

But, it is a long time in the making, and seems to have a long way to go. Istio has moved their gateway API support to Beta, but the API itself is still quite alpha, not included in kubernetes mainline at all, and the intra-mesh HTTPRoute configuration functionality that will replace VirtualService is still experimental and relies on functionality that hasn't been even approved/integrated into the alpha gateway API standards.

[zbintliff]

Oh my mistake. Thank for the docs!

[rocasx]

monimesl/istio-virtualservice-merger This repository provides a good solution for merging virtualServices.

[hakan-77]

Are there any future plans to merge virtual services at the sidecar level as well? Or is it an absolute no-go?

Our platform routing rules/hosts are quite dynamic and decoupled. It is not really possible to combine virtual services. And destination rules are not flexible enough.

To use multiple virtual services per host, we have to route all traffic through an external gateway, and this means we need an external DNS. This is causing issues at development time.

E.g. When a sidecar tries to hit http:https://localhost/some/addres/ it will look for it at 12.0.0.1 and fail. And if we add "mesh" gateway, only one virtual service per host is active instead of the 10+ we need.

As a result, we have to move all development/testing to a cloud environment, which is not ideal..

Other than that, thank you for the amazing work!

[howardjohn]

HTTPRoute, which is basically virtual service v2, already supports this. I wouldn't say it's impossible for virtual service to ever do so but it doesn't seem likely given we already have a solution. It's quite hard to add to virtual service due to an original mistake to tie ordering if routes to a linear list, which makes it unclear how to merge. HTTPRoute handles this by having an order defined based on the match itself, so it can easily be merged

…

On Thu, Jul 27, 2023, 2:31 PM Hakan ***@***.***> wrote: Are there any future plans to merge virtual services at the sidecar level as well? Or is it an absolute no-go? Our platform routing rules/hosts are quite dynamic and decoupled. It is not really possible to combine virtual services. And destination rules are not flexible enough. To use multiple virtual services per host, we have to route all traffic through an external gateway, and this means we need an external DNS. This is causing issues at development time. E.g. When a sidecar tries to hit http:https://localhost/some/addres/ it will look for it at 12.0.0.1 and fail. And if we add "mesh" gateway, only one virtual service per host is active instead of the 10+ we need. As a result, we have to move all development/testing to a cloud environment, which is not ideal.. Other than that, thank you for the amazing work! — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNVRZPKQNOG2BUOBDTXSLM47ANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hakan-77]

@howardjohn, thank you.

Just to be 100% sure, you mean HTTPRoute of gateway.networking.k8s.io right?

And not the HTTPRoute of istio Virtual Service.

[howardjohn]

Yep - thanks for the clarification. Man, every word is overloaded ....

…

On Thu, Jul 27, 2023 at 3:24 PM Hakan ***@***.***> wrote: @howardjohn <https://github.com/howardjohn>, thank you. Just to be 100% sure, you mean HTTPRoute of gateway.networking.k8s.io right? And not the HTTPRoute of istio Virtual Service. <https://istio.io/latest/docs/reference/config/networking/virtual-service/#HTTPRoute> — Reply to this email directly, view it on GitHub <#22997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXOPIJUTVTITADESN43XSLTBHANCNFSM4MJY2IBA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CharlieC3]

I am currently using kong gateway as the ingress gateway, as it has the multiple different auth plugins that I need, with the kong pods in the mesh, but then i need to route to any number of branch-deployed-integration subsets based on a header value.

@jcam I have a similar setup I'm working through right now. It's not ideal, but until the Gateway API is better supported in Istio, for now I've ended up just routing all external traffic through the Kong Gateway, then through the Istio Gateway.
That way I can still leverage the Kong plugin features while the Istio Gateway keeps mesh complexity at a minimum for traffic entering the cluster.

Were you able to find another way through this?

[jcam]

Nope. I haven't changed my setup. I spent a little time with Gateway API in one of my test environments but it broke things and the next version had breaking changes, so I ripped it all back out.

[isaac88]

Hello @howardjohn
We're facing the same issue IST0109 ConflictingMeshGatewayVirtualServiceHosts
.
Context:
We want to have different external internet services retries and timeouts configurations (VS) per Kubernetes application mesh services.

We're using different VirtualServices per Kubernetes applications inside the same namespace with the same Host (SE external internet service) where we use HTTPMatchRequest sourceLabels to isolate the routing configurations pushed to the different proxies applications inside the same namespace.

We're concerns since the istioctl analyse raise that on Error level, and we don't want to have any production issues.

What do you recommend us to handle that issue?