Spans are not connected after upgrading spring-web-starter

[lubinson]

Dear Community,

Different services spans can not be connected after upgrade below old version

opentracing-spring-jaeger-starter:0.1.4
opentracing-spring-web-starter:0.3.3

to new version

opentracing-spring-jaeger-starter:2.0.0
opentracing-spring-web-starter:2.1.0

Before the upgrading, everything is ok.

For example, Service A call B and C, we inject the tracing context to the HTTP header when calling Service B and C. The sampler is ratelimiting 1. Span Calling Service B and Calling Service C are created in Service A and under the root span Service A. Span Service B is recorded in service B. Span Service C is recorded in service C.

Before upgrading, we got below three level tracing: one root span, two level 2 spans and two level 3 spans,

Service A
  Calling Service B
    Service B
  Calling Service C
    Service C

Even we changed the sampler to any, above tracing is working as expected.

After upgrading, we can only got 1 root span and two level 2 spans which all belong to service A:

Service A
  Calling Service B
  Calling Service C

But when we tried to change the sampler to const 1 in Service A. we got three individual tracing:
1:

Service A
  Calling Service B
  Calling Service C

2:

GET
  Service B

3:

POST
  Service C

in tracing 2 and 3, there has two unknown span: GET and POST. When looked at the span detail. I found the field Component has the value java-spring-webclient which is also not seen in old version. I am guessing the Spring Webclient created a root span solely. I didn't look at the http packet to see what exactly the header are, but from the result, I think Service B and Service C treat the span as the root span. Is our injected span got overwritten? and also sometimes the results related to the sampler?

I am not sure this is the changes to new version or other changes resulted in the weird behavior. Thanks for your information and opinion.

[pavolloffay]

hi @lubinson, what client/technology are you using to create request from service A to services B and C?

Ultimately would you be able to share a small reproducer?

[lubinson]

@pavolloffay

Thanks for your reply. After a deep looking, I found the Spring Webflux Webclient is auto injected with a filter TracingExchangeFilterFunction. This actually starts a new Tracing (Tracing 2 and Tracing 3) to Service B or Service C, which the tracing starts with span GET and POST. And I think this injection overwrote the tracing context we injected so the linked spans are broken.

After I removed the auto injected filter, everything is ok as before. I think this is a change of new version. Its better to make it configurable, especially for upgrading cases. And further, despite of my case, I am not sure this injection created span will connect spans before and after?

The reproducer is a little bit hard, because it is integrated with Spring Boot and other code. But the basic case is All Services are spring boot applications instrumented with Jaeger, Client call Service A, Service A processes its business and then uses auto injected WebClient.builder() (this Webclient is the one mentioned above) to call Service B and Service C. Then you will see the happening. Pls be note that Service A processing has multiple threads switching, we use some context to take the root tracing context and inject it to the HTTP header then use Webclient to call Service B and C.

Thanks.

[geoand]

@lubinson Since you have already done an initial investigation and have an idea of how to fix the problem, care to create a PR for it?

Thanks