Thrift client throws: NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer

[dziemba]

When using the thrift client, it throws a NoSuchMethodError when running on Java 8. Running on Java 11 works fine. This has been happening since finagle 20.4.0 (20.3.0 works fine).

Expected behavior

Thrift client works as expected.

Actual behavior

It throws:

[error] (run-main-7) java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
[error] java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
[error] 	at com.twitter.finagle.thrift.Protocols$TFinagleBinaryProtocol.writeNonCompactString(Protocols.scala:289)
[error] 	at com.twitter.finagle.thrift.Protocols$TFinagleBinaryProtocol.writeString(Protocols.scala:321)
[error] 	at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:117)
[error] 	at test.Foo$FinagleClient.encodeRequest(Foo$FinagleClient.scala:56)
[error] 	at test.Foo$FinagleClient.$anonfun$foo$1(Foo$FinagleClient.scala:153)
[error] 	at com.twitter.util.Local.let(Local.scala:4978)
[error] 	at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:51)
[error] 	at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:28)
[error] 	at test.Foo$FinagleClient.foo(Foo$FinagleClient.scala:150)
[error] 	at test.Test$.delayedEndpoint$test$Test$1(Test.scala:12)
[error] 	at test.Test$delayedInit$body.apply(Test.scala:7)

Steps to reproduce the behavior

Example code:

struct FooMsg {
  1: optional string message
}

service Foo {
  FooMsg foo(1: FooMsg fooMsg)
}

val service = ThriftMux.client.newService("localhost:8080", "test")
val client = new Foo.FinagledClient(service, RichClientParam())

val requestMessage = FooMsg(Some("foo"))
Await.result(client.foo(requestMessage), Duration.fromSeconds(1))

Full example (sbt runable): https://github.com/dziemba/finagle-issue-839

It works fine on Java 11, but fails with the above exception on Java 8 (1.8.0_242)

You don't need to start a server of any kind, it throws before it tries to make a connection.

[dziemba]

I've update the issue with a full reproduction example to make debugging easier. Please let me know if you need more details. Thanks!

[dziemba]

I did some more testing: I built finagle/scrooge/util from the 20.4.0 tag both with Java 8 and Java 11. The Java 8 build also works with the above test case under Java 8. The Java 11 build fails as described.

I'm not familiar with your build/release setup, but my assumption is that you're building the artifacts with Java 11, while still targeting Java 8 bytecode.
That usually works fine, but in some cases it breaks. Apparently the stdlib in Java 11 has changed. Here's an issue describing the same problem with more details: jetty/jetty.project#3244

The official recommendation is also to build everything with Java 8 for now, see https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html

Can you change the release process to build with Java 8 (for now at least)? Or do you have an idea on how to work around this without upgrading everything to Java 11 on our side?

[dziemba]

@bryce-anderson @ryanoneill @illicitonion @yufangong
I apologize for the direct ping, but it would be super cool if someone can have a look at this before the next release, since we'd love to keep up with the latest finagle version.

Please let me know if you need more details or if I can help out in any way.

[cpretzer]

@dziemba nice work on getting to the bottom of this.

+1 for java 8 build. I can't upgrade to finagle 20.4.0 without it because scala recommends compiling with java 8

In addition this ticket states that Java 8 is the version used internally at Twitter. Is there a mismatch between what version of Java is used to build finagle and the expected Java version?

[roanta]

@dziemba thanks for posting this and the relevant details.

This release went through some hiccups in our process and environment, specifically around jdk versions, and this issue is likely a fallout from that. We're working to address this and figure out a path forward, but it sounds like 20.4.0 is a bad artifact. We will update as we know more and can give an ETA on the next release.

[cpretzer]

Awesome, thanks @roanta

[dziemba]

The issue seems to be fixed in v20.4.1. Thanks a lot @roanta! 🙂

[roanta]

That's great!

Credit to @yufangong for fixing the underlying issues and doing a followup release!