Proper way to repackage ByteBuddy into the library

[qwwdfsad]

Some context:

We at kotlinx.coroutines have a special "debug" library that uses ByteByddy to redefine some classes for the sake of better user experience.

Our API shape has two modes:

1. Programmatic API using self-attach mechanism. Everything works fine, a user has our jar (and thus ByteBuddy) in the classpath.
2. Attach mechanism. If a user already has its own application without our library in the classpath, we don't want to force them to recompile their project.
   So to make it as simple as java -jar myapp.jar -javaagent:kotlinx-coroutines-debug.jar we'll have to shade ByteBuddy into our library. But then we can mess up with other libraries that use ByteBuddy, especially in case of incompatible versions. To avoid that, we not only shade but also repackage ByteBuddy.

It works fine most of the time but fails with java.lang.UnsatisfiedLinkError: Native Library /.../jre/lib/amd64/libattach.so already loaded in another classloader when the target application uses libraries that also use ByteBuddy (e.g. Spring or Mockito).

So the question is, what is the best way to have a self-contained JAR suitable for attach and that also works with other libraries that use ByteBuddy?

[raphw]

Well, this one is a real PITA.

Technically, this has nothing to do with Byte Buddy, but with binding native libraries that do not respect Java's class loader concept but rather load a VM global state. Tools.jar is one of those.

Byte Buddy cannot know if it is already loaded on another class loader. This class loader might be isolated and to discover it, it would already need an agent present. Thus, Byte Buddy has a mechanism to inject a holder class into a well-know class loader (the system loader) once attached to avoid this conflict. Using this class, previous attachments can be discovered.

But if Byte Buddy is shaded, this won't work out as the well-known name is not known anymore. The general advice is therefore not to repackage.

I try to keep Byte Buddy API compatible for not only that reason. But of course, the version conflict remains. Maybe I should make sure Byte Buddy always puts the holder class into the net.bytebuddy namespace by obfuscating its name to shading tools.

I will explore that option.

[elizarov]

Can you provide us with instructions for shading, maybe, as to what classes shall not be shaded for this "sharing" to work?

[raphw]

It's classes in the net.bytebuddy.agent package, I think (without subpackages). Those are fairly stable, too. I'd have to experiment myself, to be honest before I can promise that this will work but it should be a big step in the right direction.

I am however confident that I can fix this in the library eventually by obfuscating the net.bytebuddy namespace eventually when those dynamic injections are made. ASM just does a startswith check in their remapper of the constant pool that every shading tool uses, it's not hard to fool.

[rmannibucau]

If it helps: sigar for intance uses a system property to say the native is already loaded and skip it for later calls, can be a trivial way to share it accross relocations for BB too.

[raphw]

Unfortunately, Byte Buddy cannot control the native code bits of tools.jar.

[rmannibucau]

@raphw it can suround its usage, will not prevent another lib to use it and fall into that bucket but enables to solve that ticket issue ;).

[raphw]

Yes, but to use the attach API, Byte Buddy needs to load tools.jar which automatically triggers loading the native library.

I think using a fixed name that survives autmatic repackaging. I'm on vacation atm, but I'm sure I'll find a solution.

[raphw]

FYI: I extended Byte Buddy agent such that it is now possible to attach to a VM without using the JDK-specific tools.jar. I simply reimplemented the attach API for OpenJ9/HotSpot on Solaris/Windows/POSIX which is now a part of Byte Buddy.

All you need to do is to include JNA for this. Note however that JNA is not shadable due to its use of JNI.

However, this mechanism is not vulnerable to being loaded by multiple class loaders. Maybe instead of shading you can isolate Byte Buddy agent and JNA in a seperate class loader and work around that way? Due to the possibility of using Byte Buddy agent in a premain state, it is not trivial to work around the shading, unfortunately but I will investigate further.

As a bonus, with the JNA-based solution, you can self-attach on non-JDK VMs if this is relevant to you.

[elizarov]

Thanks. JNA use is indeed safer. If we include it as a "plain" dependency, then there's still a risk of a version conflict on JNA. However, if you use "separate classloader" trick, then it indeed works -- JNA can be safely loaded from multiple classloaders 🎉 (I have not explicitly checked if it holds for the case of pre-main, but I don't see how it should be different). Interestly, though, behind the scenes two JNA instances share one copy of the loaded native library, but create different proxies to it.

[raphw]

Minor current issue: There seems to be a bug in the attach emulation for MacOS where the temporary directory is user-dependent. I do not own a Mac but someone else is looking into it.

[raphw]

Did you look into using JNA and class loader isolation?

[dkhalanskyjb]

Yeah, the issue with non-standard tmpdir on MacOS is still present. A simple project that shows this: https://github.com/dkhalanskyjb/byteBuddyMacBug Are there any new details?

[qwwdfsad]

We are using ForEmulatedAttachment with plain JNA dependency and repackaged ByteBuddy for now and everything works like a charm, it seems like the issue can be closed.
Thanks fir the help!

[raphw]

I'd still like to fix the repackaging for older JVMs where JNA is not an option but yes, the JNA approach does not suffer these problems.

[olka]

Hi there!
ATM I'm trying to instrument okhttp client bundled into ktor and found your "debug" library :)

The problem that I'm facing is that byte-buddy inside kotlinx-coroutines-debug is v1.10.9. My instrumentation agent and application itself compiled with java 17 but byte-buddy:1.10.9 major version is 49
So basically everything fails with

Starting....
[Byte Buddy] BEFORE_INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@7c61749d on sun.instrument.InstrumentationImpl@747f5f81
[Byte Buddy] ERROR okhttp3.Dispatcher [datadog, unnamed module @3c0ecd4b, loaded=true]
java.lang.IllegalArgumentException: Unsupported class file major version 61
	at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:196)
	at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:177)
	at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:163)
	at net.bytebuddy.utility.OpenedClassReader.of(OpenedClassReader.java:86)
	at net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:681)
	at net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:667)
	at net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:747)
	at net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:845)
	at net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:914)
	at net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8147)
	at net.bytebuddy.description.ModifierReviewable$AbstractBase.matchesMask(ModifierReviewable.java:618)
	at net.bytebuddy.description.ModifierReviewable$AbstractBase.isInterface(ModifierReviewable.java:427)
	at net.bytebuddy.description.type.TypeDescription$AbstractBase.isAssignable(TypeDescription.java:7453)
	at net.bytebuddy.description.type.TypeDescription$AbstractBase.isAssignableTo(TypeDescription.java:7489)
	at net.bytebuddy.description.type.TypeDescription$ForLoadedType.isAssignableTo(TypeDescription.java:8289)
	at net.bytebuddy.implementation.bytecode.assign.reference.ReferenceTypeAwareAssigner.assign(ReferenceTypeAwareAssigner.java:42)
	at net.bytebuddy.implementation.bytecode.assign.primitive.PrimitiveTypeAwareAssigner.assign(PrimitiveTypeAwareAssigner.java:68)
	at net.bytebuddy.implementation.bytecode.assign.primitive.VoidAwareAssigner.assign(VoidAwareAssigner.java:67)
	at net.bytebuddy.asm.Advice$OffsetMapping$ForField.resolve(Advice.java:2127)
	at net.bytebuddy.asm.Advice$Dispatcher$Inlining$Resolved$ForMethodExit.doApply(Advice.java:7950)
	at net.bytebuddy.asm.Advice$Dispatcher$Inlining$Resolved$ForMethodExit.apply(Advice.java:7911)
	at net.bytebuddy.asm.Advice$Dispatcher$Inlining$Resolved$AdviceMethodInliner.visitMethod(Advice.java:7388)
	at net.bytebuddy.jar.asm.ClassReader.readMethod(ClassReader.java:1331)
	at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:717)
	at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:401)
	at net.bytebuddy.asm.Advice$Dispatcher$Inlining$Resolved$AdviceMethodInliner.apply(Advice.java:7382)
	at net.bytebuddy.asm.Advice$AdviceVisitor$WithExitAdvice.onUserEnd(Advice.java:9684)
	at net.bytebuddy.asm.Advice$AdviceVisitor.visitMaxs(Advice.java:9464)
	at net.bytebuddy.jar.asm.ClassReader.readCode(ClassReader.java:2640)
	at net.bytebuddy.jar.asm.ClassReader.readMethod(ClassReader.java:1492)
	at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:717)
	at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:401)
	at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:3397)
	at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:1933)
	at net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:217)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10327)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10263)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10722)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:10660)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10219)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
	at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
	at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6869)
	at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7148)
	at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:6993)
	at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:4855)
	at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:9463)
	at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:9384)
	at net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:10986)
	at tools.agent.LoaderOkHttpAgent.agentmain(LoaderOkHttpAgent.java:59)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:513)
[Byte Buddy] TRANSFORM okhttp3.Dispatcher [jdk.internal.loader.ClassLoaders$AppClassLoader@251a69d7, unnamed module @31e4bb20, loaded=true]
[Byte Buddy] INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@7c61749d on sun.instrument.InstrumentationImpl@747f5f81

My suspicion is that ByteBuddy v1.10.9 doesn't know anything about java 17 so noting could be injected. On local environment without ktor instrumentation agent works just fine - so I'm 90% sure it's something related to this ticket
Any help would be appreciated

[raphw]

If you are running an agent, it is a good strategy to use a trampoline agent. The trampoline agent basically just creates a class loader which has the boot loader as its parent and adds the actual jar to it. It then invokes the actual agent's premain method using reflection.

[olka]

ATM I was using dynamic loading to inject agent in runtime without intervention into classpath on a startup. Is this approach is pretty limited and only can get this far?

[raphw]

Not sure I understand the question. What do you mean by "dynamic loading"?

[olka]

Attaching agent to a running JVM

VirtualMachine vm = VirtualMachine.attach(pid);
vm.loadAgent(jarPath, null);