HotSwap Agent (fatjar) corrupts JVM NestMembers/NestHost attributes, causing IllegalAccessError for private inner classes

[Artur-]

Disclaimer: This is an AI analysis of the problem. You can see it by running the Flow tests though.

JBR version:

OpenJDK Runtime Environment JBR-25.0.1+8-268.52-nomod (build 25.0.1+8-b268.52)
OpenJDK 64-Bit Server VM JBR-25.0.1+8-268.52-nomod (build 25.0.1+8-b268.52, mixed mode, sharing)
HotSwap Agent: 2.0.3 (bundled fatjar)

JVM flags:

-XX:+AllowEnhancedClassRedefinition -XX:HotswapAgent=fatjar

Description

When HotSwap Agent is loaded in fatjar mode with -XX:+AllowEnhancedClassRedefinition, private inner class access from enclosing classes fails with IllegalAccessError. The JVM reports that the inner class "is not a nest member" of its enclosing class, even though both are in the same classloader and were compiled normally by javac.

This violates https://openjdk.org/jeps/181 — the NestMembers attribute of the enclosing class no longer lists the inner class as a nest member after HotSwap Agent processes the class bytecode.

Steps to reproduce

1. Use JBR 25.0.1 with HotSwap Agent enabled:

export JAVA_TOOL_OPTIONS="-XX:+AllowEnhancedClassRedefinition -XX:HotswapAgent=fatjar"

2. Clone the https://github.com/vaadin/flow repository (any recent commit on main)
3. Run unit tests in the flow-server module: cd flow-server && mvn test
4. Observe 483+ test errors, all with the same root cause:

  java.lang.IllegalAccessError: class com.vaadin.flow.server.MockVaadinServletService
    tried to access private method
    'void com.vaadin.flow.server.MockVaadinServletService$MockVaadinServlet.<init>(
      com.vaadin.flow.function.DeploymentConfiguration)'
    (com.vaadin.flow.server.MockVaadinServletService and
     com.vaadin.flow.server.MockVaadinServletService$MockVaadinServlet
     are in unnamed module of loader 'app'),
    (Type com.vaadin.flow.server.MockVaadinServletService$MockVaadinServlet (loader: 'app')
     is not a nest member of type
     com.vaadin.flow.server.MockVaadinServletService (loader: 'app'):
     current type is not listed as a nest member)

5. Remove HotSwap Agent and run the same tests: JAVA_TOOL_OPTIONS="" mvn test

Result: 0 errors (all 4479 tests pass).

Affected classes (examples)

The issue affects ANY Java class with a private inner class. Two distinct examples from the Vaadin Flow codebase:

1. MockVaadinServletService$MockVaadinServlet — test helper class:

  public class MockVaadinServletService extends VaadinServletService {
      private static class MockVaadinServlet extends VaadinServlet {
          private MockVaadinServlet(DeploymentConfiguration configuration) { ... }
      }
      // Accessing MockVaadinServlet's private constructor fails
  }

2. ElementListenerMap$DomEventListenerWrapper — production code:

public class ElementListenerMap extends NodeMap {
      private static class DomEventListenerWrapper implements DomListenerRegistration {
          private DomEventListenerWrapper(ElementListenerMap map, String type,
                  DomEventListener origin) { ... }
      }
      // Accessing DomEventListenerWrapper's private constructor fails
  }

Root cause analysis

HotSwap Agent's class file transformer (loaded via JVM TI during -XX:HotswapAgent=fatjar initialization) processes classes during class loading. This processing corrupts or strips the NestMembers and NestHost class file attributes (JVMS §4.7.28, §4.7.29).

The JVM's nest-based access verification then correctly rejects access because the attributes are missing/corrupted — the inner class is no longer listed as a nest member.

Disabling individual HotSwap Agent plugins via -Dhotswapagent.disablePlugin=... does not fix the issue, confirming the corruption happens in the core agent's class file transformer, not in any specific plugin.

Additional finding

-XX:+AllowEnhancedClassRedefinition alone (without -XX:HotswapAgent=fatjar) crashes the forked JVM:
The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
This may be a separate issue — the flag appears to require HotSwap Agent to function, but should either work independently or fail gracefully.

Expected behavior

Private inner classes should remain accessible from their enclosing classes regardless of whether HotSwap Agent is loaded. The class file transformer should preserve NestMembers/NestHost attributes.

Environment

• OS: Linux 6.12.67-linuxkit
• JBR: 25.0.1+8-268.52-nomod
• HotSwap Agent: 2.0.3 (bundled in /opt/jbr/lib/hotswap/hotswap-agent.jar)
• Build tool: Maven 3.x with Surefire 3.5.4
• Test framework: JUnit 5 with Mockito 5.x / ByteBuddy

[skybber]

Hi,

HA should preserve nest attributes (NestHost / NestMembers, i.e. the “Nested*” metadata) when transforming bytecode, it is supported by internal javassist.

Since the Vaadin plugin doesn’t have unit tests I can run locally, could you please verify this by dumping both the original and transformed class bytes inside the Vaadin plugin transformer? For example:

  @OnClassLoadEvent(classNameRegexp = VAADIN_SERVICE)
  public static byte[] transformVaadinService(byte[] originalBytes, CtClass ctClass, ClassPool classPool, String className)
          throws NotFoundException, CannotCompileException, IOException {

      CtClass vaadinHotSwapper = findHotswapperClass(classPool);
      boolean hasVaadinHotSwapper = vaadinHotSwapper != null;

      StringBuilder src = new StringBuilder();
      if (hasVaadinHotSwapper) {
          String initHotswapperPluginCall = PluginManagerInvoker.buildCallPluginMethod(
                  VaadinPlugin.class, "initializeHotswapper", "hotswapper", "java.lang.Object");
          src.append("try { ")
             .append("java.util.Optional maybeHotswapper = ")
             .append(vaadinHotSwapper.getName())
             .append(".register(this);")
             .append("if (maybeHotswapper.isPresent()) { ")
             .append("Object hotswapper = maybeHotswapper.get();")
             .append(initHotswapperPluginCall)
             .append("} } catch (Exception e) { e.printStackTrace(); }");
      } else {
          src.append(PluginManagerInvoker.buildCallPluginMethod(
                  VaadinPlugin.class, "vaadinServiceInitialized"));
      }

      ctClass.getDeclaredMethod("init").insertBefore(src.toString());
      ctClass.getDeclaredMethod("destroy").insertBefore(
              PluginManagerInvoker.buildCallPluginMethod(VaadinPlugin.class, "vaadinServiceDestroyed"));

      byte[] transformedBytes = ctClass.toBytecode();

      Path dumpDir = Paths.get(System.getProperty("ha.dump.dir", "/tmp/ha-dump"));
      Files.createDirectories(dumpDir);
      Files.write(dumpDir.resolve(className.replace('/', '_') + ".orig.class"), originalBytes);
      Files.write(dumpDir.resolve(className.replace('/', '_') + ".xform.class"), transformedBytes);

      return transformedBytes;
  }

Then you can inspect whether NestHost/NestMembers disappear.

[Artur-]

Digging some more, the AI came up with this bug which may or may not be the same issue: https://github.com/Artur-/jbr-bug

[skybber]

Thanks, I’ll take a look at it. By the way, I’ve had quite mixed experiences using AI - on most projects it’s absolutely amazing, but on the HotswapAgent plugins it hasn’t helped me much so far. The likely issue is the different framework versions, which it can’t handle.

[Artur-]

This is another test project, more related to the original problem I think https://github.com/Artur-/jbr-bug-nest

[skybber]

I created https://youtrack.jetbrains.com/issue/JBR-10000 for the crash - it has higher visibility.

[Artur-]

Should there be another JBR issue for https://github.com/Artur-/jbr-bug-nest as it crashes in do_topological_class_sorting?

[skybber]

If it is the same crash, then JBR-10000 is enough

[Artur-]

I don't think it is

[skybber]

There’s a fix in this branch:

https://github.com/JetBrains/JetBrainsRuntime/commits/vladimir.dvorak/JBR-10000.jbr25/

It should be merged next week after code review. A related PR reporting the same problem is here:

JetBrains/JetBrainsRuntime#588

[Artur-]

Cool, both test projects pass with that branch. The original problem in the Flow project is also resolved!