Home » Java » Core Java » JAR File Handles: Clean Up After Your Mess!

About Janaka Bandara

Janaka Bandara
A budding software engineer in the enterprise integration space, and a simplistic guy who likes to learn, appreciate, help, share, and enjoy life

JAR File Handles: Clean Up After Your Mess!

In Ultra ESB we use a special hot-swap classloader that allows us to reload Java classes on demand. This allows us to literally hot-swap our deployment units – load, unload, reload with updated classes, and phase-out gracefully – without restarting the JVM.

Windows: supporting the forbidden land

In Ultra ESB Legacy the loader was working fine on Windows, but on the newer X-version it seemed to be having some hiccups. We are not supporting Windows as a target platform, so it didn’t matter much – until recently, when we decided to support non-production distros on Windows. (Our enterprise integration IDE UltraStudio runs fine on Windows, so Windows devs, you are all covered.)

TDD FTW

Fixing the classloader was a breeze, and all tests were passing; but I wanted to back my fixes up with some extra tests, so I wrote a few new ones. Most of these involved creating a new JAR file in a subditrectory under the system temp directory, and using the hot-swap classloader to load different artifacts that were placed inside the JAR. For extra credit on best practices, I also made sure to add some cleanup logic to delete the temp subdirectory via FileUtils.deleteDirectory().

And then, things went nuts.

And the tear-down was no more.

All tests were passing, in both Linux and Windows; but the final tear-down logic was failing in Windows, right at the point where I delete the temp subdirectory.

Being on Windows, I didn’t have the luxury of lsof; fortunately, Sysinternals already had just the thing I needed: handle64.

Finding the culprit was pretty easy: hit a breakpoint in tearDown() just before the directory tree deletion call, and run a handle64 {my-jar-name}.jar.

Bummer.

My test Java process, was holding a handle to the test JAR file.

Hunting for the leak

No. Seriously. I didn’t.

Naturally, my first suspect was the classloader itself. I spent almost half an hour going over the classloader codebase again and again. No luck. Everything seemed rock solid.

The “leak dumper”; a.k.a my Grim Reaper for file handles

My best shot was to see what piece of code had opened the handler to the JAR file. So I wrote a quick-n-dirty patch to Java’s FileInputStream and FilterInputStream that would dump acquire-time stacktrace snapshots; whenever a thread holds a stream open for too long.

This “leak dumper” was partly inspired by our JDBC connection pool that detects unreleased connections (subject to a grace period) and then dumps the stacktrace of the thread that borrowed it – back at the time it was borrowed. (Kudos to Sachini, my former colleague-intern at AdroitLogic.)

The leak, exposed!

Sure enough, the stacktrace revealed the culprit:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
id: 174 created: 1570560438355
--filter--
 
  java.io.FilterInputStream.<init>(FilterInputStream.java:13)
  java.util.zip.InflaterInputStream.<init>(InflaterInputStream.java:81)
  java.util.zip.ZipFile$ZipFileInflaterInputStream.<init>(ZipFile.java:408)
  java.util.zip.ZipFile.getInputStream(ZipFile.java:389)
  java.util.jar.JarFile.getInputStream(JarFile.java:447)
  sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:162)
  java.net.URL.openStream(URL.java:1045)
  org.adroitlogic.x.base.util.HotSwapClassLoader.loadSwappableClass(HotSwapClassLoader.java:175)
  org.adroitlogic.x.base.util.HotSwapClassLoader.loadClass(HotSwapClassLoader.java:110)
  org.adroitlogic.x.base.util.HotSwapClassLoaderTest.testServiceLoader(HotSwapClassLoaderTest.java:128)
  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  java.lang.reflect.Method.invoke(Method.java:498)
  org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
  org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
  org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
  org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
  org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
  org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
  org.testng.TestRunner.privateRun(TestRunner.java:782)
  org.testng.TestRunner.run(TestRunner.java:632)
  org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
  org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
  org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
  org.testng.SuiteRunner.run(SuiteRunner.java:268)
  org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
  org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
  org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
  org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
  org.testng.TestNG.run(TestNG.java:1064)
  org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
  org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

Gotcha!

1
2
3
4
5
java.io.FilterInputStream.<init>(FilterInputStream.java:13)
  ...
  sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:162)
  java.net.URL.openStream(URL.java:1045)
  org.adroitlogic.x.base.util.HotSwapClassLoader.loadSwappableClass(HotSwapClassLoader.java:175)

But still, that didn’t tell the whole story. If URL.openStream() opens the JAR, why does it not get closed when we return from the try-with-resources block?

01
02
03
04
05
06
07
08
09
10
11
12
try (InputStream is = jarURI.toURL().openStream()) {
            byte[] bytes = IOUtils.toByteArray(is);
            Class<?> clazz = defineClass(className, bytes, 0, bytes.length);
            ...
            logger.trace(15, "Loaded class {} as a swappable class", className);
            return clazz;
 
        } catch (IOException e) {
            logger.warn(16, "Class {} located as a swappable class, but couldn't be loaded due to : {}, " +
                    "trying to load the class as a usual class", className, e.getMessage());
            ...
        }

Into the wild: JarURLConnection, URLConnection, and beyond

Thanks to Sun Microsystems who made it OSS, I could browse through the JDK source, right up to this shocking comment – all the way down, in java.net.URLConnection:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
private static boolean defaultUseCaches = true;
 
   /**
     * If <code>true</code>, the protocol is allowed to use caching
     * whenever it can. If <code>false</code>, the protocol must always
     * try to get a fresh copy of the object.
     * <p>
     * This field is set by the <code>setUseCaches</code> method. Its
     * value is returned by the <code>getUseCaches</code> method.
     * <p>
     * Its default value is the value given in the last invocation of the
     * <code>setDefaultUseCaches</code> method.
     *
     * @see     java.net.URLConnection#setUseCaches(boolean)
     * @see     java.net.URLConnection#getUseCaches()
     * @see     java.net.URLConnection#setDefaultUseCaches(boolean)
     */
    protected boolean useCaches = defaultUseCaches;

Yep, Java does cache JAR streams!

From sun.net.www.protocol.jar.JarURLConnection:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
class JarURLInputStream extends FilterInputStream {
        JarURLInputStream(InputStream var2) {
            super(var2);
        }
 
        public void close() throws IOException {
            try {
                super.close();
            } finally {
                if (!JarURLConnection.this.getUseCaches()) {
                    JarURLConnection.this.jarFile.close();
                }
            }
 
        }
    }

If (well, because) useCaches is true by default, we’re in for a big surprise!

Let Java cache its JARs, but don’t break my test!

JAR caching would probably improve performance; but does that mean I should stop cleaning up after – and leave behind stray files after each test?

(Of course I could say file.deleteOnExit(); but since I was dealing with a directory hierarchy, there was no guarantee that things would get deleted in order, and undeleted directories would be left behind.)

So I wanted a way to clean up the JAR cache – or at least purge just my JAR entry; after I am done, but before the JVM shuts down.

Disabling JAR caching altogether – probably not a good idea!

URLConnection does offer an option to avoid caching connection entries:

01
02
03
04
05
06
07
08
09
10
/**
     * Sets the default value of the <code>useCaches</code> field to the
     * specified value.
     *
     * @param   defaultusecaches   the new value.
     * @see     #getDefaultUseCaches()
     */
    public void setDefaultUseCaches(boolean defaultusecaches) {
        defaultUseCaches = defaultusecaches;
    }

It would have been perfect if caching could be disabled per file/URL, as above; our classloader caches all entries as soon as it opens a JAR, so it never needs to open/read that file again. However, once a JAR is open, caching cannot be disabled on it; so once our classloader has opened the JAR, there’s no getting rid of the cached file handle – until the JVM itself shuts down!

URLConnection also allows you to disable caching by default for all subsequent connections:

01
02
03
04
05
06
07
08
09
10
/**
     * Sets the default value of the <code>useCaches</code> field to the
     * specified value.
     *
     * @param   defaultusecaches   the new value.
     * @see     #getDefaultUseCaches()
     */
    public void setDefaultUseCaches(boolean defaultusecaches) {
        defaultUseCaches = defaultusecaches;
    }

However, if you disable it once, the whole JVM could be affected from that moment onwards – since it probably applies to all URLConnection-based implementations. As I said before, that could hinder performance – not to mention deviating my test from cache-enabled, real-world behavior.

Down the rabbit hole (again!): purging manually from the JarFileFactory

The least-invasive option is to remove my own JAR from the cache, when I know I’m done.

And good news, the cache – sun.net.www.protocol.jar.JarFileFactory – already has a close(JarFile) method that does the job.

But sadly, the cache class is package-private; meaning there’s no way to manipulate it from within my test code.

Reflection to the rescue!

Thanks to reflection, all I needed was one little “bridge” that would access and invoke jarFactory.close(jarFile) on behalf of me:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class JarBridge {
 
    static void closeJar(URL url) throws Exception {
 
        // JarFileFactory jarFactory = JarFileFactory.getInstance();
        Class<?> jarFactoryClazz = Class.forName("sun.net.www.protocol.jar.JarFileFactory");
        Method getInstance = jarFactoryClazz.getMethod("getInstance");
        getInstance.setAccessible(true);
        Object jarFactory = getInstance.invoke(jarFactoryClazz);
 
        // JarFile jarFile = jarFactory.get(url);
        Method get = jarFactoryClazz.getMethod("get", URL.class);
        get.setAccessible(true);
        Object jarFile = get.invoke(jarFactory, url);
 
        // jarFactory.close(jarFile);
        Method close = jarFactoryClazz.getMethod("close", JarFile.class);
        close.setAccessible(true);
        //noinspection JavaReflectionInvocation
        close.invoke(jarFactory, jarFile);
 
        // jarFile.close();
        ((JarFile) jarFile).close();
    }
}

And in my test, I just have to say:

1
JarBridge.closeJar(jarPath.toUri().toURL());

Right before deleting the temp directory.

So, what’s the take-away?

Nothing much for you, if you are not directly dealing with JAR files; but if you are, you might run into this kind of obscure “file in use” errors. (That would hold true for other URLConnection-based streams as well.)

If you happen to be as (un)lucky as I was, just recall that some notorious blogger had written some hacky “leak dumper” patch JAR that would show you exactly where your JAR (or non-JAR) leak is.

Adieu!

Published on Java Code Geeks with permission by Janaka Bandara, partner at our JCG program. See the original article here: JAR File Handles: Clean Up After Your Mess!

Opinions expressed by Java Code Geeks contributors are their own.

(0 rating, 0 votes)
You need to be a registered member to rate this.
Start the discussion Views Tweet it!
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....
I agree to the Terms and Privacy Policy

Leave a Reply

avatar

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
Notify of