Recently, I was discussing with a friend, why the Java process uses more memory than the maximum heap that we set when starting the java process.
All java objects that code creates are created inside Java heap space, which its size is defined by the -Xmx option. But a java process is consisted by many spaces, not only by the java heap space. A few of spaces that a java process is consisted are the following:
- Loaded libraries (including jar and class files)
- Control structures for the java heap
- Thread Stacks
- Generated (JITed) code
- User native memory (malloced in JNI)
- … more…
In a 32-bit architecture system, the total process size cannot exceed 4GB. So, a 32-bit java process is consisted by many spaces (java heap, native memory (C-Heap) and other spaces) and its allocated space cannot exceed 4GB.
Assume on a 32-bit production system you run a java application server with -Xmx 1.5 GB (java heap is set to 1.5 GB) for a long time, with many applications deployed. After some time, customer wants to deploy on the same application server more applications. System operator(s) understands that as server will have to process more requests will also need to create more objects and do more processing. So, as a future proof solution operator(s) decides to increase maximum heap of java process to 2 GB.
OK, it looks like a good approach, but what did it really happen on this production application server in reality??? (This is a real case). The application server crashed with OutOfMemoryError !!! Can you think about the possible causes?
My first thought was that 2 GB were not enough for all these applications with this load. Unfortunately, the problem was something else. What do you think now? I will help you a little.
java.lang.OutOfMemoryError: requested 55106896 bytes for Chunk::new.
The real cause was that already deployed (old) applications were needed too large size for the native (C-Heap) memory. Before operator(s) increase the size of the heap size (from 1.5GB to 2 GB) they had not monitored the required native memory space of the old applications. The side effect of this action was to automatically decrease the available maximum size of native memory of java process (from 2.5 GB to 2GB). As the old applications were already use so large size for native memory, this change crash the server!!!
The only accepted solution on this case was to avoid increase the maximum heap size, deploy the new applications and live with less throughput. It is not a perfect solution, but it is the only one viable for this case (as our java process has to be 32-bit).
Especially in 32-bit systems, be aware of the required size of native memory of java process, before you increase the java heap size. If you are in a situation where these two spaces conflict, then the solution may not be so easy. If you cannot change your code to overcome this situation, then the most common solution is to move to a 64-bit system, where the maximum process size limit is too much larger.
There are four major things to remember:
- The maximum limit of size of a process
- The size of a java process is not only consisted of java heap
- The size of native (C-Heap) memory of a java process cannot be configured explicitly, as it is possible with the java heap space
- The size of java heap space and native (C-Heap) memory space an application requires is only defined by the application and there is not any standard ratio between these two spaces
Reference: Java heap space, native heap and memory problems from our JCG partner Adrianos Dadis at the Java, Integration and the virtues of source blog.
