Our system doing repeated FULL GC calls almost every 60 odd seconds and that was affecting performance on Production machines. I found the GC happening using JConsole which comes bundled with Java5 upwards.
This was quite puzzling, but the good thing was it was happening predictably every 60 odd seconds. Sometimes the GC was taking 2-3 sec and sometimes less.
It seemed that some library we are using on TOMCAT side – has some hard coded System GC calls.
My initial thought was – that somecode was calling System.gc() every min or so. I searched through the whole code base but found no explicit GC in our code.
Then I researched the JVM Options and found this option
-XX:+DisableExplicitGC
And decided to add it to the JVM Startup options and restart the production machines at night.
And then the FULL GC every minute vanished from the log.
Though I could never conclusively prove where the GC was coming from – but my belief was that some 3rd party library code – was calling System.gc() and once the flag was set on the JVM Startup – the GCs vanished.
Sumit Pal's Blog 
Every 1 minute ? Sounds like the RMI distributed GC kicking in. See http://jira.atlassian.com/browse/CONF-17574 for a nice discussion.
Do you use RMI? By default, DGC is run once a minute. You can tune that with sun.rmi.dgc.server.gcInterval and sun.rmi.dgc.client.gcInterval system properties, in case you do not want to disable System.gc() altogether.
Regardless, that does not mean you should always add a directive disabling garbage collection calls… many high speed/high memory processes function much better if a gc is ran before hand… simple as that… what you should do is always use explicit garbage collection intelligently, something obviously not happening somewhere.
Hi
Thanks for the comments and sorry for the late reply.
What memory processes are you talking about here. Can you give some examples.
In my mind – GC is a very non deterministic thing and it is best left to the JVM Implementors to decide when to invoke it.
So I am wondering if basing a product’s performance capabilities ( as you are saying – some high speed memory processes perform better when you call GC ) is prudent or not.
Java provides things like SoftReferences and WeakReferences to handle caching and also make sure that you do not get an out of memory type errors.
I would like to hear / know more from your end about this
thanks
If you’re working with an IDE, did you try asking for a call hierarchy on System.gc()? Inspired by your post, I tried that in Eclipse with an app I work on and found that several third-party libs were calling System.gc(). Looking over the code, I don’t think these calls will be at all frequent or harmful to performance, but it’s definitely interesting to know that they’re there.
> Though I could never conclusively prove where the GC
> was coming from
There are some informations on this topic in the net.
Even “-XX:+DisableExplicitGC” didn’t helped on some JVMs.
The cause is the distributed GC of the Java RMI subsystem.
In production environments you can use “-Dsun.rmi.dgc.client.gcInterval=x” and “-Dsun.rmi.dgc.server.gcInterval=x” (x in milliseconds) with large values to come around this problem if you use RMI.
googling for -XX:+DisableExplicitGC yields other blog post with similair sentiments and experiences and even sun bugs around this.
Have you tried using DTrace/BTrace et all to find the offending code?
Did you figure out which library was it? Thanks!
I have not yet found out the offending library. If I do – I will surely update the post. Also I like the IDE Approach to debugging it.
Since we run both JBOSS and Tomcat on separate machines and I am sure that we use DGC settings on the JBOSS Side – may be I should check whether the TOMCAT Process has dgc settings on its command line.
Good suggestions. As soon as I find time I will dig in more to figure out.
Pingback: Online Resources: Android Development, Future of Java, Careers in Software Development « AndroidBoss
RMI DGC is not GC; it does not run every minute, it runs every half lease interval; and it does not call System.gc(). The culprit was elsewhere.