How we solved – GC every 1 minute on Tomcat

Our system doing repeated FULL GC calls almost every 60 odd seconds and that was affecting performance on Production machines. I found the GC happening using JConsole which comes bundled with Java5 upwards.

This was quite puzzling, but the good thing was it was happening predictably every 60 odd seconds.  Sometimes the GC was taking 2-3 sec and sometimes less.

It seemed that some library we are using on TOMCAT side – has some hard coded System GC calls.

My initial thought was – that somecode was calling System.gc() every min or so. I searched through the whole code base but found no explicit GC in our code.

Then I researched the JVM Options and found this option

-XX:+DisableExplicitGC

And decided to add it to the JVM Startup options and restart the production machines at night.

And then the FULL GC every minute vanished from the log.

Though I could never conclusively prove where the GC was coming from – but my belief was that some 3rd party library code – was calling System.gc() and once the flag was set on the JVM Startup – the GCs vanished.

About these ads

About palsumitpal

Depending on your viewpoint, I am either a software architect who codes or a software developer who understands architecture
This entry was posted in Java and Java EE. Bookmark the permalink.

11 Responses to How we solved – GC every 1 minute on Tomcat

  1. Every 1 minute ? Sounds like the RMI distributed GC kicking in. See http://jira.atlassian.com/browse/CONF-17574 for a nice discussion.

  2. Janne Lindholm says:

    Do you use RMI? By default, DGC is run once a minute. You can tune that with sun.rmi.dgc.server.gcInterval and sun.rmi.dgc.client.gcInterval system properties, in case you do not want to disable System.gc() altogether.

  3. ag says:

    Regardless, that does not mean you should always add a directive disabling garbage collection calls… many high speed/high memory processes function much better if a gc is ran before hand… simple as that… what you should do is always use explicit garbage collection intelligently, something obviously not happening somewhere.

    • Sumit says:

      Hi

      Thanks for the comments and sorry for the late reply.

      What memory processes are you talking about here. Can you give some examples.

      In my mind – GC is a very non deterministic thing and it is best left to the JVM Implementors to decide when to invoke it.

      So I am wondering if basing a product’s performance capabilities ( as you are saying – some high speed memory processes perform better when you call GC ) is prudent or not.

      Java provides things like SoftReferences and WeakReferences to handle caching and also make sure that you do not get an out of memory type errors.

      I would like to hear / know more from your end about this

      thanks

  4. Matt Passell says:

    If you’re working with an IDE, did you try asking for a call hierarchy on System.gc()? Inspired by your post, I tried that in Eclipse with an app I work on and found that several third-party libs were calling System.gc(). Looking over the code, I don’t think these calls will be at all frequent or harmful to performance, but it’s definitely interesting to know that they’re there.

  5. > Though I could never conclusively prove where the GC
    > was coming from

    There are some informations on this topic in the net.

    Even “-XX:+DisableExplicitGC” didn’t helped on some JVMs.

    The cause is the distributed GC of the Java RMI subsystem.

    In production environments you can use “-Dsun.rmi.dgc.client.gcInterval=x” and “-Dsun.rmi.dgc.server.gcInterval=x” (x in milliseconds) with large values to come around this problem if you use RMI.

  6. Michael says:

    googling for -XX:+DisableExplicitGC yields other blog post with similair sentiments and experiences and even sun bugs around this.

    Have you tried using DTrace/BTrace et all to find the offending code?

  7. Anton says:

    Did you figure out which library was it? Thanks!

  8. palsumitpal says:

    I have not yet found out the offending library. If I do – I will surely update the post. Also I like the IDE Approach to debugging it.

    Since we run both JBOSS and Tomcat on separate machines and I am sure that we use DGC settings on the JBOSS Side – may be I should check whether the TOMCAT Process has dgc settings on its command line.

    Good suggestions. As soon as I find time I will dig in more to figure out.

  9. Pingback: Online Resources: Android Development, Future of Java, Careers in Software Development « AndroidBoss

  10. Esmond Pitt says:

    RMI DGC is not GC; it does not run every minute, it runs every half lease interval; and it does not call System.gc(). The culprit was elsewhere.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s