Strange OutOfMemoryError issue

[ady1981]

Hello,

Our e-store periodically crashes with the exception:

Jul 3, 2010 6:26:32 AM org.apache.tomcat.util.net.JIoEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: unable to create new native thread
   at java.lang.Thread.start0(Native Method)
   at java.lang.Thread.start(Thread.java:597)
   at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
   at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
   at org.apache.catalina.core.StandardThreadExecutor.execute(Unknown Source)
   at org.apache.tomcat.util.net.JIoEndpoint.processSocket(Unknown Source)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:619)
Exception in thread "Thread-3" java.lang.OutOfMemoryError: unable to create new native thread
   at java.lang.Thread.start0(Native Method)
   at java.lang.Thread.start(Thread.java:597)
   at java.util.Timer.<init>(Timer.java:154)
   at java.util.Timer.<init>(Timer.java:122)
   at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
   at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
   at java.sql.DriverManager.getConnection(DriverManager.java:582)
   at java.sql.DriverManager.getConnection(DriverManager.java:185)
   at org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS.getPooledConnection(DriverAdapterCPDS.java:177)
   at org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS.getPooledConnection(DriverAdapterCPDS.java:132)
   at org.apache.commons.dbcp.datasources.KeyedCPDSConnectionFactory.makeObject(KeyedCPDSConnectionFactory.java:158)
   at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:747)
   at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:167)
   at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)
   at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)
   at org.apache.torque.TorqueInstance.getConnection(TorqueInstance.java:749)
   at org.apache.torque.Torque.getConnection(Torque.java:289)
   at org.apache.torque.util.Transaction.beginOptional(Transaction.java:108)
   at org.apache.torque.util.BasePeer.doSelect(BasePeer.java:763)
   at com.konakart.bl.ConfigurationMgr.getConfiguration(Unknown Source)
   at com.konakart.app.KKEng.getConfiguration(Unknown Source)
   at com.konakart.app.KKEng.getConfiguration(Unknown Source)
   at com.konakart.al.ConfigCacheUpdater.updateConfigs(Unknown Source)
   at com.konakart.al.ConfigCacheUpdater.run(Unknown Source)

Does anybody know what may be the reason and solution?

Here is our extra params for tomcat:
JAVA_OPTS="-d64 -server -Xms200M -Xmx360M -XX:MaxPermSize=140M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"

[kate]

These can be difficult and very dependent on your environment.

This article might be handy:

http://www.talkingtree.com/blog/index.cfm/2005/3/11/NewNativeThread

Perhaps you could do as it suggests at the bottom?   Maybe try a smaller stack size? (-Xss)

Also interesting is:

http://webcache.googleusercontent.com/search?q=cache:http://www.egilh.com/blog/archive/2006/06/09/2811.aspx

[ady1981]

Hello,

The "-Xss128k" solution does not seem to work.

Here is our JAVA_OPTS:

JAVA_OPTS="-d64 -server -Xss128k -Xms200M -Xmx360M -XX:MaxPermSize=140M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"

And here is the random exception:

java.lang.OutOfMemoryError: unable to create new native thread
   at java.lang.Thread.start0(Native Method)
   at java.lang.Thread.start(Thread.java:597)
   at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
   at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
   at org.apache.catalina.core.StandardThreadExecutor.execute(Unknown Source)
   at org.apache.tomcat.util.net.JIoEndpoint.processSocket(Unknown Source)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(Unknown Source)
   at java.lang.Thread.run(Thread.java:619)

[ming]

You'd probably have more success searching the web for answers to this because it's more of a java/tomcat/platform issue than a KonaKart problem.

Anyway... a couple more ideas:

Have you tried an even lower Xss value such as 64k?  May as well keep going until you get stack overflows.

Also, on Linux, I believe (I've never tried this) you need to set the Linux thread stack size to the same value as the JVM stack size to get full benefits. eg  use "ulimit -s <size in kb>".
Therefore you could try a 128kb stack with your -Xss128k 

[ady1981]

Thanks for the answers.

I wrote the email to the hosting provider, they set the number of active threads into 100 and the problem seemed as solved now.

The only regular error now is:

Code Select Expand

Aug 6, 2010 5:15:28 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet action threw exception
javax.servlet.jsp.JspException: Cannot find bean: "konakartKey" in any scope
at org.apache.struts.taglib.TagUtils.lookup(TagUtils.java:935)
...

And the tomcat needed to be restarted for the last 30 days just once.

[ady1981]

Indeed the error is not fully regular (rather accidental) .