Hello,
Our e-store periodically crashes with the exception:
Jul 3, 2010 6:26:32 AM org.apache.tomcat.util.net.JIoEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.apache.catalina.core.StandardThreadExecutor.execute(Unknown Source)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(Unknown Source)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "Thread-3" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at java.util.Timer.<init>(Timer.java:154)
at java.util.Timer.<init>(Timer.java:122)
at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS.getPooledConnection(DriverAdapterCPDS.java:177)
at org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS.getPooledConnection(DriverAdapterCPDS.java:132)
at org.apache.commons.dbcp.datasources.KeyedCPDSConnectionFactory.makeObject(KeyedCPDSConnectionFactory.java:158)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:747)
at org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:167)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:669)
at org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:653)
at org.apache.torque.TorqueInstance.getConnection(TorqueInstance.java:749)
at org.apache.torque.Torque.getConnection(Torque.java:289)
at org.apache.torque.util.Transaction.beginOptional(Transaction.java:108)
at org.apache.torque.util.BasePeer.doSelect(BasePeer.java:763)
at com.konakart.bl.ConfigurationMgr.getConfiguration(Unknown Source)
at com.konakart.app.KKEng.getConfiguration(Unknown Source)
at com.konakart.app.KKEng.getConfiguration(Unknown Source)
at com.konakart.al.ConfigCacheUpdater.updateConfigs(Unknown Source)
at com.konakart.al.ConfigCacheUpdater.run(Unknown Source)
Does anybody know what may be the reason and solution?
Here is our extra params for tomcat:
JAVA_OPTS="-d64 -server -Xms200M -Xmx360M -XX:MaxPermSize=140M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
These can be difficult and very dependent on your environment.
This article might be handy:
http://www.talkingtree.com/blog/index.cfm/2005/3/11/NewNativeThread
Perhaps you could do as it suggests at the bottom? Maybe try a smaller stack size? (-Xss)
Also interesting is:
http://webcache.googleusercontent.com/search?q=cache:http://www.egilh.com/blog/archive/2006/06/09/2811.aspx
Hello,
The "-Xss128k" solution does not seem to work.
Here is our JAVA_OPTS:
JAVA_OPTS="-d64 -server -Xss128k -Xms200M -Xmx360M -XX:MaxPermSize=140M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
And here is the random exception:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.apache.catalina.core.StandardThreadExecutor.execute(Unknown Source)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(Unknown Source)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
You'd probably have more success searching the web for answers to this because it's more of a java/tomcat/platform issue than a KonaKart problem.
Anyway... a couple more ideas:
Have you tried an even lower Xss value such as 64k? May as well keep going until you get stack overflows.
Also, on Linux, I believe (I've never tried this) you need to set the Linux thread stack size to the same value as the JVM stack size to get full benefits. eg use "ulimit -s <size in kb>".
Therefore you could try a 128kb stack with your -Xss128k
Thanks for the answers.
I wrote the email to the hosting provider, they set the number of active threads into 100 and the problem seemed as solved now.
The only regular error now is:
Aug 6, 2010 5:15:28 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet action threw exception
javax.servlet.jsp.JspException: Cannot find bean: "konakartKey" in any scope
at org.apache.struts.taglib.TagUtils.lookup(TagUtils.java:935)
...
And the tomcat needed to be restarted for the last 30 days just once.
Indeed the error is not fully regular (rather accidental) :).