From The Principle of Too Much Magic (written early 2003):
Many programmers have a tendency to be Too Smart. Being Smart, they introduce various kinds of Magic into their software. This will usually lead to a software design that is more generic, dynamic and flexible than is needed, has more features than is needed, and is hence much more complex than is needed.
I learned that lesson, and I learned it really, really well. It seems that a large part of the java developer community didn’t. We were greeted with this stack trace the other day:
[2008/03/21 11:54:32.710] WARN [openjpa.Runtime] java.lang.NullPointerException: at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org.apache.openjpa.kernel.AbstractBrokerFactory.loadPersistentTypes(AbstractBrokerFactory.java:265) at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:197) at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:142) at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:192) at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:145) at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:56) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean$ManagedEntityManagerFactoryInvocationHandler.invoke(AbstractEntityManagerFactoryBean.java:375) at $Proxy11.createEntityManager(Unknown Source) at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.createEntityManager(OpenEntityManagerInViewFilter.java:161) at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:102) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75) at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:471) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568) at org.mortbay.http.HttpContext.handle(HttpContext.java:1530) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:636) at org.mortbay.http.HttpContext.handle(HttpContext.java:1482) at org.mortbay.http.HttpServer.service(HttpServer.java:909) at org.mortbay.http.HttpConnection.service(HttpConnection.java:820) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:986) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:837) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:245) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
- The servlet engine is jetty 5, the JDK is JRockit.
- This is a spring-based web application. It uses the ContextLoaderListener to initialize spring, and an applicationContext.xml file to tell spring what beans to create.
- Some other part of the app (not actually this servlet) uses JPA (with OpenJPA) for persistence, for which there is of course a persistence.xml.
- Transaction management uses spring’s <tx:annotation-driven/>.
In other words, it’s pretty much how you would write a spring-and-jpa web application after reading the spring manual and the jpa manual and following their examples. Some scenario detail:
- The stack trace happens on first servlet invocation.
- It doesn’t always happen. It seems to happen more when the application is restarted while the cluster is under load.
The quiz question: what’s happening here?
Here’s a hint: classloaders are not normally thread-safe.
The bonus question: which one of dynamic proxies, byte code engineering, runtime aspect weaving, xml-based domain specific declarative scripting language, declarative transaction management, inversion of control, separation of concerns, reflection, classloader hacking, interceptor chaining, filter chaining, resource pooling, is not used in this scenario?
The job interview question: what would you do if you had a 9-machine cluster in production with this app on it, and then 2 of the nodes suddenly suffer from this problem on a restart?
LSD::RELOAD 
Hi Leo,
Fast solution I would apply (risking the use of more magic):
– If it is a concurrency issue and IF only the OpenJPA loadPersistentTypes() method is loading those “persistence types” I would force loadPersistentTypes() to synchronize on a singleton monitor.
You can achieve this by editing its source code or by instrumenting it (I would use an around aspect via AspectJ and statically instrument the OpenJPA library to avoid even more classloader “magic”).
If another loading point exists for loadPersistentTypes() (and that is easy to find via static initializers logging the call stack… which can also be statically applied with AspectJ) I would also force the synchronization of that code on the same singleton monitor used for loadPersistentTypes().
This solution is simplistic – specially the use of a singleton as a synchronization monitor – ant it _MIGHT_ have a performance cost, but it also has the advantages of (1) having a lower chance of causing further synchronization problems and, even if that happens, they (2) should be easier to diagnose.
I must say I took a look at the loadPersistentTypes() source code and I remain a bit suspicious of its thread safety, even without the ClassLoader issues you mention:
http://svn.apache.org/viewvc/openjpa/trunk/openjpa-kernel/src/main/java/org/apache/openjpa/kernel/AbstractBrokerFactory.java?view=markup
On your claims about Magic:
More than too much magic you have magic from multiple wizards. And some of the magic being used (e.g.: bytecode instrumentation) can be quite dangerous when many wizards are providing it. (Is there a clear standard way of applying code instrumentation without causing a mess when you have multiple instrumentation providers?)
As usual, it is a matter of cost benefit with an emphasis on control. (Because Lack of Control always costs more than most people seem to expect).
I have been successfully some of the technologies you qualify as magic on your 2003 article (like AOP/bytecode instrumentation, introspection, dynamic proxies, annotations, etc.).
An example: I developed a logging library using all of these because I found this would save a lot of coding and make remaining code much more readable on applications with extensive logging needs.
– The cost: some developers will not always understand what is happening / how it works;
– The benefit: having much more readable code cuts a lot on development and maintenance costs.
Why am I “bragging” about this?
– Because I miss some exploration following the cost/benefit perspective on the 2003 article you refer.
The article is extensive but it would be more useful if you could be more explicit about “creating” a cost/benefit mindset on the reader.
Have fun,
Paulo Gaspar
(from Avalon days)
job interview answer:
step 1: be happy you had 9 nodes, now you have 7
step 2: print out your essay and leave it on the developers desks
step 3: get that stack trace printed on some toilet paper rolls and re-stock the bathrooms.
step 4: take the node out of the cluster, restart it, load it once so you know you’re the only thread hitting it, ensure its working normally, then re-introduce to the cluster.
🙂