A late night fix that saved a release
The deployment finished just before midnight. We had pizza, a green light from QA, and a room full of tired grins. Fifteen minutes later the pager went off and the logs shouted ClassNotFoundException for a library that had worked on every developer laptop. The app ran on IBM WebSphere Application Server and we had bundled a few common jars. WebSphere had its own flavor of a couple of those. The result was a quiet little war for class control.
We did not roll back. We opened the Admin Console, switched the application to Parent Last class loading, restarted the app, and watched the errors disappear. Traffic flowed, dashboards turned green, and the team got their night back. A week later we tripped on a different wire. The JDBC connection pool had been left on default values that worked in test but folded in production. Threads piled up. The database team saw spikes. We tuned pool sizes and the statement cache, set a clear purge policy, and that was that.
Two fixes. Same lesson. In WebSphere, class loaders and JDBC pools decide if your app starts clean and stays fast. Here is the playbook I wish I had printed above my desk the first time around.
Classloader tuning that avoids jar fights
WebSphere gives you knobs under Enterprise Applications then your app then Class loading and update detection. The big ones:
- Class loader order: Parent First or Parent Last.
- WAR class loader policy: Application or Module.
- Reload interval: for development only. Turn it off in production.
How to choose:
- Parent First is safe when you want WebSphere platform libraries to win. Use this when you do not ship your own XML parsers or logging frameworks.
- Parent Last is your friend when your app brings its own version of common libraries. Think commons logging, Axis, activation and mail, or your own XML jars. This setting lets the app version win and keeps you from wrestling with container jars.
- WAR class loader policy: Choose Module when each web module must be isolated from others inside the same enterprise app. This keeps two teams from stepping on each other with different versions of the same jar. Choose Application when you expect jars to be shared across modules by design.
A simple pattern that works for teams that ship their own stack: set the app to Parent Last and set the WAR class loader policy to Module. Then move shared stuff to an enterprise level library.
About shared libraries. Put vendor drivers and cross app libraries into a Shared Library at the server or node level and reference them from each enterprise app. That gives you a single upgrade point for the Oracle or DB2 driver. It also prevents one app from bundling a driver that hurts another app during class resolution. In the Admin Console open Environment then Shared Libraries, create one, and attach it to the server or the app as needed.
Watch for classic conflicts:
- commons logging versus WebSphere logging bridges
- Xerces and Xalan when you also rely on platform XML
- Axis and activation or mail jars
- Multiple versions of the same driver tucked into different WAR files
One more note for daily life. Set Reload interval only in dev and keep it low for RAD cycles. In production, disabled reload means fewer file system checks and less surprise. That also nudges you to deploy like you run, which avoids the classic it worked in my IDE line.
JDBC connection pools that stay steady under load
Defaults look friendly and then the traffic hits. The pool fills, threads wait, and error rates creep up. The good news is that WebSphere gives you the right levers on each Data source. Focus on these:
- Min and Max connections: Pick the number that matches the thread pool and the database capacity. If your servlet container has 50 possible worker threads, do not set max at 10 and hope for the best. A starting point I like is min 5 to 10 and max equal to or a little below your web container max threads. Then measure.
- Connection timeout: How long an app thread will wait for a free connection. Keep it short enough to fail fast under pressure so callers can back off.
- Unused timeout and Reap time: Pair these to clean idle connections. This keeps the pool fresh and cuts random socket errors after quiet periods.
- Aged timeout: Force close of connections that live too long. Very useful when a firewall drops long lived sockets. Set it a bit below the firewall idle drop setting.
- Purge policy: I prefer EntirePool when the database bounces or the network splits. It clears bad sockets quickly. If you must keep some connections alive, try failing connection only, but watch for slow poison.
- Statement cache size: Start around 25 per connection for Oracle, 25 to 50 for DB2, then watch memory and hit rates. A good cache cuts parse time and helps under load.
- XA versus non XA: Use XA only when you need a real global transaction across more than one resource. If the app hits one database and a single JMS session is not enlisted, a plain data source usually wins on speed.
Driver tips that have saved me time:
- Put the JDBC driver jars in a shared library rather than in the enterprise app. You get clean upgrades and no duplicate classes inside your WAR files.
- Confirm the driver and database versions with the DBA team. A tiny mismatch can show up as odd errors long after go live.
- For Oracle, set a login timeout and make sure the network team knows your keep alive and firewall rules. Pair that with aged timeout so dead sockets do not linger.
Measure with PMI. Turn on JDBC metrics like PoolSize, FreePoolSize, WaitTime, CreateCount, DestroyCount, and PreparedStatementCount. Run a repeatable load test and change one thing at a time. You will see the curve flatten as the pool hits the right size and the cache stops thrashing.
What managers should ask this week
If you lead a team that runs on WebSphere, you do not need to tweak every knob yourself. You do need a clear checklist and a shared vocabulary. Ask for these items:
- A one page class loader policy for each enterprise app with the reason for the choice
- A list of shared libraries with versions for JDBC drivers and common jars
- Documented JDBC pool settings and the numbers that justify them under expected load
- A rollback plan for class loader changes and driver upgrades
- A short PMI report from the last load test with graphs for pool growth and wait time
This turns tuning from folklore into a repeatable step. It also keeps production stable when team members rotate or when a new app arrives with a different set of libraries.
Your turn
Open your Admin Console and do a quick review:
- Check one enterprise app that matters and confirm Parent Last if you bundle common libraries. Flip a lower risk app and verify that the logs stay clean.
- Switch WAR class loader policy to Module if two web modules ship different jar sets.
- Move your driver jars into a Shared Library and attach it to the server that runs the apps.
- On one data source, set min, max, unused timeout, reap time, aged timeout, and statement cache size with intentional values. Write them down.
- Run a thirty minute soak test and watch PMI. If PoolSize sits below your worker threads and WaitTime climbs, raise max. If memory climbs and PreparedStatementCount explodes, trim the cache.
Small changes in class loaders and JDBC pools decide whether your next release is a quiet night or a long one. Make the quiet night the default.