Java Web Application Upgrade Story From 1.2 to 1.5: Lessons Learned

We were busy during past months while upgrading our ERP application. The upgrade wasn’t a piece of cake for us. There are always 2 rules about upgrade; it has a cost and it is painful. If it contains many elements, then your compatibility combat begins. I want to share our journey and some lessons learned. We transformed 70 web applications and about 1000 db tables to new version. We couldn’t upgrade to the latest versions because of license and fear of probable bugs on latest versions.

Let’s look at the versions and elements of upgrade:

Before:
Java 1.2, WebSphere 3.5, DB2 7.1, VisualAge For Java 3.5, Team Repository

After:
Java 1.5 (5), WebSphere 6.1, DB2 9.1, Rational Application Developer For WebSphere 7.5, ClearCase 7.1


1. The most changing part was application configuration: In previous WebSphere version, we were using the file system for application elements and that path was defined as classpath in application server, that was ver easy with some disadvantages. In Java EE 5, EAR and WAR package structure are used. We had to map our applications to this new package system. Session, classpath and classloader behaviours are very closely related with this mapping.

There were 2 options for application packaging:
a-Packaging each web application as WAR within one EAR(One EAR): In WebSphere, we can’t restart WARs individually and install options are generally bound to EAR structure. Because of that, we didn’t select this option.

b-Packaging each web application as one WAR and one EAR(Many EARs): We used this option. In this option, every application can be started, stopped, upgraded without stopping application server. However, there is a shared session problem. In Java EE 5 specs, every WAR has its own independent session. WebSphere supports shared session among WARs within only one EAR.

Shared Session Problem Solution
We needed shared session through all web applications and EARs. What we did was, to develop “Shared Session Support Among EARs” ourselves. It wasn’t easy to achieve this, we had to use session invalidation events and then temporarily store the objects of invalidating session. When user enters that application, we filled the new session with the previuosly created session objects. Another interesting point is that WebSphere uses same Session ID for new web applicaions. For a certion user, session objects are differenet but session ID is same.

Shared Library Problem Solution
Another problem was configuring application classpaths. We needed 2 basic classpath (now called as “shared lib” in WebSphere) requirements for library visibility:

a- Inter-Application-Level Classpath: For instance, one application needs to see a certaion application classes. We should provide that visibility without adding to global container-level classpath. Fortunately, a WebSphere extension eanbled us to make such a configuration. What we did was to add WEB-INF/classes path to the shared lib section of other application. One note about that is you should choose “ClassLoader Multiple Policy”, otherwise your application restart doesn’t take effect since classloaders are not renewed.

b-Container-Level Classpath: This is required if you need global visibility of some libraries. This is achieved in WebSphere with ClassLoader creation in Admin Console. You firstly define your library as a shared lib. Then you create a new classloader containing that shared lib entries.


2. DB2 is perfectly stable: "DB2 is DB2 is DB2." This mantra holds true. The easiest element of upgrade was DB2. What we need is to restore a 7.1 backup image to 8.1 then again to restore its backup to 9.1 version. You can’t restore a backup of 7.1 version to directly to 9.1 version. One extra work was to migrating an AIX database to the Windows system. In DB2, you can’t restore a backup image from other OSs. There is a utility to export and import form one system to another: db2move

3. The hardest part was to configure ClearCase environment: From VisualAge, we exported source codes, then imported to the RAD, that was a straightforward task. The problematic part is to build a Team environment for source code. In VAJ, there were a basic and simple Team system. We had to become familiar with versioning concepts like “Check-in”, “Check-out”, “Hijack”, “Baseline”, “Branch”, “Activity” etc. Initial setup and configuration of ClearCase server was another rough job. We had to outsource that job to an external consultant. I strongly recommend that you should consult to someone else if you are not familiar with ClearCase and Eclipse connection. There are many confusing configuration options for ClearCase. We used web-client type with snapshot views. Its performance is not good but enough for now.

4. Upgrade periodically for performance: Yes, to update frequently is a cumbersome for agile teams but becoming too late may increase upgrade cost. New versions may bring not only new features but new performance improvements. In Java 1.2, the most bothering part was GC; it was stopping all applications. With Java 1.5, “Incremantal GC” performance is excellent. After the upgrade, users were very satisfied with performance. Report execution times decreased from 10 minutes to 10 seconds. Every team should plan and reserve a time for upgrade in their agenda.

5. SSD disks versus HDD disks: We bought some new servers during upgrade. In one server configuration, we used SSD disk for its famous performance advantages for db-driven applications. One mistake of us was to buy only 2 SSD disks, and using it with RAID1(mirroring) configuration. Its performance was equivalent with 4 SAS disks with RAID5 configuration. We needed to buy 2 new SSD disk(Using RAID5) for better performance. I found a benchmark which depict the case.

As I understand, even we use 4 SSD disks, its speed won’t be superior as we expected. I read many articles mentioning 4 or 10 times performance but we didn’t get it. (Our new servers were Dell PowerEdge.). Additionally, SSD disks are highly expensive compared with classical HDDs.

6. 32 bit versus 64 bit: We were unable to use 64 bit versions for some reasons. 64 bit version should be supported by OS, DB2 and WebSphere. For WebSphere, 64 bit means more heap space more than 1.5 GB. For DB2, 64 bit means larger memory for DB which is more than 3 GB. In 64 bit, DB2 buffer pools can leverage more memory and that may result good performance gain. WebSphere is also bundled as as 32-bit and 64-bit. WebSphere Express edition currently doesn't support 64-bit, expecting 64 support in coming releases.

Java EE Web Development Without JSF

As a Java web development team, we’re frequently asked if we plan to use JSF in our web applications. After some hearings from my developer colleagues about their experiences using JSF, I decided to analyze if JSF is right technology for some problems of web development. After reading specifications, inspecting implementation libraries, I reached the conclusion of not using JSF. Even JSF 2.0 version is not enough to change this result. Here, I’ll try to give my reasons for this decision. Although Sun was pushing this technology by removing the mention of JSP form the Java EE 6 Tutorial, we are expecting JSP to live in the future. JSP technology is simple and direct solution of a certain problem. You can get same functions promised by JSF with JSP and JavaScript with less cost. JSF is a big engineering effort but the problem lies within the nature of web architecture.

I’m not alone for having this point of view. There are many voices having doubts about this technology:

http://ptrthomas.wordpress.com/2009/05/15/jsf-sucks/
http://weblogs.java.net/blog/edburns/archive/2010/01/22/analysis-peter-thomass-jsf-critical-rant
http://www.theserverside.com/news/1364127/Rethinking-JSF-The-Real-Problem
http://www.theserverside.com/news/thread.tss?thread_id=43899
http://www.chrislhardin.com/2009/08/java-server-faces-vs-ajax-frameworkswhy.html
http://weblogs.java.net/blog/2006/06/07/programmer-productivity-jsf-and-netbeans

Here are my opinions about the disadvantages of JSF:

1- JSF Introduces A Redundant UI Layer: As I mentioned in my previous posts, complete server-side UI componentization is problematic since in web applications we have 2 runtimes; server-side and client-side. If we use a heavy server-side solution, we won’t have enough energy to develop in client-side. However, same functionality can be achieved with simple JSP and JavaScript code. Some developers tried to ignore client-side and JavaScript but the client-side is a very important part of web applications and can’t be hidden. As the JavaScript engines gets better and HTML gets new exciting improvements (add AJAX move), client-side gets more importance. We need lightweight component libraries with client-side abilities. If we talk about components, we should consider client-side components. MS IE’s HTC and Firefox XBL are some efforts on this direction. In recent HTML specifications, I expect a standard for HTML components (XBL 2.0) but it seems difficult to see widespread browser implementations. We are currently using MS IE’s HTC components for some simple behaviors like number-date-time formatting, AJAX-enabled input fields etc. Some components may be client-only but some big components may also need server-side support (For instance our DataGrid row is posted to server to save its state).

2- Tag Soup: When I see some new tags invented by a web framework, I admit its failure is started. Web standards became successful since they were simple and standard. New element names, new attribute names, new behaviors and new event models terribly increases learning curve without any real benefits. What I say is don’t reinvent current HTML tags. I think current DOM, JavaScript and CSS namespaces are already too extensive that a programmer could learn.

3- Losing Extensibility and Flexibility: Component encapsulation makes it hard to modify UI behaviors. Let’s say you want to insert some HTML code into 3rd row of a data grid. That may not be available in current API set. There may be many scenarios that were simple in HTML base declarative form. I inspected many JSF list components and they are nothing more than simple HTML tables. Why do we pay extra complexity cost for little earning of encapsulation of HTML? Even most of JSF components are just repetition of current HTML elements. You can find same function set of JSF implementations within JS libraries, nothing is new.

4- Complexity of Binding, Bean Lifecycle, Navigation, Scope and Event Model: To get basic grasp of JSF is much more difficult than request-based server models. MVC simplicity is also lost. You can observe that complexity in increasing development times or difficulties in debugging and fixing problems of applications. Designing UIs is just another story, common denominator between programmer and graphics designer is lost, every programmer should be also good designer. A very strong IDE support is required to hide some complexities like WebObjects did years ago.

5- Performance: I’ve heard some complaints about JSF performance, it may be eliminated but It is a fact that it would anyway consume valuable server resources. As we develop complex components with JSF, their performance degradation would be a new area of bottleneck analysis.


In Java EE stack, some standards never got enough adoption and that lead developers to find better, simpler solutions. I think JSF is moving on the same direction with EJB. Some developers may feel themselves comfortable by using JSF; good engineering, clean and structured code but I don't.

So, what is alternative? My suggestion is to use JSPs, servlets and JavaScript libraries. You may use some web frameworks or JS libraries to get ready-use functions. (Note that, some web frameworks have similar problems with JSF) If you still need HTML componentization, you can use XBL in Firefox or HTC in Internet Explorer (later waiting XBL 2.0 standard adoption). In web application architecture, unfortunately we’ve always ignored client computing resources. PCs are getting faster, browser engines are becoming faster. It is time to leverage client-side as much as possible.

Enterprise Application Integration (EAI) with Integration Engine

Every application lives in an ecosystem of different types of software. Some of them are platform software and you use by visibly calling some APIs or invisibly behind the JVM. Application Servers, HTTP Servers, DBMSs are platform software for enterprise applications. EAI is somewhat different. It is in application to application (A2A) or business to business (B2B) style.

Although ERP systems try to meet all business application requirements of an enterprise, IT systems generally have many different applications for some reasons. Sometimes those applications need to share data because business process may span across those applications. Every application manages only a piece of business process. The need for integration of those applications depends on data flow volume. If data volume is high, users wouldn’t want to enter same data to each application again and again (sometimes this maybe impossible). Application integration prevents data errors, diminishes user data entry but it has a cost. Best option for integration is not to do it. Every integration project requires detailed analysis. Every integration decreases application functionality and increase system fragility. For these reasons, ERP systems are indispensable for their highly integrated architecture when imagining islands of many applications.

For integration requirements, ERP systems have some integration capabilities from direct table integration to web service functions. In core modules of ERP, inter-module function calls already requires an integration layer. ERP system exposes those functions to the outside users.

We also recently added this capability to our ERP system. I’m going to share some architectural design tips. At the beginning, when analyzing an integration system, we were considering using messaging system as an integration backbone. However as walking through requirements, I realized that messaging system can only be a part of integration, i.e. for EDI (Electronic Data Interchange). Messaging system usage for file integration would be somewhat cumbersome. Then, “Integration Engine” idea shined. Yes, a new engine is best solution for integration functions. We worked previously to add EDI integration to the messaging system, but we realized that “Integration Engine” is best-suited and neater for the whole integration problem.

Let me summarize, what we thought for function set of “Integration Engine”:
• Integration should be easily developed and plugged to the running system without any compilation which may require IDE. This could be possible by leveraging dynamic programming facility of Rule Engine (see my previous article for that).

• Integration should be possible for any system that Java can talk. Yes, since integration is done through Java programming, we are only limited to Java (By calling web services from Java, this border also expanded). We have supported following type of integrations; Server File, Client File (From Browser), Web Service, EDI Message, Other.

• Integration should work outbound or inbound. That means we can export data to the outer system or import data form outer system.

• Integration functions can be triggered by outer system via web service APIs.

• Integration can work as a batch process. Let’s say you need to continuously monitor a file which is filled by an outer program.

• Integration can be queued. You may need sometimes to approve data then take it to the system. Let’s say you have a web service and want to let data entry to system by controlling validity.

• When running integrations, user may need to give some extra data or edit some data. We designed some integration pages for users in order to run integration, to give parameters and to edit data coming from integration source.

• Field format can be specified. Data format is one of the problems in integration efforts. We provided to give format definition (we used Java format definitions) for numbers, date, time or timestamp.

• Excel integration. Spreadsheet applications are most-wanted applications for internal integration. We make it possible via macro calls to our web services (More portable solution for different spreadsheet applications when compared with platform-specific integration methods).

• Integration usage should be simple. We added an integration link to the module main page to easily navigate between data and integration page.






An integration engine is simply composed of 2 parts; Communicator and Processor. In both export and import situations, these 2 parts are used. These parts are dynamic Java programs and communicate with each other with generic data objects. Communicator is responsible for outer systems. It connects to outer system and pushes or pulls data. Then, it builds a generic data object and passes it to Processor or vice versa. Processor is responsible for dealing with internal system. In export, it access to the internal system via integration APIs and then builds generic data objects and pass it to the Communicator. In import, it receives data objects from Communicator then it calls integration APIs for saving data.

The architecture is as following: