Java, ERP, Software Architecture

Java Web Application Upgrade Story From 1.2 to 1.5: Lessons Learned

2011-07-16T11:28:00.000-07:00

We were busy during past months while upgrading our ERP application. The upgrade wasn’t a piece of cake for us. There are always 2 rules about upgrade; it has a cost and it is painful. If it contains many elements, then your compatibility combat begins. I want to share our journey and some lessons learned. We transformed 70 web applications and about 1000 db tables to new version. We couldn’t upgrade to the latest versions because of license and fear of probable bugs on latest versions.

Let’s look at the versions and elements of upgrade:

Before:
Java 1.2, WebSphere 3.5, DB2 7.1, VisualAge For Java 3.5, Team Repository

After:
Java 1.5 (5), WebSphere 6.1, DB2 9.1, Rational Application Developer For WebSphere 7.5, ClearCase 7.1

1. The most changing part was application configuration: In previous WebSphere version, we were using the file system for application elements and that path was defined as classpath in application server, that was ver easy with some disadvantages. In Java EE 5, EAR and WAR package structure are used. We had to map our applications to this new package system. Session, classpath and classloader behaviours are very closely related with this mapping.

There were 2 options for application packaging:
a-Packaging each web application as WAR within one EAR(One EAR): In WebSphere, we can’t restart WARs individually and install options are generally bound to EAR structure. Because of that, we didn’t select this option.

b-Packaging each web application as one WAR and one EAR(Many EARs): We used this option. In this option, every application can be started, stopped, upgraded without stopping application server. However, there is a shared session problem. In Java EE 5 specs, every WAR has its own independent session. WebSphere supports shared session among WARs within only one EAR.

Shared Session Problem Solution
We needed shared session through all web applications and EARs. What we did was, to develop “Shared Session Support Among EARs” ourselves. It wasn’t easy to achieve this, we had to use session invalidation events and then temporarily store the objects of invalidating session. When user enters that application, we filled the new session with the previuosly created session objects. Another interesting point is that WebSphere uses same Session ID for new web applicaions. For a certion user, session objects are differenet but session ID is same.

Shared Library Problem Solution
Another problem was configuring application classpaths. We needed 2 basic classpath (now called as “shared lib” in WebSphere) requirements for library visibility:

a- Inter-Application-Level Classpath: For instance, one application needs to see a certaion application classes. We should provide that visibility without adding to global container-level classpath. Fortunately, a WebSphere extension eanbled us to make such a configuration. What we did was to add WEB-INF/classes path to the shared lib section of other application. One note about that is you should choose “ClassLoader Multiple Policy”, otherwise your application restart doesn’t take effect since classloaders are not renewed.

b-Container-Level Classpath: This is required if you need global visibility of some libraries. This is achieved in WebSphere with ClassLoader creation in Admin Console. You firstly define your library as a shared lib. Then you create a new classloader containing that shared lib entries.

2. DB2 is perfectly stable: "DB2 is DB2 is DB2." This mantra holds true. The easiest element of upgrade was DB2. What we need is to restore a 7.1 backup image to 8.1 then again to restore its backup to 9.1 version. You can’t restore a backup of 7.1 version to directly to 9.1 version. One extra work was to migrating an AIX database to the Windows system. In DB2, you can’t restore a backup image from other OSs. There is a utility to export and import form one system to another: db2move

3. The hardest part was to configure ClearCase environment: From VisualAge, we exported source codes, then imported to the RAD, that was a straightforward task. The problematic part is to build a Team environment for source code. In VAJ, there were a basic and simple Team system. We had to become familiar with versioning concepts like “Check-in”, “Check-out”, “Hijack”, “Baseline”, “Branch”, “Activity” etc. Initial setup and configuration of ClearCase server was another rough job. We had to outsource that job to an external consultant. I strongly recommend that you should consult to someone else if you are not familiar with ClearCase and Eclipse connection. There are many confusing configuration options for ClearCase. We used web-client type with snapshot views. Its performance is not good but enough for now.

4. Upgrade periodically for performance: Yes, to update frequently is a cumbersome for agile teams but becoming too late may increase upgrade cost. New versions may bring not only new features but new performance improvements. In Java 1.2, the most bothering part was GC; it was stopping all applications. With Java 1.5, “Incremantal GC” performance is excellent. After the upgrade, users were very satisfied with performance. Report execution times decreased from 10 minutes to 10 seconds. Every team should plan and reserve a time for upgrade in their agenda.

5. SSD disks versus HDD disks: We bought some new servers during upgrade. In one server configuration, we used SSD disk for its famous performance advantages for db-driven applications. One mistake of us was to buy only 2 SSD disks, and using it with RAID1(mirroring) configuration. Its performance was equivalent with 4 SAS disks with RAID5 configuration. We needed to buy 2 new SSD disk(Using RAID5) for better performance. I found a benchmark which depict the case.

As I understand, even we use 4 SSD disks, its speed won’t be superior as we expected. I read many articles mentioning 4 or 10 times performance but we didn’t get it. (Our new servers were Dell PowerEdge.). Additionally, SSD disks are highly expensive compared with classical HDDs.

6. 32 bit versus 64 bit: We were unable to use 64 bit versions for some reasons. 64 bit version should be supported by OS, DB2 and WebSphere. For WebSphere, 64 bit means more heap space more than 1.5 GB. For DB2, 64 bit means larger memory for DB which is more than 3 GB. In 64 bit, DB2 buffer pools can leverage more memory and that may result good performance gain. WebSphere is also bundled as as 32-bit and 64-bit. WebSphere Express edition currently doesn't support 64-bit, expecting 64 support in coming releases.

Java EE Web Development Without JSF

2010-08-22T07:57:00.000-07:00

As a Java web development team, we’re frequently asked if we plan to use JSF in our web applications. After some hearings from my developer colleagues about their experiences using JSF, I decided to analyze if JSF is right technology for some problems of web development. After reading specifications, inspecting implementation libraries, I reached the conclusion of not using JSF. Even JSF 2.0 version is not enough to change this result. Here, I’ll try to give my reasons for this decision. Although Sun was pushing this technology by removing the mention of JSP form the Java EE 6 Tutorial, we are expecting JSP to live in the future. JSP technology is simple and direct solution of a certain problem. You can get same functions promised by JSF with JSP and JavaScript with less cost. JSF is a big engineering effort but the problem lies within the nature of web architecture.

I’m not alone for having this point of view. There are many voices having doubts about this technology:

http://ptrthomas.wordpress.com/2009/05/15/jsf-sucks/
http://weblogs.java.net/blog/edburns/archive/2010/01/22/analysis-peter-thomass-jsf-critical-rant
http://www.theserverside.com/news/1364127/Rethinking-JSF-The-Real-Problem
http://www.theserverside.com/news/thread.tss?thread_id=43899
http://www.chrislhardin.com/2009/08/java-server-faces-vs-ajax-frameworkswhy.html
http://weblogs.java.net/blog/2006/06/07/programmer-productivity-jsf-and-netbeans

Here are my opinions about the disadvantages of JSF:

1- JSF Introduces A Redundant UI Layer: As I mentioned in my previous posts, complete server-side UI componentization is problematic since in web applications we have 2 runtimes; server-side and client-side. If we use a heavy server-side solution, we won’t have enough energy to develop in client-side. However, same functionality can be achieved with simple JSP and JavaScript code. Some developers tried to ignore client-side and JavaScript but the client-side is a very important part of web applications and can’t be hidden. As the JavaScript engines gets better and HTML gets new exciting improvements (add AJAX move), client-side gets more importance. We need lightweight component libraries with client-side abilities. If we talk about components, we should consider client-side components. MS IE’s HTC and Firefox XBL are some efforts on this direction. In recent HTML specifications, I expect a standard for HTML components (XBL 2.0) but it seems difficult to see widespread browser implementations. We are currently using MS IE’s HTC components for some simple behaviors like number-date-time formatting, AJAX-enabled input fields etc. Some components may be client-only but some big components may also need server-side support (For instance our DataGrid row is posted to server to save its state).

2- Tag Soup: When I see some new tags invented by a web framework, I admit its failure is started. Web standards became successful since they were simple and standard. New element names, new attribute names, new behaviors and new event models terribly increases learning curve without any real benefits. What I say is don’t reinvent current HTML tags. I think current DOM, JavaScript and CSS namespaces are already too extensive that a programmer could learn.

3- Losing Extensibility and Flexibility: Component encapsulation makes it hard to modify UI behaviors. Let’s say you want to insert some HTML code into 3rd row of a data grid. That may not be available in current API set. There may be many scenarios that were simple in HTML base declarative form. I inspected many JSF list components and they are nothing more than simple HTML tables. Why do we pay extra complexity cost for little earning of encapsulation of HTML? Even most of JSF components are just repetition of current HTML elements. You can find same function set of JSF implementations within JS libraries, nothing is new.

4- Complexity of Binding, Bean Lifecycle, Navigation, Scope and Event Model: To get basic grasp of JSF is much more difficult than request-based server models. MVC simplicity is also lost. You can observe that complexity in increasing development times or difficulties in debugging and fixing problems of applications. Designing UIs is just another story, common denominator between programmer and graphics designer is lost, every programmer should be also good designer. A very strong IDE support is required to hide some complexities like WebObjects did years ago.

5- Performance: I’ve heard some complaints about JSF performance, it may be eliminated but It is a fact that it would anyway consume valuable server resources. As we develop complex components with JSF, their performance degradation would be a new area of bottleneck analysis.

In Java EE stack, some standards never got enough adoption and that lead developers to find better, simpler solutions. I think JSF is moving on the same direction with EJB. Some developers may feel themselves comfortable by using JSF; good engineering, clean and structured code but I don't.

So, what is alternative? My suggestion is to use JSPs, servlets and JavaScript libraries. You may use some web frameworks or JS libraries to get ready-use functions. (Note that, some web frameworks have similar problems with JSF) If you still need HTML componentization, you can use XBL in Firefox or HTC in Internet Explorer (later waiting XBL 2.0 standard adoption). In web application architecture, unfortunately we’ve always ignored client computing resources. PCs are getting faster, browser engines are becoming faster. It is time to leverage client-side as much as possible.

Enterprise Application Integration (EAI) with Integration Engine

2010-03-17T04:28:00.000-07:00

Every application lives in an ecosystem of different types of software. Some of them are platform software and you use by visibly calling some APIs or invisibly behind the JVM. Application Servers, HTTP Servers, DBMSs are platform software for enterprise applications. EAI is somewhat different. It is in application to application (A2A) or business to business (B2B) style.

Although ERP systems try to meet all business application requirements of an enterprise, IT systems generally have many different applications for some reasons. Sometimes those applications need to share data because business process may span across those applications. Every application manages only a piece of business process. The need for integration of those applications depends on data flow volume. If data volume is high, users wouldn’t want to enter same data to each application again and again (sometimes this maybe impossible). Application integration prevents data errors, diminishes user data entry but it has a cost. Best option for integration is not to do it. Every integration project requires detailed analysis. Every integration decreases application functionality and increase system fragility. For these reasons, ERP systems are indispensable for their highly integrated architecture when imagining islands of many applications.

For integration requirements, ERP systems have some integration capabilities from direct table integration to web service functions. In core modules of ERP, inter-module function calls already requires an integration layer. ERP system exposes those functions to the outside users.

We also recently added this capability to our ERP system. I’m going to share some architectural design tips. At the beginning, when analyzing an integration system, we were considering using messaging system as an integration backbone. However as walking through requirements, I realized that messaging system can only be a part of integration, i.e. for EDI (Electronic Data Interchange). Messaging system usage for file integration would be somewhat cumbersome. Then, “Integration Engine” idea shined. Yes, a new engine is best solution for integration functions. We worked previously to add EDI integration to the messaging system, but we realized that “Integration Engine” is best-suited and neater for the whole integration problem.

Let me summarize, what we thought for function set of “Integration Engine”:
• Integration should be easily developed and plugged to the running system without any compilation which may require IDE. This could be possible by leveraging dynamic programming facility of Rule Engine (see my previous article for that).

• Integration should be possible for any system that Java can talk. Yes, since integration is done through Java programming, we are only limited to Java (By calling web services from Java, this border also expanded). We have supported following type of integrations; Server File, Client File (From Browser), Web Service, EDI Message, Other.

• Integration should work outbound or inbound. That means we can export data to the outer system or import data form outer system.

• Integration functions can be triggered by outer system via web service APIs.

• Integration can work as a batch process. Let’s say you need to continuously monitor a file which is filled by an outer program.

• Integration can be queued. You may need sometimes to approve data then take it to the system. Let’s say you have a web service and want to let data entry to system by controlling validity.

• When running integrations, user may need to give some extra data or edit some data. We designed some integration pages for users in order to run integration, to give parameters and to edit data coming from integration source.

• Field format can be specified. Data format is one of the problems in integration efforts. We provided to give format definition (we used Java format definitions) for numbers, date, time or timestamp.

• Excel integration. Spreadsheet applications are most-wanted applications for internal integration. We make it possible via macro calls to our web services (More portable solution for different spreadsheet applications when compared with platform-specific integration methods).

• Integration usage should be simple. We added an integration link to the module main page to easily navigate between data and integration page.

An integration engine is simply composed of 2 parts; Communicator and Processor. In both export and import situations, these 2 parts are used. These parts are dynamic Java programs and communicate with each other with generic data objects. Communicator is responsible for outer systems. It connects to outer system and pushes or pulls data. Then, it builds a generic data object and passes it to Processor or vice versa. Processor is responsible for dealing with internal system. In export, it access to the internal system via integration APIs and then builds generic data objects and pass it to the Communicator. In import, it receives data objects from Communicator then it calls integration APIs for saving data.

The architecture is as following:

Designing Transaction Manager for Persistence Layer

2010-02-09T11:18:00.000-08:00

When writing persistence layer of our ERP applications, one of the critical part was transaction processing mechanism (OLTP). Transaction processing (TP) is critical since it affects every unit of system. If not properly designed or used, it may bring whole system down via database locks, deadlocks and isolation problems. Or it may cause data leaks and errors if not properly programmed. Transaction within database (ACID) is just one part of whole story. As application or framework designers, we are also concerned with TP behaviors in our applications. I’m going to try to give some details of designing a transaction system for a persistence framework.

Transaction Models
1- Raw JDBC Transactions: You can use Connection interface to handle database transactions. This model is simple and can be used in some certain circumstances.

2- Programmatic Transactions (Bean-Managed Transaction): You manage transaction lifecycle through methods calls; start, commit and rollback. javax.transaction.UserTransaction interface provides programmatic transactions. Programmatic transactions are flexible but more error-prone. If you don’t handle an exception of a transactional process, it may lock database since it is not committed or rolled back. It is more flexible in terms of transaction demarcation (transaction boundaries). You can control transaction boundaries more flexibly.

3- Declarative Transactions (Container-Managed Transaction): You declare transaction boundaries and attributes through definition files or annotations. EJB and Spring has many features to support that model.

Transaction Types:

1- Flat Transactions: The simplest type of transactions. Everything is atomic and processes are within one level of control. They all succeed or fail. JDBC transactions are flat.

2- Chained Transactions: Transactions run in a sequence and we can control each transaction’s behavior in commit and rollback situations. JDBC’s java.sql.SavePoint may be regarded as having similar function.

3- Nested Transactions: A nested transaction is a tree of transactions, the sub-trees of which are either flat or chained or nested transactions. Some TP systems or frameworks support it.

4- Distributed Transactions (XA): If multiple types of transactional resources are associated with a transaction, then it is said to be distributed. In this case, there are transaction managers coordinating each other to achieve 2-phase commit. A typical example is a database update query and a message delivery within the same transaction. Another example is “multiple database updates”. JTA and JTS include distributed transaction function. Note that, these transactions more expensive than flat ones.

How to Design TP Manager
We used programmatic transaction model and supported chained and nested transactions. In background we use JDBC connection transaction APIs with “Connection Passing” technique. Our persistent classes use our transaction APIs for transactional database operations. Persistent classes may be associated with a transaction vertically and horizontally like an object tree. Transaction propagation is possible in any direction. Any rollback logic can be programmed without any trick or hack. I think programmatic transactions are better for readability, control and flexibility. By hiding transaction codes for child persistent objects, we achieved simpler application code without any transactional code mess.

1- Transaction API: We used a DBTransaction object for transaction processing. It has 3 basic methods; start, commit and rollback. Transactions are not bound to current thread and can be passed through threads with no problem.


public final class DBTransaction {
    public DBTransaction(int nIsolationLevel)
    public void commit() throws DBException{...}
    public void rollback() throws DBException{...}
    public void start() throws DBException{...}
    ...
}

public static void testTransaction() throws Exception{
 DBTransaction dbTransaction = new DBTransaction();
 dbTransaction.start();
 try {
  DBEmployee dbEmployee1 = (DBEmployee)BOFactory.get(DBEmployee.class);
  //Attach dbEmployee1 to transaction manager
  dbEmployee1.setTransaction(dbTransaction);
  dbEmployee1.setName("Name1");
  dbEmployee1.setSurname("Surname1");
  dbEmployee1.setDepartmentCode("D1");
  dbEmployee1.dbSave();
 
  DBEmployee dbEmployee2 = (DBEmployee)BOFactory.get(DBEmployee.class);
  //Attach dbEmployee2 to transaction manager
  dbEmployee2.setTransaction(dbTransaction);
  dbEmployee2.setName("Name2");
  dbEmployee2.setSurname("Surname2");
  dbEmployee2.setDepartmentCode("D2");
  dbEmployee2.dbSave();
 
// 2 db operations in one transaction; Employee Insert, Employee Insert
  dbTransaction.commit();
 }
 catch(Exception e) {
  dbTransaction.rollback();
  throw e;
 }
}

2- Transactional Persistent Objects: We eased transactional programming through supporting automatic transactions within a persistent object’s child objects. That means one object’s all child objects database operations are executed within a transaction. To bypass this mechanism is also possible through removing transaction codes in child objects. We automatically propagate master transaction (executer) to the child (delegator) transactions. In this way, we got the ease of declarative model with flexibility of programmatic model.


//Persistent Object and Its Child Objects
public class DBEmployee extends DBObject {
 private int m_nNo;
 private String m_sName;
 private String m_sSurname;
 ...
 //1-1 relation
 private DBAddress m_dbAddress;
 //1-n relation
 private DBTaskList m_dbTaskList;
}

DBEmployee dbEmployee = (DBEmployee) BOFactory.get(DBEmployee.class);
dbEmployee.setName("Name1");
dbEmployee.setSurname("Surname2");
dbEmployee.setDepartmentCode("D1");

//1-1 Relatonal Object
DBAddress dbAddress = dbEmployee.getAddress();
dbAddress.setCity("City1");
dbAddress.setStreet("Street1");

//1-n Relatonal Object
DBTaskList dbTaskList = dbEmployee.getTaskList(); 
DBTask dbTask1 = (DBTask) BOFactory.get(DBTask.class);
dbTask1.setDescription("Description1");
dbTaskList.add(dbTask1);
DBTask dbTask2 = (DBTask) BOFactory.get(DBTask.class);
dbTask2.setDescription("Description2");
dbTaskList.add(dbTask2);

// Automatic transaction, no transactional code required for relational objects
// 4 db operations in one transaction; Address Insert, Employee Insert, Task Insert, Task Insert
dbEmployee.dbSave();

3- Transaction Ids and Logs: For transactional program control and debugging, transaction ID and logs are wonderful. The logs used for audit trailing also serve transactional operation list. Every database change query (DELETE, INSERT and UPDATE) should be executed within a transaction and every transaction should have a transaction ID. Transaction ID is saved within audit trail logs and we can see how many database changes occur within a certain transaction. A programmer can detect transaction leaks by using transaction logs (audit trail logs). The programmer runs a transactional process and checks whether if all expected operations are executed within one transaction id or 2 transaction ids are generated. If 2 or more transaction ids are generated than that means we don’t set transaction boundaries properly, then transactional code is analyzed and fixed.

4- Synchronizing Transaction and Persistent Object States: As transaction change its state like started, committed or rolled back, that state must be keep synchronized with persistent object states. One classic case is that a dirty object should be still dirty if a rollback occurs. In some cases, a deleted object from a list should be restored to the list again. Sometimes objects may also need to restore their changed fields to the prior values before transaction start. That kind of synchronization is important otherwise it may lead erroneous data.

5- Transactional Memory: One best practice when writing transactional code is to keep transaction boundary as short as possible. This is important for database performance and also for persistent objects. Persistent objects are associated with transactions so that their states are managed appropriately. Sometimes that leads to big memory consumption if many persistent objects enter same transaction (i.e. 500 invoice objects each having 500 invoice line objects, totally 250000 objects). To make transaction memory in object-side minimal we provide an API to flush and release transactional objects to decrease memory allocation.

5- Transaction Isolation and Locks: Higher isolation level means higher transactional safety but with performance loss disadvantage. We used lower levels and it boosted performance a lot. With the help of optimistic and pessimistic locks (not database locks, application locks), transactional inconsistencies are eliminated at the application level. We appended “read only” clauses (for DB2, it is “FOR READ ONLY”) to report queries to help avoiding transaction locks.

Reference:
Java Transaction Design Strategies

Transactions and EJB

Transaction Processing Concepts and Techniques

JTA Transactions

Customization and Extension Architecture for Java Web Applications

2009-11-13T10:10:00.000-08:00

It is a must that every ERP package has customization architecture. ERP packages are modified according to special requirements of many customers. So we needed customization architecture for our own ERP (composed of many Java EE web applications) too. I analyzed existing ERP packages(i.e. MS Dynamics Customization Architecture) to find out how we can develop such a system for Java EE web applications. There is no real clue from their architecture because of both their programming environment and architecture. I tried to analyze some extension mechanisms within some well-known Java applications like Eclipse. Eclipse plug-in system is great but it doesn’t suit Java web application extension requirements. I read many articles from Internet but there is no satisfactory answer for the problem.

After many years of thinking this issue, I found the answer. Answer was surprising because it was so close to me that I couldn’t see it; Inheritance and Dependency Injection(some may argue that this is Service Locator pattern but I preferred this term). Here in this post, I’ll present how to use inheritance (thanks to this excellent OOP feature of Java) and DI for application customization and extension requirements.

CUSTOMIZATION TYPES

Throughout this article I’ll use one term “Customization” to mean extension, plug-in or add-on etc. Let’s start with the categorization of customizations. In a typical application, there are 5 types of customizations (Cost increases from top to bottom):

1- By Data (Configuration with Parameters)
The cheapest solution for customization and can be done via system parameter files, or parameter tables.

2- By Workflow and Rule Engine
These systems provide semi-programmatic dynamic logic and can be changed without compilation in the current system on-the-fly. These solutions are cheap compared with programming customizations but it still requires some technical efforts.

3- By Addition
It means adding new applications to the system. To add a new application to existing Enterprise Application is the easiest programmatic customization since it is independent of current applications. In this method, programmer who makes addition may need to use your system libraries. When installing new versions of core applications, binary compatibility should be tested if any library of applications is used.

4- By Extension
Existing core applications functions are modified to meet new requirements without touching the base code. In this method, programmer would need to link base elements in his IDE (otherwise extension may not be compiled). We could support extension for Persistence Objects, Servlets, JSPs, Reports, JavaScript library, Framework and Business Logic. Every component type is extended with different methods, I’ll mention below. In new core application versions, both binary and functional compatibility should be tested.

5- By Modification
In this type, existing core application codes are changed. Programmer would probably need source code of applications (JSPs or JavaScript codes are already open source if not obfuscated). In new application versions, modifications would be lost. Programmer has to apply all modifications in every related new version. This is the most costly customization option.

EXTENSION OF PROGRAMMATIC ELEMENTS

In extension customization type, I said that every programming element can be extended. Here is the complete list of techniques we used for extension:

Persistence Objects
You need 2 things to achieve extension of persistence objects; firstly your persistence object can be extended. Yes, using built-in Java “extends” keyword is not enough so the problem is that does the extended persistence objects work in your Persistence Framework? Second requirement is that you should have a Factory class to return extended object in the run-time. You should build an extension lookup table to understand if this object is extended. If it is extended Factory creates extended object and returns it (DI). In this way, you can change functions of persistence objects by overriding its methods.


//in run-time DBSalesOrderExtension object is created
DBSalesOrder dbSalesOrder = BOFactory.get(DBSalesOrder.class);

public class DBSalesOrderExtension extends DBSalesOrder{
    ...
}

Servlets
Again you need 2 things to achieve this; firstly there should be no limitation to use extended Servlets in your web framework. Secondly you can forward a request to extended Servlet in “Front Controller” code. Since servlets are created by “Servlet Engines”, you can’t use Factory like persistence objects. Again, you should have a customization definition table to look up extended servlets. Your “Front Controller” checks this lookup table if current Servlet is extended. If so, it forwards, if not it continues processing. Client code or server configurations don’t need any adjustments.


//In Front Controller:
public final void service(HttpServletRequest request, HttpServletResponse response){
    ...
    String extendedSalesOrderServlet = getExtensionName(salesOrderServlet);
    if(extendedSalesOrderServlet != null){
        doForward(extendedSalesOrderServlet);
        return;
    }
    ...
}

public class SalesOrderServlet extends HttpServlet{
    ...
    public void doGet(HttpServletRequest request, HttpServletResponse response){...}
    public void doPost(HttpServletRequest request, HttpServletResponse response){...}
}

public class ExtendedSalesOrderServlet extends SalesOrderServlet{
    ...
    public void doGet(HttpServletRequest request, HttpServletResponse response){...}
    public void doPost(HttpServletRequest request, HttpServletResponse response){...}
}

JSPs
For JSP extension, you need only one thing; you should have JSP access method(s) to open/redirect JSP pages so that we can inject extended JSP name in these methods. In practice, JSPs can’t be extended with Java’s inheritance mechanism. You can only extend it by copying an existing JSP and modifying its content. Again, you should define its base and extended name within customization lookup table. In JSP access method, you check if target (base) JSP is extended. If extended you redirect to the customized JSP page.


//Original Page
SalesOrder.jsp

//Extended Page
ExtendedSalesOrder.jsp

//In this method, "ExtendedSalesOrder.jsp" is displayed
openPage("SalesOrder.jsp")

Reports
Like JSPs, reports can be extended by copying and modifying it (This is a general approach (I remember same approach within Oracle ADF), if a software element includes GUI elements, inheritance becomes useless for extension). We have company association of reports so we didn’t need a mechanism for extended reports. It is just defined as this company uses this report and company-specific report is executed when requested.

JavaScript Library
We created an empty extension file (js_extension.js) and include it in every JSP page so that JavaScript methods can me modified. In our JS library, everything is method. In JS, there is no direct support for inheritance and overriding. Extender programmer can remove a method in base JavaScript file and add its changed version to this extension file (Overriding or overloading is not possible in JavaScript) or add a wrapper function with different name and use it in his web applications. You can develop a better extension than mentioned here by simulating inheritance in JavaScript.

Framework
Some services can be extended by making new implementations. Here, DI is leveraged again. In our frameworks, we make extension possible in some functions. Implementation is defined within framework configuration file and that implementation is loaded during framework initialization.


public class MyFrameworkSystem{
    private IFrameworkService frameworkService;
    ...
    public static void initialize(){
        frameworkService = loadImplementation(getImplementationClassNameFromConfFile("FrameworkService"));
    ...
}
}

public interface IFrameworkService{
    public void extensibleMethod();
}

Business Logic
If you need only a fragment of code change or a control is used in many places in base code which are hard to locate by extenders, what do you do? By default, you can extend that object and override it. If that method is a big method, you have to copy all lines to new method and change a piece of it. Maintenance would be a nightmare in that case. A neater solution is to develop an extension injection mechanism. We developed “Extension Points” for just that requirement. If some business logic in base code is going to be modified, we can take it to an extension class to easily override it. In that way, extension can be plugged to any piece of business logic. How can we know which part is customized can’t be answered at the moment. That would be shaped by projects and core customization requirements. Again “Extension Points” should be defined in your customization lookup table. (One interesting case is that you may want to change business logic any time without compilation. Here I mentioned about Rule Engines, you can define a rule and call it in your program. In that way, you can change that rule any time you want; “Dynamic Business Logic”)


void salesOrderBusinessLogic(){
    ...
    SalesOrderBusinessLogic soBL = (SalesOrderBusinessLogic) getExtensionPoint(SalesOrderBusinessLogic.class);
    if(soBL.businessLogic())
    ...
}

public class SalesOrderBusinessLogic implements IExtensionPoint{
    public boolean businessLogic(){
        //some business logic here
    }
}

public class ExtendedSalesOrderBusinessLogic extends SalesOrderBusinessLogic {
    public boolean businessLogic(){
        //some modified business logic here
    }
}

USER CUSTOMIZATION

When talking about application customization, generally user customization is understood. In this sense, you can develop customization options for your applications. You can provide GUI-level user customization facilities. Let’s briefly see what kind of user customizations can be developed:

List Customization
HTML list customization is interestingly difficult (To see that try to change column orders). You may enable your users to customize lists; show/hide columns, column orders, column widths etc.

Input Form Customization
Some user fields may be added to forms by your users. Let’s say your user needs an extra column to track something, if you provide extra column feature (i.e. predefined user columns) they can easily add new fields. Or users may need some default values. For example you can enable them to get last entered data to a form input element. When they re-enter a form and press a shortcut key, their last entered data may be restored to that field again without requiring filling that data.

Report Customization
Your users may need to hide/show some columns etc. Or they may need to add new criteria to report parameter input page. Or they may want to change sort or group columns.

Competing Requirements: Functional versus Non-functional

2009-11-06T04:52:00.000-08:00

It is a common situation that can be encountered in many software projects that functional and non-functional requirements intersect. It is generally difficult to make correct decision among these 2 types of requirements. Moreover, some non-functional requirements can also have interrelationships.

Functional requirements are the behavior of software that a user wants to use. Non-functional requirements are invisible attributes that indirectly affects users. These attributes may be sometimes related with developer-side or user-side or system-side. Let’s remember non-functional requirements; Availability, Maintainability, Efficiency, Portability, Flexibility, Reusability, Integrity, Testability, Interoperability, Reliability, Robustness and Usability.

Even when making best selection we have to pay some trade-off cost. In our software requirements review meetings, it is a very usual such a case that an analyst wants a functional feature and programmer rejects for the benefit of non-functional such as performance considerations. What do you do in such conflicts? Performance or a user requirement. Rejecting user requirement is a very easy action but it decrease user satisfaction. We usually start a field analysis, so programmer inspects that feature in his program if a better way would be found. In second meeting, results are evaluated and final decision is taken.

Requirements gathering, elicitation and validation are very precious in software development. Everything begins with requirements. If requirements are not engineered enough, the project is deemed to failure. In this aspect, requirements analysts should be educated and well-informed about both functional and non-functional requirements. In this way, they can filter and design requirement solutions better with non-functional parts in mind. That would only filter some requirements but the final authority for non-functional aspect is programmers and architects.

Sometimes, this competition may lead to personal conflicts. For this kind of situations, we developed a rating system that analysts’ leader has 2-point vote, analyst has 1 point-vote and programmers’ leader has 2-point vote, programmer has 1-point vote and referee (manager) has 3-point vote. We just rate the feature and make the final decision.

To prevent conflicts, another important helper is to declare development priorities. What values do you give prominence as a software team? What are your strategies? Do you plan long-term software? If so, than maintainability would be important then.

You may prepare a “Requirement Documentation“ document that lists best practices of writing requirements for your team. This document may state priorities and make a common agreement for your team.

Links:
Software Requirements

More About Software Requirements: Thorny Issues and Practical Advice

Mastering the Requirements Process

From API to Solution: My One Year of Technical Blogging Experience

2009-10-29T06:52:00.000-07:00

I’ve been blogging for one year. Before starting to blog, I had read many blog experiences to understand the implications of writing a blog. Now, also writing this entry, I found many one year anniversary posts in blogsphere. Everybody lives different experiences about their blogging, me too. At the beginning, I was wondering if I would find enough motivation for writing more than a few posts. Surprisingly, I never missed a week to post during last year. I don’t know whether I’ll keep this sustainability this year.

Naming Blog: “From API to Solution”, what does it mean?
Your first work before blogging would be to find a title for your blog. Although I use different title “Java, ERP, Software Architecture” for my blog, my blog URL is real name of my log and it depicts my professional life; from API to Solution. That means during my profession, I work from micro level (API) to the macro level (Solution). Everything begins with designing an API. Then those APIs composes applications, then applications results products. Products are used via implementation projects. Those projects lead to systems. Systems turn to solutions. I worked in all areas, that name summarize my professional life.

To Blog or Not
If you are a techie like me, what I recommend is if you feel you know something valuable you may blog. Otherwise, don’t write a blog for just to have a blog. When writing something I felt the responsibility of not misleading people when they search something on search engines. If people are losing their time even by only clicking to your buzz blog, you are responsible for the distraction.

Another distraction is to have a joint personal blog with technical one. Sometimes to write about an off-topic becomes very charming, but this would be a waste of time for your readers. You may add some feeling, stories etc. to your posts, but I think this should be limited.

What to Write
Content is the key point that the other people will read your blog. You should share something you know, something you thoroughly analyzed, and something you used. In this sense, my only suggestion is to write only the topics you are sure. Some assumptions, some feelings about technical issues would be easily rejected by your blog commenters. If you can’t defense your ideas, than you feel bad and blogging loses its meaning. Your resistance to commenters would decrease and that would be end of your blog.

How to Write
Before writing a post, my behavioral pattern is like following. Firstly I find a topic and search topic on the net, on the blog posts. If somebody already wrote a post or article about it, I quit that topic and find another topic. If there is no equivalent content on the net, I read what others said about this topic. After reading other articles, I merge with my ideas and experiences and begin to write. I use MS Word for first edit. After writing the post, I read it again for correcting spelling errors and sentence structure errors. Than, I publish it. After publishing, I may change some sentences and add some new paragraphs.

How to Advertise
There are some blog post aggregator sites. You may register there to widen your audience. You may also contribute your posts to some technical portals like java.dzone (Javalobby).

Other Experiences

I don’t like adding ads to the blog: You agree or not, when I open a blog that is full of advertisements, I feel that blog writers’ only aim is to have click stream. Because of that, I have a suspicion about what is written. I feel that ads also make blogs dirty. What I like as a reader is clean and well-organized blogs.

Prefer Suggestion over Criticism: We have some strong beliefs about our technical values. Even tough we state a small criticism against an issue; we may hurt its believers. We’d better use a positive expression way. I got this point after I wrote a criticism post. Needless to say, you must avoid humiliation and curse.

Don’t Repeat Content: I see some blogs that are like news aggregators. If you want to be a web journalist, good luck, but I don’t recommend that. If you want to comment some news, that’s OK but IMHO re-publishing some content again doesn’t add value to readers. I see many blog entries on the net saying same thing, same technical issue, and same observation. Writers should search it before writing something.

Learning by Blog Writing: People say that best way to learn something is to teach it. Same applies to blogging. By writing you also realize what you know again. You check your knowledge by writing, by reading comments, by answering them. First of all, you can learn how to write, how to organize your knowledge to express.

Blogging Requires Time and Motivation: Blogging requires some time dedication. If you don’t have enough motivation, you wouldn’t allocate required time. Many corporations have technical blogs and your employees may tolerate your blog writing during work hours. Motivation may also come from your technical passions.

Blogging Reveals You: When reading technical articles you feel you have a different perspective and that may motivate you to write. By blogging you assert your technical perspective for everyone; your current and future employer, your current and future staff, your customers etc. For this reason, you should write your common denominator ideas.

Blogging in English: Blogging in English provides you to reach a wider audience. Sometimes, people argued me about not writing in my native language. If I’d been writing so, it would be restrained. It is very exciting to share something with people from very distant countries. I study to make my English writing better and beg your pardon for some grammatical and vocabulary mistakes in this blog.

Software Upgrade Process

2009-10-24T12:51:00.000-07:00

Today I’m going to try list the common patterns of upgrading DBMSs, OSs, and ERPs from my experiences during our whole 8-year project duration. Upgrade is a major task of system administrators or system support stuff. During our project life-cycle, our project team continuously contributed system management of many systems. Upgrade is an installation process to make working system better. Yes, it gets better but you have to pay its cost by sacrificing your team weekends and after-hours.

Upgrade process is an element of SCM (Software Configuration Management) and System Management. There are 2 kinds of upgrade;

Version Upgrade: You just install a newer version of an application. Software is a living being and if it didn’t end its lifecycle, it will be developed and fixed periodically.

Migration: You replace an application with a similar product from another vendor. This is a harder upgrade compared with version upgrade.

Upgrade Process Steps

1- Planning: We make requirement analysis in this phase. We answer which versions are needed, version prerequisites and dependency analysis. We should also plan upgrade execution like system down-time period and user’s appropriate times.

2- Preparation: We prepare test environment. We obtain required upgrade packages and its dependencies. We check again hardware and software prerequisites to make sure everything is alright. We should also prepare recovery plan for upgrade failures.

3- Test: We install upgrade version to the copy of production system. We note installation steps, problems and their fixes. We prepare an “Installation Notes” document in addition to vendors’ release notes.

4- Upgrade: This is the execution phase. If upgrade is offline, we close user connections and start upgrade. Before upgrade, we should backup required components for recovery. If everything goes smoothly, upgrade is tested for confirmation. If upgrade fails, then it may be re-attempted or may be rolled back. When rolling back upgrade, our recovery plan is executed.

5- After-Upgrade: We open the system and users begin to use. We should accompany users during their first usages. It is probable that some problems would arise. After we are sure that system is stable and operates correctly, that means upgrade is finished.

Upgrade Best Practices

Upgrade is painful: I placed this on top of the list. The folks who upgrade the systems know this very well. Touching a running mission-critical system is just like working in a power station.

Upgrade should be tested: If you don’t test upgrade, this is identical with Russian roulette. Everything may be bogged down after upgrade. If system availability is very important, that interruption costs a lot.

Upgrade should be phased: If upgrade will be applied to many systems, than a phased approach is very paying. You can begin with a pilot system and then continue with the rest by gaining installation knowledge. That makes upgrade process smoother.

Recovery Plan should be tested: If upgrade fails, you face with your recovery plan. Whether it is an OS backup, a DB backup or a folder backup, or an application backup, restoring existing backup is critical. Recovery process should be as simple as possible.

Upgrade should be managed: Some upgrades may require managerial decisions like to continue or not when a problem is met.

Upgraded software requires more hardware resources: This is true in most of the situations. Even if it is said that software works better in new version, it always would require more hardware to reach that goal.

Upgrade infrequently: Upgrade process requires a very valuable time even though it seems a little installation. If there is no mandatory requirement, upgrade should be given a mid-level priority.

Upgrade to one version below highest version: Safest version is always one version level below the highest version of that application. Although software vendor make many tests to make the release robust, it is generally in beta stages before fix-packs. Fix-packs require time (installations and usage) to emerge. If you don’t like extreme sports, you should not be on the edge.

Some Upgrade Memories

During our ERP project, I’ve got two unforgettable upgrade memories. One was the upgrade of DB2 installed on AIX server. It was a very critical requirement for the project to apply some fix-packs. After working for hours on Saturday, time was running out. At Saturday night, it was 2:30 AM that I succeeded to execute DB2 instance. That was a perfect moment I can’t describe it. If we can’t succeed then system admin was expected to restore OS from backup in Sunday morning.

My other amazing upgrade memory is an ERP upgrade work. We had finished branch systems’ upgrade during last month and our system support team who made all upgrades were very tired (worked for many weekends). Last upgrade was the central system upgrade. For that upgrade, we worked during weekend and it couldn’t complete it. We then convinced users to continue on Monday. However, we couldn’t finish it during the daytime. After talking with upgrade team, I supposed they will finish it before the night. But it was a wrong assumption; they called me at night for resolutions of some problems. I hurriedly went to company and support them to eliminate the problems. By means of unbelievable devotion and tenacity of that team, we had completed upgrade before Tuesday morning. It was a great moment when system was up in the morning.

Data Archiving of Enterprise Applications

2009-10-18T04:33:00.000-07:00

One of the elements of Data Lifecycle Management (DLM) or Information Lifecycle Management (ILM) is Data Archiving. In my previous post, I’ve mentioned the coverage of DLM. Archiving is the last phase for data before removal. Yes, you build your enterprise system and production database get larger and larger day by day. Dependently to volume of transactions, your production database gets bigger and performance degrades gradually. Tuning application database is not a cure for overloaded tables. In our ERP applications, we observed that a mid-scale company may need archiving after 2 or 3 years later after system goes live.

Archive requirements depend following parameters:
Volume of Transactions: High volumes of transactions (new rows) in an application signal a coming archival need. Some applications may generate little data. Data accumulation is dependent to business process and your applications table design (Normalized tables means less data). Adding 1 to 10 of daily rows would not lead to archival need up to 5 years. However, 10 to 1000 or millions of daily new rows (transactions) would eventually causes tables too loaded. In our ERP archiving process, we didn’t touch some module tables since transaction ratio is considerably low and we didn’t observe performance degradation. One of our ERP systems’ (a big factory) DB size increased from 10 GB to 20 GB in only 1 year period. For this reason, we had to archive per year.

DB Server Configuration: If your db server is a high-range server having many processors and fast disks then you might archive later. Some of our powerful ERP db systems didn’t require archiving for up to 4 years.

When to Archive

Every enterprise application would need an archival today or some time in the future. You need to get ready for archive process before you face it unexpectedly. Because archive process requires some planning and preparation, you should consider it on the first day of your system usage. Symptom of archive requirement is clear; users complain about slowness of the system. After inspecting the system you find no evident reason for the problem except data size.

How to Archive

There are 2 ways when an archive need emerges:
Build Your Own Archive Solution: If you developed the enterprise applications, your only way to follow is to develop a custom solution. One helper at this stage might be the emerging features of DBMSs for archival. Developing an archival system would not be easy because of referential integrity of your tables. You would probably face with circular dependencies of tables problem.

Find an Archive Product: Some big ERP packages have archival modules or you may find some third party solutions. If you can’t find an archive solution, you may outsource that task to the consultants companies of your DBMS.

Building an Archive System

“Data Archiving” can be achieved with different methods. When searching archiving methods, we see that some people use “Database Partitioning” for archiving. In our solution, we didn’t test and prefer that way. Database changes in new versions of applications may impose maintenance overhead when selected this way.

Some solutions also archive records incrementally and partially whereas we selected yearly archives for whole tables at the same time. We just copy current database (with copying applications as archive applications) as archive and cleanse archive data from the production database. There are 2 relations when archiving: From archive data to live data and from live data to historical. By just copying current database as archive (loaded some live data), we didn’t struggle to nullify forward references from archive data. Thus, we only solved one way references as explained following.

There are many requirements for an archival solution:

1- Extract Unused Data from Production Database to Archive Database: Needless to say, an archive program can extract unused data from production database to the archive database. It may also support opposite; loading data from archive database but we didn’t need this feature in our archive operations.

2- Properly Nullifying Relational Data: Archive system should provide archival information of relational data. One of the hardest problems of our archive module design was this problem. What to do to relational data? Just setting null to foreign columns that refers to archive records was not a good solution because user won’t understand it and some applications may produce errors for null column data. This is also impossible for “Not Null” columns. We solved this problem by keeping one level of related archival data and signing it as archive data (When user enters archived data he is warned that this data is archived). That archived data’s columns referring to archive data is nullified.

3- No New Version Maintenance Overhead: In our solution, archive data is archived with the applications. This was a controversial issue. Let’s define the problem; do you plan to run your archive data with new versions of your applications? Our answer was no. If saying yes, we had to reflect every database structure change of new application versions to the archive databases. By just archiving database with applications eliminate this overhead. We didn’t get any critical error reported form archive system that would prove new version need. If that could happen, we had to install a fix version to archive system. You don’t touch both your archive database and archive applications and that eases system maintenance. We also make archive database read-only so that we could remove archive database and load it from secondary storage (DVD) when written during first archive operation without thinking whether it is changed.

4- Archiving In Year Units: As I mentioned above archiving with applications, you would not want to archive frequently. On the other hand, when users want to take annual reports they would want to reach whole data. Business reports are generally taken within year ranges. That results archiving should be done in year units. For example, you may archive in 2 years period. Less archive process means less archive systems to manage. We generally archived last years records in the mid of current year. Before sending data to archive, it is important to analyze its liveliness. In an ERP system, last years data may be changed till the mid of current year. An archived data can’t be changed, for this reason we have to ensure the borders of historical data. Some applications also use “Closed” column flag to ensure it is candidate for archival but users were reluctant to update that status fields. Best way is to wait middle of the year.

5- Online or Offline Archiving: Archiving can be executed in 2 modes; online or offline. Online archiving is to archive when production system is open and in use. Some systems can’t be closed and might require being available 24 hours and 7 days. Online archiving would take more time compared with offline archiving. Online archiving would probably run with SQL statements. Offline archiving could use DBMS export and import utilities and that would take less time to complete. We currently preferred offline archiving and it meets our current requirements. Extracting 1 year data with SQL may take a lot of time and system performance may be affected with this background process.

6- Archive System Storage: OK, we finally finished archiving. We finally got a lighter and faster production database. What to do the archive database and applications? Now, let’s write it to a DVD and take it to the strongbox. I assure that it would not last long that your users call you and want to reach archive data. After archiving, we should prepare an archive system for users to reach it. Pervious year archive system is much more frequently visited compared with past years. Sometimes you may have to remove past years archive systems because of not enough storage on disks, but you can restore archive system when it is requested. Your application infrastructure (DBMS and AS) software may need to be retained for installing archive system again.

Links:
Database Partitioning vs. Database Archiving

DB2 Data Archive Expert

Oracle Total Recall/Flashback Data Archive

Archiving Your SAP Data

Information Life Cycle Management (Data Archiving) in Business Intelligence

Oracle Data Archiving

Application Clustering For Scalability and High Availability

2009-10-11T07:55:00.000-07:00

Application clustering is a sub-topic of “Parallel Computing”. Today many types of software support parallel computing in some forms. Why do we need parallel, clustered applications? For 2 reasons; scalability and high availability. Does every application need clustering? No. However, if your system is underperforming and not available enough to meet customer requirements, then that signs that you need clustering. There are many kind of clustering at each software tier, please note that application clustering is different. A stand-alone application may not run smoothly in some types of clustered environment and needs some tweaks that I’ll share here in this article.

Which benefits can we gain from clustering? There are 2 main benefits:
1. Load Balancing (Scalability): By distributing processing, we can make it possible vertical or horizontal scalability. In this way, we can leverage resources more effectively. One example is Java GC advantage. If we distribute user sessions to different JVMs then GC would run less and performance would be better. Also we can overcome JVM heap size limit.

2. Fail-over (High Availability): Fail-over means less and shorter service interruption. Some SLAs may require some high availability percentages like two-nines availability. A server may be down because of many reasons; system failures, planned outage, hardware or network problems etc. To achieve HA, one important technique is clustering. Fail-over may occur automatically, or manually. We may prepare a redundant backup system (Cold Failover) or our load balanced servers may also serve the fail-over function (Hot Failover). Please note that there is an engineering branch for this topic; Reliability Engineering. Moreover, fail-over is not only an issue solved by clustering but it is also related with your system design and characteristics. If you design your applications against interruptions then it is called “Fault-Tolerant”. When dealing with this issue, we shouldn’t forget the recovery time (Disaster Recovery). You may fail but it is important how fast you can make system up. Downtime is costly and its cost should be calculated to get better understanding of HA.

There are many kinds of clustering in different tiers. In every tier, it may be named differently meaning the same thing like virtualization, partitioning, mirroring. Let me try to list clustering tiers:

1. Hardware Clustering: RAID is such technology used for both performance and reliable disks.

2. OS (Server) Virtualization: We know many OS-level virtualization programs that can run many logical servers within same physical server at the same time.

3. DB Clustering: Many DBMS supports clustered database instances via different topologies. We can add JDBC-level cluster libraries to this category that simulate clustered database feature.

4. JVM Clustering: Some vendors transparently cluster JVM without knowledge of the running applications. Some of them use byte-code instrumentation for this.

5. AS Clustering: Application servers (and HTTP servers) support clustering with varying capabilities. Some of them can make session state replication across AS instances. Some of them have load-balancers to distribute coming requests. Some of them can have transparent fail-over feature etc.

6. Application Clustering: This category is what I want to explore today. Application clustering is parallel with application server clustering but not the same thing. If you don’t make your application cluster-enabled, it won’t make use of AS clustering. If you configure your servers for clustering, you may still need to change your application code to run on these clustered servers.

Application Clustering

There are some adjustments when preparing a stand-alone application to run in a clustered environment. Briefly, some of your application services need to be reorganized for state integrity. These services are probably using Singleton pattern. If you are building your application with some frameworks or libraries, they also must support clustering in a consistent way. Otherwise, you won’t make us of clustering by yourself.

To keep state integrity, you may choose one of following architectures (Some ASs may already support this infrastructure):

State Replication: You may have multiple service instances on cluster nodes and synchronize the state by replicating it. There are many replication techniques you can use; port communication, RMI, JavaSpaces. This architecture has advantage in fail-over and disadvantage in performance.

Hub-and-Spoke (Master-Slave): In this architecture, you would have only one instance of services on master cluster node and child nodes will invoke master node services through for instance RMI method calls. This architecture has advantage in performance but disadvantage in fail-over capability (if master server fails, child nodes would lose service states).

Let’s delve into some services that must support clustering:

1. Locking: Application locks should be shared across different cluster nodes. If DB locks are used then it is cluster-safe. If a singleton lock service is doing locking, then we need to make lock acquire-release calls among cluster nodes synchronously. For example; a user is running a process on AS cluster node1 and hold a pessimistic lock on Entity A, then another user’s process running on cluster node2 must receive an exception when trying to update Entity A.

2. Caches: Every cache service should be analyzed if a cluster-enablement is required. Some caches may repeat on cluster nodes but that may not cause any problem. On the other hand, let’s say we have a persistent object cache service that runs according to update information to tables and checks if any table modified. A process on another cluster may update a table and that change information should be reflected to all caches on the other cluster nodes.

3. Events: You may have designed some type of application events and those may need to be cluster-aware. For example, you have a notification event that triggers when a user sends a message to another user. If that event occurs in node01 and other user is on node02, he would never receive that event and message if that event is not cluster-enabled.

4. Identities: Let’s say we have a sequence generator service that has a cache to prevent accessing database every time a number is requested. If identity service is not cluster-enabled, identity services on different cluster nodes would generate same identity numbers and applications would get duplicate record errors.

5. Authentication: If user can login on each cluster node then some authentication information need to be shared. One example is a feature such that a user account is locked after multiple failed login tries. If failed attempts are not shared, a hacker may continue his attempts on other nodes. I don’t add authorization here since we didn’t need an adjustment for it to work in clustered environment (Your authorization service may need it).

6. Scheduling: Scheduled or batched tasks should be considered when clustering. Same scheduled processes may try to run on each instance whereas only one execution is enough.

7. Logging (File IO): You have to deal with file operations in a clustered environment. One file issue is log files. Multiple cluster nodes may try to write the same log file and make it corrupt or cause IO errors if not cluster-enabled. Our solution for this is to generate and use different log files for each cluster node.

8. Messaging (Network IO): If you are using some messaging services, you have to make sure that message service is cluster-enabled. If you are using a JMS product, you need to test in a clustered environment whether it behaves appropriately. If you wrote it and you plan to use multiple messaging service instances, then you have to use different ports if cluster nodes are on the same server. We just used only one instance of messaging service to overcome that kind of problems.

9. A Service That Need to be Clustered: Since there are many application services beyond this list, you may have a special application service that need to be cluster-aware. You should provide an API to execute in your cluster architecture compatibly. We did so. We have a very easy API to register services to run on cluster nodes seamlessly without thinking state sharing. Thus, any programmer can utilize that service without dealing with clustering issues and details.

It is very important to evaluate if a service really needs cluster-enablement. For example, ordinary persistent objects should not work in distributed and clustered architecture since it would be a very big overhead to run across network (Add extra code addition to this cost). Sometimes, by just cluster-enabling some services is good enough to utilize parallel computing.

A sample cluster topology:

Links:
http://www.theserverside.com/tt/articles/content/J2EEClustering/article.html
http://www.javaworld.com/jw-02-2001/jw-0223-extremescale.html
http://www.javaworld.com/jw-08-2001/jw-0803-extremescale2.html
http://www.redbooks.ibm.com/abstracts/sg246153.html

New Web-based ERP On Java EE

2009-10-04T08:11:00.000-07:00

It is nearly impossible to see new players in ERP market especially international ones. Big vendors are very powerful and ERP is not such an application that anybody could write when wishing so. Software vendors who want to enter this field generally buy some small players in the town. ERP is an aged product, it requires time to mature. It requires at least 10 years getting the applications ready for mid-market. For that reason ERPs are usually based on some old-technologies and always on the track to catch the new technologies. We began to write our ERP in 2001 and for that reason we could have contemporary technologies inside the ERP which is Java EE.

Our ERP project began as an in-house project to meet the requirements of a big group of companies. We finished almost 30 ERP implementations. The total time we spent during that time is approximately 8 years. It took 3 years to get first release of the ERP which is a very daunting project. Making first release doesn’t guarantee to sustain ERP package growth. We designed the applications by analyzing many ERP packages to make the right choices. We struggled to make it architecture “Made to Grow” since every ERP applications get bigger by the time with new projects.

We are currently trying to package this in-house ERP to serve outside companies. We are expecting to release that packaged version on the early months of 2010. We believe we have some unique aspects.

Current ERPs Disadvantages

1- Cost: ERPs are really very costly projects compared with other IT projects. It is not a sole software project but it is a hardware and business improvement project as well. Its selection and its consequences are another article topic.

2- Complexity: ERPs get bigger and bigger. Every piece of business logics is programmed and to manage such a big application source base is a very big challenge. Big scaled ERP applications are an extra cost for mid-size companies with no real benefit. This is something that you buy a fully-fledged gadget and you use only one function of this gadget paying for all functions. You not only pay for unused functions, you also pay for the complexity of managing and adapting this huge code base to your company.

3- Old Core Technology: Many ERPs are rooted to back 20 years. Although many ERP vendors tried to renovate their core technology, they still lack many new technologies. Web technologies are the primary problem of ERP vendors since migrating to these technologies are still a risky operation since there is still some function gaps compared with client-server architecture. To innovate in this function gaps are still very hard for most of them. For that reason, you won’t see an existence for old players in the field of SAAS market.

Our ERP Features

1- Contains Standard ERP Modules: It contains 4 suites; Common, Finance, Production and Human Resources which are totally 80 modules. We have some beta suites; Import/Export, Insurance, Free Trade Management, Catering.

2- Based on Java EE Platform: Yes, finally an ERP is written in Java EE from bottom up. As I outlined some of the features in the previous blog post, it is not as much proprietary as the others. We depend on Java EE on infrastructure side, and Java EE advantages are our applications advantages.

3- 100% Web-Based: Many ERPs claiming to be web-based are just some thin-client applications instead of core web technologies. In my perspective, web is going to be the most powerful and prevalent application model in today and would be in coming years. Web architecture advantages are our applications advantages. To convert a standard client-server application to web-based architecture is not an ordinary application migration. As I said before, it is much more difficult than to change a programming language of ERP.

4- It is Built with Best Features of Current ERPs: Oracle Fusion Application target is to mix bets of features in ERP packages, we already did it. During application design, we analyzed a lot of ERPs and tried to build best of breed modules.

5- ALM Platform for Easy Extensibility and Customization: We have a complete ALM platform so application management is very easy. Our development platform is not only a development platform, it is like an application operating system.

We are Seeking Investor Partners

We are seeking partners to invest in our Java EE platform and ERP. If you are an IT company and want to enter this market, it is a best practice to go with an existing package. Developing an ERP is a challenging job which is beyond from the technical expertise. Many ERP development projects failed because of not realizing that reality.

I remember a colleague who asked me for advice about their ERP project. I said not to see ERP development as a software project. His answer was I’m technically profound to develop any software package. What happened? They failed. Why? They had exhausted their time with infrastructure and development process issues and application modules were late to use (I guess it stemmed from the lack of knowledge to transfer business logic to applications) and their project cancelled. That kind of stories generally finish with a new ERP project bought from big vendors (To reduce a new risk and failure, project sponsors are now more generous to buy from a big vendor).

Java EE Simplification Project For Simplicity, Productivity, Agility and Cost Reduction

2009-09-27T11:40:00.001-07:00

We’ve been working on a project that will yield a product to simplify Java EE web development. Java EE 5’s leading principle was to simplify Java EE development. Simplifying EJB usage and adding some language features were an important step but there is more to do. Most of the difficulties for Java EE is about too many specifications and complexity of the technologies. To develop a software application with only pure Java EE technologies is today a costly option in terms of complexity. In fact complexity source is not only Java EE. Add web technologies to that list. A standard web developer should have many facets in addition to Java EE knowledge fighting on many fronts; JavaScript, CSS, SQL, XML, AppServers, HTTP Servers, DBMSs, IDEs, Tools etc. Most of the software teams embraced open source libraries to cure this complexity problem. We developed an ERP application with our development platform and we are getting ready to release this full-scale enterprise platform to ease enterprise web development. In this post, I am going to give you details about our ongoing project and our motivations to prepare such a platform.

This is a very fresh start. This is a solution in Java EE space but not built with only standard Java EE technologies. Our approach is pragmatic and what Java technologies we used are just according to common sense. We are trying to find a middle path in Java EE space. I think Java EE simplification will not be with a new language that is popular recently. Java simplification would be possible with API simplification and powerful tools. Instead of a new language, we aim to provide simplicity with a full-scale API set ranging from utilities to persistence with an integrated approach (Integrated Development Frameworks).

Simplicity Motivations

1- Current Java EE API Depth & Vast Web Technologies: There are numerous low-level APIs in Java language and that is a real drawback for entry-level programmers. Teams are built with senior and junior programmers. A development platform should be learned and used with senior programmers with high knowledge level. But in Java space, even many senior programmers still can not cover the details of standard technologies. A classic example for web development is lack of JavaScript knowledge. For that reason, web project teams are not agile enough to provide fast solutions that lead to cost reduction. Today, everybody knows the meaning of the cost in current economic climate.

2- Open Source Integration and Maturity Problems: Companies that are providing Java solutions and complaining about Java complexity problem usually embraced lightweight solutions which exist only in open source world (Java vendors couldn’t support these solutions because they are selling Java EE certified solutions and this is going to be conflicted with their current policy). Open source solutions come with their own problems (I mentioned in an earlier post). As they get attention from developers their lightweight solutions improved and turned into another heavy giant software stack. Library dependencies are a new nightmare to be solved in open source solutions which is one of the main integration problems. What is needed is simple, a new fresh start that adheres some Java web standards and don’t have zillions of dependencies.

3- We don’t Need a New Language for Simplicity: We don’t need a new language for simplicity, why? Current API granularity is good enough for an enterprise level programming. I analyzed many ERP domain language (i.e. ABAP, X++) but they generally lacks low-level access that what they want to hide. They are proprietary and require a new learning curve. I think Java language is good enough if a successful API simplification is done. Many developers are pondering over learning a new scripting dynamic language to simplify development but I believe that is not going to simplify development as wanted. Those languages may be used only in small web projects. As scale increase, their scripting capabilities would not meet requirements. One of the API approach advantages is the ease of switching to Java normal APIs if needed. For example, our file system APIs may not suit a special need and at that time, it is very easy to use Java’s own API set.

4- Agile Development: Today, we need fast solutions. Speed is only possible if we craft the solution easily. Development time reduction is not enough. Speed begins with requirement analysis and lies to post-production. We have to focus on total-time reduction covering all phases. Some platforms may provide to build software speedily but maintenance may be very hard or may require more time thus diminishing total gain. Total agility should be targeted.

Simplification Elements

We are trying to achieve Java EE simplification by following elements:
1- Simplified High-Level APIs with Frameworks: Our platform contains base development frameworks; application, reporting and messaging frameworks. I’ve already mentioned the details of our frameworks in my previous posts by explaining their architecture. We are targeting “Enterprise Web Development”. We are not writing an application server or IDE or database. We’ve developed application service APIs by eliminating details of many enterprise level problems. What is a simplified development API? We aimed to be used by average programmer without any problem or fuss. In our early ERP development phase we had former-IT members those were not software engineers and they today become Java developers. We always considered their COBOL-based background and wanted to make them Java developers with a very minimal Java education. A very interesting story is that a non-programmer IT member began to write applications within our simplified enterprise development environment. On the other side, software engineers complain about short learning curve.

We make simple development services for following areas:
Persistence Framework: We developed a very flexible and simple persistence framework to use. It is used as a transaction engine for our current ERP. We wrote ORM library even before many open source emerged. We tried to achieve the items mentioned in my criticism of persistence framework post.

Web Framework: We developed a web framework to develop web applications easily. It contains the elements of a standard web framework with integration advantage with persistence framework. It contains the features which I mentioned in better Java web framework post.

Messaging Framework: We wrote a simplified API set to make messaging-oriented programming easy. I mentioned about it in an early post.

Reporting Framework: We wrote a simplified API to develop reports in Java. It has many features that I also mentioned about it in an early post.

Followings are some services provided by our Application frameworks:
Connection Cache
Logging
Database Change Logging
Data Caching
Session Tracking
Locking
Profile Management
Scheduling
Batched Tasks
Resource Management
Replication
Distributed Query
Portal APIs
Identity Generation
Simplified Email API
Simplified File System API
Adapter API
Authorization Service
Authentication Service
Encryption Service
Clustering
Simplified Localization
Customization and Extension APIs
Rule Engine
Workflow Engine
Metadata Management
Mobile Support
JavaScript Library for Rich Web Applications (Many AJAX-enabled web components I mentioned in previous posts)

2- Development Tools: We developed some MDA tools that facilitate code generation. To develop a CRUD application is just in minutes, believe me. These tools are not dependent on any IDE and works as a web application. Generated codes are touchable codes and can be altered and not encryptic codes to comprehend. Simplification is possible with tools and even if your APIs are simple, this is not enough. Tools boost productivity a lot. Agile development and easy prototyping is possible with development tools.

3- ALM Tools: ALM is going to be popular in coming years as Oracle and other vendors invest more on this field. This is a natural step for software vendors because ALM means automation. Every software company needs an application set to manage their processes. These applications repeatedly developed in every software organization, remember; bug-feature tracking software, run-time management, user management, version management, license management etc. We developed many ALM modules easy to use and ready to use and integrated with development frameworks. These modules complete the simplification goal.

We are Seeking Investor Partners
Our project continues, we are the last phases to release our product (planning to release in early months of 2010). We are also working to package our in-house ERP to sell as a new ERP product. What we aim is to introduce 2 products; one is an application platform I detailed here, second is a web-based ERP product based on the application platform. This is a very unique strategy. Many ERP vendors tried to sell their ERP development platform but that was not possible without an accompanying ERP sales. They maybe didn’t have the vision to sell their platform at the beginning and always had a proprietary locked-in architecture.

We are open to hear from serious investors that are big enough to support our team and strategies. We may make some kind of partnerships if an agreement is established. Branding may be done through new partner since we are not an IT company yet. If we don’t find any partner, we are going to begin in domestic market alone.

Our projects resembles IBM’s former San Francisco project with one distinguishing difference, we’ve already developed a big ERP system using this platform and our platform already verified itself. Our platform is not dependent to the ERP system and can be independently used to develop any enterprise application. Let’s make it true that “Java is new COBOL” by powering enterprise Java projects.

Deciding Correct Problem Solution

2009-09-22T06:58:00.000-07:00

When designing a system, it is a usual situation that there are multiple solutions for same problem. Each solution may reside in different software layers. This is a very interesting pattern that can be experienced every time when facing a problem and designing its solution. Designing is really a special effort in engineering which is beyond the problem solving techniques. Designing a good solution begins with deciding the good solution choice for the problem.

At the first stages of our ERP development, I had always a mantra that I insist on whenever we meet multiple solutions for a problem which is “Single Touch Point”. As long as it is logical, keep everything in one place for “Single Touch Point” principle. It would keep programmer focus and accelerate productivity. For this slogan to become true, we should pull the programmatic artifacts to Java source code as much as possible.

Let me summarize some design decisions for that principle:

If possible, don’t use XML: If we can do our job with Java source code, why do we bother to make many software pieces that need to integrate on XML manipulation and conversion. One example for this is our rule engine definition files. Many rule engines generally used XML as a rule definition format but I feel that it should be in Java. Many rules are written by developers and why do we disturbing the programmer focus? One reason for that was dynamism that we can also achieve in Java language. There are many examples, no need to explain all of them.

Database View Construct Abandonment: Yes, we don’t use views instead we use View Java objects. View Java objects are placeholders of database views with validation and criteria support plus security. I read many articles about why views are good but I think drawing views into persistent layer has much more benefits to keep it in database layer.

Database Stored Procedure Abandonment: We also don’t use stored procedures. We have business events and programmers can write event handlers in Java instead of writing code in database non-portable proprietary language. Stored procedures are not a bad thing but we tried to keep code as much as in Java.

Server-side Validation: To make a single point of validation, we decided to use database objects the sole place for validation. For that reason we rarely used client validation. This is again a benefit for programmers for “Single Touch Point” that they don’t have to struggle with JavaScript.

There may be many solution alternatives and any software would follow a path among them. There is no single answer but selections determine the faith of the software product. For successful software, every selection would not be true but true selections should be much more than the rivals’ true selections.

Let me list some of occasions that requires delicate selections:

SQL versus Application Code: Yes we know when to write in Java and when to write in SQL but interestingly there are some cases that you can achieve same function both in Java and SQL. For example a join may be achieved with both Java and SQL, which one to use? This is an intersection point between solution contexts.

System versus Application: This is a classical case that every software team meets. You have a problem that can be solved both from your application and from system (server machine). For example your application has poor performance and you think that you made every possible performance tuning in your application. What do you do? Do you want the system administrator to buy more CPUs and memories or do you re-check your application for better performance? The decision would depend on parties involved and problem agility.

User or Application: Again, you have problem that can be solved by your application or by user’s action. Let’s say your application have a double-submit leak in an application page. What would you prefer whether to inform user that he mustn’t double-click or you solve the problem of your application? There are many examples even further the buggy situations. Second example is a user requirement for Excel-like functions in a data entry screen, would you develop that application or just want the user to use your ultra-fast input application?

Administrator or Application: Again you have a problem that needs some actions by administrator. Dou you want system administrator do the job or add it that feature in your applications? Let’s say your administrators complain about system management overhead of your applications. Would you develop a new application server embedded your application to ease system management (a desperate path) or explain them to get more training?

Spreadsheet (Excel) or Application: Let’s say that your users want a super formula system in your application? Would you add that function into your application or just lead them to use spreadsheet programs for that task?

JavaScript or Java: You have a problem that may be solved in both front-end and back-end. Which one would you prefer? For example, with AJAX technology you can load everything onto JavaScript. Let’s say you need a fast validation requirement, would you ignore the server-side validation for the sake of client-side validation while fearing the possible security vulnerabilities?
That list goes on, I’m sure there are much more to add to the list. Selections are much more important than it seems. Problems may be solved for today but we eventually face our decision results in the near future. For sound decisions to become a good “Solution Architect”, we need to learn, inspect, try and analyze and ponder over the problem more to reach best selections among multiple solutions. After a good solution selection, story continues with good solution design characteristics like extensibility, maintainability, adaptability, learnibility etc.

Even in programming, there are many alternatives to solve the same programmatic problem. Sound solutions always live longer and weak solutions may live till test phase and require a rewrite again. It is a fact that sound solution need more time and more experience.

Orchestrating Business Processes With Workflow System

2009-09-12T07:37:00.000-07:00

In my early posts, I mentioned both BPM and Rule Engine. In conjunction with these topics, today I’m going to talk about workflow systems. Workflow is an application technology that automates some functions thus eliminates to do recurring mundane tasks again and again. These repetitive tasks usually require user actions in absence of workflow system. Workflow is the streamlining (orchestrating) of business process steps according to some rules. Business steps are defined as workflow activities and workflow system runs these activities triggerred by business events.

Workflow systems may be in stand-alone or add-on form. In stand-alone workflow systems, workflow may cover the business process execution (Usually called BPM). In add-on type, workflow is triggered by external business events. Workflow system should be as loosely-coupled as possible and be flexible enough to add any type of new flow definitions. I know some workflow products are just built-in approval systems. Workflow is a technology beyond approval system. It may be only one function of workflow system.

Is workflow system a mandatory part of a business application? Simply no. We can add custom features that provide similar functions with workflow systems. However every time business process or people changes, we need to change whole application source code, this is a very fragile solution.
Let’s list some of the advantages of adding a workflow system to our applications whether you build or buy it.

Advantages of Using Workflow System

1- It enables us to define flexible business processes and change it any time we want. We can manage processes, execute them and monitor their status. In add-on workflow type, it adds a new business process layer on top of existing static application processes.

2- It facilitates “Push” technology. User tasks are brought in front of them by workflow messages. In this way, they don’t have to control new data periodically via navigating many application screens. Users don’t necessarily need to receive emails or phone calls form other users on the same business process flow. They can receive workflow messages instantly when processes occurs. Thus, business process runs faster. For example, customer representative doesn’t forget to send new sales order information to the planning department since it is automatically notified by workflow system via email.

3- It may help to EAI (Enterprise Application Integration) tasks. Let’s assume that we have a third party CRM application and workflow system runs in our ERP system. We may need to enter customer order information to the CRM system. In this case, we can add a triggering workflow action definition to automate to enter sales orders to the CRM system as soon as it is entered to the ERP system.

4- It eliminates repetitive user tasks that are performed in some certain events. For example, in your bug tracking system, you may want to notify developers when you approve new trouble tickets to resolve. By just defining a workflow action for your approval event, workflow system automatically sends email alerts to the associated developers for you.

5- Workflow system alert messages may ease to access and do some jobs. Workflow system should generate system messages as well as email messages. Email messages are useful when related user is not online in the system. For example, HR department want to hire a candidate and entered his CV to the system and waits an approval from the related department manager. If workflow system generates an email message for approval, it should be very easy to approve without requiring to login and navigate to the related candidate record. In our implementation, when manager clicks approval button, it is automatically approved.

Structure of Workflow System

Definitions and Instances: There are 2 basic elements in a workflow system; definitions and execution instances. Every workflow definition is composed of some activities. These activities reside on a flow path. Activities are forwarded in some conditions. Execution flow from one activity to another is triggered by some events. Workflow activity execution information (state) is held as workflow instances separate from definitions. These execution instances hold some activity parameters (state) thus we can remember what to forward when an event is received by workflow engine. “Workflow Engine” manages instances and their lifecycles.

Activity Actions: Every workflow activity may be registered for some type of actions by the users with some criteria. In our ERP system, we made it very easy for users to register workflow activities and choose their actions. For example, a user may register an email alert to the “Order Closed” activity of “Order Processing” workflow. User may want to know if deadlines are conformed. This user may want to receive emails only if sales order total price is bigger than 10000 bucks.

Activity States: Workflow activities may have some states like; Initial, Started, Completed, Suspended and Terminated. An activity may need many events to pass “Completed” state. For example, “Sales Order Fulfilled” activity may need some “Procurement” and “Production” events for that sales order. We should also consider the execution cancellation. Let’s say customer cancelled his order. Then this workflow passes to the “Terminated” state when receiving “SalesOrder” cancel event.

Programmatic Activities: In our workflow system custom actions may be programmed as well. We designed our workflow system so that its states and actions are programmed within rule definitions. I mentioned earlier that our programmatic rules are java codes which are dynamically compiled in run-time. This gives us great flexibility when managing activity states. I see in many workflow products that have some simple “Yes”-“No” branch conditions but real requirements are far from that including many comparisons and checks.

Achieving Platform Independence For Web Applications

2009-09-05T12:09:00.000-07:00

When mentioning cross-platform, platform independent software, Java’s motto comes to mind “Write Once, Run Anywhere”. It was one of the best features of Java language for software developers because it frees them to consider porting their application to different hardware and operating systems. Platform independence is about moving your software to different environments and starting new relations while expecting to get same platform services. I think to be portable it is not enough to be independent of operating system or hardware. There are other platform dependencies as well. Here in this post I am going to try to uncover some other platform dependencies for a typical web application.

Why do we bother to be independent of platforms? Platform independence means freedom, breaking the chain of dependence. Market has many competing platforms and our customers may select one of them. They have choices and options. While having options, they can find a platform that best suits their needs. In this sense as an application provider, to reach a wide customer base, we have to support as many platforms as possible.

What is the difference between standard library dependence and platform dependence? By saying platform, I mean a software layer that our application use highly extensive services from it. As application layers increased, many new platform dependencies emerged. One classic example is application server. For a web application, an application server is a platform since it gets many services from application server. For a web application, application server dependence is much more real then an operating system dependence since OS dependence is a matter for application server or its language platform.

Every type of application may have variable platform dependencies. To know them is a first step for platform independence. I will try to list the common platform dependencies of a web application:

1- Programming Language and Platform: The biggest dependency is programming language, and its version, its implementation. In Java, we are dependent to JRE which is mostly the JRE of Application Server. In Java, programming language dependency could only be possible in integration layer by using web services. On the.Net camp, language neutrality is somewhat possible. Knowing language version and implementation differences could enable us to write portable software in this aspect.

2- Application Server (+HTTP server): For a web application, this is also a big dependence. If application frameworks don’t isolate applications from this dependence, every application needs to comply with new application server differences. In Java, fortunately there are server-side web standard specifications that Java application servers comply with. Although there are standards, there are still some vendor-specific features that may result dependency. Porting a fully-fledged Java EE web application from one application server to another still requires some amount of work.

3- Database Server (DBMS): For a database-driven application, DBMS is very important. For portability, DBMS vendors’ standards save us. For example if you program using ANSI SQL, that means your database codes are database independent. Besides, SQL is not the only dependency point. DBMSs come with many charming vendor-specific features; replication services, reporting services, document and XML publishing services, BI services etc. When these services are used, it hurts application portability to different DBMSs.

4- Browser: Browser is the universal client of web applications. Fortunately, here again standards saves us a lot. All web standards built on common programming standards thus Internet and web could be possible. In spite of many web standards, we have to still deal with many type of browser compatibility problems. Even same browser may change its behaviors along its new versions. We used Internet Explorer’s features for rich functions in our web applications (Modal-Modeless dialogs, HTC, Print Templates) but if we want to support other browsers, we have to re-implement them for new browsers.

5- Operating System: Since Java VM eliminates us to consider OS differences by hiding low-level system calls, this topic is a minor dependence for Java web applications. Nevertheless, in some conditions we may need to know where our application is operating. For example, you want to invoke an executable application, in AIX file extension is “sh” whereas it may be “bat“ or “exe” in Windows. File and path separators are also different in Unix and Windows that should be cared when programming for file system. I think Application Server vendors have more trouble than our applications in this context. I know that in WebSphere 3.5, it has 2 different (very similar but different) code bases for Windows and AIX. Java is platform independent but as seen here applications may not be so.

When we need a specific non-portable feature from a platform but we want to remain cross-platform as much as possible, there 4 different strategies we may prefer:

1- Find A Cross-Platform Toolkit For That Service: For example, if we need to remove DBMS dependency, we may use an ORM framework that deals with different DBMS platform.

2- Isolate Dependency by Accumulating Them in One Package: If you can’t get rid of that dependency, you may refine them by accumulating platform-specific codes in one place, i.e. in a package.

3- Just Use Platform Dependent Feature: If no resource or time to question dependency and it is a big requirement, just use it. A code comment may be added for later attention if we are sure that this is a platform-dependent feature.

4- Leave Platform Dependent Feature: If it is optional to use that feature and portability would be significant for us later, than we may leave that feature and don’t use it by staying in lowest common denominator zone.

Application Logging: What, When, How

2009-08-30T08:58:00.000-07:00

Logging is a fundamental part of applications. Every application has a varying flavor of logging mechanism. A very good designed logging system is a huge utility for system administrators and developers. Especially support team knows its value very well. Logs save many valuable hours of support team or developers. As users execute programs at the front, system invisibly builds a vault of event information (log entries) for system administrators and support team.

After stating its value, let’s try to figure out logging requirements for an application. Java has a standard logging API in its new versions (java.util.logging). Log4J is also a well-known library for logging. We implemented a transparent “Logging Service” in our application framework. You may prefer some kind of logging implementations but there are some other important questions you have to answer in your application design which are what to log, when to log, how much to log, how to control logging and how correlate it with your exception system.

What to Log

Some application exceptions should be logged:
Why not to log all exceptions? Some exceptions are managed exceptions which are thrown by application as a warning or as a validation error to the user. If all validation errors or application exceptions are included in logs, logs lose their usefulness since it would contain many entries hard to reach valuable entries. Here, we should discriminate exceptions if they should be logged. We used a Loggable mark interface to determine if it is included in logs. Another best practice is to have a single global exception handler. In this way, application developers don’t worry about logging of any exception they generate. A single exception handler means a single unified and neat exception logging.

Some application events should be logged:
For major components of application we may log lifecycle events like start, stop and restart. Some security-related events may be logged such as unauthorized URL access attempts, user logins etc. Some resource thresholds may be exceeded and should be logged.

Some application states should be logged:
In some codes, we should ask that “What could go wrong here in this code”. If this state occurs, we may throw an exception or log it (if we don’t want to interrupt current process) with some levels like Error, Warning or Information. For example if a connection is normally released while it is uncommitted, that info may be logged as uncommitted connection release that points some application code problems.

Application process steps may be logged:
For some multi-step processes, we may create process log files so that we can inspect process progress. It behaves like a smart progress indicator telling you what is happening in the current step. Process logs may include errors, warnings and information.

Some debug information may be logged:
In some applications, you may have some errors and can’t find why this is happening. You may add some debug logs into your code and redeploy it to diagnose the problem. Some chronic bugs deserve debug traces which can’t be detected in development environment.

Executed SQLs may be logged:
In some conditions, we may want to get the SQL statement executed by an application. We should easily switch logging on without start/stop of system. Let’s say user complain about wrong report result when executing a report. If we don’t have a clue about that we may log SQL and trace it.

User HTTP requests may be logged:
We may need what is coming from the user with full parameter details. To achieve this kind of logging we have to plug a log service in servlets and JSP pages.

Executing threads may be logged:
I mentioned a BlackBox implementation in our applications in one of my previous posts. You may log executing thread information to that black box to figure out what may go wrong in a system crash.

JavaScript errors may be logged:
This may be considered same with first item but its implementations totally differs. You need to have a global JavaScript exception handler. In the handler, you submit JavaScript error with AJAX to log on the server (assuming it is not an AJAX error).

Some Best Practices

As I said above, exception handling system is an important plug point for logging. In our all Servlets and JSPs, main code blocks have following structure to enable global exception handling. This standard eliminates the work of logging exceptions by developers (Transparent unified logging):


try{
}
catch(Exception e){
   GlobalHandler.handleException(e);
}

We name “Log Medium” to mean where to store log entries but in Java logging API it is called Handlers. For some type of logs, database logs are handy since you can run powerful SQL queries against log table which is merely possible in files.

To assign log messages to the appropriate log levels is also important. Otherwise some wrong alerts would mislead system administrators.

We used System.err stream for application-level logging which is same place with e.printStackTrace() which may be printed with our global handler log entry at the same time. We wouldn’t need all exception’s stack trace but some may be very useful to see where the root of the problem (i.e. StackOverflowException).

When logging exceptions we used following format. Some articles recommends that we’d better to include problem failover suggestion but I think this information should be given to the user not included in the logs. Logs are rarely read but exceptions are in front of users.

Log entry should answer following questions:
Who(UserName), When (Timestamp), Where(Context, ServletOrPage,Database), What (Command), Result (Exception)

[ErrorHandlerName] UserName: userabc DatabaseName: ABC Timestamp: 15.07.2009 13:02:08 Context:Servlet Page: /prod/sales/SalesOrders.jsp Window: windwName Command: saveSalesOrder Exception: ShortDescriptionOfException
ExceptionStackTraceHere

In a clustered environment, logging should be considered. We separated same type of log files with a cluster node name suffix to the file name. Otherwise, concurrent log writes may lead to some problems. system_node01.log system_node02.log

As a last item, logging may be a very interesting performance killer. If applications begin to log a lot, application performance may severely fall down. I have a real life story for that. Once we had forgotten to include an image file in deployment package. It was being used in every page. When we started system after installment, page response was so slow. Log files seem small enough that may not incur problem. After, opening and reading log files, we realized that log frequency (stating image is missing) was high and that was causing the problem. In conclusion, your log file size and log writing frequency should be small.

Identity Generation For Primary Keys

2009-08-23T04:02:00.000-07:00

When designing a database model, there are many critical decision points regarding primary keys. These decisions have 2 affected results; database-side and application-side. Some PK structures ease application development but make it hard to use SQL for reporting these tables. These decisions may be variable that is relative to development context. For PKs, sample design differences are; surrogate keys versus composite keys, natural keys versus auto-number keys (I will refer it as “identity” here).

Natural keys are meaningful or intelligent keys that is easy to use in SQL queries. Contrarily, identity keys are composed of sequence numbers. Identity keys don’t have any meaning for end users. Identity keys are easy to use in applications but hard to use in SQL queries and reports. As I said, deciding which one to use is dependent to application context. We generally preferred natural keys for parameter tables and used identity keys for transactional table keys. Sometimes, we needed to use both structure (semi-natural) in same table according to user settings. Identity keys may be used within single column (surrogate keys) or multi-column PKs (composite keys).

If you decide to use an identity value for a table primary key then we reach another decision point here; to use database-generated keys (sequences) or application generated keys (In JPA it is called as table generator). Both have advantages and disadvantages. Database-generated keys are completely DBMS dependent and hard to use in applications (i.e. reading its value). Application generated identity columns are easy to use if persistence layer has a strong support for it. Besides, designing a simple generator class for your sole requirements would not take much time. In our persistence framework, we began with using database sequences but then we quit to use them and developed an identity table based service to use in all applications thus simplifying the solution for identity problem.

Having a single architecture (Table-based identity generation) to use in applications had served following benefits:
1- Limited and concise options make development faster: Since there is only one identity service to use in applications, developers don’t have to tackle the differences between different kinds of sequence generator techniques (Identities may bring so many problems that are hard to believe).

2- Simple identity API makes it easy to use identity generation: In our persistent objects, identity generation can be plugged to any column value whether it is primary key or not. There is only one method for application developers String nextIdentity(), String getIdentity().

3- We could cluster-enable whole applications very easily: Since all applications used the same API for identity generation, by making identity service cluster-enabled, whole applications identity generation was cluster-enabled at the same time without any additional effort. So our identity service can now run on application server clusters without any conflict.

4- Identity column replication could be possible: When replicating table data to other databases, identity values would collide. By using a ranged identity definition in identity service, identity value conflict never occurs. Many DBMSs solve identity replication conflict by using identity ranges.

There are many issues and solutions you can find on the net but I want to mention some additional issues when using identity keys:

Dependent Identity Keys: Some dependent tables may need to increase their sequence only for parent table record. Let’s say we have a parent table key “0055”, we may need to generate keys like “005501”, “005502”, “005503” etc. In your dependant table’s persistent objects, you have to read parent object’s identity key and increase according to that value.

Semi-natural Keys Having Identity Value: Some key values may have both natural (meaningful) and auto-number parts. For example, you may have a sales order key like “SO0908000045” which says that this is a sales order(“SO”) and taken at date 09, 2009(“0908”) and has a identity key “000045”. In this key, only last part is taken form identity generator.

Updating Identity Generator Values At Runtime: Sometimes, it may be required to take some identity values back. There are many circumstances that result this requirement. Some values may be increased by mistake or updated from database and primary key conflict exception may be taken from database. At these conditions, we need to update identity’s current values. We have an identity update page in our admin application to alter current sequence number. The key point when updating that value is to alter both database and cache sequence value (Identity cache is a cache that eliminates database access every time an identity is required).

Here is the architecture of identity service in our persistence framework:

The Gap Between ERP and Enterprise Java Technologies

2009-08-16T03:24:00.000-07:00

Major ERP packages (Business Applications) are dates back to 1990s. Enterprise Java takes root in Java language invention in 1995. ERP and Enterprise Java (Java EE) marriage is a long-term story. After Java language wide-adoption in server-side programming, many ERP vendors started to move Java language as their core language. At the same time in Java-side, enterprise features are added to the platform to meet the requirements of enterprise computing. Today, where are they? Has the gap between both camps closed? In this blog post, I try to give my opinion about this issue.

There is a gap because Java EE moves much more flexibly than ERP technologies. ERP packages’ technology adoption is slower than Java’s progression. Even tough ERPs are dressed with style decoration, their core architecture is difficult to adopt new trends and technologies. Let’s see why ERPs are slower in technology adoption:

1- Numerous Customer Installations: ERPs’ current customer base is a backward compatibility factor for vendors. Customers would want a stable system with enough technology options. As the ERP changes, their total cost of ownership increases because they are forced to upgrade new versions. If ERP vendor supports old versions, this is going to be a very big burden for ERP vendor. Because of this bidirectional relation, natural result is to keep change minimal thus results low-technology adoption. An example for this is SAP’s current customers’ reluctance to new versions.

2- ERPs’ Legacy Architecture and Languages: Since it is almost impossible to see new vendors in ERP software market, old players prevails the current business applications market. Old players have old guns, and they struggle to update them. New technology updates could be merely achieved in front end. As a software challenge, to change a language in an ERP software may be simpler than to change architecture. If both situations are valid, to write a new ERP from scratch is inevitable and that is a very big risk for ERP vendors to take. Because of that, Web architecture would be a roadblock for vendors if their core is not web-based. Examples are Oracle ERP proprietary core platform and Fusion Middleware, SAP ERP’s ABAP and Netweaver Platform, Peoplesoft’s SQR.

3- Technology Bubbles: Temporary trends in technology are also a factor for ERP vendor’s reluctance for change. We see a new trend in every 3-4 years. For an ERP vendor, that’s impossible to change according to these waves. At the beginning, it is impossible to guess that a technology will live in the long-term. Their current behavior is to support in the front end but not at the core. As an example, SOA was number one question asked to the vendors for their support. Nowadays, they would wrestle with “Cloud Computing”.

From the Java-side, there are some additional factors that enlarge the gap. I think that ERPs’ (Business Applications) development experience or development tools couldn’t be utilized sufficiently when making specifications. Enterprise Java is built from the technology vendor’s feedbacks and requirements with technology improvements. But they may not have got enough feedbacks. This may be regarded as usual because either ERP technology architects are not still in the job or they don’t know Java platform or they didn’t send any representative to the JCP programs. On the other hand, Enterprise Java technology leaders may not have enough ERP experience to capture the enterprise requirements. One example for this is EJB technology. If EJB were used and implemented in an ERP platform, its architectural problems could be detected earlier. From the early EJB specs, I guess a banking system experience from the leaders but this is not wide enough to include enterprises. I know many EJB-based enterprise projects that failed to meet expectations and changed their persistence architecture. Who knows, if Sun had an ERP package, Enterprise Java would be better to fit in ERP technologies or vice versa.

Optimistic and Pessimistic Locking For Application Concurrency

2009-08-06T03:59:00.000-07:00

Concurrency is a delicate subject when designing multi-user software systems. Forgetting to solve the issues of shared resource access may bring many concurrency bugs. In a typical application, there are 3 levels of concurrency; object, database and application. Each level has to be considered and handled separately. You shouldn’t use the concurrency features of one level for another. Doing so may result some design flaws. In this post, I’m going to share the architecture of locking service in our persistence framework.

Object Concurrency
In Java, there are many language-level and utility features for concurrency. Beginning with “synchronized” and “volatile” keywords, then Object’s “wait” and “notify” methods for thread synchronization and finally Java 5’s concurrency utility packages, Java has many powerful mechanisms for concurrent programming in object level. In this level, shared resource is object.
http://java.sun.com/j2se/1.5.0/docs/guide/concurrency/index.html

Database Concurrency
In this level, shared resource is data residing in containers, tables, rows and columns. Some of the terms used are transaction isolation, locks, deadlocks, time-outs. To deal with database concurrency, we have some JDBC APIs (i.e. Connection.setTransactionIsolation) and some SQL constructs (i.e. FOR UPDATE, FOR FETCH ONLY, LOCK TABLE).
http://www.ibm.com/developerworks/data/library/techarticle/dm-0406whitlark/index.html
http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db2.doc.admin/dchfd.htm

Application Concurrency
In this level, shared resource is an entity (record) that may be composed of many sub-entities and updated from some update pages by users. A very important confusion when designing application-level concurrency features is thinking application concurrency in terms of database concurrency. They are different things, their levels are different. Application entity concurrency granularity is different from database tables. For example, when we talk about sales order pessimistic lock, that lock also covers order line and order payment table data locks. Application lock structure is also different from database locks. For example, an application’s optimistic locks are maintained on Java objects and version controls are run at application-level. In this level, locker is application user.

For application concurrency requirements, we designed a locking service in our persistence framework. It both supports optimistic and pessimistic locking as described below. This service is cluster-enabled and runs on application server clusters.

Optimistic Locking:
In this locking method, multiple users can enter to update page of a record without restriction. First committer’s update is executed successfully and subsequent updates are denied by giving an optimistic lock exception.

Design notes:

This locking is executed in the background transparently without requiring extra definitions by programmers. There is no API set to configure this feature so it frees programmers from this task. It is built-in and optimistic lock works behind the scene whether pessimistic lock exists or not.

Sometimes application may receive this exception itself. This locking prevents some application code bugs.

Persistent objects’ PK values are versioned with a timestamp value in memory. No need to add a version column to tables/objects or create a version table since everything is run on memory. Locks size held in memory is not so big to worry.

Optimistic lock exception gives enough information about who updated record from which program. Programmers don’t have to build exception messages.

The only survival from this exception is leaving the current update page and re-entering it again. That provides a lock refresh.

In rarely or infrequently accessed applications, this locking is enough without requiring addition of a pessimistic lock. If optimistic recovery cost (update lost) will be high then pessimistic lock should be used.

Pessimistic Locking:
Only one user can enter an update page and other users trying to enter update page or trying to update same record from another module gets a pessimistic lock exception. Users who only want to see/read the record may be permitted by not acquiring pessimistic lock and disabling save button.

Design notes:

To use this locking, an explicit API call is required. 2 simple methods used by persistent objects for utilizing this feature; acquireLock, releaseLock.

Pessimistic lock and optimistic lock works independently and together so if a program doesn’t use pessimistic lock, then if double-update occurs it is caught by optimistic lock.

Pessimistic locks can be associated with transactions but this is not mandatory.

We used only master records for pessimistic locks in our applications. For example, if you lock a SalesOrder, you don’t need to lock all detail records. Note that all applications accessing and updating SalesOrder should use pessimistic locks for master object not detail objects. Otherwise they bypass master object lock.

Pessimistic locks don’t block the read-only queries. For example, we can get monthly sales report whereas a user is on an update page of a SalesOrder holding a pessimistic lock.

Using database locks for pessimistic locks may degrade application and database performance. Especially, some applications which have multi-conversational update pages are not appropriate for database-driven locks. Moreover database locks are hard to detect and release. If database tables are locked, we can’t discriminate if this lock is resulted from standard pessimistic locks or application SQLs (A real nightmare in a mission-critical system for DBAs). For this reason, it would be better to design and use pessimistic locks in application-level.

Some pessimistic locks may need to be persistent. We supported persistent pessimistic locks. For example some archived records should be locked when it is tried to update. A user may want to lock a record for some time.

Pessimistic locks have a timeout and a background thread cleanses stale locks. If user forgets to close his/her window or close window on update page, then lock is held unnecessarily. Locks should be released when leaving update pages. There are many possibilities to leave update pages and each should be handled properly to release lock; returning main list by saving current record or canceling, clicking a link that causes to leave update pages, logging out the system.

User can release or add locks and see other locks from a locks page. In this way, users can manage their application locks instead of calling system managers.

Database Connection Pooling: Gain Application Performance and Monitoring

2009-07-31T04:39:00.000-07:00

To use a connection pool is the number one performance tuning action in database-driven applications. To create database connections takes time depending on DBMS and server configuration. In applications, persistent objects execute unbelievably large volume of SQL statements. To have a fast and robust pool is very important. Connection pools are not just a dummy cache to hold references of connection objects. They should serve to monitor the database activities of persistent objects. When applications deployed, it is a very handy tool to diagnose the flawed persistent objects that may causes locks or transaction problems.

In JDBC, there are 2 classes to get a connection from; DriverManager and DataSource. DataSource has more features like distributed transactions, connection pooling etc. DataSource interface is generally implemented by application server vendors. There are also some open source connection pool libraries. To build a more general solution, we implemented a connection pool in our persistence framework compatible with our persistent objects and transactions.

I’ll share some of the experiences when developing this pooling service for our persistence framework. (We name it DBConnection Cache Service):

1- A connection pool enables us to inject a layer behind JDBC API. For database processes, entry point is to get a connection. After getting connection, creating Statements and iterating through ResultSets follow. When creating a statement, we can return a wrapper statement object implementing a Statement interface. Thus, we control every JDBC API call. Background operations can be easily carried out like statement caching, automatic statement closing, disabling close methods etc.

2- A connection pool makes it very easy to get a connection by eliminating to to deal with configuration tasks for developers. Otherwise every developer needs to configure his/her driver settings, connection settings etc. In a typical application what I want would be so simple. I want to connect a database and execute my queries, that’s all. Connection pools should take the responsibility of all default configurations. We even simplify it further by hiding the codes to get connection from connection pool and releasing it with getConnection and releaseConnection methods in persistent objects.


public class DBEmployee extends DBObject {
...
protected void dbInsert() throws SQLException {
 try{
  String sql = "MYUPDATE SQL HERE";
  Statement stmt = getConnection().createStatement();
  stmt.executeUpdate(sql);
  ...
 }
 finally{
  releaseConnection();
 }
}
}

3- A connection pool invisibly handles stale connections. It is a very common situation that some db connections are closed by DBAs or by some db scripts. With the help of connection pool, an application code gets never a stale connection and doesn’t need to have plumbing code to handle this. Connections are controlled just before giving it to the application by executing a simple SQL against a database table if it is still alive.

4- A connection pool prevents connection leaks like detection of uncommitted connections, automatic statement releasing etc. By using its monitor tools (Many DataSource implementations unfortunately don’t have this feature), we should see that which objects still didn’t release the connection. It is very critical to see the objects that don’t release the connections. Persistent objects may be forgotten to add a releaseConnection method (Coding problem) or a database lock may occur at that time (Database problem).

5- A connection pool helps us to solve database performance problems. For example, a query may take a lot time to finish. For example an application takes too much time to finish. There are hundred of objects accessing database in that application. How would you detect the bottleneck? Without needing a profile tool we can easily detect by just monitoring the objects that holds connections and their elapsed time.

6- A connection pool can provide information about connected active users (People). Which user is executing by using which connection can be answered. It may provide database information, say we have multiple databases. We can assure that a user is working on a specific database by monitoring connection cache.

7- A connection pool should release unused connections. Connection pool would have a max size parameter. Total created connections will be dependent to the active users and applications. Some applications may mistakenly use up all connections, in that cases applications should be fixed. Total number of connections opened would change according to persistence framework you used. To have small number of connections is always desirable.

Connection Pool Monitor Page:

By developing a connection cache service, we provided a simple API for application developers to focus on business logic and SQL queries. All the subtleties of dealing with connections are undertaken by this connection pool service.

As a footnote, so far, one minor missing in JDBC API is connection id support. Having this information could make it possible to associate a pooled connection with physical database connection. Thus, we can kill a locked connection very easily without trying to find it in database with database tools.

Audit Trail Logging: Transparently Tracking Data Changes in Persistence Layer

2009-07-24T01:20:00.000-07:00

Audit is a process that can apply many fields of an enterprise including information systems. Whether it is internal (Audit Department) or external (Outsourced), audit is important to control people, infrastructure and third party relations. Many business processes are carried out by transaction processing systems with a database backend. To track and spot the processes, we need a change history for every transaction. In enterprise applications, this is possible with data change logging. Especially, ERP applications have a built-in audit trail logging feature.

Trail logging has many benefits:
1- It helps audit personnel to detect fraud. It enables us to detect unauthorized or unapproved data manipulation. It only helps because fraud can be possible by legal and normal records if nobody sufficiently controls the business transactions.

2- It provides data change non-repudiation. That means when a user changes a record and claim he/she didn’t change the record, then its validity is precisely determined via inspecting audit trailer logs. Sometimes users may blame the applications for transaction leaks, this is a very nice tool for application developers to justify the right side.

3- Users also use this facility. When a user notices that his/her inserted records are changed, by looking audit trails he can understand who changed the records.

4- Developers also use it to determine abnormal application behavior. An application may inadvertently alter some records and this can be easily debugged if an audit trail is present.

There are 2 types of audit trail logging technique:

Audit Trailing on Table
You add some meta-columns to your tables for data change information like UserInserted, UserInsertDate, UserUpdated, UserUpdateDate.

Pros:
Easy to record trail data
Audit data size is small
Easy to fetch trail data with SQL statements

Cons:
Only one update trail data can be stored
Can’t store changed data
Data and row meta-data(trail data) is merged

Audit Trailing on External Table
An external table may be used to keep change logs. Every successful transaction should be logged to this table. This table may be taken to another database or split to multiple tables to ease management.

Pros:
Insert and every update can be logged
Changed data can be logged
Separated form the transactional data
By adding a transaction id to logs (Log table has a transaction id column), it is possible to see every change in a transaction (This is a shining feature when debugging transaction boundaries).

Cons:
Hard to fetch trail data with SQL
Trail data is bigger since PK columns and table names are also stored within every row.

We used second technique within our ORM Framework (This feature is still missing or hard to use in many other ORM frameworks). We implemented it as a transparent service plugged to persistence layer so that developers don’t have to write any log code or SQL. By the nature of a service, it can be closed and opened when necessary without system restart. It works asynchronously so that application doesn’t wait logging process. It is independent from any DBMS vendor. We named it as “Database Logging Service”.

Architecture of database logging service is as following:

Books:
Integrated Auditing of ERP Systems

Auditing Information Systems: A Comprehensive Reference Guide

Auditing Information Systems

Easy User Authorization for Web Applications

2009-07-17T05:21:00.000-07:00

Authorization is one of the primary common features of all software systems. When developing an authorization system for our ERP software, I felt that I’m re-inventing the wheel since everybody wrote it again and again in many types of software like OSs, DBMSs, HTTP Servers, File Systems etc. They have different flavor of authorization capabilities. Re-inventing is not limited with different type of systems so it is re-implemented again and again in every enterprise web applications.

One of the missing or weak features of web frameworks is easy authorization support. In Java EE standard, JAAS is used for authentication and authorization. I assume framework writers don’t want to re-implement this service that is already defined in Java EE stack. However, it would not be easy to use JAAS in web applications. In my opinion, JAAS is much more appropriate for desktop-based applications and component level services. In web applications, we need a new abstraction for authorization that is a URL-based authorization mechanism.

By saying “Easy Authorization”, I mean followings:
1- It can be used both by power users and administrators from a management module. If you have numeraous users and companies using your system, you have to give the task of managing user accounts and rigths to some power users.

2- It should not force us to declare permissions in configuration files every time a new user or role is created. System administrators can alter these files but power users can’t access and understand these files.

3- User access permissions should be grouped within role definitions (Role-based Authorization). In this way, we define an access schema only once, and assign it to new users many times easily. This is a magnificent feature. Otherwise, every time we create a user, we had to define same access rights again and again by leading to duplicate definitions which are open to errors.

4- Authorization should be a transparent service for the developers. Authorizations definitions shouldn’t be embedded in code. By transparent authorization, a developer doesn’t have to think the authorization aspect of the program since it is a built-in feature. In some edge conditions, developer should also have the chance to override and change authorization behavior.

5- URL-based authorization should be easily mapped to the part of the application thus removing the task of defining an authorization grant (URL) every time a new web application is written. For example, when a user has access rights for a module, a new sub-module added to this module is automatically granted to access for its users. To achieve this, you have to divide your application into sub-modules (Logical Parts). In our applications, we defined 3 levels: suite (A bunch of web applications), module (a web application), sub-module (a logical part of a web application). For example, “Human Resources” is a suite and “Payroll” is a module and “Advance Payment” is a sub-module.

6- Record and field-based authorization should be defined within the same screen with module (URL) access permissions. If you are using different frameworks for persistence and web flow, AFAIK that would be impossible for now. Single point of access makes it easy to define all authorization rights.

Architecture for Authorization Service

Web Application Authorization: Authorization control is carried out by an authorization service at the front of servlets. Each user request is evaluated according to its user, URL and command. If user has the authority to run this command at this URL, then command is delegated to the business layer for further execution. If user doesn’t have the permissions for the command, then a proper “Access Denied” message is returned.

Record Authorization: Authorization is also executed behind the persistence layer for record-based authorization. If user is not authorized to fetch some kind of rows, then these records are filtered out. Or a user is not authorized to edit some rows, if an update is tried on this row, then it is denied.

Validation Architecture in Our Application Framework

2009-07-10T05:40:00.000-07:00

Validation is a very important topic which is poorly handled in many frameworks. Although they tend to give “framework” name to their validation mechanism, as I will explain here, they have many drawbacks which are not easy to fix since they are architectural. Validation both enhances client interaction and eliminates some problems earlier. For instance, a database error reported back a user may not give enough information for correction. Or a missing validation may lead to erroneous data. Validation has many forms in software; data, input, requirement, architecture, class, object validation etc.

Problems in Current Validation Mechanisms

1- Validation is a Broken Link between Different Types of Frameworks: Every framework needs some sort of validation. Because of this fact, every framework has its own validation mechanism and adds a burden on developer. What we need is a single point for validation. This single validation mechanism should meet both server-side and client-side requirements. Broken link can be easily seen in validation error handling and reporting codes. Yes, when I talk about MVC integration pains (Better Java Web Framework blog post), this is a part of it.

2- Validations are not really Metadata-based: Metadata-based validation is a must needless to say. Although I will mention only database metadata, there may be many kind of metadata in other systems. Although some frameworks provide features to declare validation rules, this task is given to developer. Than they claim that they are metadata-based. This is not true, metadata-based validation is not that. Metadata-based validation is automatic validation (nullability, data type, data format, data range, data length, data security) by using the metadata knowledge of persistent objects without requiring new declaration. I sorrowfully read such documentation; “Add a ‘Not Null’ annotation to your persistent object field to make it required”. If I add this line, why do I need metadata-based validation, I just add it to the business-tier code. If that code is generated then this is OK but if we add them one by one, it is a real tedious job for programmers.

3- Persistent Object Metadata Declaration Place: Another problem is where to put object metadata. Object metadata is somewhat similar with database table metadata but may contain additional information. It is mandatory for many reasons that we must have an object metadata. At first, many frameworks used XML then recently are using annotations mixed with ugly validation annotation rules. For XML use; one of my primary programming rules is that programmer should have minimum touch-point for development, what I mean that metadata should be placed in objects; easy to reach-read metadata, easy to change. We used an array block for metadata declaration in initialize method of objects. I am also not sure about using annotations for metadata.

4- Use of Annotation and XML for Validation: We see again an overuse of something (annotations), history repeating. I believe many of you confused when reading mess of annotations lingering on top of classes or methods. Let’s do every programming job by using annotations (!). Although it may seem a good idea to use annotation (JSR 303), let me explain why I don't like this approach; you can’t declare every validation rule by annotations, that’s enough. For example, you have to validate parent object according to child objects’ data (Say “SalesOrder” with “SalesOrderLine” persistent objects, rule is “Lines’ total amount can’t be bigger than header amount”), then if rule is not met, raise a validation exception. How would you declare this validation with your annotation? Just adding a validation method but that causes to have 2 places for looking up validation rules. Instead, I strongly favor programmatic metadata-based validation; this is simple and effective.

5- Validation After Setting Persistent Object Fields: One of the major problems is that where do we store invalidated data? One pitfall of many validation solutions is to do validation after setting the fields of persistent objects. Let’s say we have a web application which has an update form. How can we validate a data that can’t be set to a field (Formatted data or String value to number field)? When user enters a data not appropriate validation, how do we rollback set data to the objects if we want to use dirty object again? Invalidated object may be replaced with a new fresh object but this is not easy in some page conditions (In sub-tabs, user may re-edit same object and you can’t get a fresh object from database). This is why Data Transfer or Value Objects are required. But that approach may add a new object layer without real benefit (Code duplication with persistent objects, same setters and getters). What we did to solve this problem is to use a validator object (Contains HashMap for fieldnames and values) attached to persistent objects to store invalidated data. If validation succeeds than these values are committed to the object, if not exception is thrown and object fields are left intact.

What to Validate
Our programmatic metadata-based approach contains following validations:
1- Object Consistency Validation: In our framework, persistent object consistency validation is done when the object is created at the first time. What do I mean by object consistency validation? This is a validation of object’s metadata information against database table structure. For example, when you declare a field as “Primary Key” or “Foreign Key” then if that declaration is different in database structure, object produces an exception. Another example is that your object has a field that doesn’t exist or removed from database. Missing database fields are reported within a validation exception. This validation is very important that provides consistency between object and database.

2- Input Data Validation: A program may take input from different kind of systems or users. Many scenarios; A user may enter some data from a web client application; a web service may be invoked by a batch program to enter data to the system. An application should validate any type of data coming from the outside. Even within same enterprise applications, different web modules should validate data coming from other module if their tables are different. Our metadata-based validation frees programmer from adding validation rules for standard validations including nullability, data type, data format, data range, data length, data security. If additional validations are required then this rules are added by programmer to the standard validation method (dbValidate). When validation runs, firstly standard validations are automatically executed then finally extra validations added by programmer are executed. If validation errors occur, these errors are reported field-based to the user. In addition to programmatic validation, we added a Rule Engine (I mentioned about it in my previous blog post) for dynamic validation rules that are the benefit of XML-based validation definition. We added changing validation rules to the Rule Engine definitions.

Validation Architecture
Let’s delve into our validation architecture:
1- Persistent Object Metadata: Every persistent class has metadata information in initialize method. We tried to use minimal metadata here. Some database metadata is taken from the database and doesn’t need to be declared here (ie. FK Table Names). This metadata is used by our validator class for general validations. A persistent object may contain other fields that are not persistent. These are not held in this declaration. Only the table-columns are included in this array list.


public class DBEmployee extends DBObject{
      private int m_nNo;
      private String m_sName;
      private String m_sSurname;
      private int m_nDepartmentNo;
      private int m_nAddressNo;
...
protected void initialize() {
      super.initialize();
      if(isInitRequired()){
                  setTableInfo(new DBTableInfo(this,new String[][]{
                             {"No","NO","true","","No","true"},
                             {"Name","NAME","true","25","Name"},
                             {"Surname","SURNAME","true","25","Surname"},
                             {"DepartmentNo","F_DEPARTMENT_NO","true","","Department No"},                             {"AddressNo","F_ADDRESS_NO","true","","Address No"}},
                      "TEST.EMPLOYEE"));
      }
}
}

Array Reference:
[FieldName],[ColumnName],[Required],[Length],[Text],[Identity],[FieldViewer],[PrimaryKey],[ForeignKey] …

2- Fetching and Setting Data to be Validated: Every persistent object has a corresponding validator object to store changed data by the user. In new record entry, validator object is empty. In update record, validator is filled with data from persistent fields (These fields are also filled from database fetch). In input form, validator data is used to display and set before validation.

Update:
DB --[FETCH]--> Persistent Object --[FETCH]--> DBValidator --[DISPLAY]--> Input Form
Insert:
Persistent Object --[NEW]--> DBValidator --[DISPLAY EMPTY]--> Input Form

A part of input form (Max length control and input text coloring for required fields is done via metadata, field-based validation reporting is possible):


<tr>
<td class="tdBgK" align="right" valign="top"><%=appLocale.getText("Name")%>:</td>
<td class="tdBgA" valign="top">
<input type="text" class="inputTxt" <%=dbEmployee.isFieldRequired("Name")?UIConstants.REQUIRED:""%> name="Name" size="25" maxlength="<%=dbEmployee.getFieldMaxLength("Name")%>" value="<%=dbEmployee.getValue("Name")%>"><span class="errorMessage"><%=dbEmployee.getError("Name")%></span></td>
</tr>

When user enters data this is stored in validator object initially. After all fields are set then validate method is called (manually with dbValidate or implicitly in dbSave method). Validation occurs only for the set values. Programmer may set persistent object fields directly, but these won’t be validated.

Save:
Input Form --[SET]--> DBValidator --[VALIDATE-SET]--> Persistent Object --[SAVE]--> DB

Servlet method handling set and validation:


private void saveEmployee() throws DBValidationException {
      DBEmployee dbEmployee = (DBEmployee) getFromSession("dbEmployee");
      dbEmployee.setValue("Name", getParString("Name"));
      dbEmployee.setValue("Surname", getParString("Surname "));
      dbEmployee.setValue("DepartmentNo", getParString("DepartmentNo"));
      dbEmployee.setValue("AddressNo", getParString("AddressNo"));
      dbEmployee.dbValidate();
}

In the code above, internal validator object is filled by setValue method.

3- Validation Execution: When user input is set then validation is executed by calling dbValidate method. In dbValidate method, standard validation is automatically done by framework without any code addition. If any extra validation is required then it can be added here thus it collects validation logic in one place.


public class DBEmployee extends DBObject{
...
public void dbValidate() throws DBValidationException{
      super.dbValidate();

      if("TestValue".equals(getValue("Name")))
                  setError("Name", "A Name Validation Error Message");

      if(!isValidated())
                  raiseValidationException();
      else
                  commitValidatedValues();
}
}

If validation error occurs, this method throws an exception. Then error is displayed at the right side of the input text. If no validation error occurs, then the values on the validator object are committed to persistent object. Object is now ready to make database save. In database save, persistent object’s field values are used.

This architecture may be plugged to any application environment. Key point is that general validation should be done by your application framework without any manual coding. For example, developer won’t find enough time to manually parsing a date field and user-friendly throwing an exception to the user. If it is skipped your application quality is deemed. A value holder object (Validator) is a necessity not redundancy. We use server-side validation by default but in some part of applications we used object metadata with a client-side validation JavaScript code to validate client-side. To reveal validation logic to client is not a good practice if it is not necessity.

Java and Database Naming Conventions

2009-06-29T01:45:00.000-07:00

There are many naming conventions used by programmers. Every programming team should build a naming standard. It boosts code readability and maintainability. It seems to be a minor issue when considering other standards but small things may influence big things.

To form our naming conventions we made use of both “Hungarian Notation” and “Camel Case Notation”. Our practice may not be appropriate for other teams but I am going to explain why some notations worked for us.

Before starting the details of naming notations, my first statement is that “There is no such thing in programming like naming”.

In an enterprise application, there are 2 kind contexts that you identify naming conventions; first is database and the other is Java programs (3rd one is JavaScript but we used same notation as Java).

Database Naming Conventions
1- Needless to say that you shouldn’t use abbreviations in columns names as long as DBMS maximum column name length is not exceeded.

2- Build a catalog of names for your column names. There are many synonymous words that may cause confusion. You should select only one word for one meaning in your project.

3- The only possible way for readable db entity names is to use “_” between words. DBMS and SQL are not case sensitive and we can’t use “Camel Case Notation” here. Add an underline character between the words of database, table, column, index, PK or FK names.

4- We use “P_” prefix for the name of multi PK columns and “F_” prefix for the name of FK column names. This eases to figure out the PK-FK columns without looking at the structure of tables from DBMS tools. For example when writing SQL statements, it is very handy. One drawback for that is you have to change column names if PK-FK columns change.

5- We use “Table Name” prefix only in FK column names to make it easy to understand the name of the FK table name. We strictly forbid using this prefix for a table’s own columns.

Let’s take a table named A and an ordinary column B and FK column C to table D.
B must not be named as A_B : A.A_B (A name is uselessly repeated when used with table name)
C should be named as D_C: A.D_C (We can figure out that C data is originated from D)

6- Don’t use plural names for table names. For example use EMPLOYEE instead of EMPLOYEES.

Java Naming Conventions
1- Use a standard package name for similar Java classes (Layers) in all your web applications(We say "module" referring Java web application). We use “db” for persistent classes, “report” for reporting classes, “mobile” for mobile classes.
Application package structure is as following:
[applicationname].[modulegroupname].[modulename].[classgroupname]
fasterp.finance.accountspayable.db

2- Use Camel Case Notation for class names, JSP page names, Java-JavaScript function and variable names. Since Java is case sensitive there is no need to use “_”.
Example: RuntimeResourceHandler

3- We used Hungarian Notation in following way. For member variables we used “m_” prefix. For some data types we used an additional prefix; s for String, b for Boolean, n for any kind of numeric type, db for persistent classes. We don’t use any data type prefix for other type of classes like objects, arrays etc keeping list shorter. Expanding this data type prefix list makes it harder to use and remember. We use data type prefixes for method parameter names but we don’t use it for method local variables.

In following example, it shows how it makes it easy to comprehend a variable membership:


public class DBEmployee {
 private int m_nNo;
 private String m_sName;
 private String m_sSurname;
 private DBDepartment m_dbDepartment;
 …
 public int dbFetch(int nNo) throws SQLException {
  int no = ...

/*when reading a code block below, it is very easy to understand where a variable belongs. Also without any mouse move,I get a general idea about its data type. The only exception is local variable “no” but its declaration should be close enough to see that information.*/

  m_nNo, nNo, no
 }
}

4- All methods that execute an SQL statement should be prefixed with a “db”. In this way, we can be aware of the cost associated calling a method.


public class DBEmployee {
 public void dbUpdatePosition (int nNewPosition) throws SQLException {
  ...
 }
 public void dbFetchByName(String sName) throws SQLException {
  ...
 }
 public double calculateSalary(){
  ...
 }
}

5- All JavaScript and Java AJAX method names should start with “rs”(Remote Scripting). This also gives a clue about the cost of calling this method.

There are many other guidelines that I don’t mention here. No need to repeat them again, just search the web for reference.

Web-based Master-Detail Grid Form Design

2009-06-27T12:16:00.000-07:00

Master-detail forms are a special kind of forms that join some tables into one page view. In this way, it is possible to see and edit some related records easily on the same page. Desktop applications have much more complicated master-detail form capabilities compared with ordinary web forms.

First problem of master-detail web forms is to display and edit detail tabs. Loading all detail tabs at once is not a neat solution. For this reason every time we click a detail tab we have to go to server. Another problem is grid component for detail editing. Grid component is very difficult to implement in web applications. The only possible way to implement it was to use AJAX technology and DOM manipulation. Inline-cell-editing is achieved with DOM manipulation and edited row data is sent to server with AJAX. Most of the web grid implementations don’t have master-detail form or server-side integration. We could design both and I’ll give you some conceptual tips.

Our implementation is as following:

1- Master (Header) may have some other tabs like detail tab. Since we display detail grid at the bottom of the page, there may not be enough space to put all master fields at the top section.

2- Some header fields may need to be disabled after header AJAX post. For example, if user is inserting a new record, we may lock (made un-editable) some certain header fields when user clicks the detail grid.

3- There may be multiple detail grids. Most of the web-based master-detail forms I see don’t have this feature. Designing these detail tabs as JSP pages is tricky. Header section may repeat in each JSP page. We solved this problem by using partial JSP pages.

4- Insert row; this is the most important element in this component. User adds new records with this input line. There is no edit link or new link to click (Most of the grids bring this extra click costs). It is ready to use. This insert line is also used for update row format.

5-6- Any type of elements can be used as cell type including popup input, checkbox, combobox etc. Reference codes can be selected by opening lookup popup. We can disable some columns if required.

Another important feature is to be able to update the row when row is clicked. If user completes update, then the row is converted to un-editable form. Just before this conversion the updated data is sent to server via AJAX. If no server-side error occurs, then it is converted to un-editable form.

1- Update row; user can make editable any row by clicking or by using keyboard up-down keys like Excel editing. Some grid implementations make all rows editable, but this is a heavy process if grid contains hundred of rows, assume that it may also contain many columns. One update row is also useful for differentiating the editable row.

2- Disabled cells; some can be disabled in some circumstances.

3- A very useful function when editing is auto-complete. Like in Excel, when user enters a text matching with current column values, it is automatically completed. Just by pressing enter key, cell is filled with that value.

We may have multiple detail tabs. When a detail tab is clicked, current detail tab row edit data is posted to server. If no error is received then user is navigated to other tab. To achieve this, we have to control tab clicks by blocking user clicks then re-running with JavaScript:

Master or header tabs may also be multiple. When user clicks a master tab, then all detail grids are closed and only the clicked tab is open. In the example below, we used secondary tab for detail of master record since all fields may not be filled on the same page with detail grids.

Web Application Input Form: Design Guidelines

2009-06-18T23:50:00.000-07:00

Before designing our UI components, we investigated many examples. We researched the samples of many UI design books for a better input form design. Many UI books are somewhat misleading for application developers because they focus on web site usability. If you are a web developer most of the things I mention will not be new for you but I hope this post may give some new clues.

Input forms are the primary elements of a web application. A standard application has 3 standard application types; parameters, transactions (data entry), and reporting. Transactions are generally occurs via data entry screens. Data entry may occur in many forms like user data entry, data importing, device input, or pulling data. In a typical application, user data entry builds the most of data. Input forms are the medium for that task. Because of that reality, input forms are very important in web application design.

In web applications, we have some drawbacks compared with standard desktop applications. For instance, UI event handling is not as easy as it is in desktop applications. Although, UI events can be executed via AJAX server calls, that kind of development is still expensive.

Let’s see the functions of a standard input form:
1- Providing input controls for data entry; input texts, check boxes, combo boxes, calendar popup, reference record lookup popup, list boxes, hidden fields etc.

2- Presenting new or existing data; user input fields should have formatting ability in both client-side and server-side. For example, numeric field should format the numbers same as the server-side Java formatting. Required fields should be distinguishable.

3- Execution of some codes upon user UI events. For example we may lock an input text field according to some events. Or we may pull some data via an AJAX call.

4- Posting and validation of data; user submits form. Client or server-side validation should be done. Validation or transaction errors should be user-friendly presented back to user. Server objects (EJB-DAO-POJO-DTO?) associated with form input should have appropriate states on failures and retry.

Here is a typical input form snapshot:

1. Don’t forget to put form title. It should say if this is a new entry or update. We placed it here. Other places of the page may be used. One important design lesson we learned is that use the same JSP page for insert and update of record. Otherwise there will be many pages that mean code duplication.

2. A master record may have relational detail records. Details are placed in other tabs. This example is not in one-page master-detail form, we split them. Master-detail form design is not easy in web applications compared with desktop applications, I’ll mention it later.

3. You may need some summary reports quickly (On-record reporting). These pages serve this purpose. When user clicks these tabs, it is easily reported.

4. Record navigator enables us to iterate the records. This is a common control in Windows forms. Our navigator doesn’t have delete-add-update-save buttons because these commands are located in other places (see my Web List Component post).

5. Reference lookup input controls are the most difficult control to develop. Why? It has many functions. First question how do you open lookup popup? In new window or dialog control or layer? How to return selection to page? How do you execute some codes upon selection? After completing popup dialog problem, user comes and says “I changed reference data but associated text at the bottom didn’t change!”. That text should be fetched via AJAX call both on data changes and page load with the same mechanism.

6. Numbers should be right-aligned. We used coloring in required fields, as you can see its background color is different than other input texts. At first, we had used ”*” but coloring is nicer.

7. One important usability rule is that you should separate data and label text. We used “Bold” font for data. Some uses colors but I think this is more elegant.

8. This is also an important feature that you should highlight the cursor position on page. Otherwise user has to look around to find it. We use an attractive background color for focused-active elements.

9. Don’t forget to format numbers, money, dates, times and timestamps. Some custom data type may need also special formatting on the screen. If it is editable, we have to format data both server-side and client-side. Users may change data and after focus lost we should format again (Or format on the fly but users may go crazy if it is not coded appropriately).

10. Finally, it is time to submit data to start a new journey, server-side processing. Command buttons should be big enough to click and grouped within the same region of page. You may do some client validation or leave that job for server-side. After validation errors, you may report it field-level like below.

Our forms have following extra functions:
1- User can navigate active input forms by entering enter key or navigation (Up-Down) keys.
2- User’s previous data entry is saved and if user press F3 key on any input text, previous entry is restored there.
3- Focused input element is remembered in the next visit.
4- We have bulk-update forms that provide users update many records at the same time.
5- Printing is possible within form pages.

Web Tab Component: Mapping Database Tables Into UI Tabs

2009-06-12T06:07:00.000-07:00

In this second installment of UI posts, I will show the “Tab Component” of our UI library. Tab components are very important part of rich interfaces. Tabs provide us partitioning the interface and the elements we want to display. Logical division in UI means better user-friendliness. It empowers easy access. Nothing more to say, in every application we use there is a tabbing function.

Yes, we need a tab component in web applications but how can model it so that we achieve minimum server roundtrip, minimum development cost with better user-friendliness. One important design difference among tab component implementations is to make tabs whether server-side or client-side or mixture of both using AJAX. Server-side tabs always go to server when tab title is clicked. Other is client-side solution which all tabs are loaded on page load at the same time. Second option’s most important disadvantage is loading all tabs content at once. This results loading all tabs although user may not click and want to see some tab content (Requires many database selects). AJAX may cure this problem and we may load tabs on click or simply use server-side option.

Mapping of database tables to web tabs require following steps:
1- Your tab component should also be mapped to database access objects for lazily loading the content of tabs. Tabs should have a similar structure with database objects and database tables or vice versa.

2- Tab navigation requires a special controller mechanism (Controller servlets). You should model the navigation mechanism between tabs. Some controls are really hard to design like when edit forms should be saved and committed to database, if multi-level tabs are used it gets much more complicated.

3- Tabs should be designed within JSP pages. Tabs may contain edit forms or lists or some kind of reports. We modeled JSP pages in such a way that every tab is a JSP page. Using tabs in a master-detail manner requires much more effort and I will mention master-detail component later.

As seen from here, designing a server-side tab component is not easy. We have to design and model database access objects, controller servlets, tab JSPs in parallel. I think client-side tabs are very easy to design but they are not as rich as server-side ones. I analyzed many JavaScript libraries that have tab components but I think they are not ready for enterprise applications since their server-side processing part is missing. This is the main problem of UI components of JavaScript libraries.

Our tab component has following features:
1- Any number of tabs can be used in one level. Tab may contain edit form, lists, report or summary pages etc.
2- Any level of tabs can be used. Adding a second level and a third level of tabs is possible (No limit, as deep as your database table levels). In the example below, you can see the second level tabs.
3- Tab pages have a record navigator (At the right top side of the page). In any tab, record navigator can enable us to traverse the records in that tab. Navigator also can be used in sub-tabs.
4- User can navigate and click all tabs at once. If user views a page in a second level tab and it contains an edit form, if user clicks a link of another tab then form is automatically submitted.
5- User can save all tab modifications at once with the “Save” button which exists in first level of tabs. Any changes in tabs are held in memory. Database access objects should support memory operations appropriately like validation, formatting etc.

Here is the mapping illustration:

Header tab page:

Detail tab page:

Second-level tabs:

Web List Component: Beyond Simple HTML Table

2009-06-04T23:57:00.000-07:00

Today, I am going to share the list component we use in our ERP applications. Web applications UI is a very important topic for web developers. UI design is an activity that mixes both functional design and graphical (artistic) design. Function richness, usability, comfort and look&feel are very important. Mixing these aspects into components determines your GUI success.

Web applications should have following components; List, Form, Grid, Tab, Popup, Calendar, Alerts, Navigation Bar, Tool Bar, Command Buttons etc. Graphically or HTMLly designing these components are not enough. We have to carefully attach functions while considering usability. When attaching functions, there is always a decision to be taken for function design; where do we provide this function, in server-side or in client-side or with Ajax?

Here is the snapshot of our sample list:

Our list is mix of a table and a layer element. In a web application, a list component is a must. We added the find&edit responsibility to the main lists. For this reason, our lists are an entry point for editing records. Let’s dig into details:

1-3 Command Icons: These icons do the functions in lists. Command icons from left to right are “Delete”, “Add”, “Copy”, “Update”, “Search”, “Documents”, “Workflows”, “Audit Trail”, “Print”, “Filter” and “Customize”. When user clicks these icons, it triggers server-side commands and execution. Every command icon is associated with a hot-key so that users can easily invoke this icon clicks without moving mouse. As a UI design note, every function on page can be used by using only keyboard. Some users prefer keyboard for its speed compared with mouse.

2 Paging: Paging is very important when the number of records in tables reach thousands or millions. Loading and displaying all records in a browser is not acceptable so we have to skip some records. The neat way of doing this is paging. (Some applications load records or pages when scrolling the list but I think this is an ugly function design) By paging, we both minimize server and client processing with the cost of reading page counts. Many database systems support paged fetch. We use a paging object in a list object for paging functions. “Total” is the number of records that a user can see, “Page No” is the number of page I am viewing, “Record” is the number of records that I want to see in the page(The default record count we are using for lists is 10). If the number of rows displayed on page overflows the height of the list, then a list scroll bar emerges instead of page scroll bar. One important design note, when user scroll rows you have to keep the command icons and column headers fixed at the top. (Many lists I analyzed don’t have fixed header feature but users would want it. We achieved fixed column header feature by using CSS expressions).

4 Fast Filter: Fast filter is my favorite feature in our lists. We added a filter line to filters instead of some search inputs at top of the page. In this way, users are not limited with the filters you added. Users can use every column for filtering. This is a very rich function. For lookup values, we add lookup icon for opening popup and user can select reference values. In fast filter list we support following filters: "Equal", "NotEqual", "Smaller", "SmallerEqual", "Bigger", "BiggerEqual", "Starting", "Including", "Ending", "Null", "NotNull", "NotStarting", "NotIncluding", "NotEnding", "Covering", "NotCovering", "ExistingInList", "NotExistingInList", "Between", "NotBetween".

Sample Filters:
Bigger: >145
Including: *2009*
NotExistingInList: !03,04,05
Between: 1000<>5000

Users can add multiple filters by opening “Filter” page (With Filter command icon, on the right top of the list).

5 Selection: Single or multi-selection is possible. Commands may require a row selection, for example “Copy” command. For single selection in our lists, user only clicks the row and row is highlighted. Clicking on checkbox for single row selection is not user-friendly. Users usually use these columns checkboxes for multi-selection. We may use CTRL keys for multi-selection but we thought that this is more usable. For example sometimes, some rows can’t be selected. With this column it is very understandable to show disabled rows.

6 Column Headers: We add a function to the column headers, “Sorting”. When user clicks a header, the list is sorted according to this column. If user clicks again to the sorted column, sort order is reversed.

7 Selection Row: When user clicks a list row it is highlighted. User can also select row by using keyboard navigation keys (Up-Down). One important function is that, when user leaves this page and then returns again this page, user’s previous selection is remembered and selected. This is very useful when user list has many rows and scroll required. By double-clicking a row, users is taken to edit page. Double-click is also useful, otherwise user had to select the row then click the edit icon.

Java Web Application Version Management

2009-05-31T11:21:00.000-07:00

Change is inevitable. So version is a result of this reality. Every application will have a version history from alpha to beta, to production, to maintenance versions. Version management requirements of software teams may vary but principles and base process are same. As a software team, we had to define our processes so that the customers would get new versions with minimum pain. We aimed to cope with the reality of “Upgrade is painful”. Defining process is not enough, you should educate your staff and provide infrastructure for better version management.

Tools for version management is still weak. Vendors are still trying to define the borders of applications (Java Module System (inactive), JSR 277). (Web application and enterprise application separation has some problems for us. When we were trying to upgrade to newer Application Server, we noticed that you can’t share user sessions among web applications unless vendor’s proprietary web configuration parameters. For enterprise applications, it is totally impossible. We are about to configure our large application to bypass this limitation.) The reason of poor version management tools may be the result of vast deployment configuration environment. To support many application server, database server etc. is not easy. The problems or variety in middleware frameworks of Java may exhaust the time required to develop such tools. Anyway we should already have had these tools in our toolbox.

As an ERP team, we built the process and the tools for effective web application version management for better serving our customers. I’ll mention about these. ERP applications have a very large code base with many database tables. To keep the requirement change in minimum is almost impossible. Every project may lead to new tables, new columns, and new business logic.

Our version management infrastructure is as following:
1- Version Management Application: We needed such a tool to bundle web application for fast deployment. This tool prevents manual database scripts and automates this process. Every database change is defined in its repository and related versions may include these changes. Then using this tool we can package web applications as a zip file (bundle of EARs). This file is installed again with this tool in target system. This tool is just like Windows’ Add/Remove Programs. One can see what versions of applications are installed in his system.

2- Development and Version Servers (Staging Servers): At the beginning of our ERP development, we heard about staging servers but we didn’t know why we needed it (We had only development server). As the time passed and we installed our application to the customer systems then we understood that we should have system that has the same configuration with our customer systems. Then, we configured a version server in addition to development server. As our version within customers systems increased, we needed new version servers. At the end we had 3 version servers. Otherwise you can’t provide local version support for your customer. Supporting more version of your applications mean more servers and more development cost. A fix to older version may bring applying it to the newer versions thus it adds a burden on your developers, testers and release team.

We have version management process like below:
1- Minor (Fix) Versions: Any bug fix causes a new version. Accumulating these fixes into a single version is a good practice. Sometimes, some fixes may be very urgent and needs a fast version release period. A fix request is entered our bug system, then development manager inspects this problem then if he thinks that it is related with code then approve it for developer’s attention. Approved bug enters to the developer’s task list. Then the developer fixes the problem locally. Then he/she uploads his version into version server. Then testers do tests. If they confirms that the bug is fixed then they notifies the release manager. Release manager packages the web application then notifies support manager. Support manager announces bug fix for system managers. System managers get the fix and installs into their systems.

2- Major Versions: We have certain major version periods like one month, 2-months. ERP vendors’ major version period is longer such as 1 year, 2 year. In some projects, these periods may be changed. Sometimes, you need to release frequently so that new developments reflected to the target system earlier for customer tests. Development team does the required changes in the applications in the determined development timeframe (4 weeks). Then test period begins, testers make the tests during the test timeframe (1 week). After the bugs are closed by the developers (if satisfactory enough for release) then release manager packages the web applications from the development server. Then this version is installed to the target version server. We package from development server and install it into version server, because installation is an issue to test. Then testers test the version server for any installation problem. If no problem is found, that means we are ready to announce and release the version.

Web applications versions have following common characteristics with other type of applications:
1- Database Change Management: Every database-driven application has many database changes accompanying versions. To automate these changes are very important, because manual process is error-prone.

2- Application Code Changes: Business logic and application may change. Changing software artifacts should be packaged and installed to the target system without any manual process. Un-installation should be supported. Software pieces versions should be maintained otherwise we have no clue which version of software has the problem in the future.

Web applications versions have following distinctive characteristics:
1- Application should be packaged according to web deployment model. Currently Java EE Application Servers minimum deployable units are enterprise applications (EAR) that may include multiple web applications (WAR).

2- Enterprise applications can be deployed and restarted without server shutdown. This is a very important feature for system availability. An application version installation doesn’t bother the rest of the other applications’ users. After web application restart, you should handle session objects appropriately. After restart of web application, classloaders renewed and your users may get ClassCastException. Objects of same class loaded with different classloaders would cause ClassCastException.

3- Web application dependency is another aspect to consider. Some web applications may have some dependencies to other libraries or web applications. Classpath settings are important. Changing these settings may bring new ClassLoader problems. In this respect, knowing the differences between system classloaders and application classloaders are important.

Primary Element of Software Project Delivery: Team Capacity

2009-05-24T10:47:00.001-07:00

To a degree, there is nothing impossible in software programs. Everything is possible; complex and large scale software systems can be written like OSs, DBMSs, Space Shuttle programs etc. This possibility is always a disadvantage for software project teams. Because of such wide possibilities of software project output, people tend to widen scope and the project gets closer to failure. So, the critical question is that what is the limit that a project team can tackle with their current capacity?

Every project team has a varying degree of capacity. Every team is unique since its members are unique. Team capacity is different then sum of members’ capacity. Sometimes adding new members to the team doesn’t increase capacity. There is a common saying that one programmer may equal to 20 programmers in some cases. Capacity may change according to learning skill, problem-solving skill, design skill and management skill. Capacity is also related with how much time you would devote to this project. Are your ready overtimes? Are you ready weekend works? Are you ready for stressful milestones?

Team capacity has 3 levels in point of finishing software projects successfully:
1- Project Manager’s Capacity: This is the most important capacity factor since he/she determines the project path. First capacity check should be done in that level when planning projects. Project manager should ask himself if he/she is capable of taking this project.

2- Team Leader’s Capacity: Team leaders also another management level that has large scale factor on projects. Architects, system designers are generally assigned as team leaders. Their personal capacity is also determinant factor of project success. They are the key personnel and should be kept until project delivery. Key personnel turnover may be very adverse affect on total capacity. Filling the empty position is a very big headache. Morale decrease of the rest of the team is another capacity problem.

3- Team Member’s Capacity: Every member shows different capacity behavior. Some of them present all his/her capacity which results with a good performance. Some of them may work under capacity and to figure out this is very hard. What you expect is honest behavior, if you become honest people will be so. Although some team members use their full capacity, this may not result sound results. However, some skillful members may change destiny of projects in some hazardous and critical situations.

Capacity has following interesting aspects:

To increase capacity, we should allocate more time to project. This may be overtime or weekend work. Sometimes deadlines may force us to expand work-times. Overtime doesn’t always result capacity increase. Sometimes team may be tired because of heavy work conditions. I think, best practice is to keep overtime period as short as possible.

Capacity may be raised by employing skilful members. Skill means fast and strong problem solving. Skill measurement is a very open topic to discuss but we should see it in results, in problem solving periods, fast and sound output.

Skill may be gained by education and capacity may be raised. Learning and skill gaining is another skill and varies among team members. We may invest on education of long-standing members.

One can increase his/her capacity by investing himself. That means we should read more, investigate more for better solving problems, designing systems. One interesting sentence I’ve heard some time ago is that “One can earn money for his life between 9 am and 6 pm, one can earn something to change his life between 6 pm and 9 pm.” We should learn more in addition to what we learn on our daily job.

Although, team capacity is much more than the sum of team members’ personal capacity, if team members don’t find an environment to show their capacity, the result may be opposite. Harmony of team members is very important. To assign true people on true tasks is important.

Web Application Design For Easy Problem Diagnosis

2009-05-15T13:17:00.000-07:00

Every software application will have some bugs. Every bug has a lifecycle. Bug reports are collected from the customer sites then fixes are released and finally bugs are closed. Number of critical bugs and bug fix elapse time are some of the quality measures for support process. Critical bugs have a tremendous adverse effect on users’ software acceptance. Its fix elapse time is very important in this respect.

How could we design application so that we can immediately troubleshoot the critical bugs as soon as possible? There are many techniques for problem diagnosis used in many big applications. Doctor is a famous name used for diagnostic tools for instance Dr.Watson, Dr. Admin. System event logs are another common technique. In Java, there are many tools including heap analyzers, thread inspectors, javacore files, profilers etc.

In problem recovery context, there are 3 types of people:
1- End User: If error is related with user process, user should be informed for corrective action if possible. User-friendly error messages are very important. It should be in non-technical language. For example, database errors should be converted to user-friendly messages, of course not all of them. We translated some common database errors for user understanding. For instance, default DBMS’ referential constraint errors is not user-friendly.

2- System Administrator: For system administrators, errors should be logged in a descriptive format. Sometimes, developers may catch exception and forget to throw it again. In these circumstances, to find problem cause is a guessing game. When logging exception, additional environmental information should be included such as current database, current window, current user etc.

3- Support Team: When a problem occurs and it is not resolved by the previous users, eventually it reaches development or support team. Support team should have the tools for system snapshot to get better clues. Support team should have thorough knowledge about not only application but also system, application server and database server. Some problems may only be resolved with the help of database snapshot tools.

We use some techniques for faster problem recovery for the mentioned people:
1- Thread Inspectors: In our admin application, thread inspection tool is marvelous tool for problem determination. This technique is simple and yet powerful. You just register running servlet to a running register list until it finishes its job. When the servlet finishes execution it is removed form running list. JSP pages also should be added to this list since sometimes some heavy tasks may be run by mistake (An infinite loop condition).

2- Connection Inspectors: Because of using our own connection cache, we could develop and use connection inspection facility easily. It is very useful to determine which database object is executing or waiting locks. In this way, we do not need to get a JVM heap snapshot from running system. Java tools for this kind of purposes may slow down system or force us to shut down system (To get a core file we have to kill java process).

3- Session Size Calculator: When Java memory gets larger and we deplete system memory resource a lot, that means we hit a memory leak problem. A simple page to show memory levels both free and available may give the information of GC activity. If free memory instantly consumed then it shows a very large object allocation. If free space is not released, then this is a bigger problem. In these circumstances, we need to see the amount of user session in memory. We could calculate user session size, object by object. In this way, we can figure out which object is oversized (Objects must be Serializable).

4- Log Files: Our applications have 2 types of exceptions, one kind is logged and one is not. Some validation errors are not needed to log. Some system errors are logged in a proper format. Sometimes we may print stack trace with error message. Log entries are classified with error, warning and information.

5- Black Box: This is the most interesting technique. Javacore files include thread stacks but sometimes it is difficult to get enough information from these files. We developed a soft-black box, inspiring aircrafts' black boxes, which is a simple file contains running process list with details like who runs, when the thread started, elapsed time etc. An independent monitor thread continuously updates this file. It is very useful in system lock conditions too since no application GUI can be reached but this file can be easily accessed and read.

Server Topologies For Enterprise Web Application Deployments

2009-05-09T13:35:00.000-07:00

When we were trying to figure out the deployment model of our large enterprise applications, the ERP system, I was unsure if an overall server consolidation was a good decision. At the beginning, we deployed our applications to one server. As the group companies began to use our system, we faced the scalability challenge. We had architected our system so that multiple companies could use the same system within one installation. Programmatic support for multiple companies were not enough, the critical point was its deployment scalability.

Eventually, we had to implement new ERP system within the group companies located other cities. The question was which system would they use, new local server in their own city or remote server in our city. We decided to use one server per city, the distributed deployment model. Server consolidation was a very popular at that time but our network WAN reliability was not satisfactory for users. A WAN connection failure means business disruption thus means cost, 1 day unavailability may cost one server price. Of course, the cost of buying and setting up a new server is not the only price but administering it is another cost. New server requires at least one administer so that enterprise applications could run smoothly.

Although web applications can be accessed from anywhere, scalability and availability should be taken into consideration when building server topology. We even used multiple servers for one city because of heavy processing in one company to split it.

Let’s summarize top-level server topology factors in terms of distribution or consolidation:
1- Application architecture to handle the transaction amount of overall system. If application is not able for handling big load, global consolidation would not be possible.

2- Server power to handle global consolidation. If one decides to build such a system, he/she should build a server system which would include many powerful servers including database server, application server, web server etc.

3- Available IT stuff is another factor for server topology. If enough personnel exist in that location, we may add a new server system.

4- Application transaction amount and availability requirements of user companies are another factor that determines if we must add a new server to the system. Local servers would be usually much powerful and much available. Locality has its own powerful aspects. In our case, some companies have many transactions than the other ones so their system requirement is over average (We measured that daily transaction count of one company is equal to transaction count of 10 companies).

Local server topology was another deployment issue to consider. How could we achieve a highly-available system? How many physical servers would we need in one server system? Which of the scalability model to use, horizontal scalability or vertical scalability? After making many research about these topics, we concluded that we’d better use one physical server for system which requires lower processing, 2 physical servers for system which requires higher processing.

In our one-physical server configuration, the server hosts all server applications; application server, web server, database server. In 2-physical server configuration, we deployed application-web server on one physical server and database server on the other. Database server is the major performance element for business applications. Database servers require both disk and CPU power a lot whereas application and web server requires CPU power and network bandwidth.

When determining local server topology following factors are important:
1- Application deployment architecture determines if horizontal or vertical scalability is possible. If application supports cluster then we can utilize vertical scalability. If we can use database cluster or partition support, then database can be configured for different scalability methods.

2- Simpler topology means less administration cost. If possible only one physical server should be used with one application server and one database having no cluster.

3- Server range is important for different processing requirements. Some companies may have many transactions per day compared with others. Some companies may have many system users whereas some small companies would have a few. For bigger companies, server should be high-range. High-range servers have many powerful hardware capabilities compared with entry-level servers. For instance, high range servers may have many CPU’s that make parallel processing possible.

Part(Item) Numbering System

2009-05-02T05:56:00.000-07:00

In ERP systems, parts are the most important atomic unit of system. Every module is linked to item(part) master table. Every part is given a number to uniquely identify it. During production phases, materials may be in many forms like raw material, semi-finished product, or finished product. As the production phases continue, raw materials turn into sub-assemblies, sub-assemblies forms assemblies and finally assemblies are combined to build final product.

During each phase of the production, we need to describe processed items to uniquely separate from others. When we need to stock items, it is necessary go give part numbers. Sometimes, some intermediate forms directly enter the other production phase and eliminates to give part number. Reducing part numbers provides better and error-free processing decreasing associated costs, but it is generally not possible. Companies may have thousands and even millions of part numbers.

There are 2 kinds of measure to figure out whether to give a new part number or not:

Interchangeability: If the item we are about to give a new part number is not equivalent and can be replaced with the already defined one, then we should give a new part number.

Form-Fit-Function: If size of the item changes or if its connection points change or a new function is added then we should give a new part number.

Part numbering system changes across companies. Every company has a different numbering policy. Every numbering mechanism has some advantages and disadvantages. Part numbers generally can’t be shared among companies residing on the same supply-chain. For this reason, unique product numbering systems among companies are very hard to build. One company detail level and perspective for an item is different than others so the same is true for part number.

There are 3 basic part numbering methods:

1- Non-significant Numbers: Part number is just a sequential number and has no logical meaning within digits. Total part number length is shorter than other methods but the possibility of duplicate code is higher. It is hard to remember part numbers for people but easy to use in barcode systems.

2- All-significant Numbers: Every digit of part code is intelligent. Part numbers are generated from some category schemas (Classification is another system that may be custom or same with other systems like Opitz, Brisch, Gildemeister, Pittler, Zafo, Vuoso, PERA,or PGM systems). It is hard to maintain category schema. Category schema change is impossible when its use begins, only addition is possible. This technique requires delicate categorization work at the initial step of system setup. It is easy to remember codes, but codes are longer then ordinary numbers. Querying codes are easy but depending only code category may be wrong since in some cases, some duplicate categories may be possible.

3- Semi-significant Numbers: This method is aimed to use advantages of above methods. First digits are intelligent and the rest is an ordinary number which is sequential or manually given. Some separators may be used to separate significant and non-significant part such as dot.

Sophisticated products contain many part numbers and maintaining a category system would be very hard. In semi-significant numbering systems, one company should use as much category-subcategory as it is able to maintain.

In BOM numbers or product numbers, these systems can be used but BOM numbers are generally shorter than part numbers.

What Drives Software Development?

2009-04-25T12:12:00.000-07:00

There are many drivers and factors in software development. Culture and focus of software organization determine key factors. These factors constitute the flavor of the products and projects. There are many trends in software world that affects the way we develop software. Any driver that changes the way we develop software should be carefully evaluated.

Here is the list of some x-driven development styles:
1- Value-Driven Development: At its core most important thing is value that we provide with our software to the customer. Software economics are always considered as a top priority check. In any software we should ask ourselves the value proposition of our solution. Is it worth the effort for the solution for the given problem? We see many bloated and non-value projects that result with failure at the end, this shows the importance of this perspective.

2- Customer-Driven Development: Customer is the central focus point. Customer involves in every phase of the projects and is given top priority. Customer feedback and communication enable end product meet the expectations better. Customer satisfaction is high since product is customer-oriented. I observe that there is a very interesting relation between customer and supplier. Sometimes customer is king, sometimes supplier is king. I favor balanced relations, otherwise balance will be lost and customer may incorrectly manage the development process then project may fail.

3- Domain-Driven Development: Domain concept is used to clear the specification of software design. Domain knowledge is used as a common vocabulary between stakeholders. Domain knowledge is really a very delicate factor for software development. Many projects failures are associated with bad requirements analysis that lacks domain emphasis.

4- Test-Driven Development: Test is the first activity of development. In this approach, test is a high priority activity whereas in many software projects it is abandoned because of tight project schedule. It is a good practice, but I think writing test-cases depends on projects and contexts. Some critical software needs many safeguards, it is useful in these cases. On the other hand, some projects may have many changing and urgent requirements and developer would not be able to write and maintain test-cases at the same time. It is a quality issue. If we have more time, we can find more chance to write tests and increase quality.

5- Behavior-Driven Development: This aims to close the communication link between developers and business users. It uses some of the best practices of existing software engineering methods.

6- Feature-Driven Development: Basically, it enables us break down the requirements to the smallest pieces, which are features. By dividing functions, systems development could be simplified.

7- Model-Driven Development: Model is the core of the software. If it is used in every phase of the software, development complexity significantly reduced. Model use can simplify development, communication, documentation etc.

As a software team, I think it is more appropriate to make use of the best practices of these approaches. Every method is developed within some problem context and there is no single answer for development problem. We may also have some best practices that depend on our own development experience (Mr. Foo’s Experience-Driven Development). Every software development project may have different priorities and contexts but we have to look at the picture from different perspectives to build better processes and to reach better results.

Best Practices For Software Project Management

2009-04-14T11:50:00.000-07:00

When I accepted a multi-year, multi-million dollar software project, I wasn’t aware of the difficulty of such a software project management. In my first PM role, I was assigned a very crucial software project. It took just 7 years to reach the last milestone. My initial estimation for project duration was very different than the result. Looking back my initial project plans, I admit that we had made really a “Death March” project plan. We had to revise many times the original project plan to handle the changing requirements and realities. Every software project is not lucky like us since top-management supported us up to now. Some long-running projects are really hard to cancel for both stakeholders and project team. Big ERP projects are such kind of projects. During this period of time, I took many kind responsibilities (in addition to PM) ranging from developer to software architect, to analyst. Having a software engineer background, I had to fill the gap, the knowledge of project management.

I want to share some of the experiences for PM candidates. My first statement will be “There is no such thing in PM role like experience”. Yes, if this is your first PM role, that would be very challenging since technical person has a very tremendous gap for PM role. This gap is not easily filled by reading some books and best practices articles. This learning doesn’t resemble to learning a new programming language. To be successful in first steps, I recommend taking the responsibility of small projects instead of risky ones. Your employer may not know the risks of the project and “Yes, I can” may be your first project failure prelude.

Another important point is that every software project is unique or has some unique aspects. If this is the first case the team is writing that kind of software, the associated risk would be very high. I think this is the major factor that software projects fail much more than other type of projects for instance construction projects. We‘ve heard many ratio for project failures like 90%. The precise ratio may not be true, but most of projects would fail to reach the initial budget, time, scope and quality targets. This is why managing a project is like riding wild horse, PM is the rodeo. There would be many unexpected exception thrown in your project. You may not be ready for them (adding your imaginary programmatic catch clause).

There are so many best practices around the net that results lots of confusion for beginners. Those items and my items may not apply your project but read and decide if it is appropriate for your project:

1. “Software Project Management” is different than standard management: Yes, standard management is not enough for PM tasks. This may be the major pitfall for many managers. Management is to manage the people so that the execution of process continues. Project management is about changing, transforming or building something with people. Project management requires much more leadership than standard management.

2. Avoid detailed plans in initial phases: I assure that you will rewrite and revise it again and again. In ERP world, this plan is known as “Master Production Plan”. Keep it simple and take the first steps as quick as possible. Instead of waterfall model, iterative model is much appropriate to see the problems earlier and fail early (to recover). If possible use prototyping that would shorten iteration period.

3. “Software Estimation” is most problematic part of planning: Just write down your plan and multiply time by 2. Accurately estimating time may be only possible to be pessimistic during planning. There are many techniques to make precise predictions. I think the best estimation is possible if we’ve already done another similar project. There are thousands of factors that may alter deadlines.

4. Do requirement engineering in analysis phase: Requirements analysis is the most important part of projects since it determines the rest of the project. Wrong analysis may lead to catastrophic results. Analysis should be checked, reviewed sufficiently. Otherwise, whole software may be rewritten again.

5. Avoid methodologies if you don’t use in previous projects: In first days of my assignment, I had a very hard time trying to figure out the relations and differences of management, projects, methodologies, software engineering. Trying a methodology may require a very deep investigation for your project thus takes time (Time is the primary resource in PM). Team size, employee profile, project characteristics etc. may change the methodology that will be used. In those days, RUP was popular and everybody was asking why we didn’t use that methodology. We tried, but it was really heavy-weight process that is meaningful with UML-based tools. My observation was that it is much more appropriate for teams larger than 100. Nowadays, we see that it is replaced by Agile processes. XP was another methodology that unfortunately used in many teams without any feasibility test. I think “Software Engineering” is a very sound base for many methodology requirements. One day you may end up a new methodology, your unique methodology.

6. Determine roles appropriately: “Separation of concerns (profession)” is a very sound team organization principle. The key question is how to divide the roles. Role division should be fair. Team members should have enough depth of responsibility for tasks. We have 2 divisions; analysts and programmers. That was enough in our case. If roles are not divided correctly, tasks would be hard to assign, team would not serve efficiently. If needed new roles may be introduced. An employee may have multiple roles.

7. Handle problems timely: Any kind of problem is important. If it is a personal or technique problem, it should be fixed timely while taking care of priorities. Long-awaited problems may bother employees up to resigning. Team organization and software organization should be built so that it doesn’t generate problems a lot.

8. Good projects (products) are result of good team members: Stuffing is also very important. If you plan a very sophisticated product, you can’t reach that goal with low-skilled and low-educated people. Every team has key members. Don’t lose them until project delivery. Stuff turnover may be deadly for project in some phases.

9. Motivation and morale should be high: Any factor effecting to lose motivation causes low-performance. That eventually reflects to overall project progress. There are many motivation techniques to use. Making a questionnaire to staff gives very good clues about team motivation factors. It also prevents wrong assumptions about people. In my observation, best motivation is giving the tasks that employee is eager to take. Another motivation is to believe in the project. First person in team to believe the project is PM. PM should make team believe in project.

10. Backup personnel are best solution for turnover problem: Software development is a mental process and development is dependent to team members. It is a very good practice to keep 2 employees for similar tasks. Sharing some tasks among 2 members may yield backup. Backup also would be very useful for vacation and illness circumstances.

11. Eliminate risk if possible: Project is itself a risk. If possible we should try to leave risky parts. Taking too much risk would harm to time, budget, scope or quality of project. It is a good behavior to say “No” in some cases.

12. Add a next version (Version2) notion: Every feature request may not be completed in first phase of project. Project may be divided into phases. We can simply add a version 2 for later attention.

There are many other best practices that can fill a book. If you are assigned a project, my first recommendation is to take a PMI’s PMP (Project Management Professional) certification or attend courses for this. This is a much better way of learning.

Recommended PM books:
Quality Software Project Management

Preparing For The Project Management Professional (PMP) Certification Exam

Corporate Software Project Management

IT Project Management: On Track from Start to Finish

Our Messaging Framework Architecture

2009-04-09T01:28:00.001-07:00

During the initial development of our ERP software, we had searched messaging solutions for our messaging requirements. We evaluated JMS providers’ packages. Shortly at the end, there was no solution fitting our case. At that time, some of the developers were criticizing JMS with the argument that messaging vendors shaped the specification in order to be complaint with their current products. Adding a new dependency (3rd party JMS provider) would increase overall cost of our solution by price and by administration. We decided to write a new messaging framework that highly integrates with our applications. We aimed to make message-based programming easy and embedded to our platform. Development cost of such a framework would not be feasible for many projects but we had time and resource at that time. On the other hand, every enterprise application would need a message-oriented middleware (MOM) for enterprise application integration (EAI) if multi-site (multi-server) deployment occurs.

I summarize the benefits that we wanted to provide with this decision (We didn’t implement JMS but our API is simpler than JMS API, JMS implementation maybe in the future):

1. High-integration to Applications: Message-driven beans (MDB) enable EJBs to utilize JMS messaging. When we evaluated, JMS providers’ solutions were hard to bind into development and deployment environment. We aimed high-integration with our applications and database objects similar to EJB-MDB model.

2. No Administration Overhead: One of the most important problems of JMS providers was administration mechanism. We couldn’t find an embedded messaging product. They were requiring installation and management. When we try to install our application on Unix-based servers, there was no support by these providers (Oops, where is my MOM?). Database support is another hindrance.

3. Both Synchronous and Asynchronous Communication: RMI was strong in synchronous communication and JMS was strong in asynchronous communication, we had tried to combine these both models. In this way, we managed to send a message to send synchronously or asynchronously.

4. Easy User-User Groups (Roles) Usage as a Message Consumer: When we inspected JMS API, the only way to make a user a message consumer is to create a new queue naming same with user name. However that mustn’t be such that hard (We had to create a new queue in JMS providers’ admin console). User groups are another type of message consumer that requires similar procedure to be a message consumer. We wanted that every system-generated message would flow in this messaging middleware reaching users, user groups or program without manual definition.

5. Workflow Integration: We benefit from our messaging framework as workflow infrastructure backbone. Users can register workflow actions and make system produce messages to them.

6. Replication: One of the most important usages of framework is in replication mechanism. We didn’t use database replication support since product-dependent. Instead, we developed a replication mechanism based on MOM and integrated with applications. Application database events are listened and if replication is defined for this operation. Then by using MOM’s synchronous transacted one-to-many message, replication is done.

7. Electronic Data Interchange (EDI): Application’s transactional data transfer among systems is a very tricky problem. In business world, there is a huge amount of document exchange among companies-departments (B2B). If systems can talk each other, paper transfer and data re-entry can be minimized and some errors may be prevented. Systems can talk each other with EDI mechanism. MOM can transfer electronic documents to other party and get response. Users should use EDI mechanism within application GUI easily by selecting for instance a purchase order and pushing send button as a sales order for receiving side. Sending and receiving sides may use same system or different system (Different semantics and dictionaries are a very big challenge here). In some circumstances, MOM should transfer this EDI document by converting different formats like XML or legacy EDI formats.

There were many transportation options but we used HTTP protocol for transportation. We had to handle timeout problems. We used XML as core message format. Needless to say, messages are stored in database tables for guaranteed delivery. Messaging middleware uses the same JVM with applications.

To make it work across systems seamlessly and reliably is hard and a must in enterprise systems. Leaks in message transactions are much harder to fix than a local application transaction leak since a problem may lock all systems at the same time. Think a geographically distributed system, the system would be very delicate to fix the problems (high shutdown-restart cost).

Our Messaging Framework Architecture:

Managing Software Complexity

2009-04-04T15:23:00.000-07:00

Many software engineers including me ponder over “Software Complexity” problem. Complexity problem is not only IT world’s problem, it is in every field of life. Any software would probably introduce this problem in a certain stage of its lifecycle by hitting a wall. Software evolves over time gradually by adding new features. Every addition to source code increases software “Size” that is an element of complexity. Every new feature increases algorithmic complexity. Every new feature might cause architecture degradation. Complexity risks non-functional requirements including maintainability, productivity and quality. Small APIs become library, library becomes platform, platform becomes system, system becomes large system, and large system becomes ultra large systems (ULS). Then we begin to think about how to tame this wild beast. Is it possible to reduce complexity? Is it possible to avoid complexity? Is it accidental or inherent?

“The challenge over the next 20 years will not be speed or cost or performance; it will be a question of complexity.” Bill Raduchel

“Our enemy is complexity, and it’s our goal to kill it.” Jan Baan

“In the presence of essential complexity, establishing simplicity in one part of a system requires trading off complexity in another. We can master essential complexity, but we can never make it go away.” Grady Booch

Some questions we occasionally ask ourselves show signals of software complexity, “We do not understand the software, its functions, behaviors, structures, algorithms and domain (context)”or “Modifying software becomes a rocket science for us” or “Nobody figures out why this code is written like this”. Let’s list formal symptoms of complexity:

Details of problem domain (zillions of requirements)

Long requirements and specifications (Poor design, feature creep,over-engineering)

Many highly-interacted components or subsystems (High-coupling)

Increasing software size (doubling or tripling every year)

Arbitrary functions (Low-cohesion, non-separable concerns)

Wrong or leaky abstractions

Deep hierarchy (Relative Primitives)

Developer’s low capacity and low domain knowledge

Complexity in software has many forms, let’s see which types of complexities we may encounter in software development:

Domain complexity (Analysis-paralysis, lack of deep knowledge)

Team complexity (Many stake holders, many sub-projects)

Algorithmic complexity (Chaotic code flows, “save the day” fixes)

Design complexity (hundreds tables-columns, hundreds of design documents, ambiguous UIs)

Programming language complexity (Unprecedented language features)

Software platform-system complexity (Embedded, real-time, distributed)

Complexity problem is in most cases inherent. However we should ask ourselves that how can we manage complexity and eliminate software chaos (entropy)? The precautions taken against complexity may hinder the problem’s adverse impacts on our software. Especially early architectural decisions are very important. Otherwise at some point, we may have to say “impossible” instead of “hard”. Let’s try to list what are the remedies for complexity malady:

Abstraction: This is the primary cure for complexity problem (Divide and understand, separate concerns, separate skills). Abstraction is possible in any level or piece of software. Abstraction should be implemented properly so that it doesn’t leak (Joel Spolsky's Leaky Abstractions Law). It should reduce complexity in one place by taking some functions to another place (Overall complexity is conserved, Matt’s software complexity law). Duplicate or repeating code blocks in software are abstraction candidate for designing new APIs. In these circumstances, we should ask if we should take this code to another block or not(or how much abstraction is good enough). We can make abstractions in our own software along with others’ abstractions we depend on such as programming languages (from Assembly to 4GLs, from functional to OOP), platforms.(Human body is an example of abstraction, we only eat and the rest of process is invisibly done without our intervention, another example is car and driver)

Layering (Hierarchy, Modularity or Partitioning): We should slice system into layers so that functions are grouped together (This is another kind of abstraction). Layers must be lowly-coupled and highly-cohesive.

Refactoring: Refactoring codes, UIs and database tables can reduce possible problems and reduce complexity. Refactoring is like preventive medicine, it heals code base. In this way, we may add new layers and new functions (like strengthening pillars of a building to add new floors). Otherwise we begin to say that “Don’t touch if it works, no more features please”. Freezing software is not possible.

Modeling: Models help us understand and comprehend systems. It is the case that is "A picture is worth a thousand words". Modeling is also an abstraction. We model database tables, UIs and interactions. Design patterns may be classified as a type of modeling. Design patterns provide a common language to understand system. Object-oriented programming is also a kind of modeling.

Team Domain Knowledge: By increasing team domain knowledge, we can understand and solve problems more powerfully. Otherwise analysis will not match requirements and that will boost complexity. Domain may be technical as well such as database systems, networking protocols, rendering algorithms, programming environment etc. Team average skill level determines how much complexity we can handle (and how complex software can be).

Software Tools: Software tools are also an important helper against complexity (we should make sure it is really so, some tools are another complexity stream). Requirement tracing tools, versioning systems, configuration software etc.

To measure complexity in software, there are many metrics. Some of them are “Halstead's Software Science”, “Henry's and Kafura's Fan-In Fan-Out Complexity”, “Function Point Analysis (FPA)”, “Full Function Points (FFP)”, “McCabe’s Cyclomatic Number”.

I want to mention the abstraction that we certainly used in our application framework. Every layer of MVC is open to make abstraction:

1. Base Database Access Object: By using Java inheritance mechanism, some functions can be abstracted to upper layers. We can also inject a new database layer into JDBC interfaces (Interfaces are very useful for layering). In this way, every database call from DAO can be collected in one channel. This JDBC call wrapping is a common technique used by many JDBC libraries.

2. Base Servlet: A common base servlet that extends HttpServlet can be used for abstraction of some functions to upper layer. In this way, every application can use this abstraction (APIs) easily. Another advantage is that all servlet calls pass through one method, service(request,response) method (Front Controller Pattern) and we can intercept and manage all application calls (security controls, logging, exception handling etc.).

3. Base JSP: JSP pages are not classes in practice and we can’t extend them. For this reason, we used JSP include mechanism (partial JSP) for abstraction. A common partial JSP is used for loading common objects on top of the page like user info object, customizer object, authorization object etc. Otherwise on top of every page, we have to define these variables. Another good practice is a adding a common try-catch-finally block in every JSP page.

4. Base JavaScript Library: In every JSP page, there are some common JavaScript tasks. These common tasks are executed when loading and running page. By adding a dynamic JavaScript include in head part of page, we could make available some JavaScript APIs to all applications.

5. Base File/Folder: Every application has a folder structure. To make it easy to access these folders and files a base file and folder technique is very useful. When application needs to access application folders, base folders or a new folder class extending base folder can be used. In addition to this, we can also shorten Java’s file access API.

Complexity is a word that human created for incomprehensible. Infinity is such a word too. Whenever I think about complexity, I want to investigate nature. There must be many clues that we may use to build better software systems. Scientists are still trying to explore many nature forms like brain and DNA.

Links:
Software Complexity: How Do We Bring Order to Chaos?(PDF)

Domain-Driven Design: Tackling Complexity in the Heart of Software

Object-Oriented Metrics: Measures of Complexity

Zero Application Maintenance

2009-03-24T00:24:00.000-07:00

“Zero” something is a perfectionist approach. 100% is generally not possible but we should strive to reach that point as close as possible. This is why we repetitively do Software Process Improvement (SPI) all the time in every software process. In business world, “Six Sigma” has similar purposes. With this process improvement technique, companies struggle to reduce the product’s defect ratio to approximately zero. Reducing software defects and bugs are just a piece of overall cost. How can we minimize maintenance tasks to reduce total cost of ownership?

Zero maintenance has 2 dimensions:
1- Minimal Bugs and Fix Versions
As reported bug count increases, fix versions increases too. Every version requires an upgrade process. Some software upgrades are easy and simple. However, some big software upgrade and fix is long and critical. ERP upgrade is such an upgrade that it should be tested before installation in staging servers. To reduce fix version count, software releases should have stable and low-changing character. Alpha and beta versions should be tested by an enough testers or end users. This topic may be associated with “Zero-Bug Software” idiom. Test-driven development also helps us to reduce bug count considerably.

2- Minimal Application Administration
How can we architect our software that it would have minimal administration tasks? When designing software, this is an important. This topic is also related with IBM’s “Autonomic Computing” initiative, it has similar targets. “Self-healing” code could reduce system management tasks. Alongside system aspect, users also should not be overlooked when dealing with this issue. How can we take action for users’ wrong behaviors or requirements?

There are some best practices for minimizing application system maintenance:

Software releases should have all configuration information thus requiring no manual setting by system administrators (Manual configuration leads to errors). Automatic upgrade may be provided like in Windows OSs.

Software should provide batch-background tasks. In this way, we can utilize spare hours of system and prevent system performance degradation.

Users should have self-service system tools. For example, in ERP application some data may be needed to change within all modules and tables. Users could use data-maintenance tools without contacting system administration.

Users use extensively reports and they need new reports all the time. They should be given some simple report wizards to prepare or modify reports.

Software log files should have some limits and alerts should be sent to administrator in log-full cases. If heavy log write occurs, this may also make system very slow. Log write may be blocked when heavy write occurs.

System health should be monitored by an independent process so that it alarms when reaching some thresholds. For example, database connection count may have a limit that a database can handle.(i.e. DB2 Performance Monitors are a perfect implementation for this)

Software error messages should be recoverable as much as possible instead of fatal one. It is very easy to deny a circumstance and throw an exception, but in system side this may result a user interruption. We should tolerate unwanted conditions as much as possible.

Software should not require restart except planned ones. For example, your software may have a memory leak and it must be restarted after a certain period of time elapsed. Or it may crash or suspends in some circumstances. For example a heavy transaction may bring down your system or a lock (Java or DB) may suspend all the system. Memory leaks and deadlocks should be removed from your software.(i.e. uncommitted application transactions may cause database locks)

Backup process should be automatic. System, database and application backup should be backed up without administrator presence. Administrator should periodically check backup if they are healthy.

“Zero Application Maintenance” will not make system administrators free, but it will make room for other urgent IT projects. Also, they will call you less for your running software.

ERP Resources

2009-03-20T00:22:00.000-07:00

If you intend to learn Java, there are plenty of resources on the net. However, this is not true for ERP. ERP learning materials are spread into vendors’ documents and business field books. To find a vendor-neutral ERP portal is almost impossible. Although ERP packages differ in technology and application architecture, their business functions are approximately same. Some high-level ERPs have much more extensive functions but their basic principles never change.

Even ERP books are rare in some topics like financial management, human resources management. To find good books in Amazon has been a tough job for me; search books, inspect TOC, read sample chapter, read comments etc.

Here is my favorite ERP (SCM+CRM) book collection:
Manufacturing Data Structures : Building Foundations for Excellence with Bills of Materials and Process Information

Fundamentals of Production Planning and Control

Manufacturing Resource Planning (MRP II) with Introduction to ERP, SCM, and CRM

Manufacturing Planning And Control Systems For Supply Chain Management : The Definitive Guide For Professionals

Enterprise Resource Planning Systems: Systems, Life Cycle, Electronic Commerce, and Risk

Maximizing Your ERP System: A Practical Guide for Managers

ERP: A-Z Implementer's Guide For Success

Class A ERP Implementation: Integrating Lean and Six Sigma

E-Business and ERP: Rapid Implementation and Project Planning

Accounting Reference Desktop

Accounting Best Practices

Essentials of Payroll Management and Accounting

Payroll Best Practices

Financial Analysis Tools and Techniques: A Guide for Managers

Handbook of Budgeting

Controllership: The Work of the Managerial Accountant

Managing Cash Flow: An Operational Focus

Designing and Managing the Supply Chain: Concepts, Strategies, and Cases

Supply Chain Logistics Management

Supply Chain Management: Strategy, Planning and Operations

Supply Chain Management and Advanced Planning: Concepts, Models, Software and Case Studies

Logistics: An Introduction to Supply Chain Management

Inventory Accounting: A Comprehensive Guide

Inventory Best Practices

Customer Relationship Management

ERP vendors’ web sites vary in richness as a learning resource. Most generous one is Oracle. Oracle has made documentation of its recently-bought ERP packages accessible to everyone:
http://www.oracle.com/technology/documentation/index.html
http://www.oracleappsblog.com/ (Oracle Applications Blog)

SAP documentation:
http://help.sap.com/
http://help.sap.com/printdocu/core/Print46c/en/Data/index_en.htm (PDF)
http://www.sap.com/community/events/2008_05_SAPPHIRE_US/presentations.epx (SAPPHIRE videos are online)
http://www.sap-press.com/ (SAP Books)

Microsoft Dynamics documentation:
http://msdn.microsoft.com/tr-tr/dynamics/ax/default(en-us).aspx

ERP portals are also rare. ERP vendors’ web sites are generally sole place for community: http://erp.ittoolbox.com/
http://www.apics.org/
http://www.technologyevaluation.com/

Magazines:
http://www.mbtmag.com/
http://www.logisticsit.com/
http://www.intelligententerprise.com/
http://www.scmr.com/
http://www.hrotoday.com/
http://www.apics.org/Resources/Magazine/
http://www.cfo.com/

Dynamic Java Programming With Rule Engine

2009-03-13T04:37:00.000-07:00

Rules are statements that define business procedures and policies. Normally, every business applications contain many embedded business rules that determines business process flow and execution. When we program these rules with static languages like Java, there is no problem at all. The problem is that some business rules change over time very frequently. It is very expensive to fix these changing parts since it requires new versions and software development cycles; develop, deploy on a test server, test, package, deploy on a production server etc. Here the “Dynamic Programming” rescues. Our program doesn’t have to be necessarily written with a dynamic language. We can use this advantage in Java with Rule Engine mechanism.

Rule Engines don’t serve only the dynamic programming. They evaluate rules and extract some inferences by using some algorithms like RETE. This inference mechanism is hardly possible with many “if-else” statements in normal program code. Rule Engines have generally 4 types of rules;

Constraint Rule: We may need to define restriction or limits for some entities. “A customer sales order amount must be lower than 100 if the plant is over capacity ”

Validation Rule: Some validation rules may change over time. “Production order can’t be accepted if current day is Sunday”

Action Rule: Some actions may be triggered. “Send a notification e-mail if a purchase order is over $1000 to boss”.

Computation Rule: Some formula may be executed. “Discount is 20% if customer is Mr. Anderson”

We can use Rule Engines on the market but even we can write our own Rule Engine to benefit the dynamic programming in Java. By using Java compilers, classloaders, it is not so much hard. My favorite rule definition format is Java source code instead of some natural languages or XML. This can also reduce the overall development cost of your custom rule engine. By writing Java rule definitions, we can also use the IDE debugger features without learning a new syntax. If we develop our own Rule Engine, we may skip the inference engine part since we only want dynamic programming feature for now. Best place for plugging rule execution is database access layer. Rules should work invisibly behind the database objects. In some cases, some rules might be invoked from static program codes.

The benefits that we have when using Rule Engine are:

Dynamic programming with hotswapping feature that provides fix&run easiness.

Minimal cost of business rule changes that occurs many times.

Accumulation of business rules in one place that may result re-usability and code repetition prevention.

Dynamic validations that may change over time.

Constraints can be defines for default values, assertion checks etc.

In any time rule may be turned off/on.

Some of the features of Rule Engine may remind us database constraints and triggers. Using database may hinder portability and require DBA involvement. I think Rule Engine is better for business rule definitions. These database features may be used for other purposes.

The critical question we should ask ourselves is which part of business rules should be taken into Rule Engine from the programs. The answer is “frequently changing” rules. Some changing parts can be defined within parameter tables (processing options) but we can’t define computation or many if-else statements in these tables. But we should prefer parameter table if it is enough for the problem.

Links:
http://java.sun.com/developer/technicalArticles/J2SE/JavaRule.html
http://www.javaworld.com/javaworld/jw-04-2005/jw-0425-ruleengine.html
http://www.javaworld.com/javaworld/jw-06-2006/jw-0612-dynamic.html
http://www.amazon.com/Principles-Business-Addison-Wesley-Information-Technology/dp/0201788934/ref=sr_1_3?ie=UTF8&s=books&qid=1236927606&sr=1-3
http://www.amazon.com/Business-Rules-Applied-Building-Approach/dp/0471412937/ref=sr_1_1?ie=UTF8&s=books&qid=1236927606&sr=1-1

Web-based Reporting In Java

2009-03-03T06:38:00.000-08:00

Every application has 3 kind of blocks; parameters, transactions and reports. Reporting is a fundamental part of applications. It begins with a “one page” printing then expands to “Business Intelligence” reporting. Reporting is interpretation and analysis (via SQL) of raw data and presenting it in readable and printable format (via Reporting Tools). As function widens, SQL and reporting tools differentiate a lot. Data warehousing (OLAP) is such a special analysis and reporting area.

Your web-based reporting journey will probably begin with figuring out that web-pages are not printer-friendly. So we need a reporting tool. Reporting tools are not just an ordinary tool in your toolbox. It determines your application success and it is important as much as transactional processing (OLTP). If you can’t show the data as desired, it has no meaning for end users.

In web-based applications, reporting has some difficulties:
1- Determining Reporting Tool: There are many commercial and FOSS solution out there but their web-enabled reporting capabilities are questionable. If it is commercial, it will introduce license cost and increase your solution cost for your customer. If it is FOSS, then you are dependent to voluntary open source developers. In our case, we had tried to use FOP (XSL-FO based) many years ago for our reporting requirements, we couldn’t overcome the problems of localized fonts and can’t get timely responses and fixes. Businesses and projects can’t wait.

2- Integrating to Your Applications: Reporting is not only composed of design tool, it has a runtime engine that you should integrate with your JDK. If it works in your application JVM, than JDK version compatibility is important. If its run-time is detached, than it is going to be another problem to use your business objects in the reporting environment. If your server is Unix-based, reporting tool platform-dependent MS-Office (Excel) features may not work.

3- Printing: This is the most problematic topic. In web-based reporting, we used Internet Explorer’s “Print Template”s that solves web-page printing problems (via MeadCo's ScriptX product). It solves graphical printing problem. Even you can write your own templates. What about dot-matrix or label printing that you have to communicate with printer directly (Raw Printing). Here, we had to write a custom ActiveX component for this purpose. Another problem is server-side printing. If your client doesn’t want to install someone’s printer driver, you have to use server-side printing. You only install printer drivers on server then your client can print via your print-server. Another point is that many reporting tools claim that they can easily handle printing problems with PDF-reports. I don’t think that PDF is a good presentation format for “default” report format. I think first format should be HTML, it is lightweight, faster and can be dynamic (i.e. drill-down). Then report should be converted to any format including PDF. A final note, we can’t use Java’s printing API since our pages are JSP (HTML).

4- Report-Design: In desktop applications, reports can be easily generated with drag&drop ease. In web reporting tools, this function is generally missing or weak. Reporting tools generally brings desktop-based design tools. Report generation is also a part of reporting. Every report producer user had to install this application that we lose web-based application advantages.

My other observation about reporting tools/frameworks:
Reporting Scripting Language: Although this is not related with web-based architecture, but I have to write about it here. Most of reporting frameworks brings a new scripting language that I thoroughly object. Why are we inventing a new language that we already have Java? A new scripting language brings a new learning curve, a new cost. “To make report programming easy” can’t be a reason since most of the report writers are programmers.

Dot-matrix Printing: Our search for text-based reporting function in Reporting Tools ended with no result. There was no direct support from Reporting Tools for this function. In our case, this was a major function that resulted to write our own reporting frameworks and tools. Your reports can be printed text-based not graphics-based. This can be achieved with some printing commands (i.e. ESC/P2) that may differ among dot-matrix vendors. Graphics-based printing within dot-matrix results low-quality prints, pagination problems, long-printing job, and high toner consumption.

XML Report Design: We developed XML design (clean and simple) feature on top of our current API set. Up to this feature, we were developing reports programmatically. XML reporting enabled us dynamic design which report designers can change every part of report without recompiling. In our XML design search, I was surprised how complicated the XML structure of other tools. I am not sure if everything can be done via GUI design tool without touching XML and its complexity adds another cost (We know in IDE GUI design, this doesn’t work all time and we have to touch to the low-level codes).

Business Process Management(BPM) In ERP

2009-02-24T00:20:00.000-08:00

BPM is another buzzword that has different meaning for every software camp. In essence, it is the effort of defining, managing, executing, monitoring and reporting of business processes. Isn’t this the promise of ERP software? ERP software intersects the function area of BPM but they are not the same. As I jump into the terminology, it was really a confusing topic to find out the differences.

ERP software has rigid built-in process definitions and it is hard to change dramatically. Although there are many parameter definitions and settings to make it flexible, it is not as flexible as it is in the BPM software area. In BPM, you can dynamically alter the definitions of processes whereas in ERP software this is achieved via hard-coded customizations.

On the other hand, BPM suits lack predefined built-in processes for enterprises. In absence of ERP system, we would have to understand and define all business processes ourselves. ERP already contains these definitions without any extra effort. ERP software is mature enough and matches most of the need of enterprise processes.

There are 2 points that ERP systems can leverage from BPM concepts:
1- Workflow: Workflow is the core part of BPM (Workflow versus BPM is another discussion topic) that enables us flexible activity (process) definitions. Many ERP systems added this capability to their package with different functional structure. Workflow can eliminate the rigid nature of ERP systems and provide an additional logical layer on top of the ERP. We can easily model process definitions cooperating with ERP processes without any coding or customization. Workflow uses ERP data events to forward the workflow activities.

2- Electronic Data Interchange (EDI): Document or data exchange amongst the systems is possible with EDI function. This function is also covered by BPM software. Message exchange is done via some messaging servers. Most common example is sales or purchase order transactions. A transaction message from one company to another transferred without manual data entry. Web services with XML format may be the new definition of EDI function. We define the activities in workflow and its remote activities can be executed via EDI. Email system is another medium used for this purpose.

Anybody entered this field is to be overwhelmed with the definitions; BPML, BAM, Workflow patterns, EAI, WfMC standards, many vendors’ varying solutions for BPM. My last comment about BPM is that it can be used without ERP but it would be much more functional when used with ERP software.

Code Generators: Robots of Software Factory

2009-02-18T03:29:00.000-08:00

There are many analogies between software development and manufacturing. “Software Factory” methodology utilizes these similarities. Another analogy is possible for construction industry. As an interdisciplinary study, these similarities are beneficial to understand our own software problem. In manufacturing industry, robots replaced the workers in some product lines and workers did controlling and managing of these robots. Robots are great examples of automation. They are configured for the current assembly-line product specifications. They don’t have to be “Artificially Intelligent” to make the job. Analogy is that code generators have similar features with industrial machinery robots. You program the generators according to your application specification and your software developers use these tools to produce the software (also named as Generative Programming). In this way, we can automate software development.

There are many types of code generators and techniques. Some of them low-level techniques that helps achieving some specific tasks. One example is language abstraction. Your high-level language is automatically translated-generated into low-level machine code. One example in Java world is JSP transformation into servlet. These generators generally invisible to the developers and can’t be considered as an automation in development.

The generators that provide automation are those generate final software that the software team develops. In business applications, final software is composed of elements like GUIs, business logic codes and database codes. Every elements of software may be taken into generators if you can capture template or pattern. Templates are some specification patterns that depend on your API set. Let’s say templates are meta-descriptions of your coding standards.

“To build a generator or use existing one” is one of the most discussed issue in your software team. Could we utilize the advantages of generators or could it be a wrong investment that ruin clean code base? Yes critical question lies behind the meaning of generated code for your team. If generated code value is high that it shortens development time by removing tedious repetitive tasks than it would be worth for investment.

Code generation can be applied in following way:

1- Determine what to generate: To find repetitive task is not hard in your development environment. If same things are done more than three times it is a candidate to generate. For example, generator experts recommend code generation for CRUD codes.

2- Capture meta-level coding structure: Capturing your meta-coding structure is important. Your coding structure should have a pattern that is shared among applications. What are the variable parts that would be parameterized by generator? How your GUI is constructed? In which way do you access database? How does your controller code manage object states? Capturing is the hardest part of the whole job. It requires understanding of the current development API set very well.

3- Build or configure your generator: There are many generators available there, but if not satisfied with them even you can write it yourself. After building it should be tested against current codes if anything is wrong. Generated code should be complete, concise and error-prone. Otherwise leaky generators make it impossible to benefit generation technology. Your coders would prefer manual way instead of wrestling your generators.

My generator recommendations:

Generators should not change generated code structure a lot. Otherwise, “Coding standards are kept” benefit would be lost.

If we find an abstraction area, we should ask if this abstraction is done via code generator or a new API.

Generated code should as short as possible. It should be supported with a high-level API set to reduce produced code.

Every part of software can be generated. There is no limit. Limit is your capturing ability in your environment.

I don’t like “Your code is here” generators. IMHO, generated code can be changed in every way. Changing it should be safe as well. Dynamic generators that can change code base after customizations may restrict this. Static generators may provide much more freedom with re-generation overwrite problem. If we change it, it should be readable and understandable.

More intelligent generated codes mean more intelligent generators. Generators complexity should be manageable. Here MDA rescues. Models are a way to achieve sophisticated generation in a meaningful way. Generators should use models (i.e. database models).

Code generation should not be made a rocket science. Begin with small steps than make improvements. It should serve you, not tie you.

Code generation speeds prototyping thus serves for “Agile Development”. Any change in software specification can be easily reflected into prototype without heavy re-write.

Links:
http://www.codegeneration.net/

Code Generation in Action

Generative Programming: Methods, Tools, and Applications

Data Models: Skeleton of Business Applications

2009-02-10T23:04:00.000-08:00

As we first began to build up data model of our ERP modules, first lesson was to understand how the data model is critical for applications. ERP applications are database-driven and work on so many tables. ERPs’ table count change according to their range. Entry level ERPs have about 500-1000 tables; mid-level ones have 1000-5000; high-level ones have 5000-50000 tables.

I am not going to write here about database normalization or ERD or CDM or PDM or UDM or DW. When you begin modeling, keep away terminology stuff as much as possible. Modeling is just capturing the real world realities (entities and processes) describing them in databases. Modeling process has 3 aspects; business domain, domain model patterns and your design skill. To model successfully these 3 aspects should work correctly. First modeler should know business domain with business requirements. Also modeler should know patterns of data model for this domain. Data models of ERPs in the market are an incredible vault to get these patterns (Not open to public since this part has core value for ERPs). Lastly modeler design skill and experience finalize the story.

Why data model is critical for applications? Because data model is the most important part of business applications. If it is not designed correctly, all project and development efforts are meaningless. Major data model changes may bring application rewrite. Even minor data model changes are a bad thing when considering its change cost (Change Management). On the other hand, minor changes such as new columns, column type changes, or column name changes are inevitable as software evolves with new requirements.

I’ll list here other issues about data models:

1- Data models are the most important part of software artifacts that is shared between programmers and analysts: Data models are a conjunction point for business users, analysts and programmers. In our practice, data models are the last point where analysts are concerned. Analysts prepare software specifications and data models and give them to developers. All programs evolve around this data model.

2- Data modeling is not a database administrator (DBA) job: Data modeling concerns business domain much more than physical database. For that reason, business analysts are much more candidate for this job. DBA may revise data models for database requirements, such as primary keys, foreign keys or naming conventions etc.

3- Search for data model patterns: Our first sales order data model had 5-level (branching) tables which was a catastrophe for the remaining modules that used these tables. Inserting data, reporting data was a nightmare. It didn’t last long we noticed the problem. Order entry models basically have 2 levels; order and order-line tables.

4- Develop data model standards: If you have many tables, it is a very good practice to build a column catalog. This catalog will have naming conventions and physical structure information. As your table count increases, you will see different names used for the same attribute(synonyms); for example size, volume, amount, count etc. Using different names bring readability-maintainability problems.

5- Primary and foreign key decisions: This is the most delicate confusing part of physical structure. There are many discussions about PK and FK design. There is no single answer too. Primary key concerns; single versus multi-column, auto-number versus meaningful key, identity generation in database or application etc. Foreign key concerns; Is FK a performance killer? , how much relational consistency is enough? Which FK constraint should be used?, FK as a dependency etc.

Recommended books about data modeling:
The Data Model Resource Book, Vol. 1: A Library of Universal Data Models for All Enterprises

The Data Model Resource Book, Vol. 2: A Library of Data Models for Specific Industries

The Data Model Resource Book: Universal Patterns for Data Modeling (Vol. 3)

Data Model Patterns: Conventions of Thought

Data Model Patterns: A Metadata Map

Data Modeling Essentials

Blurring Borderlines of ERP Software

2009-02-02T22:43:00.000-08:00

ERP (Enterprise Resource Planning) software is an enterprise management system that is not limited with planning. ERP is originated from MRP (Materials Requirements Planning) software. With addition of financials, manufacturing and human resources to MRP, today’s modern ERP systems took shape. As people involved in ERP projects and software, it is becoming a very tough question that which software is really en ERP software? We see the tendency that software packages names themselves as ERP very easily. Are they really an ERP system or just an entry level business application?

In my opinion, to be ERP software, one package should at least have following modules:
1-Logistics: Sales-Purchase Management, Warehouse Management, Distribution Planning etc.

2- Manufacturing: Shop-Floor Management, Production Planning, Materials Requirement Planning, Capacity Planning, Quality Management etc. (For non-manufacturing businesses add Service and Project Management)

3- Financials: General Ledger, Accounts Receivable-Payable, Costing, Asset Management, Cash-Flow Management, Consolidation etc.

4- Human Resources: Payroll, Recruitment, Training, Career Management, Performance Management etc.

Borderline blurring problem occurs in 3 forms:
Under-functional packages:
Some small business applications are not an ERP. I see lots of software that is roughly a portal or CMS (Content Management System) software that names itself as ERP. On the other hand, they may have some ERP modules but these modules don’t have enough features to handle business processes. One example is some open-source ERP packages.

Over-functional packages:
This is a very interesting topic because we experienced same thing with Operating Systems. Operating systems expanded their functional scope including browsers, media players, text editors, email systems etc. Don’t be surprised if ERP is bundled with OS in the coming years. For the case of ERP, this is also happening. ERP packages are expanding by including CRM, SCM and BI. Gartner called this new generation of ERPs as ERP II. Some ERP packages became monster of business applications. IMHO, ERP packages will continue to swallow every business-related application.

Mixed-functional packages:
Another blurring borderline problem occurs in the definition or border of modules. Although there is some level of standard definition in business processes, we see that every ERP package uses different classification for their modules (Especially new modules). One module may mix some functions of different processes or modules. I witness this case in SCM (Supply Chain Management) modules. Yesterday’s logistics modules are magically re-branded as SCM without solid supply-chain functions. We can conclude that ERPs tight module integration may blur module functional lines.

ERP Implementation Challenges

2009-01-26T21:56:00.000-08:00

In our ERP project, we’ve finished about 30 ERP implementations. It took nearly 3 years to finish these projects. Our main difficulty was writing our own ERP software which I don’t recommend any company. Packaged or COTS (Commercial off the Shelf) applications (ERP) are generally a better way to follow. As application technology diversity rises, it gets more difficult to support in-house applications while keeping skilled personnel. In the past, every company developed their own legacy system. Today, it is much easier to find a package that meets business processes. Every company has some unique requirements but these requirements can be solved in ERP platform. Before ERP projects, most of company says “we are unique.” This statement is not true in most cases.

On the other hand, some vertical industry functions may not be found as it is desired in ERP packages. To find out if an ERP is appropriate for business is a key work which is called “ERP Evaluation”. Sometimes, ERP packages don’t fit these vertical industries and projects end with failure. It is very important that ERP consultants have experience in this vertical in addition to ERP package vertical industry support. One another exception is some special requirements that never fit into ERP solution. For example, government and military organizations may have unique requirements and it is hard to find special solutions.

Implementation challenges have 4 main sources:
1- Business (Domain): Every project requires some level of customization. As business domain changes, ERP teams should learn characteristics and difficulties of new domain. Familiarity of domain is a very important in project success.

2- Project (Implementation): Implementation process is a very important part of ERP projects. Timelines and milestones should be followed very carefully. Which modules will be use at first and second phase of project? Which approach is suitable for implementation, big-bang or parallel? How the project tasks should be distributed to consultants?

3- People: ERP consultant team and company’s project participants are a key factor. Their harmony is important. Some key users resistance or feature requests may jeopardize project. Senior management level support is another important aspect that is valid in any kind of project. Consultants’ unplanned turnover is another key problem. Selecting true key business users provides us understanding business processes correctly.

4- ERP Software: ERP package features are another aspect that shapes project. Some missing features may stand as single-point of project failure. ERP package must bring minimum customization for projects (Vanilla approach). System installation and administration is important for non-functional requirements.

Implementation challenges:
1- Mission-criticality: Since ERP system is nervous system of companies, its interruption means business interruption. Especially ERP teams feel this on go-live period. If something goes wrong like erroneous data or a major error, and company data volume is high, this situation is a red alarm for ERP team. If problem is not urgently solved, its compensation might be impossible. To keep the business running, team work overtime is necessary until the problem is solved. Every ERP team may experience troubled times. Business interruption is something like that production stopped, transportation cancelled, orders delayed, customers lost etc.

2- ERP projects are not just another software project: This is a fallacy that some IT people who are not familiar with ERP world assume ERP projects as typical software projects. ERP project is much more business-oriented rather than technical one. IT people who knows business processes are much more immune to fall in this trap. ERP implementation is a different branch of project management that has unique characteristics. This is why ERP vendors have their own ERP implementation methodology like Oracle AIM or SAP ASAP.

2- Business Process Reengineering (BPR): Alongside with ERP package usage, some businesses have to change their working styles in accordance with ERP process style. If business changes many processes during project, this adds another burden on team to handle. New style may introduce new problems. Changes in processes may make business more advantageous and decrease customization level, but this change is a cost that is paid by ERP project teams.

3- Pessimistic and dissatisfied users’ dangers: Business employees and ERP team should have enough motivation for project. This motivation can be fostered by many factors. Users should be believed in project that it will make their life better. Some users’ bad attitude may discourage all team members and business users. Users generally follow this pattern during project; suspicion, resistance, curiosity, adoption, obligate, liking etc. Every user changes his/her state of mind among these patterns alongside the project.

4- Feature creep customization: In the initial step of project, many things are demanded. Many modules are assumed to be used. It is very important to see that what is most important and what is not for this business. Large customization impedes version upgrade later thus increases total cost of ownership. New modules-features come with new education, configuration and data entry cost.

5- ERP software performance: This challenge appears when ERP software doesn’t scale well for large volume of business users or transactions. Hardware infrastructure is important. ERP software range is key differentiator here. If it is not designed for large businesses that supports only small and medium businesses, it is difficult to tune system for large transactions. ERP technical consultants should also have enough experience to tune OS, DBMS and ERP.

Peer Reviews In Software Development

2009-01-18T22:09:00.000-08:00

“Peer Review” is a very important method used in many disciplines. In software, “Peer Review” enhances every piece of software from project schedule, to development, to testing and to quality. Surely it is not free. It takes time of team members who have different responsibility in software lifecycle.

Yes, it is a very valuable method but how do we plug it into our team and current processes by maximizing value? This question is the most critical one when we are trying to improve our development process. Some of the faulty application of this method may cause problems instead of improving development. As it is valid in every part of our life, again here “Context” is important. This method must be applied in context-driven nature. Some big projects may need many “Peer Reviews” in many levels whereas some small teams may skip some of them. Agility, project scope and team size is a very important varying factor when applying it.

Peer reviews do not only serve for quality but it has many benefits in many aspects. Let’s describe these aspects that can be utilized by peer reviews (This list may be considered as advantages of peer reviews):

1- Communication: Even if only one person is joined for review process it provides communication among team members. Each review process participants must be very carefully determined. People who are not expected to utilize from this process must be discarded. Otherwise communication noise occurs and wrongly participated members diminish the value we expect to get from this process.

2- Learning: New members of team can use this sessions as a knowledge-growing opportunity. Different people have different viewpoints and that produces richer ideas and eventually results.

3- Priorities: As reviews continue, meanwhile many types of development policies are specified. Which features are more important? Which bugs needs immediate attention? What would be the project priorities?

4- Quality Metrics: Reviews enable us to determine quality metrics. Which code block is acceptable? Which kinds of quality issues are frequently encountered? Which types of problems are tolerable for releases?

5- Negotiation Culture: “Conflict” is possible in every project. There may be many reasons. For example, analysts and developers requirements may conflict (Functional versus non-functional requirements). Team players learn how to handle gently these problems and to agree with a result (i.e. Voting may be done). This practice helps us to prevent later big quarrels that may result into losing some of members.

6- Software Standards: As software feature requests increase and software gets bigger, we need software standards. Standards evolve and are used by developers and testers. Standards should be determined by senior and expert members.

In which points of software development could we apply peer review? Here is the list we apply review process:

1- Feature and Bug Management: Initial design is done by preparing software analysis and specification documents. These documents are reviewed by an analysts committee for some corrections. After the software is released, every feature request and bug report is inspected via weekly meetings by a committee which includes related analysts and programmers and their leaders. Some major bugs may be immediately evaluated and approved for urgent fix without waiting these meetings. By these review meetings, that we call “Revision Evaluation Meeting”, every piece of software is inspected and this helps to reach software standards (consistency among parts of software) and quality.

2- Code Inspection: Software source codes are another very important asset. Its quality reflects software quality. Some developments may corrupt source code quality. Source codes can be inspected by senior developers who know the coding standards. The form of meeting may vary from team to team. In our approach, one senior member first inspects the target programmer’s codes than takes notes. These notes are later discussed in a meeting whether the problems are real or not. When everybody agrees upon the problem then it is added to the schedule of programmer with a priority for later correction.

3- Software Verification: (Are we building the software right?) Although this is not considered as a peer review, I added it here in the list. In this stage, new software features are tested and validated against requirements specification. Validation process may be done via a review process that analysts, programmer and team leaders participate. This is only done after large developments or before big releases. Participants may detect some design and development problems during this presentation meeting (like an internal demo).

4- Software Validation: (Are we building the right software?) This is the acceptance step. Our customers use the software and approve or reject for additional development. If possible, some key users may review the new features before software release. This review process may be done via a meeting hence customer communication is increased and beta version problems are reduced.

5- Project Evaluation: In some phase of projects, it is very good to arrange progress review meetings. This is also a review of current picture. Project members and some customer representatives (stakeholders) participate and evaluates project schedule and to-do list. We do this generally in a monthly period. If this is not arranged, late problems might mitigate project success.

Resources:
Peer Reviews in Software: A Practical Guide

Software Inspection

Software Quality Assurance: From Theory to Implementation

Better Java Web Frameworks

2009-01-12T00:31:00.000-08:00

I have been doing web application framework development for a long time. In my first experience, we developed a “Web Application Framework“, to ease development so that even business user could make development. As the years passed, I never see any business user entered self development because of the fact that this job belongs to programmers. Today some 4GL or DSL tools are still on the search of the same promise. Before writing our own framework, we had used WebObjects development tools, to make Java web development. It was very good platform that front-end GUI development could be done from a GUI editor with a component model. Every web element can be used as a component and it had events like “clicked”, and click handler Java code can be added with double-click in GUI editor. One disadvantage was portability that you were bound to this platform and its Application Server. This model was similar with Visual Studio ASP.NET development environment that was present years ago from .Net emergence. After years later, JSF tries same model.

Web Application Frameworks meet the web application requirements. It is hard to define its border, but it should at least support presentation layer. It may expand to “Business Logic” layer or data access layer. In Java, we have following technologies for web development; JSP, Servlet, JSF, EJB. There are many Java-based web frameworks; Struts, Spring(Spring MVC), GWT, WebWork, Tapestry, Apache Wicket etc.

Power of Java and richness of web technologies produced many web frameworks (We don’t even see many internal frameworks). It is getting harder and harder to find a compatible development environment. Even some of vendors are pushing to scripting languages for “Ease of Development”(One another interesting move is “Cloud Computing”). I don’t favor scripting languages for enterprise web application development for “Ease of Development”. On the other hand, “Ease of Development” is a very important topic for programmers (I mean this includes both “Ease of Testing” and ”Ease of Deployment”). “Ease of Development” is closely related with productivity. As development environment complexity increases, development becomes harder thus productivity decreases. Today, we witness low-productivity in many software projects. Low-productivity results low-quality and delay in projects. We need better, simpler and yet powerful web frameworks that truly understand web architecture and its requirements. It is very important to make value proposition of framework before accepting.

Choosing A Java Web Framework: A Comparison

Choosing a JVM Web Framework

The API Field of Dreams – Too Much Stuff! It’s Time to Reduce and Simplify APIs!

Rethinking JSF - The Real Problem

Criticisms of Spring, Struts, PHP, and Rails

Here are required features of a web framework according to our enterprise application development experience:

1- Includes all core application layers (MVC):
Web framework should include data access, business logic, controller and presentation layers internally. As frameworks turn out to be an integration hub, it looses value. Every integration among the core layers introduces new complexity, new glue code, new dependency, and conflicting of intersecting features. If data access layer (Model) uses another framework, presentation layer (View) uses another framework, integrating these frameworks adds a very big challenge even if frameworks support each other. Replacing any framework causes many new problems later. For example, JPA is developed for data access independence but at this time you are limited only the features of JPA. IDE is a major development tool, but at this time we need an “Integrated Development Frameworks” environment within IDE. (Similar with ERPs that brought together enterprise applications under the same umbrella)

2- Avoids heavy-componentization:
In web architecture, desktop-like componentization is heavy and inflexible. Components in desktop applications were very successful. They utilized reusability and used in IDEs. In web applications, component model doesn’t work at the same form. Efforts to convert HTML (+JavaScript) into component model will not be successful as desired. This is because HTML is dynamic (DHTML), works on client and declarative (Declarative Programming). With heavy-components, we loose declarative programming to some degree. We loose “Web Graphics Designer” ability to edit web pages because of moving from design-time to run-time and moving structure information from HTML to programs (With losing HTML Editors functions). Web editor’s favorite structure place is CSS files so what about CSS componentization? Another problem is architectural. Web GUI has 2 runtimes; server and client (Browser). At previous years, web frameworks supported only server-side-only functionality. Then today we see client-side-only approach. I think best solution is balanced mixture of both client-side (JS) and server-side code with component templates (not hard components but light partial HTML+JSP+Servlet codes). I’ll not detail further, there are already many discussions about “Component-based” versus “Action-based” frameworks on the web.

3- No new tag markup or page template:
Some web framework requires learning a new markup with no added-value. Your form inputs turn out to be strange tags. Finally, developers don’t understand HTML, JavaScript, CSS because no time left for this learning. Who will fix GUI errors? Frameworks should bring minimum or no new tags (instead we may prefer attributes). HTML tags with simple JSP expressions are enough (KISS). Isolating developers from HTML and JavaScript is not possible.

4- No XML usage:
Heavy XML usage for configurations makes programs hard to develop, hard to understand, hard to test. One example is “Page Flow” information in XML files. Another example is bean configuration. Yes, pulling this information makes it flexible but who needs it? How many times your page flow changed? How many times did we utilize flexible bean configuration? What about source code readability? I don’t like “Dependency” so “Dependency Injection”. I think dependency is not free that you have to manage its subtleties. Here is my anti-pattern “Dependency Rejection”. XML can be used in other useful places like AJAX messages or data import-export.

5- Has its own web GUI page elements:
Rich web elements (say light components) are generally found only in JS or AJAX libraries. Web frameworks should provide rich elements like; Calendar, Dialog, Menu, Popup, Progress Bar, List, Grid, Tab (With sub-levels), Master-Detail Windows, Child Windows, Record Navigator etc. Developers can easily extend these elements. We are still turning around simple features like table sorting, filtering etc. We should step ahead. There is still no desktop-like web grid components to use (I see only in JS libraries) that I mentioned in my previous blog post.

6- Code generation:
Code Generation makes “Rapid Development” possible. Every part of software should be generated (Generative Programming); CRUD data access classes, business code, controller code, and view pages. Code generation takes development one step ahead of “Drag and Drop” WYSIWYG editors. If web framework facilitates code generation, developers could jump to customization details of application instead of building everything from scratch (MDA).

7- Has its own GUI JavaScript library:
Another bleeding integration point is JavaScript libraries. JavaScript libraries are not fully-integratable with web frameworks. They try to solve the problem in client side. What we need is close cooperation with client-side and server-side. Most of web frameworks unfortunately have no or little JavaScript in their presentation layer.

8- AJAX support (Asynchronous Communication):
AJAX eliminates bothering page-refreshes. Web frameworks should properly blend AJAX functionality into their code architecture. AJAX requires server-side coding. As we make client runtime powerful with AJAX, GUI state management code is duplicated. For example, if we update and fill a combo-box with AJAX call then server-side bean that is bound to this element is not aware of this state change. We have to change server-side state as well. AJAX functionality should be implemented without code duplication (Another interesting trend is AJAX MVC).

9- Portable among application or database servers:
Application and database portability is not easy. In Application Server side, class loader policies change, session management changes, deployment model changes etc. In DBMS side, join clauses change, paging, and sequence generating changes. Web frameworks should provide portable packages for different platforms. On the other hand, some web frameworks have their IDE and Application Server (believe me even DBMS). I think we must leave this job to the famous bright products (IDEs and Application Servers in the market).

10- Input validation:
Data input validation is a very important feature. If validation doesn’t occur in application, database error occurs. Database errors are not user-friendly. Some validation errors may not be related to database. Programmers need automatic validation according to database object metadata. Custom validations should be added if needed.

11- Bug-free:
Because of bugs in frameworks, all average developers become framework expert spending valuable time to figure out the problem. “Focusing business problems” is lost. I read many open source framework hacks and workarounds in many blogs which is not the task of developer.

12- Handles exceptions user-friendly:
If error or exception occurs, user-friendly messages should be returned. Application programmer has some responsibility for this but web frameworks may ease this task.

13- Eliminates double-click, double-submission problems:
Double-click may cause double-submission. Double-submission may cause unexpected errors in application (2 threads tries to do same thing). Web frameworks can eliminate this problem even in client-side without going to server.

14- Authentication and authorization support:
User login (authentication) is still developed by programmers without knowledge of SQL-Injection attacks. Web application authorization is still missing. Who will be granted for CRUD on which application etc.?(User roles, permissions) I am sure that in every enterprise web application, application authentication and authorization is re-invented.

15- Security controls for web attacks:
Web frameworks should prevent web security attacks like; Cross-Site Scripting (XSS), SQL Injection, URL Manipulation, HTTP Injection, Session Hijacking etc. Web client data is un-trusted and open to tampering so this is why we can’t quit totally server-side validation for the sake of client-side validation.

16- Reporting integration:
Reporting integration is important. We need reporting products/frameworks integration. Would you use your data access objects in your reports? Would your reporting engine use the same JVM runtime?(Embedded Reporting)

17- Messaging and workflow integration:
Web frameworks may support easy integration with messaging (JMS) and workflow products. Workflow is one of major element of BPM (Business Process Management). In some middleware stacks, this is included (i.e. JBoss Seam jBPM). Web application frameworks may support business events and workflow activities. These events can also be used to feed messaging backbone (ESB).

18- Application to application integration (i.e. Web Services):
In Java, there is external system (EIS, legacy) integration API, which is JCA, but inter-application communication within same JVM is not standardized. Let’s say we have 2 applications and one should use some call other application code. There is no standard for this. Basic solution is just adding other application’s path into its class-path and then using other application objects. We developed an Adapter API for standardization of this. In one-application environment, this is not a problem but if many applications are required to communicate, it gets more important. You can even convert your APIs into web services when necessary (integration with remote or non-Java systems). Web frameworks may provide tools for web services code generation, deployment and monitoring.

19- Admin application for run-time process and user session monitoring:
This is very important in point of user and system management view. What are my users executing at the moment? Which applications take longer to finish? Which users are on-site? Which pages are they surfing? In each session, which objects are they created? What are the URLs that a user requested? Which SQL statement did a user execute?

20- System resource management:
If your application runs big queries that require a lot of system resource (CPU, RAM, DISK I/O), we are faced the reality that resources are limited. If applications don’t restrict user processes, then system will consume its all resources and will not respond to even small processes. For the sake of system availability some user may be rejected by system. Web framework may have such limitation API’s.

21- Cluster support:
When server load is high and performance is a major concern, load-balancing is required. Application server clustering will not suffice, web frameworks must support cluster architecture. One simple example is framework’s id generators. They will collide in clustered Application Server environment.

22- Multi-database, multi-company, multi-window, multi-session support:
Application user may need to work on multiple database instances. One user may have to work with multiple companies. User may want to use multiple GUI windows. Web framework should handle or prevent state corruption among windows. User may need to work on the system with many sessions.

23- Internationalization:
If there are global users, then i18n support is important. One key aspect here is Application Server and DBMS should also support your localization.

24- SSL support:
If web application is wanted to be secure in insecure networks, SSL-support is important. SSL deployment in HTTP Server would not be enough. Even if SSL is not used, frameworks must encrypt sensitive data between client and server, like user passwords.

25- Document attachment:
In every enterprise application, document attachment is important. Users may want images, Excel documents attached to their application records. Every programmer first search for an upload utility then tries to understand server document folders. Instead, built-in functionality saves valuable time.

26- Mobile device support (i.e. Internet Explorer Mobile):
If we want to plan mobile access to our applications, how can we do this with web technologies? Many mobile devices have built-in web browsers and we may run our applications in these browsers. Web framework mobile support would be very beneficial at such cases. Otherwise, you should explore mobile web browser limitations by yourself.

27- Portal features:
Partial web components should be supported to use in Portals or external sites. In portal terminology, its name is portlet. There are many synonyms; Widget, Mashup etc.

28- Scheduling:
Application task may be batched and scheduled. After task completion, users may see results.

29- Keyboard hot-keys:
Users, especially old TUI (Text UI) users want keyboard hot-keys. Buttons, command icons should be bound to hot-keys. Web frameworks elements can support this instead of developing in every application.

30- Alerts between users:
Users may want to send messages to each other or system admin may want to send messages to users like notifying a shutdown or an application restart. This feature will be very handy.

Since this list is long enough, I had to skip these features (plus non-functional features); email support, back button, page navigation, bookmarkability, post-get support, formatters/converters, page skins, context-sensitive helps, URL manipulation, caching, logging etc.

This feature list is application-centric. Web-site-centric feature list may have different items. In web development, these 2 camps (web-based application developers and web-site developers) are joined and have some conflicting requirements. Scripting languages (PHP, Ruby or Groovy) and RIA platforms (Flex, Silverlight or JavaFX) may be much more appropriate for web-site developers. This is a controversial issue and differentiates framework architecture. In this aspect, we say that some web frameworks are much more appropriate for web-site development and don’t fit into enterprise application development.

Another argument is “Web architecture is not appropriate for enterprise application development”. I think we overcome the barriers of web architecture with many bright standard technologies thus making it a good application platform; Powerful Browsers, JavaScript, CSS, HTML(DHTML, XHTML), XML technologies, Web Services and finally AJAX meet the application platform requirements. Let’s make Web 3.0 possible.

Implementing Master Data Management (MDM)

2009-01-02T06:34:00.000-08:00

Master Data Management (MDM) is the process of managing master data so that systems have one version of master data (one version of the truth) thus preventing errors and raising data quality. What is master data in an enterprise application? Master data is non-transactional data that doesn’t change a lot (used over the years in the system) and is shared between applications. MDM is only valid if master data is duplicated among systems. This is an EAI (Enterprise Application Integration) issue. These systems may be instances of ERP systems or instances of multi-site ERP systems or OLAP systems or in-house legacy systems or any type of other applications (i.e. CRM, PLM, SCM, and GTM).

All master data is not the same, some may have different characteristics. Some master data is used only for lookup purposes, holds a few records and doesn’t change ever, like currency type, measure unit type etc. In contrast, some master data holds many records and may change much more frequently. For example, material-product-company master data tables may have many records and they may change by the time.

MDM tools may be a part of your enterprise system, say ERP, or bought from a vendor and integrated into current application ecosystem. MDM integration is a very sensitive issue because MDM effort is an integration effort itself. For MDM projects or implementations, reality check should be done that benefits should outweigh its cost. On the other hand, MDM is a continuous process that should be sustained otherwise we may lose all our previous efforts.

MDM overhead depends on its capabilities and integration to the system. Some of them may require repetitive manual master data selection and correction. Others may need master data replication monitoring. It is a fact that, more master data is managed by MDM system, more administrator effort is required.

We are doing MDM in our ERP systems which are located 4 different cities. We added MDM features to our ERP system for systems integration. We had following benefits from our MDM efforts:
· One Version of Truth: Duplicate code may cause many problems. A master data has same meaning for all system users and subsidiary companies. One master data may be traced throughout the systems. For example, supplier rating could be done (sharing information with other group companies), or some items in chart of accounts may be cancelled in all systems.

· Easy Data Integration: Since master data is same, systems can be easily integrated. Consolidation reports or inter-system document interchange (EDI) can be made seamlessly. Databases can be easily merged when doing data warehousing.

· Automatic Synchronization: When master data is changed in one system, this change is automatically reflected to the systems. Some power users don’t have to login and enter/update a master data in all systems since it is replicated to other systems automatically by MDM tools.

Here is recommended checklist for MDM implementations:
1. MDM integration to the applications should be high: If possible, it should be in core part of system. MDM may need some triggering events. For example, during data entry MDM system is expected to control duplicate code and return validation errors. ERP vendors have different amount of MDM support.

2. Master data that is managed by MDM should be minimal: If some frequently changing entities are included in MDM system, availability and performance of applications may suffer. Even company operation can halt in some circumstances.

3. Some master data should be entered even by users: Master data criticity is different. Some of them should be entered only by an authority (Data Governance). Some of them can be entered by ordinary users. If all master data is entered-updated with only one authority, it would be impossible to overcome this work load. Additionally, master data that users enter always should be checked for data quality concerns.

4. MDM tools don’t solve all data quality problems: User factor is important. If users enter wrong master data during data entry, MDM can’t fix this problem. User-entered master data may have invalid entries.

5. Correction of master data may be too expensive: If wrong (i.e. duplicate) master data is used in enterprise applications, it may be impossible or too expensive to fix the data since its correction side effects can’t be estimated. Company codes are this kind of master data that is very hard to change in operational system.

6. MDM gets harder when different types of ERP applications are used: Company mergers and acquisitions may change one type of ERP system implementation. MDM may be only used among similar systems. Eventually systems may be migrated into one type of system.

7. In every integration project (In SOA projects), MDM should be taken into consideration: When exchanging messages among systems, master data is a key factor for interoperability.

Resources:
Demystifying Master Data Management

MDM: A Benefits Analysis

Enterprise Master Data Management: An SOA Approach to Managing Core Information

SAP NetWeaver Master Data Management

Effective Master Data Management with SAP NetWeaver MDM

Data Lifecycle Management (DLM) in ERP Applications

2008-12-28T22:59:00.000-08:00

DLM (Data Lifecycle Management) is the process of managing data from creation to disposal. There are 3 pillars of this management; storage, applications and people. Storage is the process of persisting data (Storage device and storing application mostly DBMS). Applications change the state of data thus determines lifecycle (In our case, it is ERP applications). DBAs are the responsible personnel for data lifecycle.

Data lifecycle is a process where data changes its state. This lifecycle may be related with only a row of a table, or a set of rows, or a table, or a set of tables, or a database, or a set of databases. Some data may not participate in every step of lifecycle whereas some other data does.

Data spans following states that constitutes data lifecycle:

1. Creation: In this initial state, data is raw. Sometimes, data may bypass initialization step and be ready for usage. (Some users enters raw data, like sales order)

2. Initialization: Some data needs some extra processing in order to be used. In this phase, data is prepared for usage.(i.e. some users approves some records, managers approves sales order)

3. Usage (Consuming): Data is ready for consumers. Data may be used and referenced by some other data. (Approved sales orders may be referenced and used in sales invoices)

4. Usage Suspension: In some circumstances, data needs to be suspended to block usage. Data is in suspending mode and may return to usage or termination state.(i.e. an inventory item may be passivated temporarily)

5. Usage Termination: Data usage is terminated and data is ready to end. Data can’t be referenced and used. (i.e. a production order is cancelled)

6. Ending Usage: Data is destroyed and removed from the system. This step may require termination step. (i.e. customer master record is deleted)

7. Re-usage Again: Finished, terminated or ended data may be reactivated again. Removed data can be restored again. (i.e. Deleted data may be restored again or archived sales order may be reloaded to the active database again)

8. Closing Usage: Data is closed to usage. During this interval, data can’t be accessed and used. (i.e. sales invoice is locked and can’t be used by other users until release)

9. Finishing Usage: Unused data is moved into other systems. In some cases, reloading may be needed. (i.e. accounting receipts are archived and taken to archive system)

State machine of data lifecycle is as follows (only main transition paths are included):
This basic state machine may be combined with each application’s own defined states. Every ERP application module may have its own system states or user-defined states that determine data flow and lifecycle of data. DBAs may intervene in some conditions and change data state. Sometimes data may be in a corrupt state and DBAs manually correct data.

It is notable that objects have similar lifecycle in memory. Java objects have their own initialization and termination processes that are managed by JVM.

Strategic Java Technology Management

2008-12-22T21:50:00.000-08:00

“Strategic Management” is one of the leading management practices in business world. “Strategic Management” begins with the understanding of what the strategy is. Strategy is a long term planning to achieve our goals by gaining advantageous position in the future. By applying strategic management we can answer following questions: How can we utilize strengths and overcome weaknesses? What are opportunities and threats? How can we successfully manage future risks? (SWOT Analysis and TOWS Matrix)

As a Java development team, we should apply strategic management for Java technology evaluation and selection. Which Java technology should we use for current requirements without losing in the long term perspective? Which Java technology investment has the best ROI? What is the visible and invisible cost associated with a technology usage in the short or long term? Is a technology a bubble or not? Is a technology mature enough to use in mission-critical software?

We made strategic Java technology management for our development environment by evaluating each Java EE technology for our long term success. Following evaluation is valid for our case and may differ in every software company.

Java Technologies we use:
JDBC: Very powerful and exists beginning with early Java releases. JDBC has simple and very powerful API. It is widely supported by every RDBMS.

JSP&Servlet: Matches the base and common requirements of web programming. These technologies supported by all Application Server vendors. APIs are stable and not changed over versions.

JavaMail: Its default provider meets most of our requirements. We even developed a simple email web client by using JavaMail API. Its API is very extensive.

XML API: This is a W3C interface and we are using our application server Java implementation library.

Java Technologies we didn’t use:
JSF: Although this is a new technology, we don’t plan to replace our JSP pages. Component model in UI kills declarative programming and its benefits. New tag syntax adds additional learning curve. By losing design-time capabilities, web designer tools (WYSIWYG) can’t be used efficiently like Macromedia Dreamweaver.

EJB: Programmatic model is not architectured for our needs. Its complexity, performance problems and hard-to-use nature caused this decision. We use our own ORM framework which is very simple and easy to use. This decision saved us from many problems.

JMS: It requires a provider package and may introduce additional license cost. Even if we had bought one, it is managed with a new admin GUI and this means additional maintenance cost. Its generic model doesn’t meet our requirements. We developed our own messaging framework that is simpler and much more integrated than JMS API.

Web Services Technologies: Since web services are still evolving, so Java web services technologies will change over time. We may use web services technologies for our integration requirements.

The software products we use (We may also use or support other products if needed):
WebSphere: Very successful for both in development and deployment. Its management is easy. It is very integrated with development environment (VAJ or RADW). It handles many user requests very robustly. It supports many server platforms thus achieving Java’s “Run anywhere” motto. Java EE technology implementation is very good and compatible among versions.

DB2: Very elegant product for DBMS requirements. It is one of the two market leaders in DBMS segment. It seamlessly runs with WebSphere. It is very stable over versions. As your DB2 learning curve continues, you will explore that how comprehensive DB2 is.

Internet Explorer: In our early stage of development we planned to support other browsers but we changed mind and decided to support only IE. Otherwise we have to add new codes to our JS libraries thus raising complexity and losing IE-specific features (HTC files, Dialog windows etc.). Since our web applications are not used like web-sites, we don’t need to support other browsers. Our users are already using IE coming with their PC. IE versions are very compatible and we don’t need much tweaks after new versions.

Here is a very good article from WebSphere architect for strategic Java technology management:
Developing long term strategies for using Java EE technology

Web Application Performance Engineering

2008-12-17T22:06:00.000-08:00

Performance is definitely one of the most important non-functional requirements in web applications. When we were about to select web architecture for our applications, my primary concern was performance. Would our users be satisfied with JSP page performance? Would it be fast enough to download pages over the network every time? What would be the most critical performance bottleneck that we meet? Could we overcome concurrency issues?

As we developed our applications and used across the systems, we made many performance tunings in many layers of systems. We eliminated many problems. Final problem was “fast data entry” requirement (Like Excel) that we developed and used AJAX-enabled grid component. This was a UI design problem but users perceive this as performance problem since user task completion requires many clicks and keystrokes. This shows that performance improvement begins with application architecture.

“Performance Engineering” is composed of following steps:

Measure: “If you can’t measure it, you can’t manage it” This is true for performance. If you can’t measure the components of system you can’t take right decisions. We must avoid blind guesses. For measurement purposes, there are many monitors to use like System monitor tools, Database Monitors, Transaction Monitors, Application Session or Thread Inspectors etc. In this step, we must answer the critical question “What are we going to measure?”. This question could be answered with monitoring from general to detailed, from system to component. Sometimes "there may be many participating bottleneck points that cause the performance problems (Ripple Effect).

Analyze: After getting some hints from monitors, we should elaborate for solution. Performance analysis requires knowledge of each system layer from disk IO to network, to browser caching system. After finding the cause of the problem, we move next step to develop a solution.

Improve: In this step, we develop solution for performance problem. Sometimes this may regards to code refactoring, or sometimes Database parameter tuning.

Test: Performance improvement should be tested within a test system that is as much as similar to production system. Only one parameter should be tested at a time if possible. Test measurement is another important topic since it shows the result of enhancement. This step is the last step for performance development.

Deploy: It is the step that the performance solution is applied to the production system. If previous steps are correctly implemented, success is much more probable. During deployment, only one performance parameter should be applied at a time to make sure that if the solution works.

Performance factors of web applications are different from desktop applications. One example is browser and JavaScript performance.

Here is the hardware list that performance depends on:

1- Server Hardware: Most important factor is server hardware since this is the place where most of processing is done. Here, there are 3 major component that affects performance; Disk, CPU, and RAM. Disk RPM’s and caches should be good enough. CPU count should be good enough. RAM should be as much as application requires. Workload may be distributed to many servers with some topologies for load balancing purposes.

2- Network: Server Ethernet, Network Switches and Hubs should have good speed. Network switches and hubs should be well configured so that network packages transmit within optimum route. Any intermediate server (Like Proxy Server) between server and client should not be overlooked for network performance evaluation.

3- Client PC: Client PC should be as strong as to meet client application needs. Sometimes some thin applications may require fast CPU’s as well (i.e. Displaying long table lists in browsers).

Here is the software that performance depends on:

1- Database: Most important factor if applications are database-driven. OLTP systems run many queries for both modification and reading. If database is not tuned according to application architecture then performance problem is inevitable. Database performance tuning has 2 aspects; one is DBMS tuning the other is database tables-index tuning which is very related with applications. DBMS has many parameters that are very dependent on each other and must be carefully altered. Index tuning is the hardest and most profitable part of tuning. It may affect many applications performance.

2- SQL: Application SQL’s are closely related with table-index tuning. SQL should be built and used by testing their access plan. Index usage should be high and table scan should be minimal. For many modifications, it is ideal to use batch SQL’s. Stored procedures may be used for performance goals minimizing SQL compilation (Static SQL against Dynamic SQL).

3- Application & HTTP Server: Applications runs on Application Server. Application performance is closely related with Application Server tuning. Some Application Server is faster than others. Server response time is very critical performance indicator. In addition to this, HTTP Server should be tuned for fast response. HTTP Server and Application Server communication is important.

4- Application: Application logic should be efficiently coded to increase performance. Concurrency architecture may also cause problems. Memory and object usage is another very important issue that may cause performance problems. Application design may cause problems that same query may run many times unnecessarily. Wrong API or Framework usage may cause problems. Transactions may cause problems (Isolation Levels, Deadlock, Commit-Rollback Leaks). Profiler tools may be used for runtime analysis. Code inspection can be done before performance problems emerge.

5- Browser, HTML, JavaScript and CSS: Browser rendering is very important for responsive applications. Some browsers or versions render pages very fast. Browser JavaScript engine is very important for JavaScript and CSS code execution. JavaScript and CSS should be used efficiently so shat pages are loaded and rendered very fast. Resource files (CSS files, images etc.) should be cached if possible.

It is hard to find general performance related books since every software architecture is different and requires different tuning techniques. For web applications, for better performance one should read Database tuning guides, SQL optimization techniques, Application Server Tuning Guides, Java and JavaScript Code Performance Books.

Criticism of Java Persistence Frameworks

2008-12-02T04:06:00.000-08:00

I have been working on persistence technologies for 8 years. I entered the persistence technology domain at my first company. We firstly used WebObjects’ Enterprise Objects framework (ancestor of all ORM libraries) for our projects in 2000. It was a very good framework and it was adequate for many persistence needs. We then began to develop our own web framework including persistence layer to make a software product to provide “fast application development for web applications” (this motto repeated across years in many places). After economical problems our project cancelled and then I transferred to another company in 2001 as Project Manager for developing new ERP software. I again take the responsibility of developing web framework for ERP applications. Its most important part is surely persistence layer. We developed the framework and used in ERP applications. As our in-house ERP software is developed and brought into use in many servers, our web frameworks matured and evolved. This is the journey that I am involved in persistence technologies.
During that time, EJB era ended and ORM frameworks popped. Many web applications emerged for the promise of fast or simple web application development. As our own development has continued across the years, I observed the emerging frameworks and inspected their documentation from books or articles. Let me share my thoughts about persistence solutions in Java world. Besides all criticism I state here, I appreciate the work and effort behind all these solutions. I know that it is very hard to bring these solutions into widespread use and support it.

It is a fact that we need a persistence solution. Most of applications are database-driven and need to access tables, modify data. Although there are many questions about persistence frameworks, we need them a lot when developing applications. Otherwise SQL codes scatter across applications causing an ugly and unmanageable application. Persistence frameworks build a database layer and separates concerns. Java Persistence solutions are divided into two branches. One is Java standard solutions; JDBC, EJB, JDO. The other is ORM (Object-relational Mapping) frameworks; Hibernate, Toplink, Enterprise Object Framework, iBatis etc.

Let me ask the basic question that everybody somehow asks himself “Isn’t there any other solution to persistence problem simpler and more powerful?” I have been reading many problem news about this issue. Last one was Sun’s attempt to simplify EJB. In fact, this is why new frameworks will never end. Followings are just some samples:
http://www.amazon.com/Expert-One-One-Development-without/dp/0764558315
http://www.codinghorror.com/blog/archives/000621.html
http://www.web4j.com/Java_Web_Application_Framework_Overview.jsp
http://www.theserverside.com/news/thread.tss?thread_id=37121

Instead of comparing every solution, I am going to describe most common pitfalls:

1- “Object-Relational Impedance Mismatch” was not solved appropriately:
This is the main problem. Everybody knows it but it still hasn't been solved completely. “Mismatch” is heavily lasting even in simplified-lightweight ORM frameworks. Is this inevitable? No, we can solve it but I think our attitude is wrong. We are trying to carry relational world into object world which will never occur. Objects and database technologies are different and should be left separate. For example, SQL is a very dynamic language that you can’t state it with objects. What we see is QUERY, JOIN, WHERE objects in ORM frameworks to re-implement SQL again in object world. We made this mistake in our first ORM framework but corrected in second one. The magical clue to achieve this is that not object-only or relational-only solution is true. We need a balanced mixture of those technologies by using objects when object usage is true and by using SQL when SQL usage is true. SQL abandonment is the major problem of ORM frameworks. Hiding SQL operations of persistent objects are problematic. This is why your favorite framework’s PersistenceManager class (source code) is complexity champion.

2- Frameworks didn’t suit application architecture needs (Web or Desktop):
Your persistent object model should be compatible with your application architecture; application event system, application GUI components, application flow or navigation. I wonder how much rich applications will be succeeded using current persistence frameworks. I am still skeptical about using those frameworks in complex applications. Maybe it is succeeded but could we maintain with minimal cost? Are transaction boundaries, transaction rollbacks, data validations, lazy loading mechanisms are running without problem? Across the years, I still see the same demo a simple data list and simple command buttons to add/update/delete records (Where is grid, detail grids, tabs, detail tabs etc?). That is not the applications we are trying to develop, we are expected to develop much more complex applications.

3- XML usage should be immediately abandoned:
SQL, Mapping and descriptions are held in XML files. This is the most bleeding mistake done through the years. When annotation is added to the Java, eventually XML files are abandoned. Why do we need adding another touch-point for developers decreasing development speed? Couldn’t we add them to the persistent classes without waiting annotation support? Edit hundreds of XML files without any real benefit. I remember that one of persistence solution documentation says that transaction boundaries flexibly determined via XML files. Who needs flexible transaction boundary changes, I don’t need any. Who need flexible mapping? I don’t need any. Could you change your fragile application without touching any code? Impossible. Field change is flexibly done. Does anyone remember field name or type change doesn’t change application code?

4- Database Metadata was not utilized:
Before going to database, persistent classes should validate its data. This is not the case for many persistence frameworks. We had developed generic validation rules that are executed every time when user enters data like type, range, format, length, security (i.e. SQL injection) control. Any specific validation rules can be added to persistent objects similar to database table constraint rules. There are many other usages for metadata. In some of persistence solutions, there is no API to access to metadata of persistent objects other than database metadata.

5- Object Query Languages are a wrong path:
For the sake of removing all SQL statements in objects, OQLs are invented; EJB-QL,HQL etc. Many classes are written to handle this new query language. Why do we invent a new language even though we already have SQL? I think the reason for that mapping SQL resultset confuses ORM object caches and make them same with ordinary objects. Let’s reinvent everything for OQL; syntax, access plan, index optimization etc. We don’t need this. If we need some changing criteria, basic Criteria API’s are enough for that.

6- SQL visibility is important:
SQL is here and will be here in the future. SQL is tried to be removed completely from persistence solutions (Named Transparent Persistence Framework). SQL must be contributed much more than its current use. SQL has its standards (Ansi 92 SQL) that should be enough for database independence if it is removed for the sake of database independence. If you want to use custom SQL, your trouble begins. SQL usage should be made simpler in persistence frameworks. It is very hard at the moment.

7- Multiple data source dilemma:
One of the major architectural problems of persistence frameworks is multiple data source architecture. This is why we still discuss inabilities of the persistence frameworks today. Small percent of possibility to use different data source added enormous cumbersome to the persistence frameworks. What are the data sources other than database? File, Network, EIS, XML, ESB, JMS etc. I don’t answer the question why we don’t leave this portion of problem to the other software. I think it is a rare condition and should be totally removed from persistence solutions. This will not endanger its generality.

8- Why persistent objects are so weak (POJO)?:
We were asking opposite question years ago for EJB “why EJB objects are so heavy?”. Today the question is converted. I think this is mainly caused with the aim of hiding generated code. I think there should be a middle way. Persistent objects may take much more responsibility. They may take validation responsibility, value object responsibility etc.

9- Poor Transaction Management:
Visibility of objects entered to transaction is questionable. When reading a piece of code, it is very hard to understand transaction borders if it resides in a XML file that you forget to read (Declarative Transaction Demarcation). We use automatic transaction management for mapped child objects freeing programmer to start and commit transaction for them. Is object state handled correctly (according to application needs) in commit or rollback condition? Is transactional memory footprint is manageable? Transactional memory is a very big problem when thousand of objects participate in a transaction. Is isolation level adjustable according to application needs?

10- Heavy Object Caches:
Object caches are subject to debate. Aren’t database buffer pools enough for caching? Garbage collector troubles when memory is loaded with many objects. Why do we reinvent database cache system again with object caches? Yes, it is sometimes useful to cache objects in bulk object creation but if it is a part of system, isn’t this responsibility of database system?

11- Wrong Locking Methods:
Using database locks for application lock needs is another pitfall. Since some processes require many screens and steps, to lock records across long processes leads to performance problems. Optimistic locks and pessimistic locks are still a missing feature. We also implemented optimistic locks as transparent which doesn’t require explicit API call.

12- Persistence API could be simpler:
I do not like API used for the persistence. JPA is candidate to solve this problem. To do simple database operations, we need to write many boilerplate code; Sessions, UnitOfWork, Context etc. Add your UI object to this list, DAO, DTO, value objects etc. Why do not we give this responsibility to persistent objects? I really don’t figure out why transaction mechanism is not used for the session architecture. I think this is why many scripting or domain-specific languages (DSL) are emerging for the simplicity.

DBEmployee dbEmployee = new DBEmployee();
dbEmployee.setName(“John”);
dbEmployee.setSurname(“Carpenter”);
…
dbEmployee.dbSave(); // inserts record here

13- Bytecode manipulation problems:
We know from JSP technology that runtime code generation and compilation is very expensive. When we add persistent objects to this bandwagon, our software system stability is at stake. Changing an object after its declaration at runtime or build-time may cause many problems; performance, testability, stability etc. Many of us already experienced JITC problems which it has similar mechanism.

14- Unnecessary OID (object id) columns in tables whereas PK columns are enough:
This is the behavior of some persistence framework that your DBA never agrees upon. Many frameworks can’t handle multiple PK and FK keys. This is a very big handicap.

15- Inheritance has no meaningful usage:
Wrong motivation is that; “In object world, inheritance is a great structure so we must use it in relational world.” You don’t need inheritance in relational world. We don’t implement inheritance in our ORM solution and we don’t even need it. Traversing your persistent object one level below is not a hard job. Even your tables may already have some repeating redundant columns.

16- Primary and foreign keys are held in accompanying new classes causing many unnecessary classes:
When considering 1000 tables in our ERP system, applications have many persistent objects. If we had created PK and FK classes, it is going to be a total mess. We used simple strings for the solution of problem. I see in some frameworks PKs and FKs are held in their own classes.

17- Missing cluster support:
Many leading Application Servers support cluster, but to use cluster with persistent objects would not be easy. First of all, sequence generation and locking system should support cluster. I am sure that to write clusterable persistent objects would not be easy. Every application doesn’t need cluster but as your application grows than scalability and availability concern increases than cluster becomes necessary.

18- Missing database events:
Like triggers in databases, we sometimes need database event support, like row inserted, updated, deleted. This is especially required if you want to add support of BPM or Workflow to your application.

19- Missing authorization:
Row-level authorization just recently added to the feature list of ORM frameworks. Database connection users also can be separated according to user roles for security.

20- Missing dynamic rules:
Some validations or calculations may change over time and can’t be stated with static code or parameter. Persistent objects should support dynamic rule engines or mechanisms.

21- Missing object modeling tools for code generation:
Most of ORM framework have mapping design tools. Reverse engineering from database tables are a common feature. We should design objects and generate persistent objects from this design models.

22- Missing audit trailer:
In ERP systems, audit trailing is a very important requirement. Because of this reason, every table includes columns like insert time, user inserted, last edit time, last user edited etc. There should be ways to fill these columns or other helper tables transparently without appending to SQL statements by programmers.

23- Missing persistent object debugging utilities:
Persistent objects state (insert, update, delete, identical) should be visible. At runtime, sometimes persistent objects need to be accessed to read last SQL executed or field values. Instead of attaching a profiling software to production system, some debugging information should be provided.

This is like a wish list and may be used by persistence framework developers. To build a persistent framework is very charming and looks like very easy but as you progress it requires a lot of resource. To justify your development investment, resources should be evaluated carefully thinking all the risks that your attempt may not produce result. Your initial framework will be used and then changing and evolving it be a real nightmare if you are not familiar with concepts and followed the wrong path.

Beautiful API

2008-11-25T00:01:00.000-08:00

When I first began to program in Java, I loved Java language a lot. I used to program in Pascal, Delphi ,Visual Basic and C but Java was very different and elegant. In addition to its language structure and features, its API set was very special. With its beautiful and aesthetic design, programming in Java is a pleasure. I don’t have this feeling when I program in other languages. To feel pleasure or pain is also valid when we use API sets. There are many API set we use in any development cycle coming from different frameworks or libraries. API beauty depends on designer knowledge and design capability (say artistic skill).

We use following principles when developing API or UI or product (I used the word “Beautiful” since this includes following features):
· Simplicity: This kind of API is simple enough to meet programmer expectations. No extra line, no extra method, no extra parameter. It should contain only what you need today and tomorrow. If you are developing API, you should repeatedly ask the question “Can I shorten API without losing functionality?
· Maintainability: If I provide this API, can I support it without any problem. Is any change make it depreciated in the coming development? Some API may endanger other part of software and some of them are much more exposed to error. When releasing an API, its usage should be thoroughly considered. Sometimes, API users may use it in some places that you never planned.
· Aesthetic: If we believe software development is art, aesthetic is its one of the main aspects. Goods or items in physical world have design and form aesthetic. We similarly have forms and designs in software not with materials but with API’s. To be aesthetic; API should be consistent, best names should be selected, understanding of function should be minimal, using API should make you comfortable.
· Functionality: One API set should solve as much problems as it can. Sometimes we see that similar needs are separately met by different API set whereas one set of API may be enough. This topic is somewhat related with efficiency. Designed API should provide maximum reusability. One API set should provide as much functionality as it can with minimum code. We read many lines of code everyday and we see that many boilerplate codes (smelling codes) are needed in some frameworks or API Set.
· Consistency: Same problems should be solved with the same attitude and way. Behaviors of your API should not change among classes or packages. You should use same pattern throughout your API set although there are many patterns for the same problem. For example your exception management style should be same in API set.

To design “Beautiful API”, one designer should take into consideration following points:

1- Keep it simple. Do only what is needed at the moment while thinking extensibility for future possible requirements as well. API should be easy to understand.
2- API designer must develop and empower himself in programming skills by reading others codes and writing codes continuously. One should learn as much as possible about topics like design patterns, API design principles and OOP principles. Skills should be certified with certification exams if possible. This gives some objective notion about skill development.
3- When designing API, the cost associated with API development, bringing it into use, runtime resource consumption, user (programmer) education and support should be carefully considered and planned.
4- Before delving into API design, technological analysis and research should be done. Related references should be investigated and inspected taking notes. Research documents should be read before starting API design. Other similar API’s could be read for better API design. If this reference research is not done, “inventing wheel” never ends.
5- If API design requires a lot of work without any real benefit, it should be avoided. This will be a dangerous situation for both API designer and user. Additionally, API sometimes can’t meet requirements as desired. In this situation, API may be suspended temporarily and a workaround may be found for the requirement. When bringing API into use, it should be monitored for “Bad Smell” taking feedback from programmer.
6- “Naming” should be carefully done. Names are the most important aspect of API design. Every name (DB Table - Column, Class, Method, Member and Parameter) should be considered as delicate as it is given to a newborn baby. Names must be searched in related literature.
7- Designed API should be peer-reviewed before usage if possible.
8- API should be tested and used in a reference implementation.
9- API can be used without any configuration or maintenance (Zero maintenance).
10- Unused API should be removed from API set. Any dead API confuses API designer and user in the future.
11- API should be documented and used as many places as possible. Sometimes its usage may be enforced.
12- API should be designed concretely that it must not change after release.

There are many good API design articles around the web. I recommend following resources for API designers:

http://www.infoq.com/presentations/effective-api-design
http://video.google.com/videoplay?docid=-474821803269194441
http://blogs.msdn.com/kcwalina/archive/2007/03/29/1989896.aspx
http://www.artima.com/forums/flat.jsp?forum=106&thread=142428

http://www.amazon.com/Practical-API-Design-Confessions-Framework/dp/1430209739
http://www.amazon.com/Framework-Design-Guidelines-Conventions-Development/dp/0321545613/ref=sr_1_1?ie=UTF8&s=books&qid=1227599123&sr=1-1
http://www.amazon.com/Beautiful-Code-Leading-Programmers-Practice/dp/0596510047
http://www.amazon.com/Machine-Beauty-Elegance-Technology-Masterminds/dp/046504316X

SOA Implementation in ERP

2008-11-16T23:59:00.000-08:00

Service-Oriented Architecture (SOA) is another buzzword in IT world. Everybody has different understanding of what SOA is. Trying to change all software architecture for the sake of SOA is another pitfall I witness. “Everything integrates without any mess and pain with SOA” but everything should be SOA-enabled. SOA-enabling all software is very long journey that hasn’t been taken yet. I don’t find the answer how the legacy systems will be SOA-enabled and what the cost we will pay for this. I read many SOA projects that are blue-sky projects trying to make everything a service without any return for the cost. This is why many projects are subject to failure.

I have following observations about SOA:
SOA is meaningful and feasible in integration (i.e. In Enterprise Application Integration (EAI)): Best place for SOA is to use it in integration functions. This integration may be at application, system or business process level. In Java world same mistake (making everything SOA) is done in EJB component architecture. Every component thought to be accessible remotely with RMI-enablement making them very heavy without any remote need. Some components may be used remotely but this is a rare condition. Eventually EJB components supported local-only options but it took years to fix the architecture problem. I think in integration spots we need SOA with web service or interface form. Other parts of software should not need SOA-enablement. Let’s say every API doesn’t necessarily be a service (i.e. web service). But if we need to integrate applications, then their integration API can be made SOA-enabled. If your application is locally used why do you need to bring its API to ESB?

SOA technologies are still evolving (Is there an end?): Web services technologies are aimed to replace CORBA but it is still evolving and changing. Sometimes same problem solution set has 2 different standard bodies (i.e. different security standards lead by IBM and Microsoft) destroying interoperability. On the other hand, XML taxonomies and vocabularies have many duplicate branches ruining common definitions. New alternatives are emerging to solve the same problem like REST against SOAP. I think some mature SOA technologies may be used as a wrapper of current defined application interfaces. Application interfaces don’t depend on technological changes. Adopting all SOA technologies and making a huge investment on them must be evaluated carefully. SOA technology changes affect platforms, products, tools and finally your software.

We implemented SOA in 2 parts of our ERP architecture:
1- Application Infrastructure Services: We know that service architecture is used in MS Windows OS. From the System Management, this OS services are registered, unregistered, started, stopped and disabled. We adopted same manner and in our ERP runtime, every application infrastructure services can be managed transparently from applications. This enables fault-tolerance, fail-over and managed services. In this way, we are using the benefits of SOA as architectural principle not the technology stack.

We have following “Application Infrastructure Services” managed from admin module similar to Windows OS. This is why we call our platform as Application Operating System (AOS):
· Object Cache Service
· DB Connection Cache Service
· Message Service
· Database Log Service
· Data Control Service
· Log Service
· HTTP Control Service
· Persistent Session Service
· Authorization Service
· Authentication Service
· Lock Manager Service
· Identity Service
· Email Service
· Profile Service
· Schedule Service
· Meta Data Cache Service
· Resource Profiler Service
· Replication Service
· Rule Service
· Workflow Service

2- Application Function Interfaces: Integration needs among ERP modules or applications are much more than any other software. Therefore every module needs to access (CRUD) other modules tables and functions (i.e. Purchase module may update Sales module’s tables). We achieved this by defining application interfaces (Adapters) without direct table access. Direct table access has problems like lock bypassing, business function duplication etc. These interfaces can be accessed from any application directly in local process. Remote access is achieved via our Messaging backbone (ESB). We plan to “web service”-enable this adapters to make them integration solution for third party integration needs.

Foundation of Enterprise Java Frameworks

2008-11-11T05:35:00.000-08:00

Frameworks became one of the hottest topics in Java community. Then, Microsoft adopted similar development architecture and later .Net platform was born (MS Enterprise Library). I am not going to describe what the framework is but try to frame the framework libraries. I think Java Framework boom starter was EJB. EJB problems lead to new seeking to find simpler and better alternatives. ORM libraries solved many problems of EJB while staying outside of the Java standard body. As time goes on, standard body changed its standards adopting these library technologies like Hibernate.

Frameworks provided many useful utilities and eased software abstraction and layering providing best practices of design patterns.

First of all, let me classify enterprise Java frameworks describing their functions (I think these are fundamental blocks and every enterprise application needs these frameworks):

1- Application Framework: Application framework contains functions and API’s for enterprise applications. These frameworks provide a run-time (like VM) for applications and hide the details of some services and functions. There are 3 main functions; User interface (View), server interaction-business logic execution (Controller) and persistence (Model). Some of them are complete application framework like MVC frameworks whereas some provides functions only for one aspect of application like UI, persistence or controller. It is very hard to find a complete and fully-fledged application framework. Today these functions are scattered to many frameworks thus hardening integration.

2- Messaging Framework: Messaging framework enables to communicate different process or servers in synchronous or asynchronous way with tightly or loosely-coupled manner. RMI and JMS are examples in Java. Depending on requirements, RMI or JMS could be used. JMS needs an implementation library as well. ESB (Enterprise Service Bus) is just a new acronym for this framework type. Web services and EDI (Electronic Data Interchange) can run with this framework functions.

3- Reporting Framework: Reporting is laying out the data for printing or publishing purposes. Reporting frameworks provides layout arrangement, paging and printing to different printers with different document formats. There are many open-source or commercial reporting frameworks with different capabilities. Many of them developed their own proprietary report writing script languages and XML formats with an API set.

4- Utility Framework: There are some special needs like XML parsing, String manipulation, source code generation, etc. These requirements are met by utility frameworks. A well-known example is Apache Commons.

Frameworks have following aspects:
1- API: Every framework provides API’s. Maturity and simplicity of these API are very important. There are many other characteristics of good API which will be held within another blog post. Frameworks are tightly-coupled application blocks. API is glue of these blocks. API is the most important aspect of frameworks which determines success.

2- Tools: Frameworks must be supported with tools. Sometimes frameworks are not usable in the absence of good tools. Tools depend on frameworks, any major code and API change brings many pains to tool writers. Tool support is one of the drawbacks of framework writers since no time remains from framework development jobs. One of the tooling misunderstandings is IDE dependency. Tools don’t necessarily require an IDE, for example code generators can be independent applications.

3- Run-time Management: Frameworks haven’t come to this level yet. I see only a few JMX support but this is not enough. On the other hand, JMX support is very hard for framework developers (Only Application Servers could use JMX). This job is left to profiler vendors but profiling in a production system is nearly impossible. Which job is running using which object through which connection can’t be answered in a running system, I haven’t encountered such tool yet. Session inspection, thread management and connection pool management do still not exist.

4- Standards: With so many options, developer toolbox is flooded with frameworks. Keeping standards and consistency across developer ecosystem is another important issue. If frameworks are utilized with careful planning, we can build and keep standards across programs and projects. Otherwise unmanaged development environment will be costly.

Frameworks boosted developer productivity by encapsulating functions and dividing expertise zone. Framework selection is a key part of any development project. Frameworks should be evaluated very carefully with the above aspects. In some circumstances, “Build” decision may be taken if we have enough resources (time, expertise). This was our case and eventually our libraries matured meeting ERP application expectations. Our frameworks turned out to be an Application Operating System (AOS) for web applications.

Simple and Practical Model Driven Architecture (MDA)

2008-11-03T23:18:00.000-08:00

One of the leading tenets of MDA is its help to easily understand systems thus lowering complexity. As I read over the literature about MDD and MDA over the years, I am amused how it turned out to be another factor favoring complexity instead of diminishing it. I think OMG ‘s MDA initiative is not successfully utilized. This may be the result of problems on specifications or implementations.

We utilized MDA in our ERP platform different from what OMG offers. First of all, I want to describe the problems of current MDA usage then I am going to show MDA practical usage in our ERP architecture.

The problems I see are as follows (I am surprised that similar problems are listed even in wikipedia entry):

OMG’s MDA specifications are not usable by every software company: Small software companies can’t afford to implement specifications such as MOF, UML, XMI and CWM. PIM and PSM should be only used as an abstraction layer not a specification set. I am not sure about tooling vendor ability to implement those specs and its benefit to the companies both vendor and user companies.

Overuse of UML again: MDA can be used without UML diagrams. Let code remain as code. Forcing to describe software artifact in UML adds another complexity layer. Some artifacts may be in document format, or SQL script format or reverse-engineered DB Table diagrams or code. As productivity is a concern in development, we have to throw repetition with the hope we will use in the future that will never happen (How many times do we need code generation to both .Net and Java? This may occur rarely). I think to bring understandability we can’t victim simplicity and productivity.

Where is “Data Model”: In enterprise software, data model is the most important part of system. In MDA initiative, “Data Model” is blurred into PIM and PSM. In enterprise applications, isn’t “Data Model” just the modal we mean in MDA term? I think to make association is not easy and that causes some confusion. I think this is the reason that Domain Driven Design (DDD) emerged.

We utilized MDA but in a very simplistic form. Let me describe our case:

1. Take DB Tables as Model: In our approach, everything begins with database table design. One of the artifacts of analysis team is tables and programmers uses these tables as their development models. DB tables serves as PIM and programmers make many transformation (code generation) into PSM like DAOs, JSPs, Servlets, ReportFiles, MobileCodes etc.

2. Generate Code from Models (Tables): Every application (ERP module) uses a set of transactional tables. These tables are designed according to business requirements. Created tables and their relations (Physical) are used as without UML representation. UML may remain in analysis documents but not in coding phase. Because table structure and coding is much more converged. As near as your efforts close to database, as much as your application success increases. This is why we preferred table-first approach instead of object-first approach. Tables are much more important than programs in enterprise applications since they hold precious data. We generate a complete enterprise application (module) from table models (MDA Transformation). This is the final point in MDA premise (Executable UML). What our programmers do is just adding extra custom requirements which are very little at the early stage of software lifecycle. Re-generation is also supported by our tools to cope with changes. I think generation success depends on API used in generated code. We use minimalist approach, as little as generated code to provide understandability and change ability for programmers. Our frameworks’ API’s supports this minimalist approach.

3. Use Data Model Patterns: We are using best practices of ERP table designs. Many enterprise applications use the same patterns. In any application design, we search for patterns before finishing our design. Order, Order Line, Processing Option, Product, Company, Item, Part, Product, Material are just common patterns found in any data model pattern catalog. If you don’t build your data model correctly, then development efforts can’t produce results (whatever tools, methodologies and architecture you use). Correct data model is just an output of healthy business analysis. ERP’s are founded on experiences of years of analysis.

I read many articles stating strong concerns about MDA. I think “modeling” is a mandatory part of system design and its formal definition is a good thing. As software builders, we must be much more result-oriented and focus on utilizing concepts (Principles) rather than getting lost in specifications (Specification evaluation is another topic). Concepts live longer then specifications and standards.

Effective Application Lifecycle Management (ALM)

2008-10-28T06:12:00.000-07:00

Production companies use ERP as process automation software. In software industry, we should use ALM software to manage software processes. Let’s dig into how successful we are at that point. Sometimes ALM term is interchangeably used with Software Development Lifecycle (SDLC) but I think ALM is a broader term which is not limited to only development. I give you an example from ERP world. If ALM is ERP of software companies, SDLC is only manufacturing part of ERP (Shop Floor Management). ERP enables companies not only manage manufacturing but also sales, purchases, marketing, materials, supply-chain, customers, stocks and products. Product lifecycle Management (PLM) has also similar features with ALM in terms of scope. PLM is just a part of ERP systems. My last example is Software Factory or Software Product Lines methodology which is inspired from manufacturing factories and their process management. There are many similarities between production-manufacturing processes and software processes which are beyond this blog entry scope.

Do we really use effective tools for ALM today? Are there really qualified applications for software automation? As every ERP vendor built their own tool-set, we ended up this end. Finding a satisfactory ALM suite is really hard or impossible. Every software house should not build its own toolset which requires lots of resources. On the other hand, there is no single ALM toolset to manage all processes. Using different tools for different processes has many difficulties and problems. “Software Engineering” remains only in our processes but couldn’t be automated good enough as it is desired with ALM tools.

Let’s try to capture what are the problems of today’s ALM tooling:
1- Software process management tools are not integrated: We need an “Integrated Application Management” suite like production-manufacturing companies needed years ago and ERP had emerged. Integrated Development Environment had emerged for similar requirement. But today’s requirement is much wider. Application lifecycle begins from idea into project then to modeling. It continues with development, release, deployment, maintenance and configuration (SCM). What we have today is separate products for every phase. There are many efforts to integrate with IDE adding for example requirement-bug management but has problems and limited. I think we need a mature end-to-end ALM product defining process borders and close to technologies that the applications are running.
2- Existing tools are not mature (or not existing): Existing tools are not mature and not functional well enough to gain widespread usage and popularity. There are many creeping technology waves causing re-write and disallowing advancement in true direction. Luckily we see some build tools emerged recently but no release management at the moment. Reaching to the SCM area is still a long way to go.
3- ALM is thought separate from DLM (Data Lifecycle Management): Many enterprise applications are data-intensive. DLM is left to Database Administrators (DBA) and its importance is neglected. What is the relation between DLM and ALM? Every software release includes database structure changes, data transformation operations. Application run-time behavior is adjusted with some parameters. It is a common conflict that occurs between DBAs and developers about data modeling (Application versus DBMS requirements). Especially in data-centric applications, DLM should be considered earlier in software process (this is also generally an afterthought like security). Data archiving is a branch of DLM and closely related with application design.
4- Framework and Tool Islands/Silos Floating In Enterprise Sea: As enterprise ALM needs are not met by a unified solution, granularity increases and causes complexity and management dilemma. Most of readers should have experienced many integration and management problems of their frameworks when trying to use with Data Modeling Tools, Code Generators, Built Tools, Test Tools, Deployment Models and Profiling Tools etc.
5- ALM should not be a documentation tool: Adding another burden on developers is not true for the sake of quality without real benefit. ALM is not only the drawing of requirements, models, object interactions, execution with UML. ALM should provide requirement traceability that each requirement should be easily related with release plans, release notes etc. Data Models should produce database configuration scripts. Application Generators should be used for every part of software with appropriate framework API’s. Otherwise ALM tools will not be effective.

Since ALM is a broad term, let’s try to define borders and how ALM tools can be in these areas:

1- Project Management: Software projects should be tracked with this tool. This is used for entire lifecycle of projects from development to implementation. Project resources can be managed with this tool (Time/Budget/Scope).
2- Requirement Management: Once project and product requirement specification (SRS) is documented then these requirements should be loaded this tool for future traceability. Requirement planning and priority assignment is done here. New requirements may emerge as the software is taken into use.
3- Modeling: Project or product is defined and modeled in software design specification (SDS), this information is used to model application. For DB applications, this is the data model and behavior of applications (functions).
4- Development: Editing/Debugging/Profiling/Testing environment for software development. This is commonly an IDE.
5- Testing: Some tools for quality team. Any type of tests should be supported (Not only unit test but also integration and stress tests must be included). Automated test is final point to reach.
6- Release Management: After tests are completed with acceptable results, application is ready to package. Package and Deployment tools are needed for version and change management.
7- Documentation: Shipped applications should be documented with tools. One example tool is JavaDoc. Packaging tools should support document inclusion into packages such as help documents, release notes etc.
8- Deployment (Configuration Management): Application should be easily deployed to the target system without pain. Upgrade or new system installation should be done to the running system with acceptable or zero down-time with automated tools. For example it is error-prone to apply database table changes manually.
9- Runtime Management: Applications run-time should be easily managed. Application processes, threads, memory usage and connections should be manageable (start-stop-terminate-kill-allocate-deallocate).
10- Performance Management: Performance is one of the biggest problems of enterprise applications since these applications are composed of many related tables. Java memory management is another performance hot-spot. Tools should help us to find bottleneck without compromising system availability.
11- Bug Management: As the system goes alive, users find many bugs. Sometimes, bugs are reported only in system logs. These are all captured and should be managed and corresponded with new software fix releases. This must be corresponded with quality management.
12- Data Management: Application data requires maintenance like erroneous data correction. Each application data correction may need different procedures. Some data can’t be changed after some status. Another topic is master data (MDM) may need to be replicated across systems to provide consistency of data.

Today, we don’t see a complete ALM suite that solution set spans all of these areas. Middleware companies mostly provide development-oriented tools. ERP vendors’ software platforms are not generic enough to use out of ERP packages (ERP Lock-in). If we intend to build a “Software Factory”, we should have as much as automation that manufacturing factories have today. With ever-changing software technologies and trends, this is really a difficult goal.