How to Mavenize big, monolithic legacy Java projects

Last year a customer asked me to modularize a rather critical legacy Java system. It has grown over the years and was based on a rather monolithic Ant based build process. The architecture of the system clearly defined layers and components, but this architecture was not supported by the project layout. Now, due to the monolithic layout the well-defined architecture became more and more corrupted, because nothing in the build process really enforced the rules and restrictions of the architecture. My customer did not feel comfortable with this situation and decided to invest some time and money in a major refactoring of the system. The goal of the refactoring was to break up the monolithic project structure. Instead, multiple smaller modules with clearly defined dependencies on each other should be constructed.

After some investigation we decided very early in the refactoring process to move the whole thing to Maven. This was not an easy decision. If you decide to use Maven, you have to adhere to the conventions and rules defined by the Maven build process. Projects that have their own, Ant-based build process typically do not follow any other rules than their own. So converting such a project to Maven can be a daunting task. After some prototyping I was sure that I would be able to accomplish it anyhow, and here is how I did it:

A very good starting point for the refactoring was the existing software architecture, which defined layers and components. The goal was to more or less define one separate module per component. Additionally, some infrastructure modules were necessary that capsuled for example the configuration files. However, even with this basic strategy in mind, the practical transformation of the one big project to about 50 modules is a complicated thing. The main problem are the dependencies between the various modules. In a Maven POM, you explicitly define all dependent modules. If this dependency set is incomplete, the resulting project won't build. If you only have 2 or 3 modules, you can figure out the dependencies by try and error. This won't work in a reasonable time frame for 50+ modules.

Another pitfall are cyclic dependencies. Maven won't build projects with cyclic dependencies between modules. A cyclic dependency occurs for example when Module A is dependent on B which again is dependent on A. The problem is that in most cases the cycle is not so easy as in this example. Typically, you have something like A->B->C->D->A. Since Maven resolves dependencies transitively, it will detect this cycle. And immediately cancel the build (which is perfectly all right IMO, because cyclic dependencies are a bad thing and should never occur in a well-structured project).

At short, I was faced with two problems. First, I needed a possibility to cut the big project in the "right" modules and to find out the dependencies between the modules. And second, I had to identify and break up any cyclic dependencies between the new modules. To work efficiently, all this must be done in a kind of simulation, before actually performing the refactoring and breaking up the project.

Luckily there are some tools out there that are able to support these kind of tasks. I chose SonarJ for this specific project. SonarJ allows you to define a target architecture for a given software system. In my case, I defined the planned 50+ modules as target architecture. Next I defined the target dependencies between the modules using a graphical modelling tool within SonarJ. The cool thing is that SonarJ is now able to identify all modules, packages and even classes that break these rules. Often the violation are caused by wrongly placed packages (someone placed the code in component A, but this is simply nonsense, it belongs to component B). Now, I was able to fix any errors by virtually moving the package to the right component. This movement occurs only in the architecture model of SonarJ, no source code is changed (yet).

Additionally, SonarJ has a very powerful tool set to identify and break up cyclic dependencies. In most cases, the break-up is also done by virtually moving classes or packages around in the model. Of course, some cycles are caused by real "bad" code that needs some thorough refactoring. In these cases, one can mark the culprit classes on a todo list with some additional notes for later, manual rework.

As soon as my SonarJ model was fully defined, all cycles resolved and all dependencies defined, I "only" needed to actually perform the refactoring:

1. Create new Maven modules for all modules defined in SonarJ. This task included the creation of the necessary parent modules that group modules of the same type. As a result of this task, a project structure with all Maven POMs was created, but still without any content (source code, resources, etc.). Of course, the newly created POMs included all dependencies defined in SonarJ.
2. Copy all source-code files, resources, config-files, test cases etc. from the old monolithic Ant project to the correct new Maven module.

For a project of this size one cannot perform these tasks manually. Since I am a big fan of Groovy, I decided to write some scripts that interpret the SonarJ model file (which is luckily a well-defined XML-file). The first set of scripts created the Maven module structure. To keep thing as simple as possible, I first created a set of simple Maven Archetypes that contained empty templates for all needed module types. The script itself simply iterated through the SonarJ file and invoked the "right" archetype with some parameters. The actual creation of the directories, POM files, etc. was performed by the Maven Archetype Plugin.  A second set of scripts performed the copy tasks using AntBuilder. AntBuilder is incredibly powerful in copying things and hence the script fits on a single page.

Basically I wrote a generator that interpreted the SonarJ model to perform the refactoring. This approach allowed me to easily try out variants or fix problems in the model that showed up after the generation. After I finished everything I realized that this was the first time I used a model-driven approach successfully! I'm still not a fan of MDSD, but hey, in this case it really worked OK. Of course, it was not real software development but "just" a refactoring with some Groovy scripts, but I have the feeling that any other approach would have been much more painful.

Pros and cons of programming language inflation

Here in Germany a quite popular developer magazine called "JavaMagazin" is published once a month. I'm a regular reader and a few days ago the latest issue was in the mail (yes, the "real" mail :)). The headline on the front pagewas about Clojure, a quite new language for the JVM. A few issues before, the headline was about Scala. Also a quite new language for the JVM. Again before it was Groovy. Personally I get the feeling that some kind of programing language inflation takes place. The question comes in mind if this makes sense or not.

I've read several posts and articles about this phenomenon. Sometimes it is discussed in conjunction with the term "polyglot programmer". The idea behing this term is that you as a developer should learn permanently new languages. By doing so you are supposed to somehow look behind the scenes and get a better programmer by learning from the different abstractions that are used in the different languages. This sounds not bad to me. If you are a skillful programmer you probably are able to improve your skill level this way. You probably could do so by reading books about programming concepts, too, but learning new languages is definitely more fun.

Additionally, the concepts of new languages are sometimes really able to improve the quality of our work. This happens, in my opinion, when a new language is able to transport a theoretic concept out of the computer science ivory tower to real world. In my opinion, Java did exactly this for object oriented programing (and some other concepts like garbage collection and virtual machine execution). Now, if people invent languages like Clojure, they enable programmers to easily use the concepts of functional programming in their projects. Since Clojure is based on the JVM it has a lot more chances to really succeed in this task than, for example, Lisp. Who knows, maybe in ten years from now the majority of the development community has moved on to functional languages. One possible reason for that move might be that increased parallelized hardware simply called for new programming concepts. From this point of view, the invention of new languages is a must and absolutely makes sense.

On the other hand, the inflation of programming languages is a massive problem for businesses that consider software as an investment. Even if a new language decreases the overall costs of a new software system by, lets say, 20% (which is a LOT and not realistic), the savings will be eaten up in the maintenance phase quite fast. You'll need either more people with different language skills or the same amount of people with increased skills. Both variants are more expensive that maintaining systems just in one language.

Today its easy to focus on Java or .NET or even C++ as the main programming language in a company. At least my customers try to focus their IT investments this way. The more programming languages evolve, the more difficult it is to enforce a focused technology selection. A lot of discussions pop up around the language selection process right now and these discussion will be more intense the more languages enter the market. From this point of view, language inflation is anything from positive. It makes IT projects even more complicated and risky than they are right now (risky, because you easily could choose the wrong language for your problem domain).

As a software architect I cannot simply put away the arguments of the people that are focused on the figures. Developing systems is fun, but at the end of the day they are just investments. So here's how I handle this dilemma. I agree with people that argue that a developer should learn new languages regularly. Probably not a new language a year or even more - I simply am not able to handle this. But every two to three years I try to grasp one of the popular languages and see what happens. A few years ago I did this with C# an threw it away afterwards. It simply is not relevant for my business. The last time I chose Groovy and this was certainly one of my best decisions lately. Groovy has saved me tons of times in several projects since then. Just if you are curious: I never used Groovy for a production system until now. But I use it regularly to pimp my development infrastructure. This infrastructure is only needed during the development phase, not for maintenance. If something of the Groovy stuff is really relevant for maintenance, I convert it to Java before shipping it to the customer. I do this because I believe that in maintenance phase things must be simple to be cost efficient. Two programming languages (Java and Groovy) for one system are not simple, hence I avoid this.

Now, if someone argues we should use a brand new language like Clojure for a given project, because it is cool and certainly better than good old Java, I am very cautious. Java is a suitable language for most requirements in IT business I am aware of, so introducing something new for productive use must be evaluated very carefully. Typically, this evaluation results in keeping Java. But if the arguments are valid, like, for example, the need for massive parallel execution of the internal processing, I certainly would not disallow the usage of Clojure just because it is new.