Updating or Starting Spring 3.0 Project

Monday, March 1, 2010

On December 16, 2009, Spring framework version 3.0 was finally released.

For those with Spring projects already in progress, as well as those starting new projects, a number of new features are available, and some changes must be made to take advantage of them. It is strongly recommended that the full release with documents is downloaded as the changes are sweeping and sometimes drastic. That said, you shouldn’t need to do much differently than before to get the same stuff out of your Spring applications. At the very least, a slew of compiler warnings, if not errors, will be introduced if you just take a “replace the JAR files” approach to upgrading as many of the Controllers and other elements have been deprecated or otherwise reworked.

Downloading the full release with documentation brings forth a new 800-page reference document, which we should all read, but we know not all of us will. This article will endeavor to point out some pitfalls and help make a quick conversion of an existing web project to the new paradigm as well as try to show how to quickly make a new web project from scratch using the new Spring 3.0 framework.

Swing by the Spring download page (http://www.springsource.com/download/community) and grab the release archive, if you haven’t already. Expand it so the files will be at the ready. If you grab the with-docs version (which is recommended at least once), peruse the reference documents for areas you expect to have trouble.

The assumption made is that you’re at least familiar with Spring enough to recognize some of the generalities and translate them to your own project; a word of warning–this isn’t intended to be a “learn Spring” tutorial.

Existing Project Upgrade

This bit assumes you’ve got a Spring 2.x project and you’re looking to upgrade it to the new framework. Presumably if you’ve been developing with the Spring 3.0 beta and release candidates these problems have already been addressed; perhaps not, though, or perhaps there’s some clarity you seek. Hopefully this section helps in either case.

In your project, we’ll assume a standard structure of a source directory for the Java and such, and a web content directory for the web-served bits. In the web content directory lives our magic WEB-INF folder, and in the WEB-INF/lib folder is where the Spring JAR files are located. If your configuration is different, you’ll have to translate, and should probably consider this more concise approach instead.

In a Spring 2.x project, the spring-beans.jar, spring-core.jar, spring-web.jar, and whatever other Spring JAR files you’ve been using will need to be removed to avoid conflicting with the new framework files. Remove also any .tld or .xsd or other files that may have been copied from the old framework folders into the project. A flurry of compiler errors should be evident as we’ve removed core classes required by the project. Don’t worry about correcting any of these just yet as we’ve got to put the replacement JARs in place, which will remove most of those errors, and give us just a few warnings in their place.

As we look in the new Spring 3.0 archive’s dist folder, we see immediately that the file naming format has been changed. For the most part, once you get comfortable with the new format it starts to make sense. In most cases the classes have been packaged inside the JAR that starts with the same package name. For example, the starting point for nearly every Spring web application is the DispatcherServlet. The DispatcherServlet is in the org.springframework.web.servlet package. There should be a org.springframework.web.servlet-3.0.0.RELEASE.jar file in the dist folder. In the Spring 2.x framework, the DispatcherServlet is in the spring-webmvc.jar file. Other classes we may use are likewise tucked into their package-named JAR files. For example, if we use the org.springframework.beans.factory.config.PropertiesFactoryBean, we can find that it is hiding in the org.springframework.beans-3.0.0.RELEASE.jar file.

With this in mind, for each of the missing class errors that our compiler is giving us, copy the appropriate package-named JAR file to our WEB-INF/lib folder. As we recompile, we’ll lose all or most of the errors. I leave the caveat that your project may be doing something different than mine, and that perhaps there’s a dependency that you have that also requires an upgraded JAR…so upgrade those as well. The project I’m basing this on uses Hibernate and some Apache Commons bits, and none of them required any changes to support the upgrade of Spring.

When done, there shouldn’t be any need to change any source files to satisfy any of the errors created by changing the JAR files. In some of the Spring-related XML files, like the applicationContext.xml (or whatever you named yours), there may be a bit that says something like “spring-beans-2.5.xsd” that needs to be changed to “spring-beans-3.0.xsd” or “spring-context” or whatever other bits you’ve used in your application. Change those to match the new version, and that should be it for the change.

It should be the case that the application will again compile and execute as before, without any other changes. Of course, there may be some configuration file quirks, again depending on the complexity of your project, or how hard you banged on Spring to get it to work.

If you’re not already using annotations, though, a diligent eye will notice that we’re now probably left with some compiler warnings. Most notably, any class that extends a Spring Controller will complain that the Controller it’s extending has been deprecated. Spring 3.0 pushes us hard into using annotations.

I was already using annotations for most of my controllers, but I did have one “catch-all” controller for URLs that I hadn’t mapped. The two bits in my Spring configuration that I changed included removing the bean for the AnnotationMethodHandlerAdapter (which in retrospect I may not have needed anyway), and removing the SimpleControllerHandlerAdapter and SimpleUrlHandlerMapping beans I’d configured for my catch-all, replacing that bean instead with @Controller and @RequestMapping(“/*”) annotations which did the same default URL handling.

This reduced the Controller/URL handling parts of my applicationContext.xml to just this bit.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:p="http://www.springframework.org/schema/p"
	xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
	<bean
		class="org.springframework.web.servlet.view.InternalResourceViewResolver"
		p:prefix="/WEB-INF/jsp/" p:suffix=".jsp"
		p:viewClass="org.springframework.web.servlet.view.JstlView" />
	<context:component-scan base-package="tld.domain.project.controllers" />
</beans>

Two lines, with wrapping, that controls all of our URL and JSP handling. The first tells the ViewReslover where the related JSP files are, and the second tells Spring in which package I’ve tucked my annotated Controllers. I also have some Hibernate configuration, Locale resolvers, and ResourceBundle bits, but they didn’t change a bit from 2.5 to 3.0, and this is enough to satisfy the annotated Controller discussion. Your paths may vary, and that’s a horrible package name.

If you’re not using annotations, here’s where it gets a little tricky: you’re probably using a specific Controller for specific tasks. Perhaps you’re a fan of the SimpleFormController or you roll your own with the AbstractController. The difficulty in making the change from those to the annotated Controller is going to depend on the complexity of the Controllers used and the functions therein. Let’s take a simple SimpleFormController as an example and convert it to an annotated Controller instead, to give a quick example.

First, one of our application context XML files would contain a bit not unlike this defining our bean and telling Spring to handle all otherwise unmapped URLs with our controller.

	<bean id="exampleSimpleFormController"
		class="tld.domain.project.ExampleSimpleFormController">
		<property name="commandClass" value="tld.domain.project.CommandClass" />
		<property name="commandName" value="Command" />
		<property name="formView" value="index" />
		<property name="successView" value="index" />
	</bean>
	<bean
		class="org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter" />
	<bean class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping">
		<property name="order" value="99" />
		<property name="mappings">
			<props>
				<prop key="/*">exampleSimpleFormController</prop>
			</props>
		</property>
	</bean>

Our Controller class would look something like this, presumably doing a little bit more work between the call and return, but this actually meets our criteria for the example.

public class ExampleSimpleFormController extends SimpleFormController {
    @Override
    protected ModelAndView handleRequestInternal(final HttpServletRequest httpServletRequest, final HttpServletResponse httpServletResponse) throws Exception {
	// Do your work here
        return super.handleRequestInternal(httpServletRequest, httpServletResponse);
    }
}

We can convert this to an annotated Controller without too many changes and losing no functionality. Simply remove those bits in the application context XML file related to this bean, make sure it’s in a package covered by the annotated class scan (the context:component-scan bit above). Make the following changes, and you’ve converted the controller to an annotated one.

@Controller
public class ExampleSimpleFormController {
    @RequestMapping("/*")
    public String handleRequestInternal(final HttpServletRequest httpServletRequest,
		final HttpServletResponse httpServletResponse) throws Exception {
	// Do your work here
        return “index”;
    }
}

Again, of course, the complexity of your Controller will make that more difficult. This class, however, will function exactly the same in both versions. The name of the class can be changed, as can the name of the method handling our default request, with consideration to use elsewhere.

One obnoxious thing about this example is that it’s tediously simple. All it’s really going to do is handle any URL request not matched by another Controller and return the rendered WEB-INF/index.jsp page. The original Spring configuration implied, as a SimpleFormController would, that some action would be taking place, as we had defined a command object and had to define both success and failure view names. Let’s make a tougher example and convert it from a SimpleFormController to an annotated Controller with some simple form input.

Let’s begin with a JSP fragment that contains this simple login form:

<div id="form">
<form:form action="login.ext" commandName="Login" method="POST">
<div><label for="user">Name</label> <form:input path="user" /></div>
<div><label for="password">Password</label> <form:input path="password" /></div>
<div><input id="submit" type="submit" value="Log-in" /></div>
</form:form>
</div>

Note that the .ext used in the action should match whatever URL mapping done in the web.xml tying the URL to the DispatcherServlet. It could be that /* is handled by the DispatcherServlet, but that’s not recommended. We’d back this with an Object that presumably had a pair of member Strings, for the name and password, clumsily shortened for brevity like this:

public class Login {
    public String user = null;
    public String password = null;
}

We’d reference this class as our Controller’s command object, giving it the same name as the one in our form, all tied into our bean definition in an appropriate XML file. The Controller class then probably would have getters and setters to set the object. We can eliminate most of that busy work with our new annotated Controller that does a little trivial validation.

@Controller
@RequestMapping("/login")
public class LoginController {

    @RequestMapping(method = RequestMethod.GET)
    public String handleGET() {
        return “login”;
    }

    @RequestMapping(method = RequestMethod.POST)
    public String handlePOST(@ModelAttribute("Login") final Login login) {
        if ((login != null) && (login.user != null) && (login.password != null)) {
	      return “index”;
        }
       return “login”;
    }
}

Note that while the .ext was specified in the form, we can leave it out (or put it in) in the RequestMapping, which will by default then map all login.* and login/ requests to this Controller unless a better match to the URL is found. When the Controller gets a GET request, it simply renders the login.jsp page. When it gets a POST request, it verifies that the user and password fields have values (I did say trivial), and then renders the index.jsp page, otherwise it re-renders the login.jsp page.

A more comprehensive example will follow as we set up a new simple web application in the next section, but that’s the crux of a quick transformation from Spring 2.x to Spring 3.0 that will work with most projects. A little work to transform deprecated Controllers to annotated Controllers, but that can be postponed for a bit and done as time allows.

New Project Basics

Sadly, in my opinion, the Spring documentation spreads out all of the bits and pieces necessary to make a project from nothing. Additionally, what strong suggestions there are tend to start with “use the sample project and remove what you don’t need,” which is both tedious and intimidating if you’re not sure what is safe to remove. I prefer an approach of starting with nothing and add the minimum necessary to get things going.

Of course, a bit of up front design is always nice; it’s hard to make a project without any kind of intention behind it. For the purposes of this example, we’ll make a trivial in-memory Twitter clone out of just a couple of annotated Spring Controllers and the JSPs that render the pages. No security, graphics, or style sheets to clog the works, just simple annotations and tags.

For the view, we’ll have a simple one-page interface that will give us a simple form to add a post, a simple search form, and a list of the posts thus far, ordered in descending date (newest on top).

For our controllers, we’ll need something to handle the search, something to handle the post, and something to give us the list of previous posts. We’ll use a simple multi-action controller with a different target for each of our form actions.

For our model, we’ll simply have a post object, and we’ll maintain a small collection of them (so we don’t overwhelm our example system or run out of memory.

We’ll assume and discuss as if the whole world uses Java6, Tomcat v6, and Eclipse, and that all of the paths are correctly configured, and that Eclipse has a Tomcat server configured. In Eclipse, make a new Dynamic Web Project; give it a name (I’ll call it “microblog”), associate it with the Tomcat server, let it create source and web content folders, and generate a default web.xml file.

Since we know we’re going to use Spring for our framework, let’s put that in our application. Start with editing the WEB-INF/web.inf file in the web content folder. By default the XML contains a description that has the display-name of the application and a generous list of welcome-files. Reduce the welcome-file-list to just one, like index.htm and create an empty file in the root of the web content folder of the same name; a quirk in Tomcat will throw a 404 error if the file doesn’t exist, even though it won’t be used as we’ll replace it with a Spring Controller in just a bit.

A quick fix to another little Tomcat quirk, add a context-param for the webAppRootKey to avoid collisions with other applications. Give it a unique name so that Tomcat won’t complain every time the app starts, nor will any application fail because suddenly it’s looking in the wrong folder. The param-name needs to be webAppRootKey, but the param-value should be unique within your Tomcat server. Put this between the display-name and welcome-file-list.

	<context-param>
		<param-name>webAppRootKey</param-name>
		<param-value>microblog.appRoot</param-value>
	</context-param>

Then since we’re going to be making a really simple Spring application, we’ll start with the textbook minimum configuration. We need to declare our Servlet to handle our requests, and map URLs to the servlet. Put this between the context-param (or description if you and welcome file list

	<servlet>
		<servlet-name>microblog</servlet-name>
		<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
		<init-param>
			<param-name>contextConfigLocation</param-name>
			<param-value>/WEB-INF/applicationContext.xml</param-value>
		</init-param>
	</servlet>

	<servlet-mapping>
		<servlet-name>microblog</servlet-name>
		<url-pattern>*.htm</url-pattern>
	</servlet-mapping>

The servlet definition tells Tomcat that there’s a servlet named “microblog” of the type DispatcherServlet, with the provided parameter. The parameter will be passed to the DispatcherServlet and tell it where to find the context information; we’re going to tuck it away in the WEB-INF folder where web browsers can’t get to it. The servlet-mapping definition tells Tomcat that all URLs reaching this application that end in .htm will be handled by our microblog Servlet.

This gives us a simple WEB-INF/web.xml file that looks like this one.

<?xml version="1.0" encoding="UTF-8"?>
<web-app
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="http://java.sun.com/xml/ns/javaee"
	xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
	xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
	id="microblog_ID" version="2.5">
	<display-name>microblog</display-name>
	<context-param>
		<param-name>webAppRootKey</param-name>
		<param-value>microblog.appRoot</param-value>
	</context-param>
	<servlet>
		<servlet-name>microblog</servlet-name>
		<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
		<init-param>
			<param-name>contextConfigLocation</param-name>
			<param-value>/WEB-INF/applicationContext.xml</param-value>
		</init-param>
	</servlet>
	<servlet-mapping>
		<servlet-name>microblog</servlet-name>
		<url-pattern>*.htm</url-pattern>
	</servlet-mapping>
	<welcome-file-list>
		<welcome-file>index.htm</welcome-file>
	</welcome-file-list>
</web-app>

Since we’ve declared that we’re going to use the DispatcherServlet, we’ll need to copy the appropriate Spring JAR file to our WEB-INF/lib folder. DispatcherServlet is in the org.springframework.web.servlet package, which most closely matches the org.springframework.web.servlet-3.0.0.RELEASE.jar, so copy that one. You can verify by searching for the type and Eclipse should be able to find it.

Although we’re going to be annotating our Controllers and such, some may point out how we’re sticking with the old XML way of configuring Spring. This is due in part to the comfort we have with the XML configurations, the trivial configuration we’re going to have, and finally because the Spring documentation that tells us that annotating the configuration is not a 100% replacement. Rather than anything confusing at this point, we’ll stick with the trivial XML we need to make our application.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
	xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
	<bean
		class="org.springframework.web.servlet.view.InternalResourceViewResolver"
		p:prefix="/WEB-INF/jsp/" p:suffix=".jsp"
		p:viewClass="org.springframework.web.servlet.view.JstlView" />
	<context:component-scan base-package="com.objectpartners.microblog" />
</beans>

This was discussed before, but so you don’t have to scroll back, this essentially two-line XML file (one bean, one context element) tells our DispatchServlet what we’re going to use for our view resolver (to render our JSPs), and to scan our package for annotations. Since we’ve stated that our JSPs are going to be in WEB-INF/jsp, we should take this opportunity to create that folder. Additionally, we’ve declared our package to scan, so we should create that path in our folder, too; in Eclipse we just create an empty package. Our two additional classes are in the org.springframework.web.servlet package with our DispatcherServlet, so we don’t need to copy an additional JAR file just yet.

Where to go next is a choice of style and discipline; controller, model, or view, or all at once. I think we need somewhere for the first URL request to go, so let’s build a simple catch-all controller to deliver our default page, and see where that leads us.

Our project is going to be terribly small, so despite any other design patterns, we’re going to put everything right in our previously created package. Let’s create a class named CatchAll to catch everything that isn’t otherwise handled.

package com.objectpartners.microblog;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class CatchAll {
	@RequestMapping("/*")
	public String catchAll() {
		return "index";
	}
}

Curious, the org.springframework.stereotype.Controller isn’t found in a JAR named after the package, and it doesn’t give any hint that this is an annotation class, either. We can find this is the right fully-qualified class name in the Spring API documentation, but finding it in the JAR files is a bit tedious. It’s hiding in the org.springframework.context-3.0.0.RELEASE.jar which is one of the few deviations from the new naming scheme we’ll run into. The RequestMapping annotation is correctly in the org.springframework.web-3.0.0.RELEASE.jar file. Add both of those files to our WEB-INF/lib and any compile errors so far will be resolved.

Our handler is woefully simple, and will tell Spring to render the index.jsp for every URL received. We told Spring before that these files would exist in the WEB-INF/jsp folder, so let’s make one right now; we’ll revisit and make it useful in a moment. Simply add a “hello, world” JSP so we can test that everything works so far; using Eclipse, just add a new JSP page to the folder, name it index.jsp, and add some text between the body tag.

So far, all looks well, and while not meeting our project goals, we should be at an acceptable starting point. That is, the app looks like it completes all of the required configuration elements. Add the project to the configured Tomcat server in Eclipse and start it up. Tomcat should start with no problem, but if we try to visit it (http://localhost:8080/microblog/), we’ll end up with an exception because we’re missing some classes we didn’t declare, but that Spring needs to function. There are going to be a lot of missing classes, so be prepared. In our Tomcat console, and probably the web page, we should see a stack trace that warns us that there’s a missing class.

java.lang.NoClassDefFoundError: org/springframework/beans/BeansException

We need to add org.springframework.beans.BeansException, which is in the org.springframework.beans-3.0.0.RELEASE.jar file, so copy that to the WEB-INF/lib folder and restart Tomcat. A second try, another exception for another missing class.

java.lang.NoClassDefFoundError: org/springframework/core/NestedRuntimeException

Copy the org.springframework.core-3.0.0.RELEASE.jar file to the WEB-INF/lib folder, restart Tomcat, and try again. We could have almost guessed we’d need beans and core, but we’re trying to add as few files as possible, remember. Trying again, and, yes, another failure. This time from a dependency Spring has for logging.

java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory

Quickly scanning the documentation, we can see that this is left to us to decide for ourselves which implementation we can or may want to use. Some application servers provide such logging, some environments need a little extra configuration. We could rebuild Spring without the dependency, or the easy solution, get the latest JAR from Apache Commons Logging (http://commons.apache.org/logging/) and add it to our WEB-INF/lib folder. Restart, retry, re-fail.

java.lang.NoClassDefFoundError: org/springframework/asm/ClassVisitor

Copy org.springframework.asm-3.0.0.RELEASE.jar to WEB-INF/lib, restart, reload.

java.lang.NoClassDefFoundError: org/springframework/expression/PropertyAccessor

This one is hiding in org.springframework.expression-3.0.0.RELEASE.jar so copy that, restart, reload.

The next one gets a little tricky.

java.lang.NoClassDefFoundError: javax/servlet/jsp/jstl/core/Config

While this should be a part of Tomcat, it isn’t. I suppose it’s possible that the tag library might not be used by enough Servlet or JSP applications to make it a default. If you’re not using Tomcat, you might have this in your J2EE engine’s classpath already. Since we are running Tomcat in our example, grab the Taglibs from the Apache website (http://tomcat.apache.org/taglibs/standard/). Grab the version 1.1 file (1.2 isn’t quite done yet) and copy the JAR files from the downloaded lib folder to your WEB-INF/lib folder. Restart Tomcat.

Finally, we should be rewarded with the text of our index.jsp page. Our annotated Spring web application works for now. Back to making it useful.

Since we were last editing the index.jsp, let’s add the tags and form elements to display a list of our micro-blogging posts and some simple forms for submitting a post and searching.

<%@ page language="java" contentType="text/html; charset=UTF-8"
	pageEncoding="UTF-8"%><%@ taglib prefix="form"
	uri="http://www.springframework.org/tags/form"%><%@ taglib prefix="c"
	uri="http://java.sun.com/jsp/jstl/core"%><!DOCTYPE html
	PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
	"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Microblog</title>
</head>
<body>
<div><form:form action="post.htm"
	commandName="Post" method="POST">
	<div><form:textarea path="post" cols="60" rows="3" /></div>
	<div><input id="submit" type="submit" value="Post!" /></div>
</form:form></div>

<div><form:form action="search.htm"
	commandName="Search" method="POST">
	<div><form:input path="post" maxlength="60" size="40" /></div>
	<div><input id="submit" type="submit" value="Search!" /></div>
</form:form></div>

<div><c:forEach var="post" items="${Posts}">
	<div>
	<div>${post.post}</div>
	<div>${post.time}</div>
	</div>
</c:forEach></div>
</body>
</html>

Near the top we added a couple taglib lines telling the JSP renderer to use the Spring form and JSTL core tags. The first form is a big text area with a button to submit; it’s aiming for the post.htm action, and will use a page bean named “Post” for storing its data. The second form is an input and a submit button; it’s aiming for the search.htm action and will use a page bean named “Search” for storing its data. Finally, there’s a forEach loop that will simply spew whatever comes out of the array or collection page bean named “Posts.”

Let’s throw together a quick bean to satisfy the forms. We’ll cheat and use the same little guy, for brevity. A simple bean with two Strings.

package com.objectpartners.microblog;

public class Post {
	private String post = null;

	private String time = null;

	public String getPost() {
		return post;
	}

	public String getTime() {
		return time;
	}

	public void setPost(String post) {
		this.post = post;
	}

	public void setTime(String time) {
		this.time = time;
	}
}

We’ve already got our CatchAll Controller, so let’s just tweak that to handle these new requests.

package com.objectpartners.microblog;

import java.util.LinkedList;
import java.util.Calendar;
import java.util.List;

import javax.servlet.http.HttpServletRequest;

import org.springframework.stereotype.Controller;
import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class CatchAll {

	@RequestMapping("/*")
	public String catchAll(final HttpServletRequest httpServletRequest,
			final ModelMap modelMap) {
		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof List<?>) {
			modelMap.addAttribute("Posts", object);
		}

		return "index";
	}

	@RequestMapping("/post")
	@SuppressWarnings("unchecked")
	public String post(final HttpServletRequest httpServletRequest,
			@ModelAttribute("Post") final Post post, final ModelMap modelMap) {

		LinkedList<Post> posts = null;
		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof LinkedList<?>) {
			posts = (LinkedList<Post>) object;
		} else {
			posts = new LinkedList<Post>();
			httpServletRequest.getSession().getServletContext().setAttribute(
					"Posts", posts);
		}
		if (post.getPost() != null && !post.getPost().trim().isEmpty()) {
			post.setTime(Calendar.getInstance().getTime().toString());
			posts.offerFirst(post);
		}
		while (posts.size() > 100) {
			posts.pollLast();
		}
		modelMap.addAttribute("Posts", posts);

		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		return "index";
	}

	@RequestMapping("/search")
	@SuppressWarnings("unchecked")
	public String search(final HttpServletRequest httpServletRequest,
			@ModelAttribute("Search") final Post search, final ModelMap modelMap) {

		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof List<?>) {
			if (search.getPost().trim().isEmpty()) {
				modelMap.addAttribute("Posts", (List<Post>) object);
			} else {
				final List<Post> posts = new LinkedList<Post>();
				for (final Post post : (List<Post>) object) {
					if (post.getPost().toLowerCase().contains(
							search.getPost().toLowerCase())) {
						posts.add(post);
					}
				}
				modelMap.addAttribute("Posts", posts);
			}
		}

		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		return "index";
	}
}

We’ve upgraded our default request method, catchAll(), adding a couple parameters that Spring will autowire for us. We need the HttpServletRequest to gain access to the collection we’re maintaining in memory, or more specifically in the Servlet context’s memory (we could use a static variable). We need the ModelMap to store our page beans for the form. We could annotate these a little better, getting directly to the individual beans, but that actually adds as much extra class material to build what will ultimately result in a bunch of empty objects. The catchAll() simply grabs the collection from the ServletContext, if it exists, and sends back that collection and some empty beans to satisfy the JSP.

We added a post() method, and mapped that to the /post request to handle when the user taps the “Post!” button from the form. Note the ModelAttribute annotation telling Spring to associate the form data for a form named “Post” with the parameter. This method will create the collection if it doesn’t already exist. It validates that the post contains text (and nothing more) and adds it to the head of the list. It also truncates the list to be sure our simple list won’t get too long. Like catchAll(), it puts the necessary elements in the ModelMap to render the form.

Finally, the search() method will handle the /search request when the user taps the “Search!” button. Like the post() method it has an annotation allowing Spring to autowire the form data. This method searches for all posts containing the string as input. If the string is empty, all post elements are returned. The ModelMap is populated and the form is rendered.

In our trivial application, we have (after much trial-and-error correction) the minimum Spring and other JAR files necessary to support our application. We’ve got one annotated Controller class, handling all of the actions our one JSP page can dish out. It isn’t pretty, but it works.

Why Hadoop?

Monday, February 22, 2010

Hadoop is garnering a lot of attention these days as companies ponder how to address problems involving large amounts of data (tera to petabytes). It also gathers attention from companies interested in distributed computing. Lastly, it is seen as an alternative to an RDBMS. So what is Hadoop actually and how would it be used in the enterprise now and in the near future?

What is Hadoop?
Briefly, Hadoop is an Apache Open Source project that mirrors technologies developed by Google to power its search engine and other products (Google Maps, Analytics, etc.). Hadoop consists of a number of sub-projects but the main ones are Hadoop File System (HDFS), Map Reduce and Hbase (multi-dimensional sorted map used as a data store). The HDFS creates a massive, durable, cost-efficient file system by using three way replication of all files across a cluster of commodity servers.
For more on Hadoop: http://hadoop.apache.org

Use Case #1: Searching large quantities of unstructured data quickly.

Hadoop gathers most of its interest from companies that would like to search, analyze and store a large amount of data, in the tera to petabyte range. As Google’s Google File System (GFS) is the basis for Google’s search engine, Hadoop’s HDFS is the basis for Yahoo!’s search engine. Since the impetus for creating Hadoop was Internet search, obviously, the main usage for Hadoop has been in searching unstructured data such as web pages or other documents.

Use Case #2: Accessible, programmable distributed computing paradigm.

Distributed computing has long been seen as a shining light at the end of the processing tunnel, but has largely been unused in enterprise environments due to its complexity. The Map Reduce framework is starting to change that paradigm as it is a conceptually simple programming model that brings distributed computing down to a reachable level. Map Reduce combined with HDFS has proven successful by abstracting away any knowledge of where the data lives. Map Reduce works by pushing the processing to the data rather than moving the data to the processing. It has only become viable as computing power and memory has increased to the point where complex computations could be completed quickly on commodity hardware.

Use Case #3: RDBMS replacement for large data sets.

There would be little reason to replace your RDBMS with Hadoop if you have a reasonable database size (GB range). Where Hadoop starts looking legitimate is when the cost/benefit of purchasing larger and larger servers to run the RDBMS crosses the cost/benefit of purchasing and maintaining a large cluster of commodity servers. Throw in retraining your development resources from SQL based development to NOSQL based and that further complicates the picture against Hadoop. Where Hadoop wins out is when the RDBMS can no longer support the queries taking place on a massive data set. A restructure of the data set to Hadoop and Hbase can return the query times to a more reasonable level. Obviously, this is not a trivial amount of work but necessary if other alternatives fail.

Framework for a Multi-stage Spring Property Loader Extension Allowing Dynamic Updates of Properties via JMX

Monday, February 22, 2010

Overview

It is commonplace in enterprise applications to allow application properties to be loaded from configuration files. When leveraging Spring, this is typically achieved using a PropertyPlaceholderConfigurer instance within the application. As is stated in the JavaDocs for this class,  it is “A property resource configurer that resolves placeholders in bean property values of context definitions. It pulls values from a properties file into bean definitions.” (see http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/beans/factory/config/PropertyPlaceholderConfigurer.html)

Properties are loaded at application startup, prior to loading other spring beans for the application. Other beans in the application can then leverage the properties as part of their initialization, allowing externalization of various properties. This can be accomplished within Spring context files using placeholders containing the desired property keys, or via annotations in the classes themselves.

A common and vital pattern leveraged in many applications is to allow for environment-specific properties file. This means that different properties files may get loaded depending on the environment in which the application is running. There are a handful of strategies for supporting this pattern, and a full discussion is outside of the scope of this discussion. The one strategy that will be discussed is as follows: specify a location external to the application itself (such as a configuration folder living on the system where the application will be deployed) that will contain a consistently named property file (e.g. myApp.properties). When configuring the PropertyPlaceholderConfigurer spring bean, specify that location as the location of the properties file to load. This might look like:


<bean id="placeholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="location" value="file:/my/app/config/myApp.properties" />
</bean>

Even more useful is to leverage multiple properties files, allowing for property overrides. This might look like:


<bean id="placeholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="locations">
        <list>
            <value>classpath:defaults.properties</value>
            <value>file:/my/app/config/myApp.properties </value>
         </list>
    </property>
</bean>

When using a list of property files for a PropertyPlaceholderConfigurer, entries later in the list take precedence over entries earlier in the list when there are property key collisions. In the above example, we specify an internal property file within the distribution called default.properties, and a property file external to the application. In the event that there is overlap of property keys between the files, the values in the external properties file would be used (because it is last in the list). Furthermore, the full set of properties loaded would be the union of the properties contained in the two files (e.g., there is no stipulation that the properties contained in the files correlate at all, unless you desire it). There are many advantages to this simple approach to property loading, including:

  • The default (internal) property file can contain default values for all properties, if desired, ensuring that there will not be “missing” properties in your application
  • The external property file can be updated manually on the server without requiring redeployment. However, an application restart would be required to pick up the modifications to the file.
  • Properties that should never be modified without an application redeployment can be exclusively contained within the internal properties file. This might be important if there are properties that you need to be configurable for unit testing purposes, but otherwise need to be consistent in all deployments regardless of environment.

At this point a developer with some knowledge of Spring is probably saying “so what”. This is nothing that isn’t already outlined in Spring’s documentation and dozens of examples out there on the web. So, here is where we can introduce some extensions that make the configuration framework more flexible.

Enabling Multi-stage Property Loading

At a recent client, there was a requirement for an application that we were writing that many of it’s properties be configurable from a central configuration database. Essentially, the database represented servers, applications on those servers, and key-value property pairs for each application/server combination. The idea is that it is a one-stop shop for managing the configurations for all applications (and in fact all instances of each application) in the enterprise. In contrast to that requirement (or in conjunction with it), from a development and testing perspective we wanted:

  • An internal properties file to represent defaults and properties that should not change from environment to environment (e.g. should not be configurable without an application redeployment, as they may substantially affect some piece of logic within the application)
  • An external, environment-specific properties file that could be changed at will (at least in local and dev environment), and could contain overrides for a number of the properties in the internal properties file.

Given these three sources for properties, we needed to put together an extension to the property placeholder framework that could:

  • Load properties from multiple sources
    • Including a database
  • Allow staged property loading with flexible override ordering. In other words, we wanted to be able to specify whether the internal, external, or database properties should take precedence when configuration keys overlap.

As we saw in the previous section, dealing with the internal and external properties files is easy. However, dealing with the database properties files takes a little more work..

For beginners, we need a basic extension to PropertyPlaceholderConfigurer to work with. We will start with the following class, StagedPropertyPlaceholderConfigurer, which extends PropertyPlaceholderConfigurer.


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;

    private Resource internalProperties;
    private Resource externalProperties;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;

        // if this constructor is used, a data source will not be used for property loading
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }
}

As you can see, this class contains properties for the internal and external property files, and a basic constructor taking these arguments. The configurer is set up to be used without the database at all, for the moment. Some flags to indicate where to load properties from for reference during the actual loading.

Next, we need to override the loadProperties method from the PropertyPlaceholderConfigurer class. The Spring framework will call this method to load the properties. For the moment, lets make it simple, since we are not considering the database quite yet:


@Override
protected void loadProperties(final Properties props) throws IOException {
    List<Resource> resources = new ArrayList<Resource>();

    if(loadFromInternalPropertiesFile) {
        resources.add(internalProperties);
    }

    if(loadFromExternalPropertiesFile) {
        resources.add(externalProperties);
    }

    Resource[] locations = new Resource[resources.size()];
    resources.toArray(locations);
    setLocations(locations);

    super.loadProperties(props);
}

At this point, we have essentially re-invented the wheel, so lets go a bit further. Now we are going to add the ability to configure a data source to load properties from, and specify how to load them. In order to do this in as generic a fashion as we can, we will define a DatasourceProperyLoaderStrategy interface as follows:



/**
 * Interface to be implemented for defining a data source-based property loading strategy to use in conjunction
 * with the StagedPropertyPlaceholderConfigurer.
 *
 */
 public interface DatasourcePropertyLoaderStrategy {

    /**
     * provide any configuration needed to connect to the data source. This should include creation of the data source
     * and any other initialization necessary.
     */
     public void configure(Properties properties);

    /**
     * Load properties from the configured data source. If overrideExisting is true,
     * replace any existing properties in the {@link Properties} instance with the ones loaded from
     * the data source. Otherwise, the existing properties take precedence in the case of property key collisions
     *
     * @param properties {@link Properties} instance containing any propreties that have already been loaded
     * @param overrideExisting if true, the properties loaded from this data source take precedence in the case of key collisions.
     */
    public void loadProperties(Properties properties, boolean overrideExisting);

    /**
     * Any cleanup required after properties are loaded, for instance disposing of the data source
     */
    public void tearDown(Properties properties);

}

An implementation of this interface is necessary for defining how properties would be loaded from a data source. This implementation would need to define:

  • How to connect to the data source (e.g., would need to be aware of data source attributes such as the database urls, drivers, credentials, and so on).
  • How to load properties from the database (e.g. where are properties loaded, in what format, how to process them)
  • How to merge properties pulled from the data source with the other properties (e.g., is manipulation of keys required, are there case-sensitivity concerns, should the properties loaded from the database override the previously loaded properties)
  • How to clean up the data source and dispose of it when property loading is completed.

Note that all methods on this interface take the Properties instance. Obviously this is needed to merge properties from the database with the previously loaded properties. However, the fact that they are passed to the configure() and teardown() methods allows previously loaded properties to be leveraged while determining how to load the additional properties. For instance, we can put the data source attributes in the external, environment-specific property file. This file would be loaded first, and therefore the data source attributes contained within that configuration file can be used to establish the data source to pull additional properties from the database.

Another useful aspect of this interface is that it would not be limited to pulling properties from a database. You could easily create an implementation that pulled configuration from another source, such as a web service.

If you were to implement a strategy to load from a database, the org.springframework.jdbc.datasource.DriverManagerDataSource class is a safe bet for creating a data source.

Okay, now back to the StagedPropertyPlaceholderConfigurer itself. With an implementation of DatasourcePropertyLoaderStrategy assumed to be defined, lets make the following modifications (in bold):


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;
    private boolean databaseTakesPrecedence = true;

    private Resource internalProperties;
    private Resource externalProperties;

    private DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties, DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy, boolean databasePrecedence) {
        super();
        this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        this.databaseTakesPrecedence = databasePrecedence;

        if(databasePropertyLoaderStrategy == null) {
            loadFromDB = false;
            databaseTakesPrecedence = false;
        }

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    @Override
    protected void loadProperties(final Properties props) throws IOException {
        List<Resource> resources = new ArrayList<Resource>();

        if(loadFromInternalPropertiesFile) {
            resources.add(internalProperties);
        }

        if(loadFromExternalPropertiesFile) {
            resources.add(externalProperties);
        }

        Resource[] locations = new Resource[resources.size()];
        resources.toArray(locations);
        setLocations(locations);
        loadProperties(props);

        if(loadFromDB) {
            try {
                databasePropertyLoaderStrategy.configure(props);
                databasePropertyLoaderStrategy.loadProperties(props, databaseTakesPrecedence);
                databasePropertyLoaderStrategy.tearDown(props); 
            } catch (Exception e) {
                // just log the exception? This happen if the datasource is unavailable or the data returned
                // cannot be processed correctly. In this case, the DatasourcePropertyLoaderStrategy should be responsible for
                // any necessary exception handling.
            } 
        } 
    }
}

So far so good. We have a property loader that can take properties from multiple sources and merge them. It can pull property from an external data source, whether that is a database, a web service, or anything else. It can be configured to allow different property sources to take precedence over others. And the only real work involved is in implementing the DatasourcePropertyLoaderStrategy. But, as long as we have gone this far, lets go a bit further.

Exposing a Configuration Bean for Use Throughout the Code Base

What has been discussed up to this point allows for configuration modifications that would only be picked up with an application restart. It would be nice to be able to dynamically update properties via JMX without an application restart. This can be very useful for tuning applications (if you have properties that are considered “tunable”), or allowing dynamically modifiable behavior. For example, if your application posts content over http to some URL, you might want to be able to modify that URL without restarting the application or having to monkey around with the network.

In order to facilitate this, we will further modify the property placeholder configuration framework, enabling the creation of a “configuration bean” that can be leveraged throughout the application. This configuration bean would get populated with the final output of the StagedPropertyPlaceholderConfigurer. With some modifications to how you access your properties in your code, the bean can then be leveraged to get access to properties. And the bean can be exposed as a managed resource to allow dynamic modification of the properties.

First, lets will define a ConfigurationBean interface as follows:


public interface ConfigurationBean {

    public Map<String, String> getProperties();
    public void setProperties(final Properties props);
    public String getValue(String key);
    public Integer getIntValue(String key);
    public Long getLongValue(String key);
    public Boolean getBooleanValue(String key);
    public String setProperty(String key, String value);
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception;
}

And a base implementation of this interface:


public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map<String, String> props = new HashMap<String, String>();

    /**
     * Gets the properties.
     *
     * @return the properties
     */
    public Map<String, String> getProperties() {
        if (props == null) {
            return null;
        }

        return Collections.unmodifiableMap(props);
    }

    /**
     * Sets the properties for the configuration bean based on Properties loaded by the Property Placeholder Configurer
     *
     * @param props
     */
    public void setProperties(final Properties props) {
        for (Iterator iterator = ((Map) props).keySet().iterator(); iterator.hasNext();) {
            String key = (String) iterator.next();
            this.props.put(key, props.getProperty(key));
        }
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

    /**
     * Gets the value for the specified key as an Integer
     *
     * @param key
     * @return the value
     */
    public Integer getIntValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Integer.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Long
     *
     * @param key
     * @return the value
     */
    public Long getLongValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Long.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Boolean
     *
     * @param key
     * @return the value
     */
    public Boolean getBooleanValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Boolean.valueOf(val);
    }

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap<String, String>();
        }

        return props.put(key, value);
    }

    /**
     * Used to populate the collection target, with values of type classType. classType must have a single String argument constructor to create new instances.
     *
     * Example usage
     *
     * List<Long> toFill = new ArrayList<Long>();
     *
     * loadIntoCollection("propertyKey",toFill, Long.class);
     *
     * @param key
     * @param target The collection to load config data into cannot be null
     * @param classType The class type to cast the config's String value to. Should be the class type assigned to the Collection target
     *
     * @throws Exception
     **/
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception {
        if (props == null) {
            return;
        }

        if (target == null) {
            throw new IllegalArgumentException("Argument target cannot be null");
        }

        if (classType == null) {
            throw new IllegalArgumentException("Argument classType cannot be null");
        }

        String val = props.get(key);
        if (val == null || val.length() == 0) {
            return;
        }

        String[] vals = val.split(",");
        Constructor con;

        try {
            con = classType.getConstructor(String.class);
        } catch (NoSuchMethodException e) {
            throw new IllegalArgumentException(classType + " does not have a single String argument constructor ");
        }

        for (int i = 0; i < vals.length; i++) {
            try {
                target.add(con.newInstance(vals[i]));
            } catch (Exception e) {
                throw new Exception("Config Value: " + vals[i] + " cannot be converted to " + classType.getName());
            }
        }
    }
}

This interface can of course be extended to handle additional data types, or to provide more granular methods for getting and setting specific properties. In fact, it would be a good idea for any application using this framework to create a ConfigurationBean implementation that extends ConfigurationBeanImpl, just for the flexibility it would provide.

Now we need to define a singleton instance of the ConfigurationBean that will be used within the StagedPropertyPlaceholderConfigurer (and elsewhere in the application). For instance:


<bean id="configBean" class="com.objectpartners.sandbox.config.ConfigurationBeanImpl" scope="singleton"/>

Next, the StagedPropertyPlaceholderConfigurer needs to be updated to populate the ConfigurationBean instance. There are a few things that need to be added to this class to support this:

  • We need a handle on the bean factory, as we will need to look up the ConfigurationBean instance. This can be accomplished by:
    • Defining a private data member for the BeanFactory (There is a BeanFactory instance on the superclass PropertyPlaceholderConfigurer, but it is hidden):
    • 
      private BeanFactory beanFactory;
      
    • Overriding the setBeanFactory method as follows:
    • 
      @Override
      public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
          this.beanFactory = beanFactory;
          super.setBeanFactory(beanFactory);
      }
      
      
  • We need to know the id of the ConfigurationBean singleton in order to be able to find it in the bean factory.
    • Add the string name as a data member:
    • 
      private String configurationBeanName;
      
      
    • Update all constructors to take the configuration bean name as a parameter. E.g.
    • 
      
      public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties,
                                                 DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy,
                                                 boolean databasePreceduence, String configurationBeanName) {
          super();
          this.configurationBeanName = configurationBeanName;
          this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
          this.internalProperties = internalProperties;
          this.externalProperties = externalProperties;
          this.databaseTakesPrecedence = databasePreceduence;
      
          if(databasePropertyLoaderStrategy == null) {
              loadFromDB = false;
              databaseTakesPrecedence = false;
          }
      
          if(internalProperties == null) {
              loadFromInternalPropertiesFile = false;
          }
      
          if(externalProperties == null) {
              loadFromExternalPropertiesFile = false;
          }
      }
      
      
  • We need to implement a method to populate the ConfigurationBean instance with the output of the property loading process:


/**
 * Populates the Configuration bean with the properties that have been loaded.
 *
 * @param props the props
 */
protected void loadServerConfigBean(final Properties props) {
    ((ConfigurationBean) beanFactory.getBean(this.configurationBeanName, ConfigurationBean.class)).setProperties(props);
}

  • And, finally, we need to add a call to loadServerConfigBean after all of the properties have been loaded/merged. Add the following as the last line of code in the loadProperties method:

loadServerConfigBean(props);

Okay, that was a lot of little things, but now we are done with the property placeholder configurer and populating the configuration bean. The next step is to add a bean to the Spring context to define the property placeholder.  This bean definition, along with the configuration bean definition and the definition of a DatasourcePropertyLoaderStrategy, might look like:



<bean id="configBean" class="com.objectpartners.sandbox.config.ConfigurationBeanImpl" scope="singleton"/>

<bean id="datasourcePropertyLoaderStrategy" class="com.objectpartners.sandbox.config.DatabasePropertyLoaderStrategy" scope="singleton"/>

<bean id="placeholderConfig" class="com.objectpartners.sandbox.config.StagedPropertyPlaceholderConfigurer">
    <constructor-arg index="0" value="classpath:internal.properties"/>
    <constructor-arg index="1" value="file:/my/app/config/external.properties"/>
    <constructor-arg index="2" value="datasourcePropertyLoaderStrategy "/>
    <constructor-arg index="3" value="true"/>
    <constructor-arg index="4" value="configBean"/>
</bean>

After application startup, the configuration bean would now contain the current values of all of the application properties. In fact, you could extend the framework so that the configuration bean will ultimately contain more than the union of the contents of all of the configuration files and data sources. One example might be to load in configuration values for the host ip and host name into configuration based on an InetAddress instance created during application startup.

Now we need to discuss the usage of the configuration bean. This will require a little change in programming paradigm where you want to leverage the configuration bean. First of all, in order to access properties in the configuration bean, you need to know the property keys. Therefore, it is useful to create a “configuration constants” class or interface whose sole purpose is to contain constants for every configuration key that will be of interest. For instance, if you have a property with a key of “my.first.property”, define a constant:


public static final String MY_FIRST_PROPERTY_KEY = “my.first.property”;

Next, let suppose that we have a class Foo that leverages the current value of “my.first.property” in a method bar. We can get leverage the configuration bean by wiring it into the class and using the methods on the configuration bean to access the value. For instance



package com.objectpartners.sandbox.config;

import org.springframework.beans.factory.annotation.Autowired;

public class Foo {

    @Autowired
    private ConfigurationBean configurationBean;

    public static final String MY_FIRST_PROPERTY_KEY = "my.first.property";

    public void bar() {
        System.out.println("value of my.first.property = " + configurationBean.getValue(MY_FIRST_PROPERTY_KEY));
    }
}

The Last Step – JMX

Now we are down to the last step – allowing the configuration values to be updated dynamically via JMX. The quickest way to do this will be to make the Configuration Bean a managed resource.

Add the following to your Spring configuration:


<context:mbean-export />
<context:annotation-config />
<context:component-scan base-package=<package> />

Next, add the following to your implementation of the ConfigurationBean, right before the class declaration (it is assumed that you have subclassed ConfigurationBeanImpl, extended the ConfigurationBean interface, or both):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")

Finally, for any methods that you want to be exposed via JMX, add the following before the method declaration:


@ManagedAttribute

For any getters or setters decorated with the @ManagedAttribute, you will now be able to view the return values of those methods as attributes on a configuration mbean using your favorite JMX client (I would recommend JVisualVM for day-to-day use). Additionally, any getters and setters decorated with the @ManagedAttribute annotation will be presented as an operation on this mbean. Lets suppose that we go with the base implementation of ConfigurationBean, and add these annotations (entire class not listed, just the pertinent portions):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")
public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map<String, String> props = new HashMap<String, String>();

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    @ManagedAttribute
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap<String, String>();
        }

        return props.put(key, value);
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    @ManagedAttribute
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

In the JMX console, you will now be able to retrieve the current value for any property key using the getValue() operation, and will be able to dynamically update the value of any property via the setProperty() operation.

Additional Considerations and Final Thoughts:

There are some considerations that should be kept in mind if you plan on leveraging this framework or extending it:

  • Allowing any configuration value to be updated dynamically via JMX might not be desired functionality. If there are certain values that you might want to update via JMX, and certain ones that you would not want to, then you can implement more granular methods in your configuration bean, and only expose those methods in JMX.
  • It should be noted that the @ManagedAttribute annotation is only valid for getters and setters. If you want to expose an operation for some other type of method, use the @ManagedOperation annotation.
  • There are certainly situations where it does not make sense to allow dynamic updates of configuration values, or where it would have no effect. For instance, suppose your database driver is a configuration value, and it is used to set up your data source. That is probably only going to happen once, at application start up. Changing the configuration value after application startup is not going to have any affect in such a situation.
  • There would also be a multitude of types of configuration that you would not want to be available for dynamic update at all. This is why it generally would make sense to extend (or replace) the ServerConfig interface, and only include those operations that you absolutely need exposed in JMX for your application.
  • Remember that configuration values do not need to be single values. It is common to have comma-separated lists for configuration values. Managing these types of configurations via JMX may take some specialized operations (e.g., implementation of some operations from the List interface, exposed as JMX operations)
  • This is by no means a perfect framework, and could be refined and extended in many ways. For instance:
    • Rather than only allowing one internal property file, one external property file, and one Datasource property loading strategy, you could easily update the framework to allow multiples of each

It should also be noted that there have been a lot of updates in relation to how properties can be loaded within Spring 3.  Those capabilities are well worth looking at as well.

Enabling JMX monitoring of SEDA Queue Depths in Apache Camel

Monday, February 15, 2010

Apache Camel’s SEDA (Staged Event-Driven Architecture) endpoints (http://camel.apache.org/seda.html) provide a useful and quick mechanism to implement asynchronous, event-driven processing within your applications. See http://www.eecs.harvard.edu/~mdw/proj/seda/ for the original description of the SEDA architecture. For those not very familiar with Camel, implementing a SEDA route can be as simple as adding the following to a route builder class:


from(“seda:start”).beanRef(“someBean”, “someMethod”);

Exchanges (messages) put to the seda:start endpoint will be processed through the remainder of the Camel route. The SEDA endpoint encapsulates an in-memory queue used as an entry point to the route. All messages are processed asynchronously. In the example above, the messages would be routed to the someMethod method on a spring bean named someBean.

From a monitoring perspective, the throughput and other statistics for this route can be viewed via Camel’s JMX functionality (assuming that JMX is enabled). However, it should be noted that the queue depth for the SEDA queue is not exposed by Camel’s JMX framework. This queue depth may be of interest for application monitoring purposes. The following describes a method for exposing SEDA queues for monitoring. Additionally, this method resolves an issue inherent to the current implementation of the SedaEndpoint in Camel: the fact that the maximum queue depth is bounded.

Step 1: Define the endpoint in the camel context

In the Camel context, define the SEDA endpoint that will be the input to the route:


<camel:camelContext id="camel">
    <camel:package>com.objectpartners.camel</camel:package>
    <camel:jmxAgent id="agent" createConnector="true" connectorPort="1099" />

    <camel:endpoint id="startEndpoint" uri="seda:start"/>
</camel:camelContext>

When the context is loaded, this will instantiate the SEDA endpoint and place a bean in the spring registry for it.

Step 2: Create the route builder class

In the route builder class, wire in the endpoint and create the route as follows:


package com.objectpartners.camel;

import org.apache.camel.Endpoint;
import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.seda.SedaEndpoint;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;

public class Routes extends RouteBuilder {

    @Autowired
    @Qualifier(value = "startEndpoint")
    private Endpoint startEndpoint;

    /**
     * Configure the routes
     **/
    @Override
    public void configure() throws Exception {
        from(startEndpoint).beanRef(“someBean”, “someMethod”);
    }
}
 

Step 3: Alter the properties of the SEDA endpoint

At this point, we must change the properties of the SEDA endpoint to make the depth unbounded (this step would be optional, if you are okay with the depth being bounded. The default maximum depth is 1000, but can be configured explicitly on the endpoint.), and to allow the queue depth to be accessible via JMX. JMX exposure is accomplished by exposing the SEDA endpoint’s backing queue as a managed resource. Enabling unbounded queue depth can be accomplished by changing the blocking queue implementation used by the SEDA endpoint. The following code assumes that the following have been placed in your application context:


<context:mbean-export />
<context:annotation-config />
<context:component-scan base-package=<package> />

The final code for the route builder class:


package com.objectpartners.camel;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

import org.apache.camel.Endpoint;
import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.seda.SedaEndpoint;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.jmx.export.annotation.ManagedAttribute;
import org.springframework.jmx.export.annotation.ManagedResource;
import org.springframework.stereotype.Component;

@Component
@ManagedResource(objectName="example:name=TestRoutes", description = "Camel Test Routes")
public class Routes extends RouteBuilder {

    @Autowired
    @Qualifier(value = "startEndpoint")
    private Endpoint startEndpoint;

    // the new blocking queue for the SEDA endpoint
    BlockingQueue<Exchange> startEndpointQueue = new LinkedBlockingQueue<Exchange>();

    /**
     * Configure the routes
     **/
    @Override
    public void configure() throws Exception {

        // change the BlockingQueue implementation for the seda endpoint
        ((SedaEndpoint) startEndpoint).setQueue(startEndpointQueue);

        from(startEndpoint).beanRef(“someBean”, “someMethod”);

    }

    @ManagedAttribute
    public long getQueueDepth() {
        return startEndpointQueue.size();
    }
}

The Groovy Spaceship Operator Explained

Monday, February 8, 2010

While extending an existing Grails application bits of research around sorting lists using Groovy ensued. My first stop was Groovy In Action wherein the operator <=> was shown in a closure used for sorting, but with no explanation. Under Operators in the index, it became known that <=> is “The Spaceship Operator” – no doubt due to its similarity in look to a flying saucer. Sadly the referenced section of text also offered little in explaining what this bugger did exactly.

The spaceship operator has it’s roots in Perl and has found it’s way in to languages like Groovy and Ruby. The spaceship is a relational operator that performs like Java’s compareTo() comparing two objects and returning -1, 0, or +1 depending on the value of the left argument as compared to the right. If the left argument is greater than the right, the operator returns 1. If the left argument is less than the right, the operator returns −1. If the arguments are equal, 0 is returned. When using Groovy/Grails, if the arguments are not of comparable types, a ClassCastException is thrown.

While the spaceship could be used wherever you would use compareTo(), it seems most references to it are found in sorting. For example the following two lines would have identical results:

things.sort{ Thing a, Thing b -> b.beginDate <=> a.beginDate }

things.sort{ Thing a, Thing b -> b.beginDate.compareTo( a.beginDate ) }

Other examples of its behaviour from the Groovy Console:

groovy> def d = new Date()
groovy> println d <=> d
groovy> println 3 <=> 4
groovy> println “doberman” <=> “dachshund”
groovy> println d <=> new Date()

0
-1
1
-1

Perhaps the greatest advantage of using the Groovy comparison operators is the graceful handling of nulls such that x <=> y will never throw a NullPointerException:

groovy> println “dachshund” <=> null
groovy> println null <=> “dachshund”
groovy> println null <=> null

1
-1
0

In addition when comparing numbers of different types the type coercion rules apply to convert numbers to the largest numeric type before the comparison. So the following is valid in Groovy1:

groovy> Byte a = 27
groovy> Double b = 9
groovy> println a instanceof Byte
groovy> println b instanceof Double
groovy> println a > b

true
true
true

We come in peace – happy comparisons! For more on Groovy operator overloading look here.

1 “Groovy – Operator Overloading” Available at: http://groovy.codehaus.org/Operator+Overloading 26 January 2010

Immutable Data Structures in Concurrent Java Applications

Monday, February 1, 2010

Concurrent applications have multiple threads running simultaneously. Access to data shared by multiple threads requires synchronization which is often a source of fragile and hard to maintain code, hard to find bugs, and performance issues. You can minimize synchronization and the headaches that go with it using immutable data structures. In this article I’ll demonstrate how to create and use immutable data structures to reduce and localize the need for synchronization.
(more…)