Object Partners Inc » Blog

Updating or Starting Spring 3.0 Project

jwarren — Mon, 01 Mar 2010 14:47:11 +0000

On December 16, 2009, Spring framework version 3.0 was finally released.

For those with Spring projects already in progress, as well as those starting new projects, a number of new features are available, and some changes must be made to take advantage of them. It is strongly recommended that the full release with documents is downloaded as the changes are sweeping and sometimes drastic. That said, you shouldn’t need to do much differently than before to get the same stuff out of your Spring applications. At the very least, a slew of compiler warnings, if not errors, will be introduced if you just take a “replace the JAR files” approach to upgrading as many of the Controllers and other elements have been deprecated or otherwise reworked.

Downloading the full release with documentation brings forth a new 800-page reference document, which we should all read, but we know not all of us will. This article will endeavor to point out some pitfalls and help make a quick conversion of an existing web project to the new paradigm as well as try to show how to quickly make a new web project from scratch using the new Spring 3.0 framework.

Swing by the Spring download page (http://www.springsource.com/download/community) and grab the release archive, if you haven’t already. Expand it so the files will be at the ready. If you grab the with-docs version (which is recommended at least once), peruse the reference documents for areas you expect to have trouble.

The assumption made is that you’re at least familiar with Spring enough to recognize some of the generalities and translate them to your own project; a word of warning–this isn’t intended to be a “learn Spring” tutorial.

Existing Project Upgrade

This bit assumes you’ve got a Spring 2.x project and you’re looking to upgrade it to the new framework. Presumably if you’ve been developing with the Spring 3.0 beta and release candidates these problems have already been addressed; perhaps not, though, or perhaps there’s some clarity you seek. Hopefully this section helps in either case.

In your project, we’ll assume a standard structure of a source directory for the Java and such, and a web content directory for the web-served bits. In the web content directory lives our magic WEB-INF folder, and in the WEB-INF/lib folder is where the Spring JAR files are located. If your configuration is different, you’ll have to translate, and should probably consider this more concise approach instead.

In a Spring 2.x project, the spring-beans.jar, spring-core.jar, spring-web.jar, and whatever other Spring JAR files you’ve been using will need to be removed to avoid conflicting with the new framework files. Remove also any .tld or .xsd or other files that may have been copied from the old framework folders into the project. A flurry of compiler errors should be evident as we’ve removed core classes required by the project. Don’t worry about correcting any of these just yet as we’ve got to put the replacement JARs in place, which will remove most of those errors, and give us just a few warnings in their place.

As we look in the new Spring 3.0 archive’s dist folder, we see immediately that the file naming format has been changed. For the most part, once you get comfortable with the new format it starts to make sense. In most cases the classes have been packaged inside the JAR that starts with the same package name. For example, the starting point for nearly every Spring web application is the DispatcherServlet. The DispatcherServlet is in the org.springframework.web.servlet package. There should be a org.springframework.web.servlet-3.0.0.RELEASE.jar file in the dist folder. In the Spring 2.x framework, the DispatcherServlet is in the spring-webmvc.jar file. Other classes we may use are likewise tucked into their package-named JAR files. For example, if we use the org.springframework.beans.factory.config.PropertiesFactoryBean, we can find that it is hiding in the org.springframework.beans-3.0.0.RELEASE.jar file.

With this in mind, for each of the missing class errors that our compiler is giving us, copy the appropriate package-named JAR file to our WEB-INF/lib folder. As we recompile, we’ll lose all or most of the errors. I leave the caveat that your project may be doing something different than mine, and that perhaps there’s a dependency that you have that also requires an upgraded JAR…so upgrade those as well. The project I’m basing this on uses Hibernate and some Apache Commons bits, and none of them required any changes to support the upgrade of Spring.

When done, there shouldn’t be any need to change any source files to satisfy any of the errors created by changing the JAR files. In some of the Spring-related XML files, like the applicationContext.xml (or whatever you named yours), there may be a bit that says something like “spring-beans-2.5.xsd” that needs to be changed to “spring-beans-3.0.xsd” or “spring-context” or whatever other bits you’ve used in your application. Change those to match the new version, and that should be it for the change.

It should be the case that the application will again compile and execute as before, without any other changes. Of course, there may be some configuration file quirks, again depending on the complexity of your project, or how hard you banged on Spring to get it to work.

If you’re not already using annotations, though, a diligent eye will notice that we’re now probably left with some compiler warnings. Most notably, any class that extends a Spring Controller will complain that the Controller it’s extending has been deprecated. Spring 3.0 pushes us hard into using annotations.

I was already using annotations for most of my controllers, but I did have one “catch-all” controller for URLs that I hadn’t mapped. The two bits in my Spring configuration that I changed included removing the bean for the AnnotationMethodHandlerAdapter (which in retrospect I may not have needed anyway), and removing the SimpleControllerHandlerAdapter and SimpleUrlHandlerMapping beans I’d configured for my catch-all, replacing that bean instead with @Controller and @RequestMapping(“/*”) annotations which did the same default URL handling.

This reduced the Controller/URL handling parts of my applicationContext.xml to just this bit.

Two lines, with wrapping, that controls all of our URL and JSP handling. The first tells the ViewReslover where the related JSP files are, and the second tells Spring in which package I’ve tucked my annotated Controllers. I also have some Hibernate configuration, Locale resolvers, and ResourceBundle bits, but they didn’t change a bit from 2.5 to 3.0, and this is enough to satisfy the annotated Controller discussion. Your paths may vary, and that’s a horrible package name.

If you’re not using annotations, here’s where it gets a little tricky: you’re probably using a specific Controller for specific tasks. Perhaps you’re a fan of the SimpleFormController or you roll your own with the AbstractController. The difficulty in making the change from those to the annotated Controller is going to depend on the complexity of the Controllers used and the functions therein. Let’s take a simple SimpleFormController as an example and convert it to an annotated Controller instead, to give a quick example.

First, one of our application context XML files would contain a bit not unlike this defining our bean and telling Spring to handle all otherwise unmapped URLs with our controller.

	
		
		
		
		
	
	
	
		
		
			
				exampleSimpleFormController

Our Controller class would look something like this, presumably doing a little bit more work between the call and return, but this actually meets our criteria for the example.

public class ExampleSimpleFormController extends SimpleFormController {
    @Override
    protected ModelAndView handleRequestInternal(final HttpServletRequest httpServletRequest, final HttpServletResponse httpServletResponse) throws Exception {
	// Do your work here
        return super.handleRequestInternal(httpServletRequest, httpServletResponse);
    }
}

We can convert this to an annotated Controller without too many changes and losing no functionality. Simply remove those bits in the application context XML file related to this bean, make sure it’s in a package covered by the annotated class scan (the context:component-scan bit above). Make the following changes, and you’ve converted the controller to an annotated one.

@Controller
public class ExampleSimpleFormController {
    @RequestMapping("/*")
    public String handleRequestInternal(final HttpServletRequest httpServletRequest,
		final HttpServletResponse httpServletResponse) throws Exception {
	// Do your work here
        return “index”;
    }
}

Again, of course, the complexity of your Controller will make that more difficult. This class, however, will function exactly the same in both versions. The name of the class can be changed, as can the name of the method handling our default request, with consideration to use elsewhere.

One obnoxious thing about this example is that it’s tediously simple. All it’s really going to do is handle any URL request not matched by another Controller and return the rendered WEB-INF/index.jsp page. The original Spring configuration implied, as a SimpleFormController would, that some action would be taking place, as we had defined a command object and had to define both success and failure view names. Let’s make a tougher example and convert it from a SimpleFormController to an annotated Controller with some simple form input.

Let’s begin with a JSP fragment that contains this simple login form:



Name 
Password

Note that the .ext used in the action should match whatever URL mapping done in the web.xml tying the URL to the DispatcherServlet. It could be that /* is handled by the DispatcherServlet, but that’s not recommended. We’d back this with an Object that presumably had a pair of member Strings, for the name and password, clumsily shortened for brevity like this:

public class Login {
    public String user = null;
    public String password = null;
}

We’d reference this class as our Controller’s command object, giving it the same name as the one in our form, all tied into our bean definition in an appropriate XML file. The Controller class then probably would have getters and setters to set the object. We can eliminate most of that busy work with our new annotated Controller that does a little trivial validation.

@Controller
@RequestMapping("/login")
public class LoginController {

    @RequestMapping(method = RequestMethod.GET)
    public String handleGET() {
        return “login”;
    }

    @RequestMapping(method = RequestMethod.POST)
    public String handlePOST(@ModelAttribute("Login") final Login login) {
        if ((login != null) && (login.user != null) && (login.password != null)) {
	      return “index”;
        }
       return “login”;
    }
}

Note that while the .ext was specified in the form, we can leave it out (or put it in) in the RequestMapping, which will by default then map all login.* and login/ requests to this Controller unless a better match to the URL is found. When the Controller gets a GET request, it simply renders the login.jsp page. When it gets a POST request, it verifies that the user and password fields have values (I did say trivial), and then renders the index.jsp page, otherwise it re-renders the login.jsp page.

A more comprehensive example will follow as we set up a new simple web application in the next section, but that’s the crux of a quick transformation from Spring 2.x to Spring 3.0 that will work with most projects. A little work to transform deprecated Controllers to annotated Controllers, but that can be postponed for a bit and done as time allows.

New Project Basics

Sadly, in my opinion, the Spring documentation spreads out all of the bits and pieces necessary to make a project from nothing. Additionally, what strong suggestions there are tend to start with “use the sample project and remove what you don’t need,” which is both tedious and intimidating if you’re not sure what is safe to remove. I prefer an approach of starting with nothing and add the minimum necessary to get things going.

Of course, a bit of up front design is always nice; it’s hard to make a project without any kind of intention behind it. For the purposes of this example, we’ll make a trivial in-memory Twitter clone out of just a couple of annotated Spring Controllers and the JSPs that render the pages. No security, graphics, or style sheets to clog the works, just simple annotations and tags.

For the view, we’ll have a simple one-page interface that will give us a simple form to add a post, a simple search form, and a list of the posts thus far, ordered in descending date (newest on top).

For our controllers, we’ll need something to handle the search, something to handle the post, and something to give us the list of previous posts. We’ll use a simple multi-action controller with a different target for each of our form actions.

For our model, we’ll simply have a post object, and we’ll maintain a small collection of them (so we don’t overwhelm our example system or run out of memory.

We’ll assume and discuss as if the whole world uses Java6, Tomcat v6, and Eclipse, and that all of the paths are correctly configured, and that Eclipse has a Tomcat server configured. In Eclipse, make a new Dynamic Web Project; give it a name (I’ll call it “microblog”), associate it with the Tomcat server, let it create source and web content folders, and generate a default web.xml file.

Since we know we’re going to use Spring for our framework, let’s put that in our application. Start with editing the WEB-INF/web.inf file in the web content folder. By default the XML contains a description that has the display-name of the application and a generous list of welcome-files. Reduce the welcome-file-list to just one, like index.htm and create an empty file in the root of the web content folder of the same name; a quirk in Tomcat will throw a 404 error if the file doesn’t exist, even though it won’t be used as we’ll replace it with a Spring Controller in just a bit.

A quick fix to another little Tomcat quirk, add a context-param for the webAppRootKey to avoid collisions with other applications. Give it a unique name so that Tomcat won’t complain every time the app starts, nor will any application fail because suddenly it’s looking in the wrong folder. The param-name needs to be webAppRootKey, but the param-value should be unique within your Tomcat server. Put this between the display-name and welcome-file-list.

	
		webAppRootKey
		microblog.appRoot

Then since we’re going to be making a really simple Spring application, we’ll start with the textbook minimum configuration. We need to declare our Servlet to handle our requests, and map URLs to the servlet. Put this between the context-param (or description if you and welcome file list

	
		microblog
		org.springframework.web.servlet.DispatcherServlet
		
			contextConfigLocation
			/WEB-INF/applicationContext.xml
		
	

	
		microblog
		*.htm

The servlet definition tells Tomcat that there’s a servlet named “microblog” of the type DispatcherServlet, with the provided parameter. The parameter will be passed to the DispatcherServlet and tell it where to find the context information; we’re going to tuck it away in the WEB-INF folder where web browsers can’t get to it. The servlet-mapping definition tells Tomcat that all URLs reaching this application that end in .htm will be handled by our microblog Servlet.

This gives us a simple WEB-INF/web.xml file that looks like this one.



	microblog
	
		webAppRootKey
		microblog.appRoot
	
	
		microblog
		org.springframework.web.servlet.DispatcherServlet
		
			contextConfigLocation
			/WEB-INF/applicationContext.xml
		
	
	
		microblog
		*.htm
	
	
		index.htm

Since we’ve declared that we’re going to use the DispatcherServlet, we’ll need to copy the appropriate Spring JAR file to our WEB-INF/lib folder. DispatcherServlet is in the org.springframework.web.servlet package, which most closely matches the org.springframework.web.servlet-3.0.0.RELEASE.jar, so copy that one. You can verify by searching for the type and Eclipse should be able to find it.

Although we’re going to be annotating our Controllers and such, some may point out how we’re sticking with the old XML way of configuring Spring. This is due in part to the comfort we have with the XML configurations, the trivial configuration we’re going to have, and finally because the Spring documentation that tells us that annotating the configuration is not a 100% replacement. Rather than anything confusing at this point, we’ll stick with the trivial XML we need to make our application.

This was discussed before, but so you don’t have to scroll back, this essentially two-line XML file (one bean, one context element) tells our DispatchServlet what we’re going to use for our view resolver (to render our JSPs), and to scan our package for annotations. Since we’ve stated that our JSPs are going to be in WEB-INF/jsp, we should take this opportunity to create that folder. Additionally, we’ve declared our package to scan, so we should create that path in our folder, too; in Eclipse we just create an empty package. Our two additional classes are in the org.springframework.web.servlet package with our DispatcherServlet, so we don’t need to copy an additional JAR file just yet.

Where to go next is a choice of style and discipline; controller, model, or view, or all at once. I think we need somewhere for the first URL request to go, so let’s build a simple catch-all controller to deliver our default page, and see where that leads us.

Our project is going to be terribly small, so despite any other design patterns, we’re going to put everything right in our previously created package. Let’s create a class named CatchAll to catch everything that isn’t otherwise handled.

package com.objectpartners.microblog;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class CatchAll {
	@RequestMapping("/*")
	public String catchAll() {
		return "index";
	}
}

Curious, the org.springframework.stereotype.Controller isn’t found in a JAR named after the package, and it doesn’t give any hint that this is an annotation class, either. We can find this is the right fully-qualified class name in the Spring API documentation, but finding it in the JAR files is a bit tedious. It’s hiding in the org.springframework.context-3.0.0.RELEASE.jar which is one of the few deviations from the new naming scheme we’ll run into. The RequestMapping annotation is correctly in the org.springframework.web-3.0.0.RELEASE.jar file. Add both of those files to our WEB-INF/lib and any compile errors so far will be resolved.

Our handler is woefully simple, and will tell Spring to render the index.jsp for every URL received. We told Spring before that these files would exist in the WEB-INF/jsp folder, so let’s make one right now; we’ll revisit and make it useful in a moment. Simply add a “hello, world” JSP so we can test that everything works so far; using Eclipse, just add a new JSP page to the folder, name it index.jsp, and add some text between the body tag.

So far, all looks well, and while not meeting our project goals, we should be at an acceptable starting point. That is, the app looks like it completes all of the required configuration elements. Add the project to the configured Tomcat server in Eclipse and start it up. Tomcat should start with no problem, but if we try to visit it (http://localhost:8080/microblog/), we’ll end up with an exception because we’re missing some classes we didn’t declare, but that Spring needs to function. There are going to be a lot of missing classes, so be prepared. In our Tomcat console, and probably the web page, we should see a stack trace that warns us that there’s a missing class.

java.lang.NoClassDefFoundError: org/springframework/beans/BeansException

We need to add org.springframework.beans.BeansException, which is in the org.springframework.beans-3.0.0.RELEASE.jar file, so copy that to the WEB-INF/lib folder and restart Tomcat. A second try, another exception for another missing class.

java.lang.NoClassDefFoundError: org/springframework/core/NestedRuntimeException

Copy the org.springframework.core-3.0.0.RELEASE.jar file to the WEB-INF/lib folder, restart Tomcat, and try again. We could have almost guessed we’d need beans and core, but we’re trying to add as few files as possible, remember. Trying again, and, yes, another failure. This time from a dependency Spring has for logging.

java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory

Quickly scanning the documentation, we can see that this is left to us to decide for ourselves which implementation we can or may want to use. Some application servers provide such logging, some environments need a little extra configuration. We could rebuild Spring without the dependency, or the easy solution, get the latest JAR from Apache Commons Logging (http://commons.apache.org/logging/) and add it to our WEB-INF/lib folder. Restart, retry, re-fail.

java.lang.NoClassDefFoundError: org/springframework/asm/ClassVisitor

Copy org.springframework.asm-3.0.0.RELEASE.jar to WEB-INF/lib, restart, reload.

java.lang.NoClassDefFoundError: org/springframework/expression/PropertyAccessor

This one is hiding in org.springframework.expression-3.0.0.RELEASE.jar so copy that, restart, reload.

The next one gets a little tricky.

java.lang.NoClassDefFoundError: javax/servlet/jsp/jstl/core/Config

While this should be a part of Tomcat, it isn’t. I suppose it’s possible that the tag library might not be used by enough Servlet or JSP applications to make it a default. If you’re not using Tomcat, you might have this in your J2EE engine’s classpath already. Since we are running Tomcat in our example, grab the Taglibs from the Apache website (http://tomcat.apache.org/taglibs/standard/). Grab the version 1.1 file (1.2 isn’t quite done yet) and copy the JAR files from the downloaded lib folder to your WEB-INF/lib folder. Restart Tomcat.

Finally, we should be rewarded with the text of our index.jsp page. Our annotated Spring web application works for now. Back to making it useful.

Since we were last editing the index.jsp, let’s add the tags and form elements to display a list of our micro-blogging posts and some simple forms for submitting a post and searching.

<%@ page language="java" contentType="text/html; charset=UTF-8"
	pageEncoding="UTF-8"%><%@ taglib prefix="form"
	uri="http://www.springframework.org/tags/form"%><%@ taglib prefix="c"
	uri="http://java.sun.com/jsp/jstl/core"%>



Microblog



	
	



	
	



	
	${post.post}
	${post.time}

Near the top we added a couple taglib lines telling the JSP renderer to use the Spring form and JSTL core tags. The first form is a big text area with a button to submit; it’s aiming for the post.htm action, and will use a page bean named “Post” for storing its data. The second form is an input and a submit button; it’s aiming for the search.htm action and will use a page bean named “Search” for storing its data. Finally, there’s a forEach loop that will simply spew whatever comes out of the array or collection page bean named “Posts.”

Let’s throw together a quick bean to satisfy the forms. We’ll cheat and use the same little guy, for brevity. A simple bean with two Strings.

package com.objectpartners.microblog;

public class Post {
	private String post = null;

	private String time = null;

	public String getPost() {
		return post;
	}

	public String getTime() {
		return time;
	}

	public void setPost(String post) {
		this.post = post;
	}

	public void setTime(String time) {
		this.time = time;
	}
}

We’ve already got our CatchAll Controller, so let’s just tweak that to handle these new requests.

package com.objectpartners.microblog;

import java.util.LinkedList;
import java.util.Calendar;
import java.util.List;

import javax.servlet.http.HttpServletRequest;

import org.springframework.stereotype.Controller;
import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class CatchAll {

	@RequestMapping("/*")
	public String catchAll(final HttpServletRequest httpServletRequest,
			final ModelMap modelMap) {
		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof List) {
			modelMap.addAttribute("Posts", object);
		}

		return "index";
	}

	@RequestMapping("/post")
	@SuppressWarnings("unchecked")
	public String post(final HttpServletRequest httpServletRequest,
			@ModelAttribute("Post") final Post post, final ModelMap modelMap) {

		LinkedList posts = null;
		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof LinkedList) {
			posts = (LinkedList) object;
		} else {
			posts = new LinkedList();
			httpServletRequest.getSession().getServletContext().setAttribute(
					"Posts", posts);
		}
		if (post.getPost() != null && !post.getPost().trim().isEmpty()) {
			post.setTime(Calendar.getInstance().getTime().toString());
			posts.offerFirst(post);
		}
		while (posts.size() > 100) {
			posts.pollLast();
		}
		modelMap.addAttribute("Posts", posts);

		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		return "index";
	}

	@RequestMapping("/search")
	@SuppressWarnings("unchecked")
	public String search(final HttpServletRequest httpServletRequest,
			@ModelAttribute("Search") final Post search, final ModelMap modelMap) {

		final Object object = httpServletRequest.getSession()
				.getServletContext().getAttribute("Posts");
		if (object instanceof List) {
			if (search.getPost().trim().isEmpty()) {
				modelMap.addAttribute("Posts", (List) object);
			} else {
				final List posts = new LinkedList();
				for (final Post post : (List) object) {
					if (post.getPost().toLowerCase().contains(
							search.getPost().toLowerCase())) {
						posts.add(post);
					}
				}
				modelMap.addAttribute("Posts", posts);
			}
		}

		modelMap.addAttribute("Post", new Post());
		modelMap.addAttribute("Search", new Post());

		return "index";
	}
}

We’ve upgraded our default request method, catchAll(), adding a couple parameters that Spring will autowire for us. We need the HttpServletRequest to gain access to the collection we’re maintaining in memory, or more specifically in the Servlet context’s memory (we could use a static variable). We need the ModelMap to store our page beans for the form. We could annotate these a little better, getting directly to the individual beans, but that actually adds as much extra class material to build what will ultimately result in a bunch of empty objects. The catchAll() simply grabs the collection from the ServletContext, if it exists, and sends back that collection and some empty beans to satisfy the JSP.

We added a post() method, and mapped that to the /post request to handle when the user taps the “Post!” button from the form. Note the ModelAttribute annotation telling Spring to associate the form data for a form named “Post” with the parameter. This method will create the collection if it doesn’t already exist. It validates that the post contains text (and nothing more) and adds it to the head of the list. It also truncates the list to be sure our simple list won’t get too long. Like catchAll(), it puts the necessary elements in the ModelMap to render the form.

Finally, the search() method will handle the /search request when the user taps the “Search!” button. Like the post() method it has an annotation allowing Spring to autowire the form data. This method searches for all posts containing the string as input. If the string is empty, all post elements are returned. The ModelMap is populated and the form is rendered.

In our trivial application, we have (after much trial-and-error correction) the minimum Spring and other JAR files necessary to support our application. We’ve got one annotated Controller class, handling all of the actions our one JSP page can dish out. It isn’t pretty, but it works.

Why Hadoop?

Joel Crabb — Mon, 22 Feb 2010 14:59:14 +0000

Hadoop is garnering a lot of attention these days as companies ponder how to address problems involving large amounts of data (tera to petabytes). It also gathers attention from companies interested in distributed computing. Lastly, it is seen as an alternative to an RDBMS. So what is Hadoop actually and how would it be used in the enterprise now and in the near future?

What is Hadoop?
Briefly, Hadoop is an Apache Open Source project that mirrors technologies developed by Google to power its search engine and other products (Google Maps, Analytics, etc.). Hadoop consists of a number of sub-projects but the main ones are Hadoop File System (HDFS), Map Reduce and Hbase (multi-dimensional sorted map used as a data store). The HDFS creates a massive, durable, cost-efficient file system by using three way replication of all files across a cluster of commodity servers.
For more on Hadoop: http://hadoop.apache.org

Use Case #1: Searching large quantities of unstructured data quickly.

Hadoop gathers most of its interest from companies that would like to search, analyze and store a large amount of data, in the tera to petabyte range. As Google’s Google File System (GFS) is the basis for Google’s search engine, Hadoop’s HDFS is the basis for Yahoo!’s search engine. Since the impetus for creating Hadoop was Internet search, obviously, the main usage for Hadoop has been in searching unstructured data such as web pages or other documents.

Use Case #2: Accessible, programmable distributed computing paradigm.

Distributed computing has long been seen as a shining light at the end of the processing tunnel, but has largely been unused in enterprise environments due to its complexity. The Map Reduce framework is starting to change that paradigm as it is a conceptually simple programming model that brings distributed computing down to a reachable level. Map Reduce combined with HDFS has proven successful by abstracting away any knowledge of where the data lives. Map Reduce works by pushing the processing to the data rather than moving the data to the processing. It has only become viable as computing power and memory has increased to the point where complex computations could be completed quickly on commodity hardware.

Use Case #3: RDBMS replacement for large data sets.

There would be little reason to replace your RDBMS with Hadoop if you have a reasonable database size (GB range). Where Hadoop starts looking legitimate is when the cost/benefit of purchasing larger and larger servers to run the RDBMS crosses the cost/benefit of purchasing and maintaining a large cluster of commodity servers. Throw in retraining your development resources from SQL based development to NOSQL based and that further complicates the picture against Hadoop. Where Hadoop wins out is when the RDBMS can no longer support the queries taking place on a massive data set. A restructure of the data set to Hadoop and Hbase can return the query times to a more reasonable level. Obviously, this is not a trivial amount of work but necessary if other alternatives fail.

Framework for a Multi-stage Spring Property Loader Extension Allowing Dynamic Updates of Properties via JMX

nadelman — Mon, 22 Feb 2010 14:30:00 +0000

Overview

It is commonplace in enterprise applications to allow application properties to be loaded from configuration files. When leveraging Spring, this is typically achieved using a PropertyPlaceholderConfigurer instance within the application. As is stated in the JavaDocs for this class, it is “A property resource configurer that resolves placeholders in bean property values of context definitions. It pulls values from a properties file into bean definitions.” (see http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/beans/factory/config/PropertyPlaceholderConfigurer.html)

Properties are loaded at application startup, prior to loading other spring beans for the application. Other beans in the application can then leverage the properties as part of their initialization, allowing externalization of various properties. This can be accomplished within Spring context files using placeholders containing the desired property keys, or via annotations in the classes themselves.

A common and vital pattern leveraged in many applications is to allow for environment-specific properties file. This means that different properties files may get loaded depending on the environment in which the application is running. There are a handful of strategies for supporting this pattern, and a full discussion is outside of the scope of this discussion. The one strategy that will be discussed is as follows: specify a location external to the application itself (such as a configuration folder living on the system where the application will be deployed) that will contain a consistently named property file (e.g. myApp.properties). When configuring the PropertyPlaceholderConfigurer spring bean, specify that location as the location of the properties file to load. This might look like:

Even more useful is to leverage multiple properties files, allowing for property overrides. This might look like:



    
        
            classpath:defaults.properties
            file:/my/app/config/myApp.properties

When using a list of property files for a PropertyPlaceholderConfigurer, entries later in the list take precedence over entries earlier in the list when there are property key collisions. In the above example, we specify an internal property file within the distribution called default.properties, and a property file external to the application. In the event that there is overlap of property keys between the files, the values in the external properties file would be used (because it is last in the list). Furthermore, the full set of properties loaded would be the union of the properties contained in the two files (e.g., there is no stipulation that the properties contained in the files correlate at all, unless you desire it). There are many advantages to this simple approach to property loading, including:

The default (internal) property file can contain default values for all properties, if desired, ensuring that there will not be “missing” properties in your application
The external property file can be updated manually on the server without requiring redeployment. However, an application restart would be required to pick up the modifications to the file.
Properties that should never be modified without an application redeployment can be exclusively contained within the internal properties file. This might be important if there are properties that you need to be configurable for unit testing purposes, but otherwise need to be consistent in all deployments regardless of environment.

At this point a developer with some knowledge of Spring is probably saying “so what”. This is nothing that isn’t already outlined in Spring’s documentation and dozens of examples out there on the web. So, here is where we can introduce some extensions that make the configuration framework more flexible.

Enabling Multi-stage Property Loading

At a recent client, there was a requirement for an application that we were writing that many of it’s properties be configurable from a central configuration database. Essentially, the database represented servers, applications on those servers, and key-value property pairs for each application/server combination. The idea is that it is a one-stop shop for managing the configurations for all applications (and in fact all instances of each application) in the enterprise. In contrast to that requirement (or in conjunction with it), from a development and testing perspective we wanted:

An internal properties file to represent defaults and properties that should not change from environment to environment (e.g. should not be configurable without an application redeployment, as they may substantially affect some piece of logic within the application)
An external, environment-specific properties file that could be changed at will (at least in local and dev environment), and could contain overrides for a number of the properties in the internal properties file.

Given these three sources for properties, we needed to put together an extension to the property placeholder framework that could:

Load properties from multiple sources
- Including a database
Allow staged property loading with flexible override ordering. In other words, we wanted to be able to specify whether the internal, external, or database properties should take precedence when configuration keys overlap.

As we saw in the previous section, dealing with the internal and external properties files is easy. However, dealing with the database properties files takes a little more work..

For beginners, we need a basic extension to PropertyPlaceholderConfigurer to work with. We will start with the following class, StagedPropertyPlaceholderConfigurer, which extends PropertyPlaceholderConfigurer.


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;

    private Resource internalProperties;
    private Resource externalProperties;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;

        // if this constructor is used, a data source will not be used for property loading
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }
}

As you can see, this class contains properties for the internal and external property files, and a basic constructor taking these arguments. The configurer is set up to be used without the database at all, for the moment. Some flags to indicate where to load properties from for reference during the actual loading.

Next, we need to override the loadProperties method from the PropertyPlaceholderConfigurer class. The Spring framework will call this method to load the properties. For the moment, lets make it simple, since we are not considering the database quite yet:


@Override
protected void loadProperties(final Properties props) throws IOException {
    List resources = new ArrayList();

    if(loadFromInternalPropertiesFile) {
        resources.add(internalProperties);
    }

    if(loadFromExternalPropertiesFile) {
        resources.add(externalProperties);
    }

    Resource[] locations = new Resource[resources.size()];
    resources.toArray(locations);
    setLocations(locations);

    super.loadProperties(props);
}

At this point, we have essentially re-invented the wheel, so lets go a bit further. Now we are going to add the ability to configure a data source to load properties from, and specify how to load them. In order to do this in as generic a fashion as we can, we will define a DatasourceProperyLoaderStrategy interface as follows:



/**
 * Interface to be implemented for defining a data source-based property loading strategy to use in conjunction
 * with the StagedPropertyPlaceholderConfigurer.
 *
 */
 public interface DatasourcePropertyLoaderStrategy {

    /**
     * provide any configuration needed to connect to the data source. This should include creation of the data source
     * and any other initialization necessary.
     */
     public void configure(Properties properties);

    /**
     * Load properties from the configured data source. If overrideExisting is true,
     * replace any existing properties in the {@link Properties} instance with the ones loaded from
     * the data source. Otherwise, the existing properties take precedence in the case of property key collisions
     *
     * @param properties {@link Properties} instance containing any propreties that have already been loaded
     * @param overrideExisting if true, the properties loaded from this data source take precedence in the case of key collisions.
     */
    public void loadProperties(Properties properties, boolean overrideExisting);

    /**
     * Any cleanup required after properties are loaded, for instance disposing of the data source
     */
    public void tearDown(Properties properties);

}

An implementation of this interface is necessary for defining how properties would be loaded from a data source. This implementation would need to define:

How to connect to the data source (e.g., would need to be aware of data source attributes such as the database urls, drivers, credentials, and so on).
How to load properties from the database (e.g. where are properties loaded, in what format, how to process them)
How to merge properties pulled from the data source with the other properties (e.g., is manipulation of keys required, are there case-sensitivity concerns, should the properties loaded from the database override the previously loaded properties)
How to clean up the data source and dispose of it when property loading is completed.

Note that all methods on this interface take the Properties instance. Obviously this is needed to merge properties from the database with the previously loaded properties. However, the fact that they are passed to the configure() and teardown() methods allows previously loaded properties to be leveraged while determining how to load the additional properties. For instance, we can put the data source attributes in the external, environment-specific property file. This file would be loaded first, and therefore the data source attributes contained within that configuration file can be used to establish the data source to pull additional properties from the database.

Another useful aspect of this interface is that it would not be limited to pulling properties from a database. You could easily create an implementation that pulled configuration from another source, such as a web service.

If you were to implement a strategy to load from a database, the org.springframework.jdbc.datasource.DriverManagerDataSource class is a safe bet for creating a data source.

Okay, now back to the StagedPropertyPlaceholderConfigurer itself. With an implementation of DatasourcePropertyLoaderStrategy assumed to be defined, lets make the following modifications (in bold):


public class StagedPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {

    private boolean loadFromDB = true;
    private boolean loadFromExternalPropertiesFile = true;
    private boolean loadFromInternalPropertiesFile = true;
    private boolean databaseTakesPrecedence = true;

    private Resource internalProperties;
    private Resource externalProperties;

    private DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy;

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties, DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy, boolean databasePrecedence) {
        super();
        this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        this.databaseTakesPrecedence = databasePrecedence;

        if(databasePropertyLoaderStrategy == null) {
            loadFromDB = false;
            databaseTakesPrecedence = false;
        }

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties) {
        super();
        this.internalProperties = internalProperties;
        this.externalProperties = externalProperties;
        loadFromDB = false;

        if(internalProperties == null) {
            loadFromInternalPropertiesFile = false;
        }

        if(externalProperties == null) {
            loadFromExternalPropertiesFile = false;
        }
    }

    @Override
    protected void loadProperties(final Properties props) throws IOException {
        List resources = new ArrayList();

        if(loadFromInternalPropertiesFile) {
            resources.add(internalProperties);
        }

        if(loadFromExternalPropertiesFile) {
            resources.add(externalProperties);
        }

        Resource[] locations = new Resource[resources.size()];
        resources.toArray(locations);
        setLocations(locations);
        loadProperties(props);

        if(loadFromDB) {
            try {
                databasePropertyLoaderStrategy.configure(props);
                databasePropertyLoaderStrategy.loadProperties(props, databaseTakesPrecedence);
                databasePropertyLoaderStrategy.tearDown(props); 
            } catch (Exception e) {
                // just log the exception? This happen if the datasource is unavailable or the data returned
                // cannot be processed correctly. In this case, the DatasourcePropertyLoaderStrategy should be responsible for
                // any necessary exception handling.
            } 
        } 
    }
}

So far so good. We have a property loader that can take properties from multiple sources and merge them. It can pull property from an external data source, whether that is a database, a web service, or anything else. It can be configured to allow different property sources to take precedence over others. And the only real work involved is in implementing the DatasourcePropertyLoaderStrategy. But, as long as we have gone this far, lets go a bit further.

Exposing a Configuration Bean for Use Throughout the Code Base

What has been discussed up to this point allows for configuration modifications that would only be picked up with an application restart. It would be nice to be able to dynamically update properties via JMX without an application restart. This can be very useful for tuning applications (if you have properties that are considered “tunable”), or allowing dynamically modifiable behavior. For example, if your application posts content over http to some URL, you might want to be able to modify that URL without restarting the application or having to monkey around with the network.

In order to facilitate this, we will further modify the property placeholder configuration framework, enabling the creation of a “configuration bean” that can be leveraged throughout the application. This configuration bean would get populated with the final output of the StagedPropertyPlaceholderConfigurer. With some modifications to how you access your properties in your code, the bean can then be leveraged to get access to properties. And the bean can be exposed as a managed resource to allow dynamic modification of the properties.

First, lets will define a ConfigurationBean interface as follows:


public interface ConfigurationBean {

    public Map getProperties();
    public void setProperties(final Properties props);
    public String getValue(String key);
    public Integer getIntValue(String key);
    public Long getLongValue(String key);
    public Boolean getBooleanValue(String key);
    public String setProperty(String key, String value);
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception;
}

And a base implementation of this interface:


public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map props = new HashMap();

    /**
     * Gets the properties.
     *
     * @return the properties
     */
    public Map getProperties() {
        if (props == null) {
            return null;
        }

        return Collections.unmodifiableMap(props);
    }

    /**
     * Sets the properties for the configuration bean based on Properties loaded by the Property Placeholder Configurer
     *
     * @param props
     */
    public void setProperties(final Properties props) {
        for (Iterator iterator = ((Map) props).keySet().iterator(); iterator.hasNext();) {
            String key = (String) iterator.next();
            this.props.put(key, props.getProperty(key));
        }
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

    /**
     * Gets the value for the specified key as an Integer
     *
     * @param key
     * @return the value
     */
    public Integer getIntValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Integer.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Long
     *
     * @param key
     * @return the value
     */
    public Long getLongValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Long.valueOf(val);
    }

    /**
     * Gets the value for the specified key as a Boolean
     *
     * @param key
     * @return the value
     */
    public Boolean getBooleanValue(String key) {
        if (props == null) {
            return null;
        }

        String val = props.get(key);
        if (val == null) {
            return null;
        }

        return Boolean.valueOf(val);
    }

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap();
        }

        return props.put(key, value);
    }

    /**
     * Used to populate the collection target, with values of type classType. classType must have a single String argument constructor to create new instances.
     *
     * Example usage
     *
     * List toFill = new ArrayList();
     *
     * loadIntoCollection("propertyKey",toFill, Long.class);
     *
     * @param key
     * @param target The collection to load config data into cannot be null
     * @param classType The class type to cast the config's String value to. Should be the class type assigned to the Collection target
     *
     * @throws Exception
     **/
    public void loadIntoCollection(String key, final Collection target, Class classType) throws Exception {
        if (props == null) {
            return;
        }

        if (target == null) {
            throw new IllegalArgumentException("Argument target cannot be null");
        }

        if (classType == null) {
            throw new IllegalArgumentException("Argument classType cannot be null");
        }

        String val = props.get(key);
        if (val == null || val.length() == 0) {
            return;
        }

        String[] vals = val.split(",");
        Constructor con;

        try {
            con = classType.getConstructor(String.class);
        } catch (NoSuchMethodException e) {
            throw new IllegalArgumentException(classType + " does not have a single String argument constructor ");
        }

        for (int i = 0; i < vals.length; i++) {
            try {
                target.add(con.newInstance(vals[i]));
            } catch (Exception e) {
                throw new Exception("Config Value: " + vals[i] + " cannot be converted to " + classType.getName());
            }
        }
    }
}

This interface can of course be extended to handle additional data types, or to provide more granular methods for getting and setting specific properties. In fact, it would be a good idea for any application using this framework to create a ConfigurationBean implementation that extends ConfigurationBeanImpl, just for the flexibility it would provide.

Now we need to define a singleton instance of the ConfigurationBean that will be used within the StagedPropertyPlaceholderConfigurer (and elsewhere in the application). For instance:

Next, the StagedPropertyPlaceholderConfigurer needs to be updated to populate the ConfigurationBean instance. There are a few things that need to be added to this class to support this:

We need a handle on the bean factory, as we will need to look up the ConfigurationBean instance. This can be accomplished by:
- Defining a private data member for the BeanFactory (There is a BeanFactory instance on the superclass PropertyPlaceholderConfigurer, but it is hidden):
- Overriding the setBeanFactory method as follows:

We need to know the id of the ConfigurationBean singleton in order to be able to find it in the bean factory.

Add the string name as a data member:


private String configurationBeanName;

Update all constructors to take the configuration bean name as a parameter. E.g.



public StagedPropertyPlaceholderConfigurer(Resource internalProperties, Resource externalProperties,
                                           DatasourcePropertyLoaderStrategy databasePropertyLoaderStrategy,
                                           boolean databasePreceduence, String configurationBeanName) {
    super();
    this.configurationBeanName = configurationBeanName;
    this.databasePropertyLoaderStrategy = databasePropertyLoaderStrategy;
    this.internalProperties = internalProperties;
    this.externalProperties = externalProperties;
    this.databaseTakesPrecedence = databasePreceduence;

    if(databasePropertyLoaderStrategy == null) {
        loadFromDB = false;
        databaseTakesPrecedence = false;
    }

    if(internalProperties == null) {
        loadFromInternalPropertiesFile = false;
    }

    if(externalProperties == null) {
        loadFromExternalPropertiesFile = false;
    }
}

We need to implement a method to populate the ConfigurationBean instance with the output of the property loading process:



/**
 * Populates the Configuration bean with the properties that have been loaded.
 *
 * @param props the props
 */
protected void loadServerConfigBean(final Properties props) {
    ((ConfigurationBean) beanFactory.getBean(this.configurationBeanName, ConfigurationBean.class)).setProperties(props);
}

And, finally, we need to add a call to loadServerConfigBean after all of the properties have been loaded/merged. Add the following as the last line of code in the loadProperties method:


loadServerConfigBean(props);

Okay, that was a lot of little things, but now we are done with the property placeholder configurer and populating the configuration bean. The next step is to add a bean to the Spring context to define the property placeholder. This bean definition, along with the configuration bean definition and the definition of a DatasourcePropertyLoaderStrategy, might look like:

After application startup, the configuration bean would now contain the current values of all of the application properties. In fact, you could extend the framework so that the configuration bean will ultimately contain more than the union of the contents of all of the configuration files and data sources. One example might be to load in configuration values for the host ip and host name into configuration based on an InetAddress instance created during application startup.

Now we need to discuss the usage of the configuration bean. This will require a little change in programming paradigm where you want to leverage the configuration bean. First of all, in order to access properties in the configuration bean, you need to know the property keys. Therefore, it is useful to create a “configuration constants” class or interface whose sole purpose is to contain constants for every configuration key that will be of interest. For instance, if you have a property with a key of “my.first.property”, define a constant:


public static final String MY_FIRST_PROPERTY_KEY = “my.first.property”;

Next, let suppose that we have a class Foo that leverages the current value of “my.first.property” in a method bar. We can get leverage the configuration bean by wiring it into the class and using the methods on the configuration bean to access the value. For instance



package com.objectpartners.sandbox.config;

import org.springframework.beans.factory.annotation.Autowired;

public class Foo {

    @Autowired
    private ConfigurationBean configurationBean;

    public static final String MY_FIRST_PROPERTY_KEY = "my.first.property";

    public void bar() {
        System.out.println("value of my.first.property = " + configurationBean.getValue(MY_FIRST_PROPERTY_KEY));
    }
}

The Last Step – JMX

Now we are down to the last step – allowing the configuration values to be updated dynamically via JMX. The quickest way to do this will be to make the Configuration Bean a managed resource.

Add the following to your Spring configuration:

/>

Next, add the following to your implementation of the ConfigurationBean, right before the class declaration (it is assumed that you have subclassed ConfigurationBeanImpl, extended the ConfigurationBean interface, or both):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")

Finally, for any methods that you want to be exposed via JMX, add the following before the method declaration:


@ManagedAttribute

For any getters or setters decorated with the @ManagedAttribute, you will now be able to view the return values of those methods as attributes on a configuration mbean using your favorite JMX client (I would recommend JVisualVM for day-to-day use). Additionally, any getters and setters decorated with the @ManagedAttribute annotation will be presented as an operation on this mbean. Lets suppose that we go with the base implementation of ConfigurationBean, and add these annotations (entire class not listed, just the pertinent portions):


@Component
@ManagedResource(objectName="example:name=configuration", description = "Configuration")
public class ConfigurationBeanImpl implements ConfigurationBean {

    /** The props configuration key-value pairs */
    private Map props = new HashMap();

    /**
     * Sets the specified property with the specified value
     *
     * @param key
     * @param value
     * @return the previous value of the property
     */
    @ManagedAttribute
    public String setProperty(String key, String value) {
        if (props == null) {
            props = new HashMap();
        }

        return props.put(key, value);
    }

    /**
     * Returns the value for the specified key, or null if no such key exists.
     *
     * @param key
     * @return the value
     */
    @ManagedAttribute
    public String getValue(String key) {
        if (props == null) {
            return null;
        }

        return props.get(key);
    }

In the JMX console, you will now be able to retrieve the current value for any property key using the getValue() operation, and will be able to dynamically update the value of any property via the setProperty() operation.

Additional Considerations and Final Thoughts:

There are some considerations that should be kept in mind if you plan on leveraging this framework or extending it:

Allowing any configuration value to be updated dynamically via JMX might not be desired functionality. If there are certain values that you might want to update via JMX, and certain ones that you would not want to, then you can implement more granular methods in your configuration bean, and only expose those methods in JMX.
It should be noted that the @ManagedAttribute annotation is only valid for getters and setters. If you want to expose an operation for some other type of method, use the @ManagedOperation annotation.
There are certainly situations where it does not make sense to allow dynamic updates of configuration values, or where it would have no effect. For instance, suppose your database driver is a configuration value, and it is used to set up your data source. That is probably only going to happen once, at application start up. Changing the configuration value after application startup is not going to have any affect in such a situation.
There would also be a multitude of types of configuration that you would not want to be available for dynamic update at all. This is why it generally would make sense to extend (or replace) the ServerConfig interface, and only include those operations that you absolutely need exposed in JMX for your application.
Remember that configuration values do not need to be single values. It is common to have comma-separated lists for configuration values. Managing these types of configurations via JMX may take some specialized operations (e.g., implementation of some operations from the List interface, exposed as JMX operations)
This is by no means a perfect framework, and could be refined and extended in many ways. For instance:
- Rather than only allowing one internal property file, one external property file, and one Datasource property loading strategy, you could easily update the framework to allow multiples of each

It should also be noted that there have been a lot of updates in relation to how properties can be loaded within Spring 3. Those capabilities are well worth looking at as well.

Enabling JMX monitoring of SEDA Queue Depths in Apache Camel

nadelman — Mon, 15 Feb 2010 14:20:53 +0000

Apache Camel’s SEDA (Staged Event-Driven Architecture) endpoints (http://camel.apache.org/seda.html) provide a useful and quick mechanism to implement asynchronous, event-driven processing within your applications. See http://www.eecs.harvard.edu/~mdw/proj/seda/ for the original description of the SEDA architecture. For those not very familiar with Camel, implementing a SEDA route can be as simple as adding the following to a route builder class:


from(“seda:start”).beanRef(“someBean”, “someMethod”);

Exchanges (messages) put to the seda:start endpoint will be processed through the remainder of the Camel route. The SEDA endpoint encapsulates an in-memory queue used as an entry point to the route. All messages are processed asynchronously. In the example above, the messages would be routed to the someMethod method on a spring bean named someBean.

From a monitoring perspective, the throughput and other statistics for this route can be viewed via Camel’s JMX functionality (assuming that JMX is enabled). However, it should be noted that the queue depth for the SEDA queue is not exposed by Camel’s JMX framework. This queue depth may be of interest for application monitoring purposes. The following describes a method for exposing SEDA queues for monitoring. Additionally, this method resolves an issue inherent to the current implementation of the SedaEndpoint in Camel: the fact that the maximum queue depth is bounded.

Step 1: Define the endpoint in the camel context

In the Camel context, define the SEDA endpoint that will be the input to the route:



    com.objectpartners.camel

When the context is loaded, this will instantiate the SEDA endpoint and place a bean in the spring registry for it.

Step 2: Create the route builder class

In the route builder class, wire in the endpoint and create the route as follows:


package com.objectpartners.camel;

import org.apache.camel.Endpoint;
import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.seda.SedaEndpoint;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;

public class Routes extends RouteBuilder {

    @Autowired
    @Qualifier(value = "startEndpoint")
    private Endpoint startEndpoint;

    /**
     * Configure the routes
     **/
    @Override
    public void configure() throws Exception {
        from(startEndpoint).beanRef(“someBean”, “someMethod”);
    }
}

Step 3: Alter the properties of the SEDA endpoint

At this point, we must change the properties of the SEDA endpoint to make the depth unbounded (this step would be optional, if you are okay with the depth being bounded. The default maximum depth is 1000, but can be configured explicitly on the endpoint.), and to allow the queue depth to be accessible via JMX. JMX exposure is accomplished by exposing the SEDA endpoint’s backing queue as a managed resource. Enabling unbounded queue depth can be accomplished by changing the blocking queue implementation used by the SEDA endpoint. The following code assumes that the following have been placed in your application context:

/>

The final code for the route builder class:


package com.objectpartners.camel;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

import org.apache.camel.Endpoint;
import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.seda.SedaEndpoint;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.jmx.export.annotation.ManagedAttribute;
import org.springframework.jmx.export.annotation.ManagedResource;
import org.springframework.stereotype.Component;

@Component
@ManagedResource(objectName="example:name=TestRoutes", description = "Camel Test Routes")
public class Routes extends RouteBuilder {

    @Autowired
    @Qualifier(value = "startEndpoint")
    private Endpoint startEndpoint;

    // the new blocking queue for the SEDA endpoint
    BlockingQueue startEndpointQueue = new LinkedBlockingQueue();

    /**
     * Configure the routes
     **/
    @Override
    public void configure() throws Exception {

        // change the BlockingQueue implementation for the seda endpoint
        ((SedaEndpoint) startEndpoint).setQueue(startEndpointQueue);

        from(startEndpoint).beanRef(“someBean”, “someMethod”);

    }

    @ManagedAttribute
    public long getQueueDepth() {
        return startEndpointQueue.size();
    }
}

The Groovy Spaceship Operator Explained

bkarels — Mon, 08 Feb 2010 14:19:55 +0000

While extending an existing Grails application bits of research around sorting lists using Groovy ensued. My first stop was Groovy In Action wherein the operator <=> was shown in a closure used for sorting, but with no explanation. Under Operators in the index, it became known that <=> is “The Spaceship Operator” – no doubt due to its similarity in look to a flying saucer. Sadly the referenced section of text also offered little in explaining what this bugger did exactly.

The spaceship operator has it’s roots in Perl and has found it’s way in to languages like Groovy and Ruby. The spaceship is a relational operator that performs like Java’s compareTo() comparing two objects and returning -1, 0, or +1 depending on the value of the left argument as compared to the right. If the left argument is greater than the right, the operator returns 1. If the left argument is less than the right, the operator returns −1. If the arguments are equal, 0 is returned. When using Groovy/Grails, if the arguments are not of comparable types, a ClassCastException is thrown.

While the spaceship could be used wherever you would use compareTo(), it seems most references to it are found in sorting. For example the following two lines would have identical results:

things.sort{ Thing a, Thing b -> b.beginDate <=> a.beginDate }

things.sort{ Thing a, Thing b -> b.beginDate.compareTo( a.beginDate ) }

Other examples of its behaviour from the Groovy Console:

groovy> def d = new Date()
groovy> println d <=> d
groovy> println 3 <=> 4
groovy> println “doberman” <=> “dachshund”
groovy> println d <=> new Date()

0
-1
1
-1

Perhaps the greatest advantage of using the Groovy comparison operators is the graceful handling of nulls such that x <=> y will never throw a NullPointerException:

groovy> println “dachshund” <=> null
groovy> println null <=> “dachshund”
groovy> println null <=> null

1
-1
0

In addition when comparing numbers of different types the type coercion rules apply to convert numbers to the largest numeric type before the comparison. So the following is valid in Groovy¹:

groovy> Byte a = 27
groovy> Double b = 9
groovy> println a instanceof Byte
groovy> println b instanceof Double
groovy> println a > b

true
true
true

We come in peace – happy comparisons! For more on Groovy operator overloading look here.

¹ “Groovy – Operator Overloading” Available at: http://groovy.codehaus.org/Operator+Overloading 26 January 2010

Immutable Data Structures in Concurrent Java Applications

jmcclure — Mon, 01 Feb 2010 14:39:26 +0000

Concurrent applications have multiple threads running simultaneously. Access to data shared by multiple threads requires synchronization which is often a source of fragile and hard to maintain code, hard to find bugs, and performance issues. You can minimize synchronization and the headaches that go with it using immutable data structures. In this article I’ll demonstrate how to create and use immutable data structures to reduce and localize the need for synchronization.

Consider an application that uses a set of contacts. The contacts are stored in a map with the contact name as the key and a Contact object as the value. Various threads in the application access the contacts. Both the Contact class and the map require synchronization. Here is thread safe implementation of the Contact class:

public class Contact {
    private volatile String name;
    private volatile String email;
    private volatile String phone;

    public String getName() {return name;}
    public void setName(String name) {this.name = name;}

    public String getEmail() {return email;}
    public void setEmail(String email) {this.email = email;}

    public String getPhone() {return phone;}
    public void setPhone(String phone) {this.phone = phone;}
}

I could have chosen to synchronize all the setters and getters but making the fields volatile is a simpler solution. Using this implementation for the contact objects and ConcurrentHashMap for the contact map provides a thread safe solution. Usually thread safety at the class level alone is not sufficient to support application logic as demonstrated in the following method:

public void handleGmailNotification(Contact contact) {
    if (contact.getEmail().endsWith("gmail.com")) {
        sendGmailNotification(contact.getEmail());
    }
}

This method is supposed to send a special email to contacts with a GMail address. If another thread changes the contact between the execution of the first two lines of code the GMail specific email may be sent to a non-GMail account. That may only happen once a month and will likely be a difficult bug to find especially if the code that changed the email was added by a developer unaware of the use above. More course grained synchronization can be used to fix the problem but that requires that all code that deals with the Contact’s email address be correctly synchronized. Multiply that by the many different pieces of data being shared by multiple threads in a highly concurrent application and you get fragile, bug ridden, and hard to maintain code. Making Contact immutable alleviates these issues:

public final class Contact {
    private final String name;
    private final String email;
    private final phone;

    public Contact(String name, String email, String phone) {
        this.name = name;
        this.email = email;
        this.phone = phone;
    }

    public String getName() {return name;}
    public String getEmail() {return email;}
    public String getPhone() {return phone;}
}

Using the immutable version I can be sure that the Contact object I’m interacting with will not change. If I need to make a change to the email address I create a new Contact object and put it in the contact map in place of the existing one.

The ConcurrentHashMap, while thread safe, can cause similar issues. This code is supposed to send a text message and an email to every contact:

public void sendMessages(Map contactMap) {
    sendTextToPhone(contactMap.values();
    sendEmail(contactMap.values();
}

If another thread updates the contact map while this is executing we will have another hard to find bug. You could make the collection unmodifiable using Collections.unmodifiableMap but its not truly immutable. The underlying map can still be modified. Also, you have no way to change the contact list. A map implementation that creates a copy when its modified is a nice alternative. Here’s a simple implementation:

public class ImmutableMap implements Map {
    private final Map map;

    public ImmutableMap() {
        this(Collections.EMPTY_MAP);
    }

    public ImmutableMap immutablePut(K key, V value) {
        Map newMap = new HashMap(map);
        newMap.put(key, value);
        return new ImmutableMap(newMap);
    }

    public ImmutableMap immutableRemove(K key) {
        Map newMap = new HashMap(map);
        newMap.remove(key);
        return new ImmutableMap(newMap);
    }

    private ImmutableMap(Map delegate) {
        this.map = Collections.unmodifiableMap(delegate);
    }

    // All the map methods are simply delegated to the map field.
    // To conserve space they are not shown here.
}

The immutablePut and immutableRemove methods return a copy of the map with the changes, leaving the original map unchanged. All the methods from the Map interface delegate to the underlying HashMap. Since it is unmodifiable any mutating methods will throw UnsupportedOperationException. Immutable versions of other mutating map methods can be implemented using the same pattern as immutablePut and immutableRemove. You can safely use any instance of this type in any way with no concern about causing issues in other parts of the application or vice-versa.

Its likely the application we’ve been using as an example will require a central reference to the contact list. Here’s a contact service implementation using the ImmutableMap:

public class ContactService {
    private final ReentrantLock lock = new ReentrantLock();
    private volatile ImmutableMap contacts = new ImmutableMap();

    public void addContact(Contact contact) {
        lock.lock();
        try {
            contacts = contacts.immutablePut(contact.getName(), contact);
        } finally {
            lock.unlock();
        }
    }

    public ImmutableMap getContacts() {
        return contacts;
    }
}

There is some synchronization required to ensure that if multiple threads want to add a contact to the central list all additions are maintained correctly. Without the lock, if two threads called addContact at the same time they would each add a different contact to the existing map and the last one assigned would be the only one saved. The other would be lost. The contact must be volatile to ensure all threads calling getContacts will get the updated reference. The synchronization required in this case is very localized and easy to maintain. Any code that gets the contacts from the service can interact with them without synchronization concerns.

This paradigm works best in cases where updates are infrequent and the data set is relatively small. If updates are frequent or the data set is very large then the copying required can become a performance issue. Don’t let that deter you from adopting this paradigm where you can. Immutable data structures make concurrent programming a much simpler and safer endeavor. It is a proven approach that is fundamental in functional programming languages such as Scala and gaining traction in the object oriented world.

Using JPA and JAXB Annotations in the Same Object

smccoole — Mon, 25 Jan 2010 14:23:05 +0000

Recently, I started working on some projects where I had to prototype various service end point technologies, including REST and AMF. During this process, I worked out a fairly nice prototype project template that makes setting up a project fairly simple and quick. I plan to write about the template more fully in a forthcoming post, but for now wanted to share a small gotcha that took me a bit to figure out.

I’m using JPA for persistence and JAXB to produce XML to some of the end points. I’ve also been using annotations to minimize the amount of XML configuration that I have to do. Things were going really well, until I wanted to customize the XML representation of one of my domain object members as well as persist it. The member was a date and I wanted a simple month-day-year display rather than JAXB’s default locale formatting. I started out with:

@Entity
@XmlRootElement
public class Person implements Serializable {

  @Temporal(TemporalType.DATE)
  private Date birthDate;

  public Date getBirthDate() {
    return birthDate;
  }

  public void setBirthDate(Date birthDate) {
    this.birthDate = birthDate;
  }
}

This worked but when I retrieved my XML via the REST end point my dates looked like:

1987-11-01T00:00:00-06:00

Kind of ugly. Since formatting dates is quite common in the problem domain that these technologies would eventually be used in, I decided to add an example of it to the prototype. I wrote a quick date adapter class:

import java.text.SimpleDateFormat;
import java.util.Date;

import javax.xml.bind.annotation.adapters.XmlAdapter;

public class DateAdapter extends XmlAdapter {
 // the desired format
 private String pattern = "yyyy-MM-dd";

 public String marshal(Date date) throws Exception {
 return new SimpleDateFormat(pattern).format(date);
 }

 public Date unmarshal(String dateString) throws Exception {
 return new SimpleDateFormat(pattern).parse(dateString);
 }
}

And simplistically tried to set it up by adding the annotation to the member declaration.

@Entity
@XmlRootElement
public class Person implements Serializable {

  @Temporal(TemporalType.DATE)
  @XmlJavaTypeAdapter(value=DateAdapter.class)
  private Date birthDate;

  public Date getBirthDate() {
    return birthDate;
  }

  public void setBirthDate(Date birthDate) {
    this.birthDate = birthDate;
  }
}

When I went to marshall the object, I hit an IllegalAnnotationException.

Caused by: com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 1 counts of IllegalAnnotationExceptions
Class has two properties of the same name "birthDate"

It took a bit to figure out what was going on, and there was surprisingly little information or examples that tried to do this sort of thing when I searched around. Eventually, I figured out what might be obvious to some. The trick is to separate the two types of annotations and only use the JPA annotation on the member declaration and the JAXB annotation on the accessor (get) method. When I altered my code to this:

@Entity
@XmlRootElement
public class Person implements Serializable {

  @Temporal(TemporalType.DATE)
  private Date birthDate;

  @XmlJavaTypeAdapter(value=DateAdapter.class)
  public Date getBirthDate() {
      return birthDate;
  }

  public void setBirthDate(Date birthDate) {
    this.birthDate = birthDate;
  }

}

Everything started working as expected. Hopefully this can be helpful to anyone that encounters a similar situation.

Agile Project Inception, Success and Pitfalls

mwaldal — Thu, 21 Jan 2010 13:45:10 +0000

Agile projects have been around for years now, but for companies that are still new to the concepts and process, some simple pitfalls can delay your project. I have seen projects at companies that have run into challenges so here are some ideas for successful starts and pitfalls to avoid.

What could be so hard about starting a software project? Right! Well sometimes when pulling a group together to form an agile team you will get different ideas from each person about what agile means to them. In addition, being on a long project may only expose you to the project inception once a year perhaps. To give some ideas of the activities involved in an agile project inception, I thought I would give some suggestions for getting off to a good start, and list a few pitfalls to avoid. This isn’t a comprehensive list, I only wanted to share a few activities that are key. As a project manager there may be more activities to do. So here are a few important ones.

To get your Agile project off the ground create a project charter with a problem statement or opportunity statement and then a context document for the project. The context document will explore and document the domain and shows the users and their goals and actions that they will perform on the domain. I like to see a User Goal Diagram or Use Case Context used to achieve this, and I have found this type of document allows all business persons to describe what will need to be created or perhaps what doesn’t need to be created. I have seen some spreadsheets used in place of a context diagram that list out the features and or components to be created, but usually they do not describe dependencies or relationships between the items in the domain. I would suggest a visual document so that all on the team may understand and see the ownership and relationships. From this point we come up with a list of stories or Use Cases that represent all that is in scope for the project. So reviewing this document is a good way to flush out missed stories, and prevents missed scope before an estimate is done. The pitfall here is missed functionality and inaccurate estimates, which leads to missing the project target dates.
Explore the domain with the business contacts and subject matter experts to gain knowledge and gather ideas for your context diagram. Create a context diagram to map out your domain organizing users to their goals or actions they will perform. The more involvement by the subject matter experts and stakeholders for the project the smoother the project will go. Holding working sessions and a review sessions with stakeholders and subject matter experts is imperative to get this task done. One pitfall that arises often is no commitment to complete the definition from your business folks who are too busy to be on your project. Not always, but on some of my projects the companies either didn’t prioritize the project high enough to allow these business resources to have the time needed to create these important documents or the subject matter expert was overbooked and had no time to meet. Commitment from company resources is necessary so the project is not delayed and the content of the diagram and stories is accurate. Other resources are over looked during this important discovery period like including the QA staff during discussions so they have a reference when writing test cases. It is also good to have the tech lead available so they can answer questions about design possibilities and helping with setting a priority for stories.
Hold an agile project kickoff meeting for sharing the rules of engagement and agile process that will be used. This documents the behaviors and gives all team members an idea of what to expect while being a member of the project. If you miss this step the team will not know what behaviors they should follow and the project will miss out on standards for operation. Development standards are mentioned here as well as rolls are assigned for the project. This is a nice way to create basic order in your project around the flexibility of the scope and features in the project.
Get involvement from your subject matter experts or product managers. As much as you can get business and subject matter experts (SMEs) to participate, encourage them meet and help describe all the user’s goals and actions. As a Business Analyst, ask as many questions as possible to get your SMEs to think of alternatives and missing content. The more a business person talks through functionality the clearer the picture gets in their head. Business Analysts should get signoff on stories for content and completeness. No signoff review or further discussions may mean missing components or features. Often having more team members available during these reviews will spar important discussions. After this update the project charter with necessary scope changes.
Create a list of the stories or use cases that are needed to start the process of estimation and then prioritization of the list (the stakeholders will need to decide the precision and content of the schedule and estimate for the project). It is good to give a measure of the accuracy of the estimate. Then a priority can be assigned for the stories based on business priority and technical cost. Some churning may happen around the estimation based on the feature complexity and scope desired. One pitfall that comes up with the estimate is the level of details known in the early story list isn’t enough to give a good estimate. In some cases additional meetings maybe needed to clarify the scope and functionality so that a better estimate can be given. Often a sign of minimal involvement from the SMEs.
Before the first release and iterations are planned we need to come to a decision about the resources needed, project costs, and schedule. The three are related and must balance as planning is done. One pitfall here is that a schedule has been predetermined without regard to staffing and cost. The size of the project can sometimes limit the number of developers able to work on the same project at the same time and therefore the earliest point when the project can be completed. One other pitfall is not giving the business analyst time to detail out the stories for the first iteration(s). For determining the schedule, extra time after the developers are done coding for QA to test out the last components developed. QA needs a few weeks to finish the remaining testing and retesting. If the estimate and schedule looks good to the project sponsor, then the team may start to plan out the first iteration and release for the project.
Plan for project setup and planning. Planning the first iteration and release for the project is important to get your project off to a good start. During the first part of the project an iteration 0 is needed to get an environment setup and to do a proof of concept for the technology to give the stakeholders confidence in the project. Pitfalls for planning out the first iterations and releases are not having the resources available to plan out the amount of work that can be done in the time period for the iteration. A good method of planning during the first iterations is to layout subsequent iterations to get an idea of how many total iterations will be needed and to make future planning easier for business to look and decide story or use case priority. This planning is NOT set in stone and is only a tool for the BA or PM to use to plan out what stories to detail out next.

Hopefully this has given you an idea of how you can get your agile project started and everyone moving in the same direction. Agile projects may have different work products and process from start to completion and an agile project may be different at different companies. I try to keep a consistent process for starting out a project and share my vision with the team so they know what to expect and how to participate. These steps have given me success on the projects I have done and may help you to get your project off to a good start.

Subversion and Hudson From “Scratch”

jwarren — Mon, 18 Jan 2010 14:22:38 +0000

Here we’ll try to get a Hudson Continuous Integration server building from a monitored Subversion directory by adding these softwares and their Apache HTTP and Tomcat hosts.

Assumptions

Much of this discussion assumes you’re working with Linux (particularly assumed is that you’re running Ubuntu or other Debian-based distributions), but with a little care it can be used for Windows, Mac, Solaris, or other Linux distributions. The particular differences will most likely be around the installation paths and methods used to add prerequisites (e.g., what you do instead of “apt-get” or using Synaptic).

Additionally, it is assumed that the system on which this is being attempted has at least the basic software building tools. An installation of Ubuntu (the test system used to document this) comes with at least the basic C compiler, headers, and libraries necessary to do this. The exceptions encountered are noted, and should be enough to indicate where trouble might be.

Apache Serving Subversion

For the moment we’re assuming that an Apache web server is not installed, or that it’s acceptable to replace what is installed with one that has the required modules to make Subversion happy.

Get the Apache HTTP (web) server source right from the Apache HTTPD site. I recommend getting the latest production-ready source, which as of this writing is 2.2.14. Download the UNIX source, expand it (anywhere, really; It’ll be safe to deleting it later), with the likes of “tar -xzf httpd-2.2.14.tar.gz” and it’ll by default create a folder with the version name, in this case, httpd-2.2.14 in the current directory.

Change to that expanded directory and run “./configure –help” to get a list of the things that can be added. Use this list to double-check the names of the plug-ins and such in the following sections, adjusting where they’ve made changes. These changes are necessary for Subversion only, so if other things are needed for your installation, feel free to add them as well.

Ensure the option for mod_dav is enabled by adding “–enable-dav” (or whatever the –help exposed) to the ./configure command, such as “./configure –enable-dav” as this is one of the requirements for Subversion to work through the Apache server.

Another requirement is mod_ssl, which is likewise enabled by adding “–enable-ssl” to the ./configure command. This requires that an SSL package (and its headers) be installed. A good one is the OpenSSL. Using the likes of Ubuntu, just add the OpenSSL and libopenssl-dev packages using the Synaptic Package manager before trying to make. Installing OpenSSL from scratch is a little more involved than this document has room for, but it’s the basic “./configure && make && sudo make install” deal, if you do it from source.

By default, Apache will install to /usr/local. A last handy option to know (and look for in the –help output) is to change the output directory, using the “–prefix=(dir)” option, as in “–prefix=/opt/apache” or “–prefix=~/apache”. One good reason to change the install path is to avoid conflicting a pre-existing installation. Another is if you don’t have (or want to use) super-user rights on the system. A final good one is to have several Apache versions at the same time, perhaps for different development needs.

When all of the desired options are decided, run the configure command with the options, which will at a minimum be “./configure –enable-dav –enable-ssl” to create the Makefile. Run “make” in that directory to be sure everything builds. Running make as your user instead of the super user will create all of the files with your permissions, making clean-up and re-running easier.

Run “sudo make install” (or skip the sudo if you’ve chosen a folder to which your user has permission) to copy the files into place.

Go to the /usr/local/apache2/bin folder (or the bin directory inside the one in which you installed Apache), verify installation by running “./apachectl -l” and make sure mod_dav.c, mod_ssl.c, and http_core.c are in the list. While we didn’t specify HTTP as an option, it is on by default; it’s assumed you want HTTP in the HTTPD server, so it does that for you.

Adding Subversion

Subversion will require the Apache be configured with the minimum discussed above. It will also require a XML parsing library, such as expat or libxml is installed and available. You can get the libxml from source at xmlsoft.org, or if you prefer using a package manager, find and add the “expat” and “libexpat-dev” (for example by running sudo apt-get install expat libexpat-dev from a shell prompt) packages before configuring Subversion. Again, the building of these from scratch is outside of our scope today.

Download both Subversion Source and Subversion Dependencies from subversion.tigris.org, currently version 1.6.6, and expand them both into the same folder (which happens by default if you un-tar them in the same folder), and go to the directory (e.g. subversion-1.6.6).

Checking in that expanded folder, double-check the INSTALL file (with the likes of “more INSTALL”) and find the ./configure command suggestion which would look something like “./configure –with-apr=/usr/local/apache2 –with-apr-util=/usr/local/apache2″ so we can be sure we have the right APR parameter names. We’ll be using the path of the Apache server, configured above.

Also, you can run “./configure –help” to see other options that might be helpful, again such as “–prefix=dir” (for the same reasons as above) to put the subversion in a different directory than the default, which is /usr/local.

Run the ./configure command, changing the apr paths to match the Apache installation path from above (if you changed it with the –prefix= option). Watch for errors and peek at the INSTALL folder for tips. If the Apache is configured as described and an XML library is found, there should be no errors.

Run “make” to build the files, and “sudo make install” to add the Subversion programs and libraries to the system, and the modules to Apache. Again, if not installing to a location requiring root permissions, the sudo is not required.

Quickie Repository

Quickly make a Subversion repository so that Apache has something to show us for our efforts. Decide where the repository should go, and run the command “svnadmin create /var/svn/testrepository” using the correct path and permissions. Note if you chose to install Subversion in a different directory than the default, you may need to specify the full path to the executable, such as ~/subversion/bin/svnadmin or add the bin folder to your path. You also may have to create the path and correct the permissions (e.g., “mkdir /var/svn && chmod 777 /var/svn”) before executing the command. The svnadmin program will silently succeed, so you can check for the existence of the repository by double-checking “ls /var/svn/testrepository” to be sure it exists with all kinds of subversion goodness in it.

The permissions need to be set to allow access for the user under which your Apache runs. This user will need full access. A few different ways to do this include changing the owner or group on the folder, or as the example above does, simply allow everyone in there. Securing the Subversion through Apache is a whole other topic to be handled a whole other day.

Joining Subversion and Apache

A slight tweak to the Apache configuration needs to be done so that Apache knows to serve the subversion repository. Edit (as super-user if necessary) the Apache conf/httpd.conf file and add the following block, correcting the path to match the one above.


    DAV svn
    SVNParentPath /var/svn


    Order allow,deny
    Allow from all

Using the SVNParentPath will allow this single configuration to host all of the repositories created in the /var/svn folder. Whenever a repository is created in that same folder, it will automatically appear in the Apache server without further configuration, nor will a restart be required. If you use the SVNParentPath, however, note that you can’t visit /svn (or whatever you named it)

One quick note: if you’re not going to be starting Apache as root (e.g., on a system you don’t have sudo access), as long as you’re in the httpd.conf file, change the port from the default 80 to something bigger than 1024, and remember to use that port whenever accessing this server. And if you’re going that far, you can avoid the “can’t find the server name” warning by finding and setting the ServerName in the configuration file.

Run “/usr/local/apache2/bin/apachectl -t” (again, correct for the path used to install above) to check the syntax of the updated Apache configuration see if there are any errors. (An error I encountered meant I had to edit /usr/local/apache2/bin/envvars and add /usr/local/lib in the LD_LIBRARY_PATH because it was giving me an error “httpd: Syntax error on line 53 of /usr/local/apache2/conf/httpd.conf: Cannot load /usr/local/apache2/modules/mod_dav_svn.so into server: libsvn_repos-1.so.0: cannot open shared object file: No such file or directory.”)

If all is good, run “sudo /usr/local/apache2/bin/apachectl start” to fire up Apache. Fire up a web browser and visit http://localhost/ to get a brief message that should tell you that it worked. Again, skip the sudo if you don’t need it, and add the port if you changed it, of course.

Using a web browser to visit http://localhost/svn/testrepository (using the Location noted in the Apache config, and the name of the repository from the svnadmin command) should show your repository (finally!). Initially it might just indicate the revision number as zero, and show no files; but that’s right as we didn’t add anything to the repository yet.

Installing Tomcat

Tomcat requires that a Java runtime be installed and available to the system. This is almost assumed these days, but if in doubt, visit java.sun.com and get a JRE or JDK and install it. Note that previously Tomcat required a JDK be installed, but now it uses its own compiler for JSPs, and so a JDK is not required. However, since we’re using this as a tutorial for preparing for use with Hudson, get the JDK to be sure that whatever Hudson tries to build will work.

Visit the Apache Tomcat website at tomcat.apache.org and download the latest version. Expand the downloaded file (untar or unzip as appropriate for what you downloaded). Done. There’s nothing that needs to be done to install Tomcat.

Depending on where you want Tomcat to live, you may need to do the expanding and other bits as the super-user. Common places to put the Tomcat distribution are /usr/local (requires super-user) and /opt (usually doesn’t require super-user). There’s no reason this couldn’t work from your user’s home directory, either.

Additional configuration is probably a good idea (such as changing/adding the manager name and password in the tomcat-users.xml, changing the path in the server.xml, etc), but nothing more is required to make the rest of this work.

A quick peek at the conf/server.xml file and we can see that by default the WAR files can be added to the webapps folder and that they will automatically be deployed.

Start the server now (or after copying the hudson.war file in the next section) by simply executing the bin/startup.sh script. Visiting http://localhost:8080/ should bring up a confirmation page that Tomcat is running.

Finally, Hudson

Get Hudson from hudson-ci.org and download the latest and greatest hudson.war. Put the war file under Tomcat webapp directory. Simply copy the file. If Tomcat is running, after a moment Tomcat will see it and expand and deploy it.

Start the Tomcat server (if not already running) and go to http://localhost:8080/hudson/ and the Hudson interface (or “loading, please wait” message) will let you know it’s been installed.

Hudson Job

The final piece of the puzzle is to put a project into Subversion for Hudson to find and act on. Let’s first create a simple “hello world” project, with an Ant build script, commit it to Subversion, and make a job for Hudson to run.

A few assumptions will be stated here (to avoid all of the potential asides, like this one). First is that you’ve installed things to the default or as mentioned above. That is, the HTTPD is running on port 80 and Tomcat on 8080, which are their defaults. Also that you’ve chosen to use the “parent” view for subversion as noted above, so we can just create repositories on a whim. We’re also assuming everything is running on one machine, so the URLs will be localhost, which you will have to change if you’re doing this on separate machines.

Make a folder for the project (e.g., mkdir hudsonProject). This will become the root of our project.

In that folder, create a source folder (src) and in there create a trivial class to print “hello world” or some other such simple stuff. I’ll leave out the Javaness (e.g., “name the file HelloWorld.java” and so on, as I’m assuming you’re that far along.

class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, world.");
    }
}

Also create a simple Ant build script (or Maven if you prefer, but I’m documenting Ant…). Put this in the root of our project, and name it, of course, build.xml.

  
    
    
  

This is about as simple of a build.xml as one can have; it will only compile our Java source files found in the src folder into the build folder. In the end we should have the .java file we created, and the .class file the Ant script creates. We haven’t installed Ant yet, so unless you’ve taken that initiative, you can’t test the script. That’ll happen in a bit and we’ll use Hudson to do our debugging for that…gives us something to play with.

Now we need to introduce this project to Subversion. The following command will prepare everything for our initial project creation. We could have created the project empty, without any files at all, which makes a fine “revison 0″ but I’ll leave discussions of repository management theory for another post.

svnadmin create /var/svn/helloWorld
svn import . http://localhost/svn/helloWorld -m "Initial import"
svn list http://localhost/svn/helloWorld

The first line will create the repository in our parent folder (unless you want to use the one created before…then just change the name as necessary). The second line will copy the contents of our current directory (the dot) to the repository through our web server. The final line will show us what’s in the repository, which should mimic our trivial src/HelloWorld.java and build.xml structure. We can verify the repository exists and has our files by visiting http://localhost/svn/helloWorld with the web browser and the same stuff should be there.

Now we’ve got our project in Subversion; let’s create a Hudson job to work on it. Fire up a web browser and visit http://localhost:8080/hudson again. By now it should be running and greet you with a “create jobs” message. We’ve got a little housekeeping to do to make it work with our install.

On the right side find and click the “manage Hudson” link. At the top of the list in the middle, click the “configure system” link. Near the middle, find and click the “add jdk” button. De-select the “install automatically” if it’s selected, and put the path to the JDK installed above (e.g. /home/youruser/jdk1.6.0_17 if you unpacked it in your home folder), and give it a name.

Since we’re doing an Ant project, we need Ant “installed” and configured. The latest versions of Hudson look like they’ll try to install it for us, but I can’t get it to work, so make a quick trip to ant.apache.org and download the latest (currently 1.7.1). Simply extract that file somewhere. Back in our Hudson configuration, click the “add Ant” button, de-select the “install automatically” box, if it’s selected, and put the path to the Ant as unpacked (e.g., /home/youruser/apache-ant-1.7.1) and give it a name.

Hit the save button at the bottom of the form.

Bonk the “new job” link and start the process. Give the job a name (something like “helloWorld” — it isn’t really related to the poject) and select “build freestyle” to use the Ant job, and bonk the “ok” button. Much of the busy next form is optional, and we’ll leave Hudson management to another article, focusing now on the required fields to make this work.

Find and select Subversion. This will spring open the configuration bits necessary for our job. Put in the URL to the repository (in our example, http://localhost/svn/helloWorld).

Find the “add build step” drop down and select Ant from the list, then select the Ant installation by the name used when we added the configuration above (the “default” never seems to work, even if only one Ant is installed). By default it will use the build.xml in the root of the project, and the project’s default target. For ours, recall, the simple script has only one target, and it is the default. Nothing more is required.

Bonk the save button at the bottom of the form. The job will be saved. The Hudson view will now show the current state of the helloWorld job, which is pretty blank. On the left side is a “build now” link. Bonk that to trigger Hudson to do an SVN check-out and run the Ant script.

A few moments will pass and some activity will occur in the “build history” area on the left side. If we’ve done all of our configuration correctly, and there are no errors in our Java or Ant files, we should be rewarded with a successful build. Click on the build version (by date and time) from the “build history” area and we can see what happened to the build. For the most information, given our simple job configuration, check the console output (on the left) to see what Hudson did.

Had our source and build script had more, like unit tests and build a WAR or JAR, or if there are errors in the source or build script, Hudson would show us all that went right or wrong.

Remaining Work

Subversion and Hudson together with their appropriate Apache tools is a powerful and easy-to-use suite of version control and continuous integration. Brushing over as we did to get a skeleton working, the projects are exposed to the world. Both of the Apache servers, Subversion, and Hudson all have security to help control and restrict access to the servers, repositories, and jobs. Hudson has plug-ins for monitoring unit tests and code coverage and can put notices on Twitter and events to Google Calendar…

Much more than there’s time for here…but, heck, this is already five pages long…

Nine Simple Tips for Greener Consulting

bkarels — Thu, 14 Jan 2010 13:45:45 +0000

As consultants we often find ourselves in a new office several times a year and it is up to us to carry our green habits with us as we go. Programs offered at offices are most often by, for, and about employees leaving consultants and contractors out of the loop on available facilities and programs. With this just being the way of things, here are nine simple, portable things we can do to perpetuate a green lifestyle wherever we find ourselves working on a given day.

1. Kill the lights
While many offices have installed savvy motion sensing light systems for conference rooms and other common area – most have not and rely on the common wall switch. For as simple as it seems to turn off the lights in unused area you will often be confronted with strange looks as you reach out and hit the switch. Many disregard energy use at the office as they do not directly foot the bill. However, businesses represent a significant portion of the demand in the supply and demand of the energy sector. As we continue to reduce demand we all pay and pollute less.

2. Enhance Computer Efficiency
Some estimates place energy waste from idle computers in the business sector at over $1 billion dollars a year. Turn off your computer and the power strip it is connected to whenever possible. You will want to check with the IT department to ensure that system updates and/or backup are not done overnight. It is also a good idea to adjust your power settings to send your computer into a suspended state after a brief period of being idle – and do not forget to disable the screen saver. Remember screen savers do not save energy. If you cannot turn your computer off overnight, at least power off the monitor – many illuminate small LEDs to let you know that your monitor is off and that consumes power. Odds are a blank screen is indication enough that your monitor is off.

3. Do not print, print less, print smart
The Sierra Club estimates that the average American office worker goes through about 10,000 sheets of paper annually at the office – that’s about 1.5 pounds a day. Whenever possible, do not print. Instead, bring your laptop to the meeting, transfer files to a smart phone or other portable device, or make nonsensitive material available through a web based interface for access anywhere. If you must print, print smart. Print multiple pages per sheet and use double sided printing where possible. This alone can cut your paper use by 50-90%.

4. Watching what and how you eat
Make a point to bring your own reusable mug and other dish ware to the office. When you need caffeine (and most of us do), opt for coffee or tea instead of soda. Not only will you cut out a lot of unnecessary sugar, you will avoid consuming liquid that has been shipped cross country to get to that vending machine.

When running out to grab lunch, avoid establishments that use Styrofoam to go containers. Instead choose to eat at the restaurant or perhaps grab a sandwich that can be wrapped in paper. Additionally, consider a vegetarian meal from time to time to help reduce your carbon footprint.

5. Traveling while traveling
When traveling for work, consider how you will travel during your stay. When attending a conference, try to stay at a hotel within walking distance of the event. For other work, commit to walking, busing, taking the subway, or other transit while on the road. When renting a car is necessary, seek out and rent a fuel efficient vehicle – most rental agencies now have hybrids available upon request. Of course using technology to make travel unnecessary is the ultimate solution – get creative and see what is possible.

6. The Commute
As always, carpooling, biking, or using transit are great ways to use fewer resources and reduce congestion. However, as technology workers nothing beats telecommuting – commit to working remotely wherever and whenever possible. If driving cannot be avoided, perhaps consider a car sharing service like Zipcar or similar.

7. Write this down…
…but do so in pencil. A good mechanical pencil can last for years if taken care of whereas the average ball point pen will find its way to the trash in a couple of weeks. The pencil also has the advantage of not becoming trash if you leave the cap off.

8. Composting
You have brought your lunch, your snacks, and other foods in your reusable containers – now take it a step further. Put your apple cores, banana peels, orange rinds, that last bit of salad, etc. into your empty lunch container and bring it home to be properly composted. While most offices have every form of recycling bin available, almost none have containers for compostable material. Bring it home and make your good garden great! You do have a garden right?

9. Bring a Plant to Work
For longer term engagements, bring a plant with you. Plants brighten up the atmosphere, eat indoor pollutants, consume CO₂, and produce oxygen. Stay away from flowers and other plants that produce allergens; and what ever it is, do not forget to water it – a dead plant offers little benefit in an office setting.

So, whether you can do one, some, or all of these things; remember that every bit helps and no good deed is too small. Do what you do and encourage others to do the same. Lead by example and show how little it takes to make a difference. A positive person doing good things for the benefit of everyone is hard to ignore and even harder to disappoint.