My New Favorite Toys

OK, I’m going to admit it.  I tend to get really excited about Configuration Management (CM) type work and I have no idea why.  This is one of those.

Machine Images/Virtual Machines

Ok, these have been around.  If you don’t know what a machine image is, then stop and go download Oracle’s VirtualBox, because it is something that everyone should know.  These were revolutionary when they first were introduced.   So much so that most Operating Systems support some form of images (disk or machine) right out of the box.  Especially because a number of companies, including VMWare and Parallels, found out that they could manage your hardware better than the Operating System (which was originally supposed to do just that, but has really turned into something much more complex and more like a user interface). So, it turned out that if you used machine images, you could have a snapshot to a point in time. Sure, it still required some configuration after getting it going, but after that, it was cake. IT departments (and Dell/Norton, with their Ghost Image partition) were loving it. It really cut the mind-numbing monotony of setting up new boxes.

Puppet and Chef

Essentially, Puppet and Chef do the same thing.  Both start on the assumption that you have a base image installed on your box. Both represent a language that gives you the ability to install the software, users, and whatever else is necessary on a box to have it ready for tasking.  Both have claimed to support all major OS options.  This way, you could install the same users on the Windows boxes and Unix boxes in one script!  Of course the multiplatform argument is crap, but it still is a great option to finish what Machine Images started.

Take passwords for example.  Each OS handles this so completely differently that you would have to create a different module for the password for each OS.  Too much hassle.  Most companies are going to pick an OS and stick with it.  Heck, most are going to pick the same hardware too. This is the suggested use for almost all companies anyways.  Google, Yahoo, Amazon, etc. all use proprietary hardware.  But it’s THE SAME proprietary hardware.  Every machine has the same specifications.  (If you want, check out the Google server machines or the data center).  And I’m guessing most of them aren’t Windows boxes.  Just sayin’.

So, you create a new machine configuration for each type of machine.  One for your Web servers, one for the database servers, etc. Makes scripting all boxes that you might want to make really easy. Plus, in a cloud environment, like Amazon EC2 or privately held HDFS clusters, this makes setting up servers a breeze.

The difference between Puppet and Chef is their target audience.  Although they are technically competing products, both companies have pretty much stated that their real competition is the rats net of batch/shell scripts that you’ve probably written, or even better yet the documented instructions like the following.  We’ve all written these before.  And they even sometimes work.  Sometimes.

Vagrant

Another fun toy. So now, assuming you start using the Puppet/Chef to start requisitioning your servers, everything is great. Every server that gets started is always the same. Starts from the same image. Now you know that every server will run the same. Huzzah! But as every web developer knows, we don’t test enough for IE. Why? Because as a developer, I f’ing hate it. Because I don’t use it, I probably miss bugs.  Same applies to servers.  I’m missing bugs because I work in Windows or OS X, but deploy to a totally different environment.

Vagrant is a fun new tool that takes that really nice puppet script and image that you’d made, load it into a virtual machine, and run it in the background.  That way, you have the exact same environment that the server has, but on your local machine.  So, if you need to, you can ssh into it.  You can update your puppet scripts and push changes to it.  It also does port forwarding from your machine to your virtual machine, so that any exposed REST service or port is fully accessible via a web browser.  (If you want to play with it, the best install guide I found is here).

Now, you can avoid the annoying statement from your coworkers that is “Well… it works on my machine.”  Because I don’t care if it works on your machine.  Now, I only care that it works on the production environment.  You have it locally, so make sure to test it there.

Boxen

My new toys just got better.  Your IT department can now set up your computer for your project within minutes by using a new product called Boxen.  This is also based on Puppet and brought to you by the GitHub folks (who also brought you Hubot).  This software was originally designed to install project specific files and programs on OS X, but since Puppet is very much a OS independent tool, I’m guessing it will work on Linux (Good luck Windows guys.  Although Puppet might work, I doubt the tool will, but only time will tell.)

Overview

So, the tools are changing to help us stand up and deploy 1 to 1,000 servers with ease. Developers can run the exact same environment is working there. Now, we should wait and hope for Internet Explorer to give up on their web engine, much like Opera did (some say for performance reasons) and we might eventually get to a uniform presentation platform as well.  Wouldn’t that be nice?

Groovy Scripting or Gradle – check if binding variable (or Gradle property) is defined

In scripting, its helpful to create global variables.  It’s definitely not a good idea for any form of scalability, obviously, but Groovy has done a good job in creating this slight separation of function.

1
2
3

myBindingVar = 'checkme' //binding occurs, so that it is available everywhere
def myLocalVar =  'not useful' //this is a local variable that is only available to the local scope
String myLocalString =  'also not useful' //this is also a local variable that is only available to the local scope

The difference is that the following won’t work:

1
2
3
4
5

def myLocalVar =  'not useful' //this is a local variable that is only available to the local scope
def myFunc() {
    println myLocalVar //this will throw an error, because it is out of scope
}
myFunc()

but the following will work:

1
2
3
4
5

myBindingVar = 'checkme' //binding occurs, so that it is available everywhere
def myFunc() {
    println myBindingVar
}
myFunc()

This can cause some problems in scripting if you need to either provide default values or if you need it to provide a better exception if the variable isn’t found.

An easy solution to this is the following:

1
2
3

if (binding.variables.containsKey("bindingVar")) {
    // do something
}

Or if you’d like to get a null value for an optional binding:

1
2
3
4

def optVar = binding.variables.get("bindingVar")
if (optVar) {
    // do something
}

Similarly, when using Gradle, which is based off of Groovy, you can define one or more gradle.properties files. An example gradle properties would be as simple as:

1

myProp=checkMe

use the following to determine if a property is defined:

1
2
3

if (gradleProject.properties.containsKey("myProp")) {
    // do something
}

Or if you’d like to get a null value for an optional property:

1
2
3

if (gradleProject.properties['myProp']) {
    // do something
}

Making the Maven Version and Subversion Version Match

Let me start this out – this is strongly discouraged. Not only by me, but by Maven and any standards that you will ever come across. Your release version should almost never be your SCM version. Some tools allow you to pull in the SCM version as a variable and use it as a maven variable. This is great, because you can inject it into places to tie your release version to your project. Note the difference. I even wrote a great article about injecting variables into WAR files here and you can use the buildnumber plugin to inject your build numbers!

But, some groups (we’ll just nickname them “Oogle” for now) are being annoying and don’t want to do full blown releases, as that might mean that they have to support old releases. I totally understand that principle, so I’m not going to knock it. But, if you don’t have a release, how does one differentiate between versions?

Well, at this mythical company Oogle, they have stated you should use their subversion revision as the way to refer to their releases. Well, we can do that, but God help us if you switch to Git or some other form of DVCS. One thing about release numbers is that they should be sequential, so that you know which version precedes another, and human readable. In many of the new DVCS flavors, they use the SHA-1 hash, which is some long string of numbers and letters that is not sequential, nor human readable. But, for now, we can work with Subversion.

So, now we have a predicament. We need to do something that Maven strongly discourages (and can’t really work with by itself – it needs to know the version, so no variables are allowed in the project version tag). Thinking about this, we could run two maven pom files. One to set the version in the second pom file, and the other does the release. Problem is when you are working with Continuous Integration. Many of the CI servers (Hudson, Jenkins, etc.) expect one pom file to be executed for a release. Hudson/Jenkins can execute an ant command followed by a maven command though… It’s getting sticky.

Rather than you following me down the rabbit hole, let me just show you the solution that I’ve come up with.

First, create a template pom file. This will overwrite your pom.xml every time the build.xml is run.

1
2
3
4
5
6
7
8
9

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.github.jlgrock</groupId>
	<artifactId>myId</artifactId>
	<version>@SVN_VERSION@</version>
	<packaging>pom</packaging>
…
</project>

Next, create an Ant Script as build.xml at the root level of your maven project.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<project name="MyProject" default="createPomFromTemplate" basedir=".">
 
	<description>
    	This will replace the version number in the maven file with the current revision 
		within Subversion.  This requires a machine with a c-shell and subversion installed.
    </description>
 
	<property name="url">http://whatever.my.com/server/address/is/</property>
 
	<target name="createPomFromTemplate" depends="svnInfo" description="Will copy the pom-template.xml to pom.xml, making the appropriate replacements.">
		<copy file="pom-template.xml" tofile="pom.xml" filtering="yes" overwrite="yes">
			<filterset>
				<filter token="SVN_VERSION" value="${build.version}-SNAPSHOT" />
			</filterset>
		</copy>
	</target>
 
	<target name="svnInfo" description="Open a c-shell and execute a subversion command to get the current version">
		<exec executable="sh" outputproperty="build.version">
			<arg value="-c" />
			<arg value="svn --non-interactive info ${url} | grep Revision | awk '{print $2}'" />
		</exec>
	</target>
 
</project>

OK, what does this do? The svnInfo goal (btw – I think this only works on linux-style machines) will execute, via c-shell, a command to query the server for svn info. This will return a large block of text, of which, we only need a small amount of text. So, we pass this through to the grep command, followed by the awk command. The result of the grep command will be something like “Revision: 4233” and the result of the awk command should be something like “4233”. You can test this locally if you want to see each of those in action.

Next, it takes that value and shoves it into a variable (${build.version}). Then, it does a copy of the template file, searching through the template file, where I have no version but have put @SVN_VERSION@ as a placeholder. It replaces the string “@SVN_VERSION@” with “4233-SNAPSHOT”.

OK, next we have to create the pom-template that it can work on.

Now the Continuous Integration environment can execute the release plugin on the Maven pom and the release version will match the subversion version. You can test that by running ant. It will create a pom.xml for you.

Next, I do a full blown maven release with the following:

1

mvn clean release:prepare release:perform

This will release with the subversion number to the Nexus or https://github.com/jlgrock/ClosureLibrary

Is it pretty? No. Does it work and is it maintainable? Very much so.

Using Maven to Package JavaScript Library

Hopefully, the article about using JavaScript libraries should have swayed you should use Maven or some other repository to disseminate your libraries. But now you’ve hit a point that you have found a library that doesn’t exists in any Maven Repository (at least not ones that you have access to). What to do now? No worries. Publishing an artifact is easy.

First, assuming your code is in the location ${basedir}/src/main/javascript, create a file in ${basedir}/src/main/resources called “assembly.xml”. This file should contain the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
  <id>bin</id>
  <formats>
    <format>tar.gz</format>
    <format>tar.bz2</format>
    <format>zip</format>
  </formats>
  <fileSets>
    <fileSet>
      <directory>${basedir}/src/main/javascript</directory>
      <outputDirectory>/</outputDirectory>
      <includes>
        <include>*</include>
      </includes>
    </fileSet>
  </fileSets>
</assembly>

This file will tell the assembly plugin how to package (into a zip, gzip-tarball, and bzip2-tarball) and what to include (“*”, or everything – you can include only specific things or exclude specific things. Please see the Assembly Plugin for more information about configuring includes/excludes).

Next, you need to tell maven to execute the Assembly plugin. Do this by adding the following to your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-assembly-plugin</artifactId>
	<version>${maven.assembly.version}</version>
	<executions>
		<execution>
			<id>package-closure</id>
			<goals>
				<goal>single</goal>
			</goals>
			<phase>package</phase>
			<configuration>
				<descriptors>
					<descriptor>src/main/resources/assembly.xml</descriptor>
				</descriptors>
				<attach>true</attach>
				<appendAssemblyId>false</appendAssemblyId>
			</configuration>
		</execution>
	</executions>
</plugin>

What if the code isn’t yours, though? OK, now we are in a strange place. First, make sure that the licensing allows you to republish this if you are publishing this somewhere public. If it is Apache 2 or Gnu, you won’t have an issue republishing it unmodified. But check to be sure.

First, you need to download the code from the external repository. The following is code that will download Google’s Closure JavaScript library. This uses the SCM plugin. This is the same plugin that acts on your code in your SCM section, allowing you to do mvn scm commands. Why don’t we put it in the SCM section? Well, cause now we know that your code will live in a different place than the SCM plugin. So you are going to need to fill out the SCM section with your SCM details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-scm-plugin</artifactId>
	<version>1.7</version>
	<executions>
		<execution>
			<id>checkout</id>
			<goals>
				<goal>checkout</goal>
			</goals>
			<phase>generate-resources</phase>
		</execution>
	</executions>
	<configuration>
		<useExport>true</useExport>
		<checkoutDirectory>${project.build.directory}/checkout</checkoutDirectory>
		<connectionType>connection</connectionType>
		<connectionUrl>scm:svn:http://closure-library.googlecode.com/svn/trunk/</connectionUrl>
		<developerConnectionUrl>scm:svn:http://closure-library.googlecode.com/svn/trunk/</developerConnectionUrl>
	</configuration>
</plugin>

Ok, that will have downloaded this into your target/checkout folder. Now, jut make a quick tweak to what we’ve already done. Change the directory that the assembly.xml is pointing at to:

1

<directory>${project.build.directory}/checkout</directory>

And Voila! Now, if you just use the Maven release plugin or follow standard deployment procedures, this will be released for everyone in your company (or in the world, depending on where you are publishing to)!

How to use a Maven JavaScript Dependency

Using a Maven Repository to handle your dependencies is just a good idea. This is what it does best. Due to the central repository idea, many other build tools have copied it. Arguably, some would say they have improved upon it as well, but the idea is the same. Ivy (for Ant) , Buildr, and Gradle, for example, use the same artifacts that Maven does. Node Package Modules(npm) is another form for direct JavaScript inclusion, though it requires that you are tied into Node.js. It’s probably a good idea to publish to npm as well for a JavaScript dependency, but I’ll have to post another article on that later.

So if you take away anything, it is that setting up a standard for your dependency is a good idea. It allows people to easily just add a couple of lines and they are using your library without a problem. The reasons are these:

1. You are able to guarantee that no one has gone in and fiddled with the library that you’ve downloaded (yes – it has happened to me in production, which causes a licensing nightmare for some products, especially if the Junior developer wasn’t smart enough to know that you can’t just change someone else’s files without adjusting the licenses.)
2. Design by extension is the correct way to develop anyways. If you want to change their code, take a class that they are using and wrap it to do what you want. That way, when the version changes and you upgrade, you still have the changes. Looking at #1 – if one developer has made this change and then the team is required to upgrade the library, things will start breaking and you won’t know why.

So most people will be with me at this point, but if you ask them how to do it, they are now going to stop and say “Wait… isn’t Maven for Java?” My response is a simple “No.” It was built originally with Java in mind, and has heavy adoptance in the Java world and the plugins are written in Java, but Maven itself is actually language agnostic. For example, if you want to use Maven for C/C++, I would suggest using the Nar Plugin or if you want to use like to use Google’s Closure Tools, I suggest using the Closure JavaScriptFramework or if you want to compile with Dalvik down to and Android app, you can use the Maven Android plugin. You get the general idea. It can handle quite a bit, once you know what you are doing.

So, in this example, I’m going to use Dojo in my development (since they have done a really good job of publishing to Maven Central, and it’s a great open source JavaScript Library – eat it Sencha, with your ridiculous Developer’s license). To include it as part of my build, just add the code below. Feel free to change any of the variables to be whatever you need (especially in the outputDirectory tag):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-dependency-plugin</artifactId>
	<executions>
		<execution>
			<id>unpack dojo</id>
			<phase>generate-sources</phase>
			<goals>
				<goal>unpack</goal>
			</goals>
			<configuration>
				<artifactItems>
					<artifactItem>
						<groupId>org.dojotoolkit</groupId>
						<artifactId>dojo</artifactId>
						<version>${dojo.version}</version>
						<type>zip</type>
					</artifactItem>
				</artifactItems>
				<outputDirectory>${project.build.directory}${my.outputDir}</outputDirectory>
			</configuration>
		</execution>
	</executions>
</plugin>

This will download the zip file (note the type tag with a value of “zip”) and unpack it using the maven dependency plugin to the location specified in the value of the output tag.
That’s it! Now don’t you feel silly for having not done this before?

Maven WAR Filtering

So it looks like you can adjust your war plugin to use filtering, as such:

       org.apache.maven.plugins
       maven-war-plugin
       ${maven.war.version}
       
             ${deploy.directory}
             true
       

This will automatically scan your deployment discriptors, including web.xml.

So we put the following in a web.xml:

<web-app>
       <context-param>
             <param-name>ApiVersion</param-name>
             <param-value>${project.version}</param-value>
             <description>My API version</description>
       </context-param>
</web-app>

At which point you can use this in your JSP:

<% String apiVersion = getServletContext().getInitParameter("ApiVersion"); %>

This will then resolve in any jsp page you have, and you can just use the “apiVersion” variable in any way you want.