Setting up AWS keys for Eclipse

One somewhat annoying thing about running JUnit tests in Eclipse is that they do not inherit your system's environment variables. There are good reasons for this, but we pass our AWS credentials to all of our applications via system variable, and it's a pain to add these to every single run configuration that needs them. This gets especially tedious when a significant number of your JUnit tests require AWS access.

As a workaround, you can add "Default VM Arguments" to the JVM you use to run your tests. Simply go to "Preferences->Java->Installed JREs" and edit your default JVM. Right under the JRE name is a space to add default VM arguments. I simply added "-DAWS_SECRET_ACCESS_KEY=foo -DAWS_ACCESS_KEY_ID=bar", and now I no longer need to manually edit individual run configurations.

This method seems a bit hacky to me, but until I can get a global run configuration, it definitely beats manually setting common environment variables for individual tests.

Dependency management for Scala scripts using Ivy

I'm quickly becoming a huge fan of Scala scripting. Because Scala is Java-compatible, we can easily use our existing Java code base in scripts. This is especially convenient as we're moving our reporting to Hive, which supports script-based Hadoop streaming for custom Mappers and Reducers.

The one very annoying thing about Scala scripting is managing dependencies. My initial method was to have my bash preamble manually download the required libraries to the current directory and insert them onto the Scala classpath. So, my scripts looked something like this:


#!/bin/sh

if [ ! -f commons-lang.jar ]; then
s3cmd get [s3-location]/commons-lang.jar commons-lang.jar
fi

if [ ! -f google-collect.jar ]; then
s3cmd get [s3-location]/google-collect.jar google-collect.jar
fi

if [ ! -f hadoop-core.jar ]; then
s3cmd get [s3-location]/hadoop-core.jar hadoop-core.jar
fi

exec /opt/local/bin/scala -classpath commons-lang.jar:google-collect.jar:hadoop-core.jar $0 $@

!#
(scala code here)


This method has some rather severe scaling problems as the complexity of the dependency graph increases. I was about to step into the endless cycle of testing my script, finding the missing or conflicting dependencies, and re-editing it to download and include the appropriate files.

Fortunately, there was an easy solution. We're already using Ivy to manage our dependencies in our compiled projects, and Ivy can be run in standalone mode outside of ant. The key option to use is the "-cachepath" command line option, which causes Ivy to write a classpath to the cached dependencies to a specified file. So, now the preamble of my scripts looks like this:


#!/bin/bash

tempfile=`mktemp /tmp/tfile.XXXXXXXXXX`

/usr/bin/java -jar /mnt/bizo/ivy-script/ivy.jar -settings /mnt/bizo/ivy-script/ivyconf.xml -cachepath ${tempfile} > /dev/null

classpath=`cat ${tempfile} | tr -d "\n\r"`

rm ${tempfile}

exec /opt/local/bin/scala -classpath ${classpath} $0 $@

!#
(scala code here)


Now all I need is a standard ivy.xml file living next to my script, and Ivy will automagically resolve all of my dependencies and insert them into the script's classpath for me.

Crisis averted. Life is once again filled with joy and happiness.

Pruning EBS Snapshots

We've been using Amazon's Elastic Block Storage (EBS) for some time now. In a nutshell, EBS is like a "hard drive for the AWS cloud". You simply create an EBS volume and then mount it on your EC2 instance. You then read/write to it as if it were local storage. For a good intro to EBS, check out this RightScale blog post.

The snapshots feature of EBS is especially handy as it allows you to easily backup the data on your EBS volume. AWS provides an API that allows you to request a snapshot. The API call will return immediately and then, in the background, the backup will occur and eventually be uploaded to S3.

While the snapshots feature is useful, one of the issues that you will likely run into is the snapshot limit. A standard AWS account allows you to have 500 EBS snapshots at any given time. After this limit has been reached, you will no longer be able to create new snapshots. So, you will need to have a strategy to 'prune' (remove) snapshots.

I wasn't able to find any scripts for pruning EBS snapshots on the web so I ended up writing a little Ruby script to accomplish the task.

You can get the script here. It requires the excellent right_aws ruby gem.

custom map scripts and hive

First, I have to say that after using Hive for the past couple of weeks and actually writing some real reporting tasks with it, it would be really hard to go back. If you are writing straight hadoop jobs for any kind of report, please give hive a shot. You'll thank me.

Sometimes, you need to perform data transformation in a more complex way than SQL will allow (even with custom UDFs). Specifically, if you want to return a different number of columns, or a different number of rows for a given input row, then you need to perform what hive calls a transform. This is basically a custom streaming map task.

The basics

1. You are not writing an org.apache.hadoop.mapred.Mapper class! This is just a simple script that reads rows from stdin (columns separated by \t) and should write rows to stdout (again, columns separated by \t). It's probably worth mentioning this again but you shouldn't be thinking Key Value, you need to think about columns.

2. You can write your script in any language you want, but it needs to be available on all machines in the cluster. Any easy way to do this is to take advantage of the hadoop distributed cache support, and just use add file /path/to/script within hive. The script will then be distributed and can be run as just ./script (assuming it is executable), or 'perl script.pl' if it's perl, etc.

An example

This is a simplified example, but recently I had a case where one of my columns contained a bunch of key/value pairs separated by commas:

k1=v1,k2=v2,k3=v3,...
k1=v1,k2=v2,k3=v3,...
k1=v1,k2=v2,k3=v3,...

I wanted to transform these records into a 2 column table of k/v:

k1	v1
k2	v2
k3	v3
k1	v1
k2	v2
...

I wrote a simple perl script to handle the map, created the 2 column output table, then ran the following:

-- add script to distributed cache
add file /tmp/split_kv.pl

-- run transform
insert overwrite table test_kv_split
select
  transform (d.kvs)
    using './split_kv.pl'
    as (k, v)
from
  (select all_kvs as kvs from kv_input) d
;

As you can see, you can specify both the input and output columns as part of your transform statement.

And... that's all there is to it. Next time... a reducer?

Load testing with Tsung

One of the big issues with building scalable software is making tests scale along with the application. A high performance web application should be tested under heavy loads, preferably to the breaking point. Of course, now you need a second application that can generate lots of traffic. You could use something simple like httperf; however, this doesn't work so well with complex systems, since you're only hitting one URL at a time.

Enter Tsung. Tsung is a load testing tool written in Erlang (everybody's favorite scalable language) that can not only generate large amounts of traffic, but it can parametrize requests based on data returned by your web application or with data pulled from external files. It also can generate very nice HTML reports using gnuplot.

Here's how we're running Tsung on Ubuntu in EC2:

1. Start a new instance. We're using an Ubuntu Hardy instance build by Alestic.


2. Download, configure, compile, and install Erlang.


3. Get and install the Tsung dependencies: gnuplot and perl5.
   
4. Download, configure, compile, and install Tsung.


5. Install your favorite web server. I prefer Apache HTTPD...others in this office perfer nginx. If you want to be really Erlang-y, install Yaws or Mochiweb.


6. Configure your ~/.tsung/tsung.xml configuration file for your test. The Tsung user manual has pretty good documentation about how to do this. Note that you do NOT want to use vm-transport for heavy loads, as this prevents Erlang from spawning additional virtual machines, which limits the number of requests you can use at a time. This does require you to set up passwordless ssh access to localhost.


7. Point your web server at "~/.tsung/log/". Each test you run will log the results in a subdirectory of this location.


8. Start your test with the "tsung start" command.


9. Set the report-generating script /usr/lib/tsung/bin/tsung_stats.pl to run in the appropriate log directory every 10 seconds. You can do this via crons or simply having a "watch" command running in the background.

Now, you can just browse over to your machine to view the latest test report. Tsung exposes all of the statistics you would expect (req/sec, throughput, latency, etc) both in numerical and graphical form. All of the graphs can be downloaded as high quality postscript graphs, too.

If you want to generate truly large amounts of traffic, Tsung supports distributed testing environments (as you might expect from an Erlang testing tool). Just make sure that you have passwordless SSH set up between your test machines and configure the client list in your tsung.xml file appropriately.

custom UDFs and hive

We just started playing around with Hive. Basically, it lets you write your hadoop map/reduce jobs using a SQL-like language. This is pretty powerful. Hive also seems to be pretty extendable -- custom data/serialization formats, custom functions, etc.

It turns out that writing your own UDF (user defined function) for use in hive is actually pretty simple.

All you need to do is extend UDF, and write one or more evaluate methods with a hadoop Writable return type. Here's an example of a complete implementation for a lower case function:

package com.bizo.hive.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public final class Lower extends UDF {
  public Text evaluate(final Text s) {
    if (s == null) { return null; }
    return new Text(s.toString().toLowerCase());
  }
}

(Note that there's already a built-in function for this, it's just an easy example).

As you've probably noticed from the import statements, you'll need to add buildtime dependencies for hadoop and hive_exec.

The next step is to add the jar with your UDF code to the hive claspath. The easiest way I've found to do this is to set HIVE_AUX_JARS_PATH to a directory containing any jars you need to add before starting hive. Alternatively you can edit $HIVE_HOME/conf/hive-site.xml with a hive.aux.jars.path property. Either way you need to do this before starting hive. It looks like there's a patch out there to dynamically add/remove jars to the classpath, so, hopefully this will be easier soon.

example:

# directory containing any additional jars you want in the classpath
export HIVE_AUX_JARS_PATH=/tmp/hive_aux

# start hive normally
/opt/hive/bin/hive

Once you have hive running, the last step is to register your function:

create temporary function my_lower as 'com.bizo.hive.udf.Lower';

Now, you can use it:

hive> select my_lower(title), sum(freq) from titles group by my_lower(title);

...

Ended Job = job_200906231019_0006
OK
cmo 13.0
vp 7.0

Although it's pretty simple, I didn't see this documented anywhere so I thought I would write it up. I also added it to the wiki.

Force.com's SOAP/REST library for Google App Engine/Java

As long as I'm reflecting on our Google I/O experiences, I also want to point out what looks like a very useful library from Salesforce. The Force.com Web Services Connector is a toolkit designed to simplify calling WSDL-defined SOAP and REST services. The best part is that they have a version that works on Google App Engine for Java! (Make sure that you use wsc-gae-16_0.jar, not the regular version.)

I haven't had the chance to do a lot of development on GAE/J, but my colleagues have definitely had some headaches getting SOAP and REST calls working around the GAE/J whitelist. Maybe one of them can comment after we give this toolkit a whirl.