new version of s3-simple

Just committed some small changes to the s3-simple library for specifying ACLs, and/or arbitrary request headers/meta-data while storing keys.

Example usage:

S3Store s3 = new S3Store("s3.amazonaws.com", ACCESS_KEY, SECRET_KEY);
s3.setBucket("my-bucket");

// upload an item as public-read
s3.storeItem("test", new String("hello").getBytes(), "public-read");

// upload a js file, with a cache control-header
final Map<String, List<String>> headers = new HashMap<String, List<String>>();
headers.put("Cache-Control", Collections.singletonList("max-age=300, must-revalidate"));
headers.put("Content-Type", Collections.singletonList("application/x-javascript"));

s3.storeItem("test2.js", new String("document.write('hello');").getBytes(), "public-read", headers);

Download it here.

Currently, you can only do this while storing keys, and there's no way to retrieve this data later. Still, it was enough of a pain to get this working correctly with the request signing, so I figured I'd share the code anyway.

Work @ Bizo

We’re looking for an out-of-the-box thinker with a good sense-of-humor and a great attitude to join our product development team. As one of five software engineers for Bizo, you will take responsibility for developing key components of the Bizographic Targeting Platform, a revolutionary new way to target business advertising online. You will be a key player on an incredible team as we build our world-beating, game-changing, and massively-scalable bizographic advertising and targeting platform. In a nutshell, you will be working on difficult problems with cool people.


The Team:
We’re a small team of very talented people (if we don’t say so ourselves!). We use Agile development methodologies. We care about high quality results, not how many hours you’re in the office. We develop on Mac, run on the Cloud, and use Google Apps. We don’t write huge requirements documents or TPS reports. We believe there is no spoon!


The Ideal Candidate:

• Self-motivated


• Entrepreneurial / Hacker spirit


• Track record of achievement


• Hands on problem solver that enjoys cracking difficult nuts


• Enjoys working on teams


• Experience working with highly scalable systems


• Linux/Unix proficiency


• Amazon Web Services experience


• Bachelors degree in Computer Science or related field – points for advanced degrees


• Willing to blog about Bizo Dev :)


• Gets stuff done!


• Bonus points for MDS (Mad Bocce Skills)

Technical Highlights:

• Languages: Java (90%), Javascript, Ruby (thinking about Scala, Clojure and Erlang too!)


• All Clouds, all the time: Amazon Web Services, Google App Engine


• Frameworks/Libraries: Hadoop, Thrift, Google Collections, Spring

Send me your resume and a cover letter if you are interested in joining our team.

Spring MVC on Google App Engine

I've been developing an application on Google App Engine and reached a point where I really wanted to be able to use Spring and Spring MVC on the server side.

To my surprise, I discovered that it was relatively easy to get Spring running on Google App Engine (at least for the basic functionality that I'm using it for). However, I did come across the javax.naming.InitialContextFactory issue.

The root cause of this particular issue is that, under certain circumstances, Spring will attempt to load a bean that depends on javax.naming.InitialContextFactory. The problem is that this class is not on GAE's JRE whitelist, and, therefore, results in a ClassNotFoundException.

For example, this occurs when you're using annotation based configuration and have a line similar to the following in your config:

<context:component-scan base-package="com.bizo.accounting.ui.server" />

I searched the forums to see if anyone else had found a solution. This post discusses a potential solution.

I ended up trying the suggested solution and found that it worked. The solution involves loading a 'dummy' bean with the same id as the bean that depends on javax.naming.InitialContextFactory (prior to the context:component-scan). This tricks the container into thinking that the bean has already been loaded.

For example, I put the following line at the top of my config:

<bean id="org.springframework.context.annotation.internalPersistenceAnnotationProcessor" class="java.lang.String"></bean>

I have no doubt that I'll run into other issues with GAE, but I was pleasantly surprised to find that basic Spring integration works without too much difficulty.

google app engine (java) and s3

After struggling for way too long, I finally (sort of) got app engine talking to s3.

Background

I've used the python app engine before for a few small personal projects, and it rocks. Not having to worry, at all, about physical machines, or deployment is a huge win. And, on top of that, they provide some really nice services right out of the box (authentication, memcache, a data store, etc.). So, when they announced the availability of app engine for java, we were all really excited.

Of course, there are some limitations. No Threads, no Sockets (this really sucks), and not all JRE classes.... BUT, it's probably enough for most applications...

And, it just so happens I'm working on a project where all this sounds okay.

Local Environment

They provide a really nice local development environment. Of course, there's not 100% correlation between things working locally and things working remotely. It's to be expected, but can be a pain.

Some things to watch out for:

• crypto problems with the local Mac SDK -- luckily you can work around this by disabling all permission checking, but unfortunately this means that all permission checking will be disabled locally :).


• You can't seem to use PUT or POST locally with the java.net connection services.


• XPathFactory.newInstance() fails.

Connecting to S3

We normally use jets3t for all of our s3 access. It's a great library. However, it's not a great app engine choice because it uses Threads, and Sockets... It seemed like a big task to modify it to work for app engine... I thought using a simpler library as a base would be better.

The author of jets3t has some s3 sample code he published as part of his AWS book. After making some small changes to get over the XPath problem, I just couldn't get it to work. The code worked fine locally, and it would work when I first deployed to app engine, but after that it would fail with IOException: Unknown... Everything looked pretty straightforward... I even tried setting up a fake s3 server to see if there was some weird issues with headers, or what... nothing...

So, I decided to try out some even simpler approaches. After all, it's just a simple REST service, right? That lead me to two different paths that (sort-of) worked.

J2ME and J2SE Toolkit for Amazon S3 -- this is a small SOAP library designed for use in a J2ME environment. This works in app engine! At least for GETs (all I tested). It is very basic and only has support for 1MB objects.

S3Shell in Java -- This is a small REST library designed for use as a shell. There's a small bug (mentioned in the comments), and you need to remove some references to com.sun classes (for Base64 encoding), but otherwise it seems to work pretty well! You will have problems using PUTs locally, but it works fine in production.

Putting it all together

I decided to go with the S3Shell code. I had to make a few changes (as mentioned above), but so far so good. I put up a modified copy of the code on github, or you can download the jar directly. This code should work fine as-is on google app engine. As mentioned, there's an issue with local PUTs (Please vote for it!).

The functionality is pretty basic. If you add any other app-engine supported features (would definitely like to get some meta-data support), definitely let me know.