Work @ Bizo

We’re looking for an out-of-the-box thinker with a good sense-of-humor and a great attitude to join our product development team. As one of five software engineers for Bizo, you will take responsibility for developing key components of the Bizographic Targeting Platform, a revolutionary new way to target business advertising online. You will be a key player on an incredible team as we build our world-beating, game-changing, and massively-scalable bizographic advertising and targeting platform. In a nutshell, you will be working on difficult problems with cool people.


The Team:
We’re a small team of very talented people (if we don’t say so ourselves!). We use Agile development methodologies. We care about high quality results, not how many hours you’re in the office. We develop on Mac, run on the Cloud, and use Google Apps. We don’t write huge requirements documents or TPS reports. We believe there is no spoon!


The Ideal Candidate:

• Self-motivated


• Entrepreneurial / Hacker spirit


• Track record of achievement


• Hands on problem solver that enjoys cracking difficult nuts


• Enjoys working on teams


• Experience working with highly scalable systems


• Linux/Unix proficiency


• Amazon Web Services experience


• Bachelors degree in Computer Science or related field – points for advanced degrees


• Willing to blog about Bizo Dev :)


• Gets stuff done!


• Bonus points for MDS (Mad Bocce Skills)

Technical Highlights:

• Languages: Java (90%), Javascript, Ruby (thinking about Scala, Clojure and Erlang too!)


• All Clouds, all the time: Amazon Web Services, Google App Engine


• Frameworks/Libraries: Hadoop, Thrift, Google Collections, Spring

Send me your resume and a cover letter if you are interested in joining our team.

Spring MVC on Google App Engine

I've been developing an application on Google App Engine and reached a point where I really wanted to be able to use Spring and Spring MVC on the server side.

To my surprise, I discovered that it was relatively easy to get Spring running on Google App Engine (at least for the basic functionality that I'm using it for). However, I did come across the javax.naming.InitialContextFactory issue.

The root cause of this particular issue is that, under certain circumstances, Spring will attempt to load a bean that depends on javax.naming.InitialContextFactory. The problem is that this class is not on GAE's JRE whitelist, and, therefore, results in a ClassNotFoundException.

For example, this occurs when you're using annotation based configuration and have a line similar to the following in your config:

<context:component-scan base-package="com.bizo.accounting.ui.server" />

I searched the forums to see if anyone else had found a solution. This post discusses a potential solution.

I ended up trying the suggested solution and found that it worked. The solution involves loading a 'dummy' bean with the same id as the bean that depends on javax.naming.InitialContextFactory (prior to the context:component-scan). This tricks the container into thinking that the bean has already been loaded.

For example, I put the following line at the top of my config:

<bean id="org.springframework.context.annotation.internalPersistenceAnnotationProcessor" class="java.lang.String"></bean>

I have no doubt that I'll run into other issues with GAE, but I was pleasantly surprised to find that basic Spring integration works without too much difficulty.

google app engine (java) and s3

After struggling for way too long, I finally (sort of) got app engine talking to s3.

Background

I've used the python app engine before for a few small personal projects, and it rocks. Not having to worry, at all, about physical machines, or deployment is a huge win. And, on top of that, they provide some really nice services right out of the box (authentication, memcache, a data store, etc.). So, when they announced the availability of app engine for java, we were all really excited.

Of course, there are some limitations. No Threads, no Sockets (this really sucks), and not all JRE classes.... BUT, it's probably enough for most applications...

And, it just so happens I'm working on a project where all this sounds okay.

Local Environment

They provide a really nice local development environment. Of course, there's not 100% correlation between things working locally and things working remotely. It's to be expected, but can be a pain.

Some things to watch out for:

• crypto problems with the local Mac SDK -- luckily you can work around this by disabling all permission checking, but unfortunately this means that all permission checking will be disabled locally :).


• You can't seem to use PUT or POST locally with the java.net connection services.


• XPathFactory.newInstance() fails.

Connecting to S3

We normally use jets3t for all of our s3 access. It's a great library. However, it's not a great app engine choice because it uses Threads, and Sockets... It seemed like a big task to modify it to work for app engine... I thought using a simpler library as a base would be better.

The author of jets3t has some s3 sample code he published as part of his AWS book. After making some small changes to get over the XPath problem, I just couldn't get it to work. The code worked fine locally, and it would work when I first deployed to app engine, but after that it would fail with IOException: Unknown... Everything looked pretty straightforward... I even tried setting up a fake s3 server to see if there was some weird issues with headers, or what... nothing...

So, I decided to try out some even simpler approaches. After all, it's just a simple REST service, right? That lead me to two different paths that (sort-of) worked.

J2ME and J2SE Toolkit for Amazon S3 -- this is a small SOAP library designed for use in a J2ME environment. This works in app engine! At least for GETs (all I tested). It is very basic and only has support for 1MB objects.

S3Shell in Java -- This is a small REST library designed for use as a shell. There's a small bug (mentioned in the comments), and you need to remove some references to com.sun classes (for Base64 encoding), but otherwise it seems to work pretty well! You will have problems using PUTs locally, but it works fine in production.

Putting it all together

I decided to go with the S3Shell code. I had to make a few changes (as mentioned above), but so far so good. I put up a modified copy of the code on github, or you can download the jar directly. This code should work fine as-is on google app engine. As mentioned, there's an issue with local PUTs (Please vote for it!).

The functionality is pretty basic. If you add any other app-engine supported features (would definitely like to get some meta-data support), definitely let me know.

Calling SOAP Web Services in Google App Engine

We have taken the plunge and are begining to develop a couple of internal apps using GWT and Google App Engine (GAE) for Java.

The app that I'm writing needs to make SOAP calls to get external data from a service. The service provides a set of WSDLs and we use Apache Axis to generate code to call this service. However, when I tried using the generated code in the Google Eclipse Plugin environment, I received the following exception:

Caused by: java.security.AccessControlException: access denied (java.net.SocketPermission xxx.xxxx.com resolve)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
at java.security.AccessController.checkPermission(AccessController.java:427)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:76)
at java.lang.SecurityManager.checkConnect(SecurityManager.java:1031)
at java.net.InetAddress.getAllByName0(InetAddress.java:1134)
at java.net.InetAddress.getAllByName(InetAddress.java:1072)
at java.net.InetAddress.getAllByName(InetAddress.java:1008)
at java.net.InetAddress.getByName(InetAddress.java:958)
at java.net.InetSocketAddress.(InetSocketAddress.java:124)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.(SSLSocketImpl.java:350)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:69)
at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:92)
at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
... 45 more

This was annoying, but not a huge surprise as it is well known and documented that GAE has its own JVM implementation which prevents things like the creation of socket connections and threads. I did some investigation to see if others had solved this particular problem, calling a SOAP Web Service from GAE, but didn't really find any good answers.

In any case, I understood that HTTP communication in GAE must occur through either the URL Fetch Service or java.net.URL class. So, I needed to find a way to make Axis use one of these methods instead of opening sockets directly.

After learning more about how Axis works and then doing more searching, I came across SimpleHTTPSender. This class is an Axis Handler that uses only java.net.URL and friends for HTTP communication.

Problem solved.

Now, on to solving the next one... :)

Public <=> Private DNS names in EC2

Every instance in Amazon's EC2 cloud has both a public and a private DNS name. When dealing with my instances, I often find that I have one of these two names, but really want the other. For example, one of tools lists the status of our servers by their private DNS names (domU-11-22-33-44-55-66.compute-1.internal). If I need to log on to one of these machines to do something, I need to convert this to a public DNS name (ec2-111-222-333-444.compute-1.amazonaws.com). My solution so far to mapping between public and private DNS names has been to simply write down this information on a glass panel by my desk whenever I start an instance.

Needless to say, this solution works fine whenever I'm dealing with my developer instances from my desk, but it doesn't work so well when I'm working from home or dealing with instances other people have started. We have a few general-purpose tools that will pull down instance meta-data, but it's a bit of a pain to use. So, I finally gave in and wrote a simple Ruby script that pulls down all instance metadata and does a simple regex pattern match against the public and private DNS names. Just pass it an initial substring of the private DNS name (eg, "domU-11-22-33-44-55-66", "domU-11-22-33-44-55-66.compute-1.internal", or simply "domU"), and it will spit out the public DNS name for all matching instances. It works in reverse, too, if you want the private DNS name for a given public DNS name.

Note that you'll need to have the amazon-ec2 gem installed, which you can get by simply running the command "gem install amazon-ec2".

IvyDE: Eclipse plug-in for Ivy

If you're using Eclipse and Ivy (as we do here at Bizo), you definitely want the IvyDE plug-in. IvyDE will create a classpath container containing all of your Ivy dependencies, making it trivial to keep your Eclipse classpath synchronized with your ivy file.

For automatic installation, point Eclipse's software updates at the following URL:

http://www.apache.org/dist/ant/ivyde/updatesite

Once installed, you should go to Eclipse Preferences->Ivy and, in the first text field under "Global settings" point the plug-in at your ivyconf.xml file. While you're there, you can add your organization info at the bottom of the dialog, too. Now, it's simply a matter of right-clicking on your ivy.xml file and selecting "Add Ivy Library" to include all of your Ivy dependencies in the project. Alternatively, you can edit your Java build path and (in the Libraries section) select "IvyDE Managed Dependencies" from the "Add Library..." dialog.

Prior to joining Bizo, I used Maven and m2eclipse for the one-two punch of dependency management and Eclipse integration. With the addition of this plug-in, I now feel like Ant/Ivy/IvyDE gives me all of the convenience of Maven/m2eclipse with a great deal more flexibility. Of course, some of that may be a result of the common-build system we use for all of our projects. Hopefully, we'll have some time to talk about common-build in the near future....

'On Demand' javascript reloading with jQuery

So I was writing some "exploratory" javascript the other day. What I mean by "exploratory" is that I wanted to pull some data out of the DOM, but I didn't know the exact location of the data I wanted to retrieve.

I used Firebug to examine the DOM, figure out what I wanted to retrieve, and then wrote some javascript to actually retrieve the data I needed. However, I wanted to be able to perform this cycle without re-loading the page. In other words, I needed a way to be able to reload my javascript file on demand.

One really simple way to do this is to create a button on your page and then call the jQuery.getScript function in the 'onclick' handler. For example:

<input value="Reload Me" onclick="jQuery.getScript('http://localhost/my.js');" type="button">

This function will asynchronously load your javascript file and execute it. The only issue I had in doing this is that it broke Firebug's ability to debug the code. However, it was nice to be able to modify my javascript code in Eclipse and then immediately run it in my browser.

If anyone has other/better ways to do this I'd love to hear about them.