Calling SOAP Web Services in Google App Engine

We have taken the plunge and are begining to develop a couple of internal apps using GWT and Google App Engine (GAE) for Java.

The app that I'm writing needs to make SOAP calls to get external data from a service. The service provides a set of WSDLs and we use Apache Axis to generate code to call this service. However, when I tried using the generated code in the Google Eclipse Plugin environment, I received the following exception:

Caused by: java.security.AccessControlException: access denied (java.net.SocketPermission xxx.xxxx.com resolve)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
at java.security.AccessController.checkPermission(AccessController.java:427)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:76)
at java.lang.SecurityManager.checkConnect(SecurityManager.java:1031)
at java.net.InetAddress.getAllByName0(InetAddress.java:1134)
at java.net.InetAddress.getAllByName(InetAddress.java:1072)
at java.net.InetAddress.getAllByName(InetAddress.java:1008)
at java.net.InetAddress.getByName(InetAddress.java:958)
at java.net.InetSocketAddress.(InetSocketAddress.java:124)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.(SSLSocketImpl.java:350)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:69)
at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:92)
at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
... 45 more

This was annoying, but not a huge surprise as it is well known and documented that GAE has its own JVM implementation which prevents things like the creation of socket connections and threads. I did some investigation to see if others had solved this particular problem, calling a SOAP Web Service from GAE, but didn't really find any good answers.

In any case, I understood that HTTP communication in GAE must occur through either the URL Fetch Service or java.net.URL class. So, I needed to find a way to make Axis use one of these methods instead of opening sockets directly.

After learning more about how Axis works and then doing more searching, I came across SimpleHTTPSender. This class is an Axis Handler that uses only java.net.URL and friends for HTTP communication.

Problem solved.

Now, on to solving the next one... :)

Public <=> Private DNS names in EC2

Every instance in Amazon's EC2 cloud has both a public and a private DNS name. When dealing with my instances, I often find that I have one of these two names, but really want the other. For example, one of tools lists the status of our servers by their private DNS names (domU-11-22-33-44-55-66.compute-1.internal). If I need to log on to one of these machines to do something, I need to convert this to a public DNS name (ec2-111-222-333-444.compute-1.amazonaws.com). My solution so far to mapping between public and private DNS names has been to simply write down this information on a glass panel by my desk whenever I start an instance.

Needless to say, this solution works fine whenever I'm dealing with my developer instances from my desk, but it doesn't work so well when I'm working from home or dealing with instances other people have started. We have a few general-purpose tools that will pull down instance meta-data, but it's a bit of a pain to use. So, I finally gave in and wrote a simple Ruby script that pulls down all instance metadata and does a simple regex pattern match against the public and private DNS names. Just pass it an initial substring of the private DNS name (eg, "domU-11-22-33-44-55-66", "domU-11-22-33-44-55-66.compute-1.internal", or simply "domU"), and it will spit out the public DNS name for all matching instances. It works in reverse, too, if you want the private DNS name for a given public DNS name.

Note that you'll need to have the amazon-ec2 gem installed, which you can get by simply running the command "gem install amazon-ec2".

IvyDE: Eclipse plug-in for Ivy

If you're using Eclipse and Ivy (as we do here at Bizo), you definitely want the IvyDE plug-in. IvyDE will create a classpath container containing all of your Ivy dependencies, making it trivial to keep your Eclipse classpath synchronized with your ivy file.

For automatic installation, point Eclipse's software updates at the following URL:

http://www.apache.org/dist/ant/ivyde/updatesite

Once installed, you should go to Eclipse Preferences->Ivy and, in the first text field under "Global settings" point the plug-in at your ivyconf.xml file. While you're there, you can add your organization info at the bottom of the dialog, too. Now, it's simply a matter of right-clicking on your ivy.xml file and selecting "Add Ivy Library" to include all of your Ivy dependencies in the project. Alternatively, you can edit your Java build path and (in the Libraries section) select "IvyDE Managed Dependencies" from the "Add Library..." dialog.

Prior to joining Bizo, I used Maven and m2eclipse for the one-two punch of dependency management and Eclipse integration. With the addition of this plug-in, I now feel like Ant/Ivy/IvyDE gives me all of the convenience of Maven/m2eclipse with a great deal more flexibility. Of course, some of that may be a result of the common-build system we use for all of our projects. Hopefully, we'll have some time to talk about common-build in the near future....

'On Demand' javascript reloading with jQuery

So I was writing some "exploratory" javascript the other day. What I mean by "exploratory" is that I wanted to pull some data out of the DOM, but I didn't know the exact location of the data I wanted to retrieve.

I used Firebug to examine the DOM, figure out what I wanted to retrieve, and then wrote some javascript to actually retrieve the data I needed. However, I wanted to be able to perform this cycle without re-loading the page. In other words, I needed a way to be able to reload my javascript file on demand.

One really simple way to do this is to create a button on your page and then call the jQuery.getScript function in the 'onclick' handler. For example:

<input value="Reload Me" onclick="jQuery.getScript('http://localhost/my.js');" type="button">

This function will asynchronously load your javascript file and execute it. The only issue I had in doing this is that it broke Firebug's ability to debug the code. However, it was nice to be able to modify my javascript code in Eclipse and then immediately run it in my browser.

If anyone has other/better ways to do this I'd love to hear about them.

Another win for findbugs: is Math.abs broken?

I was wrapping up some code today, and ran into a weirdly interesting findbugs warning: RV_ABSOLUTE_VALUE_OF_HASHCODE.

Bill Pugh has a more in-depth post on his blog from 2006 about this. But, if you've ever written code like:

Object x = bucket[ Math.abs(y.hashCode()) % bucket.length]

you have a bug. Or, maybe it's a bug in Math.abs, depending on your point of view.

You would probably be surprised to learn that Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE, but it's true (go ahead and try it out).

So, about one time out of 4 billion, you'll get an ArrayIndexOutOfBoundsException, when your object's hashCode returns Integer.MIN_VALUE, and you end up with a negative array index.

This seems like a fairly common bit of code to write. In fact, back when Bill wrote this post, he uncovered this occurring 7 times in eclipse, 6 times in jboss, and twice in tomcat, among others.

This behavior sort of makes sense when you think about it. There's really no other value Math.abs could return for Integer.MIN_VALUE, and, it is, in fact, documented. Still, I'd bet that this bug exists somewhere in code you are using.

Anyway, it's worth reading Bill's post to check out some suggested fixes, and dig a little deeper into this.

stupid simple xml parsing

In the last few months, I've been doing a lot of XML parsing. It's been mostly small little parsers, in both java and python, trying to get some stats on different data sets. This is all on huge data sets, so streaming/stax type parsers. It's also kind of 3rd party data files, so of course the structure is crazy and weird :), or at least, I can't change it...

Anyway, last night I was working on probably my 5th or 6th parser, and man, was it getting repetitive. With these types of parsers, it's a lot of just keeping track of what state you're in, and then shoving it into your model.

Some pseudocode illustrating the repetitive nature of this:

case START_ELEMENT:
  if (name == "company") { inCompany = true; }
  if (name == "domains") { inDomains = true; }
  if (name == "value" && inDomains) { inDomainValue = true; }
case CHARACTERS:
  if (inDomainValue) { company.addDomain(characters) };
case END_ELEMENT:
  if (name == "company") { inCompany = false; }
  if (name == "domains" && inCompany) { inDomains = false; }
  if (name == "value" && inDomain) { inDomainValue = false; }

Anyway, really repetitive... and just a pain to keep testing these similar parsers over and over again.

So, last night, I tried to come up with a super simple generic parser. In fact, I decided that all I really needed was some way to get the Strings I'm interested in into java objects. Initially, my model object had Integers, enums, sub objects. I decided that's too much -- No complex binding, just get the strings our of XML into java, and then from there I can form more complex objects if I need to, do filtering, etc.

I've also been wanted to take the time and learn more about java annotations. Of course, I've used them before, but I'd never created my own, or parsed them.

Here's the new model:

@XmlPath("company")
public class Company {
  @XmlPath("name")
  private String name;
  @XmlPath("domains/value")
  private List domains;
....
}

Now, you can just write

new SimpleParser(consumer, Company.class).parse(in);

The parser will inspect the Company class' annotations to figure out what paths it cares about, then internally keep a simple queue of its state. Each time it comes across a path the class cares about, it will either set the value, or add to the List of values depending on the field type. Again, only strings! Once it reaches the end element, it will pass off the last constructed to the consumer, then create a new bean.

Anyway, surprisingly, it worked pretty well! I realize this isn't going to solve all XML parsing needs, but it seems like for 90% of my use cases, it's "good enough."

I did a few searches, and didn't really find anything as simple, although StaXMate looks pretty interesting...

I'd love to get some feedback on this. I wonder if there's already something else out there? Or a better way to do this?

I put the code up on github, as stupid-xml. Quick warning: this is just a late night experiment. It's not production code. I did almost no checking of anything, there are no comments, it's probably crazy inefficient, and I'm probably doing a ton of dumb things. If so, please feel free to fork the code and submit changes, or send feedback, etc.