Want to work on some interesting problems with a great development team?
We're looking to hire a junior developer and a quantitative engineer.
Send us your resume! Or if you know someone in the Bay Area that you think might be a good fit, please forward the posting to them.
Friday, June 25, 2010
Tuesday, June 8, 2010
Accessing Bizo API using Ruby OAuth
During a recent HackDay, I wrote a search-engine-like frontend to our multi-dimensional business demographic database.
Among other things, the interface provides search term suggestions based on business title classification, using the Classify operation of the Bizo API.
I chose to use Ruby and the lightweight Sinatra web framework for fast prototyping and since the Bizo API uses OAuth for authentication, I reached out for the excellent OAuth gem.
Now, while the documentation is good it took me a little time to grok the OAuth API and figure out how to use it. The Bizo API does not use a RequestToken; instead we use an API key and a shared secret. Since the OAuth gem documentation didn't include any example for this use-case, I figured I'd post my code here as a starting point for other people to reuse.
Without further ado, here's the short code fragment:
Hopefully this is useful to Rubyists out there needing quick OAuth integration using HTTP GET and a query string and don't need to go through the token exchange process.
Among other things, the interface provides search term suggestions based on business title classification, using the Classify operation of the Bizo API.
I chose to use Ruby and the lightweight Sinatra web framework for fast prototyping and since the Bizo API uses OAuth for authentication, I reached out for the excellent OAuth gem.
Now, while the documentation is good it took me a little time to grok the OAuth API and figure out how to use it. The Bizo API does not use a RequestToken; instead we use an API key and a shared secret. Since the OAuth gem documentation didn't include any example for this use-case, I figured I'd post my code here as a starting point for other people to reuse.
Without further ado, here's the short code fragment:
If you're curious, here's the JSON response for the "VP of Market..." title classification,
require 'rubygems'
require 'oauth'
require 'oauth/consumer'
require 'json'
key = 'xxxxxxxx'
secret = 'yyyyyyyy'
consumer = OAuth::Consumer.new(key, secret, {
:site => "http://api.bizographics.com",
:scheme => :query_string,
:http_method => :get
})
title = "VP of Marketing"
path = URI.escape("/v1/classify.json?api_key=#{key}&title=#{title}")
response = consumer.request(:get, path)
# Display response
p JSON.parse(response.body)
{
"usage" => 1,
"bizographics" => {
"group" => { "name" => "High Net Worth", "code" => "high_net_worth" },
"functional_area" => [
{"name" => "Sales", "code" => "sales" },
{"name" => "Marketing", "code" => "marketing" }
],
"seniority" => {"name" => "Executives", "code" => "executive" }
}
}
Hopefully this is useful to Rubyists out there needing quick OAuth integration using HTTP GET and a query string and don't need to go through the token exchange process.
Tuesday, May 11, 2010
Hackday: dependency searching using scala, jersey, gxp, mongodb
For my hackday project, I thought I would try to build an internal tool to let us more easily search our dependency repository. We use ivy for dependency management, and maintain our own repository in s3. It can be kind of a pain to track down the latest version of library X, especially if you're not sure what the organization is, or maybe you know the org and not the name. It seemed like a fun, useful project that I could tackle in a day, and that would allow me to play around with a couple of things I was interested in. To build it, I used jersey, gxp, and mongoDB. The whole thing was written using scala.
I borrowed the main layout from the SpringSource Enterprise Bundle Repository. I'm pretty happy with the results:
And the detail view:
There's also a browse view.
I've been really happy using scala and jersey, and I wanted something simple and easy for this project, so I thought it was worth a shot. After adding GXP for templating support, I have to say the combination of scala/jersey/GXP makes a pretty compelling framework for simple web apps.
As an example, here's the beginning of my 'Browse' Controller:
It's using nested paths, so /b/o is the main browse by organization page, /b/o/G would be all organizations starting with 'G'.
Then, I have a simple MessageBodyWriter that can render a GxpClosure:
And, that's really all there is to it. Nice, simple, and lightweight.
Last but not least, mongodb. It was probably overkill for this project, but I was looking for an excuse to play with it some more. I use it to store and index all of the repository information. I have a separate crawler process that lists everything in our repository s3 bucket, then stores an entry for each artifact. As part of this, it does some basic tokenizing of the organization and artifact names for searching. Searching like this was a little disappointing compared to lucene. Overall though, I'm pretty happy with it. Browsing and searching are both ridiculously fast. Like I said, it was probably overkill for the amount of data we have.... but it can never be too fast. speed is most definitely a feature.
Anyway, that's the wrap-up.
I'd be interested to other thoughts/experiences on mongodb from anyone out there.
I borrowed the main layout from the SpringSource Enterprise Bundle Repository. I'm pretty happy with the results:
And the detail view:
There's also a browse view.
I've been really happy using scala and jersey, and I wanted something simple and easy for this project, so I thought it was worth a shot. After adding GXP for templating support, I have to say the combination of scala/jersey/GXP makes a pretty compelling framework for simple web apps.
As an example, here's the beginning of my 'Browse' Controller:
@Path("/b")
class Browse {
val db = new RepoDB
@Path("/o")
@GET @Produces(Array("text/html"))
def browseOrg() = browseOrgLetter("A")
@Path("/o/{letter}")
@GET @Produces(Array("text/html"))
def browseOrgLetter(@PathParam("letter") letter : String) = {
val orgs = db.getOrgLetters
val results = db.findByOrgLetter(letter, 30)
BrowseView.getGxpClosure("Organization", "o", orgs, letter, results)
}
...
It's using nested paths, so /b/o is the main browse by organization page, /b/o/G would be all organizations starting with 'G'.
Then, I have a simple MessageBodyWriter that can render a GxpClosure:
@Provider
@Produces(Array("text/html"))
class GxpClosureWriter extends MessageBodyWriter[GxpClosure] {
val context = new GxpContext(Locale.US)
override def isWriteable(dataType: java.lang.Class[_], ...) = {
classOf[GxpClosure].isAssignableFrom(dataType)
}
override def writeTo(gxp: GxpClosure, ...) {
val out = new java.io.OutputStreamWriter(_out)
gxp.write(out, context)
}
...
And, that's really all there is to it. Nice, simple, and lightweight.
Last but not least, mongodb. It was probably overkill for this project, but I was looking for an excuse to play with it some more. I use it to store and index all of the repository information. I have a separate crawler process that lists everything in our repository s3 bucket, then stores an entry for each artifact. As part of this, it does some basic tokenizing of the organization and artifact names for searching. Searching like this was a little disappointing compared to lucene. Overall though, I'm pretty happy with it. Browsing and searching are both ridiculously fast. Like I said, it was probably overkill for the amount of data we have.... but it can never be too fast. speed is most definitely a feature.
Anyway, that's the wrap-up.
I'd be interested to other thoughts/experiences on mongodb from anyone out there.
Wednesday, May 5, 2010
Improving Global Application Performance, continued: GSLB with EC2
This is an unofficial continuation of Amazon's blog post on the use of Amazon CloudFront to improve application performance.
CloudFront is a great CDN to consider, especially if you're already an Amazon Web Services customer. Unfortunately, it can only be used for static content; the loading of dynamic content will still be slower for far-away users than for nearby ones. Simply put, users in India will still see a half-second delay when loading the dynamic portions of your US-based website. And a half-second delay has a measurable impact on revenue.
Let's talk about speeding up dynamic content, globally.
The typical EC2 implementation comprises instances deployed in a single region. Such a deployment may span several availability zones for redundancy, but all instances are in roughly the same place, geographically.
This is fine for EC2-hosted apps with nominal revenue or a highly localized user base. But what if your users are spread around the globe? The problem can't be solved by moving your application to another region - that would simply shift the extra latency to another group.
For a distributed audience, you need a distributed infrastructure. But you can't simply launch servers around the world and expect traffic to reach them. Enter Global Server Load Balancing (GSLB).
A primer on GSLB
Broadly, GSLB is used to intelligently distribute traffic across multiple datacenters based on some set of rules.
With GSLB, your traffic distribution can go from this:
To this:
GSLB can be implemented as a feature of a physical device (including certain high-end load balancers) or as a part of a DNS service. Since we EC2 users are clearly not interested in hardware, our focus is on the latter: DNS-based GSLB.
Standard DNS behavior is for an authoritative nameserver to, given queries for a certain record, always return the same result. A DNS-based implementation of GSLB would alter this behavior so that queries return context-dependent results.
Example:
User A queries DNS for gslb.example.com -- response: 10.1.0.1
User B queries DNS for gslb.example.com -- response: 10.2.0.1
But what context should we use? Since our goal is to reduce wire latency, we should route users to the closest datacenter. IP blocks can be mapped geographically -- by examining a requestor's IP address, a GSLB service can return a geo-targeted response.
With geo-targeted DNS, our example would be:
User A (in China) queries DNS for geo.example.com -- response: 10.1.0.1
User B (in Spain) queries DNS for geo.example.com -- response: 10.2.0.1
Getting started
At a high level, implementation can be broken down into two steps
1) Deploying infrastructure in other AWS regions
2) Configuring GSLB-capable DNS
Infrastructure configurations will vary from shop to shop, but as an example, a read-heavy EC2 application with a single master database for writes should:
- deploy application servers to all regions
- deploy read-only (slave) database servers and/or read caches to all regions
- configure application servers to use the slave database servers and/or read caches in their region for reads
- configure application servers to use the single master in the "main" region for writes
This is what such an environment would look like:
When configuring servers to communicate across regions (app servers -> master DB; slave DBs -> master DB), you will need to use IP-based rules for your security groups; traffic from the "app-servers" security group you set up in eu-west-1 is indistinguishable from other traffic to your DB server in us-east-1. This is because cross-region communication is done using external IP addresses. Your best bet is to either automate security group updates or use Elastic IPs.
Note on more complex configurations: distributed backends are hard (see Brewer's [CAP] theorem). Multi-region EC2 environments are much easier to implement if your application tolerates the use of 1) regional caches for reads; 2) centralized writes. If you have a choice, stick with the simpler route.
As for configuring DNS, several companies have DNS-based GSLB service offerings:
- Dynect - Traffic Management (A records only) and CDN Manager (CNAMEs allowed)
- Akamai - Global Traffic Management
- UltraDNS - Directional DNS
- Comwired/DNS.com - Location Geo-Targeting
DNS configuration should be pretty similar for the vendors listed above. Basic steps are:
1) set up regional CNAMEs (us-east-1.example.com, us-west-1.example.com, eu-west-1.example.com, ap-southeast-1.example.com)
2) set up a GSLB-enabled "master" CNAME (www.example.com)
3) define the GSLB rules:
- For users in Asia, return ap-southeast-1.example.com
- For users in Europe, return eu-west-1.example.com
- For users in Western US, return us-west-1.example.com
- ...
- For all other users, return us-east-1.example.com
If your application is already live, consider abstracting the DNS records by one layer: geo.example.com (master record); us-east-1.geo.example.com, us-west-1.geo.example.com, etc. (regional records). Bring the new configuration live by pointing www.example.com (CNAME) to geo.example.com.
Bizo's experiences
Several of our EC2 applications serve embedded content for customer websites, so it's critical we minimize load times. Here's the difference we saw on one app after expanding into new regions (from us-east-1 to us-east-1, us-west-1, and eu-west-1) and implementing GSLB (load times provided by BrowserMob):
Load times before GSLB:
Load times after GSLB:
Reduced load times for everyone far from us-east-1. Users are happy, customers are happy, we're happy. Overall, a success.
It's interesting to see how the load is distributed throughout the day. Here's one application's HTTP traffic, broken down by region (ELB stats graphed by cloudviz):
Note that the use of Elastic Load Balancers and Auto Scaling becomes much more compelling with GSLB. By geographically partitioning users, peak hours are much more localized. This results in a wider difference between peak and trough demand per region; Auto Scaling adjusts capacity transparently, reducing the marginal cost of expanding your infrastructure to multiple AWS regions.
For our GSLB DNS service, we use Dynect and couldn't be more pleased. Intuitive management interface, responsive and helpful support, friendly, no-BS sales. Pricing is based on number of GSLB-enabled domains and DNS query rate. Contact Dynect sales if you want specifics (we work with Josh Delisle and Kyle York - great guys). Note that those intending to use GSLB with Elastic Load Balancers will need the CDN Management service.
Closing remarks
Previously, operating a global infrastructure required significant overhead. This is where AWS really shines. Amazon now has four regions spread across three continents, and there's minimal overhead to distribute your platform across all of them. You just need to add a layer to route users to the closest one.
The use of Amazon CloudFront in conjunction with a global EC2 infrastructure is a killer combo for improving application performance. And with Amazon continually expanding with new AWS regions, it's only going to get better.
@mikebabineau
CloudFront is a great CDN to consider, especially if you're already an Amazon Web Services customer. Unfortunately, it can only be used for static content; the loading of dynamic content will still be slower for far-away users than for nearby ones. Simply put, users in India will still see a half-second delay when loading the dynamic portions of your US-based website. And a half-second delay has a measurable impact on revenue.
Let's talk about speeding up dynamic content, globally.
The typical EC2 implementation comprises instances deployed in a single region. Such a deployment may span several availability zones for redundancy, but all instances are in roughly the same place, geographically.
This is fine for EC2-hosted apps with nominal revenue or a highly localized user base. But what if your users are spread around the globe? The problem can't be solved by moving your application to another region - that would simply shift the extra latency to another group.
For a distributed audience, you need a distributed infrastructure. But you can't simply launch servers around the world and expect traffic to reach them. Enter Global Server Load Balancing (GSLB).
A primer on GSLB
Broadly, GSLB is used to intelligently distribute traffic across multiple datacenters based on some set of rules.
With GSLB, your traffic distribution can go from this:
To this:
GSLB can be implemented as a feature of a physical device (including certain high-end load balancers) or as a part of a DNS service. Since we EC2 users are clearly not interested in hardware, our focus is on the latter: DNS-based GSLB.
Standard DNS behavior is for an authoritative nameserver to, given queries for a certain record, always return the same result. A DNS-based implementation of GSLB would alter this behavior so that queries return context-dependent results.
Example:
User A queries DNS for gslb.example.com -- response: 10.1.0.1
User B queries DNS for gslb.example.com -- response: 10.2.0.1
But what context should we use? Since our goal is to reduce wire latency, we should route users to the closest datacenter. IP blocks can be mapped geographically -- by examining a requestor's IP address, a GSLB service can return a geo-targeted response.
With geo-targeted DNS, our example would be:
User A (in China) queries DNS for geo.example.com -- response: 10.1.0.1
User B (in Spain) queries DNS for geo.example.com -- response: 10.2.0.1
Getting started
At a high level, implementation can be broken down into two steps
1) Deploying infrastructure in other AWS regions
2) Configuring GSLB-capable DNS
Infrastructure configurations will vary from shop to shop, but as an example, a read-heavy EC2 application with a single master database for writes should:
- deploy application servers to all regions
- deploy read-only (slave) database servers and/or read caches to all regions
- configure application servers to use the slave database servers and/or read caches in their region for reads
- configure application servers to use the single master in the "main" region for writes
This is what such an environment would look like:
When configuring servers to communicate across regions (app servers -> master DB; slave DBs -> master DB), you will need to use IP-based rules for your security groups; traffic from the "app-servers" security group you set up in eu-west-1 is indistinguishable from other traffic to your DB server in us-east-1. This is because cross-region communication is done using external IP addresses. Your best bet is to either automate security group updates or use Elastic IPs.
Note on more complex configurations: distributed backends are hard (see Brewer's [CAP] theorem). Multi-region EC2 environments are much easier to implement if your application tolerates the use of 1) regional caches for reads; 2) centralized writes. If you have a choice, stick with the simpler route.
As for configuring DNS, several companies have DNS-based GSLB service offerings:
- Dynect - Traffic Management (A records only) and CDN Manager (CNAMEs allowed)
- Akamai - Global Traffic Management
- UltraDNS - Directional DNS
- Comwired/DNS.com - Location Geo-Targeting
DNS configuration should be pretty similar for the vendors listed above. Basic steps are:
1) set up regional CNAMEs (us-east-1.example.com, us-west-1.example.com, eu-west-1.example.com, ap-southeast-1.example.com)
2) set up a GSLB-enabled "master" CNAME (www.example.com)
3) define the GSLB rules:
- For users in Asia, return ap-southeast-1.example.com
- For users in Europe, return eu-west-1.example.com
- For users in Western US, return us-west-1.example.com
- ...
- For all other users, return us-east-1.example.com
If your application is already live, consider abstracting the DNS records by one layer: geo.example.com (master record); us-east-1.geo.example.com, us-west-1.geo.example.com, etc. (regional records). Bring the new configuration live by pointing www.example.com (CNAME) to geo.example.com.
Bizo's experiences
Several of our EC2 applications serve embedded content for customer websites, so it's critical we minimize load times. Here's the difference we saw on one app after expanding into new regions (from us-east-1 to us-east-1, us-west-1, and eu-west-1) and implementing GSLB (load times provided by BrowserMob):
Load times before GSLB:
Load times after GSLB:
Reduced load times for everyone far from us-east-1. Users are happy, customers are happy, we're happy. Overall, a success.
It's interesting to see how the load is distributed throughout the day. Here's one application's HTTP traffic, broken down by region (ELB stats graphed by cloudviz):
Note that the use of Elastic Load Balancers and Auto Scaling becomes much more compelling with GSLB. By geographically partitioning users, peak hours are much more localized. This results in a wider difference between peak and trough demand per region; Auto Scaling adjusts capacity transparently, reducing the marginal cost of expanding your infrastructure to multiple AWS regions.
For our GSLB DNS service, we use Dynect and couldn't be more pleased. Intuitive management interface, responsive and helpful support, friendly, no-BS sales. Pricing is based on number of GSLB-enabled domains and DNS query rate. Contact Dynect sales if you want specifics (we work with Josh Delisle and Kyle York - great guys). Note that those intending to use GSLB with Elastic Load Balancers will need the CDN Management service.
Closing remarks
Previously, operating a global infrastructure required significant overhead. This is where AWS really shines. Amazon now has four regions spread across three continents, and there's minimal overhead to distribute your platform across all of them. You just need to add a layer to route users to the closest one.
The use of Amazon CloudFront in conjunction with a global EC2 infrastructure is a killer combo for improving application performance. And with Amazon continually expanding with new AWS regions, it's only going to get better.
@mikebabineau
Friday, April 2, 2010
GWT MVP Tables
I'm building a GWT MVP app here at Bizo. Despite its upsides (primarily insanely fast unit test), GWT MVP is new, under documented, and, for me anyway, has had a non-trivial learning curve.
So, in the interest of sharing tricks I've learned, one I've come across is how to elegantly implement dashboard-style tables.
Dashboard-style tables are those where each row has a lot of interactivity, e.g. a user can start/stop/cancel the entity in the row, see the icons change, and other various UI things that you generally want to have under test in presenters.
For awhile, I was stumped because most GWT MVP tables favor data-style tables, where the focus is more on efficiently displaying 100s/1000s of rows of basically non-interactive data. There are a few counter examples, but none I had seen got to the point of pushing testable HasXxx interfaces back into a presenter for business logic to hook up to.
I finally got a suitable solution by realizing that each row is essentially its own view, and so instantiating a new RowXxxPresenter for each domain object in the value. This means there is a RowXxxView, which can expose all the HasXxx interfaces it wants to the presenter. And the RowXxxPresenterTest becomes a treat to write.
For more details and code examples, please jump over to the more elaborate post over on my personal blog.
Wednesday, March 24, 2010
Bizo Job - Designer
Position: Web / UI / UX Designer (San Francisco)
Summary:
We’re looking for an out-of-the-box thinker with a good sense-of-humor and a great attitude to join our product development team. As the first in-house Web / UI / UX Designer for Bizo, you will take responsibility for developing easy, powerful, consistent and high velocity web and interaction designs across all Bizo web products as well as marketing materials related to the Bizographic Targeting Platform, a revolutionary new way to target business advertising online. You will be a key player on an incredible team as we build our world-beating, game-changing, and massively scalable bizographic advertising and targeting platform. In a nutshell, you will be the voice of reason in all design and usability aspects of Bizo.
The Team:
We’re a small team of very talented people (if we do say so ourselves!). We use Agile development methodologies. We care about high quality results, not how many hours you’re in the office. We believe in strong design that helps people get stuff done!
The Ideal Candidate:
- Self-motivated
- Entrepreneurial / Hacker spirit
- Experience/Expertise with Adobe Illustrator (and/or similar design tools)
- Strong CSS skills
- Strong HTML skills
- Strong Javascript skills
- Understands the value of mock-ups (points for Balsamiq experience)
- Flash experience (bonus but not required)
- Enjoys working on teams
- Educational background or industry experience in Design or related field – points for advanced degrees
- Gets stuff done!
Please send a resume, cover letter and link to online portfolio to: donnie@bizo.com
Wednesday, March 17, 2010
Introducing Cloudviz
Amazon CloudWatch exposes a variety of useful metrics for EC2 instances, Elastic Load Balancers, and more. Unfortunately, it is tedious to query directly and the results can be difficult to interpret.
Like most operational metrics, CloudWatch data provides the most insight when graphed. While there are existing tools to graph CloudWatch data, they are only available as part of a proprietary suite or service and, generally, they sacrifice customization and flexibility for ease-of-use.
Here at Bizo, we wanted to incorporate CloudWatch data into operational dashboards. Nothing we found was flexible enough to meet our needs, so we decided to write our own. We are now releasing it to for all to use.
I'm pleased to introduce cloudviz, an open source tool for creating embeddable CloudWatch graphs.
Specifically, cloudviz is a data source that exposes CloudWatch data for graphing by Google Interactive Charts (formerly Visualization API). It's written in Python using Google's Data Source library and Mitch Garnaat's excellent AWS interface, boto.
With cloudviz, it's easy to create graphs like these:
I encourage you to check out the project on GitHub here. There's a fairly detailed README and plenty of examples, but feel free to drop me a line if you have any questions, mike@bizo.com.
Happy graphing!
Subscribe to:
Posts (Atom)