Friday, March 5, 2010

SSH to EC2 instance ID

I often find myself looking up EC2 nodes by instance ID so I can grab the external DNS name and SSH in. Fed up with the extra “ec2-describe-instance , copy, paste” layer, I threw together a function (basically a fancy alias) to SSH into an EC2 instance referenced by ID.

Assuming you’re on Mac OS X / Linux, just put this somewhere in ~/.profile, reload your terminal, and you’re good to go.

Alternatively, you can use the shell script version.


(note: cross-posted here)

Thursday, March 4, 2010

Example git/git-sh config

I've been using git, git-svn, and git-sh while working on Bizo's internal projects and really enjoying it. Per requests from some other devs, here is my git/git-sh config.

First, you should start with git-sh. It adds some bash shell customizations like a nice `PS1` prompt, tab completion, and incredibly short git-specific aliases. I'll cover some of the aliases later, but this is the thing that started me down the "how cool can I get my git environment" path.

I've included commented versions of my .gitconfig and .gitshrc below, but you can find raw versions here and here. I also cross-posted this on my personal blog if you're so inclined as to read it twice.

Example Shell Session



A lot of my customizations are around aliases, so this is a quick overview, and then the aliases are defined/explained below.

Here is a made up example bash session with some of the commands:


# show we're in a basic java/whatever project
$ ls
src/ tests/

# start git-sh to get into a git-specific bash environment
$ git sh

# change some things
$ echo "file1" > src/package1/file1
$ echo "file2" > src/package2/file2
$ echo "file3" > src/package3/file2

# see all of our changes
$ d
# runs: git diff

# see only the changes in package1
$ dg package1
# runs: git diff src/package1/file1

# stage any path with 'package' in it
$ ag package
# runs: git add src/package1/file1 src/package2/file2 src/package3/file3

# we only wanted package1, reset package2 and package3
$ rsg package2
# runs: git reset src/package2/file2
$ rsg package3
# runs: git reset src/package3/file3

# see what we have staged now (only package1)
$ p
# runs: git diff --cached

# commit it
$ commit -m "Changed stuff in package1"
# runs: git commit -m "..."


That is the basic idea.

Most of the magic is from the [alias] section of .gitconfig, along with my .gitshrc allowing the git prefix to be dropped.

.gitconfig



The .gitconfig file is in your home directory and is for user-wide settings.

Here is my current .gitconfig with comments:


[user]
name = Stephen Haberman
email = stephen@exigencecorp.com
[alias]
# 'add all' stages all new+changed+deleted files
aa = !git ls-files -d | xargs -r git rm && git ls-files -m -o --exclude-standard | xargs -r git add

# 'add grep' stages all new+changed that match $1
ag = "!sh -c 'git ls-files -m -o --exclude-standard | grep $1 | xargs -r git add' -"

# 'checkout grep' checkouts any files that match $1
cg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git checkout' -"

# 'diff grep' diffs any files that match $1
dg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git diff' -"

# 'patch grep' diff --cached any files that match $1
pg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git diff --cached' -"

# 'remove grep' remove any files that match $1
rmg = "!sh -c 'git ls-files -d | grep $1 | xargs -r git rm' -"

# 'reset grep' reset any files that match $1
rsg = "!sh -c 'git ls-files -c | grep $1 | xargs -r git reset' -"

# nice log output
lg = log --graph --pretty=oneline --abbrev-commit --decorate

# rerun svn show-ignore -> exclude
si = !git svn show-ignore > .git/info/exclude

# start git-sh
sh = !git-sh
[color]
# turn on color
diff = auto
status = auto
branch = auto
interactive = auto
ui = auto
[color "branch"]
# good looking colors i copy/pasted from somewhere
current = green bold
local = green
remote = red bold
[color "diff"]
# good looking colors i copy/pasted from somewhere
meta = yellow bold
frag = magenta bold
old = red bold
new = green bold
[color "status"]
# good looking colors i copy/pasted from somewhere
added = green bold
changed = yellow bold
untracked = red
[color "sh"]
branch = yellow
[core]
excludesfile = /home/stephen/.gitignore
# two-space tabs
pager = less -FXRS -x2
[push]
# 'git push' should only do the current branch, not all
default = current
[branch]
# always setup 'git pull' to rebase instead of merge
autosetuprebase = always
[diff]
renames = copies
mnemonicprefix = true
[svn]
# push empty directory removals back to svn at directory deletes
rmdir = true


.gitshrc



This is my .gitshrc file, heavily based off Ryan Tomayko's original.

Ryan's original comments are prefixed with #, I'll prefix my additions with ###, most of which are aliases to my [alias] entries above and some git-svn aliases.


#!/bin/bash
# rtomayko's ~/.gitshrc file
### With additions from stephenh

# git commit
gitalias commit='git commit --verbose'
gitalias amend='git commit --verbose --amend'
gitalias ci='git commit --verbose'
gitalias ca='git commit --verbose --all'
gitalias n='git commit --verbose --amend'

# git branch and remote
gitalias b='git branch -av' ### Added -av parameter
gitalias rv='git remote -v'

# git add
gitalias a='git add'
gitalias au='git add --update'
gitalias ap='git add --patch'
### Added entries for my .gitconfig aliases
alias aa='git aa' # add all updated/new/deleted
alias ag='git ag' # add with grep
alias agp='git agp' # add with grep -p
alias cg='git cg' # checkout with grep
alias dg='git dg' # diff with grep
alias pg='git pg' # patch with grep
alias rsg='git rsg' # reset with grep
alias rmg='git rmg' # remove with grep

# git checkout
gitalias c='git checkout'

# git fetch
gitalias f='git fetch'

# basic interactive rebase of last 10 commits
gitalias r='git rebase --interactive HEAD~10'
alias cont='git rebase --continue'

# git diff
gitalias d='git diff'
gitalias p='git diff --cached' # mnemonic: "patch"

# git ls-files
### Added o to list other files that aren't ignored
gitalias o='git ls-files -o --exclude-standard' # "other"

# git status
alias s='git status'

# git log
gitalias L='git log'
# gitalias l='git log --graph --pretty=oneline --abbrev-commit --decorate'
gitalias l="git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative"
gitalias ll='git log --pretty=oneline --abbrev-commit --max-count=15'

# misc
gitalias pick='git cherry-pick'

# experimental
gitalias mirror='git reset --hard'
gitalias stage='git add'
gitalias unstage='git reset HEAD'
gitalias pop='git reset --soft HEAD^'
gitalias review='git log -p --max-count=1'

### Added git svn asliases
gitalias si='git si' # update svn ignore > exclude
gitalias sr='git svn rebase'
gitalias sp='git svn dcommit'
gitalias sf='git svn fetch'

### Added call to git-wtf tool
gitalias wtf='git-wtf'


Since I defined most of the interesting aliases in the .gitconfig [alias] section, it means they're all usable via git xxx, e.g. git ag foo, but listing alias ag='git ag' in .gitshrc means you can also just use ag foo, assuming you've started the git-sh environment.

It results in some duplication, but means they're usable from both inside and outside of git-sh, which I think is useful.

Get Your Speed Tracer On!

I first saw Speed Tracer in action at Google I/O 2009 and was pretty amped about it. While we have been using GWT 2.0 features for a few months now (e.g. OOPHM, UiBinder, ClientBundle), I had not tried out Speed Tracer until tonight. Speed Tracer is a Chrome plugin that is as a web performance profiling tool on steroids. The level of profiling information that you can get from this tool is truly amazing. If you develop web apps then I highly recommend that you check it out. I guarantee it will be something you will want to have in your toolbox. Installation instructions can be found here.

Friday, February 19, 2010

Triggering post-Elastic MapReduce steps as parameterized jobs in Hudson

Here at Bizo, the combination of Hudson for cron management, Hive for report generation, and Elastic MapReduce for provisioning compute power has greatly simplified our data processing. Periodically and automatically, our Hudson cron instance generates Hive scripts for us and launches them in EC2.

The main inconvenience with this process is that the results of our Hive jobs are left as one or more obscurely named files in S3. These often need some post-processing to put them into a more friendly form. Unfortunately, EMR doesn't have an easy hook for launching these post-processing tasks -- although we could implement them as MapReduce steps, we'd need to write our own workflows, losing the simplicity of using EMR's simple "--hive-script" flag.

Our solution is to use SimpleDB to store some basic metadata about jobs. Using this metadata, a Hudson job periodically checks the EMR API to determine whether tasks have completed. If so, it then triggers other Hudson jobs that are responsible for processing the results.

Here are some tools that make this process work:

  • Simple script to put data into SimpleDB. Our metadata scheme is to use the jobflow ID as item names and the name/parameters of the jobs to trigger as attributes.

  • The Hudson parameterized build feature. It's not really feasible to create a new Hudson job for each individual report that runs, so we pass parameters to Hudson so the post-processing step can figure out where the results are in S3 and what to do with them. It's not well-documented how to do this programatically (as opposed to from the web interface); the solution is to send some JSON to the build url.

  • The Trigger Script. This is the script that periodically runs on our cron server to check if a post-processing step should be triggered. The JSON format for parameterized jobs is described in the comments of this file.



The end result is that a job can run an EMR job and configure a post-processing step for itself with the following commands:


JOB_ID=`elastic-mapreduce --create --hive-script --arg ${s3.location}` | grep "Created job flow" | awk '{ print $4 }' -`

simpledb-put.rb -d ${metadata.domain} -i $JOB_ID "next_on_cron_server_job_name=post-processing-step" "next_on_cron_server_job_params={\"parameter\": [{\"name\":\"PARAM1\", \"value\":\"VALUE1\" }]}" "next_on_cron_server_triggered=false"


This launches the hive script in the specified s3.location and configures the "post-processing-step" job on the cron server to run with the parameter "PARAM1=VALUE1".

Thursday, January 7, 2010

Scala Supports Non-Local Returns

Writing some Scala code today, I found myself using non-local returns without even thinking about it. After realizing what I had done, I dug a little deeper to see what was really going on.

Take this completely made up, nonsensical example:


object Foo {
def main(args: Array[String]) {
foo(List(1, 2, 3))
}

def foo(l: List[Int]): Int = {
l.foreach { (i) =>
println(i)
return 5
}
return 10
}
}


This code will print "1" and then exit.

Perhaps this is obvious, that the "return 5" applies to the "foo" method, so the values 2 and 3 in "l" will not have a chance to be printed.

However, think about what is going on under the covers--Scala is passing the foreach method an anonymous inner class with a "void apply(int i)" method. And inside of that "apply" method is the code between the "{ (i) => ... }".

So, how does code inside of the "apply" method cause its caller to perform an early exit, without the "foreach" even knowing about it?

Exceptions.

Here is the decompiled version of "print":


public int print(List l) {
Object localObject = new Object();
int exceptionResult1 = 0;
try {
l.foreach(new AbstractFunction1() {
public static final long serialVersionUID = 0L;

public final Nothing. apply(int i) {
Predef..MODULE$.println(BoxesRunTime.boxToInteger(i));
// here is the "return"--it puts "5" into an exception
throw new NonLocalReturnException(
this.nonLocalReturnKey1$1,
BoxesRunTime.boxToInteger(5));
}
});
return 10;
} catch (NonLocalReturnException localNonLocalReturnException) {
if (localNonLocalReturnException.key() == localObject) {
// get "5" back out of the exception
return BoxesRunTime.unboxToInt(localNonLocalReturnException.value());
}
throw localNonLocalReturnException;
}
}


I think the decompiler got a little confused with "nonLocalReturnKey", but you can see the basic idea is that any early return inside of a closure is converted into an exception that is then caught outside of the closure where a proper return call can be done.

I personally think this is handy, once you know what is going on. But from what I've picked up, any closures that make it into Java 7 will not support non-local returns and instead disallow the "return" keyword inside of closures. Which, I guess, at this point any Java closures are better than no closures at all.

Tuesday, December 15, 2009

amazon ec2 spot instances

Yesterday Amazon announced EC2 Spot Instances. The idea is that you can bid on unused EC2 instance time. The 'Spot price' is determined periodically by Amazon based on availability and demand for the instances. If your bid is higher than the spot price, you will get an instance and only pay the spot price. Of course, your instance may be terminated at any time, but the nice thing is that unlike the normal ec2 pricing, here you only pay for full hours of usage.

To check out the price history of small linux instances, download the new release of the ec2-api-tools and run:


ec2-describe-spot-price-history --instance-type m1.small -d Linux/UNIX -H


Running this last night, I saw prices that looked like (times PST):



It looks like there's a substantial discount here with prices ranging from $0.025 to $0.035 per hour (the normal ec2 price is $0.085/hr).

Since I'm in the middle of reading How to Cheat at Everything, one of my first thoughts was why not just bid say $0.10/hour? In this way, you're unlikely to get outbid, but you'll probably stand to save significantly for a large part of the day. Now I'm thinking this probably isn't quite a free market... If amazon needs capacity to satisfy reserved instances, or even regular ec2 instances, maybe they'll just kill off these machines to make room.

Still, this is really very cool. A great option for doing a lot of offline batch processing. I hope we start to see support for taking advantage of this type of model in hadoop. It's also exciting to think that one day maybe we'll see something like this across providers -- bid for time across amazon, sun, etc.

Update: Some nice charts by Tim Lossen at cloudexchange.org.

Friday, December 4, 2009

github spam?

I just happened to land on the github recent repositories page, and noticed a ton of spam:



A bunch of different users and projects advertising movie downloads. There's no project content, of course, just a "homepage" that points to a target url...

At first I was thinking, wow, these are some crazy spammers -- using git as a tool for spam! But on closer look, it seems like they're just hitting the website automating account signup and new repository actions.

Still, spam on github? Crazy! I guess no website is safe these days. If you're hosting user generated content, you need to think about detecting and blocking spam and automation of user activity.