tag:blogger.com,1999:blog-52610569071326405542024-03-01T10:57:22.235-08:00bizo developer bloglarry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.comBlogger133125tag:blogger.com,1999:blog-5261056907132640554.post-56887103164119609952013-08-26T13:44:00.000-07:002013-08-26T13:44:18.830-07:00SCM Migration<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We happily used Atlassian’s hosted OnDemand service for source code management with the following setup</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<ul>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Subversion: source control management</span></li>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">FishEye: source code browsing</span></li>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Crucible: code reviews</span></li>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Jenkins (hosted on EC2): continuous integration and periodic jobs (http://dev.bizo.com/2009/11/using-hudson-to-manage-crons.html)</span></li>
</ul>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">However, Atlassian is ending their OnDemand offering for source code management in October so it was time for a change. The good news: we were wanting to migrate to git anyway. The bad news: we had around hundreds projects in our subversion repository and needed to break them up into separate git repositories.</span></div>
<b id="docs-internal-guid--e254b12-bc4c-3a96-52b4-1117e73f568b" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We switched on a Thursday morning with minimal developer interruptions, now we're on a new setup</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<ul>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Bitbucket: source control management and code browsing</span></li>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Crucible (hosted on EC2): code reviews</span></li>
<li><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Jenkins (hosted on EC2): continuous integration and periodic jobs</span></li>
</ul>
<br />
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How'd we do it? Read on, my friend.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Problem</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Move hundreds of projects (some with differing branching structures) to an equivalent number of git repositories. And change hundreds of Jenkins job configurations from pulling code out of subversion to pulling code from git. And set up a new Crucible instance for code reviews for the hundreds of repos. All without disrupting the dev team's work. For subversion, this meant moving the code, including branches, and commit history from subversion into Bitbucket. For Jenkins, it meant changing the job configs to point at the equivalent git repository with the same code and branch as the old subversion configuration. This blog post focuses on the subversion to git migration. Fixing the Jenkins configs will be covered in a later blog post.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Subversion to Git</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Converting a single repository from subversion to git is fortunately straight forward due to the terrific tool git-svn (https://www.kernel.org/pub/software/scm/git/docs/git-svn.html).</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The challenging part was determining how each project configured branches. In subversion, branches are just another subdirectory the repository. Basically any level of the directory hierarchy can support branches. You can pretty much put them anywhere. Git, however, only supports branches at the root of the repository. Git-svn allows you to tell git what directory the branches are in, but first you have to find what directory that is.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our subversion repositories followed two primary branching structures: branch at the module level or branch at the project level.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One layout that I will call "module level". Module level projects had a separate branch point for each module in the project. These projects were usually several loosely connected modules that could be deployed separately or libraries that were related but could be imported independently. Module level projects looked like this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/trunk/<module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/trunk/<module2></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<module1>/<branch1_for_module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<module1>/<branch2_for_module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<module2>/<branch1_for_module2></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">"module level" projects mapped into a separate git repo for each module using this git-svn command:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt; text-indent: -9pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git svn clone <svn_root> --trunk <project>/trunk/<module> --branches <project>/branches/<module> --tags <project>/branches/<tags> <module></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The other branching structure I’ll call "project level". These projects also had multiple modules, but the branches were defined such that each branch contained the entire project. These projects were usually separate modules for the domain layer, application layer and web layer or closely related applications that use the same database. Parts could perhaps be deployed separately but they often need to be deployed at the same time such as when the database schema changed. Project level projects looked like this:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/trunk/<module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/trunk/<module2></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<branch1>/<module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<branch1>/<module2></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<branch2>/<module1></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- svn/<project>/branches/<branch2>/<module2></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">"project level" projects mapped into a single git repo containing all modules using a git-svn command:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt; text-indent: -9pt;">
<span style="background-color: transparent; color: black; font-family: 'Courier New'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git svn clone <svn_root> --trunk <project>/trunk --branches <project>/branches --tags <project>/branches <project></span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To automate the git-svn clones, I wrote a ruby script that used "svn ls" to find the list of all projects. Each project was assumed to be "module level" unless it was in a hard-coded list of known "project level" projects. It was important for this to be fully automated as the list of "project level" projects was not complete until near the end of the migration. It took several tries to make sure the migration was correct. Some projects unfortunately used both branching structures, which is not supported by git-svn. Some of these branches were abandoned anyway, but others were moved using "svn mv" to fit that project's standard branch structure.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Local Git to Bitbucket</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Atlassian provided a jar (https://go-dvcs.atlassian.com/display/aod/Migrating+from+Subversion+to+Git+on+Bitbucket) to push a git-svn repository up to Bitbucket. The jar also can create an authors file from the subversion repository to map a subversion user to the values git needs for a committer - first name, last name and email address. This made scripting the Bitbucket upload for each repository straightforward. The jar also handles syncs to an existing Bitbucket repository so developers could continue committing to their svn projects and Bitbucket would automatically get updated. Note this only does fast forward syncs so the incremental sync stops working once commits were made directly to Bitbucket.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Crucible</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Crucible is a tool to facilitate code reviews. It imports commits from your SCM tool, allows inline comments on the diffs and manages the code review life cycle of assigning reviewers, tracking who has approved the changes, and closing the review once approved. Crucible setup is fairly straightforward with a couple of caveats.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Crucible needs to access your repositories to pull in the commit history. There is no native support for pointing crucible at a Bitbucket team account and having Crucible automatically import each repository. There is an free add-on (https://marketplace.atlassian.com/plugins/com.atlassian.fecru.reposync.reposync) that works for an initial import, but initially it did not bring in new repositories that are added to the team account after the initial import. It turns out the update did not work because I was using a Bitbucket user that could not access the User list from the Bitbucket API. Changing the Bitbucket user to one with access to this API end point solves this problem. Incremental updates to the repository list are now working.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While Crucible supports ssh access to git repositories in general, I ran into the problem described here https://answers.atlassian.com/questions/34283/how-to-connect-to-bitbucket-from-fisheye. Basically, Crucible does not support Bitbucket's ssh URL format. Instead of using ssh, I had to use https to connect to the Bitbucket repositories. This means each repository configuration requires the Bitbucket username and password to be specified separately, which is not ideal.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Testing</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After running git-svn clone on a few projects, I went ahead and pulled all the projects down with git-svn. The distributed nature of git helped testing because the entire repository could be represented locally without needing to upload it to any server to test the initial clones. However, cloning all the repositories took about 24 hours. During this time there was minimal CPU and I/O load so I multithreaded the cloning jobs using 16 threads. This improved the time to just 1.5 hours on only a dual core machine.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I was initially hesitant to upload all the repositories to Bitbucket because I did not want to have to manually delete the repos if there was a problem. However, I found the Bitbucket REST API (https://confluence.atlassian.com/display/BITBUCKET/Use+the+Bitbucket+REST+APIs). It is pretty well put together and was easy to use because it generally follows REST conventions. I've yet to find anything that can be done in the UI that can not be done in the API, which has been outstanding for adding additional niceties like adding commit hooks to push changes to crucible for each repository. For the purposes of migration, the best feature was deleting repositories. Knowing I could automatically clean up any mistakes provided the confidence to just let it rip. I actually ended up using this to clean up two false start migrations:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">git-svn has a "show-ignore" command to translate files ignored by subversion into a .gitignore file. I initially added .gitignore to the git repositories. However, this meant every repository had a commit in Bitbucket and so would no longer accept changes from subversion. This was resolved by adding .gitignore to subversion before the conversion.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">the first authors file I created was missing a few users. This was not discovered by noticing the Bitbucket commit history did not look as nice. It was nice to be able to just wipe it all out with a single command, fix the authors file, and redo the upload with a single command.</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Post-migration</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The time following the initial migration was when the automation really came in handy. A couple developers were out of the office during the cut over. They were able to make commits of their local work to subversion and then I could re-sync just those repositories even after other developers had begun working on other repositories in Bitbucket. This went very smoothly with no hand wringing or diff patching required to make sure local work was not lost.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Wrap-up</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Overall the migration went off with no hiccups. We're still tweaking our preferred settings for git pushes and pulls to get to our ideal workflow, but we're happy to be using Bitbucket. Crucible does not integrate with Bitbucket as nicely as it did with subversion in our old setup. Hopefully Atlassian will continue to make improvements to this integration as we really like the Crucible code review workflow. I'm always impressed how automation begets automation. Once you've taken the step of automating part of the process, it is so much easier to see the next step. We are already seeing some benefits from the time spent interacting with the Bitbucket API as we're now able to add and modify commit hooks on all the repositories easily.</span></div>
<div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-38376159247526656042013-06-20T11:41:00.001-07:002013-06-20T11:53:36.026-07:00Using AWS Custom SSL Domain Names for CloudFrontAWS recently announced the limited availability of <a href="http://aws.typepad.com/aws/2013/06/custom-ssl-domain-names-root-domain-hosting-for-amazon-cloudfront.html">Custom SSL Domain Names for CloudFront</a>. You have to <a href="http://aws.typepad.com/aws/2013/06/custom-ssl-domain-names-root-domain-hosting-for-amazon-cloudfront.html">request an invitation</a> in order to start using it but I am guessing it won't be long until it has been rolled out to all customers.<br />
<br />
We've been asking/waiting for Custom SSL on CloudFront for years and were excited when it finally came out. The sign up was easy and we were approved a day or two later. <br />
<br />
<h3>
Existing Setup</h3>
Our main use case for Custom SSL on CloudFront involves replacing a service that proxies secure requests to our non-secure CloudFront distro. We proxy secure requests because we didn't want the secure CloudFront domain leaking out to our customers for various reasons including:<br />
<br />
<ul>
<li>We wanted to be able to point the domain elsewhere if we needed to</li>
<li>We wanted to keep our branding consistent on domains. </li>
</ul>
It basically looks like the following diagram:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhV_jI2teI6gcOUrdG2jEKn1hB8BvnjtB5UnxTRNoit-7bXiP6mPkohj1nD2ylqH5a-D5BaPabsP46HL6iEztApIM8o4O8MRTjgw-f2jqPZ1DY-g7q4SH85C40VTXfuDlTSHTykJtXkAAg/s1600/proxy.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhV_jI2teI6gcOUrdG2jEKn1hB8BvnjtB5UnxTRNoit-7bXiP6mPkohj1nD2ylqH5a-D5BaPabsP46HL6iEztApIM8o4O8MRTjgw-f2jqPZ1DY-g7q4SH85C40VTXfuDlTSHTykJtXkAAg/s640/proxy.jpg" width="640" /></a></div>
<br />
The problem with having a proxy is two fold:<br />
<br />
<ol>
<li>We have to operate that proxy which goes against our general rule to "never operate services when AWS can do it for you"</li>
<li>We get subpar performance relative since requests are no longer served from a distributed geo-located CDN.</li>
</ol>
<div>
But we needed the flexibility and branding mentioned above so we dealt with it. Not anymore...</div>
<div>
<br /></div>
<h3>
Migrating to Custom SSL Domain Names for CloudFront</h3>
<div>
Once we got approval for custom SSL, the migration was pretty straightforward. I am not going to regurgitate the <a href="http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecureConnections.html#CNAMEsAndHTTPS">detailed documentation</a> but will summarize the process.</div>
<div>
<ol>
<li>Upload your SSL cert and make sure path starts with "/cloudfront" (<i>This was annoying because we couldn't reuse our existing certificates that we were already using for ELBs</i>)</li>
<li>Update your CF distro (I did so via the AWS Console): </li>
<ol>
<li>add the domain name you want to support (e.g. secure-example.bizographics.com from above)</li>
<li>choose the SSL cert that you uploaded in the first step</li>
<li>Save</li>
</ol>
<li>Wait for the CF distro to redeploy the configuration change</li>
<li>Update your Route53 DNS to point at the CF CNAME rather than the ELB endpoint</li>
<li>Wait for DNS to Update</li>
<li>Shut down ELB of Proxy</li>
</ol>
<div>
As you can see this was pretty easy. Most of the time was spent waiting for the CF distro the re-deploy (10s of minutes max) and DNS to update (which can take several days). </div>
</div>
<div>
<br /></div>
<div>
All-in-all, the minor annoyance of having two copies of the same SSL cert was worth the win of not having to operate the proxy and getting better performance for our customers. Check out the graph below showing the improved performance:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFbq8yVkdIoodWLBmcIXOFma1VsbPrRiRpYvKh_VrhjkM6r-68tPD5xQTXRjQTB88KzEpRo4Oe6dT0iZy2tqR3kDRAyy-ztLyH6oJ8WONwDLbXYL6xjgoc5NnXKchO7zwqq2x_mcEviPI/s1600/response-time.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFbq8yVkdIoodWLBmcIXOFma1VsbPrRiRpYvKh_VrhjkM6r-68tPD5xQTXRjQTB88KzEpRo4Oe6dT0iZy2tqR3kDRAyy-ztLyH6oJ8WONwDLbXYL6xjgoc5NnXKchO7zwqq2x_mcEviPI/s1600/response-time.jpg" /></a></div>
<br /></div>
<h3>
Note on Cost</h3>
<div>
The <a href="http://aws.amazon.com/cloudfront/pricing/">cost of custom SSL on CF </a>seems ok but could be better and the wording is not totally clear: "<span style="background-color: white; font-family: verdana, arial, helvetica, clean, sans-serif; font-size: 12px; line-height: 18px;">You pay $600 per month for each custom SSL certificate associated with one or more CloudFront distributions." </span> We have the same cert setup for multiple CF distros but I am not sure if we will be charged $600 for each disto using the cert or $600 for each cert regardless of how many distros are using it. (Will try to get clarification...) AWS claims the pricing is comparable to other similar offerings. That doesn't seem to jive with their usual practice of driving costs much lower but is livable for now. </div>
<div>
<br /></div>
<br />
<br />Donniehttp://www.blogger.com/profile/13599133732419522440noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-8704754197086680022013-04-22T13:28:00.001-07:002013-09-16T11:18:21.922-07:00Scala Command-Line Hacks<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">Do you like command-line scripting and one-liners with <a href="http://www.unixguide.net/unix/perl_oneliners.shtml" target="_blank">Perl</a>, <a href="http://reference.jumpingmonkey.org/programming_languages/ruby/ruby-one-liners.html" target="_blank">Ruby</a> and the like? </span><br />
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
For instance, here's a Ruby one-liner that uppercases the input:</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">% echo matz | ruby -p -e '$_.tr! "a-z", "A-Z"'</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">MATZ</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br />
<i>You like that kind of stuff?</i> Yes? Excellent! Then I offer you a hacking idea for Scala.<br />
<br /></div>
<div style="background-color: white; color: #222222; font-size: 12.666666984558105px;">
<span style="font-family: arial, sans-serif;">As you may know, Scala offers similar capability with the </span><span style="font-family: Courier New, Courier, monospace;">-e</span><span style="font-family: arial, sans-serif;"> command-line option but it's fairly limited in its basic form because of the necessary boilerplate code to set up iteration over the standard input... it just begs for a simple HACK!</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
Using a simple bash wrapper,</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white;">
<div>
<br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#!/bin/bash</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"># Usage: scala-map MAP_CODE</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">code=$(cat <end font=""></end></span><span style="color: #222222; font-family: 'courier new', monospace; font-size: x-small;"><<end span=""></end></span><span style="color: #222222; font-family: 'courier new', monospace; font-size: x-small;"><END</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">scala.io.Source.stdin.getLines map { $@ } foreach println</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">END</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">)</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">scala -e "$code"</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"><br /></span>
</div>
</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
then we can express similar one-liners using Scala code and the standard library:</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">% ls | scala-map _.toUpperCase</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">FOO</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">BAR</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">BAZ</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">...</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">% echo "foo bar baz" | scala-map '_.split(" ").mkString("-")'</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<span style="font-family: courier new, monospace;">foo-bar-baz</span></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
Nifty, right? Here's another script template to fold over the standard input,</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white;">
<div>
<br />
<br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#!/bin/bash</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"># Usage: scala-fold INITIAL_VALUE FOLD_CODE</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"># where the following val's can be used in FOLD_CODE:</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"># `acc` is bound to the accumulator value</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"># `line` is bound to the current line</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">#</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">code=$(cat <<end end="" font=""></end></span><span style="color: #222222; font-family: 'courier new', monospace; font-size: x-small;"><END</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">println(scala.io.Source.stdin.getLines.foldLeft($1) { case (acc, line) => $2 })</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">END</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">)</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;">scala -e "$code"</span><br />
<span style="color: #222222; font-family: courier new, monospace; font-size: x-small;"><br /></span>
</div>
</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
Now if you wanted to calculate the sum of the second column of space-separated input, you'd write:</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<div>
<span style="font-family: courier new, monospace;">$ cat | scala-fold 0 'acc + (line.split(" ")(1).toInt)'</span></div>
<div>
<span style="font-family: courier new, monospace;">foo 1</span></div>
<div>
<span style="font-family: courier new, monospace;">bar 2</span></div>
<div>
<span style="font-family: courier new, monospace;">baz 3 </span><br />
<span style="font-family: courier new, monospace;">(CTRL-D)</span></div>
<div>
<span style="font-family: courier new, monospace;">6</span></div>
</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.666666984558105px;">
You get the idea ... hopefully this inspires you to try a few things with Scala scripting templates!<br />
<br />
<b><i>Disclaimer: I am not advocating these hacks as replacement to learning other <a href="http://shop.oreilly.com/product/9780596003302.do" target="_blank">Unix power tools</a> like grep, sed, awk, ... I am simply illustrating that Scala can be turned into an effective command-line scripting tool as part of the vast array of Unix tools. Use what works best for you.</i></b></div>
Anonymousnoreply@blogger.com2tag:blogger.com,1999:blog-5261056907132640554.post-67177824913035812013-04-19T11:54:00.003-07:002013-04-19T14:53:50.008-07:00Efficiency & Scalability<b id="internal-source-marker_0.9677338404580951" style="font-weight: normal; line-height: 1.15;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Software engineers know that distributed systems are often hard to scale and many can intuitively point to reasons why this is the case by bringing up points of contention, bottlenecks and latency-inducing operations. Indeed, there exists a plethora of reasons and explanations as to why most distributed systems are inherently hard to scale, from the </span><a href="http://en.wikipedia.org/wiki/CAP_theorem" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CAP theorem </span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">to scarcity of certain resources, e.g., RAM, network bandwidth, ...</span></b><b id="internal-source-marker_0.9677338404580951" style="font-weight: normal;"></b><br />
<b id="internal-source-marker_0.9677338404580951" style="font-weight: normal;"><br style="line-height: normal;" /><span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="internal-source-marker_0.9677338404580951" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">It's said that </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">good engineers </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">know how to identify resources that may not appear to be relevant to scaling initially but will become more significant as particular kinds of demand grow. If that’s the case, then </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">great engineers </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">know that system architecture is often the determining factor in system scalability </span></b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="font-weight: normal; line-height: 1.15;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> that a system’s own architecture may be its worse enemy <b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px; white-space: normal;">—</b> so they define and structure systems in order avoid fundamental flaws.</span></b></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In this post, I want to explore the relationship between system efficiency and scalability in distributed systems; </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">they are to some extent two sides of the same coin. We’ll consider specifically two common system architecture traits: </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">replication </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">and </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">routing</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Some of this may seem obvious to some of you but it’s always good to back intuition with some additional reasoning.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Before we go any further, it’s helpful to formulate a definition of efficiency applicable to our context:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">efficiency</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is the extent to which <i>useful</i> work is performed relative to the total work and/or cost incurred.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We’ll also use the following definition of scalability,</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">scalability</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is the ability of a system to accommodate an increased workload by repeatedly applying a cost-effective strategy for extending a system’s capacity.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">So, scalability and efficiency are both determined by cost-effectiveness with the distinction that scalability is a measure of marginal gain. Stated differently, </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">if efficiency decreases significantly as a system grows, then a system is said to be non-scalable.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.15;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Enough rambling, let’s get our thinking caps on! Since we’re talking about distributed systems, it’s practically inevitable to compare against traditional single-computer systems, so we’ll start with a narrow definition of system efficiency:</span></div>
<div style="line-height: 1.15;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<span style="font-family: Courier New, Courier, monospace;"><b><span style="line-height: 17px; vertical-align: baseline; white-space: pre-wrap;"> average work for processing a request on a single computer
Efficiency = </span></b></span><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><br />
<b style="font-family: 'Courier New', Courier, monospace;"><span style="line-height: 17px; vertical-align: baseline; white-space: pre-wrap;"> average work for processing a request in distributed system</span></b><br />
<div style="line-height: 1.15;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div style="line-height: 1.15;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<div dir="ltr" style="display: inline !important; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
</div>
<b style="font-weight: normal; line-height: 1.15;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This definition is a useful starting point for our exploration because it abstracts out the nature of the processing that’s happening within the system; it’s overly simple but it allows us to focus our attention on the big picture.</span></b><br />
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">More succinctly, we’ll write:</span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><b>(1) Efficiency = Wsingle / Wcluster</b></span></span></div>
<b><span style="font-family: Arial; font-size: 15px; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<h2 dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 4pt; margin-top: 18pt;">
<span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Replication Cost</span></h2>
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Many distributed systems replicate some or all of the data they process across different processing </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">nodes</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (to increase reliability, availability or read performance) so we can model:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(2) Wcluster = Wsingle + (r x Wreplication)</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">where </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">r</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is the number of </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">replicas</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> in the system and <i>Wreplication</i> is the work required to replicate the data to other nodes. <i>Wreplication</i> is typically lower than <i>Wsingle</i>, though realistically they have different cost models (e.g., <i>Wsingle</i> may be CPU-intensive whereas <i>Wreplication</i> may be I/O-intensive). If </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">n</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> is the number of nodes in the system, then </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">r</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> may be as large as (n-1), meaning replicating to all other nodes, though most systems will only replicate to 2 or 3 other nodes </span><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> for good reason </span><b style="background-color: white; font-family: sans-serif; font-size: 12.800000190734863px; line-height: 19.200000762939453px;">—</b><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> as we’ll discover later.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We’ll now define the replication coeffient, which expresses the relative cost of replication compared to the cost of processing the request on a single node:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(3) Qreplication = Wreplication / Wsingle</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Solving for <i>Qreplication</i>, we get:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(4) Wreplication = Qreplication x Wsingle</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If we substitute <i>Wreplication</i> in (2) by the equation formulated in (4), we obtain:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><b>(5) Wcluster = Wsingle x [ 1 + ( r x Qreplication * Wsingle ) ]</b></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We now factor out <i>Wsingleon</i> the left side:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(6) Wcluster = Wsingle x [ 1 + r * Qreplication ]</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Taking the efficiency equation (1) and substituting <i>Wcluster</i> from (6), the equation becomes:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(7) Efficiency = Wsingle / [ Wsingle x ( 1 + r * Qreplication ]</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We then simplify <i>Wsingle</i> to obtain the final efficiency for a replicating distributed system:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><i><b>(8) Efficiency (replication) = 1 / [ 1 + (r x Qreplication) ]</b></i></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">As expected, both </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">r </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">and <i>Qreplication</i> are critical factors determining efficiency.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Interpreting this last equation and assuming <i>Qreplication</i> is a constant inherent to the system’s processing, our two takeaways are:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><i style="line-height: normal;"></i><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;"><span style="line-height: normal;">
<li dir="ltr" style="font-style: italic; list-style-type: decimal; vertical-align: baseline;"><div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">If the system replicates to all other nodes (i.e., </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">r = n - 1) </span><span style="vertical-align: baseline; white-space: pre-wrap;">it becomes clear that the efficiency of the system will degrade as more nodes are added and will approach zero as </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">n </span><span style="vertical-align: baseline; white-space: pre-wrap;">becomes sufficiently large.</span><span style="vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;">To illustrate this, let's assume </span>Qreplication<span style="font-style: normal;"> is 10%,</span></span><i style="font-style: normal; line-height: normal;"></i></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="font-family: Arial; font-size: 15px; font-style: normal;">
</div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 1, n = 2) = 91%</span></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 2, n = 3) = 83%</span></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 4) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 4, n = 5) = 71%</span></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 5, n = 6) = 67%</span></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">...</span></span></div>
<div dir="ltr" style="font-family: Arial; font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;">In other words, <b>fully-replicated distributed systems don't scale.</b></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="white-space: pre-wrap;"><br /></span></div>
</li>
<li dir="ltr" style="font-style: italic; list-style-type: decimal; vertical-align: baseline;"><div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;">For a system to scale, the replication factor should be a (small) constant.</span><span style="font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;">Let's illustrate this with </span>Qreplication<span style="font-style: normal;"> fixed at 10% and using a replication factor of 3,</span></span></div>
<div dir="ltr" style="font-family: Arial; font-size: 15px; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 4) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 5) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 6) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 7) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">Efficiency (r = 3, n = 8) = 76%</span></span></div>
<div dir="ltr" style="font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;">...</span></span></div>
<div dir="ltr" style="font-family: Arial; font-style: normal; font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="font-style: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">As we can see, <b>fixed-replication-factor distributed systems scale</b> </span></span><span style="background-color: white; line-height: 19.200000762939453px;">—</span><span style="font-family: Arial, Helvetica, sans-serif; line-height: normal; white-space: pre-wrap;"> although, as you might expect, they do not exhibit the same efficiency as a single-node system. At worse, the efficiency will be 1/r </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 19.200000762939453px;">—</span><span style="font-family: Arial, Helvetica, sans-serif; line-height: normal; white-space: pre-wrap;"> as you would intuitively expect.</span></div>
</li>
</span></ol>
<i style="line-height: normal;">
</i>
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<h2 dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 4pt; margin-top: 18pt;">
<span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Routing Cost</span></h2>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">When a distributed system routes requests to nodes holding the relevant information (e.g., a partially replicated system, <i>r</i> < <i>n</i>) its working model may be defined as,</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><i><span style="font-family: Courier New, Courier, monospace;"><b>(9) Wcluster = (r / n) * Wsingle + (n-r)/n * (Wrouting + Wsingle)</b></span></i></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The above equation represents the fact that </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">r</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> out of </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">n </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">requests are processed locally whereas the remainer of the requests are routed and processed on a different node.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Let’s define the routing coefficient to be,</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><b>(10) Qrouting = Wrouting / Wsingle</b></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Solving for <i>Wrouting</i> in (9) by (11) to obtain,</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><b>(12) Wcluster = (r/n) * Wsingle + (n-r)/n * [ (Qrouting * Wsingle) + Wsingle ]</b></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">and taking the efficiency equation (1), substituting <i>Wcluster</i> from (12), the simplified equation becomes:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Courier New, Courier, monospace;"><b>(13) Efficiency (routing) = n / [ n + (n - r) * Qrouting ]</b></span></span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Looking at this last equation, we can infer that:</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><i style="font-weight: normal; line-height: normal;"></i><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;"><i style="font-weight: normal; line-height: normal;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;">As the system grows and n goes towards infinity, the efficiency of the system can be expressed as 1 / (1 + </span>Qrouting<span style="font-style: normal;">). The efficiency is not dependent on the actual number of nodes within the system therefore </span></span><span style="font-style: normal; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">routing-based systems generally scale.</span><span style="font-style: normal; vertical-align: baseline; white-space: pre-wrap;">(But you knew that already)
</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;">If the number of nodes is large compared to the replication factor (n >> r) and </span>Qrouting<span style="font-style: normal;"> is significant (1.0, same cost as </span>Wsingle<span style="font-style: normal;">), then the efficiency is ½, or 50%. This matches the intuition that the system is routing practically all requests and therefore spending half of its efforts on routing. The system is scaling linearly but it’s costing twice as much to operate (for every node) compared to a single-node system.
</span></span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;">If the cost of routing is insignificant (</span>Qrouting<span style="font-style: normal;"> = 0), the efficiency is 100%. That’s right, if it doesn’t cost anything to route the request to a node that can process it, the efficiency is the same as a single-node system.
</span></span><br />
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-style: normal;"><br /></span></span></div>
</li>
</i></ol>
<i style="font-weight: normal; line-height: normal;">
</i>
<br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Let’s consider a practical distributed system with 10 nodes (n = 10), a replication factor of 3 (r = 3), and a relative routing cost of 10% (<i>Qrouting</i> = 0.10). This system would have an efficiency of 10 / 10 + (7 * 10%) = 93.46%. As you can see, routing-based distributed systems can be pretty efficient if Qrouting is relatively small.</span></div>
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<h2 dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 4pt; margin-top: 18pt;">
<span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Where To Now?</span></h2>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Well, this was a fun exploration of system scalability in the abstract. We came up with interesting equations to describe the scalabilty of both data-replicating and request-routing architectures. With some thinkering, these can serve as a good basis for reasoning about some of your distributed systems.</span></div>
<br style="line-height: normal;" />
<span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: normal; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In real life, however, there are many other aspects to consider when scaling systems. In fact, it often feels like a whack-a-mole hunt; you never know there the next performance non-linearity is going to rear its ugly head. But if you use either (or both) the data-replicating and request-routing style architecture with reasonable replication factors and you manage to keep your replication/routing costs well below your single-node processing costs, you may find some comfort in knowing that at least you haven’t introduced a fundamental scaling limitation unto your system.</span></div>
<div dir="ltr" style="font-weight: normal; line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; vertical-align: baseline;"><i><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;">PS: With </span><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;">apologies</span><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> for the formatting of the formulas ... Blogger wasn't exactly friendly with my equations imported from Google Docs so I had to go down the ASCII route. Thanks for reading and making it through!</span></i></span></div>
<div style="font-weight: normal; line-height: 1.15;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<br />Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-5640325334397794482013-04-15T07:59:00.000-07:002013-04-15T07:59:57.586-07:00<b id="internal-source-marker_0.8663988301996142" style="font-weight: normal;"></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="internal-source-marker_0.8663988301996142" style="font-weight: normal;"><span style="font-family: Arial; font-size: 24px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Sensible Defaults for Apache HttpClient</span></b></div>
<b id="internal-source-marker_0.8663988301996142" style="font-weight: normal;"><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 19px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Defaults for HttpClient</span></div>
<br /><span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Before coming to Bizo, I wrote a web service client that retrieved daily XML reports over HTTP using the Apache </span><a href="http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpClient.html" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DefaultHttpClient</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Everything went fine until one day the connection simply hung forever. We found this odd because we had set the connection timeout. It turned out we also needed to set the socket timeout (HttpConnectionParams.SO_TIMEOUT). The default for both connection timeout (max time to wait for a connection) and socket timeout (max time to wait between consecutive data packets) is infinity. The server was accepting the connection but then not sending any data so our client hung forever without even reporting any errors. Rookie mistake, but everyone is a rookie at least once. Even if you are an expert with HttpClient, chances are there will be someone maintaining your code in the future who is not.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Another problem with defaults using HttpClient is with </span><a href="http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/PoolingClientConnectionManager.html" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PoolingClientConnectionManager</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. PoolingClientConnectionManager has two attributes: MaxTotal and MaxPerRoute. MaxTotal is the maximum total number of connections in the pool. MaxPerRoute is the maximum number of connections to a particular host. If the client attempts to make a request and either of these maximums have been reached, then by default the client will block until a connection is free. Unfortunately the default for MaxTotal is 20 and the default MaxPerRoute is only 2. In a SOA, it is common to have many connections from a client to a particular host. The limit of 2 (or even 1) connections per host makes sense for a polite web crawler, but in a SOA, you are likely going to need a lot more. Even the 20 maximum total connections in the pool is likely much lower than desired.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If the client does reach the MaxPerRoute or the MaxTotal connections, it will block until the connection manager timeout (ClientPNames.CONN_MANAGER_TIMEOUT) is reached. This timeout controls how long the client will wait for a connection from the connection manager. Fortunately, if this timeout is not set directly, it will default to the connection timeout if that is set, which will prevent the client from queuing up requests indefinitely.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 19px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What would a better set of defaults be?</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">A good default is something that is "safe". A safe default for a connection timeout is long enough to not give up waiting when things are working normally, but short enough to not cause system instability when the is down. Unfortunately safe is context dependent. Safe for a daily data sync process and safe for an in thread service request handler are very different. Safe for a request that is critical to the correct functioning of the program is different than safe for a some ancillary logging that is ok to miss 1% of the time. A default for timeouts that is safe in all cases is not really possible.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Safe defaults for PoolingClientConnectionManager's MaxTotal and MaxPerRoute should be big enough that they won’t be hit unless there is a bug. New to version 4.2 is the </span><a href="http://hc.apache.org/httpcomponents-client-ga/fluent-hc/index.html" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">fluent-hc</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> API for making http requests. This uses a PoolingClientConnectionManager with defaults of 200 MaxTotal and 100 MaxPerRoute. We are using these same defaults for all our configurations.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Note that the fluent-hc API is very nice, but requires setting the connection timeouts on each request. This is perfect if you need to tune the settings for each request but does not provide a safety check against accidentally leaving the timeout infinite.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 19px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">How can you help out a new dev implementing a new HTTP client?</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If you can't have a safe default and the existing defaults are decidedly not safe, then it is best to require a configuration. We created a wrapper for PoolingClientConnectionManager that requires the developer to choose a configuration instead of letting the defaults silently take effect. One way to require a configuration is to force passing in the timeout values. However, it can be a hard to know the right values especially when stepping into a new environment. To help a developer implementing a new client at Bizo, we created some canonical configurations in the wrapper based on our experience working in our production environment on AWS. The configurations are:</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr">
<table style="border-collapse: collapse; border: none;"><colgroup><col width="239"></col><col width="101"></col><col width="97"></col><col width="111"></col><col width="124"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Configuration</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Connection timeout</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Socket timeout</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">MaxTotal</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">MaxPerRoute</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">SameRegion</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">125 ms</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">125 ms</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">200</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">100</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">SameRegionWithUSEastFailover</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">1 second</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">1 second</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">200</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">100</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">CrossRegion</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">10 seconds</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">10 seconds</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">200</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">100</span></div>
</td></tr>
<tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">MaxTimeout</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">1 minute</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">5 minutes</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">200</span></div>
</td><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">100</span></div>
</td></tr>
</tbody></table>
</div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Clients with critical latency requirements can use the SameRegion configuration and need to make sure they are connecting to a service in the same AWS region. Back end processes that can tolerate latency can use the MaxTimeout configuration. Now when a developer is implementing a new client, the timeouts used by other services are readily available without having to hunt through other code bases. The developer can compare these with the current use case and choose an appropriate configuration. Additionally, if we learn that some of these configurations need to be tweaked, then we can easily modify all affected code.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Commonly the socket timeout will need to be adjusted for a specific service. After a connection is established, a service will not typically start sending its response until it has finished whatever calculation was requested. This can vary greatly even for different parameters on the same service endpoint. The socket timeout will need to be set based on the expected response times of the service.</span></div>
<br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">It is easy to miss a particular setting even if you know it is there. At Bizo, we are always looking for ways to solve a problem in one place. We are hopeful that this will eliminate any issues we have had with bad defaults in our HttpClients.</span></div>
<div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
</b>Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-80584953458574104692013-02-18T14:07:00.000-08:002013-02-18T14:07:30.471-08:00Map-side aggregations in Apache Hive<b id="internal-source-marker_0.5484890048392117" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">When running large scale Hive reports, one error we occasionally run into is the following:</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;">
<b id="internal-source-marker_0.5484890048392117" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Possible error:<br class="kix-line-break" /> Out of memory due to hash maps used in map-side aggregation.<br class="kix-line-break" /><br class="kix-line-break" />Solution:<br class="kix-line-break" /> Currently hive.map.aggr.hash.percentmemory is set to 0.5. Try setting it to a lower value. i.e 'set hive.map.aggr.hash.percentmemory = 0.25;'</span></b></div>
<b id="internal-source-marker_0.5484890048392117" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">What’s going on is that Hive is trying to optimize the query by performing a map-side aggregation. This is a map-side optimization that does a partial aggregation inside of the mapper, which results in the mapper outputting fewer rows. In turn, this reduces the amount of information that Hadoop needs to sort and distribute to the reducers.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Let’s think about what the Hadoop job looks like with the canonical word count example.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In the word count example, the naive approach is for the mapper to tokenize each row of input and output the key-value pair (#{token}, 1). The Hadoop framework will sort these pairs by the tokens, and the reducer sums the values to produce the total counts for each token.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Using a map-side aggregation, the mappers would instead tokenize each row and store partial counts in an in-memory hash map. (More precisely, the mappers are storing each key with the corresponding partial aggregation, which is just a count in this case.) Periodically, the mappers will output the pairs (#{token}, #{token_count}). The Hadoop framework again sorts these pairs and the reducers sum the values to produce the total counts for each token. In this case, the mappers will each output one row for each token every time the map is flushed instead of one row for each occurrence of each token. The tradeoff is that they need to keep a map of all tokens in memory.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">By default, Hive will try to use the map-side aggregation optimization, but it falls back to the standard approach if the hash map is not producing enough of a memory savings. After processing 100,000 rows (modifiable via </span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">hive.groupby.mapaggr.checkinterval</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">), Hive will check the number of items in the hash map. If it exceeds 50% (modifiable via </span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">hive.map.aggr.hash.min.reduction</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">) of the number of rows read, the map-side aggregation will be aborted.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Hive will also estimate the amount of memory needed for each entry in the hash map and flush the map to the reducers whenever the size of the map exceeds 50% of the available mapper memory (modifiable via </span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">hive.map.aggr.hash.percentmemory</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">). This, however, is an estimate based on the number of rows and the expected size of each row, so if the memory usage is per row is unexpectedly high, the mappers may run out of memory before the hash map is flushed to the reducers.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In particular, if a query uses a count distinct aggregation, the partial aggregations actually contain a list of all values seen. As more distinct values are seen, the amount of memory used by the map will increase without necessarily increasing the number of rows of the map, which is what Hive uses to determine when to flush the partial aggregations to the reducers.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Whenever a mapper runs out of memory, a group by clause is present, and map-side aggregation is turned on, Hive will helpfully suggest that you reduce the flush threshold to avoid running out of memory. This will lower the threshold (in rows) of when Hive will automatically flush the map, but it may not help if the map size (in bytes) is growing independently of the number of rows.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Some alternate solutions include simply turning off map-side aggregations (</span><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">set hive.map.aggr = false</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">), allocating more memory to your mappers via the Hadoop configuration, or restructuring the query so that Hive will pick a different query plan.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">For example, a simple </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> select count(distinct v) from tbl</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">can be rewritten as </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> select count(1) from (select v from tbl group by v) t</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This latter query will avoid using the count distinct aggregation and may be more efficient for some queries.</span></b>Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-5261056907132640554.post-17662464327514375582013-02-15T09:10:00.001-08:002013-02-15T09:17:38.541-08:00Reader Driven DevelopmentIn this talk on <a href="http://vimeo.com/14313378">Effective ML</a>, <a href="http://cufp.org/users/yminsky">Yaron Minsky</a> talks about Reader Driven Development. That is, writing your code with the reader in mind. Making decisions that will make the code more easily read and understood by other developers down the line.
<blockquote>
The interest of the reader always pushes in the direction of clarity, simplicity, and the ability to change the code later.
In most real projects, code is read and changed many more times than it is written. The readers interest are paramount in that regard.
</blockquote>
When writing code the interests of the reader and writer may be at odds, and when faced with a decision, always err in the direction of the reader. The reader is always right.
Regardless of team size, it's helpful to program this way. Even code you've written yourself may not be as clear 6 months or a year later otherwise.
Great perspective, and I think it fits in nicely with previous posts here on <a href="http://dev.bizo.com/2012/06/golden-rule-of-programming-style.html">programming style</a> and <a href="http://dev.bizo.com/2012/03/on-code-reviews-and-developer-feedback.html">code reviews</a> (tend to agree with your reviewers, they are the audience!).larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-47122413195873390222013-01-23T15:03:00.000-08:002013-01-23T15:03:27.691-08:00Asanban: Lean Development with Asana and Kanban<br />
On Bizo's External Apps team (aka. 'xapps'), we've been using a Kanban system to manage our work. All of Bizo Engineering uses Asana to track tasks, which isn't specifically designed for Kanban. We've settled on a set of of conventions that we use in Asana which enable our Kanban system. These conventions also help us to track metrics like the average lead time from month to month.<br />
<h2>
Background</h2>
<a href="http://en.wikipedia.org/wiki/Kanban">Kanban</a> is a second-generation Agile software development methodology. The focus is on finding and fixing bottlenecks, as well as removing waste by limiting work-in-progress. (The "WIP" limits referenced in this post are the number of work items that are allowed to be in a particular stage of the system at one time.) Adopting a Kanban system has made things easier for engineers, increased efficiency, and is very popular with our Product Management folks as well. We are now focused on delivering value incrementally rather than specifying and implementing larger chunks of work. If you're interested in adopting Kanban, I recommend reading David Anderson's seminal book on the topic: <a href="http://amzn.com/0984521402">Kanban: Successful Evolutionary Change for Your Technology Business</a><br />
<h2>
Conventions</h2>
Each stage of work in our value chain is a priority heading in Asana. The name of the priority header follows the convention: "{STEP NAME} ({WIP}):", eg. "Dev Ready (10):". The steps that are earliest in the value chain are at the bottom of our Asana project, with tasks moving upwards through each stage until they reach "Production (15):" at the top when the functionality described by the task has been delivered to production. Once Product Management has verified that the functionality described by a task is functioning correctly in production, they mark the task as complete. We use tags to represent work item types, although its fairly limited at present.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-uK37WexeuU0/UQBrYTufwFI/AAAAAAAAAGA/YIQyJ4At83Y/s1600/Screen+Shot+2013-01-23+at+2.59.18+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="193" src="http://2.bp.blogspot.com/-uK37WexeuU0/UQBrYTufwFI/AAAAAAAAAGA/YIQyJ4At83Y/s320/Screen+Shot+2013-01-23+at+2.59.18+PM.png" width="320" /></a></div>
<br />
<h2>
Metrics</h2>
One of the most basic metrics to track in a Kanban system is the average amount of lead time (the time it takes from when a task gets added to the input queue until value is delivered). I have created some tooling that allows us to accomplish this systematically. I'll first describe what it does, and then how you can use it.<br />
<br />
The first piece of the tooling bulk loads task data from the Asana API into MongoDB. The API returns JSON and I just store JSON as-is in MongoDB, which works out well since MongoDB speaks JSON natively. One hiccup is that once tasks are archived in Asana, you can no longer obtain information about them through the API. Accordingly, the bulk load needs to be scheduled to run on a regular basis (in our case, every night) so that we don't lose information about archived tasks. Furthermore, we have a policy that tasks should not be archived until they have been completed for at least 24 hours, so that the bulk loader will always run at least once after a task has been completed before it gets archived. After loading the task data, the bulk loader will create data describing how much time each task spent in each state, as well as how long each task took (in days) to complete from start to finish (lead time).<br />
<br />
The other piece is a Sinatra web service that runs a map-reduce against the lead time data created by the bulk loader and serves lead times by month as JSON. It can also aggregate by year or day (but I don't think aggregating by day is useful).<br />
<div>
<br /></div>
<div>
<div>
I have packaged up both of those pieces into a gem called "asanban", which you can use. The source code and instructions for installation and usage are here: <a href="https://github.com/patgannon/asanban">https://github.com/patgannon/asanban</a></div>
</div>
<div>
<h2>
Pain Points</h2>
<div>
There are a couple of problems I've run into using Asana with a Kanban system. The first is that there's no way to enforce WIP limits. Users just have to be mindful of the limits shown in the priority headers. I have been thinking about writing a nightly report that uses the data created by the bulk loader to find violated WIP limits and send out emails, but I haven't gotten to it yet. (This tooling is essentially a hack day project at this point.) There is also no functionality to facilitate different classes of service (SLAs and WIP break-downs), but maybe those could be supported using the same kind of nightly report.</div>
<div>
<br /></div>
<div>
Another problem I've run into is that task sizes can be all over the place, which reduces the meaningfulness of the metrics. Some Kanban practitioners use hierarchical work items to address this kind variability in size. Stories can be grouped into epics and/or broken down into "grains". Asana does support sub-tasks, so I may recommend that we use those to break down large work items in the future, at which point the bulk loader would be modified to track metrics by sub-task (for tasks that have them, which would be assumed to be epics).</div>
<h2>
Next Steps</h2>
<div>
As we fine tune our Kanban process, we'll use these lead time metrics to verify that when we've made an adjustment (changing a WIP limit or adding a buffer, for example), that our performance improves. I'd like to have more metrics so that we can have even better insight into our system in the future. For example, I'd like to see the average time tasks spend in particular steps, the average amount of total WIP, as well as WIP in each step (shown over time) and failure load.</div>
<div>
<br /></div>
<div>
The first order of business moving forward on this will probably be an improved charting interface on top of the existing metrics. Also, it would be nice if the bulk loader used a scheduling library so that folks don't have to manually schedule it in cron. It also could use some automated tests!!! As I mentioned previously, I've just been working on this on hack days, so if there's something you'd like to see done soon, well... pull requests will be gladly accepted! :)</div>
</div>
<div>
<br /></div>
Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-4121670593228708592013-01-21T12:03:00.001-08:002013-01-21T12:03:22.766-08:00What Makes Spark Exciting
<p>At <a href='http://www.bizo.com'>Bizo</a>, we’re currently evaluating/prototyping <a href='http://www.spark-project.org'>Spark</a> as a replacement for <a href='http://hive.apache.org/'>Hive</a> for our batch reports.</p>
<p>As a brief intro, Spark is an alternative to Hadoop. It provides a cluster computing framework for running distributed jobs. Similar to Hadoop, you provide Spark with jobs to run, and it handles splitting up the job into small tasks, assigning those tasks to machines (optionally with Hadoop-style data locality), issuing retries if tasks fail transiently, etc.</p>
<p>In our case, these jobs are processing a non-trivial amount of data (log files) on a regular basis, for which we currently use Hive.</p>
<h2 id='why_replace_hive'>Why Replace Hive?</h2>
<p>Admittedly, Hive has served us well for quite awhile now. (One of our engineers even built a custom “Hadoop on demand” framework for running periodic on-demand Hadoop/Hive jobs in EC2 several months before <a href='http://aws.amazon.com/elasticmapreduce/'>Amazon Elastic Map Reduce</a> came out.)</p>
<p>Without Hive, it would have been hard for us to provide the same functionality, probably at all, let alone in the same time frame.</p>
<p>That said, it has gotten to the point where Hive is more frequently invoked in negative contexts (“damn it, Hive”) than positive.</p>
<p>Personally, I admittedly even try to avoid tasks that involve working with Hive. I find it to be frustrating and, well, just not a lot of fun. Why? Two primary reasons:</p>
<b id='1_hive_jobs_are_hard_to_test'>1. Hive jobs are hard to test</b>
<p>Bizo has a culture of excellence, and for engineering one of the things this means is testing. We really like tests. Especially unit tests, which are quick to run and enable a fast TDD cycle.</p>
<p>Unfortunately, Hive makes unit testing basically impossible. For several reasons:</p>
<ul>
<li>
<p>Hive scripts must be run in a local Hadoop/Hive installation.</p>
<p>Ironically, very few developers at Bizo have local Hadoop installations. We are admittedly spoiled by Elastic Map Reduce, such that most of us (myself anyway) wouldn’t even know how to setup Hadoop off the top of our heads. We just fire up an EMR cluster.</p>
</li>
<li>
<p>Hive scripts have production locations embedded in them.</p>
<p>Both our log files and report output are stored in S3, so our Hive scripts end up with lots of “s3://” paths scattered throughout in them.</p>
<p>While we do run dev versions of reports with “-dev” S3 buckets, still relying on S3 and raw log files (that are usually in a compressed/binary-ish format) is not conducive to setting up lots of really small, simplified scenarios to unit test each boundary case.</p>
</li>
<li>
<p>Hive scripts do not provide any abstraction–they are just one big HiveQL file. This means its hard to break up a large report into small, individually testable steps.</p>
</li>
</ul>
<p>Despite these limitations, about a year ago we had a developer dedicate some effort to prototyping an approach that would run Hive scripts within our CI workflow. In the end, while his prototype worked, the workflow was wonky enough that we never adopted it for production projects.</p>
<p>The result? Our Hive reports are basically untested. This sucks.</p>
<b id='2_hive_is_hard_to_extend'>2. Hive is hard to extend</b>
<p>Extending Hive via custom functions (UDFs and UDAFs) is possible, and we do it all the time–but it’s a pain in the ass.</p>
<p>Perhaps this is not Hive’s fault, and it’s some Hadoop internals leaking into Hive, but the various <a href='http://hive.apache.org/docs/r0.5.0/api/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspector.html'>ObjectInspector</a> hoops, to me, always seemed annoying to deal with.</p>
<p>Given these shortcomings, Bizo has been looking for a Hive-successor for awhile, even going so far as to prototype <a href='https://github.com/aboisvert/revolute'>revolute</a>, a Scala DSL on top of <a href='http://www.cascading.org/'>Cascading</a>, but had not yet found something we were really excited about.</p>
<h2 id='enter_spark'>Enter Spark!</h2>
<p>We had heard about Spark, but did not start trying it until being so impressed by the Spark presentation at AWS re:Invent (the talk received <a href='https://amplab.cs.berkeley.edu/news/sparkshark-a-big-hit-at-aws-reinvent/'>the highest rating of all non-keynote sessions</a>) that we wanted to learn more.</p>
<p>One of Spark’s touted strengths is being able to load and keep data in memory, so your queries aren’t always I/O bound.</p>
<p>That is great, but the exciting aspect for us at Bizo is how Spark, either intentionally or serendipitously, addresses both of Hive’s primary shortcomings, and turns them into huge strengths. Specifically:</p>
<b id='1_spark_jobs_are_amazingly_easy_to_test'>1. Spark jobs are amazingly easy to test</b>
<p>Writing a test in Spark is as easy as:</p>
<pre class='brush:scala'><code>class SparkTest {
@Test
def test() {
// this is real code...
val sc = new SparkContext("local", "MyUnitTest')
// and now some psuedo code...
val output = runYourCodeThatUsesSpark(sc)
assertAgainst(output)
}
}</code></pre>
<p>(I will go into more detail about <code>runYourCodeThatUsesSpark</code> in a future post.)</p>
<p>This one liner starts up a new <a href='http://spark-project.org/docs/latest/api/core/index.html#spark.SparkContext'>SparkContext</a>, which is all your program needs to execute Spark jobs. There is no local installation required (just have the Spark jar on your classpath, e.g. via Maven or Ivy), no local server to start/stop. It just works.</p>
<p>As a technical aside, this “local” mode starts up an in-process Spark instance, backed by a thread-pool, and actually opens up a few ports and temp directories, because it’s a real, live Spark instance.</p>
<p>Granted, this is usually more work than you want to be done in an unit test (which ideally would not hit any file or network I/O), but the redeeming quality is that it’s <em>fast</em>. Tests run in ~2 seconds.</p>
<p>Okay, yes, this is slow compared to pure, traditional unit tests, but is such a huge revolution compared to Hive that we’ll gladly take it.</p>
<b id='2_spark_is_easy_to_extend'>2. Spark is easy to extend</b>
<p>Spark’s primary API is a Scala DSL, oriented around what they call an <a href='http://www.spark-project.org/docs/0.6.0/api/core/#spark.RDD'><code>RDD</code></a>, or Resilient Distributed Dataset, which is basically a collection that only supports bulk/aggregate transforms (so methods like <code>map</code>, <code>filter</code>, and <code>groupBy</code>, which can be seen as transforming the entire collection, but no methods like <code>get</code> or <code>take</code> which assume in-memory/random access).</p>
<p>Some really short, made up example code is:</p>
<pre class='brush:scala'><code>// RDD[String] is like a collection of lines
val in: RDD[String] = sc.textFile("s3://bucket/path/")
// perform some operation on each line
val suffixed = in.map { line => line + "some suffix" }
// now save the new lines back out
suffixed.saveAsTextFile("s3://bucket/path2")</code></pre>
<p>Spark’s job is to package up your <code>map</code> closure, and run it against that extra large text file across your cluster. And it does so by, after shuffling the code and data around, <em>actually calling your closure</em> (i.e. there is no <a href='http://msdn.microsoft.com/en-us/library/vstudio/bb397926.aspx'>LINQ</a>-like introspection of the closure’s AST).</p>
<p>This may seem minor, but it’s huge, because it means there is no framework code or APIs standing between your running closure and any custom functions you’d want to run. Let’s say you want to use <code>SomeUtilityClass</code> (or the venerable <a href='http://commons.apache.org/lang/api-2.5/org/apache/commons/lang/StringUtils.html'><code>StringUtils</code></a>), just do:</p>
<pre class='brush:scala'><code>import com.company.SomeUtilityClass
val in: RDD[String] = sc.textFile("s3://bucket/path/")
val processed = in.map { line =>
// just call it, it's a normal method call
SomeUtilityClass.process(line)
}
processed.saveAsTextFile("s3://bucket/path2")</code></pre>
<p>Notice how <code>SomeUtilityClass</code> doesn’t have to know it’s running within a Spark RDD in the cluster. It just takes a String. Done.</p>
<p>Similarly, Spark doesn’t need to know anything about the code you use witin the closure, it just needs to be available on the classpath of each machine in the cluster (which is easy to do as part of your cluster/job setup, you just copy some jars around).</p>
<p>This seamless hop between the RDD and custom Java/Scala code is very nice, and means your Spark jobs end up reading just like regular, normal Scala code (which to us is a good thing!).</p>
<h2 id='is_spark_perfect'>Is Spark Perfect?</h2>
<p>As full disclosure, we’re still in the early stages of testing Spark, so we can’t yet say whether Spark will be a wholesale replacement for Hive within Bizo. We haven’t gotten to any serious performance comparisons or written large, complex reports to see if Spark can take whatever we throw at it.</p>
<p>Personally, I am also admittedly somewhat infutuated with Spark at this point, so that could be clouding my judgement about the pros/cons and the tradeoffs with Hive.</p>
<p>One Spark con so far is that Spark is pre-1.0, and it can show. I’ve seen some stack traces that shouldn’t happen, and some usability warts, that hopefully will be cleared up by 1.0. (That said, even as a newbie I find the codebase small and very easy to read, such that I’ve had <a href='https://github.com/mesos/spark/pull/352'>several</a> <a href='https://github.com/mesos/spark/pull/351'>small</a> <a href='https://github.com/mesos/spark/pull/362'>pull requests</a> accepted already–which is a nice consolation compared to the daunting codebases of Hadoop and Hive.)</p>
<p>We have also seen that, for our first Spark job, moving from “Spark job written” to “Spark job running in production” is taking longer than expected. But given that Spark is a new tool to us, we expect this to be a one-time cost.</p>
<h2 id='more_to_come'>More to Come</h2>
<p>I have a few more posts coming up which explain our approach to Spark in more detail, for example:</p>
<ul>
<li>Testing best practices</li>
<li>Running Spark in EMR</li>
<li>Accessing partitioned S3 logs</li>
</ul>
<p>To see those when they come out, make sure to subscribe to the blog, or, better yet, <a href='http://bizo.theresumator.com/'>come work at Bizo</a> and help us out!</p>
Stephen Habermanhttp://www.blogger.com/profile/05412274950722949930noreply@blogger.com2tag:blogger.com,1999:blog-5261056907132640554.post-14664736769154130922012-09-26T06:00:00.000-07:002013-06-26T15:21:32.810-07:00Grouping pageviews into visits: a Scala code kata<b id="internal-source-marker_0.15028386609628797" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The basic units of any website traffic analysis are pageviews, visits, and unique visitors. Tracking pageviews is simply a matter of counting requests to the server. Calculating unique visitors usually relies on cookies and unique identifiers. Visits, however, require a bit more work. For our purposes, a single visit is defined as a sequence of pageviews where the interval between pageviews is less than a fixed length like 15 minutes.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I thought that the problem of grouping pageviews into visits would make an interesting </span><a href="http://codekata.pragprog.com/"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">code kata</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Here’s the statement of the problem that I worked from:</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<blockquote class="tr_bq">
<b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Given a non-empty sequence of timestamps (as milliseconds since the epoch), write a function that would return a sequence of visits, where each visit is itself a sequence of timestamps where each pair of consecutive timestamps is no more than N milliseconds apart.</span></b></blockquote>
<b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">As a starting point, I decided to take a straightforward procedural approach:</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">def doingItIteratively(pageviews: Seq[Long]): Seq[Seq[Long]] = {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val iterator = pageviews.sorted.iterator</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val visits = ListBuffer[ListBuffer[Long]]()</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> var previousPV: Long = iterator.next</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> var currentVisit: ListBuffer[Long] = ListBuffer(previousPV)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> for (currentPV <- iterator) {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> if (currentPV - previousPV > N) {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits += currentVisit</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> currentVisit = ListBuffer[Long]()</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> currentVisit += currentPV</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> previousPV = currentPV</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits += currentVisit</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits map (_.toSeq) toSeq</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">}</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">So, we simply iterate through the (sorted) events tracking both the current visit and the previous pageview. If the current pageview represents a new visit, push the previous visit into the list of all visits and start a new one. Then push the current pageview into the (potentially new) visit.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">It actually felt a bit odd to write procedural code like this and ignore the functional parts of Scala. Using a fold cleans the code up a bit and gets rid of the mutable state.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">def doingItByFolds(pageviews: Seq[Long]): Seq[Seq[Long]] = {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val sortedPVs = pageviews.sorted</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (Seq[Seq[Long]]() /: sortedPVs) { (visits, pv) =></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val isNewVisit = visits.lastOption flatMap (_.lastOption) map {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> prevPV => pv - prevPV > N</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> } getOrElse true</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> if (isNewVisit) {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits :+ Seq(pv)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> } else {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits.init :+ (visits.last :+ pv)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">}</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Here, we’re starting with an empty list of visits and folding it over the sorted pageviews. At each pageview, we decide if we need to start a new visit. If so, we append a new visit containing the pageview to the accumulated visits. If not, we pop off the last visit, append the pageview, and put the last visit back on the tail of the accumulated visits.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">One part that’s still a bit messy is comparing the current timestamp to the previous one. We can improve that by iterating through the intervals between pageviews instead of the actual pageviews.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">def slidingThroughIt(pageviews: Seq[Long]): Seq[Seq[Long]] = {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val intervals = (0L +: pageviews.sorted).sliding(2)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (Seq[Seq[Long]]() /: intervals) { </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (visits, interval) =></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> if (interval(1) - interval(0) > N) {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits :+ Seq(interval(1))</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> } else {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits.init :+ (visits.last :+ interval(1))</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">}</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Here, we’re prepending a “0L” timestamp (and assuming that none of the pageviews happened in the early 70s) and using the “sliding” method to pair each timestamp with the previous one.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">So far, we’ve been using a sequence of pageviews as a visit. What happens if we add an explicit Visit type? This lets us convert all pageviews into Visits at the start, then focus on merging overlapping Visits. One nice benefit is that this is a map-reduce algorithm that can be easily parallelized instead of one that must sequentially iterate over the pageviews (either explicitly or with a fold).</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">case class Visit(start: Long, end: Long, pageviews: Seq[Long]) {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> def +(other: Visit): Visit = {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> Visit(min(start,other.start), max(end, other.end),</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (pageviews ++ other.pageviews).sorted)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">}</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">def doingItMapReduceStyle(pageviews: Seq[Long]): Seq[Visit] = {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> pageviews.par map { pv =></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> Seq(Visit(pv, pv+N, Seq(pv))</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> } reduce { (visit1, visit2) =></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> val sortedVisits = (v1 ++ v2) sortBy (_.start)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> (Seq[Visit]() /: sortedVisits) { (visits, next) =></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> if (visits.lastOption map(_.end >= next.start) getOrElse false)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits.init :+ (visits.last + visit)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> } else {</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> visits :+ visit</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> }</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">}</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The map-reduce solution is fun, but in a production system, I’d probably stick with the sliding variation and add a bit more flexibility to track actual pageview objects instead of just timestamps.</span></b>Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-74742923187831864772012-09-19T06:00:00.000-07:002012-09-19T06:00:11.057-07:00Using GROUP BYs or multiple INSERTs with complex data types in Hive.<b id="internal-source-marker_0.4339277243707329" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In any sort of ad hoc data analysis, the first step is often to extract a specific subset of log lines from our files. For example, when looking at a single partner’s web traffic, I often use an initial query to copy that partner’s data into a new table. In addition to segregating out only the data relevant to my analysis, I use this to copy the data from S3 into HDFS, which will make later queries more efficient. (Using maps as our log lines is how we support </span><a href="http://dev.bizo.com/2011/02/columns-in-hive.html"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">dynamic columns</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.)</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">create external table if not exists </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">original_logs(fields map<string,string>) location “...” ;</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">create table if not exists </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">extracted_logs(fields map<string,string>) ;</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">insert overwrite table extracted_logs</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">select * from original_logs where fields[“partnerId”] = 123 ;</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If I’m doing this for multiple partners, it’s tempting to use a multiple-insert so Hadoop only needs to make one pass of the original data.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">create external table if not exists </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">original_logs(fields map<string,string>) location “...” ;</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">create table if not exists </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">extracted_logs(fields map<string,string>) </span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">partitioned by (partnerId int);</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">from original_logs</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">insert overwrite table extracted_logs partition (partnerId = 123)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">select * from original_logs where fields[“partnerId”] = 123</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">insert overwrite table extracted_logs partition (partnerId = 234)</span><br /><span style="font-family: 'Courier New'; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">select * from original_logs where fields[“partnerId”] = 234</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Unfortunately, in Hive 0.7.x, this query fails with the error message “</span><span style="background-color: white; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Hash code on complex types not supported yet.” A multiple-insert statement uses an implicit group by, and Hive 0.7.x </span><a href="https://github.com/apache/hive/blob/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L500"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">does not support grouping by complex types</span></a><span style="background-color: white; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">This bug was partially addressed in 0.8, which added support for arrays and maps, but structs and unions are </span><a href="https://github.com/apache/hive/blob/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L500"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">still not supported</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">At an initial glance, it does look like adding this support should be straightforward. This could be a good candidate for our next </span><a href="http://dev.bizo.com/2012/04/dev-days-hacking-open-source-and-docs.html"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">open source day</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></b>Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-4220557650942563782012-07-07T12:44:00.001-07:002012-07-07T12:47:19.760-07:00mdadm: device or resource busyI just spent a few hours tracking an issue with <a href="http://en.wikipedia.org/wiki/Mdadm" target="_blank">mdadm</a> (Linux utility used to manage software RAID devices) and figured I'd write a quick blog post to share the solution so others don't have to waste time on the same.<br />
<br />
As a short background, we use <span style="font-family: 'Courier New', Courier, monospace;">mdadm</span> to create RAID-0 stripped devices for our Sugarcube analytics (OLAP) servers using Amazon EBS volumes.<br />
<br />
The issue manifested itself as a random failure during device creation:<br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --create /dev/md0 --level=0 --chunk 256 --raid-devices=4 /dev/xvdh1 /dev/xvdh2 /dev/xvdh3 /dev/xvdh4</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: Defaulting to version 1.2 metadata</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: ADD_NEW_DISK for /dev/xvdh3 failed: Device or resource busy</span><br />
<div>
<br /></div>
<div>
I searched and searched the interwebs and tried every trick I found to no avail. We don't have <a href="http://www.linuxmanpages.com/man8/dmraid.8.php" target="_blank">dmraid</a> installed on our Linux images (Ubuntu 12.04 LTS / Alestic cloud image) so there's no possible conflict there. All devices were clean, as they are freshly created EBS volumes and I knew none of them were in use. </div>
<div>
<br /></div>
<div>
Before running <span style="font-family: 'Courier New', Courier, monospace;">mdadm --create</span>, <span style="background-color: white;"><span style="font-family: 'Courier New', Courier, monospace;">mdstat</span> was clean:</span></div>
<div>
<span style="background-color: white;"><br /></span></div>
<div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ cat /proc/mdstat</span></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]</span></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">unused devices: <none></span></div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
And yet after running it the devices were assigned to two different devices instead of just <span style="font-family: 'Courier New', Courier, monospace;">/dev/md0</span>:</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ cat /proc/mdstat</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">md127 : inactive xvdh4[3](S) xvdh3[2](S)</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 1048573952 blocks super 1.2</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">md0 : inactive xvdh2[1](S) xvdh1[0](S)</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 1048573952 blocks super 1.2</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">unused devices: <none></span></div>
</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
Looking into <span style="font-family: 'Courier New', Courier, monospace;">dmesg</span> didn't reveal anything interesting either:</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ dmesg </span></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">...</span></div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.552493] md: bind<xvdh1></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.553011] md: bind<xvdh2></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.553040] md: could not open unknown-block(202,115).</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.553052] md: md_import_device returned -16</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.566543] md: bind<xvdh3></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[3963010.731009] md: bind<xvdh4></span></div>
</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
And strangely, the creation or assembly would sometime work and sometime not:</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --manage /dev/md0 --stop</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: stopped /dev/md0</span></div>
</div>
<div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="background-color: #93c47d; font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ sudo mdadm --assemble --force /dev/md0 /dev/xvdh[1234]</span></div>
<div>
<span style="background-color: #93c47d; font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: /dev/md0 has been started with 4 drives.</span></div>
</div>
<div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --manage /dev/md0 --stop</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: stopped /dev/md0</span></div>
</div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="background-color: #e06666; font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ sudo mdadm --assemble --force /dev/md0 /dev/xvdh[1234]</span></div>
<div>
<span style="background-color: #e06666; font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: cannot open device /dev/xvdh3: Device or resource busy</span></div>
</div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --manage /dev/md0 --stop</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: stopped /dev/md0</span></div>
</div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="background-color: #e06666; font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ sudo mdadm --assemble --force /dev/md0 /dev/xvdh[1234]</span></div>
<div>
<span style="background-color: #e06666; font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: cannot open device /dev/xvdh1: Device or resource busy</span></div>
<div>
<span style="background-color: #e06666; font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: /dev/xvdh1 has no superblock - assembly aborted</span></div>
</div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<div style="background-color: white;">
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --manage /dev/md0 --stop</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: stopped /dev/md0</span></div>
</div>
<div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="background-color: #93c47d; font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ sudo mdadm --assemble --force /dev/md0 /dev/xvdh[1234]</span></div>
<div>
<span style="background-color: #93c47d; font-family: 'Courier New', Courier, monospace; font-size: x-small;">mdadm: /dev/md0 has been started with 4 drives.</span></div>
</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
I started suspecting I was facing some kind of underlying race condition where the devices would get assigned/locked during the device creation process. So I started googling for "mdadm create race" and I finally found a <a href="http://permalink.gmane.org/gmane.linux.raid/34027" target="_blank">post</a> that tipped me off. While it didn't provide the solution, the post put me on the right track by mentioning <a href="http://en.wikipedia.org/wiki/Udev" target="_blank">udev</a> and it took only a few more minutes to narrow down on the solution: <i>disabling udev events during device creation to avoid contention on device handles</i>.</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
So now our script goes something like:</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ udevadm control --stop-exec-queue</span></div>
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ mdadm --create /dev/md0 --run --level=0 --raid-devices=4 ...</span></div>
<div style="background-color: white;">
<div style="background-color: white;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">$ udevadm control --start-exec-queue</span></div>
<div>
<br /></div>
</div>
<div style="background-color: white;">
And we now have consistent reliable device creation.</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
Hopefully this blog post will help other passers-by with a similar problem. Good luck!</div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
<br /></div>
<div style="background-color: white;">
</div>
</div>
</div>Anonymousnoreply@blogger.com9tag:blogger.com,1999:blog-5261056907132640554.post-27368014348685404822012-07-03T08:45:00.002-07:002012-07-03T08:45:56.867-07:00Amazon Web Services Outages: 4 Steps for Survival(Cross-post from the Bizo Blog)<br />
<br />
<span style="color: #404040; font-family: Arial; font-size: 13px; line-height: 20px; text-align: left;">Another Amazon Web Services (AWS) cloud outage over the past weekend took down some pretty major services such as Netflix, Heroku, Pinterest and Instagram. At Bizo, a company that provides business marketing services for hundreds of F1000 clients, we serve billions of requests a day across tens of thousands of websites, and have our entire infrastructure on the AWS cloud, but didn’t have any downtime. The simple reason is that we take our customers’ uptime and site performance seriously, and have built tools and services on AWS to ensure high-availability (HA) and low-latency (LL) services. Despite the </span><a href="http://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt" style="color: #44aadf; font-family: Arial; font-size: 13px; line-height: 20px; text-align: left; text-decoration: none !important;" target="_blank">FUD</a><span style="color: #404040; font-family: Arial; font-size: 13px; line-height: 20px; text-align: left;"> created by many of the industry blogs and press, it is possible to create HA and LL services on AWS if you follow some simple steps.</span><br />
<br />
<a href="http://blog.bizo.com/blog/engineering/surviving-amazon-web-service-outages">Amazon Web Services Outages: 4 Steps for Survival</a>Donniehttp://www.blogger.com/profile/13599133732419522440noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-26511928366926126932012-06-14T20:33:00.001-07:002012-06-14T20:38:13.280-07:00AWS Billing Info in Hive<p>Amazon recently (finally!) launched <a href="http://docs.amazonwebservices.com/awsaccountbilling/latest/about/programaccess.html">programmatic access to your AWS billing data</a>.</p>
<p>Once you turn it on, select a bucket, grant access to the AWS system user, you'll get a .csv file with your estimated billing for the month. The files are delivered daily, but they contain month-to-date information, and will replace the file from the previous day.</p>
<p>It's easy enough to view this information in excel (or similar), but I thought it would be fun to take a look in hive, especially once we start having data for a few months to aggregate over.</p>
<p>Amazon delivers the data to the root of your bucket. I decided to start moving it to a hive-partitioned path, to make it easier to query once we start have more data. I wrote a simple scala script to move the data to <code>[bucket]/partioned/year=[year]/month=[month]/[file]</code>. Here's <a href="https://gist.github.com/2933771">some example code</a>.</p>
<p>Ok, now we're ready to read the data in Hive.</p>
<p>Here's a <a href="https://gist.github.com/2933779">hive schema for the AWS billing information</a>. It uses the <a href="https://github.com/ogrodnek/csv-serde">csv-serde</a> (make sure you add that jar before running the create table statement). Run <code>alter table aws_billing recover partitions;</code> to load in the partitions (one per year/month), and you're ready to query.</p>
<p>Like I said, it's overkill to use hive to read this data for a month or so, but it's just so addictive having a SQL interface to arbitrary S3 data :).</p>
<p>Here are some example queries to get you started.</p>
<p>
<h2>Costs by Service</h2>
<pre>
select ProductCode, UsageType, Operation, sum(TotalCost)
from aws_billing
where RecordType in ("PayerLineItem", "LinkedLineItem")
group
by ProductCode, UsageType, Operation
;
</pre>
</p>
<p>
<h2>EC2 usage, by size (across EC2/EMR)</h2>
<pre>
select ProductCode, UsageType,sum(TotalCost)
from aws_billing
where RecordType in ("PayerLineItem", "LinkedLineItem")
and UsageType like "BoxUsage%"
group
by ProductCode, UsageType
;
</pre>
</p>
larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-16072675645508385622012-06-13T16:10:00.001-07:002012-06-13T16:10:28.467-07:00the golden rule of programming style<p>There's an interesting page on the subject of <a href="http://docs.scala-lang.org/style/files.html">compilation units per file</a> over at the <a href="http://docs.scala-lang.org/style/overview.html">scala style guide</a>.</p>
<p>The guideline, is, delightfully vague, which I will paraphrase as:
<quote>Mostly use
single files, unless you can't, or unless it's better if you don't.<quote></p>
<p>The author(s) go on to expand on the reasoning behind breaking the guideline:
<blockquote>
Another case is when multiple classes logically form a single, cohesive group, sharing concepts to the point where maintenance is greatly served by containing them within a single file. These situations are harder to predict… Generally speaking, if it is easier to perform long-term maintenance and development on several units in a single file rather than spread across multiple, then such an organizational strategy should be preferred for these classes.
</blockquote>
</p>
<p>This touches on what I consider to be the golden rule of programming style: <b>Make your intent clear and the code easy to read.</b></p>
<p>Software spends most of its life in maintenance, which is why we have style guides and coding standards. It's valuable to have consistent looking code to promote a shared vocabulary, improve readability, and steer away from confusing or error-prone constructs.</p>
<p>It is just as important to be able to understand, both as an author and as a reviewer, that in certain cases following the letter of the law goes against the main goal of improving readability and maintenance. A one-size-fits-all rule does not always work, and as the authors of this particular guideline mention, "these situations are harder to predict."</p>
<p><b>Make your intent clear and the code easy to read.</b></p>
larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-44774135479625069122012-04-20T17:19:00.000-07:002012-04-20T17:19:13.617-07:00Scala Test Plug-in for Sublime Text 2I have documented and put some polish on the <a href="http://dev.bizo.com/2012/04/creating-plug-ins-for-sublime-text-2.html">Sublime Text 2 plug-in I blogged about previously</a>. It l<span style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">ets you run a single Scala Test, or all tests in your project. It also lets you quickly navigate to any scala files in your project folder, and switch back and forth between a class and its test. Check it out here: </span><span style="color: #222222; font-family: arial, sans-serif; font-size: x-small;"><a href="https://github.com/patgannon/sublimetext-scalatest">https://github.com/patgannon/sublimetext-scalatest</a></span><br />
<span style="color: #222222; font-family: arial, sans-serif; font-size: x-small;"><br /></span>Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-79602970822874310702012-04-20T16:21:00.002-07:002012-04-20T16:21:48.450-07:00Dev Days: Hacking, Open Source and Docs<h3>
Dev Days</h3>
Every month we have a "Dev Day" where engineers take a break from their projects and work on "other stuff". Most start-up engineering teams have a "Hack Day" where everyone gets to hack on anything they want as long as they ship and share it with the rest of the team. Of course we have Hack Days but we also have other types of Dev Days too. In fact, we have three types of Dev Days:<br />
<br />
<ul>
<li>Hack Days </li>
<li>Open Source Days </li>
<li>Doc Days</li>
</ul>
<h3>
Open Source Days</h3>
You know what Hack Days are so I'll move on quickly to Open Source Days. Just like most companies these days, Bizo uses a lot of open source software (OSS). We love OSS and the community of developers and companies that share it. Over the last few years, we've used plenty of OSS but we've also created and <a href="http://code.bizo.com/">given back lots of code</a> as well. <br />
<br />
Actually today is one of our Open Source Days so all the engineers are working on both new and old open source projects. You can check out our (growing) list of projects by visiting <a href="http://code.bizo.com/">code.bizo.com</a>. Over the years, we've created a lot of tools around AWS including <a href="https://github.com/aboisvert/s3cp">s3cp</a>, <a href="https://github.com/stephenh/fakesdb">fakesdb</a>, <a href="https://github.com/floodfx/aws-tools">aws-tools</a> (package of all CLI tools). We've also built a lot of stuff for Hadoop (Hive, etc) including <a href="http://ogrodnek.github.com/csv-serde/">csv-serde</a>, <a href="https://github.com/balshor/gdata-storagehandler">gdata-storagehandler</a> and our latest is a scala query language called <a href="https://github.com/aboisvert/revolute">revolute</a> (still in development). In addition, we've have a wide variety of other awesome code including the awesome <a href="http://joist.ws/">Joist</a>, <a href="https://github.com/jcarver989/dependence.js">dependence.js</a>, <a href="http://softwarebyjosh.com/raphy-charts/">raphy-charts</a> and other <a href="https://github.com/t-pleasure/mighty-csv">fun stuff</a>!<br />
<h3>
Doc Days</h3>
<div>
The third type of Dev Day we have is called Doc Days. I know that you are thinking but Doc Days are extremely valuable days for engineering and everyone else for that matter. On Doc Day the entire engineering team works on wiki pages, code documentation, design docs, architecture docs and even blog posts. It really is better than it sounds! </div>
<div>
<br /></div>
<div>
If you've read my post on "<a href="http://dev.bizo.com/2011/03/on-building-kick-ass-engineering-team.html">building a kick ass engineering team</a>", you know that one of the keys is the 3Cs... Communication, Communication, Communication! (My high school baseball coach taught me that one.) As a engineering team, we believe that communication is one of the best things we can do for each other. As any company grows communication becomes a larger and larger part of day to day and we see Doc Days as a way to ensure that we are communicating as clearly and accurately as we can.</div>
<h3>
Conclusion</h3>
<div>
These Dev Days have been a huge success for Bizo engineering. We've even inspired other departments to have similar days (Marketing in particularly like these documentation days!). We challenge you to go beyond the "Hack Day" and start thinking about other Dev Days that your engineering organization can benefit from. </div>Donniehttp://www.blogger.com/profile/13599133732419522440noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-73738394679130160562012-04-05T16:36:00.001-07:002012-04-05T16:43:06.185-07:00Implementation driven interfaces?<p>I've recently encountered some <i>interesting</i> pagination in the Google Groups admin interface.
It starts off simple enough, nothing exciting here...
</p>
<p>
<img src="http://com-bizo-public.s3.amazonaws.com/blog/groups_pagination/gg_page1.png" />
</p>
<p>Instead of the usual 'Previous', we see 'First' on the next page.</p>
<p>
<img src="http://com-bizo-public.s3.amazonaws.com/blog/groups_pagination/gg_page2.png" />
</p>
<p>Are they just being clever? Knowing that there's only one previous page? No such luck…</p>
<p><img src="http://com-bizo-public.s3.amazonaws.com/blog/groups_pagination/gg_page3.png" /></p>
<p>We've reached the end of the list. I hope you've found what you're looking for, otherwise start over from the beginning!</p>
<p><img src="http://com-bizo-public.s3.amazonaws.com/blog/groups_pagination/gg_last.png" /></p>
<p>One has to wonder, who designed this interface? You can only go forward. If you overshoot, it's back to square one, then click, click, click… It's clear it does not have users in mind at all.</p>
<p>My guess is that it's based on some limitation in the backend storage or query mechanism. The system only allows forward navigation of query results, so the interface simply mirrors that…</p>
<p>What an incredibly frustrating experience! I'll never take simple pagination for granted again.</p>
<p>It's a good reminder to think about your users and how they will interact with the system. Mirroring the programming interface rarely works.</p>
larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-21609220244137904522012-04-04T08:56:00.006-07:002012-04-04T09:34:48.584-07:00Capturing Client Side JS Errors on AWS<span style="font-weight: normal; font-size: 100%; ">I saw a </span><a href="http://news.ycombinator.com/item?id=3796869" style="font-weight: normal; font-size: 100%; ">post go by on Hacker News</a><span style="font-weight: normal; font-size: 100%; "> this morning discussing </span><a href="http://openmymind.net/2012/4/4/You-Really-Should-Log-Client-Side-Error/" style="font-weight: normal; font-size: 100%; ">capturing and reporting on client side errors</a><span style="font-weight: normal; font-size: 100%; ">. We have been doing this for a long time and I wanted to share our approach.</span><br /><br /><span style="font-weight: normal; ">Background</span><br /><span style="font-weight: normal; font-size: 100%; ">Quick background, we have two major types of javascript that our customers and partners may use: analytics tags and ad tags. Both tags are javascript and share the same error capture code. </span><br /><br /><span style="font-weight: normal; font-size: 100%; ">Another quick note is that we run on <a href="http://aws.amazon.com">Amazon Web Services</a> so this approach is based on some of these services including S3, CloudFront and EMR.</span><br /><br /><span style="font-weight: normal; ">Implementation</span><br /><span style="font-weight: normal; font-size: 100%; ">Our client side JS is compiled from Coffeescript. I've created a couple of gists to show you what the error logging code looks like in Coffeescript.</span><br /><br /><script src="https://gist.github.com/2303307.js?file=error_log.coffee"></script><br /><script src="https://gist.github.com/2303403.js"> </script><br /><br /><span style="font-weight: normal; ">Details</span><br /><span style="font-weight: normal; font-size: 100%; ">The example shows our ad tags trying to execute surrounded by a try/catch that captures the error and eventually results in loading an image appending the relevant error metadata. </span><br /><br /><span >AWS Details</span><br /><span style="font-weight: normal; font-size: 100%; ">The image that is loaded actually lives on CloudFront. The CloudFront distribution is setup with logging which means that requests are logged and delivered to a specified S3 bucket (usually within 24 hours). Every day we run an EMR job against the CloudFront request logs that generates a report summarizing the errors. And that is it. Pretty simple and this approach has worked for us.</span><br /><br /><span style="font-weight: normal; "><span >Pre-emptive "this isn't perfect" response</span></span><br /><span style="font-weight: normal; font-size: 100%; ">Some of you may be thinking, "you may not get all requests!". CloudFront logs are not supposed to be used for 100% accurate reporting (although nothing is really 100%). In our case, we don't need to capture all errors rather we are looking for directional information.</span>Donniehttp://www.blogger.com/profile/13599133732419522440noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-29639260868468894202012-04-02T11:48:00.000-07:002012-04-02T13:47:04.518-07:00Creating Plug-ins for Sublime Text 2<br />
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
I have been trying out <a href="http://www.sublimetext.com/2" style="color: #1155cc;" target="_blank">Sublime Text 2</a> as my text editor lately, and I'm loving the simplicity, so I figured I would try out creating a plug-in for it. I was pleasantly surprised at how easy it is, which is an important step towards it becoming my new editor of choice. I wanted to take some steps towards creating something along the lines of <a href="http://rinari.rubyforge.org/" style="color: #1155cc;" target="_blank">rinari</a>, but for Scala... in Sublime Text. I was able to fairly easily easily create a plug-in that allowed me to run the Scala Test that was currently open in the editor, or run all Scala Tests in the (inferred) project folder, or switch back and forth between a test and the code under test, or quickly navigate to any scala file in the project folder with a few keystrokes. This post will show you how to create a new plug-in for Sublime Text 2, which uses all the API features that I needed to implement that functionality.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h2>
Create a new plug-in</h2>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<strong>Step 1</strong>. Install Sublime Text 2 (see link above). Its free to try, and fairly cheap to buy. A month or so after you download it, it basically becomes nag-ware until you finally manage to overcome your stingy developer impulses and plunk down the $59 to buy it. Also, unlike other similar text editors (ahem.. TextMate!) it actually runs on Windows and Linux, as well as Mac OSX.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<strong>Step 2</strong>. Create a new folder for your plug-in. On Mac OSX, this goes under your home folder in ~/Library/Application Support/Sublime Text 2/Packages/{PLUGIN_NAME} (where in my case, {PLUGIN_NAME} was "ScalaTest").</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<strong>Step 3</strong>. Create a python file which will contain the code for the plug-in. (Name it whatever you want, as long as it ends in ".py" ;-) Here is a really basic plug-in (borrowed from <a href="http://sublimetext.info/docs/en/extensibility/plugins.html" style="color: #1155cc;" target="_blank">this plug-in tutorial</a>, which you should read after this):</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;">import sublime, sublime_plugin
class ExampleCommand(sublime_plugin.<wbr></wbr>TextCommand):
def run(self, edit):
self.view.insert(edit, 0, "Hello, World!")
</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
Right, so, as I mentioned: Sublime Text 2 plug-ins are written in Python. Don't worry too much if you're not familiar with Python... I wasn't either prior to starting this experiment, and it didn't prove to be too much a problem. (I did have a couple Python books laying around, but I'm sure the same information is on the tubes.) Its fairly easy to pick up, and has some similarities to Ruby, in case that helps. So the code above creates a command called "example" which is defined by a class that inherits from Sublime Text's "TextCommand" class. (Sublime Text 2 maps the title-case class names to underscore-delimited command names, and strips the "Command" suffix.) All the plug-in does is insert the text "Hello, World!" at the beginning of the file open in the editor.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
(Note: Sublime Text 2 will detect that you created a Python file under its plug-in folder and automatically loads it.)</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<strong>Step 4</strong>. Run your example. Hit Ctrl+Backtick to open the python interpreter within Sublime Text 2. Run your command by typing in this:</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
view.run_command("example")</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
The open buffer will now include the aforementioned greeting. You could bind it to a key-combination easily enough, but hey, it doesn't do anything cool yet, right?, so we'll hold off on the key bindings until the end.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h2>
Make it do something cool</h2>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
So that you can see these approaches in action, I uploaded my nascent ScalaTest plug-in to github:<a href="https://github.com/patgannon/sublimetext-scalatest" style="color: #1155cc;" target="_blank">https://github.com/patgannon/<wbr></wbr>sublimetext-scalatest</a>. Note that this plug-in will currently only work with projects that use Bizo's standard folder structure, and has a hard coded path to the scala executable, so its not ready to be used as-is. I hope to clean it up in the future and make it more generically applicable, but for now, I've only shared it to add a bit more color to the code snippets in this section.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h3>
Run a command on the current file</h3>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
The name of the file currently open in the editor can be obtained with this expression: self.view.file_name(). In my plug-in, I use that to infer a class name, the project root folder, and path to the associated test (using simple string operations).</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
You can create an output panel (in which to render the results of running a command on the open file) by calling: self.window.run_command("show_<wbr></wbr>panel", {"panel": "output.tests"}) <em>(where "output.tests" is specific to your plug-in)</em>. In my plug-in, I created the helper methods below to show the panel and clear out its contents. (See the BaseScalaTestCommand class in run_scala_test.py). Note that this code was derived from code I found in the<a href="https://github.com/maltize/sublime-text-2-ruby-tests" style="color: #1155cc;" target="_blank">Sublime Text 2 Ruby Tests</a> plug-in.</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;"> def show_tests_panel(self):
if not hasattr(self, 'output_view'):
self.output_view = self.window().get_output_<wbr></wbr>panel("tests")
self.clear_test_view()
self.window().run_command("<wbr></wbr>show_panel", {"panel": "output.tests"})
def clear_test_view(self):
self.output_view.set_read_<wbr></wbr>only(False)
edit = self.output_view.begin_edit()
self.output_view.erase(edit, sublime.Region(0, self.output_view.size()))
self.output_view.end_edit(<wbr></wbr>edit)
self.output_view.set_read_<wbr></wbr>only(True)</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
(Note: I don't recommend copy/pasting code directly from this blog post, because the examples are pasted in from github, which messes up the indentation, which is a real problem in Python; instead, clone the github repository and copy/paste from the real file on your machine.)</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
To actually execute the command, I use this code in my run method, after calling show_tests_panel defined above (note that you will need to import 'subprocess' and 'thread' at the top of your plug-in file):</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;"> self.proc = subprocess.Popen("{my command}", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
thread.start_new_thread(self.<wbr></wbr>read_stdout, ())
</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
...where {my command} is the shell command I want to execute, and read_stdout is a method I defined which copies the output from the process and puts it into the output panel. Its defined as follows (and calls the append_data method, also defined below):</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;"> def read_stdout(self):
while True:
data = os.read(self.proc.stdout.<wbr></wbr>fileno(), 2**15)
if data != "":
sublime.set_timeout(functools.<wbr></wbr>partial(self.append_data, self.proc, data), 0)
else:
self.proc.stdout.close()
break</pre>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;"> def append_data(self, proc, data):
self.output_view.set_read_<wbr></wbr>only(False)
edit = self.output_view.begin_edit()
self.output_view.insert(edit, self.output_view.size(), data)
self.output_view.end_edit(<wbr></wbr>edit)
self.output_view.set_read_<wbr></wbr>only(True)</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
(Note: Depending on the command you're running, you may also want to capture the process' stderr output, and also put that into the output panel, using a variation of the approach above.)</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h3>
Using the "quick panel" to search for files, and opening files</h3>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
The "quick panel" (the drop-down which lists files when you hit command-T in sublime-text) can be extended to have plug-in specific functionality, which I used to create a hot-key for quickly navigating to any Scala file under my project folder. (See the JumpToScalaFile class in run_scala_test.py.) One of the plug-in examples I saw using the quick panel sub-classed sublime_plugin.WindowCommand instead of TextCommand. This results in a plug-in which can be run without any files being open. The flip side of that, though, is you don't get the file name of the currently open file, which in my case, is required to infer the base project folder for which to search for files. Thus, all my plug-ins sub-class TextCommand. To open the quick panel, execute: sublime.active_window().show_<wbr></wbr>quick_panel(file_names, self.file_selected). file_names should be a collection of the (string) <em>entries</em> to show in the quick panel. Note that the entries don't have to be file paths, just a convenient identifier to show the user (in my case, the class name). file_selected is a method you will define which will be called when a user selects an entry in the quick panel. Here's how I defined it:</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;"> def file_selected(self, selected_index):
if selected_index != -1:
sublime.active_window().open_<wbr></wbr>file(self.files[selected_<wbr></wbr>index])
</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
self.files is an array I created when populating the quick panel which maps an index in the quick panel to a file path. I then use sublime.active_window().open_<wbr></wbr>file to open that file in Sublime Text.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
I also used that same method (open_file) in the plug-in that automatically navigates back and forth between a test file and the code under test. That plug-in also makes use of the sublime.error_message method, which will display an error message to the user (if no test is found, for example).</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h2>
Create keystroke bindings</h2>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
To bind your new plug-in commands to keystrokes, create a file in your plug-in folder called <strong>Default (OSX).sublime-keymap</strong>. This will contain the keystrokes that will be used on Mac OSX. (You would create separate files for use on Windows and Linux.) It is a simple JSON file that maps keystrokes to commands. Lets see an example:</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;">[
{ "keys": ["super+shift+e"], "command": "jump_to_scala_file" }
]</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
This example will bind Command+Shift+e to the "jump_to_scala_file" command (defined in the JumpToScalaFileCommand class in any plug-in). If you have multiple key-mappings, you would create multiple comma-delimited entries within the JSON array. (See the example in my plug-in.) In order to reduce the possibility of defining keystrokes that collide with keystrokes from other plug-ins, I defined mine in such a way that they're only available when the currently open file is a Scala file. Here is the rather verbose (ahem, powerful) syntax that I used to do that:</div>
<pre style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; white-space: pre-wrap;">[
{ "keys": ["super+shift+e"], "command": "jump_to_scala_file",
"context" : [{"key": "selector", "operator": "equal", "operand": "source.scala", "match_all": true}]}
]</pre>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h2>
Conclusion</h2>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
Over the years, I've grown to prefer light-weight editors (such as emacs or Sublime Text 2) over more heavy-weight IDEs (such as Eclipse or Visual Studio) because they don't tend to lock up in the middle of writing code and/or crash sporadically, and I generally don't need a lot of whiz-bang features when I'm coding these days. I used emacs (and rinari) for doing rails development for a year or so, but the basic key-strokes (compared to the de-facto text editing standard key-strokes) and the undo/redo functionality always seemed a bit awkward, especially when you wind up switching back and forth between that and other text editors. Also, the language for creating plug-ins is Scheme (a dialect of Lisp), which to me isn't very convenient for these sort of things.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
I was really pleased with my foray into creating plug-ins for Sublime Text 2, and combined with its general ease of use, I've decided its now my new favorite editor. Using an editor that's this easy to significantly customize seems like it could be a real productivity win over time. Given the fairly rich list of plug-ins already available, I think the future is bright for Sublime Text 2. Below are a list of resources I found helpful during this process, including said list of plug-ins.</div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
</div>
<h2>
Resources</h2>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<b>Unofficial documentation: <a href="http://sublimetext.info/docs/en/index.html" style="color: #1155cc;" target="_blank">http://sublimetext.info/docs/<wbr></wbr>en/index.html</a> especially<a href="http://sublimetext.info/docs/en/extensibility/plugins.html" style="color: #1155cc;" target="_blank">http://sublimetext.info/docs/<wbr></wbr>en/extensibility/plugins.html</a><br />Official plug-in examples (sparse): <a href="http://www.sublimetext.com/docs/plugin-examples" style="color: #1155cc;" target="_blank">http://www.sublimetext.com/<wbr></wbr>docs/plugin-examples</a><br />Official API reference: <a href="http://www.sublimetext.com/docs/2/api_reference.html" style="color: #1155cc;" target="_blank">http://www.sublimetext.com/<wbr></wbr>docs/2/api_reference.html</a></b></div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<b>Helpful examples:<br /><a href="https://github.com/maltize/sublime-text-2-ruby-tests" style="color: #1155cc;" target="_blank">https://github.com/maltize/<wbr></wbr>sublime-text-2-ruby-tests</a><br /><a href="https://github.com/noklesta/SublimeRailsNav" style="color: #1155cc;" target="_blank">https://github.com/noklesta/<wbr></wbr>SublimeRailsNav</a><br /><a href="https://github.com/luqman/SublimeText2RailsRelatedFiles" style="color: #1155cc;" target="_blank">https://github.com/luqman/<wbr></wbr>SublimeText2RailsRelatedFiles</a><br /><a href="https://github.com/rspec/rspec-tmbundle" style="color: #1155cc;" target="_blank">https://github.com/rspec/<wbr></wbr>rspec-tmbundle</a></b></div>
<div style="background-color: rgba(255, 255, 255, 0.917969); color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<b>Unofficial list of plug-ins:<br /><a href="http://wbond.net/sublime_packages/community" style="color: #1155cc;" target="_blank">http://wbond.net/sublime_<wbr></wbr>packages/community</a></b></div>Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-21817812730750127432012-03-13T17:24:00.001-07:002012-03-13T17:24:14.093-07:00A Short Script for Logging into Interactive Elastic MapReduce ClustersElastic MapReduce is great, but the latencies can be painful. For me, this is especially true when I'm in the early stages of developing a new job and need to make the transition from code on my local machine to code running in the cloud -- the ~5 minute period between starting up a cluster and actually being able to log on to it is too long to sit there staring at a blank screen and too short to effectively context switch to something else in a useful way.<br />
<br />
My current solution is to allow myself to get distracted but to drag myself back to my EMR session as soon as it's available. Adding some simple polling plus a sticky <a href="http://growl.info/extras.php#growlnotify" target="_blank">growl</a> notification to my interactive-emr-startup script does the trick quite nicely:<br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">#!/bin/bash</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">if [ -z "$1" ]; then</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "Please specify a job name"</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> exit 1</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">fi</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">elastic-mapreduce \</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> (... with all of my favorite options ...) \</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">| tee ${TMP_FILE}</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">JOB_ID=`cat ${TMP_FILE} | awk '{print $4}'`</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">rm ${TMP_FILE}</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"># poll for WAITING state</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">JOB_STATE=''</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">MASTER_HOSTNAME=''</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">while [ "${JOB_STATE}" != "WAITING" ]; do</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> sleep 1</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo -n .</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> RESULT=`elastic-mapreduce --list | grep ${JOB_ID}`</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> JOB_STATE=`echo $RESULT | awk '{print $2}'`</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> MASTER_HOSTNAME=`echo $RESULT | awk '{print $3}'`</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">done</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">echo Connecting to ${MASTER_HOSTNAME}...</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">growlnotify -n "EMR Interactive" -s -m "SSHing into ${MASTER_HOSTNAME}"</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">ssh $MASTER_HOSTNAME -i ~/.ssh/emr-keypair -l hadoop -L 9100:localhost:9100</span><br />
<div>
<br /></div>
<br />
One of my personal productivity goals for the year is finding little places like this that I can optimize with a short script. This particular one has rescued me from the clutches of HN more than once!Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-61074368939803826262012-03-13T10:53:00.001-07:002012-03-13T11:48:18.215-07:00On Code Reviews and Developer Feedback<p>There's a great post from last week at 37signals, <a href="http://37signals.com/svn/posts/3124-give-it-five-minutes">Give it five minutes</a>:
<blockquote>
<p>While he was making his points on stage, I was taking an inventory of the things I didn’t agree with. And when presented with an opportunity to speak with him, I quickly pushed back at some of his ideas. I must have seemed like such an asshole.</p>
<p>His response changed my life. It was a simple thing. He said “Man, give it five minutes.” I asked him what he meant by that? He said, it’s fine to disagree, it’s fine to push back, it’s great to have strong opinions and beliefs, but give my ideas some time to set in before you’re sure you want to argue against them. “Five minutes” represented “think”, not react. He was totally right. I came into the discussion looking to prove something, not learn something.</p>
<p>There’s also a difference between asking questions and pushing back. Pushing back means you already think you know. Asking questions means you want to know. Ask more questions.</p>
</blockquote></p>
<p>
This is such a great outlook and a great way to approach the discussion of feedback for code reviews and design reviews.
</p>
<p>
It's surprising how little time development teams devote to training, or even internal discussion on effective feedback. As developers, we are constantly engaged in this kind of communication: white-boarding sessions, spec reviews, design reviews, code reviews. We're expected to give and receive feedback on a daily basis, but few of us are properly prepared for it. Not only do we lack the training, but we have many negative examples to draw from. Who hasn't been a part of a design review where tempers flare? Properly giving feedback is something that requires constant attention and practice. Receiving feedback can be just as difficult.
</p>
<h1><span style="font-size: large;">Culture of Communication</span></h1>
<p>One of the major pillars of our <a href="http://dev.bizo.com/2011/03/on-building-kick-ass-engineering-team.html">engineering culture at bizo</a> is "the 3 Cs": Communication, Communication, Communication.</p>
<p>We've tried hard to build a team of engineers that are eager to receive feedback, humble about their abilities, objective and gracious with their feedback, and freely giving of their own knowledge and experience. We see communication as a prerequisite for building a world-class team and developing high-quality code. You often hear the phrase "<a href="http://bobsutton.typepad.com/my_weblog/2006/07/strong_opinions.html">strong opinions, weakly held</a>," and that is the kind of culture we have tried to build.</p>
<p>Communication is hard. It takes real team agreement and commitment to continued work to keep this culture alive and well. It's important the team views effective communication as important and that the culture supports it.</p>
<p>
<h1><span style="font-size: large;">Code Reviews</span></h1>
<p>
Code reviews are something that can easily be approached from the wrong perspective, both as an author or reviewer.</p>
<p>As a reviewer, it can be easy to jump in and argue, to try and push 'your' solution (even though it may be equivalent), to push back instead of asking questions and trying to understand.</p>
<p>As an author, it's far too easy to get attached to your code, to your specific solution/naming/etc. It's also easy to feel like each comment is an attack on your ability, and that by accepting the feedback, this somehow means that you were wrong or did a bad job. Of course, nothing could be further from the truth!</p>
<p>At Bizo, we perform code reviews for every change. They are a major part of our culture of communication. In order to perform effective code reviews, it's important to have some shared guidelines that help support effective communication.</p>
<p>Here are some guidelines we've found to be helpful for performing code reviews:</p>
<h2><span style="font-size: large;">What is a code review</span></h2>
<p><ul>
<li>A careful line-by-line critique of code by peers</li>
<li>happens in a non-threatening context</li>
<li>goal is cooperation and mutual learning, not fault finding</li>
</ul></p>
<p>
Code reviews are a team exercise to improve understanding and make the code better!
</p>
<p>When people think of code reviews they usually think of catching bugs. Code reviews do occasionally catch bugs or potential performance problems, but this is rare.</p>
<p>Just as important is fostering a shared understanding of the code and exposure to new approaches, techniques, and patterns. Seeing how your peers program is a great way to learn from them.</p>
<p>Ensuring coding standards and style guides is another way code reviews help. Working on a team it's important to keep readability and quality high using a shared vocabulary.</p>
<h3><span style="font-size: medium;">As an author</span></h3>
<p>
As an author, it's important to view each comment as a new opportunity to improve your code. Instead of jumping into defense mode, take a step back and think. Try to approach the code again for the first time with this new perspective. Your team has a lot of experience and varied backgrounds -- draw from them! They are there to help you. Use the gift of their experience and knowledge to improve the code.
</p>
<p>
Trust the team, and view all comments as action items. Some changes can seem arbitrary, especially when it comes to naming and organization. Unless there's a strong reason, tend to agree with your reviewers. If a reviewer finds something confusing, it is confusing! Code spends most of its life in maintenance and programming is a team sport. Remember that they are your audience, and you want them to be able to understand your code at 4am after a system crash.
</p>
<h3><span style="font-size: medium;">As a reviewer</span></h3>
<p>
As a reviewer, it's important to take the time to understand the code, think, and ask questions to understand the code before providing feedback. The author probably spent a lot more time thinking about the problem and the approach over the course of the project.
</p>
<p>Be strict on coding standard and style guide violations. The real cost of software is maintenance (<a href="http://www.oracle.com/technetwork/java/javase/documentation/codeconventions-139411.html#16712">80% according to Sun</a>). It's important the code is easily understood by the team.</p>
<p>Be gentle on personal preferences. If it's not a standard violation and just a matter of personal preference, defer to the author. It's okay to present your perspective, but mention that it's just a preference and not meant to be taken as an action item.</p>
<p>
Trust the author. It's often the case that there are many valid approaches to a problem. It's great to present alternative approaches and discuss pros/cons of various approaches. If you see alternative solutions, bring them up! When discussing alternatives, make sure to listen to the author. Remember they are the subject matter expert and you are working together on the same team.
</p>
<h3><span style="font-size: medium;">It takes work!</span></h3>
<p>Communication is hard! It's easy to screw-up. It's easy to go into attack or defense mode when you're passionate about what you're doing. It's really something we all need to remind ourselves to work on every day. It's something we need to periodically remind ourselves as a team. Try to view each review as an opportunity to practice these guidelines. Just remember to take a step back, think, and ask questions.
</p>
larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-58119741373913740502012-03-12T14:03:00.004-07:002012-03-12T14:03:40.236-07:00Fault Tolerant MongoDB on EC2While working on a project at <a href="http://bizo.com/">Bizo</a> I needed to connect a Rails app to a MongoDB backend both of which run in Amazon's Cloud (EC2). At Bizo we have a policy to not use non Amazon services when possible (to limit risk) - so we normally run most of our services straight off of EC2. I'd like to share what I've learned as best practices throughout the experience as I hope it might save some time and frustration for others.<br />
<h1>
<span style="font-size: large;"> Primer</span></h1>
<a href="http://www.mongodb.org/display/DOCS/Replica+Sets">Replica sets</a> are the preferred way to run a distributed, fault tolerant MongoDB service. But as with any distributed system, nodes will eventually fail. Now replica sets are pretty good at handling failures, but they can't save you if too many nodes fail.<br />
Specifically a replica set requires a minimum of two nodes to function at all times (1 primary and 1 secondary node). Thus a good rule of thumb is to run **at least 3 nodes** in a replica set, that way if a node fails your database service doesn't go down with it. The Rails app I was working with doesn't experience enormous amounts of traffic so 3 m1.large (64bit) nodes were sufficient for my needs. What follows is a rundown of our setup and how it handles common needs of fault tolerant systems.<br />
<h1>
<span style="font-size: large;"> Best Practices</span></h1>
<br />
<h1>
<span style="font-size: large;"> Minimize Failure with AutoScaling, Availability Zones, CloudWatch and EBS Volumes<br />
</span></h1>
<ul>
<li> Use Autoscaling Groups, CloudWatch and EBS Volumes to replace failed nodes as soon as they go down. Since we run three nodes, our replica set is insulated from failure due to a single node crashing. But if two nodes crash the replica set goes with them. To solve this we use Cloudwatch alarms to trigger the Autoscaling Group whenever a node goes down - that way a new replacement node is automatically brought online within a few minutes of a failure to reduce the risk of nodes sequentially failing. Additionally each node stores it's data on an EBS Volume (network attachable hard drive) - that way when a node fails, it's replacement doesn't startup with missing data - it simply mounts the previous node's EBS. </li>
<li> To protect against multiple nodes failing simultaneously run each node in a separate availability zone. The above isn't sufficient to protect against things like hardware failures as all 3 instances could wind up on the same hardware. Running each node in a separate availability zone guarantees that our mongo instances run with a reasonable amount of separation (eg. they don't all end up on the same hardware box). Ideally you'd run each node in its own region (separate data center), but this causes headaches trying to configure firewalls as Amazon does not allow security groups to be used across multiple regions (see security below). So unless you want to setup a VPN for cross region communication - you're probably better off just running in separate availability zones.<br />
Assuming you've created the group and are running three nodes, each in a separate availability zone you can configure the auto scaling group using Amazon's <a href="http://aws.amazon.com/developertools/2535">command line tools</a> like so:<br />
<pre><code>
as-update-auto-scaling-group my-mongo-service \
--region us-east-1 \
--availability-zones us-east-1a us-east-1b us-east-1c \
--max-size 3 \
--min-size 3 \
--desired-capacity 3
</code>
</pre>
Now if any node fails then a new one will startup to take its place in the proper availability zone.<br />
</li>
</ul>
<h1>
<span style="font-size: large;"> If Everything Fails, have backups</span></h1>
It's always good to have backups just in case something really bad happens. Fortunately since we use EBS Volumes this is really easy - we create nightly snapshots of the primary node's EBS Volume on our cron server using Amazon's command line tools (ec2-create-snapshot). These snapshots are persisted to S3 and we can easily restore our replica set from these backups.<br />
<h1>
<span style="font-size: large;"> Use Elastic IPs</span></h1>
As nodes fail and are replaced you want both your Replica set and your database clients to be able to find and connect to the new nodes. The easiest way to do this in Amazon is to use Elastic IPs - special* static ip addresses that can be assigned to individual instances. Since each instance runs in a separate availability zone we just need one Elastic IP per zone. When a new node starts up to replace a failed instance, it checks which zone it was started in and assigns itself the matching Elastic IP. Both the client and replica set configuration should point at the Elastic IP address - that way failures and startups of new nodes will be seamless to your app. This is because cross-security group openings in the firewall need to use internal (not external) addresses and the DNS url in the console will resolve to an internal ip address from an EC2 instance.<br />
<h1>
<span style="font-size: large;"> Security</span></h1>
This is where the headaches can start. Ideally you want to restrict access to your MongoDB instances to just your client application using Amazon's security groups. The way we normally set this up is to give your mongo instances a security group, say mongo-db-prod and your client app a security group, say cool-app-prod. Then mongo-db-prod would grant access on port 27017 (default mongodb port) to security group: cool-app-prod. Unfortunately what's not documented very well is that if you use the **external Elastic IP** addresses in your configuration it **will not work** with security groups! Instead you have to use the Elastic IPS DNS url (found in Amazon's web console) for security groups to work properly.<br />
<h1>
<span style="font-size: large;"> A Final Caveat</span></h1>
One thing to be careful of is if you require more than 5 nodes in a replica set you'll run into a problem using Elastic IPs - Amazon by default only allows 5 eips per region. You'll need to either ask Amazon to increase this limit on your account or seek out an alternative setup.<br />
Well, that's it but If you have another setup for running MongoDB on EC2 I'd love to here it. Until next time.Josh Carverhttp://www.blogger.com/profile/15167764329841650102noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-87495162637299193352012-02-17T09:21:00.000-08:002012-02-17T09:30:24.483-08:00Building a Product in Just 8 HoursRecently at Bizo, we decided to try a new kind of hack day. Previously during hackdays our engineers worked individually on their own project(s). But on our last hack day we decided to try something new – The 8 Hour Product Challenge.<br />
We would build and launch a completely new product in the course of a normal workday (9-5pm). “Launching” meant this product had to be running publicly on the internet by 5pm – no excuses, no “Wait! I need 5 more minutes” – whatever was there had to be deployed. In short the experience was fantastic and I can’t wait to do it again. Here’s a breakdown of the experience:<br />
<h2>
Initial Meeting 9:30am</h2>
Organizing developers for a meeting of any kind is like trying to heard cats. But if it’s a meeting before 11:00am you’re not herding regular cats, you’re herding sleepy, fat cats with one leg and half an ear. Somehow after a lot of cattle prodding by our VP of engineering our team eventually managed to shuffle its way into the conference room like the decaffeinated zombies we were and get to work.<br />
We decided to build a stealth product. The system would use Bizo’s rich business data to personalize special content for visitors based on things like their industry, company size and seniority. The goal for the end of the day was to have a small webapp up and running on Amazon’s servers.<br />
After a bit more discussion, we decided to split tasks up into five groups of two engineers:<br />
<ul>
<li><strong>Data Discovery Team</strong> – Find relevant items for users by using our B2B business data & network.</li>
<li><strong>Scraping Team</strong> – given a url representing an item scrape the page contents and store them for later use</li>
<li><strong>Data Classification Team</strong> – extract relevant data from the HTML source of the previously scraped urls.</li>
<li><strong>Backend Team</strong> – backend architecture for the webapp that fetches serves the content from that was generated in the previous steps</li>
<li><strong>Frontend Team</strong> – frontend design + javascript that makes the app functional</li>
</ul>
Engineers were assigned more or less randomly, with the exception of myself – I was assigned to the frontend team directly.
During the course of our initial planning meeting, we (myself included) often found ourselves becoming sidetracked with feature bloat, premature scalability concerns and a myriad of other things not essential to our MVP. Fortunately one of my coworkers (Stephen) was smart enough to enforce timeboxing the meeting to one hour – eventually we got things back on topic and designed the critical components before time ran out.<br />
<h2>
Start Work 10:30am</h2>
Once the teams were assigned, we all jumped in and started to work on our relevant tasks. My partner, Darren and I immediately started out by sketching ideas on paper for our design. I can’t stress enough how important sketching is for being able to rapidly prototype a product – a trick I picked up back when I interned over at ZURB. Only after we had some solid sketches did we move into Photoshop mockups. Meanwhile the other teams were all furiously programming their parts of the application:<br />
<ul>
<li>Data Discovery Team Worked out a simple ranking algorithm for data and started writing the Hive script to extract it.</li>
<li>URL Scraping Team Started out in Scala hacking up a script to scrape & download url content.</li>
<li>Data Extraction Team Decided to try out the Pismo gem to extract summaries and titles from scraped html content.</li>
<li>Backend Team Was working on getting a sweet Scalatra webapp up and running</li>
</ul>
<h2>
Lunch Time & Status Updates 12:30pm</h2>
By lunchtime everything seemed to be coming along nicely. On the frontend had completed our Photoshop mockup and had just began writing some basic css styles. All the other teams reported making good progress on their tasks, with no major snags in the foreseeable future (betcha you never heard that one before…).<br />
<h2>
Afternoon 1:30pm</h2>
My frontend partner and I powered through our post lunch food coma and were able to move into begin wiring up the ui using CoffeeScript in conjunction with Dependence.js. My teammate and I decided to give pair programming a shot. He has always been more of an Emacs kind of guy, while I prefer vi, but in the interests of learning I decided to try Emacs for the rest of the day – I now know why he’s always so worried about contracting carpel tunnel syndrome :).<br />
Using some sample data generated by the first three teams we were able to get a rough ui working pretty quickly. Our side of things turned out to be pretty straightforward and involved three AJAX requests. One was to retrieve a list of items grouped by segment from the Scalatra web server, one to get a list of the current targetable segments from the Bizo API and another call to the API to retrieve a visitor’s business segments (bizographics).<br />
The only snag we hit was running into a race condition – originally we attempted executing all three requests simultaneously. In reality we had to wait to get the list of segments before getting the visitor’s profile. Darren and I just looked at each other and shrugged, then we indented the third API request a few spaces in our CoffeeScript code – race condition solved! Yes that’s correct we fixed a race condition by indenting some code, don’t judge – it was a hackday.<br />
The other teams all seemed to be doing well, the scraping team discovered the Scala Collection’s magical par(), which turns normal data structures into parallel ones, they almost peed their pants with joy. At this point the backend team had completed the Scalatra app and was working on setting up our eventual deployment to EC2 using our custom infrastructure, cowboy.<br />
<h2>
The Home Stretch & Deployment 4:30pm</h2>
Right around 4:30 we ran into a major problem. There had been a miscommunication regarding the necessary format of the JSON file needed by the frontend, and our data was coming through to the app in a format that just wouldn’t work. We had to scramble and hash things out with the other teams quickly before we hit our 5pm deadline. Thanks to a major push by the Data Extraction Team we were finally able to get everything in place. The product worked! – it wasn’t the most polished app ever, but what we had accomplished in just one day was pretty amazing. The product was deployed on EC2 and presented internally within Bizo – it was met with a lot of excitement and Bizo will probably be releasing it publicly in the weeks to come.<br />
<h2>
Closing Thoughts</h2>
Overall the experiment turned out to be a smash hit. Looking back on the experience there are a bunch of things we could have done better. In retrospect we got sidetracked a bit too much on non essential feature ideas when we really should have been spending time clarifying the format of the data as it passed through each team’s project – this was something that came back and bit us near the end of the day. But mishaps aside it’s really amazing what you can accomplish with 9 other talented people in a single day.<br />
I can’t recommend trying this with your team enough – what do you think, is your engineering team up for The 8 Hour Product Challenge? If you’re interested in the technologies we used throughout the day, see below.<br />
<h2>
Appendix, Technologies Used</h2>
<ul>
<li>News Discovery Team – Hive, Amazon EC2, Amazon S3</li>
<li>URL Scraping Team – Scala</li>
<li>Data Extraction Team – jRuby, gems of note: pismo, right_aws (for Amazon S3)</li>
<li>Backend Team – Scalatra, Amazon S3</li>
<li>Frontend Team – Photoshop, HTML5, Sass, CoffeeScript, jQuery, dependence.js</li>
</ul>Josh Carverhttp://www.blogger.com/profile/15167764329841650102noreply@blogger.com0tag:blogger.com,1999:blog-5261056907132640554.post-21265811800889868572012-01-30T19:38:00.001-08:002012-01-30T19:50:57.437-08:00work at Bizo (looking for some good engineers)<p>We’re a small, disciplined team that gets a lot done. Our platform processes billions of page views monthly and 100s of terabytes of data so we have lots of fun problems to tackle. We believe in <a href="http://dev.bizo.com/2011/03/on-building-kick-ass-engineering-team.html">teamwork and communication</a>: comments, design reviews, code reviews for every change, weekly tech talks. We believe in giving developers ownership over projects. We believe Engineering is more than coding. We have fun and keep the beer fridge well stocked.</p>
<p>We have customers, are well funded and recently named the forth fastest growing private company in the San Francisco Bay Area.</p>
<p>We are looking for motivated problem solvers with an entrepreneurial / hacker spirit.</p>
<p>If you're a reader of this blog, you already know our technology stack. Some highlights: Scala, Java, Javscript, Ruby, AWS (pretty much every service), Hadoop/Hive, GWT, MongoDB, Solr, etc.</p>
<p>If you're interested, please <a href="http://careers.stackoverflow.com/jobs/16330/are-you-a-mind-bending-engineer-bizo">apply on stackoverflow</a>.
</p>
larry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.com0