Scala Test Plug-in for Sublime Text 2

I have documented and put some polish on the Sublime Text 2 plug-in I blogged about previously.  It lets you run a single Scala Test, or all tests in your project.  It also lets you quickly navigate to any scala files in your project folder, and switch back and forth between a class and its test.  Check it out here: https://github.com/patgannon/sublimetext-scalatest

Dev Days: Hacking, Open Source and Docs

Dev Days

Every month we have a "Dev Day" where engineers take a break from their projects and work on "other stuff".  Most start-up engineering teams have a "Hack Day" where everyone gets to hack on anything they want as long as they ship and share it with the rest of the team.  Of course we have Hack Days but we also have other types of Dev Days too.  In fact, we have three types of Dev Days:

• Hack Days 
• Open Source Days 
• Doc Days

Open Source Days

You know what Hack Days are so I'll move on quickly to Open Source Days.  Just like most companies these days, Bizo uses a lot of open source software (OSS).  We love OSS and the community of developers and companies that share it.  Over the last few years, we've used plenty of OSS but we've also created and given back lots of code as well.

Actually today is one of our Open Source Days so all the engineers are working on both new and old open source projects.  You can check out our (growing) list of projects by visiting code.bizo.com.  Over the years, we've created a lot of tools around AWS including s3cp, fakesdb, aws-tools (package of all CLI tools).  We've also built a lot of stuff for Hadoop (Hive, etc) including csv-serde, gdata-storagehandler and our latest is a scala query language called revolute (still in development).  In addition, we've have a wide variety of other awesome code including the awesome Joist, dependence.js, raphy-charts and other fun stuff!

Doc Days

The third type of Dev Day we have is called Doc Days.  I know that you are thinking but Doc Days are extremely valuable days for engineering and everyone else for that matter.  On Doc Day the entire engineering team works on wiki pages, code documentation, design docs, architecture docs and even blog posts.  It really is better than it sounds!  

If you've read my post on "building a kick ass engineering team", you know that one of the keys is the 3Cs...  Communication, Communication, Communication!  (My high school baseball coach taught me that one.)  As a engineering team, we believe that communication is one of the best things we can do for each other.  As any company grows communication becomes a larger and larger part of day to day and we see Doc Days as a way to ensure that we are communicating as clearly and accurately as we can.

Conclusion

These Dev Days have been a huge success for Bizo engineering.  We've even inspired other departments to have similar days (Marketing in particularly like these documentation days!).    We challenge you to go beyond the "Hack Day" and start thinking about other Dev Days that your engineering organization can benefit from.  

Implementation driven interfaces?

I've recently encountered some interesting pagination in the Google Groups admin interface. It starts off simple enough, nothing exciting here...

Instead of the usual 'Previous', we see 'First' on the next page.

Are they just being clever? Knowing that there's only one previous page? No such luck…

We've reached the end of the list. I hope you've found what you're looking for, otherwise start over from the beginning!

One has to wonder, who designed this interface? You can only go forward. If you overshoot, it's back to square one, then click, click, click… It's clear it does not have users in mind at all.

My guess is that it's based on some limitation in the backend storage or query mechanism. The system only allows forward navigation of query results, so the interface simply mirrors that…

What an incredibly frustrating experience! I'll never take simple pagination for granted again.

It's a good reminder to think about your users and how they will interact with the system. Mirroring the programming interface rarely works.

Capturing Client Side JS Errors on AWS

I saw a post go by on Hacker News this morning discussing capturing and reporting on client side errors. We have been doing this for a long time and I wanted to share our approach.

Background
Quick background, we have two major types of javascript that our customers and partners may use: analytics tags and ad tags. Both tags are javascript and share the same error capture code.

Another quick note is that we run on Amazon Web Services so this approach is based on some of these services including S3, CloudFront and EMR.

Implementation
Our client side JS is compiled from Coffeescript. I've created a couple of gists to show you what the error logging code looks like in Coffeescript.




Details
The example shows our ad tags trying to execute surrounded by a try/catch that captures the error and eventually results in loading an image appending the relevant error metadata.

AWS Details
The image that is loaded actually lives on CloudFront. The CloudFront distribution is setup with logging which means that requests are logged and delivered to a specified S3 bucket (usually within 24 hours). Every day we run an EMR job against the CloudFront request logs that generates a report summarizing the errors. And that is it. Pretty simple and this approach has worked for us.

Pre-emptive "this isn't perfect" response
Some of you may be thinking, "you may not get all requests!". CloudFront logs are not supposed to be used for 100% accurate reporting (although nothing is really 100%). In our case, we don't need to capture all errors rather we are looking for directional information.

Creating Plug-ins for Sublime Text 2

I have been trying out Sublime Text 2 as my text editor lately, and I'm loving the simplicity, so I figured I would try out creating a plug-in for it. I was pleasantly surprised at how easy it is, which is an important step towards it becoming my new editor of choice. I wanted to take some steps towards creating something along the lines of rinari, but for Scala... in Sublime Text. I was able to fairly easily easily create a plug-in that allowed me to run the Scala Test that was currently open in the editor, or run all Scala Tests in the (inferred) project folder, or switch back and forth between a test and the code under test, or quickly navigate to any scala file in the project folder with a few keystrokes. This post will show you how to create a new plug-in for Sublime Text 2, which uses all the API features that I needed to implement that functionality.

Create a new plug-in

Step 1. Install Sublime Text 2 (see link above). Its free to try, and fairly cheap to buy. A month or so after you download it, it basically becomes nag-ware until you finally manage to overcome your stingy developer impulses and plunk down the $59 to buy it. Also, unlike other similar text editors (ahem.. TextMate!) it actually runs on Windows and Linux, as well as Mac OSX.

Step 2. Create a new folder for your plug-in. On Mac OSX, this goes under your home folder in ~/Library/Application Support/Sublime Text 2/Packages/{PLUGIN_NAME} (where in my case, {PLUGIN_NAME} was "ScalaTest").

Step 3. Create a python file which will contain the code for the plug-in. (Name it whatever you want, as long as it ends in ".py" ;-) Here is a really basic plug-in (borrowed from this plug-in tutorial, which you should read after this):

import sublime, sublime_plugin

class ExampleCommand(sublime_plugin.TextCommand):
  def run(self, edit):
    self.view.insert(edit, 0, "Hello, World!")

Right, so, as I mentioned: Sublime Text 2 plug-ins are written in Python. Don't worry too much if you're not familiar with Python... I wasn't either prior to starting this experiment, and it didn't prove to be too much a problem. (I did have a couple Python books laying around, but I'm sure the same information is on the tubes.) Its fairly easy to pick up, and has some similarities to Ruby, in case that helps. So the code above creates a command called "example" which is defined by a class that inherits from Sublime Text's "TextCommand" class. (Sublime Text 2 maps the title-case class names to underscore-delimited command names, and strips the "Command" suffix.) All the plug-in does is insert the text "Hello, World!" at the beginning of the file open in the editor.

(Note: Sublime Text 2 will detect that you created a Python file under its plug-in folder and automatically loads it.)

Step 4. Run your example. Hit Ctrl+Backtick to open the python interpreter within Sublime Text 2. Run your command by typing in this:

view.run_command("example")

The open buffer will now include the aforementioned greeting. You could bind it to a key-combination easily enough, but hey, it doesn't do anything cool yet, right?, so we'll hold off on the key bindings until the end.

Make it do something cool

So that you can see these approaches in action, I uploaded my nascent ScalaTest plug-in to github:https://github.com/patgannon/sublimetext-scalatest. Note that this plug-in will currently only work with projects that use Bizo's standard folder structure, and has a hard coded path to the scala executable, so its not ready to be used as-is. I hope to clean it up in the future and make it more generically applicable, but for now, I've only shared it to add a bit more color to the code snippets in this section.

Run a command on the current file

The name of the file currently open in the editor can be obtained with this expression: self.view.file_name(). In my plug-in, I use that to infer a class name, the project root folder, and path to the associated test (using simple string operations).

You can create an output panel (in which to render the results of running a command on the open file) by calling: self.window.run_command("show_panel", {"panel": "output.tests"}) (where "output.tests" is specific to your plug-in). In my plug-in, I created the helper methods below to show the panel and clear out its contents. (See the BaseScalaTestCommand class in run_scala_test.py). Note that this code was derived from code I found in theSublime Text 2 Ruby Tests plug-in.

 def show_tests_panel(self):

  if not hasattr(self, 'output_view'):

   self.output_view = self.window().get_output_panel("tests")

  self.clear_test_view()


  self.window().run_command("show_panel", {"panel": "output.tests"})



 def clear_test_view(self):

  self.output_view.set_read_only(False)

  edit = self.output_view.begin_edit()


  self.output_view.erase(edit, sublime.Region(0, self.output_view.size()))

  self.output_view.end_edit(edit)

  self.output_view.set_read_only(True)

(Note: I don't recommend copy/pasting code directly from this blog post, because the examples are pasted in from github, which messes up the indentation, which is a real problem in Python; instead, clone the github repository and copy/paste from the real file on your machine.)

To actually execute the command, I use this code in my run method, after calling show_tests_panel defined above (note that you will need to import 'subprocess' and 'thread' at the top of your plug-in file):

  self.proc = subprocess.Popen("{my command}", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

  thread.start_new_thread(self.read_stdout, ())

...where {my command} is the shell command I want to execute, and read_stdout is a method I defined which copies the output from the process and puts it into the output panel. Its defined as follows (and calls the append_data method, also defined below):

 def read_stdout(self):

  while True:

   data = os.read(self.proc.stdout.fileno(), 2**15)



   if data != "":

    sublime.set_timeout(functools.partial(self.append_data, self.proc, data), 0)


   else:

    self.proc.stdout.close()

    break

 def append_data(self, proc, data):

  self.output_view.set_read_only(False)

  edit = self.output_view.begin_edit()

  self.output_view.insert(edit, self.output_view.size(), data)


  self.output_view.end_edit(edit)

  self.output_view.set_read_only(True)

(Note: Depending on the command you're running, you may also want to capture the process' stderr output, and also put that into the output panel, using a variation of the approach above.)

Using the "quick panel" to search for files, and opening files

The "quick panel" (the drop-down which lists files when you hit command-T in sublime-text) can be extended to have plug-in specific functionality, which I used to create a hot-key for quickly navigating to any Scala file under my project folder. (See the JumpToScalaFile class in run_scala_test.py.) One of the plug-in examples I saw using the quick panel sub-classed sublime_plugin.WindowCommand instead of TextCommand. This results in a plug-in which can be run without any files being open. The flip side of that, though, is you don't get the file name of the currently open file, which in my case, is required to infer the base project folder for which to search for files. Thus, all my plug-ins sub-class TextCommand. To open the quick panel, execute: sublime.active_window().show_quick_panel(file_names, self.file_selected). file_names should be a collection of the (string) entries to show in the quick panel. Note that the entries don't have to be file paths, just a convenient identifier to show the user (in my case, the class name). file_selected is a method you will define which will be called when a user selects an entry in the quick panel. Here's how I defined it:

 def file_selected(self, selected_index):

  if selected_index != -1:

   sublime.active_window().open_file(self.files[selected_index])

self.files is an array I created when populating the quick panel which maps an index in the quick panel to a file path. I then use sublime.active_window().open_file to open that file in Sublime Text.

I also used that same method (open_file) in the plug-in that automatically navigates back and forth between a test file and the code under test. That plug-in also makes use of the sublime.error_message method, which will display an error message to the user (if no test is found, for example).

Create keystroke bindings

To bind your new plug-in commands to keystrokes, create a file in your plug-in folder called Default (OSX).sublime-keymap. This will contain the keystrokes that will be used on Mac OSX. (You would create separate files for use on Windows and Linux.) It is a simple JSON file that maps keystrokes to commands. Lets see an example:

[

 { "keys": ["super+shift+e"], "command": "jump_to_scala_file" }

]

This example will bind Command+Shift+e to the "jump_to_scala_file" command (defined in the JumpToScalaFileCommand class in any plug-in). If you have multiple key-mappings, you would create multiple comma-delimited entries within the JSON array. (See the example in my plug-in.) In order to reduce the possibility of defining keystrokes that collide with keystrokes from other plug-ins, I defined mine in such a way that they're only available when the currently open file is a Scala file. Here is the rather verbose (ahem, powerful) syntax that I used to do that:

[

 { "keys": ["super+shift+e"], "command": "jump_to_scala_file", 

  "context" : [{"key": "selector", "operator": "equal", "operand": "source.scala", "match_all": true}]}


]

Conclusion

Over the years, I've grown to prefer light-weight editors (such as emacs or Sublime Text 2) over more heavy-weight IDEs (such as Eclipse or Visual Studio) because they don't tend to lock up in the middle of writing code and/or crash sporadically, and I generally don't need a lot of whiz-bang features when I'm coding these days. I used emacs (and rinari) for doing rails development for a year or so, but the basic key-strokes (compared to the de-facto text editing standard key-strokes) and the undo/redo functionality always seemed a bit awkward, especially when you wind up switching back and forth between that and other text editors. Also, the language for creating plug-ins is Scheme (a dialect of Lisp), which to me isn't very convenient for these sort of things.

I was really pleased with my foray into creating plug-ins for Sublime Text 2, and combined with its general ease of use, I've decided its now my new favorite editor. Using an editor that's this easy to significantly customize seems like it could be a real productivity win over time. Given the fairly rich list of plug-ins already available, I think the future is bright for Sublime Text 2. Below are a list of resources I found helpful during this process, including said list of plug-ins.

Resources

Unofficial documentation: http://sublimetext.info/docs/en/index.html especiallyhttp://sublimetext.info/docs/en/extensibility/plugins.html
Official plug-in examples (sparse): http://www.sublimetext.com/docs/plugin-examples
Official API reference: http://www.sublimetext.com/docs/2/api_reference.html

Helpful examples:
https://github.com/maltize/sublime-text-2-ruby-tests
https://github.com/noklesta/SublimeRailsNav
https://github.com/luqman/SublimeText2RailsRelatedFiles
https://github.com/rspec/rspec-tmbundle

Unofficial list of plug-ins:
http://wbond.net/sublime_packages/community