Panasonic Youth rob sanheim writes about software, business, ruby, music, stuff and things

Testing Velocity Part 2 - Why do we test?

A couple weeks ago, I began a series on keeping your test suite fast and effective. I now am going to digress a bit, take a step back and view the big picture to establish context.

Before addressing test performance and what makes up a good test, we should ask ourselves why is it that we write tests at all? If we want to be effective, we should always stay conscious of the overall goal of testing, as well as the specific goals behind each test in context.

Some would argue that tests are primarily a design tool. Or that tests are a living, breathing, specification for our code. Others would say it’s primarily a means to drive and maintain quality. Some may say that tests are useful to ensure that the really difficult parts of our system work, or to keep lax developers in line.

Testing is *only* valuable and useful insofar as it supports software as a cooperative game. When you think “cooperative game”, imagine rock-climbing, or ? To quote Alistair Cockburn:

Software development is a (resource-limited) cooperative game of invention and communication. The primary goal of the game is to deliver useful, working software. The secondary goal, the residue of the game, is to set up for the next game. The next game may be to alter or replace the system or to create a neighboring system.

So is testing for design, or quality, or correctness, or communication? The answer is that it’s for *all* of those things (and more) as long as it helps deliver working software and prepare for the next game! So when someone asks, “Why do you write tests? Why do you care about the speed of your tests?” The first answer is, “It depends.” It depends on the software you are delivering, on the teammates and domain you are working with, and on what the next game (if any) is. Since every software project will be different, clearly how you write the code and its tests will differ. An embedded system targeting your phone will have a much different test suite than a large enterprise web app.

This digression is to clarify debates I’ve heard (and been involved in) over issues like what should a “proper” unit test should do, how much setup is okay, mocks or stubs or fake objects, and when it’s okay to mock/stub versus when it’s not okay. There are not hard and fast answers to any of these questions. You need to consider context: What is being built? What are the current technical issues? Does the client want to run (or create) acceptance tests? How slow is the test suite currently, and is it impacting dev speed? How large is the system, and how much larger do you expect it to be? Who will be maintaining the system after it’s released, and what will their skill level be?

Answer those types of questions before making statements like “this spec is doing too much setup and runs too slow,” or “we shouldn’t be stubbing in a functional test like this.” It will ensure that your discussions and debates will stay grounded and useful instead of becoming endless religious debates. Keep context and the cooperative game model of software in the back of your mind, and I will too as I continue this series and try to lay out some practical, overall guidelines.

CapGun Released! Super simple Capistrano deployment notifications

Tell everyone about your releases! Send email notification after Capistrano deployments! Rule the world!

Drop your ActionMailer configuration information into your deploy.rb file, configure recipients for the deployment notifications, and setup the callback task.

Setup and configuration are done entirely inside your deploy.rb file to keep it super simple. Your emails are sent locally from the box performing the deployment, but CapGun queries the server to grab the necessary release info.

This even includes the Net::SMTP TLS hack inside as a vendored dependancy to allow super easy email sending without setting up an MTA.

This is the “Rock n Roll McDonalds” release - 0.0.1.

INSTALL

• sudo gem install cap_gun and gem unpack into your vendor/plugins
• or just grab the tarball from github (see below)
• see the readme and rdocs for more config notes

URLS

• Log bugs, issues, and suggestions on Trac: http://opensource.thinkrelevance.com/wiki/cap_gun
• View source: http://github.com/relevance/cap_gun/tree/master
• Git clone source: git://github.com/relevance/cap_gun.git
• RDocs: http://thinkrelevance.rubyforge.org/cap_gun/

history meme onwards…

rsanheim@ares:~$ history 1000 | awk '{a[$2]++}END{for(i in a){print a[i] ” ” i}}’ | sort -rn | head
71 gst
67 l
38 ****
38 c
28 git
22 ss
21 gca
13 gsd
13 :
12 rake


I like keeping it short. The 3rd one thats bleeped out is a shortcut to my main client project right now. The aliases can be found in our open source’d etc stash on github.

Tagging Muness, Jason, Aaron, and Chris.

If you think Huddle is ripping off Campfire

…then you haven’t seen mIRC, an irc client app that has been around for years:

See anything familiar in the screen shot above?

Tabs on the top, list of participants in the right hand pane, and chat in the main window. Change the font to Lucida Grande 12 pt and put in a browser and you pretty much have the Campfire l&f, minus a whole lot of polish and design love. Chat applications tend to follow a pretty standard UI model, and the fact that Huddle took the 37 Signals style is a good thing for Jason Fried and co.

Here’s why: The look and feel from Basecamp and Campfire have become the defacto “template” for web 2.0 apps without a design of their own. There is a good reason for that: 37 Signals develops kickass UI’s. As long as they stay on top of their game and keep their customers happy, they have nothing to worry about from people ripping off their design. The customer who leaves your app for a free one because the CSS is the same is a customer you don’t really want anyways.

Here’s the standard process whenever this becomes public and everyone gets up in arms about it:

• 37 Signals gets a ton of free press and attention.
• Paid users for the app in question (campfire in this instance) shrug and say “huh, I’m pretty happy with my service, and I’m not going to switch.
• People new to both apps who are in the market try both and choose on their merits.
• The user experience for the 37 Signals’ apps remains superior in most cases, as they are continuously tweaking and improving while copycats are stagnant or trying to keep up.

What exactly is the problem here?

The more interesting point about Huddle, and Google App Engine in general, is the lack of diversity the platform provides for. But please, drop the “Google ripped off 37 Signals oh noess!!!!” angle.

Zero to a Fully Git Enabled Rails App in the time it takes to drink an espresso

updated: now uses the real Rails git master at github, now that its live.

So you want to set up a fresh Rails app in a fresh git repo, with proper ignores setup, with vendor/rails using a git submodule (which enables switching to any Rails branch or tag locally)? This isn’t rocket science or anything, but I figured I’d post it to see what things could be done better.

Assumptions

• You have a working, recent version of git installed.
• You have the git-rails gem installed.
• You have the bash aliases at the bottom of the post.


rails app_name
cd app_name

cd vendor/rails
gba # pick branch of Rails you wanna use
git co BRANCH # where BRANCH is the specific branch you want, unless you want the default of going against edge


You’re done!

Here’s those aliases you need:


alias g=’git’
alias gb=’git branch’
alias gba=’git branch -a’
alias gc=’git commit -v’
alias gca=’git commit -v -a’
alias gd=’git diff | mate’
alias gl=’git pull’
alias gp=’git push’


log_buddy 0.0.2 released

Description

log_buddy is your friendly little log buddy at your side, helping you dev, debug, and test.

Synopsis

Call LogBuddy.init to use log_buddy. It will add two methods to object instance and class level: “d” and “logger”. You can
use your own logger with Logbuddy by passing it into init’s options hash:

LogBuddy.init :default_logger => Logger.new(’my_log.log’)

Now you have your logger available from any object, at the instance level and class level:

obj = Object.new
obj.logger.debug(”hi”)
class MyClass; end
MyClass.logger.info(”heya”)

You also have a method called “d” (for “debug”) on any object, which is used for quick debugging and logging of things while you are developing. Its especially useful while using autotest. When you call the “d” method with an inline block, it will log the name of the things
in the block and the result. Examples:

Changes

0.0.2
* rdocs
* support for multiple statements in one “d” call separated by semicolons

Installation

sudo gem install log_buddy
full docs here

Rails is moving to Git (and Lighthouse)

The Rails core team is making a big move to Git (and Github) for source control and Lighthouse for issue tracking. I love the move to github - it will make managing patches and your small tweaks to Rails much easier to keep in sync with your chosen branch of Rails. The move to Lighthouse I’m not sold on, only because I haven’t felt the big benefits it offers over Trac, though I haven’t really used Lighthouse much over the past few months.

We’ve been moving all internal things at Relevance to Git and Github, and so far we’ve been very happy with the move. Local commits are great, git is blazing fast, and github is the bees knees. If you haven’t learned Git yet, now is the time.

How to move your domains off GoDaddy

Its common internet knowledge that GoDaddy is one of the worst, most unfriendly registrars out there. Go to NoDaddy for all sorts of nasty things that GoDaddy that make them completely disreputable, or google “godaddy sucks” for a thousand other reasons.

What’s less common is knowing how to actually get your domains off GoDaddy once you’ve had enough and have smartened up about finding a good registrar. Its simple once you know the process and can ignore the five thousand gratuitous options that godaddy has on their control panel.

First, you must make sure your domain is unlocked.

• go to your domain control panel
• click the checkbox(s) next to the domains you want to unlock
• click the “Locking” button
• choose “unlock” and click ok

That will take a few minutes to process…in the interim go sign up for a NameCheap account (referral link) so you can have a nice, usable control panel and a reputable registrar. I’ve also heard good things about dotster, but have not used them personally.

Now, once your domain has been unlocked at godaddy, go back to the control panel there and get an authorization code:

• Click on the domain name itself that you want to transfer
• You are now on the “show details” page from hell — the link you want is Authorization Code: Send by Email - click on that “send by email” link, then click OK in the confirmation page that comes up.
• Check your email in a few minutes, and you’ll have the auth code (aka EPP Code) you need! Use that code at the new registar you want to transfer to, and you’ll be off godaddy soon enough

There is a helpful page here showing this process with screenshots, but it loads very slow. Here’s a cached copy that may work better.

I’m happy to say the last of my domains will be off godaddy by tomorrow, and I’ll be 100% godaddy free.

log_buddy Released - your helpful little dev buddy

LogBuddy is your friendly little log buddy at your side, helping you dev, debug, and test. It plays well with Rails and plain old Ruby projects. To use it, sudo gem install log_buddy, then require 'log_buddy' and call LogBuddy.init. It will add two methods to object instance and class level: "d" and "logger". You probably only want to use it in non-production environments.

• The "logger" method is just a typical logger - it will use the Rails logger if its available.
• The "d" method is a special helper that will output the code in the block and its result - note that you *must* use the bracket block form - do...end is not supported.

Examples

More Details

• Log bugs/issues/suggestions here: http://opensource.thinkrelevance.com/wiki/log_buddy
• Source: http://github.com/relevance/logbuddy/tree/master
• git clone git://github.com/relevance/logbuddy.git
• rdocs: http://thinkrelevance.rubyforge.org/log_buddy/

Testing Velocity - Keeping your test suite fast, Part 1

If you are a Ruby or Rails developer, and you know what you are doing, you are writing tests or specs. Tests express the intent of your code, help verify correctness, and aid in design and exploration. Rails gives you helpful conventions to follow and functional and integration testing support for free out of the box. Ruby's power and flexibility allows you to reach in and alter objects at runtime so you can easily mock and stub external dependancies. In all the discussion and attention giving to tests versus specs, mocks and stubs, fixtures, and test coverage, one thing that doesn't nearly get enough attention is the importance of your test suite's speed.

Why does test speed matter? Consider at the most basic level what a test is: the smallest level of automated feedback in your application. You can view a typical agile application as a series of concentric feedback circles, the smallest being unit tests at the method level, up to story level tests (ie acceptance or integration), on to daily standups, iteration planning/review and release planning and retrospectives. The more you can tighten and tune all those feedback loops, from the micro level to macro, the more you can quickly respond to change and complete quality features that your app's users will really use and love.

If you've ever worked on an larger project with a lot of slow tests, you've probably seen the "slow tests antipattern": developers stop running the whole suite after small changes, because it takes too long and they get bored or distracted while they wait. Changes that require many small steps, such as major refactorings or API changes, become too burdensome to do regularly. Design, quality, and the code base as a whole suffer as a result. In the worst cases, developers stop running the test suite interactively and let a nightly build run it, or maybe the tests are abandoned altogether. Tests aren't run before checking code, and all the value from automated tests/specs is lost. Abandon all hope, ye who enter here.

Ruby, particularly in the current 1.8.6 MRI implementation, isn't exactly known for its peformance. Just ask the Rails Envy guys and they will tell you - "Rails doesn't scale". Maybe Rubinius or JRuby can rescue your slow tests a year or two from now when new virtual machines bring major improvements, but for right now it primarily comes down to you and your team's development practices.

What is a "fast" test suite? It depends on project size of course, but generally unit tests should run in the tens of seconds or less, and the full suite (including functional+integration) should be at most a few minutes. Browser-based acceptance level tests using something like Selenium are typically outside of the "interactive tests" group that I'm considering here, and take much longer. If your unit tests alone take a minute or two to run you will have some serious obstacles, if not now then later as your code base and technical debt increase.

So your test speed matters if you anticipate your app growing beyond 500 lines of code. Ruby is built for developer productivity and not machine productivity, but in order to sustain developer productivity we need to keep our tests running fast.

Future posts in this series will discuss different ideological and technical approaches to keeping you tests running fast. What does Rails consider a "unit" and "functional" test, and does it make sense? What is the actual goal of a good unit test? Should we use unit_record, nulldb, or something else? What about plain Rails2 fixtures versus fixture replacement, or maybe a home grown factory? Do YARV or JRuby offer some hope for the near future for some easy, free improvements in runtime?

I'm curious as to know what the "average" Rails test suite clocks in at, and what level of coverage it attains. Is your test slowness hampering your development? Do you rely on autotest, focused tests, or something else to keep the tests you run relevant? Please do share your thoughts on test speed and what sorts of issues this series should cover in upcoming posts.