# Articles

• ### Using Ruby Enumerators for Resumable Iterators

March 11, 2018

Here's a power technique: You can use Ruby Enumerators to create "resumable" iterators when you're extracting data from the application console.

• ### On Call Duty

March 11, 2018

Recently, there was a flurry of tweets about on-call and SRE duty schedules. It triggered a memory of a conversation at a manager meeting about a new product offering:

• ### Our Hiring Process

July 30, 2017

Here's how the engineering hiring process works at PacerPro.

• ### Do Not Push to Red Builds

July 8, 2017

There's a pattern to disasters: When you examine the post mortems, its not just one failure, but a cascade of multiple failures that overwhelm the system's safety features. Pushing to a red build risks piling one failure upon another.

• ### Wanted: Jaguar Priest

July 7, 2017

My job is to be the dumbest person in the room; I want to work with intelligent, opinionated people who can make the world a little better. Then iterate.

• ### Performant Large-Scale Postgres Migrations

November 30, 2016

Recently I tried to add a column to one of our very big tables, and as you might expect from the title of this blog, it took so long that I aborted it after an hour. Can you spot the problem?

• ### Sacrificial Test Classes

November 10, 2016

Today I learned that there’s a difference between a throw-away class and a sacrificial class. When writing rspec tests, you want the latter; otherwise you might create some very hard-to-debug test pollution.

• ### Memory Leak Whack-a-Mole

October 27, 2016

Memory leaks happen. They shouldn’t, but you just can’t squash enough of them. Or the leak is so slow that it takes hours to manifest. The trouble with memory leaks is that they are permanent for the life of a process. Once a process has expanded its VSZ, there’s no way in Unix to shrink it. So your process starts to bloat, eventually your servers page and thrash. Performance tanks. Then your pager starts going off and there goes your beauty sleep.

• ### Digital Preservation Lasts Until the Heat Death of the Universe or Five Years

October 4, 2016

(Whichever comes first)

• ### Anatomy of an Apology

September 22, 2016

I’ve written my fair share of apologies. It’s a skill, like any other, one that you wish you didn’t have to practice. Yesterday, I had to write a mea culpa to our users. It was our fault, and we came clean about it. I got compliments on the letter! I think I increased customer loyalty as a result. Fuck-ups don’t have to hurt your business.

• ### Guidelines for Using Pivotal Tracker

July 25, 2016

I am often asked about the meaning of things in Pivotal Tracker, such as “What’s the difference between a bug and a chore?” Or, “How can I tell the difference between a bug and a feature?” Another oldie but a goodie is, “What story should I pick from the backlog?” Here are my biased answers.

• ### The Syntax Error That Killed Our Build for a Week.

December 12, 2015

Back in the day when I wrote in C and we used X11 with Motif, I was creating a new module based on an existing one, so I just copied and pasted the file from one xterm to another:

• ### Zero Downtime Migrations

August 15, 2015

For any sufficiently large application, you want to minimize interruptions to service while deploying new code. This is especially challenging for migrations on Heroku and other Platforms-as-a-Service, where you have a Catch-22 problem; you want to run your migrations, first, but you can’t run them until you’ve deployed the migrations to production. Usually, at this point, your new migration-dependent features are in the deploy, too. So when your application restarts, and the migration hasn’t run, yet, your poor application will trip some exceptions, and perhaps, create problems for your users. Of course, you could put the app into maintenance mode, but that creates more downtime for your users.

• ### Unintended consequences (and a disaster averted)

January 6, 2015

I want to tell you a scary-as-shit story, mostly so that my friends that have CCW permits might read it and not get killed by good intentions, because I love you all.

• ### Timecop time and database time

June 8, 2014

Timecop is a great addition to your testing toolbox, but if you’ve ever tried to use Timecop to interact with a database, and then got the most mysterious of existential errors:

June 1, 2014

Once a week, we run a query on our production system that takes a very long time to complete. Invariably, it times out because the default settings on Heroku are (quite reasonably) for web / API interactions; anything longer than 30 seconds is too long and the connection should be closed to let other requests through. How do you get around this, then? You could set up your database.yml file so that you connect to your production database from a developer workstation. This is actually quite useful, but opens up a hellacious security hole. Let’s not get started about the consequences of accidentally running rake db:drop:all! Moreover, you want your system to be automated as much as possible, and that means running your tasks on a Heroku instance. Fortunately, with a little hackery, you can modify the connection timeout settings within your application. Do this carefully, and make sure that it can not leak into your running web application dynos.

• ### When you need a rabbit, spec it

February 7, 2014

On my current project, we're using RabbitMQ. It's a bit of infrastructure that has to be present, and if it isn't, our integration tests will fail with mysterious error messages. We want our tests to be informative, so let's write a test that asserts that we have the requisite infrastructure in place.

• ### Speed up your integration tests with a jig

February 7, 2014

If you've written enough integration tests (with Capybara et al.), you must have noticed how much time your tests spend just logging into your web app. Even if it takes 1 second each time, it starts to add up. Here's a solution that I've written several times, now. I create a test "jig" that allows me to authenticate into my application with a single visit.

• ### Stop thinking like an employee

November 5, 2013

A few weeks ago, some of the managers got together for drinks after work. I started to reminisce about an old employer who had, what I thought, was an incredibly unique and reasonable way to handle the “vacation liability” problem. The old firm was a consulting firm, mostly military contracts, in the DC area. We were paid for every hour we billed, and we accrued vacation hours for every hour we were paid. It created an environment where the top performers could rack up so much vacation that they could not use it all. Many companies handle this with the familiar “use it or lose it” policy, which caps the accrued vacation hours to some limit. Instead, the firm let vacation accrue forever, with one modification: When you got a raise, your vacation hours were reduced by exactly the same percentage.

• ### Why Pivotal Labs

May 26, 2013

It is a time-honored tradition for Pivots to blog about their first few months at Pivotal. A typical day at Pivotal is strong work. It’s different from any previous job. It’s exhausting. After six weeks or so, however, the Pivots find their rhythm. I’m not going to write any more about that. I’ll include some links, below, so you can read them for yourself. I’m here to write about the two-years-later; When most developers are itching to move on to the next big thing. I’m still happily learning new stuff every day.

• ### Sencha Touch BDD - Part 5 - Controller Testing

May 18, 2013

Part 4 Introduced PhantomJS as an easy and faster alternative to headful Jasmine testing. Part 3 added jasmine-ajax so we can verify that stores and models react properly to back-end data. We also learned how to use stores to test views, without depending on a back-end server. In Part 2 I showed you how to unit test Sencha model classes in Jasmine. In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework.

• ### Sencha Touch BDD - Part 4 - PhantomJS

May 10, 2013

Part 3 added jasmine-ajax so we can verify that stores and models react properly to back-end data. We also learned how to use stores to test views, without depending on a back-end server. In Part 2 I showed you how to unit test Sencha model classes in Jasmine. In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework.

• ### Sencha Touch BDD - Part 3 - Testing Views and Mocking Stores

May 5, 2013

In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework. In Part 2 I showed you to unit test Sencha model classes in Jasmine. In this part, we’ll test “views” and show how to mock data stores.

• ### Sencha Touch BDD Part 2 -- Unit Testing Models

April 26, 2013

In Part 1 I showed you how to set up your Sencha Touch development environment to use the Jasmine JavaScript test framework. We’re going to take a bit of a breather from all the hard work we did last week. In this blog, I’m going to show you how to test simple models.

• ### Sencha Touch BDD Part 1

April 17, 2013

(tl;dr) A multi-part series of articles on how to test Sencha Touch applications. It uses Jasmine for unit testing and Siesta for integration testing.

• ### method_missing hazardous to your module?

February 22, 2013

We built an(other) object factory module for our current project and it looks a lot like all the others: After a while, we noticed that the create methods were all exactly the same. Time for some dynamic refactoring!

• ### It's The Volatility That Will Kill You

February 19, 2013

Volatility is what Pivotal Tracker uses to measure the consistency of your team’s work output. You can use that number to help you estimate the first approximation to answer the eternal question, “Will I make the deadline?”

• ### From customer requirements to releasable gem

May 12, 2012

One of the many pleasures of working at Pivotal Labs is that we are encouraged to release some of our work as open source. Often during the course of our engagements, we write code that might have wide-spread use. Due to the nature of our contracts, we can not unilaterally release such code. Those rights belong to the client. And rightly so. So, it is an even greater pleasure when one of our clients believes in "giving back" to the community, as well.

• ### Deploy Strategies for HerokuSan

May 12, 2012

If you look at the network graphs of heroku_san on github, you'll see a number of branches where the only change is the deletion of the following line from the deploy task:

• ### TDD Action Caching in Rails 3

March 28, 2012

On my current project, we needed to prove that an action cache was working as expected. Alas, the blogosphere had either out-of-date or unhelpful information. So, after many experiments, we came up with an RSpec test that does what we want. It seems ugly to me, and I hope there's a better way. The names have been changed to protect the guilty. Any resemblances to actual classes and methods are purely coincidental.

• ### Dry DevOps with HerokuSan

March 25, 2012
1. How many times (each day) have you typed this at your console?
git push heroku master and then forgotten to run
heroku run rake db:migrate --app yellow-snow-3141 or
heroku ps:restart
2. Does your script support a multi-stage environments?
3. Do you remember how to get to the application's console process?
4. Is your application's configuration consistent across all stages?
5. Are you deploy scripts tested?

• ### DropBox + Git => Designer <3

April 5, 2011

One of the thornier problems in our workflow is knowing when assets are delivered from the designer and keeping them in sync with our application as they change. We used to use e-mail, Skype or sticky notes. The trouble is that the designer's file naming and directory structure were never quite the same as the application's /public/images directory, so direct comparisons were impossible and we ended with a lot bookkeeping to make sure that we didn't lose any changes. Our solution is to clone the project's git repository into a folder inside a shared Dropbox folder.

• ### When to do User Acceptance Testing?

November 18, 2010

"What does Scrum say about User Acceptance Testing? I am wondering if it should best be done within 24 hours of delivery, or at the end of a sprint..."

• ### The Elevator Speech

February 24, 2007

The purpose of the "speech" is to condense the project definition into something that would fit into the time it takes to ride an elevator. This is where all good ideas are pitched, apparently; you stalk an unwary VC until he or she is trapped inside the elevator has no choice but to listen to your prattling. If your idea has any merit, pray that the VC doesn't call on someone else to launch the project and leave you out in the cold.

• ### Using CVS for Web Server Farms

September 19, 2000

Let's assume that you've adopted the techniques described in Using CVS for Web Development (or [Dreilinger]) and you are the Release Master. Now imagine that instead of one web server you have a web server farm of 10 or 100. How do you prevent chaos and carpal tunnel syndrome from keeping you chained at your desk for days on end? Does this look familiar:

• ### Crack Keys in Your Spare Time

September 19, 2000

Moore's Law has had some unintended side effects: What to do with all that raw power sitting next to, on top of, or behind your desktop (or laptop or Mazda Miata). Most people fritter it away on silly things like screen savers, those flying toasters from hell. There are, however, a few, but growing number that are putting idle hardware into "productive" use.

• ### A Modest Housing Proposal

September 19, 2000

I've been thinking about the next logical step in Silicon Valley human resource management. Already there are campuses where you never have to leave for (almost) all of your personal needs: the dry-cleaner will pick up and deliver; cafeterias, webvan.com and take-out-taxi/ meals on wheels keep you fed; health clubs, masseuse and the roving chiropractor keep your body functional; showers and locker rooms keep personal hygiene up to acceptable standards. So what's next in this reinvention of the company town?

• ### Go West, Young Man, Go West

August 16, 1999
Maybe wax in your ears isn't such a bad thing after all.

• ### The First Full Year of Wishful Thinking

December 24, 1998

It’s hard to believe that it’s been over a year since we left the house in Los Gatos and moved aboard the S/V Wishful Thinking. We made trips to Angel Island, South Beach Harbor, Richmond Marina, Pier 39, Vallejo YC, Golden Gate YC… covering something over 800 miles this year.

• ### The Smartest Boat in the Marina

November 10, 1998

Day 1: Moved in to my new digitally-maxed out Hunter 450 at last. Finally, I live on the Smartest Boat in the Marina. Everything's networked. The cable TV is connected to our phone, which is connected to my personal computer, which is connected to the shore-power lines, all the appliances and the security system. Everything runs off a universal remote with the friendliest interface I've ever used. Programming is a snap.

• ### A Travelogue of a Dive Vacation to the Bahamas, Hurricane and Shipwreck Included at No Extra Charge

November 6, 1998

• ### Fleet Week at Treasure Island

October 11, 1998

An eventful weekend, that's what we had. It started Friday evening about 6 PM as that was our window of opportunity to escape from the shallows of the marina entrance in the South bay. The winds had been blowing 25+ out of the north since early afternoon so it was going to be wet and splashy, but we really wanted to stay at Treasure Island Marina that night and on to Angel Island early the next morning. So, it was now or never. Wishful Thinking was actually the last of three to leave. Gypsy, a Union 38, left about 5 PM and Dragon Lady, a Cheoy Lee 40, left about 5:30. We gave them a call once we were underway. "Hope you guys are reefed" came their answer. "Reefed" I said, "hell we're motoring!" "We're making fine progress just quartering these seas a bit and heading into the 25+ knots".

• ### Treasure Island CyberCruise

June 1, 1998

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### Renaming Ceremony

April 26, 1998

• ### Projects List

April 23, 1998

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### Letter Home

April 10, 1998

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### Three Bridges

March 16, 1998

Berkeley Yacht Club, California, US

• ### The Birthday Month

February 27, 1998

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### On a Lighter Note

November 28, 1997

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### Musing on a Boating Life

November 27, 1997

Home port, Peninsula Marina, Redwood City, US
37N29.55' 122W15.98'

• ### Top 10 Tools for Lonely Sys Admins

October 15, 1995