A Parable About Allies
June 23, 2019
A person falls into a deep, dark, scary hole.
Our CTO Sleeps Well At Night
July 30, 2018
I am PacerPro’s chief technical officer, which means that I get the first phone call at 3 am when things go pear-shaped. Fortunately, I don’t get many of those calls: I’ve built a development team and process that delivers reliability as a core feature.
Using Ruby Enumerators for Resumable Iterators
March 11, 2018
Here's a power technique: You can use Ruby Enumerators to create "resumable" iterators when you're extracting data from the application console.
On Call Duty
March 11, 2018
Recently, there was a flurry of tweets about on-call and SRE duty schedules. It triggered a memory of a conversation at a manager meeting about a new product offering:
Our Hiring Process
July 30, 2017
Here's how the engineering hiring process works at PacerPro.
Do Not Push to Red Builds
July 8, 2017
There's a pattern to disasters: When you examine the post mortems, its not just one failure, but a cascade of multiple failures that overwhelm the system's safety features. Pushing to a red build risks piling one failure upon another.
Wanted: Jaguar Priest
July 7, 2017
My job is to be the dumbest person in the room; I want to work with intelligent, opinionated people who can make the world a little better. Then iterate.
Performant Large-Scale Postgres Migrations
November 30, 2016
Recently I tried to add a column to one of our very big tables, and as you might expect from the title of this blog, it took so long that I aborted it after an hour. Can you spot the problem?
Sacrificial Test Classes
November 10, 2016
Today I learned that there’s a difference between a throw-away class and a sacrificial class. When writing rspec tests, you want the latter; otherwise you might create some very hard-to-debug test pollution.
Memory Leak Whack-a-Mole
October 27, 2016
https://www.flickr.com/photos/tpapi/2765541278Memory leaks happen. They shouldn’t, but you just can’t squash enough of them. Or the leak is so slow that it takes hours to manifest. The trouble with memory leaks is that they are permanent for the life of a process. Once a process has expanded its VSZ, there’s no way in Unix to shrink it. So your process starts to bloat, eventually your servers page and thrash. Performance tanks. Then your pager starts going off and there goes your beauty sleep.
Digital Preservation Lasts Until the Heat Death of the Universe or Five Years
October 4, 2016
(Whichever comes first)
Anatomy of an Apology
September 22, 2016
I’ve written my fair share of apologies. It’s a skill, like any other, one that you wish you didn’t have to practice. Yesterday, I had to write a mea culpa to our users. It was our fault, and we came clean about it. I got compliments on the letter! I think I increased customer loyalty as a result. Fuck-ups don’t have to hurt your business.
Guidelines for Using Pivotal Tracker
July 25, 2016
I am often asked about the meaning of things in Pivotal Tracker, such as “What’s the difference between a bug and a chore?” Or, “How can I tell the difference between a bug and a feature?” Another oldie but a goodie is, “What story should I pick from the backlog?” Here are my biased answers.
The Syntax Error That Killed Our Build for a Week.
December 12, 2015
Back in the day when I wrote in C and we used X11 with Motif, I was creating a new module based on an existing one, so I just copied and pasted the file from one xterm to another:
Zero Downtime Migrations
August 15, 2015
For any sufficiently large application, you want to minimize interruptions to service while deploying new code. This is especially challenging for migrations on Heroku and other Platforms-as-a-Service, where you have a Catch-22 problem; you want to run your migrations, first, but you can’t run them until you’ve deployed the migrations to production. Usually, at this point, your new migration-dependent features are in the deploy, too. So when your application restarts, and the migration hasn’t run, yet, your poor application will trip some exceptions, and perhaps, create problems for your users. Of course, you could put the app into maintenance mode, but that creates more downtime for your users.
Unintended consequences (and a disaster averted)
January 6, 2015
I want to tell you a scary-as-shit story, mostly so that my friends that have CCW permits might read it and not get killed by good intentions, because I love you all.
Timecop time and database time
June 8, 2014
Timecop is a great addition to your testing toolbox, but if you’ve ever tried to use Timecop to interact with a database, and then got the most mysterious of existential errors:
The slow road
June 1, 2014
Once a week, we run a query on our production system that takes a very long time to complete. Invariably, it times out because the default settings on Heroku are (quite reasonably) for web / API interactions; anything longer than 30 seconds is too long and the connection should be closed to let other requests through. How do you get around this, then? You could set up your
database.ymlfile so that you connect to your production database from a developer workstation. This is actually quite useful, but opens up a hellacious security hole. Let’s not get started about the consequences of accidentally running
rake db:drop:all! Moreover, you want your system to be automated as much as possible, and that means running your tasks on a Heroku instance. Fortunately, with a little hackery, you can modify the connection timeout settings within your application. Do this carefully, and make sure that it can not leak into your running web application dynos.
When you need a rabbit, spec it
February 7, 2014
On my current project, we're using RabbitMQ. It's a bit of infrastructure that has to be present, and if it isn't, our integration tests will fail with mysterious error messages. We want our tests to be informative, so let's write a test that asserts that we have the requisite infrastructure in place.
Speed up your integration tests with a jig
February 7, 2014
If you've written enough integration tests (with Capybara et al.), you must have noticed how much time your tests spend just logging into your web app. Even if it takes 1 second each time, it starts to add up. Here's a solution that I've written several times, now. I create a test "jig" that allows me to authenticate into my application with a single
Stop thinking like an employee
November 5, 2013
A few weeks ago, some of the managers got together for drinks after work. I started to reminisce about an old employer who had, what I thought, was an incredibly unique and reasonable way to handle the “vacation liability” problem. The old firm was a consulting firm, mostly military contracts, in the DC area. We were paid for every hour we billed, and we accrued vacation hours for every hour we were paid. It created an environment where the top performers could rack up so much vacation that they could not use it all. Many companies handle this with the familiar “use it or lose it” policy, which caps the accrued vacation hours to some limit. Instead, the firm let vacation accrue forever, with one modification: When you got a raise, your vacation hours were reduced by exactly the same percentage.
Why Pivotal Labs
May 26, 2013
It is a time-honored tradition for Pivots to blog about their first few months at Pivotal. A typical day at Pivotal is strong work. It’s different from any previous job. It’s exhausting. After six weeks or so, however, the Pivots find their rhythm. I’m not going to write any more about that. I’ll include some links, below, so you can read them for yourself. I’m here to write about the two-years-later; When most developers are itching to move on to the next big thing. I’m still happily learning new stuff every day.
Sencha Touch BDD - Part 5 - Controller Testing
May 18, 2013
Sencha Touch BDD - Part 4 - PhantomJS
May 10, 2013
Part 3 added
Sencha Touch BDD - Part 3 - Testing Views and Mocking Stores
May 5, 2013
Sencha Touch BDD Part 2 -- Unit Testing Models
April 26, 2013
Sencha Touch BDD Part 1
April 17, 2013
(tl;dr) A multi-part series of articles on how to test Sencha Touch applications. It uses Jasmine for unit testing and Siesta for integration testing.
method_missing hazardous to your module?
February 22, 2013
We built an(other) object factory module for our current project and it looks a lot like all the others: After a while, we noticed that the
createmethods were all exactly the same. Time for some dynamic refactoring!
It's The Volatility That Will Kill You
February 19, 2013
Volatility is what Pivotal Tracker uses to measure the consistency of your team’s work output. You can use that number to help you estimate the first approximation to answer the eternal question, “Will I make the deadline?”
From customer requirements to releasable gem
May 12, 2012
One of the many pleasures of working at Pivotal Labs is that we are encouraged to release some of our work as open source. Often during the course of our engagements, we write code that might have wide-spread use. Due to the nature of our contracts, we can not unilaterally release such code. Those rights belong to the client. And rightly so. So, it is an even greater pleasure when one of our clients believes in "giving back" to the community, as well.
Deploy Strategies for HerokuSan
May 12, 2012
If you look at the network graphs of
heroku_sanon github, you'll see a number of branches where the only change is the deletion of the following line from the
TDD Action Caching in Rails 3
March 28, 2012
On my current project, we needed to prove that an action cache was working as expected. Alas, the blogosphere had either out-of-date or unhelpful information. So, after many experiments, we came up with an RSpec test that does what we want. It seems ugly to me, and I hope there's a better way. The names have been changed to protect the guilty. Any resemblances to actual classes and methods are purely coincidental.
Dry DevOps with HerokuSan
March 25, 2012
- How many times (each day) have you typed this at your console?
git push heroku masterand then forgotten to run
heroku run rake db:migrate --app yellow-snow-3141or
- Does your script support a multi-stage environments?
- Do you remember how to get to the application's console process?
- Is your application's configuration consistent across all stages?
- Are you deploy scripts tested?
- How many times (each day) have you typed this at your console?
DropBox + Git => Designer <3
April 5, 2011
One of the thornier problems in our workflow is knowing when assets are delivered from the designer and keeping them in sync with our application as they change. We used to use e-mail, Skype or sticky notes. The trouble is that the designer's file naming and directory structure were never quite the same as the application's
/public/imagesdirectory, so direct comparisons were impossible and we ended with a lot bookkeeping to make sure that we didn't lose any changes. Our solution is to clone the project's git repository into a folder inside a shared Dropbox folder.
When to do User Acceptance Testing?
November 18, 2010
A former client asked:
"What does Scrum say about User Acceptance Testing? I am wondering if it should best be done within 24 hours of delivery, or at the end of a sprint..."
The Elevator Speech
February 24, 2007
The purpose of the "speech" is to condense the project definition into something that would fit into the time it takes to ride an elevator. This is where all good ideas are pitched, apparently; you stalk an unwary VC until he or she is trapped inside the elevator has no choice but to listen to your prattling. If your idea has any merit, pray that the VC doesn't call on someone else to launch the project and leave you out in the cold.
Using CVS for Web Server Farms
September 19, 2000
Let's assume that you've adopted the techniques described in Using CVS for Web Development (or [Dreilinger]) and you are the Release Master. Now imagine that instead of one web server you have a web server farm of 10 or 100. How do you prevent chaos and carpal tunnel syndrome from keeping you chained at your desk for days on end? Does this look familiar:
Crack Keys in Your Spare Time
September 19, 2000
Moore's Law has had some unintended side effects: What to do with all that raw power sitting next to, on top of, or behind your desktop (or laptop or Mazda Miata). Most people fritter it away on silly things like screen savers, those flying toasters from hell. There are, however, a few, but growing number that are putting idle hardware into "productive" use.
A Modest Housing Proposal
September 19, 2000
I've been thinking about the next logical step in Silicon Valley human resource management. Already there are campuses where you never have to leave for (almost) all of your personal needs: the dry-cleaner will pick up and deliver; cafeterias, webvan.com and take-out-taxi/ meals on wheels keep you fed; health clubs, masseuse and the roving chiropractor keep your body functional; showers and locker rooms keep personal hygiene up to acceptable standards. So what's next in this reinvention of the company town?
Go West, Young Man, Go West
August 16, 1999
Maybe wax in your ears isn't such a bad thing after all.
The First Full Year of Wishful Thinking
December 24, 1998
It’s hard to believe that it’s been over a year since we left the house in Los Gatos and moved aboard the S/V Wishful Thinking. We made trips to Angel Island, South Beach Harbor, Richmond Marina, Pier 39, Vallejo YC, Golden Gate YC… covering something over 800 miles this year.
The Smartest Boat in the Marina
November 10, 1998
Day 1: Moved in to my new digitally-maxed out Hunter 450 at last. Finally, I live on the Smartest Boat in the Marina. Everything's networked. The cable TV is connected to our phone, which is connected to my personal computer, which is connected to the shore-power lines, all the appliances and the security system. Everything runs off a universal remote with the friendliest interface I've ever used. Programming is a snap.
A Travelogue of a Dive Vacation to the Bahamas, Hurricane and Shipwreck Included at No Extra Charge
November 6, 1998
Fleet Week at Treasure Island
October 11, 1998
An eventful weekend, that's what we had. It started Friday evening about 6 PM as that was our window of opportunity to escape from the shallows of the marina entrance in the South bay. The winds had been blowing 25+ out of the north since early afternoon so it was going to be wet and splashy, but we really wanted to stay at Treasure Island Marina that night and on to Angel Island early the next morning. So, it was now or never. Wishful Thinking was actually the last of three to leave. Gypsy, a Union 38, left about 5 PM and Dragon Lady, a Cheoy Lee 40, left about 5:30. We gave them a call once we were underway. "Hope you guys are reefed" came their answer. "Reefed" I said, "hell we're motoring!" "We're making fine progress just quartering these seas a bit and heading into the 25+ knots".
Treasure Island CyberCruise
June 1, 1998
April 26, 1998
April 23, 1998
April 10, 1998
March 16, 1998
Berkeley Yacht Club, California, US
The Birthday Month
February 27, 1998
On a Lighter Note
November 28, 1997
Musing on a Boating Life
November 27, 1997
Top 10 Tools for Lonely Sys Admins
October 15, 1995
Reprinted from ;login: v20,n5, October 1995.