How 5 peoplewith 4 day jobsin 3 time zonesenjoyed 2 yearswriting 1 book

Ian DeesOpen Source Bridge 2011

This is the story of a big collaborative writing project. I’m telling it for therapeutic reasons—butif you’re a freelancer, a project manager, a telecommuter, or a member of a remote team, you may find the arc of this project familiar. Perhaps some of the tools and techniques we used along the way will be helpful to you as well.

Ian Nick Charlie

TomOla sketches byLukas Ketner

5 authors

The five authors are: Charles Nutter and Tom Enebo, the two JRuby project leads;Nick Sieger and Ola Bini, the other two members of the JRuby core team; andme (Ian Dees), a happy JRuby user and the only non-core team member in the group.

1. Sun (Charlie, Nick, Tom)

2. Engine Yard (Charlie, Nick, Tom)

3. ThoughtWorks (Ola)

4. Tektronix (Ian)

4 day jobs

Collectively, we’ve worked at these four companies...

1. Pacific (Ian)

2. Central (Charlie, Nick, Tom)

3. Central European (Ola)

3 time zones

...in these three places.

2 years

June 2008 - 2010

We started in June of ’08 and finished writing in August of ’10.

1 book

The book is Using JRuby, the definitive guide to the Java-based Ruby implementation.

IT’S NOT A

BOOKIT’S A SOFTWARE PROJECT

Since this is such a code-heavy book, I should have realized much earlier on that it’s as much a software project as a book. We could have used practices that have helped software teams, such as working in iterations and having frequent standup meetings. (Later on, we figured this out.)

the startup curve

http://adam.heroku.com/past/2008/4/23/the_startup_curve

Before I tell you our story, let me show you this famous graph drawn by Paul Graham and Trevor Blackwell at the Y Combinator headquarters.

the startup curve

The Process

TechCrunchof Initiation

Wearing Offof Novelty

Trough ofSorrow

Wiggles ofFalse Hope

The PromisedLand!

Acquisitionof Liquidity

Upside ofBuyer

Releasesof Improvement

Crash ofIneptitude

Here’s a cleaned-up version. It’s meant to represent the ebb and flow of web traffic at a startup.

the book curve

Establishmentof Rhythm

Conservationof Momentum

Release!

Nagging ofStakeholders Abandonment?

Harshnessof Reality

Trough ofSorrow

Wiggles ofFalse Hope

Euphoriaof Kickoff

The Process

It also fits the ebb and flow of morale during the writing of our book. This curve should feel familiar to anyone who’s worked on a multi-year, multi-person effort.

euphoria of kickoff

Harshnessof RealityEuphoria

of Kickoff

Let’s talk about how it all began.

dramatis personae

First, a word about my co-conspirators.

OlaThe ProspectorTECHNOLOGY SURVEYS, (RE-)STARTING PROGRESS

Ola is like a prospector. You can turn him loose in Testing Canyon, and he’ll suddenly turn up a couple of weeks later with a twelve-pound chunk of gold, saying, “Smelt that, baby.” Someone this good at working without a map is crucial for getting projects unstuck.

TomThe AnalystBREAKING DOWN HUGE QUESTIONS

Tom has an analyst’s mindset. He likes to consider every angle of a subject, write some code, study some more, and then weave a compelling narrative. He doesn’t flinch at huge topics that would overwhelm most people, such as “Tell me everything about user interfaces in JRuby.”

CharlieThe ProfessorBUILDING GREAT DEMONSTRATIONS

Charlie has a library of examples available to fit every occasion. His writing style is very demonstrative: “In this example, you will encounter problem X. To solve it, use technique Y. For instance: ....” He also has a good sense of what things we should explain in the book, and what things we should just fix in JRuby itself.

NickThe ConversationalistKEEPING THE READER ENGAGED, FINDING GAPS IN FLOW

Nick is a natural conversationalist. He connects material to previous ideas, then challenges the reader to absorb the new information. Someone like this is crucial for finding gaps in flow.

IanThe Yak ShaverTHE THOUSAND LITTLE THINGS THAT NEED DOING

And finally, there’s the yak shaver—the guy who sets out to write a chapter, creates some code examples to talk about, and eventually falls down a rabbit hole poring through the JRuby implementation. There’s even a place in most projects for a scatterbrain like this: taking care of the thousand little things like maintaining tools, talking to the editor, and so on.

the old planI. Introduction

1. IntroductionII. JRuby and Java

2. Ruby 1013. Java from Ruby4. Ruby from Java

III. Environment5. Standard Libraries/YAML6. Configuring/Running7. Inside JRuby

a. Architectureb. Interpreterc. Compilerd. Java Integratione. JRuby APIf. Extending JRuby

V. Working With Data8. JDBC9. XML10. Web Services/SOAP

V. Rails11. Overview/Tutorial12. JDBC13. Deploying

VI. Testing14. JUnit/Test::Unit/Mocking15. RSpec16. Benchmarking/Debugging

VII. Other Topics17. Security18. Application Servers/

Frameworks19. Swing20. Tools

In late 2007, the JRuby core team decided it was time for an official book. Here was an early outline. This would have made for a great book, but we didn’t have the time to write a work this ambitious. We needed to negotiate scope.

IT’S NOT A

WRITING GIGIT’S PROJECT MANAGEMENT

Lesson #2: This project was as much about management—breaking down work, making hard decisions—as it was about writing. This was yet another thing I failed to realize early on. We spun our wheels for a while because no one was making project-management decisions. One thing we did get right though: we narrowed the scope down to something we could finish together.

At RailsConf in June 2008, the five of us sat down at the awesome Doug Fir Lounge in Portland to see if there was enough rapport for us to work together. This is a scan of the original notes I excitedly took.

notes, transcribedI. JRuby

1. Basics / Installation2. Java from Ruby3. Ruby from Java4. Compiler

II. Libraries5. Rails6. Hibernate7. Rake8. Swing9. JMX10.JMS

III. AppendicesA. Ruby 101B. Contributing

This is what those notes look like when arranged hierarchically. It’s about half the size of the original plan.

the final bookI. JRuby

1. Basics / Installation2. Java from Ruby3. Ruby from Java4. Compiler

II. Libraries5. Rails6. Hibernate + Databases7. Rake + Deployment8. Testing9. Swing10.JMS

III. AppendicesA. Ruby 101B. ContributingC. ConfigurationD. C CodeE. JMX + Sysadmins

And here’s the final structure of the book. We were able to stay pretty faithful to our vision for two years, because it all came from our simple mantra: using Java from Ruby, and Ruby from Java.

ON A SIMPLE

RALLYING CRY

FOCUS

Lesson #3: avoid scope creep by asking of each proposed addition, “Does it help us do the thing we set out to do?”

work vs. “work”

After the meeting, we set up our tools based on the way we expected to collaborate: IRC for conversation, Google Groups for files and wiki pages, and so on. Alas, none of these things are work.

TOOLSDON’T CHOOSE YOUR

BEFORE YOU KNOW WHY YOU NEED THEM

It should have been a yellow flag that we were choosing tools based on how we thought we might work, rather than on how we were actually working.

things we tried

• IRC

• Google Groups

• Skype + Audio Hijack

IRC was great for helping real JRuby users, but for a small group like us, there was never a critical mass in the channel for our book. Google Groups was also overkill for the kinds of informal things we should just have used e-mail or AWS for. Skype helped us get unstuck, but there was little point in recording it.

things we didn’t try

• actually writing

Okay, so maybe we wrote after all—it just didn’t feel like it.

YOU DIDN’T EXPECTAN EXPERIMENT THAT PROVES A RESULT

is still

A SUCCESS

It’s okay that IRC and some of the other tools we tried didn’t work out for us. It was still a valid experiment to try, as long as we cut our losses and moved on.

harshness of reality

It’s at this point that we near the second phase of the project...

trough of sorrow

Abandonment?Nagging ofStakeholders

Wiggles ofFalse Hope

Trough ofSorrow

...the Trough of Sorrow. This is where it looks like there’s no progress—either because there is no progress, or because it’s all on things invisible to the reader.

July 2008

Jackie: What’s happening with JRuby?

Ian: Nothing. I have no authors. Couldn’t track anyone down....

Jackie: Can you try again this weekend? Don’t get discouraged just yet.

This phase is best told in conversation. Notice that at this point, I still haven’t learned the famous Apple dictum: when you’re the one coordinating the project and it goes off kilter, “reasons don’t matter.”

August 2008

Jackie: Anything happening with JRuby?

Ian: Good news from Sweden. Ola’s starting the chapter on Hibernate....

Aha! Forward progress!

September 2008

Jackie: BTW, if nothing is happening, we should talk about it soon.

Ian: Ola just released a bunch of code... following up to see if he’s ready to write about it.... I’ve just proposed a “bookfest” where... we sit and do nothing but write....

Infrastructure progress isn’t always visible to the reader, though. Note that this is the first time we get the idea to write in person together. We’ll revisit that idea several months later.

October 2008Ola: It seems that the book will take a long time to get finished..., and I really want to get going....

Ian: It's just been difficult [for the team] to justify writing about JRuby when they could either be hacking JRuby or helping someone.... On the other hand, a phone call gives us a solid block of time with each other, where we have (nearly) 100% of one another's attention.

Four months in, and even the five authors are getting antsy—but we don’t know what to do about it yet. We have, however, discovered that voice chat is much less distraction-prone than text chat.

February 2009

Jackie: Is [Ola’s testing chapter] ready for me to review? .... This looks really good!

Not much progress expected or seen during the Thanksgiving/Christmas time period. And then, in late January: a wiggle of hope!

April 2009

Jackie: How is it coming along?

Ian: I feel a bit stalled..., and now things just seem... slow.

Jackie: How can I help you get back on track? Are you getting discouraged?

Two short months later, and we were close to abandoning the project.

http://www.flickr.com/photos/danielygo/5242003647

I’d love to be able to tell my past self: in any long-running project, there will be times when you feel alone. You’ll log into Skype, and no one will be there. You’ll send e-mail, and no one will answer. Don’t give up, don’t despair, and don’t give up.

http://www.harpercollins.com

In The Care of the Soul, Thomas Moore explains that there are a lot more facets to the Narcissus myth than just some vain guy staring at his reflection.

NARCISSISMIS A FORM OFSELF-DOUBT

In particular, self-doubt is a form of narcissism. Telling oneself, “I’m terrible at this; might as well give up” is just looking for an excuse to give up. Who would blame a terrible writer for not writing? It’s important to acknowledge these feelings for what they are, and then push on.

establishment of rhythm

Establishmentof Rhythm

Conservationof Momentum

Release!

Right after near disaster, a few things happened that helped us establish a rhythm—both in the sense of playing well together, and in the sense of delivering at regular intervals.

a personalinflection point

First, JRuby helped me out of a jam at work. I started using it as my primary Ruby, instead of using it as an interesting alternative. When you write about something you use all the time, you can write with much more authority. See Charlie’s blog entry on writing vs. implementing: http://blog.headius.com/2007/09/are-authors-technological-poseurs.html

D O GEAT YOUR OWN

FOODCompanies call this “dogfooding”—actually using the product you’re trying to persuade people to adopt.

three thingsto establish rhythm

There were three more things we did together as a team that were crucial to getting us moving again.

HAVE A HEART(-BEAT)

The first was to establish a heartbeat for the project. (No wonder the patient almost died—we’d never taken the pulse!)

Pivotal Tracker

We broke our work down into a piece at a time—no time estimates—and entered them as stories into Pivotal Tracker. For example, a story title might be “Section on Maven describes Rake integration.” After a few weeks, Tracker would tell us if we were going to make each deadline. We got better at making them.

MEET IN

PERSONAT LEAST ONCE

The next thing we did was finally follow up on our plan to sit in person and write. I flew to Minneapolis and wrote with Charlie and Nick around dining room tables, coffee shops, restaurants, and bars all over the twin cities.

Google Docs

We needed something more immediate than revision control so we could see each other’s changes. We turned to Google Docs. Notice we’re choosing our tools based on need now, not guesswork. We wrote about JRuby as we wanted it to be; if a feature wasn’t implemented yet, we’d throw a TODO in the text and keep writing.

HABITSFORM GOOD

NOT EMPTY OBLIGATIONS

The third thing we did was to establish Skype calls every Monday, Wednesday, and Friday after the writefest. We weren’t overly rigid about this. If we had nothing to talk about, we’d skip a call. If one of us had to be somewhere loud, we’d use iChat instead. The point was to form good habits, not empty obligations.

August 2010

Ian: The final chapter is in our editor's hands. The book is written. Tonight, I'll try this "sleep" thing I've heard about.

We kept these habits up for a year. During that year, we announced the book, released several beta versions, and made tons of improvements based on our readers’ feedback.

lessons and hopes

In the course of writing this book, we learned the importance of recognizing software management when we see it. We learned the importance of in-person meetings, even for remote teams. We learned how to establish and keep a rhythm, even when we’re geographically separate and extremely busy. We hope that there’s something in here that you can apply on your next freelance, collaborative, or remote project.