Twittex - From Idea To Live in Seven Days

Twittex:From Idea To Live in 7 Days

Stuart HerbertTechnical Managerwww.gradwell.com

[email protected]@stuartherbert.com

Welcome to my talk for the PHPNW 08 conference.

Let me introduce myself. My name is Stuart Herbert. You have probably come across my blog, which is syndicated on Planet PHP.

I’ve been a professional software engineer since 1994, and I’ve been working almost exclusively on web-based projects since 1993. My career has given me the opportunity to gain a lot of real-world experience with UK household names over the years, and I’ve also been an active contributor to open-source software since the early 1990’s.

I am currently the Technical Manager for Gradwell, a role I started in April 2008. We are what is fashionably called a Unified Communications company. We do premium quality broadband, email, hosting, and Voice-over-IP (VoIP). VoIP is the majority of our business in 2008, and we are the UK’s third largest VoIP provider, with an annual turnover in excess of two million GBP. Celebrating our 10th anniversary this year, the company was founded, and is run by Peter Gradwell, who has been very active in the UK internet scene for the past decade.

One of the services we offer, and the subject of this talk, is Twittex (http://twittex.com). Twittex allows you to receive SMS messages from your Twitter feed.

All you have to do is to register your twitter account and your mobile phone with Twittex.

Decide which friends you want to watch, and when they post a status update to Twitter, you will receive an SMS containing their status update.

Twittex is a very simple, and very straight forward service.

Twittex.com TeamTwittex was built from scratch in a seven day sprint by a team from Gradwell.

From left to right, back row: me (project management, queue servers, queue management, provisioning), Ben Cole (Twittex.com website), Daniel Leech (Gradwell symfony components), Matt Park (Twittex logo).

From left to right, front row: Ben Smithurst (SMS integration), Paul Thomas (server provisioning, server monitoring), Tim Douglas (Twittex.com website).

Objectives

1. How We Built Twittex In Just Seven Days

2. What We Got Wrong

3. What We’d Do Differently Next Time

In this talk, I want to share with you our experience of building Twittex.com. I’ll start off by explaining the key factors that allowed us to build this service from scratch in just seven days.

Objectives




Although we think it was a great achievement to build Twittex in just seven days, with hindsight we made a few mistakes along the way. Some of them were obvious, and should have been avoidable, but not all of them were, and I’d like to share those with you too.

Objectives




Finally, if we could go back in time and do it all over again, what would we change?

Questions Welcome!

Please feel free to ask any questions you have at any point during the talk. There will also be time at the end of the talk for questions.

Background

1. Introducing Twitter

2. Twitter and SMS

3. The Opportunity

4. The Competition

Not everyone has heard of Twitter - surprising as that seems - so let’s start by introducing Twitter and explaining what it does.

Twitter is a microblogging service hosted at http://twitter.com. Anyone can signup for free.

Once you’ve logged in, you can post short messages to the Twitter website (known as the public timeline). It’s a great way to broadcast to the world what you are currently up to.

Each user on twitter gets their own “feed” - a list of messages from people that you choose to follow. Here’s a screenshot from my Twitter feed. Your feed is also available via the Twitter API.

As well as keeping in touch with your friends without having to keep up with the latest blog posts, Twitter is also proving to be very useful for creating online communities. One example can be found on the PHPNW 08 conference website, where all posts tagged with #phpnw automatically appear in the right sidebar.

Background


2. Twitter and SMS

3. The Opportunity

4. The Competition

So that’s what Twitter is. But where does SMS come into it?

Twitter allows you to register your mobile phone in the Devices tab.

Once you have registered a device, you simply navigate to a Twitter user that you’re following, and click where it says “Device Updates OFF”. Twitter will then send you an SMS every time that Twitter user posts a status update. And, best of all, these SMS messages are free.

Once you have registered a device, you simply navigate to a Twitter user that you’re following, and click where it says “Device Updates OFF”. Twitter will then send you an SMS every time that Twitter user posts a status update. And, best of all, these SMS messages are free.

This is a great service, and at Gradwell we’ve recently integrated it into our customer information feeds. Our customers have long been keen on getting SMS alerts about service-related problems, so we decided to achieve this by publishing our status information through Twitter.

Background


2. Twitter and SMS

3. The Opportunity

4. The Competition

A couple of weeks after we’d relaunched our GradwellStatus service, Twitter made some changes to their service. These changes left us with a bit of a problem, but also with an opportunity to launch a new service of our own.

On Wed 13th August, Twitter decided that it could no longer afford to provide free SMS alerts for several countries, including the UK where Gradwell operates. If memory services, I believe these SMS alerts were costing Twitter 1,000 USD per Twitter user per year. That’s a lot of money for a firm with no revenue model!

The story was quickly picked up by TechCrunch (if you work on the web, and you don’t read TechCrunch, you should), and from there spread like wildfire around the ’net.

Background


2. Twitter and SMS

3. The Opportunity

4. The Competition

Twitter’s withdrawal of free SMS alerts suddenly created a new market, one that possibly had real potential. Gradwell wasn’t the only company to recognise this by any means.

First to decide that they were going to launch a service was tweetSMS. They got the word out very quickly and very successfully.

tweetSMS was quickly followed by Zygo, which is run by former colleagues of mine from the old New Media department at Orange.

By 22nd of August - just nine days after Twitter withdrew their service - no less than eight rival services were competing in this new market. That’s a lot of competition!

We Can Do Better!(Or Have Fun Trying :)

At Gradwell, we decided that we wanted to compete in this market too. As we were the only telco launching a service (all our competitors are new media companies), we believed that we could provide a better service. And even if we failed, it would be a lot of fun to try, and it would also be a great way for me to learn a lot about my staff.

Timeline #1:Actual Events

Let’s start by looking at a timeline of what actually happened, and when. This timeline was compiled using BeeDocs, and at the PHPNW 08 conference, it was presented as an animated timeline.

I’ve split the timeline up into two streams. Along the top is what was happening out on the web, and we start with Twitter withdrawing their SMS service on the 13th August.

TechCrunch picked up the story on the same day, and began to spread the word far and wide.

On the very next day, TweetSMS announced their plans on Mashable (another website you should add to your Google Reader feed if you don’t have it already).

Zygo announced their service on the following day ...

... by the 22nd of August, there were eight competing services analysed by the OnlineJournalismBlog.

The second stream in the timeline looks at what we did at Gradwell. Our story starts on the 14th of August, when I came across the TechCrunch story in my Google Reader lists. I immediately thought that we could plug the gap created by Twitter, and that it was something we should do.

By the middle of the afternoon, Ben Smithurst and I had sketched out the architecture we wanted to build for the service, and felt confident with our underlying design.

By 7pm on the 14th, I phoned Peter up, and left him a voice mail asking for his approval to build what would become Twittex.

As Peter was camping out in the wilds of Scotland, we had to wait until he drove to the nearest town for some milk before he had a phone signal. This didn’t happen until the following morning.

With approval secured, we settled down to getting on with the work. Ben Cole immediately set to work on building the website.

We spent a few hours coming up with a name for the service, finally settling on twittex. As soon as I approved the name, we registered the domain to make sure no-one else came up with the same name :)

Shortly after, we had ourselves a logo. Although Gradwell doesn’t have a graphic designer in-house, we’re very lucky that one of our staff can turn his hand to this sort of work, and that he was very happy to make the time to do so.

How long would it take you to get a new logo for a service that you wanted to launch? Days? Weeks? Longer? In a show of hands at the PHPNW 08 conference, many of the audience said that it would take weeks or longer to get a logo designed.

We carried on working into the evening, and by 7pm we had the ability to send SMS messages out to mobile phones.

By 9:30pm, we’d finished provisioning the servers to run the queues on (more on the queues later on), and had finished compiling and installing MySQL onto the servers.

So to recap: on the Wednesday, Twittex announced that it was withdrawing the SMS service. By the end of the Friday, we had our servers, our domain, our logo, and the ability to send out SMS messages.

The team continued to work throughout the weekend. Whilst work was continuing on the web site, we spent Saturday and Sunday writing the queue management code.

We also sorted out billing and payments on the Saturday.

On Sunday, we adopted the 960 grid system by Nathan Smith, and began using it to prototype the user-visible parts of the website.

On the Monday morning, whilst work on the website was continuing, we put the server monitoring in place.

And, on the Tuesday, we’d finished the website, and opened it up to staff for internal testing. We were kept very busy finding and fixing corner cases in our code that day!

With all the obvious bugs squashed, on the Wednesday, we were ready to launch. We contacted TechCrunch to announce our service ...

... we also contacted Twitter’s own PR people ...

... but heard nothing back from them at all. At the same time, we sorted out an introductory offer price ...

... and put the service live.

Shortly after, Ben and I had to jump in the car and drive up from Bath to Sheffield for an important meeting the following day. Unfortunately, there was a serious accident which closed the M5 for four hours, giving us plenty of time to do even more bug fixing whilst we waited for the road to re-open. Laptops and mobile broadband are essential tools for any developer who travels!

So, we launched our service just 7 days after Twitter announced they had withdrawn theirs, and what’s more we were the first folks to have a working service live. Were we swamped by demand? No. One of the reasons was that our initial pricing was badly wrong. We had to reduce our pricing from 10p per SMS message to 5p per SMS message, and we had to do it quickly.

With our meeting in Sheffield out the way, we headed our separate ways (Ben back up north, me back to South Wales), doing more bug fixing on the way. We improved the log messages that Twittex writes, to help us diagnose problems quicker.

We also created a new Twitter account just for testing twittex with. Up til now, we’d all be using our personal Twitter accounts, which put a bit of a limit on the type of test messages we wanted to send.

Our friends at PlusNet kindly mentioned us on their blog - our very first bit of publicity!

As we still hadn’t heard anything from TechCrunch or Twitter at all, and hadn’t been mentioned by them either, we decided to widen the net by contacting The Register and Mashable. Sadly, neither website wrote anything about us :(

On the following Monday, Jemima Kiss wrote a piece for the Guardian’s website about Twitter and SMS messages. We dropped her a quick email about our service. Jemima was kind enough to write back - the only one who did out of all the organisations we contacted - letting us know that we’d basically missed the boat.

By keeping below the radar until we had a service to launch, we’d completely missed the news cycle.

But Twittex Really Started Months Ago!

So that’s how we built Twittex in just seven days. Except - it’s not the whole story. Work on Twittex had actually started *years* before.

Huh?

Years before? What do I mean by that?

Twitter.com

Twittex.com

SecPayNetsuite Gradwell

Let’s start by looking at the architecture for Twittex.com. We have the Twittex.com website, built in just seven days.

Twitter.com

Twittex.com


It pulls feeds down from Twitter ...

Twitter.com

Twittex.com


... and then re-uses Gradwell’s existing telco infrastructure.

Twitter.com

Twittex.com


An important part of this infrastructure is our ERP/CRM system, Netsuite. As a company with a multi-million pound turnover, Gradwell has to produce audited accounts every year. A company of our size simply cannot create a new service, hook it up to a PayPal shopping cart, and leave it at that.

Twitter.com

Twittex.com


For payments, we use SecPay, because it’s the only payment gateway in the UK that’s certified to work with Netsuite at this time.

PHP ServersPHP Servers MySQL Server

PHP ServersQueue Servers

Software

Load-balancers

Software

Load-balancers

That’s the architecture for Twittex. If we look inside the Twittex.com box, we’ll find a pair of PHP servers.



Software

Load-balancers

Software

Load-balancers

These servers sit behind a pair of load-balancers.



Software

Load-balancers

Software

Load-balancers

Twittex uses MySQL to hold subscriber information.



Software

Load-balancers

Software

Load-balancers

And finally we have a set of queue servers which manage the job of polling Twitter for new messages.

At the heart of our infrastructure is VMWare ESX, which we adopted late 2007. ESX gives Gradwell the ability to provision new servers extremely quickly. We don’t have to wait for new hardware to arrive, and we don’t have to have someone physically installing kit into racks in a data centre.

We’ve been using symfony for over a year now as our framework of choice. We chose symfony because we felt it had a more responsive and more supportive community that its competitors.

As I mentioned, we use MySQL for our databases. I can’t imagine there’s anyone reading this who hasn’t heard of MySQL, but I’ve included it for completeness :)

We adopted the 960 grid system for our prototyping, and our live Twittex.com site also uses 960 gs. If you haven’t come across it before, check it out. We found it extremely useful. Using pure CSS, you can create pages that are 960 pixels wide, cut up into 12 or 16 columns. These columns can then be merged to form the final layout. The only change we’d love to see Nathan make to his system would be to add a 20 column version, to allow for even more precise layouts.

For the queues, we used the Q4M storage engine for MySQL 5.1.

Q4M allows you to create first-in, first-out queues that can be accessed via SQL SELECT statements. The really nice thing about using Q4M is that our own code doesn’t have to handle any sort of locking at all. We can have multiple processes emptying the queues, and Q4M ensures that any one piece of data from the queues isn’t presented to more than one of these processes.

I came across Q4M when it was selected for the architecture of a Japanese rival to Twitter.

Why Queues?

But why use queues at all? Let’s take a look at that.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue

No Twitter Message :(

Twitter Message :)

Twittex User

We start off with an individual Twittex user.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

We put the data about this user into a queue. We check all data in this queue every minute.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

Eventually, our Twittex user makes it to the front of the queue.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

We go and check Twitter, to see if there any new messages for this user. If there are new messages ...

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

SMS Sent

... we send out the messages as SMS’s, and put the user’s record back onto the end of the queue.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

Eventually, the user once again makes it to the front of the queue.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

What happens where there are no new Twitter messages? Well, we won’t want to poll Twitter every minute for every user. That’s a lot of wasted effort.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

Instead, we have additional queues, which are checked less and less frequently.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

The user’s record works its way through these queues, until eventually it hits our 5 minute queue.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

If there are still no Twitter messages for the user, the user’s record stays in the five minute queue. We used to have 10 minute and 15 minute queues too, but during internal testing we decided that they made for a poor user experience.

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

If there are any messages for the user on Twitter ...

1 Min Queue

2 Min Queue

3 Min Queue

4 Min Queue

5 Min Queue


Twitter Message :)

... we move them straight back into the 1 minute queue. This makes Twittex very responsive if you strike up a conversation through Twitter. The user’s record will then work its way back down the queues as before until there’s another message on Twitter for the user.

Timeline #2:Groundwork

Our architecture and our infrastructure re-uses technology that we adapted long before we had the idea to create Twittex. Without this re-use, we could not have built Twittex in just 7 days.

Here’s another timeline, this time looking at all the things we did in the past that made Twittex possible.

We’ve been using AQL for sending and receiving SMS messages since at least October 2003. AQL provide a very easy to use RESTful API, and we’ve had a lot of practice with it over the years.

Do excuse this rant, but if you’re going to build an API for any sort of service, do make it RESTful. SOAP APIs should never be the only API you offer. For a start, PHP’s SOAP client is a toy, and currently does not implement enough of the SOAP standard to make it inter-operate with real enterprise SOAP APIs. Secondly, too many SOAP API authors need to learn that WSDL is not documentation - it is specification. And finally, SOAP APIs also suffer from poor design, because they focus on exposing objects rather than business services. So, please please please, stick with simple but effective RESTful APIs :)

We adopted Netsuite for ERP and billing in May 2007. As a company, we already have an established platform to handle new products, and the financial accounting that goes with them.

We’ve been using symfony since June 2007, and in that time we’ve built many re-usable components for it. We’re also very lucky that we have an internal champion for symfony, who not only advocates for its use but also that it’s used in the proper manner. This keeps our code lean, clean, and very easy to reuse.

We made ESX our main platform earlier this year. ESX gives us flexibility that we couldn’t afford if we were still working solely with physical hardware. How would you launch a new service in just 7 days if you had to wait for new hardware to be delivered first?

The Twitter APIs are very easy to use, being REST based, but nonetheless we’d already had experience working with them. We’d used them in July to integrate our GradwellStatus.com website with Twitter.

Twittex was the first time we’d used Q4M at Gradwell, but it wasn’t something we had to go out and find on the day. A big part of my job as Technical Manager (CTO/CIO in US language) is to have a wide knowledge of what’s available in the open source community and beyond just in case we ever have a need to introduce something new to our established technology stack. I rely heavily on Google Reader to help me keep on top of hundreds of RSS feeds from around the net, as well as attending conferences, trade shows, and talking to my peers.

What Made It Possible

1. We Were Prepared

2. We Were Up For It

3. An Extension Of What We Already Do

So, to recap the elements that made Twittex possible. First and most importantly, we were well prepared. We have been consistent over the years in adopting technology that is simple to use and that gives us increased flexibility.


1. We Were Prepared



It had already been an extremely hectic week, but with no prior notice, my team willingly gave up their evenings and the weekend to work on this project. Do you think we would have been able to build this with a 9-5 mentality? We would have been too late to market, and the project would probably have failed, because we would not have managed to maintain the momentum in the first place.

In questions after the talk, I was asked why my guys were up for it. I think it comes down to two key things: they thought it was an interesting project to do, and to them programming isn’t a job, it’s a vocation. It’s what they love to do.


1. We Were Prepared



Finally, we believe that a critical factor was the nature of the work. Gradwell is an established and successful telco, and is already used to handling this sort of messaging. We’re not a New Media company suddenly trying to get our head around a different paradigm. This is stuff we know how to do, so we can just get on and do it.

What We Got Right

1. Recognised Opportunity

2. Authority To Proceed

3. Technology

4. Delivery Of Working Solution

What did we get right? First of all, we very quickly recognised that there was an opportunity to be tackled. We needed to do something, because we had been relying on Twitter for SMS updates to our customers. We didn’t um and ah about it, we got our heads down straight away on solving the problem.

What We Got Right



3. Technology


Despite the amusing story of how long it took to reach Peter for approval, I’m actually very pleased about how quickly we had authority to proceed with the project. We had the green light just 24 hours after coming up with the idea.

In my own experience, most companies are utterly crap at making decisive decisions, especially in such a short period of time. The feedback I had after the talk was that many companies couldn’t make decisions this quickly.

What We Got Right



3. Technology


I’m very happy indeed with our choice of technology. From ESX through to Q4M, from symfony through to the 960 grid system, everything we used made the job much easier - not much harder. At no point were we having to fight with the technology. We were able to just get on with creating the service.

What We Got Right



3. Technology


Let’s not forget - we actually launched a working service. It works very well.

But unfortunately, it wasn’t successful in attracting a large user base ...

What We Would Change

1. Marketing

2. Pricing

3. Project Management

The first thing we got wrong was our marketing. We got this badly wrong. We deliberately kept off the radar until we had a working service. We hadn’t done a sprint like this before, so we weren’t certain how long it would take us to build a launch service. By the time we wanted to tell the world about our marvellous creation, the world was no longer interested. The news cycle had moved on, and we had missed the boat. Those behind the news cycle, it turns out, have neither the time nor the attention span to care about working services :)

In my mind, tweetSMS did exactly the right thing. They put up a ‘sign up now - coming soon’ page, and they put it up very quickly. This got them coverage, and no doubt it allowed them to hoover up many of the early adopters that Twittex was also aimed at.


1. Marketing

2. Pricing


Boy, did we get this wrong too :) My initial pricing, at 10p a message, was designed to cover our costs. That was a mistake. We had to drop our prices after the first day, once we realised that our competitors were planning on launching services that were around the 5p a message mark.

Instead, I should have made the price as low as possible, and used pricing and our existing bulk discount to gain the largest market share over our competitors.

As it turns out, the market for paid SMS messages from Twitter in the UK appears to be tiny. No-one has made a killing on this at all. The wholesale price for SMS messages is just too expensive. I’m extremely interested in making Twittex a free service supported by advertising. If there’s anyone out there who’d be interested in making this possible, do get in touch :)


1. Marketing

2. Pricing


I made some obvious mistakes with the project management too. The main one was that we only started prototyping the web pages themselves three days into the work on the website. What I should have done was had a bit more balls, and committed extra resources on the Thursday (before we had approval to proceed) to create the prototypes up front.

Timeline #3:A Better Timeline

If I could go back in time and do this project all over again, knowing what I know now, what would be different?

Twitter would still have stopped sending SMS updates out on the 13th ...

... and TechCrunch would still have spread the word on the same day.

TweetSMS would still have been first to get the word out that they were launching a replacement service.

But if I could do this all over again, we would have gotten the word out to anyone and everyone as quickly as possible too. The earliest we could have done this would have been the 15th, after the approval to proceed.

Would Zygo (and all the other competitors) have announced too, if we had made our intentions known? That’s impossible to speculate on.

... but I can’t help but wonder whether we’d have ended up with a two horse race instead of an eight-way finish.

Inside Gradwell, some things would stay the same. I’m very happy with how early we recognised the opportunity.

Sketching out the architecture up front is a necessary step, and I don’t think we could have achieved it any sooner than we did.

But one thing I would change would be to prototype the website up front too - even before having approval to proceed.

I don’t think we could have made the project request any sooner. We needed to have confidence that we could build the service first.

We got approval to proceed very quickly.

If we had had the prototypes up front, the website would have been built even quicker. We would probably have taken a day out of the timeline.

We should have created the twitter test account up front, so that we could do a wider range of testing during development. Whether or not this would have allowed us to launch earlier is hard to say, but it would have perhaps reduced the amount of bug fixing we did post launch.

I’m very happy with how quickly we came up with a name and purchased the domain.

We should have paid more attention to the commercials up front. We should have had a clearer idea about them, so that we got our launch pricing right first time.

I’m very happy with how quickly we had the logo done for the site.

Our integration with AQL was done very quickly, and I’m very happy with how that went.

I’m also extremely happy with how quickly we had working servers to deploy code onto. In the past, I worked with organisations who rely on third parties such as Rackspace to provision new servers. The lead times are often 20 working days or longer, because new hardware must be provided.

We probably couldn’t have built the queue management code any quicker ...

... and we probably couldn’t have had the billing arrangements in place much quicker.

I’m happy with the time it took to get the server monitoring in place. This couldn’t have been done earlier, as we needed to provision the servers first before hooking them up to our monitoring system.

If we’d had the prototypes up front, the site would probably have been ready for internal testing a day earlier.

... and we would have been able to launch the site at least one day earlier. Who knows? Just maybe we might have been able to launch on the Monday :)

I doubt we would have been able to avoid being stuck in the traffic jam on the M5.

We would have been able to avoid having to reduce our prices, if we’d got the commercials right up front.

I’m comfortable with having had to improve our log messages after the service had launched. I think it made sense to focus on getting the customer-visible side of the service working first, and then going back and polishing up things behind the scenes to reduce the support burden.

I’d like to think that our friends at PlusNet would still have mentioned us when they did.

Your Thoughts?

That’s what we think would have happened had we done a few things differently. What do you think?

Thank you for viewing this presentation. I hope you’ve found it interesting and that you’ve learned something from it. We certainly learned a lot from doing this project. If you have any questions or feedback, please get in touch with me [email protected].

Business

Twittex - From Idea To Live in Seven Days