The unsolved scaling challenge
Everyone knows that scaling websites so that they can serve millions of users a day is hard. Most Twitter users have seen the famous Fail Whale often enough, which usually shows up because some component of Twitter's website isn't handling an increase in traffic so well. Many people familiar with web services can also appreciate the additional challenge that comes with providing an API, especially when, as is the case of Twitter, it has to handle 10 times the traffic of the website.
What people often miss is the huge amount of non-technical work that is required to keep the consumers of a website happy as the user base grows into the tens of millions. If you're charging users you can factor support into the cost and grow your support team to ensure that every user can talk to someone when they need to. Many services don't have that luxury, but are able to assign teams of smart people to building knowledge bases that can answer almost all user questions via their websites. So what happens when, like Twitter, more than 90% of your traffic isn't coming through the website but is coming via over 50,000 applications using your API? You end up with a relatively small group of developers that need highly specialist support to ensure that they in turn can provide their own users with a reasonable level of support. You could charge that group to ensure that you can build a skilled developer support team to answer every request in person. However, what do you do if, like Twitter, you are aware that this developer community is adding a huge amount of value to your ecosystem and you don't want to slow its growth?
The question of how to scale a developer community like Twitter's is one I've been asking myself for over a year now. It's led to me exploring various ideas and meeting a host of interesting people. I started 2009 by organising the world's first events for Twitter developers, the Twitter Developer Nest, of which there have since been six in London and one in New York, involving over 350 members of the Twitter developer community. I spent the majority of the year working with startups and agencies integrating with Twitter, experimenting with new features as they have been rolled out and building a Twitter based service of my own. Towards the end of the year I had the opportunity to spend a day talking with several members of Twitter's Platform team as well as Ev, Biz and Dick.
As a result of my experience I've built a comprehensive understanding of the inner workings of Twitter's ecosystem and developed a wealth of material on the subject that I'd like to share through a series of blog posts. Topics I'll cover include:
- The unique properties and challenges of Twitter-like developer communities
- A hierarchy of platform developer needs
- Identifying key community members
- Measuring success
- Managing support requests
- Maintaining developer support sites
- Choosing developer support tools
- Keeping the team in touch with the community
- Managing product development requests
- Organising developer events
- Long term developer evangelism
While I'm under no illusion that there will ever be a simple solution for solving the challenge of scaling a developer community, there are two things I'd like to share with you now which I believe are particularly important. Both platform providers and members of developer communities need to be continually asking themselves if they are being as open and kind as they can be. The success of Twitter's Platform to date can be traced back to the openness of Twitter's early API and the kindness of early 3rd developers, but there is much more that can be done as the community grows.
Be Open
If you are a platform provider like Twitter there is always an opportunity to be more open with your developer community. A group of smart people have invested a massive amount of their time and often money into your product without any kind of service level agreement. Sometimes those developers can seem like they don't appreciate your work, more often than not it's because they don't understand what it is that you do. The more information you can provide them with, the greater their ability to assess the risks they have exposed themselves to and appreciate the value that you add. Once they have a better idea of where they stand they will be far more likely to take the added risks of working with new features and innovating on top of your platform.
Every time a developer is surprised by something you announce publicly, they'll lose a little bit of confidence in their ability to work with your platform while staying on top of the risks. If your developer community is a key competitive advantage it's not worth damaging it as a whole in order to make short term gains in the press and with a handful of partners. Special relationships with certain partners will inevitably exist as is always the case in business. By making all developers aware of steps required to become a special partner you can guide them down a path that can help you meet your strategic goals.
Be Kind
If you are a developer working with a Twitter-like platform there is always an opportunity to be kinder to the providers of the platform and other members of the community. You're getting something for free and the individuals that keep it working are human beings just like you. Sometimes they can get overwhelmed to the point of not being able to provide you with human responses. Be patient and encouraging while giving constructive feedback. You'll gain far more from taking time to build relationships with platform providers and working together to tackle problems than you will from publicly attacking them. More often than not other members of the community can be of more help than providers are able to be. By reciprocating this help in whatever way you can you can help build the community into a more rewarding environment for everyone.
These two things are surprisingly easy to take for granted while being critical to the success of a developer community. As such I am making 'Open Kind' the mantra of this series of blog posts. Of every option we have along the journey of scaling a developer community we should ask "is it open kind?", "are we being open kind?" and "can others use it to be open kind?".
What are your thoughts on this unsolved challenge? My next post will focus on the the things that set developer communities like Twitter's apart from more traditional developer communities, please let me know if you feel there are any particular aspects I should focus on. I'm looking forward to discussing this whole issue further with people in person over the course of this year, please do say hello if you spot that our paths cross. I'll be at SXSWi in March. In April I'll be attending Chirp, The Official Twitter Developer Conference in San Francisco followed by speaking at #140conf in New York. In May I'm organising WarbleCamp, The Twitter Developer Unconference and throughout the year I'll continue to participate in The Twitter Developer Nest (DevNest)
5 comments
I would add 'Be Accountable'. I think you, (and twitter) know well enough my views that there should be better governance for the community to understand what they can expect from twitter in order to proceed with allocating time and money towards development. I wrote a 'twitter developers manifesto' - which I presented at #devnest last June ( http://www.nuxnix.com/social-/19-collaboration/40-my-twitter-software-developers-manifesto.html ) to give some lessons from the recent computing past that illustrate this requirement.
As Twitter grows up I think it needs to publish service levels in a properly constituted partner and developer program which show an even playing field for developers and don't for example favour startups with funding from the same angels/people/institutions.
Consider getting funding for a serious startup - would you get it based on uncertainty, potential for lock out, conflict and ability for twitter to swipe any idea they deem nice enough for them.
This makes this issue unavoidable for twitter and I really hope they will grasp this nettle sooner rather than later, and it is a fundamental part of the scaling of the community. If they truly intend to compete with 'text messaging' then they will have to face this or face inevitable law suits.
I explained all this to @rsarver when I visited them in the fall and to make it clear to twitter that I wish them well and would like nothing more than to have this level of clarity. I do think it was understood and #chirp is part of a response to devnest and the manifesto - I just wish there was more.
Its interesting that we see so much focus in scaling applications, and scaling ones customer facing support that the whole developer side of things is often neglected.
Some companies do this well, for example Flickr does a good job of supporting its developer community however in their case, their platform is pretty static - you write a few good API docs, post code samples and open up a forum and you are pretty much good to go, much less of a scaling effort required.
Another example, Amazon, monetize their API which really focuses them to think about the 'Be Accountable' part and publish SLAs. It also allows them scale their support based on a customers pricing tier - those who pay the most get personal support.
As Twitter has grown up, I think its still in an immature state with regard to its developer offerings. Its platform is supporting thousands of developers' applications, and it is by no means static like say Flickr. The API is growing to support the products roadmap, and I think this is where 'Be Open' is most important. Like you say, you don't want to surprise developers with new features, either by changing how they interact with the platform or by stepping on their toes. Its a tough balancing act.
You're right about openness; it is essential. And, of course, the mutual respect.
That said, it can be infuriating from a developer's perspective when certain illogical rules and policies exist and when they are defended purely on the merits of "because it is so". The issue I had with the oAuth/source parameter policy recently is a prime example: my app, Feathers, is being hurt by not having a source parameter. Twitter's response is simply "use oAuth or no source parameter for you". It's hard to remain calm in the face of illogical policies like this. (I look forward to implementing xAuth in a future update; as soon as I'm allowed onto the beta – of which I haven't had any news yet.)
The other issue I had recently that got me very frustrated was the lack of personal care application developers currently get. Both the @feathers and @feathersapp accounts are unused on Twitter and yet my request to take them over for Feathers was denied with a form response that any _user_ gets. No special treatment for developers. (So, at the moment, I have to use the less-than-ideal @feathers_app name and no one has even responded to the trademark dispute I filed over a week ago).
It just feels like when you put so much time and effort into making a Twitter app, Twitter can make a few exceptions for you and at least not treat you with the same policies that apply to all users.
At the very least, it would go a long way towards forging an amicable relationship if form responses were not used as often in communicating with developers. A link to a policy page is probably the worst response you can give to someone who has just taken a lot of time to explain to you the specifics of an issue as they pertain to his/her app.
Leave a comment...
