Where Customer Service Goes to Die

“Sorry to bother you…”, the email from my client began. It was Monday morning, and I knew the company this person worked for always had conference calls with their customers on a Monday. No doubt she was prepping for a difficult one.

The email went on, “but could you help us with the query below?”

The email chain that followed contained a discussion between my client and one of their customers about an obscure feature on one of their enterprise software products. The customer didn’t like the way it worked and was clearly trying to paint this as a bug rather than a feature. I had recently done some unrelated work on that product at the request of that same customer, and I could tell they were gearing up to say, in effect, “We’re not going to pay for those changes because the product still has this ‘bug’ and we want you to ‘fix’ that too.”

This is a classic manoeuvre in enterprise software support. The customer identifies a broad or complex problem. They propose a small change to solve that problem. Then, once there are people working on it, they conveniently forget about the specific change they requested and insist more work be done because their broad problem is not solved yet.

This sort of thing happens frequently enough to convince me that it’s a deliberate ploy by many corporate managers. The only way to prevent doing work for free as a software vendor is to employ people to keep a very close eye on the scope of every change and argue back with the customer if they try this kind of thing.

A customer service expert might suggest spending more time talking to the customer about their broader problem, and seeking to solve it all from the beginning. Sometimes that works, sometimes that just looks like up-selling and the customer rejects it.

The real issue is that enterprise software exists mostly to join all sorts of disparate parts of an organisation together. It’s plumbing, and plumbing is messy. Plumbing needs continual patching up and tweaking. It’s an ever moving target. It’s also very political. It’s hard enough getting several departments to agree on how they’re going to work together, by the time they’ve agreed a process, getting that implemented in software is the easy bit!

So the companies that make enterprise software spend their days locked in conflict with their own customers trying to keep a lid on all the complexity arguing about who’s going to pay for all the changes they keep having to make. Enterprise software is where customer service goes to die.

And any potential way to improve the level of service you can give effectively amounts to, not selling enterprise software any more. This includes approaches like:

  • Simplifying what you offer
  • Focusing on doing one thing really really well
  • Trying not to get drawn into customers’ complex inter departmental problems
  • Trying to have more customers so you’re not so beholden to the few that you do have
  • Saying no when customers ask for complex things
  • Charging for your time rather than for the software

If your business is enterprise software then I feel bad for you. Fortunately there is hope. The oncoming tidal wave of SaaS applications is making enterprise IT more a matter of composing disparate services rather than building or buying large behemoth systems. Over time, most business functions are becoming productised and then owned by 3 or 4 dominant SaaS apps.

Need content management? Try WordPress, Joomla, Durpal or Umbraco. Need CRM? Try Salesforce, Sage or Highrise. Need to send lots of email? Try Mailchimp, Constant Contact, Sendgrid or Campaign Monitor. All of these products offer some customisation, but they differ from old style enterprise products in that they can be setup and configured by users in a browser. Somehow the vendor has hit that sweet spot of product positioning, sensible defaults and customisation capability and as a result they’ve carved out a market for themselves without armies of implementation experts and customisations.

If you’re selling enterprise software, my advice is to stop. Your industry will eventually be eaten by a few major SaaS vendors. So either carve out a niche and become one of those vendors, or switch to making a living gluing things together for a daily rate. Your customers will be happier. You’ll be happier, and you can stop wasting your Mondays arguing over edge cases and who’s funding the ongoing cost of tweaking things.

Perfection Considered Harmful

People are always making mistakes. People send emails to the wrong person, or forget an attachment. They give you the wrong change at the checkout or bump into you in the street. Every day, people make little mistakes. For the most part we’re quite forgiving of each other. We understand that the other human being we’re looking at is doing their best and as long as they try to fix things quickly we cut them a bit of slack. But our forgiving nature is based on our ability to relate to the person who’s made the mistake as being like us, a fellow human being. We are notably less forgiving when someone is diving and needs to switch lanes before their junction. On the road, it is not a fellow human being who’s made a mistake, it’s “some idiot in a BMW”.

When it comes to software systems we are even less likely to relate to the person who’s made them. All software is designed, written and tested by people, but we don’t see it like that. We think of software as some alien entity that arrived on planet Earth fully formed. And like the evil BMW who wants to get in front of us, any mistakes that it makes are utterly unreasonable.

Within the software development world, we prevent mistakes by having people check things. Designs are reviewed, code is tested, the content of those tests are themselves reviewed, code is deployed to small groups of people first and monitored carefully, the list of things that went wrong is reviewed and fixes are made, tested, reviewed again, and then finally released to more people. If all goes well it will work perfectly and nobody who uses it will ever experience a problem.

All, never goes well.

Each layer of testing and checking is done by people, and people make mistakes. Each layer of checking just reduces the probability of a mistake getting through, but no amount of checking ever guarantees that the software is perfect. Playing the lottery every week will increase your chances of winning, but there’s no magic number of weeks you need to play before you’re guaranteed a jackpot. What is more, like playing the lottery, each time you get someone to check something it costs you money.

As a supplier of software to others, your job is to find the right balance when customers demand new features, developed quickly, with no bugs, on a budget. The cost of checking and re-checking each new bit of code that goes out the door must be traded off against the value of the new features you’re developing. If you had the new feature 3 weeks earlier, but had to refund 3 customers who found errors in it, would that be worth it? Or does the cost of having 3 angry customers outweigh the benefit of getting the feature sooner?

There is no easy answer here. Be assured; the site will not stay up 100% of the time, there will be bugs, occasionally a sale will be lost or a refund will have to be made. This is a natural consequence of having human beings do work for you. But the important thing to remember is; do not allow people to think its reasonable to expect perfection. Involve the customers/users in the trade off decision making process. Try to put a human face on the software you’ve created. Be present as much as possible. Smile. Talk in terms of probabilities.  Be humble. Say “sorry” early when things go wrong. And if you do that really well, you may actually go up in people’s minds when things go wrong, not down.

What Are You Optimising For?

Very few software designs/architectures are completely and objectively bad. Most designs optimise for something. Here’s a [probably incomplete] list of things that you could optimise for in software architecture:

  • Developer Time Now
    When we say we’re doing something “quick and dirty”, or that we are taking out “Technical Debt”, then we are optimising the development time now.
  • Developer Time Later
    This is what people traditionally mean by “good” code. A developer wants to write code that will be easy to work with several years from now. This is based on the perception that most of the hours a developer spends working, they are maintaining an existing system.
  • Product Iterations
    In simple terms this means building something that is highly flexible and is likely to meet a broad range of future (as yet unknown) requirements. By making the system flexible we reduce the number of iterations we are likely to need before the customer is happy with the product.
  • Testing Time
    While optimising for Product Iterations means making a flexible product, optimising for testing time is almost the exact opposite. The ideal system to test is one with a single button labelled “Do {Something}” which, when clicked, does exactly what it says it will. The more options, configuration, flexibility and programmability you introduce to a system the more time it will take to test to a reasonable level of satisfaction.
  • Backwards Compatibility
    This means designing a system to accommodate existing users who potentially don’t want to have to think about the new features. Or perhaps have lots of existing data in a structure that is not conducive and to the new features you want to introduce. By optimising for backwards compatibility, we’re building something in a way we wouldn’t normally like, but that will make it easier to roll out.
  • Deployment Risk
    In these days of hosted services and sand boxed apps this is becoming less of an issue. But anyone who’s had to think about rolling out an upgrade to a user community where you don’t know exactly what data they have or what the environment its running in looks like, then this will be familiar. Depending on the context you might make users go through long-winded upgrade procedures or pass up on framework upgrades and new technologies just to be sure that your next version will not fall over when it hits real world user machines.
  • Deployment Effort
    Sometimes the solution you want to build will be a nightmare to setup in production. It may require lots of servers, or perhaps a combination of services for caching, queuing, persistence, messaging and load balancing. In some cases, you may want to spend extra development effort building automated installation tools or even take a performance hit so that rolling it out is easier.
  • Support Time
    Sometimes it’s preferable to take extra developer and testing time to introduce automated installations, lots of logging and diagnostics, remote crash reporting, user friendly failure messages with troubleshooting guides, extra documentation, lots of context sensitive sensible defaults or reduced flexibility. All so that it’s less work to support once it goes out the door.
  • End User Learning Curve
    There’s often a market for an entry level version of apps that maybe don’t have all the power of the competitor products but which is easy to pick up and use. Alternatively, you could trade development and testing time and build an easy to use product that has a lot of flexibility behind the scenes for the power users.
  • End User Usage Effort
    If you’ve ever used VIM, you’ll understand what I’m getting at. VIM sacrifices End User Learning Curve in order to optimise for End User Usage Effort. Sometimes developer and tester time is taken to build hundreds of special business logic that will make the system do exactly what the user needs exactly when they need it.
  • Performance
    This often means sacrificing Developer Time Now/Later in order to make the system run faster. For example, using an off the shelf ORM tool will reduce the code you need to write and maintain, but to get that extra performance people often resort to hand crafting the database queries they need.
  • Scale
    The ability to handle millions of users or Terra bytes of data often comes at the expense of developer time, deployment effort or single user performance. If you really need to handle that scale then you’ll have to sacrifice something else on the list to do it.
  • Budget
    If you have very deep pockets then most other problems can be made to go away. If you’re willing to build your own data centre with 10,000 servers in and an army of people to keep your site running then you can defer having to think about writing scalable code or deployment effort. This    is sometime a logical approach. This is not always true though. A baby takes 9 months no matter how many women you have.

I find that conversations between engineers about the best architecture to use are always easier and less emotive when people are clear about what they’re optimising for, why, and what they’re willing to sacrifice in the process. It’s helpful to keep asking: what are you optimising for?


Pick Something to Fail At

One of the consequences of technology is that theres an almost infinite number of things you could work on, both in and out of work. For example:

  • If someone I haven’t seen in years un-friends me on Facebook, I feel bad for not keeping in touch with them more. After all, I have all the tools at my disposal.
  • My PVR automatically records episodes of Sitcoms that I wouldn’t have worried about missing, but now feel I should watch them.
  • Colleagues can assign tasks to me instantly and easily. It’s so easy to create work, all it tasks is a few seconds to write an email out, send it to 4 or 5 people, and you’ve assign several hours’ worth of work. People used to have to fill out a form if they wanted someone to do something.
  • My task list (or Inbox) is with me continually though the power of MS Exchange Integration.
  • Between my RSS reader and Hacker News, I have more career/productivity/life enhancing information to read every day than I could realistically read in a week.
  • Online reputation systems like StackOverflow.com provide the perfect forum for others to beat you to that next exciting job. Maintaining at least a modest level of reputation now seems a prerequisite for not missing out a few years down the line.
  • Automated error/traffic logging means no unexplained error message or referrer can go unnoticed. They must all be investigated.
  • The number of tools and resources available is staggering. If I wanted to produce a community newsletter, mix a dance record, write that novel, attend that Yale class on ancient Greece, learn how to re-tile my bathroom, create a graphic design portfolio or influence others with my views on how to reduce the budget deficit… I can, and at virtual no cost. There’s no excuse.

There is more work out there than you can ever do, one way or another, you’re not going to get it all done. This may make you feel sad, angry or depressed. If so, well done, you’re probably not a psychopath. Now, lets talk about how you’re going to manage this task overload.

Time to Grow up

You’re going to need a prioritisation strategy, and none of them are perfect, so you’re going to have to choose the best of a bad bunch. I think maturity is, in part, about consciously deciding what kind of a person you’re going to be and taking responsibility for the consequences. Let’s look at the options.

Last In Wins

“Do what’s immediately in front of you. If something crops up drop what you’re doing and do the new thing instead”. Nobody chooses this strategy, but it’s what happens if you’re not paying attention. It generally gets the small and mechanical tasks done while leaving everything else half done. It doesn’t take account of priority either. This is our baseline, any improvement on this model is progress.

First In Wins (the simple Queue)

“Do tasks in the order in which that come in; never start something until you’ve finished what you were doing”. When people have had enough of the Last in Wins model, this is often the first port of call. It’s simple and easy to defend when someone wants something from you, so it can be quite appealing. The simplicity comes from the fact that it takes no account priority, it’s entirely time based. This is also its biggest weakness.

The Magpie

“Do whichever task you feel like doing (or looks the shiniest)”. This approach is actually a lot more effective that it might sound to begin with. Our brains usually have one thing that it’s working on in the background, and it’s usually the most challenging or important thing. Issues relating to food, sex or money and conflict often rate highly, as do emotionally charged issues. The more senior you are in a business the more likely it is that those are the issues you need to sort out. That’s probably why so many entrepreneurs are self-confessed magpies. 

The Selfless Hero

“Do whichever tasks seem most important to other people first”. This is an approach I see most often in low level managers. They understand their job is to clear the roadblocks and make the troops in the trenches as productive as they can, but they do this at the expense of their own career/reputation/sanity.  

The VIP Express Queue

“Do the things that a certain select few people ask for, or that will most impress those people”. This approach is easier said than done, it won’t win you many friends and may get you a reputation as an ass kisser. But if you can get that promotion before your peers turn on you then you could be on to a winner.

The Job You Want, Not the Job You Have Approach

“Drop all the tasks you feel are beneath you and focus your attention on the things you’d rather be doing”. For some, task overload is an opportunity not a problem. Having too much to do is the perect excuse for not doing “menial” work and focusing your attention where you can “add the most value”. After a few months you can ask for that promotion on the basis that you’re doing the job already. It can be dangerous though, by definition you have to deliberately fail to do your actual job for a while.

The Shareholder Approach

“Imagine you’re a shareholder, now decide what you think is the least effective use of employees’ time, then skip those tasks”. If you’re a real shareholder, this is what you’d really like employees’ to be doing, the only problem is, there’s really no incentive for employees to adopt this approach. It won’t get you that promotion, if won’t impress any of the people you interact with on a daily basis and it won’t give you an easy life.

Pick Something to Fail At

The one thing that all of these approaches have in common is that they involve choosing something to neglect – sometimes consciously, often unconsciously. My advice is, if you’re going to choose, then choose consciously and accept the consequences of your choice.

Design Notes are not Functional Specs

At my company, we don’t write Functional Specs, we write Design Notes. This post explains the differences and the thinking behind those differences.

Design Notes contain some Balsamiqs, some text, and the occasional table definition. Many people, when they see a Design Notes document think of it as a Functional Spec. They exist purely for the time before the software is written as an aid for developers and testers. Once there is software, the software is the spec. This contrasts with Functional Specs where any differences in the software and the spec should be automatically considered a bug, although it practice there are often things that got added without the spec being updated because it was too much work to update the spec.

Things change, and we want to spend our time building software, not continually updating documents about the software we’d like to build. When specs have to be kept up to date, it becomes incredibly difficult to finish the spec and actually start development. Every new idea from users, every question from developers, every clarification that testers ask for, they all require a new version of the document and every new version has to be sent out for review which triggers lots more feedback. While this is going on, it{781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}u2019s tempting to halt development to avoid “unnecessary rework” and to allow the analysis to “complete”. Let’s get one thing straight: analysis is never complete and software will always evolve. Any process that assumes otherwise is a broken process.

The overhead of having to keep the spec 100{781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f} accurate can be so tortuous that getting to a final version of the spec often involves pressuring some stakeholders into accepting things they don’t like just so everyone can agree and move on. Having done that, it becomes politically very difficult to re-open the spec later on. So even the most obvious and important changes have to wait until “Version 2” because we can{781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}u2019t possibly change the spec once it{781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}u2019s been signed off.

Design Notes do present a challenge for testers. Traditionally a tester’s job was to ensure the software met the spec. Design Notes require the tester to be more involved in the design process and to think about whether something is really a bug before raising it. Often this will involve talking things through with analysts, developers and customers. We recognize that this is not easy, but we think this is treating testers as more equal partners in the development process. On the plus side, testers are not constrained by a spec; if something looks wrong, doesn’t flow well or is inconsistent, even if it’s doing what the Design Notes say it should, they are still free to raise it. Saying {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}u201Cit meets the spec{781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}u201D is not the end of the conversation.

A la Carte Software

The traditional way of designing then costing a new piece of software is broken. All too often a someone spends months refining what they want, talking to everyone involved and throwing any feature ideas that they think they might need into the melting pot. Until eventually, they think they’ve covered everything.

Then; they ask someone how much it will cost.

The first problem is that software feels like it should be cheaper than it is. People outside of IT see software as being an advanced version of a spreadsheet. And Terry over there in the corner knocked up a really useful spreadsheet in something like two weeks, so surely a bit of new software shouldn’t take any more than 4 weeks to build!

Another problem is that the list of feature requirements that they ask the potential development team to quote on contains way more than they really need. The vast majority of the value of the new system will be contained in a very small proportion of the features. But people rarely ask development teams to help them decided what they need. Probably because those unscrupulous software developers would use that as an opportunity to sell them stuff they didn’t need.

Finally, potential customers don’t understand the non-linear way that extra features effect the price of a development project. When they see a quote saying it’ll cost $500,000 to build the 20 features they need they (quite reasonably) think that each feature costs $25,000 and that even the 5 features they really cannot live without would cost $125,000 which is still way too much. Sometimes, they simply walk away at that point convinced that the price is so far from what they can afford there’s no possibility of a deal.

There are two ways of fixing this. The good way and the bad way.

The bad way would be to be recklessly optimistic when estimating. You quote the customer a ridiculously low ball offer, they accept, you then fail to deliver but the customer is locked in, you then try to claw back some sort of profit through the change control process. In other words, how a lot of software development companies work. In fact it’s interesting to wonder whether this is the reason why this industry has a chronic underestimation problem, perhaps it’s because the business model depends on it!

The good way would be to be get involved earlier in the process and to give the potential customer instant feedback on how their decisions affect the price. One way to do this would be through a menu of software modules.

Such a menu would have things like “Business Entity”, “Audit features (per entity)”, “Content Management (per page type)”, “Workflow features (per entity)”, “Custom Reports (per report)”, “Backend API”, “Document Management” and so on. Each item would have a price there in black and white and you sit with the customer, explaining how they can combine some of the items on the menu to achieve the effect they need, all the while showing them what that does to the bottom line.

Imagine what it would be like to work with a potential customer who’d bought into this process and was starting thinking creatively about how to solve their business problems with the minimum number of software components, now that’s what I call fun!