There’s no such thing as a fixed price software development project

Fixed price software development contracts are a bad idea because they fail to limit the customer’s costs and they worsen the quality of what is delivered. In this article I will explain why.

What makes software development hard is that a group of people must clarify exactly what they want a computer to do in many thousands of situations. Between a senior executive having an idea in the shower one morning and a computer executing that idea, many people will be involved. Each person might have a different interpretation of that idea and every conversation about that idea might introduce mutations to that idea. Modern software development methodologies have evolved to help groups of people move towards agreement on how software should work by introducing techniques to specify how it will work. These include functional specification techniques (User Stories, UML), visual specification techniques (UI and UX design), iterative specification techniques (Agile processes), and formalised specification techniques (Test Driven Development).

Those techniques are all useful, but a piece of software is never fully specified until it’s been written. The developer takes these artefacts and makes the final set of decisions about what the computer should do. We can tell this is true because it is often necessary to ask a developer “What will the software do in this situation that we’ve just thought of?”. The developer will then check the code to find out.

Any software development project is therefore a collaborative exercise in deciding what should happen in each situation and writing that into code. This is what makes software development hard, not the ability to write code per say. Over years many attempts have been made to make coding easier by making the syntax more like English. But while languages like ADA, COBOL, Python and Ruby made coding more accessible, they didn’t make software development projects significantly easier, because they didn’t change the fundamental nature of the problem.

During a fixed price software development contract, any individuals on the supply side of the commercial relationship have a strong incentive to influence the decision making in favour of simplifying the software, to make it easier to write and test. The people on the customer side, on the other hand, have an incentive to remove manual tasks for their employees or make their customers’ lives better. They will try and influence the behaviour of the software in that direction. Sooner or later there will be a disagreement about what the software should do. The supplier will say that the behaviour being discussed was not made clear at the start of the project and they want to re-negotiate the price. The customer will respond that any reasonable reading of what was specified at the outset should include the behaviour being discussed and that the price should not change.

Although the supplier on a fixed price software project many not succeed in renegotiating the price every time they want to, they will succeed some of the time. And when they do they will take the opportunity to make up failed attempts to renegotiate the price earlier on. Ultimately, the customer engaged the supplier because they need them to build their software for them, and they will not be able to resist every attempt to renegotiate the price. So they will be forced to pay what the supplier says they need in order to complete the project. In very rare circumstances the customer may cancel the contract, but this is costly for the customer and just starts the whole project again, albeit with a new supplier.

Does that mean that customers should request fixed price contracts anyway? On the basis that it might work, but if it doesn’t they’re no worse off than if they had accepted a time and materials contract? The answer is no, because having two camps in an adversarial battle over how the software should work does not make for good quality software. Sometimes these battles are conducted in good natured discussions. Sometimes they are passive aggressive moves by the supplier to quietly implement their interpretation and see if a bug is raised, and sometimes there are full blown arguments with both sides threatening to call in lawyers. However it’s conducted, it’s not helpful. Good quality software comes from everyone involved being focused on the goal of making the user’s life easier.

In summary, no software development project is ever be fixed price because the scope is not defined until the code already exists, and having some of the team trying to avoid making the users’ lives easier does not make for good software.

Five Software Architect Antipatterns

I believe that all software design comes down to trade-offs, and the only way for software architects to get to the right decisions on those trade-offs is for them to have a very broad and senior role within their organisation. I’ve observed some people in the industry who are called architects but, for one reason or another, are not in a position to design software well. Here are five types of software architects that are ill placed to actually design software.

1. “Well Paid Programmer” Architects

Sometimes, giving people the job title of Architect is just a way of giving them a pay rise without their job materially changing. Some architects are really just programmers who reached the top of their pay scale and didn’t want to manage people. In theory they’re supposed to be setting the technical direction but in reality they’re no more influential than an opinionated developer. Quite often they’re locked in conflict with other developers about the direction of the system being built and lack the necessary clout or managerial backing to enforce architectural decisions.

Lesson: Software requires tough choices to be taken, once you’ve found the right person for the job, they need management backing and they need the strength of their convictions to own that role.

2. “COBOL Programmer” Architects

Some architects are programmers who got stuck on an old technology that their company doesn’t use any more; But making them an architect is a way of keeping them around in case those old systems fall over. These architects spend a lot of time getting involved in non-work related bits and bobs. You’ll find them on the fire safety committee, they’ll be one of nominated first aiders in the office, a regular organiser of corporate charity fundraising events, they always have plenty of time to help the support team with the aforementioned COBOL system and they will always have an apt anecdote from the days of 80 column card interfaces or EBCDIC encoding. They don’t really know how to code in the languages that are used by the developers these days and they have an annoying habit of referring to IE as “Mosaic”, but they can just about cost up a new server, given a few days notice.

Lesson: Software Architecture is a real job that people need doing, its not a way to put old techies out to pasture. If you want to keep those people around then fine, but call them “Consultants” or something and hire a proper architect.

3. One Tick Architects

Some architects know one thing, like Networking Configuration or Relation Schema Design really really well but little else. Designs get written which explain in fine detail how the subnets of the servers’ network cards are going to be setup, but skip casually over whole areas of detail with statements like “A suitable object relational mapping technology (Linq2SQL/EF/NHibernate) should be used”.

Lesson: You’re skills in one discipline may have been enough to get you the architect job, but the value of the architect is in their cross disciplinary skills; To do your new job well you need to start learning news skills, and fast.

4. Back Room Architects

Some architects lack the basic personal skills to be allowed out in front of customers or users and consequently come to view their role as a purely technical one. Unaware of the political and budgetary landscape their projects are operating in, their designs are masterpieces in resilient, scalable, flexible and buzz word filled architecture which, if actually followed, would cost millions to complete. These architects are perpetually locked in battles with project managers over how much time it is reasonable to spend building each system. Look out for phases like “one day we’ll actually build a system right from day one” i.e. they see software architecture as a binary good vs. bad issue as opposed to a series of trade offs.

Lesson: All decisions in software design are compromises between development time, hardware cost, supportability, usability, maintainability, tooling costs, recruitment considerations, staff retention considerations, the aspirations of the people paying for it, the actual needs of the users and many other things, you can’t understand all those competing influences by sitting behind a desk running Emacs, you have to get out there and talk to people.

5. Non-Technical Architects

These architects are really just business analysts. They are great at talking to customers, they spend lots of time capturing the non-functional requirements of the system and writing documents that list those non-functional requirements for the benefit of the developers and server engineers. But as far as solutions go they haven’t a clue. They are full of handy suggestions like “we should do some performance testing” but when asked what to do with the results, they fall back on sledgehammer solutions like “buy a better server” or “get the programers to add some caching”.

Lesson: If you’ve somehow made it into an architect role and you’re NOT a C Hacker/SQL Guru/Bash Ninja then you have a problem. The techies will soon start to figure you out and your authority over them is dependant on you reaching some minimal level of technical ability. Find a way to start adding value to what they do before they give up on you.

Ansl – A .NET Search Library

I’ve written an implementation of the a simple search library in C#. It’s called Ansl (A .NET Search Library) and it implements the TF-IDF algorithm for indexing text documents.

By default it stores it’s index in memory but its storage engine is pluggable and it comes with an implementation of a folder storage class too so indexes can be built up and stored in the file system for later searching.

In time I’d like to add a Vector Space Model implemenation and then have a FindSimilar(string documentId) method for finding similar documents. I’m not getting much free time at the moment though so that may take a while.

Fixing MacVim .gvimrc Encoding Errors

For weeks I’ve been going mad trying to figure out why most of the settings in my .gvimrc file were being ignored on MacVim.

At first I thought it must be some strange quirk of MacVim that it didn’t support the same settings a GVim (on Windows) but everything I read online suggested that wasn’t it.

I have my .gvimrc and .vim folders sync’d using Dropbox. I’ve setup symbolic links on my Mac, like this


ln -s ~/Dropbox/vim/.gvimrc ~/.gvimrc
ln -s ~/Dropbox/vim/.vim ~/.vim

And on my windows machine, like this


cd C:Usersme
mklink /H .gvimrc Dropboxvim.gvimrc
mklink /D vimfiles Dropboxvim.vim

This keeps my windows and mac vim things in sync. But it wasn’t working on the mac.

By chance, I decided to step up the mvim script that comes with MacVim, and I notices that when running mvim from the terminal I was seeing lots of errors like this being output to the terminal


E474: Invalid argument: modelines=0^M
line   10:
E474: Invalid argument: tabstop=4^M
line   11:
E474: Invalid argument: shiftwidth=4^M
line   12:
E474: Invalid argument: softtabstop=4^M
 

This was a clue, at first I thought that windows style rn line ending might be causing it. But it turns out is was that the file was saved using DOS encoding. Clearly it had first been created on the Windows machine. To fix it I opened my .gvimrc file, went to the Edit menu, File Settings, and selected File Format… I clicked on the Unix encoding and now both my MacVim and Windows GVim installs are working fine.

On Developers and Managers

I recently got drawn into a bit of a debate on a LinkedIn discussion group about whether a Development Manager should be spending time writing code or not. This is something I feel quite strongly about and the following is a heavily reworked version of my response.

If I asked a group of sales managers whether a head of sales should still be involved in some selling, I think most of them would say yes. Not full time of course, managing things does take time, but nobody thinks its strange for a head of sales to still do some sales meetings and prepare a few of the most important proposals. Same goes for Law, Medicine, Architecture, Project Management, Civil Engineering, Advertising, Fashion Design, Cooking and Teaching. But a Development Manager who writes the odd bit of code? Surely he/she needs to learn to let go!

Why is this? My theory is that it stems from a polarisation of the development and non-development IT cultures. 

Most new developers undervalue the non-coding parts of the development life cycle like analysis, testing and support. Probably like many freshly trained young professionals they view their supporting functions as (a) optional and (b) something they could do themselves if they felt so inclined. Unlike other industries though, this never really gets beaten out of them. Which leads to a strong developer sub-culture and removes incentives to participate in and ultimately master the usual political/interpersonal shenanigans that exist in any workplace. They simply choose not to participate, instead aiming to exercise influence by ignoring that world and just building what they know to be the right thing anyway. 

For their part, the rest of the IT industry often seems happy to let the developers run their own affairs without much of a challenge. It is very rare to see a non-developer attempt to argue with a developer on their own turf. Very few non-developers make any attempt to learn the basics of programming, read up on the fundamental principles or apply critical thinking to interrogate a programmer’s assertions. The rest of the industry give developers an easy ride and they shouldn’t. Ultimately both sides are equally culpable for allowing these two separate IT cultures to exist.

I don’t believe for a second that developers are not capable of the kinds of communication, persuasion and manipulation that it takes to progress into management. Very few of the developers I’ve worked with are truly autistic or at all confused about how to operate in political work environment they just look down on it. Similarly, I hate the almost religious aversion that many non-developers have towards writing anything remotely like code. I’ve seen intelligent people who can design processes, think logically and make Excel do amazing things be presented with the most straight forward of SQL queries and insist point blank they aren’t cut out for coding and therefore can’t (i.e. won’t) do it. It makes me want to cry sometimes. 

It’s this polarisation of cultures that leads to so few developers being asked to manage their teams and so few of those that are asked accepting the job long term. With so many non-developers managing developers it’s little wonder they seek to define the role as predominantly non-technical and purely “managerial”. 

This is a shame, because: 

  1. Many developers would benefit from some actual hands on management (they might not enjoy it at first though). They would be forced to learn. Their performance would be more accurately assessed and their design choices would be regularly challenged.
  2. Many development teams would benefit if someone with a bit of clout in the business spent some time in the trenches and saw what was happening. 
  3. Many managers would benefit from having coding skills. In fact most profession people would benefit from having some coding skills. Coding is not magic, it’s like selling or design or being able to chair a meeting really well, if you’ve got it in your armoury you’ll be more productive. 

My current role combines some technical architecture and customer consulting, but even just looking at the managerial parts, I use coding to help me get everything done.

Recently I’ve: 

  • Written a greasemonkey script to make our time keeping and reporting app easier for me to use. 
  • Learnt how to use the TFS power tools command line app and PowerShell’s XML commands to analyse and report on our progress. 
  • Used lots of regexs to search for things in documents 
  • Used lots of Excel functions to build planning ‘apps’ in Excel. 

In conclusion, if you’re a developer I recommend you don’t opt out of departmental politics. People are only a bit less predictable/knowable than complex systems but they are much more rewarding to hack. If you’re a manager in charge of developers I think you owe it to your team to get stuck in and really manage what they’re up to. Finally, if you’re just someone who uses computers for a living, learn to code. You’re sitting in front of one of the most powerful devices humanity has ever constructed, you’ll save yourself so much time!

Copying Work Item Templates Across Projects in TFS

To copy a work item template definition from one TFS project to another, install the TFS Power Tools and then type the following commands in the Visual Studio 2010 Command Prompt:

 


witadmin exportwitd /collection:http://tfsserver:8080 /p:"SourceProject" /n:"Work Item Type Name" /f:"C:PathToLocalFile.xml"
witadmin importwitd /collection:http://tfsserver:8080 /p:"DestinationProject" /f:"C:PathToLocalFile.xml"

 

The Concerned Observer Manoeuvre

In any crisis, after a few days the press run out of interesting things to say and resort to the “we told you so” story. The story goes something like this; “The people in charge of [whatever it was that’s gone wrong] were told that stuff might go wrong by [un-connected outside person/group] several years ago”. The recent problems at Japan’s nuclear reactors are no exception.

We’ve seen the same story before, in numerous forms…

Right now, someone, somewhere is organising a meeting or writing a report warning that just about everything that could conceivably go wrong is going to go wrong. If aliens landed on top of the Buckingham Palace tomorrow, by Sunday you’d be reading pieces in the press on how assorted think tanks, crackpots or publicity hungry consultants had warned of just such an eventuality and were now lampooning the government for not having prepared for it.

In retrospect, the reason for all this neigh-saying is obvious: There is almost no downside and lots of upside.

Consider this example, think of project at work that you are aware of but not directly involved in, is there a slight possibility it might fail to deliver or miss its deadline? If so, why not organise a meeting, invite the person who’s job it is to ensure the project is a success, plus as many others at you can get to come along. Once at the meeting, express your concern that the project may fail. Be sincere and polite, but if pressed for details be vague and over simplify the issues. Do not, whatever you do, suggest actual solutions or tasks, otherwise you open yourself up to being given work to do. The trick is to appear conscientious and pro-active in raising these issues without assuming any responsibility for avoiding them. Get as many of the by-standers as possible to agree with you that this is a potential problem while at the same time deferring to the “experts” and “those best placed to decide” when it comes to choosing what to do about it. If the project is a success after all, you can claim credit for having helped steer it away from disaster. If it does fail, you can claim to have had foresight and to have warned those in charge that they were heading for trouble. Either way, you look good without having to do any actual work.

It’s what I call, the “Concerned Observer Manoeuvre” and it’s just such an easy trick to pull off. Little wonder then that so many people are doing it.

Cleaning Source Control Bindings from Source Code With Powershell

I needed to prepare some code for release to someone outside our company. The code was a Visual Studio 2005 solution and I don’t have VS2005 installed any more. This made removing the source control bindings and debug files a pain as I couldn’t just open the solution and use Visual Studio to do it.

So instead, I came up with the following powershell code to remove the source control bindings and remove unnecessary files, which worked quite nicely:


# Remove unnecessary files  
get-childitem . -include *.vssscc,*.user,*.vspscc,*.pdb,Debug -recurse |   
    {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}{   
        remove-item $_.fullname -force -recurse   
    }  

# Remove the bindings from the sln files  
get-childitem . -include *.sln -recurse |   
    {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}{   
        $file = $_;   
        $inVCSection = $False;  
        get-content $file |   
        {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}{   
            $line = $_.Trim();   
            if ($inVCSection -eq $False -and $line.StartsWith('GlobalSection') -eq $True -and $line.Contains('VersionControl') -eq $True) {   
                $inVCSection = $True   
            }   
            if ($inVCSection -eq $False) {   
                add-content ($file.fullname + '.new') $_   
            }   
            if ($inVCSection -eq $True -and $line -eq 'EndGlobalSection') {   
                $inVCSection = $False  
            }  
        }  
        mv ($file.fullname + '.new') $file.fullname -force   
    }  

# Remove the bindings from the csproj files  
get-childitem . -include *.csproj -recurse |   
    {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}{   
        $file = $_;   
        get-content $file |   
        {781c323b309d6e174625241ec7481c1d30d3b1f9955792182ab0278b30a7fb7f}{   
            $line = $_.Trim();   
            if ($line.StartsWith('<Scc') -eq $False) {  
                add-content ($file.fullname + '.new') $_   
            }  
        }  
        mv ($file.fullname + '.new') $file.fullname -force   

    }
 

Automatically Creating a Database When Using EF Code First CTP5

I’ve been messing about with the new Entity Framework Code First CTP5 that was released recently. One thing that’s not obvious from the Gu’s blog post is how to automatically create your database if your not using SQLServer Express or Compact Edition.

Adding the following code you your Application_Start event in Global.ascx.cs seems to do the trick:


MyDbContextClass myDb = new MyDbContextClass();
myDb.Database.CreateIfNotExists();