I am in the middle of a non-computer related project. Last week we have decided to get rid of carpeting in my home office and library and to put in the hardwood floor. My wife picked the wood and color, I picked the contractor, with my son's help we moved out the furniture and the renovation started. The most painful thing was shutting down and disconnecting all the computers and servers - only the lonely router is hanging on the wall and suffering the dust and noise of the power tools. As result, my family is now server-less and until we finish and connect them back, we have to survive using the notebooks only. Fortunately, I have at least one notebook available for each household member, except the cat. Even more fortunately, she (the cat) does not mind.
Surprisingly, this project reminded me of something that is so familiar when you are working in enterprise IT environment and deal with real life systems and real companies: you cannot do just one thing. After we uncovered the carpets, we have discovered few hidden issues that had to be fixed before we could put the hardwood in. In process of fixing it, another issue popped up and we ended up replacing the plywood that was under the carpet. As result, what was supposed to be done in 2-3 days, will take twice as long and will carry according price tag. In all this we are extremely lucky having very good, capable, hard working and honest contractor doing the work.
So, does all this renovation have anything else in common with software engineering? I think it does. Year or so ago, we had this discussion about metaphors, paradigms and models for the software development process. One of the discussion's topics was why is our (presumably engineering) profession so bad in estimating the time and effort required. Look at the guys building the houses - Steve challenged us - they can tell with 5-10% precision when they will be finished and how much will it cost. Even better, when customer changes his/her mind in the middle of the project, the impact of the change on timeline and price can be determined with fairly high accuracy.
We have talked about obvious differences between building a house and building a software system. The house is pretty much always the same - can be bigger or smaller, but the variations are very small compared to possibilities of software systems. There is very small selection of materials, the components are well standardized and the process of building has been tested and fine tuned on millions of houses built. The workforce in construction business is very much standardized as well - the productivity can be easily measured and deviations between the most and least productive worker are much smaller than in software world. The laws of physics help too - by making the completely wrong combinations of components more obvious and easier to spot. In short, construction of a house is bad metaphor for building a software system.
What seems to be much better metaphor (if we have to stay in the realms of construction business) is house renovation. Here you have to make assumptions and will make discoveries as you progress - and this maps nicely to the process of requirement "gathering" (whoever created the term gathering was idealist, in reality it is much closer to digging for requirements, but back to the topic). You start assuming that your job is to put in hardwood floor - or to integrate two software systems. In reality, your job is to resolve unknown number of problems of unknown complexity in order to even start.
You have the legacy of existing house, plumbing, infrastructure - similar as you have legacy enterprise systems, databases, data feeds, business owners and so on. You may be aware of some them before you start, but their true size, nature, status and complex interdepencies will became fully understood only during the process of renovation (or integration).
So what is the moral of the story and what is the right approach ? I have no answer that would work in every situation. What worked for me - is honest communication and transparent decisions. Make sure that the customer understands what is included in the estimate, what are your assumptions and what can potentially be hidden issues if there is an indication you may find some. Whenever you discover something that may change the scope, do not delay telling the client and make sure he/she understands all possible pathways and their consequences.
Be generous - if you need to invest extra time to make sure the recommended decision is the best option for the client, invest that effort regardless whether the cost will be covered by the client. If you go that extra mile, in long run you will be always rewarded - with repeated business, better rate or at least the good karma for doing the right thing.
Document all agreements, discoveries, options presented and decisions - they may be great reference when facing similar decision in your next assignment. And sometimes the document trail you leave behind may be only existing documentation of the system in question. The contractor that comes next, will thank you for that. If you do your part properly, that somebody may be the future you.