Up-Front Design in Agile Projects

The last few things I've read have got me thinking about where design work goes in Agile project management and whether it is consistent with creating the most value for the end user.

Agile methodologies of course claim to deliver value and seem, by many measures, to be a significant improvement over the traditional waterfall approach at doing so. But are they the best? I have a few significant worries about Agile applied in its most rapid, aggressive forms and they are:

1. People who make high level business decisions need good estimates of cost and schedule up-front to effectively plan and support a business strategy. There is a simple correlation between the amount of planning and design done and the quality of an estimate, which means better estimates possibly mean less agility.
2. It is during the up-front design phase that important core interaction ideas should be conceived, rather than allowing iterated, piecemeal attempts at GUI by the developers to calcify into the final product with limited clear vision. True value to the end user is in a usable product more than its feature list.
3. How long does iterative design take? We need to account for costs and time required to elaborate our user stories along the way and, just like code, some stories are stickier than others and can conceal enormous, costly problems.

(Before I go any further, I should make it clear that I realise that the perceived accuracy in an estimate driven by a very detailed up-front design is subject to a very large variance due to the unpredictable nature of the development process, so please don't stop reading because you assume I don't understand the value of Agile in controlling this!)

As with all problems concerning conflicting forces, the answer lies in a careful balance: a reasonable amount of up-front design (greater than the bare minimum) such that a clear vision for the software usage can be established and some estimates of cost and schedule can be made that are in line with the expectations of the stakeholders making the Big Decisions. I'd go so far as to say that the interaction vision should be non-negotiable after this point, but we need to accept that some of our design decisions may be called into question along the way and reconsider with care and reference to the initial design and those that created it. (Strategy and tactics: never lose sight of the main goal!)

As for accounting for the design time, this is as hard as estimating development time, so we should adopt the same methods for dealing with the uncertainty: just do it carefully (remembering to ask those that will do the design to create the estimate) and try to calibrate our estimating procedures over time with historical data.