Most software projects overrun. However, the interesting thing is that almost none of them overrun at the end of their development cycle. At the end of the cycle, projects are usually in tip-top shape. Everyone is working really hard, there's momentum and clarity and a list of bugs or work items in a database, programmers are programming, testers are testing, and all is really running quite efficiently. Note: We're talking here about healthy projects that overrun by some margin, not engineering disasters. It's not the activities at the end of the development cycle that cause a project to overrun, and little could be done to improve them.
Rather, typical projects overrun by several months at the beginning of their development cycle. At the beginning projects are usually a mess: Nobody knows what to do. The architect or designer is holding everyone up. People are afraid of building UI features in case the design changes. Testers are not contributing. There's no momentum or clarity. The worst thing is that pieces that stress each other out, for example back-end and UI, are not working together and so the people writing each part don't get to observe the more subtle bugs in their designs. It's clearly at this time that the slip occurs, and the fact that it's only acknowledged and considered a crisis at the end is just human unwillingness to face hard facts.
Now, most traditional software project management wisdom is about improved tracking of some kind or another: Better estimating, not losing tasks, requirements tracing, not gold-plating, etc. This looks to me like an attempt to improve the conditions that are typical of the later stages of a project - which are not the problem. What is needed, I think, are systems that achieve as much clarity and momentum as quickly as possible at the beginning of the project. And there the problem leaves the field of software engineering and enters organisational management.
The work loading of a textbook software project is teardrop-shaped, with the more pointy bit at the start. This is supposed to reflect the idea that at the start you're figuring out what to build, or how, so you assign an architect or designer and one or two others to set up a solid foundation. When that is ready you "ramp up" the project to its full staff. Well, that never works in practice. Management balks (correctly) at the total elapsed time of the teardrop project, and developers come off some other project and join you anyway. The end result is the staff loading of the project is actually box-shaped, but the productivity is still teardrop-shaped, and the mismatch eventually manifests as a slip.
To make a considerable number of developers productive at the start of the project, as per the realistic, box-shaped loading, you need to be able to run a chaotic project. People need to be able to work without direction, and build things that will probably end up as pieces of the solution, although no-one knows exactly what the solution is yet. Running a project this way is a tricky and not yet well understood management task, and may make organisations quite nervous. I think the agile/XP schools have codified many of the right practices: Tasks of interchangeable size, close proximity with the customer, empowerment, test-driven programming, refactoring, constant review. At the engineering level, these concepts are probably solid enough to demonstrate effectiveness and command respect.
My guess is that higher-level concepts of a similar nature would benefit software firms organisationally. The key point is if the firm really wants to achieve as much engineering productivity as possible in a short time, it has to delegate a large amount of control over what is produced. People have to be able to make, essentially, local decisions about how to solve pieces of the problem, which would then be brought together to make the complete solution. Inevitably, such local engineering decisions will affect the end product. That presents a problem with traditional product management, which, for some very good reasons, wants to release coherent and controlled products to the market.
I've only got a vague idea of what the solution might be like, and that is to create a more agile concept of product management. The key observation is that bottom-up engineering decisions, although perhaps not optimal, would usually be quite competent. You might need to change one or two but most would be good. Some means is needed to connect this competent but unpredictable way of creating products with the needs of the users, in the way that XP does by having a "customer" representative but at a larger scale to target a whole market. I have a faith-based belief that individual customers tolerate, or even like, a large amount of iteration in the products they use, so long as the products remain solid and there is a robust update mechanism. Getting an organisation to adopt such iterative or agile product management, though, is an interesting organisational challenge.