I've been doing a lot of training recently, and I've seeing a lot of developers doing lots of different things, all with ColdFusion. But, you know, I hear a lot of the same things from these different developers. They have the same problems-things like scope creep and clients who won't give them the information they need.

But by far the worst problem-the really lethal problem-occurs when they finally deliver the software-only to have the client tick off a list of "gotta-have" items-items that their software doesn't have and that probably which were never even discussed!

Haven't we all been there? Demo'd the software we had worked so hard and long on only to have the client indicate not rapture but disappointment?

What in the world happened? How did we get to this point? Don't you remember those long-gone days when the project began and everyone was talking about how great it was gonna be? There was love in the air and not a worry in sight.

And that, my dear fellow developers, is the problem: Not a worry in sight. I'm going to try to convince you that we developers start off a project wrong-headedly. Naively. Blindly. But it may not take great rhetorical skill on my part-the odds are good that you've experienced this yourself and know something is wrong.

The odds, in fact are about 70% that you've been involved in a software project failure. Where have we gone wrong? And it's not like we haven't been trying. Over the years, people have done a LOT of work on the architecture and coding phase of the project.

Methodologies have improved. Object orientation, once an academic R&D project, has become mainstream. In languages, we have a richer selection than ever to choose from. And those languages work together better today than ever before. Many of you routinely build web applications that combine SQL, ColdFusion, JavaScript, C++ COM objects, and Java-sometimes all on the same page!

But we still have that awful statistic confronting us: 70% failure rate. May I suggest that part of the problem we have is that, helpful as the efforts have been to improve the architecture and coding phase of a project, we have left the real leverage point in software projects virtually untouched. We need help-but it's not primarily in areas of coding and architecture. In fact, I'll go so far as to say that the success or failure of a job is usually determined before a line of code is ever written.

Does that sound like nonsense? Well, think back on a failure you've experienced yourself and ask yourself why it occurred. Was it because the coders weren't able to implement a sort efficiently enough? Or just couldn't figure out how to connect to a datasource? Or couldn't master sort two-dimensional arrays.

I'm willing to bet it had nothing to do with the coding. It had to do delivering something that failed to solve the client's problem that was the failure, no matter how elegantly coded. And that is a failure in the requirements and design sections. All the improvements of architecture and coding won't help us with this simple fact: we aren't giving the customer what they want.

But just look at the language we use. We call the initial phase "requirements gathering", a name that suggests that requirements are scattered along our path and require nothing more than stooping to gather them into our basket, like so many Easter eggs set out for children to find.

And so, when we find only a few waiting for us, we assume we're gotten them all and that we're ready to do the one thing we all really love: start coding! But wait-I'm being too simplistic in my analysis. Sure, we love to code, but haven't we all heard how important requirements gathering is? Haven't we bravely suffered through interminable meetings with clients so that we could avoid the failures so common to our industry? Yes, we have. Then why does the problem persist? I was musing over this one day when a thought struck me:

Clients can't tell you what they want until they see it.

It struck me with the force of revelation. "Of course they can't" I realized. "If they could, they wouldn't need me; they could just go hire a coder directly to work out the details."

1. Attempts to force the client into decisions only cover up problems that will come out at the worst possible time-when the software is delivered.

2. It's my job to put something in the client's hands as soon as possible so that they can react to it.

3. The client reacts in phases. The development process, to be successful, must recognize and respect this fact. The system of software development must be built around this, rather than asking developers to try to shoehorn clients into a flawed and rigid methodology.

4. Developers aren't typically good at this "drip-by-drip" requirements process. We do love to code-and this isn't coding.

5. Coding itself should wait until we have gone through the process of determining what the client wants.

So, I began experimenting with different ways of helping clients identify what they wanted. I tried high-tech, low-tech, and no-tech. I read anything about the requirements phase that looked remotely promising. I read research on the types of development efforts that were most-and least-successful.

I tried formal use cases; I tried using strict UML; I tried role-playing with clients. I tried everything I could think of. The most success seemed to occur when the clients could actually see the application itself. So I thought of beginning with prototyping. But starting off with prototyping was too expensive; I didn't have enough information and I was building a complete prototype that was nothing more than a stab in the dark.

I thought of using one of the WYSIWYG editors to create a skeletal prototype-something I came to dub a wireframe. I thought this would be great, so I began cobbling things together using NetObjects Fusion. "Don't worry about the graphics right now," I told the clients. "Just concentrate on the functionality. What do you think?"

And they told me what they thought-that the buttons should be rounded and the blue should be more of a cornflower blue and not so much like slate. And that they'd like to have that 3-D look to the buttons. And maybe the navigation should be vertical instead of horizontal. What I eventually discovered was this: If I removed ALL graphics from the page, I got results. We could discuss what the application needed to do. I found that simpler was better. I eventually pared it down to the point that I had a corresponding wireframe page for each page in the application and each of those pages had only those things it was responsible for, (its content) and a list of possible ways out of that page or exit points as I called them.

It went like this: I would meet with a client, get an overview of what they wanted and then return to build a wireframe. Then I would meet with them again to go over the wireframe. I might have to go back and forth between revising the wireframe and showing it to the client again. This was definitely better, but it meant I was doing a lot of HTML work-that and running back and forth to go over the latest version of the wireframe with the client.

I thought of having someone else do the wireframe-after all it was just simple HTML, but I found the notes I took on what clients wanted were indecipherable by others, so I had to do this myself. I was stuck. What I had worked, but it took too long. I needed a better way. I thought that better way might be building a program to generate the wireframe automatically, so I put something together quickly to try out the idea. I thought that if I worked from a central text file-something very easy to put together, but something the computer could read, I might actually be able to make the wireframe while I was with the client. That would be fantastic, as it would cut down on the sales trips back and forth.

I borrowed the Windows .ini file format, as I knew ColdFusion had a couple of functions for reading and writing to this format. Now, on one page, I could build the structure of the entire wireframe. I saved the file with a .wir extension. Each page had a corresponding section. Each section consisted of the page's responsibilities along with its exit points-links or conditions that would cause a new server request.

The title would be displayed on the wireframe page when it was generated and the link would point to another section in the ini file. Now, I was ready to take this to the client to try out in front of them. To my great joy, I found that I could create the wireframe while I was with them. I knew this was important-it's hard to get all decision makers to come to several different meetings-especially when, in their minds, they've already explained what they need. This eliminated the need for repeated meetings by giving them the ability to react-and giving me the ability to revise in almost real time.

I'd like to show you how quickly you can generate a wireframe using this technique.

[DEMO]

So, are we ready to code yet? No. The purpose of the wireframe is just to get us to the point where we can begin the process of creating a prototype. The purpose of the prototype is to create something that will eventually look like the finished application. It's sort of like a multi-stage rocket where each phase propels the entire rocket farther and then falls off when no longer needed.

I create the prototype almost entirely in HTML, with the barest possible ColdFusion code. By avoiding a coded prototype, I can have members of the development staff who aren't coders work on this. In fact, both wireframe and prototype have been designed so that they do not require the efforts of a programmer, something both programmers and clients are grateful for-programmers because they don't have to be bothered with this phase and clients because they don't have to pay for programmers to do needs assessment.

There are going to be many iterations through this, and I've learned to welcome these, as revisions now mean I won't hear about missed requirements later. And changes at this point are cheap. All that needs to change is some simple HTML. The process of reacting and revising may continue on for weeks or even months and it forms, by far, the largest part in my overall development plan. But how do we know when we're done? Or does the project never get coded-forever remaining a prototype?

The rule is simple: We're done with the prototype with the client says we're done. Clients can decide on whether the most important factor is having the application full and complete, or having something up quickly, with possible later versions to enhance it. But once they agree the prototype is done-something I call "prototype freeze"-the changes stop.

And sometimes, going through the prototype process will reveal to the client that they don't need the application at all. This may seem undesirable from the developer's point of view, but I haven't found this to be the case. The client pays us for the prototype. We avoid any problems later when the client realizes they don't need it and begins looking for a way to wiggle out of payment and it gives us goodwill for future applications the client will need.

Now this doesn't happen very often, but it's part of an overall approach that says that clients should be able to manage the risk of building a software application-and that's something they which they appreciate.

Computers are incorporated in modern ice cream vending machines to enhance their functionality. Ice Cream Vending machines are manufactured by many companies. Your competition will try to overcome all requests for high-tech ice cream vending machines and credit card acceptors

While the prototype is going through its iterations, you will need some way of making notes. I decided that what I was doing-scribbling notes on whatever paper was nearby-wasn't really working. I could see myself explaining to the client that yes, I had made a note of what they said-and just to ignore the phone bill it was written on. What I came up with was something I call DevNotes. Not a great name, but a very useful tool for asking/answering questions about the project and keeping them in a centralized location. I use DevNotes in conjunction with prototyping, though other developers have extended it to work in a number of applications. The tool itself is very simple; so rather than describe it, let me show it to you.

[DEMO]

This information is stored in a database. Now, initially, clients won't use this. They're used to calling you up and telling you their ideas and concerns while you hunt for the back of an envelope to write on. I've found that by leading them to the prototype and the DevNotes ("Great point-have you entered that into the prototype notes yet?"), they quickly learn to record their thoughts, ideas, and concerns here where they won't be forgotten, overlooked, or misplaced.

This has another benefit. Once we do arrive at prototype freeze, the DevNotes are removed from the prototype, a sort of rite of passage that tells the client, "The time for changes is past." At one point, in another life I led, I worked in the construction industry as a cabinetmaker. Now, the building trade long ago worked out this problem of how to translate the fuzzy requirements inside a client's head into an actual product.

People have been building houses for a long time and we've worked out a very good system. If you go to an architect to have them design a house, they will provide you with a set of plans and specifications. Based on this, any competent builder can build the house. You aren't tied to one particular builder.

And builders don't have to have people trained by a particular architect-they just hire carpenters, masons, etc. who have in common the fact that they can read plans. The plans govern the building of the house-they let the client know what they will get and they let the builder know what to build.

Plans are a visual tool developed in a time when most people did not know how to read. They were developed to be both simple and easy to understand, while conveying a wealth of detail. I think we could do far worse than adopt these as virtues in our own industry. Certainly, we don't hear about 70% failure rates with buildings!

If we built houses the way we built software, plans would be replaced with lengthy documents with wording like: The structure shall consist of 2 floors, 8 rooms, 2 doors, 10 windows and shall have a peaked roof. The problem is that this description fits an enormous number of houses-but only one of those possibilities is the one the client is willing to pay for. Do you really want to take the odds that you'll hit on the right one?

Personally, I want a surer method for success. If we emulate the building trade, then the prototype serves as our plans and the DevNotes serve as our specifications. The old adage is right: a picture is worth a thousand words and a prototype is worth a thousand pages of wordy specifications.

Now, back to our hard-working architect. The building industry does this a bit different from custom software world. In our world, the design is often done without charge to the client. Architects-building architects, that is-don't give their work away in hopes that you'll have them build the house. They charge for those plans and specs and, in a sense that is the building. That's where the creativity and brainpower go. Competent workers can do the actual building of it.

Shouldn't we be doing the same? Are we really doing our clients any favor by not charging for the design process-and then having to rush what is, by its nature, a protracted process because we aren't getting paid for it? I find that clients appreciate it when I explain the development process and that, almost always, they understand that my charging them for the design is really in their best interests.

Those of you who have heard me or read my work know that I'm very concerned with the quest to place software development on a firmer footing, to continue the building analogy. This can only happen when all of the participants in the process get what they want.

Clients need applications that meet their real needs, even when they can't initially articulate those needs. Coders need to concentrate on coding, not trying to discern the unknown thoughts of an absent client. (We learned from the sad fate of the Psychic Friends Network how difficult this can be.) And finally, the development companies need to be able to make both their coders and their clients happy. And everybody needs to make money doing it.

That's our challenge, as I see it-and it's something far more important that arguing over JSP v. ASP v. CFML v. tomorrow's favorite language. Languages change. Companies change-as we found out recently. A far surer course than wedding ourselves to a particular technology is to put our efforts into solving these larger problems, for problems are never in short supply and great opportunity awaits for people who can solve them.