Home Career

This issue deals with planning for proposal evaluation. First it covers the evaluation criteria and the importance of preparing them before any documents are issued to potential providers. It then goes on to discuss the evaluation weighting process. Then it discusses the issues of subjective or objective evaluations and their applicability in certain circumstances.

Evaluation criteria

Although much has been written on the subject of preparing Requests for Proposals (RFP) and Invitation to Tender (ITT) documents, there are still many people who imagine that they know the principles of these acquisition methods and who prove by their actions that they do not.

An example of this are the people who prepare the RFP or ITT, send it out to their prospective providers, and then build their evaluation criteria. All of the sensible publications on this subject state the importance of developing the evaluation criteria first, before they construct the RFP or ITT. At the risk of being tedious to many of my readers, I shall repeat the reasons.

However detailed your documents, and however much emphasis they place on technical solutions, they are intended to help you to select a vendor or provider who will satisfy a particular business requirement. Before you can describe the technical solution, you must be aware of what that business need is. You must also be aware of its importance to your organization and of the outcomes that will be produced as a result of it being satisfied.

It is the completion of the definition of the business need, its importance and its outcomes that will enable you to prepare first the evaluation criteria and then the proposal documentation.

Let us look at retail systems as an example. The proposal documents may be set up to describe in detail how the retail systems may be expected to operate, based on a conceptual model of retail systems that is itself based on what technology solutions are currently available. This often happens. The result is that the proposal documents ask vendors to provide a solution that the customer knows is the solution that the vendors provide: the customer will describe the vendors' current Point of Sale (POS) systems. This focus on the technical aspects may completely overlook the business requirements of the retail sector. Let us look even closer at what some of these are. In a retail business, it is not enough to record sales and to issue orders to replenish goods when they reach a certain level, which are all things that POS systems do. Retailers also want to know when a customer has been into a store and asked for something that they do not usually stock or that is out of stock or that is not available to that customer's specification (where the customer wants it in red but there's only blue) in the store. Standard POS systems do not handle these situations:

• they do not record demand substantially different from supply when the customers says "I want personal notepaper but you only have business stationery," even though this may identify new stock lines that the store should consider stocking;
• they do not record demand beyond supply when the customer says "I want E4 envelopes but that shelf is empty," although they may take orders; and
• they do not record demand marginally different from supply when the customer says "I want white C4 envelopes but you only have manila," although, again, they may take orders.

Taking orders is all very well, but what of the customer who leaves the store without making a purchase or placing an order? That's lost business.

So retailers need systems that will reflect these business requirements, and they need to describe those requirements in their proposal documents. They should also avoid editing by their technical staff to remove requirements to which there seems to be no technical solution.

Once the full list of business requirements has been developed, the proposal documents can be completed. If there is a need to include technical issues, such as interfaces and performance requirements, then they can be included in the document. Meanwhile, the business analysts can set to work on the list of business requirements, applying a priority to each of them.

Evaluation weighting

It is useful to complete the evaluation weighting before the proposal documents are sent out to potential providers, but is not usually necessary.

In some instances, I have known the team responsible for the evaluation weighting - the "weighting team" - give a weight of zero to an item in the proposal documents. This meant that, however well and fully the potential providers answered the question, their answer would be worth nothing during the evaluation. To avoid this happening, it may be useful to complete the weighting before the proposal documents are sent out.

This will also avoid the difficulties that occur when the weighting team, who are usually senior management, wish to include a new criterion for decision-making. If the proposals documents have already gone out, some means will have to be found of marking against that criterion on the basis of existing questions, designed for other criteria.

Of course, a proposal that has been properly constructed by professional business analysts should not suffer from either of these problems, so it may not be necessary to complete the weighting before the proposal documents go out. However, I prefer to be sure rather than sorry.

Whatever the order in which the work is done, it is essential that evaluation weighting is completed.

Where there are straightforward, "single level" questions to be answered, the weighting can also be straightforward and at a single level. For example, if one of the questions to be resolved is "Can we trust them?", with no supplementary questions, then that question can be given a simple weight of zero to ten, depending on its importance to the business and its objectives.

Had there been a section called "Trust" that contained a series of questions, each of which was to be answered individually by the potential providers, then a more complex, "multi-level" weighting scheme would be required. In a multi-level scheme, the topic (such as "trust") is given a weight relative to the other topics, and each question within a topic is given a weight relative to the other questions within that topic. The weight to be used for any question is the product of the weights for the question within the topic and for the topic. This may produce enormous numbers when the scoring is done, but that should not be a problem. For one proponent to score 17,865,460 is not remarkable if the total possible score is 30,000,000.

Incidentally, experience has taught me that the best range for both weights and "marks" is from zero to ten. Both the weighting and the marking teams can have difficulty when dealing with percentages, and arguments can arise as to whether the appropriate figure is 63% or 65%. With a range of zero to ten, these arguments disappear: I don't know why. It just happens to be my experience that the weighting and evaluation teams can more easily settle on a figure of 6 or 7 than one of 60% or 70%.

When marking, it is also better to use zero as "no" and ten as "yes" in a question that can only be answered with "yes" or "no". This means that every question has been answered on the same range of zero to ten and makes construction of spreadsheets formulae easier.

Subjective or objective evaluations

It is my contention that the RFP (or ITT) process should use objective evaluations and that the MoU process should use subjective evaluations. The Topic "001" provides some material that differentiates between the RFP and MoU processes, but does not go into this area in depth.

t001

The RFP process is designed for rigidity. Its one major advantage over the MoU process is its rigidity, which can be essential in some cases. In information systems, for example, financial systems may be acquired through the RFP process: it is highly unlikely that the MoU process would be used.

As a result of this rigidity, the RFP process needs objective evaluation. If any question has to be marked on the basis of how people feel about the responses to that question, then the RFP process is devalued. The questions demand factual responses, often backed by evidence.

The MoU process, on the other hand, is designed for flexibility. Although some of responses to the questions will be factual, most of them will be marked in a subjective manner, based on how people feel about the responses and how the responses match their previous experiences.

Whilst no-one objects that the RFP process is objective, some people are unhappy about the subjectivity of the MoU process. They should look at what the MoU process is trying to achieve.

First, it is attempting to find a solution that will deliver specific objectives, without prescribing how this is to be achieved. It is likely that each proponent will put forward a description of how they intend to deliver the objectives, but these methods may be so different from each other that it would be very difficult to differentiate between them on a purely objective basis. The markers will have to differentiate between them on a subjective basis: how successful do they feel that a given solution will be?

Second, and especially in outsourcing, a single solution, regardless of how it compares with other solutions, may be so beyond the experience of the markers that they cannot mark it objectively. An outsourcer is required to be innovative and to apply specialist skills across a wide range of customers, skills that those customers could not themselves deploy. As a result, the current business, systems and methods of the customer organization may be markedly different from the improved, stream-lined processes offered by an outsourcer. How, then, is the marking team to judge such new, innovative approaches?

Of course, if the vendors and providers who complain about the lack of objectivity of the MoU process were to base this on the fact of a lack of innovation in their own proposals, I would be more prepared to listen to their complaints.