Home Career

Usually people build just one evaluation matrix. This matrix is a "preferences matrix" and will be discussed first. There is one other matrix which is almost always useful (a "requirements matrix") and a third (a "qualifications matrix") that may be useful. These two matrices will be described later in this issue.

Preferences matrix

The preferences matrix is the classic evaluation matrix.

Criterion	Weight	Max score	Vendor A's score
Integrated general ledger	8	1	1
Report writer	6	1	1
Flexibility	8	100	80

Table 52.1: Classic evaluation matrix

This example shows three criteria. The first two are "yes/no" criteria. The score for them can be either "0" or "1". They are differentiated by their weights: the first has a weight of 8 (out of 10) and the second of 6.

The third criterion is marked on a percentage. In this case, Vendor A has scored 80%.

The weighted score for the first criterion will be 800, as "yes/no" answers are either 0% or 100%.

The weighted score for the second will be 600.

The weighted score for the third will be 640.

This method ensures that preferences that are important receive a higher weighting than those which are less important. What it does not deal with are "drop dead" criteria. There is no point in giving "drop dead" criteria a higher weighting. That will only increase their relative score. It will not ensure that they are actually present. What is needed is a mechanism for ensuring that a score of less than 100% on any "drop dead" criterion will trigger an action. (Often, this action will be rejection of the proposal, but there are alternatives, as we shall see.)

Requirements matrix

A requirements matrix will serve this purpose.

The matrix contains requirments rather than preferences. There is no weighting. Each proposal is evaluated against the requirements matrix first. If it score full marks it can then be evaluated against the preferences matrix.

There are some issues about what you can do if a proposal does not score full marks on the requirements matrix. The usual action is to reject the proposal. I do not believe that this is very productive. Where possible, I prefer to ask a vendor whose proposal has missed only one criterion to confirm its intentions or to submit a correction for the proposal. The omission may have been unintentional, and I am not happy with rejecting a proposal over a minor issue that could be cleared up easily. If there are more than three omissions, I reject the proposal. For two or three omissions, I will consider whether to reject the proposal or to work with the vendor to correct the omissions.

Remember that many of the issues that are critical to you, as a customer, are routine to the provider and it is easy for a provider to omit something that is obvious to it. The justification is sometimes given for rejection is that "if the provider can't get the proposal right, what is their service going to be like?". This is spurious: the business of the provider is to provide services, not to write proposals. Mistakes and omissions can be expected, and accepted.

In some situations, none of this is possible, and the rules have to be adhered to. Any proposal which does not fully satisfy the requirements matrix will be autimatically rejected. I understand the need to do this where the fairness and contestability of the selection process have to be transparent. I still think that it is a pity to reject what could be the best proposal because of a minor error.

However you wish to approach these matters, a requirements matrix will often be a useful way of determining whether proposals should continue into the evaluation based on preferences.

Qualifications matrix

There are occasions when it can be useful to split the preferences matrix in two, forming a new qualifications matrix.

The qualifications matrix shows all the preferences on which subjective evaluations have been made; the objective evaluations are retained in the preferences matrix. The two matrices are scored separately.

The preferences and qualifications matrices can be used in a number of ways.

Where the decision is primarily objective, they can be used together, so that a preferred provider can be selected based on their overall scores. In these circumstances, it is easy to demonstrate which proportion of the overall scores were subjective. Ideally, use of subjective evaluations should not alter the relative positions of the providers but should serve to increase the differentiation between them.

Provider	Objective score	Subjective score	Total score
A	100	100	200
B	90	70	160
C	70	40	110

Table 52.2: Qualifications matrix

This is demonstrated above. The total scores widen the gaps between the providers without altering the order, based on the objective scores.

If the scoring process can actually be subjective, then the following table is acceptable.

Provider	Objective score	Subjective score	Total score
A	80	100	180
B	90	70	160
C	70	40	110

Table 52.3: Objective and subjective scoring

Often, management may declare at the outset that the process will involve subjective evaluation and may baulk at this form of result. They may, after all, prefer a purely objective process. Whatever the feelings of the evaluators may be about this, the use of the two matrices does enable the process to be shifted towards an objective evaluation, in which Provider B would be the preferred provider instead of Provider A.

Of course, if the evaluation process is intended to be purely objective, then use of the two matrices will enable criteria to be shifted to the qualifications matrix during evaluation, whenever the evaluators become aware that they cannot evaluate on wholly objective grounds. This will enable decisions to be made on whether the criteria moved to the qualifications matrix can be dismissed from the evaluation process or whether they should be retained. If they should be retained, then a means will have to be found for the evaluation to be objective. This might mean obtaining more information from the providers before the evaluation can be completed.

Whichever set of circumstances the two matrices are used in, they provide additional insights that use of a single preferences matrix cannot.