Q33

Q33. How do we measure Output Quality?

Measuring output quantities alone will not give a fair indication of performance. These measurements relate only to workload or demand related responses. That is, the greater the demand for a particular service such as training, the greater the number of trainees trained. Measuring the quality of the output is equally important if we want to get a balanced picture of performance. It is pointless to boast of the numbers trained if, after the training, there has only been a negligible increase in the knowledge and skills of the trainees.

For example, the number of patients treated becomes irrelevant if the patients are not cured of their ailments. Hence, the rate of relapse despite treatment would be a good indicator of the quality of the treatment.

Similarly, good indicators of the workload of the Prisons Department include statistics on the number of prisoners attending rehabilitation programs, and the number of existing rehabilitation programs.

However, this does not attest to the quality of the rehabilitation service. The recidivism rate (the percentage of released prisoners who commit crime) is a better indicator of the quality of the rehabilitation program. A marked reduction in recidivism would signify that the program is working. An increase would, all things being equal, mean that the program has not had the intended effect. (Such a conclusion must necessarily assume that there are no other causes precipitating a relapse into crime.)

Quality indicators must also be specified, and should be ascertained from the client’s perspective of what they would consider to be a quality output or a quality service. The following are some of the questions that program managers can ask to identify quality indicators:

· “What are the features of the output that clients will mostly be interested in or most likely to complain about?” For example, one of the final outputs of the Meteorological Department is the weather forecasts that it provides for public use. The quality aspects of this output that the public will likely be most concerned would be:

q Accuracy: It would be pointless to say that it would not rain when it does rain, to the inconvenience and chagrin of those who had relied on that forecast;

q Understandability: A forecast that is couched in technical jargon or is ambiguous as to the expected weather condition at a particular time serves no purpose to the layman.

q Coverage: A weather forecast for a limited geographical area, such as Kuwait City, does little justice to the people in other parts of the country. Likewise, a forecast that says 20% to 30% of the country will experience rain tomorrow is meaningless for the population at large without further qualification. Even the statement ‘it will rain here and there in Kuwait’ is of little use to the people, as the actual area that will experience rain remains unidentified and the extent of the rainfall is not given.

q Timeliness: Is the forecast given in advance, with time enough to permit the public to prepare for the expected weather?

q Cost: At what costs are these forecasts given? What is the subsidy involved?

Another example:

For a public transport system, the final output is the number of passengers transported. The quality features of the service would be:

q Punctuality and/or reliability: It would be a great disservice to the clients if public transportation was inconsistent with its official schedule;

q Safety: A public bus system where the buses are driven by ‘kamikaze’ drivers is clearly undesirable;

q Cleanliness and passenger comfort: A bus full of litter, torn seats, and over-crowded does not offer a pleasant and a comfortable ride.

q Convenience: A meandering route that does not transport passengers efficiently, or a service that is irregular and unreliable can, at best, be described as highly inconvenient.

q Cost: At what cost is the service provided, and what is the average cost per passenger per kilometer? What is the subsidy, if any?

Once quality features have been identified, the next step will be to develop performance indicators that will help measure them.

· We can also ask “At what level of quality does a training program, for example, produce its final outputs?” This question would give us indicators of quality as, for example:

§ Percentage achievement of the course objectives;

§ Scores obtained by trainers;

§ Scores achieved for training techniques and training material.

Direct measures

Wherever possible, indicators should be able to measure the quality aspects directly. For example:

· Accuracy of weather forecasts could be measured by comparing the forecasts with actual outcomes. Here one possible performance indicator might be the percentage of materialised forecasts. Any indicator developed by an international meteorological organization or by other countries could be adopted as an index of accuracy.

· The average number and/or percentage of passengers who are forced to travel on buses or trains while standing can be a measure of passenger comfort

· The number of accidents, injuries and or deaths per kilometre travelled can monitor the safety of public transport system.

· The proportion of buses arriving on time can measure the reliability of the public transport system.

· Cost can by monitored by the unit price charged per passenger per kilometre and the unit cost of providing the service. The latter can be computed by dividing the total cost by the number of passengers per kilometres travelled.

Random Surveys

Not all quality features can be directly measured. Sometimes, we may need to seek the judgement of others.

For example, how does one determine whether weather forecasts are easily understood? We could ask the general public through periodic random surveys whether the forecasts make sense. However, collecting data through periodic surveys can be costly and the benefits would need to be evaluated before such an exercise is carried out.

Proxy Indicators

Proxy indicators or surrogate measures are another means of measuring quality. For example, the quality of teaching provided in schools can prove difficult to measure directly. We could carry out a survey among students and parents on teaching quality in a particular school. That could be expensive. It may even be unreliable given the possible lack of maturity among students to make a reasoned judgement on this aspect. Proxy indicators can provide an indirect indication of teaching quality. Some commonly suggested indicators include:

· Staff - student ratio: Research suggests that the ideal ratio for a primary school is 15 to 25. So, quality teaching can be inferred, in most cases though not all, if a teacher is assigned a class that has not less than 15 and not more than 25 students;

· Years of teaching experience and the level of teaching and academic qualifications: The rationale here is that the more experienced and qualified the teacher the better the quality of teaching;

· The extent teachers update their teaching techniques, materials and attend refresher teaching courses: Such an indicator might indicate a conscientious teacher dedicated to improving his or her teaching;

· Level of teacher morale: This itself can either be directly measured by a survey or through proxy indicators like the extent of moonlighting (giving of private tuitions for example), rate of sick leave, or the rate of early retirements. The rationale behind this is that the higher the morale the higher the quality of teaching, all things being equal.

· Percentage of student passes. The higher the passing rates the higher the teaching quality, all things being equal.

Proxy indicators are assumptive in nature. They do not necessarily always hold true. Notwithstanding, they are rough and ready measures of quality with least expense in measurement.

Ministry of Planning

1 October 2003