Garnett et all
Reviewed by Karen Pesse
The purpose of this article, as stated by its authors, is “to assist readers to critically assess and appropriately interpret the results of modelling exercises” which underlie the evaluation of results of specific interventions and that are used for policy decisions. The authors partially succeed to do so, since the article provides important inputs on the (potential) role of models for disease control programme evaluation and a detailed, albeit sometimes difficult to understand, description of methodologies for assessing the quality of models. It includes a checklist of the main items to be considered for a rigorous analysis of a specific model, but it warns the reader at the same time that fulfilling these criteria does not necessary mean that the model is appropriate, in the sense that it produces accurate outputs.
The authors explain the implications of uncertainty (due to complexity, unavailability of reliable data and variations among cases /individuals) and its effects on modelling, but the level of technicality might frighten some readers. One needs a reasonable level of epidemiological knowledge to follow all details of the authors’ statements. Nonetheless, even without such specialised knowledge, the article is interesting for many researchers and decision-makers, since it clearly makes the point on the importance of models… and their limitations. An important point is the recognition that models may often be presented very implicitly, especially when linear and simple relationships are assumed. The danger of this oversimplification of complex realities is that the model might turn out useless.
This last point represents, at least to my understanding, also the main weakness of the article: it tackles exclusively epidemiological, cause – effect, models, referring to mainly the “natural history” of specific diseases. The authors do not take into consideration social factors that influence the interventions for its control but also its definition and development. The complexity of any disease control program is thus underestimated; its multiple relations with the whole health system are never mentioned. For this reason, I would interpret some of the authors’ statements cautiously, i.e. the usefulness of models for predicting expected outcomes exclusively relying on data of intermediate outputs. Much more emphasis should be made on the importance of considering (local) contextual factors in these predictive exercises, even if no quantitative data can be retrieved for these aspects. An in-depth discussion on the limitations of these kind of data (not everything can, nor should, be put into numbers) is also missing; authors seem to think that numbers are always objective, even recognizing that all models are based on a certain “scientific paradigm” that, as such, could be challenged.
A very interesting and useful part of the article is the detailed description and discussion of different kinds of models, their use, strengths and weaknesses. Although the authors again refer exclusively to quantitative and epidemiological models, they show the diverse aspects that need to be taken into consideration for the development of models. Some of them, such as the difference between a deterministic and a stochastic approach, should be kept in mind while designing more socially- orientated models too.
In summary: an interesting paper for anybody struggling with the development of models that supports research and/or decision–making, even if not all technical nuances presented by the authors are applicable for the analysis of health policies or systems.