Real-world evidence in health technology assessment of high-risk medical devices: fit for purpose?

Health technology assessment (HTA) of medical devices (MDs) increasingly rely on real-world evidence (RWE). The aim of this study was to evaluate the type and the quality of the evidence used to assess the (cost-)effectiveness of high risk MDs (Class III) by HTA agencies in Europe (four European HTA agencies and EUnetHTA), with particular focus on RWE. Data were extracted from HTA reports on the type of evidence demonstrating (cost-)effectiveness, and the quality of observational studies of comparative effectiveness using the Good Research for Comparative Effectiveness principles. 25 HTA reports were included that incorporated 28 observational studies of comparative effectiveness. Half of the studies (46%) took important confounding and/or effect modifying variables into account in the design and/or analyses. The most common way of including confounders and/or effect modifiers was through multivariable regression analysis. Other methods, such as propensity score matching, were rarely employed. Furthermore, meaningful analyses to test key assumptions were largely omitted. Resulting recommendations from HTA agencies on MDs is therefore (partially) based on evidence which is riddled with uncertainty. Considering the increasing importance of RWE it is important that the quality of observational studies of comparative effectiveness are systematically assessed when used in decision-making.