Deliverables Overview

D1.1 Set-up of Focus Groups and Delphi study

This document presents the methodology adopted for setting up the Focus Groups and the Delphi Study (including stakeholders/experts engagement) and the preliminary surveys developed to enable the realization of an intermediate report about the usage of terminologies in Europe (and beyond), with a special focus on SNOMED CT.

This goal is expanded in several objectives:

  • Collect information about the past, current and future usage of SNOMED CT in Europe at all the involved levels (Cross-Borders/National/Regional); with a view also to non-EU experiences.
  • Collect information about current and future possible policies of IHTSDO, including cooperation with other SDOs, and about the usage of SNOMED CT within other standards
  • Assess whether SNOMED CT satisfies the criteria listed in annex II of the EU standardization regulation
  • Identify - among those realized - appropriate use cases suitable for performing an evidence-based assessment;
  • Provide an overview of the commonly recognized advantages and drawbacks of SNOMED CT.

The aim of the ASSESS CT country focus groups is to solicit European Views on current and future use of terminology in the health care sector with special focus on the use of SNOMED CT. Four themes where selected to underpin all country focus or discussion groups:

  • Current use of terminology
  • Barriers for extended terminology adoption and use
  • Enabling factors for extended terminology adoption and use
  • Recommendations

D1.2 Report from the Focus Groups and questionnaires

The scope of deliverable D1.2 is to report the results, gathered by mid July 2015, of country based focus groups and of the questionnaires developed for investigating the usage of terminologies in Europe (and beyond), with a special focus on SNOMED CT. The results of the EU/US focus group and of the First Validation Workshop (as part of the initially agreed Delphi Study approach) are only referenced here since they are documented in the ASSESS CT deliverable D1.1. Details about the overall methodology, including goal and objectives of work package 1 “Current Use of SNOMED CT” are also provided in D1.1. The structure of this deliverable is organised per activity accomplished, in particular it provides:

  • The relevant results of the focus group activities as available at the time of the delivery of this document. (Section 4 "Report on Focus Groups”). The focus group reports are documented in  Annex 1: Focus Groups Reports
  • A report of the All Stakeholder Questionnaire based on the answers received by July 9th, 2015, coming from 14 countries. (See section 5.1).
  • The complete results are included in:Annex 2: All stakeholder questionnaire
  • A report of the Country Overview Questionnaire based on the answers received by July 9th, 2015, coming from 14 countries. (See section 5.2)
  • The complete results are included in:Annex 3: Country overview questionnaire
  • A report of the EU / US discussion group, as reference to the results documented in D1.1. (Section 6.1 Report on EU US discussion group)
  • A report of the 1st Revision Workshop, as reference  to the results documented in D1.1 (Section 6.2 Report on 1st revision workshop)

Several facts, opinions, experiences, and beliefs have been gathered through the national focus group investigations:

  • the role of SNOMED CT as a reference terminology and mappings broker;
  • use cases for adopting SNOMED CT: e.g. terminology have to be used and assessed within specificied purposes or contexts of usage, and should not be used beyond their purposes
  • the current lack of evidence of benefits: SNOMED CT has in general a limited usage, consequently there is a low availability of evidence/best practices/examples.
  • (low) market maturity and potential impact of the adoption of SNOMED CT: e.g. SNOMED CT may play a role in the standardization of the EHR content, and therefore in the EHR-S market;
  • strategic long term benefits: IHTSDO assures a transparent and robust maintenance process;
  • pre conditions for pursuing semantic interoperability: e.g. well defined strategies, supporting policies and jurisdictional commitment;
  • usability and users acceptance: e.g. the usability – in all their different perspectives - is a critical factor in the adoption and acceptance process;
  • licensing and cost issues: e.g. the SNOMED CT licence cost is a critical barrier in the decisional / start-up phase when the potential benefits of this change have not been yet completely evaluated / experienced;
  • suggested approaches for introducing SNOMED CT: a step-wise, use case based, incremental approach is the suggested method for the introduction of SNOMED CT, possibly starting from inadequately covered domains.

D1.3 Current and Future Use of SNOMED CT - Interim Report

The "Current and Future Use of SNOMED CT" interim report documents:

  • The country overview questionnaire updates received in the period from October 2015 to March 2016;
  • The results of the questionnaire extended to the non-European IHTSDO member countries, the updates from the Portuguese and French Focus Groups, whose activities ended in January 2016;
  • The results of the four cases studies selected and of the literature review; the additional inputs collected during the second revision workshop;
  • An overview of the IHTSDO policies including the collaboration with other SDOs.

The report also completes the picture presented in D1.1 and D1.2 on the use of terminologies in Europe and worldwide. It will be completed and integrated into the final deliverable D1.4 with (a) an updated review of the assessment of SNOMED CT against Annex II of the European Standardization Regulations, and (b) results from an EU / US meeting in May 2016, and (c) any update concerning the assessment of case studies and the country overview questionnaires.

Having identified specific areas or topics where there are gaps in knowledge or a strong need to explore evidence, this report will feed into D4.3 to provide recommendations on further actions.

The conclusions of the report revise:

  • The role of SNOMED CT as a reference terminology and mappings broker
  • Use cases for SNOMED CT adoption
  • The current lack of evidence of benefits
  • Market maturity and potential impact of the adoption of SNOMED CT
  • Strategic long-term benefit
  • The approach towards the semantic interoperability
  • Usability and user acceptance
  • Licensing and cost issues
  • Suggested approaches for introducing SNOMED CT

D1.4 Current and Future Use of SNOMED CT

ASSESS CT has investigated the suitability of SNOMED CT to act as a core reference terminology supporting EU-wide eHealth deployments. WP1 has contributed to this by investigating existing experiences in the use of this and other large-scale terminology systems. Actual experiences have had to be complemented by expert consensus, because empirical evidence and formal evaluations are still limited.
Given the complexity of this topic a mixed methods approach was used including a literature review, online and interview questionnaires, focus groups, workshops and feedback on interim findings during conference presentations and invitation only events.
This report will interplay/feed into D4.3 and D4.4 to support recommendations on further actions. In the context of PHC34 cooperation the final version will be used to provide input to several future eStandards deliverables including D2.2, D2.3, D3,4, D3.5, D5.3, and D6.3

D2.1 Multilingual and multidisciplinary study of terminology coverage and quality - interim report

Workpackage 2 of ASSESS CT seeks empirical evidence for the fitness for purpose of SNOMED CT, compared to other terminology settings. As a testbed for the measurement of (i) concept coverage, (ii) term coverage, and (iii) inter-annotator agreement as quality indicators, two experiments were conducted: manual terminology annotation of a parallel corpus of clinical text snippets in six languages, and manual binding of terminology codes to clinical information models.

For the annotation use case, a SNOMED CT-only terminology setting was compared to a hybrid terminology, based on an extended subset of the Unified Medical Language System (UMLS) meta-thesaurus. For the binding use case, SNOMED CT was compared to a hybrid of four international terminologies.

The results of the text annotation experiment showed no significant superiority of the extended UMLS terminologies compared with SNOMED CT for languages in which a full translation of SNOMED CT is available (English and Swedish). The coverage of translations of SNOMED CT subsets (in French and Dutch), however, was lower than available alternatives. The benefit of the availability of synonyms could also be clearly shown. Apart from the English alternative scenario, analogously built alternative terminologies in Dutch, French, and Swedish showed much lower concept and term coverage compared with what could be shown in the English SNOMED CT scenario.

The terminology binding experiment, constructed to reflect a key standardized terminology use case, showed a better performance of SNOMED CT both regarding concept coverage and agreement compared with a set of the four widely used international standard terminologies ATC, ICD-10, LOINC, and MeSH.

The fact that SNOMED CT is a single-source product, with periodic releases, downwards compatibility and a uniform licence management, issued by an international non-for-profit organization is already an advantage over hybrid terminology settings, such as those constructed for these experiments which are partly dependent on non-European sources and provide a good coverage only for English. The outcome of our experiments suggests that SNOMED CT is capable of meeting the needs of annotating free text and binding to clinical models, in languages for which a translation exists, at least as well as an alternative hybrid solution, and better in the case of clinical model binding.

However, the restriction to SNOMED CT subsets as an alternative to large-scale terminology localisation (as done in Belgium for French and Dutch) must be carefully checked against the use cases to be addressed. 

D2.2 Use of terminologies for representing structured and unstructured clinical content – interim

Workpackage 2 of ASSESS CT targets empirical evidence for the fitness for purpose of SNOMED CT, compared to other terminology scenarios. In a variety of experiments using terminology settings reflecting the ASSESS CT scenarios, aspects of terminology use have been studied, and results have been collected and analysed. Results related to terminology coverage and inter-annotator agreement have been reported in the ASSESS CT interim deliverable D2.1. This deliverable focuses on preliminary results from an experiment applying Natural Language Processing (NLP) techniques for automatic annotation of clinical free-text documents, a qualitative analysis of annotation disagreement, and a comparative study of the results from the terminology binding (structured) and free-text annotation (unstructured) experiments.

For the NLP experiment, as a basis for gathering evidence, a SNOMED CT-only setting was compared to a hybrid terminology, based on a subset of the Unified Medical Language System (UMLS) meta-thesaurus. Results from this experiment are preliminary.

Both the annotations made in the free-text annotation and the terminology binding study have been studied qualitatively to identify categories of terminology use and specifically types of disagreement. Similar types of disagreement occur in both free-text and binding experiments, most likely due to features of the terminologies used.

Deliverable 2.2 describes the methodology of applying Natural Language Processing (NLP) techniques for automatic annotation of clinical texts, a qualitative analysis of annotation disagreement, and a comparative study of the results from the terminology binding (structured) and free-text annotation (unstructured) experiments.. The analyses of this deliverable are still in early phases of development and a richer analysis will be provided in the final deliverable.

D2.3 Multilingual and multidisciplinary study of terminology coverage and quality

This document describes two studies that scrutinize the fitness for purpose of SNOMED CT, compared to other terminology settings. Terminologies were tested for coverage and agreement in clinical text annotation, as well as in manual binding to clinical information models. The latter use case showed a better performance both regarding concept coverage and agreement for SNOMED CT. The former showed equivalence of English SNOMED CT free text annotations to an alternative, UMLS-based scenario. The Swedish SNOMED CT version was superior, but the French and Dutch versions, where only translations of a subset were available, were inferior to the alternative.

D2.4 Use of terminologies for representing structured and unstructured clinical content

Deliverable 2.4 introduces the notion of user interface terminologies in contrast to reference terminologies. Three terminology settings (SNOMED CT against an alternative, UMLS-derived hybrid
terminology and a local terminology collection), are analysed under user interface terminology aspects. It investigated the coverage of such interface terms in these three terminology scenarios against clinical text samples in six languages, using natural language processing and manual annotations. It furthermore proposes a method on semi-automated creation of user interface terms and describes its implementation, ongoing maintenance and formative evaluation.

D3.1 Assessment Framework: List of Cost and Benefit Indicators

Using results from literature review, survey, interviews, focus groups and workshops, ASSESS CT is reviewing the current state of use of SNOMED CT, the fulfilment of semantic interoperability (SIOp) use cases, known technical and organisational drawbacks and the way the terminology is improved and maintained. ASSESS CT scrutinises the adoption of SNOMED CT Europe-wide scenario (ADOPT) against two alternative scenarios: to abstain from actions at the EU level (ABSTAIN), or to devise an EU-wide semantic interoperability framework alternative without SNOMED CT (ALTERNATIVE).

This deliverable is based on discussions and work from weekly WP3 meetings, input and comments from external experts through the first ASSESS CT workshops, and input from the other ASSESS CT WPs. It focuses on the definition and listing of potential costs and benefits when adopting a clinical terminology, with the particular, yet by no means exclusive, example of SNOMED CT. It provides a structured, balanced analysis of possible benefits and cost categories and respective indicators as discussion ground for a final socio-economic assessment method. 

The numerous cost indicators have been grouped into four categories / groups of indicators as described in this section. This categorisation is preliminary and will be reviewed in the upcoming months as required, in order to better reflect the main types of costs related to terminology adoption.

  • Costs related to organisational infrastructure
  • Costs related to technical infrastructure
  • Costs related to people
  • Costs related to tooling

Initial benefit indicator categories have been identified and listed, however, both the categories and the indicators with their initial description are an early stage draft, subject to detailed review, revision and validation in the coming months of the project:

  • Benefits for healthcare: improved care processes, raised productivity and efficiency of healthcare;  improved health impact and patient outcomes
  • Organisational, administrative & management benefits: easier billing and reporting; simplified management
  • Benefits from better information and knowledge management: availability of high quality data; wide field coverage across multiple disciplines, support of registries; benefits for research
  • International dimension benefits: multiple languages/multi-lingual, multi-cultural aspects; enhanced collaboration on national or international level; impact on eHealth market 

D3.2 Cost-benefit analysis and impact assessment – interim report

This Deliverable D3.2 serves the purpose of presenting interim results, i.e. work in progress leading ultimately to the final deliverable D3.3. It updates D3.1 in a number of ways towards achieving WP3’s goals.

First results clearly indicate that much of the work behind a) describing and defining cost and benefits of implementing clinical terminologies as part of semantic interoperability strategies and b) the initial data collection methods and data generation has advanced our understanding of how to economically assess and plan for the adoption of clinical terminologies. From a scientific point of view, the literature review in Section 2.1 underlines the void in the literature, while most salient benefits mentioned in the focus group discussions and stakeholder questionnaire responses stress the need for quantification and real observed economic benefits (and costs).

In particular, the following progress has been achieved:

  • The potential benefits described in D3.1 have been revised, enriched through focus group discussions and feedback from the stakeholder questionnaire, and categorised to form a systematic set of benefit indicators
  • The cost indicators described in D3.1 have been further improved by adjusting their descriptions and adding the formulae used to calculate their impact.

In addition, further data towards the Assessment Framework for SNOMED CT and alternative terminologies have been continuously collected, analysed and interpreted by the work group of WP3.

In search for information to be fed into the Assessment Framework, WP3 has looked into a number of cases studies. The usable information in form of figures and estimates based on real cases will be used to quantify the socio-economic impact of SNOMED CT adoption.

D3.3 Cost-benefit analysis and impact assessment – Final Report

This deliverable addresses the need to provide economic evidence to serve as the basis for investigating the fitness of the international clinical terminology SNOMED CT as a potential standard for EU-wide eHealth deployments. The literature analysis showed that previous work focuses predominantly on technical issues of SNOMED CT implementation, and wherever costs and benefits are discussed, they are descriptive and not quantified. The IHTSDO through a business case study has taken significant steps towards filling the gap by
providing costs and benefits complemented by first quantification attempts and evidence collection. Our work builds on previous efforts in the area but goes beyond by offering a robust framework for
assessment of costs and potential benefits – with the aim to provide quantifiable figures to serve decision and policy makers in Europe.

D4.1 Portfolio of (best) practices

This deliverable consolidates the experience so far gained, in Europe and the US, in adopting SNOMED CT and other large scale clinical terminology systems. It serves as a guideline or check-list of issues to be considered and approaches to be defined when designing a coherent and holistic strategy for the advancement of semantic interoperability through large-scale terminology systems.

This adoption guideline is organised in sections that seek to help answer the following questions.

  • What have proved to be the drivers (the most desired business benefits) across the EU Member States for the adoption of a large-scale terminology as part of an overall strategy for better semantic interoperability?
  • What have been the key features of that adoption strategy, in terms of the initial roles for the terminology, the areas within health care services that it has been applied, and any national level governance measures that have been needed?
  • What are the key features of the operational strategy for adoption, including what concrete choices have had to be made about the terminology itself, other standardization and interoperability decisions, the specific operational steps needed to put the terminology system into use, within national infrastructures and health system products?
  • Where have significant costs been incurred or where are they planned for (since in many cases they have not yet been incurred, but are anticipated)? If not yet incurred, what do experts advise to be the important areas of investment that Member States should budget for?
  •  
  • What have been the success strategies attempted thus far, or what strategies are recommended by experts, to be used to help ensure the smooth and effective scale up of terminology implementation, use and acceptance?
  • What have proved, so far, to be the major areas of challenge, and issues, that need to be tackled when adopting the terminology system and complementary semantic interoperability measures, in particular with SNOMED CT?

D4.2 Policy Workshop Validation Report with Experts and Member States

This deliverable reports on the results from the policy workshop focusing on success criteria and operational strategies in SNOMED CT implementation. The findings of the workshop, validated by expert partners from member states and clinical terminology domain, underpin the quality and the acceptance of the ASSESS CT conclusions and recommendations.

D4.3 Policy and strategy recommendations – interim report

This deliverable makes recommendations to the European Commission and the eHealth Network on the strategic choices they should consider, at a European level and at Member State level, around the adoption of SNOMED CT, other terminology systems and other components of a coherent strategy to advance the level of semantic interoperability of health data across Europe.
This interim version of the deliverable reflects input from the consortium and consultation with international experts at meetings and via electronic means during spring 2016. The final version of this deliverable will be published in September 2016 following a further period of international consultation.

D4.4 - Policy and strategy recommendations – final report

This deliverable makes recommendations to the European Commission and the eHealth Network on the strategic choices they should consider, at a European level and at Member State level, around the adoption of SNOMED CT, other terminology systems and other components of a coherent strategy to advance the level of semantic interoperability of health data across Europe. This final version of the deliverable reflects input from the consortium and consultation with international experts at meetings and via electronic means until August 2016.