Open Access

Validation of a guideline-based decision support system for the diagnosis of primary headache disorders based on ICHD-3 beta

Contributed equally
The Journal of Headache and Pain201415:40

https://doi.org/10.1186/1129-2377-15-40

Received: 23 April 2014

Accepted: 10 June 2014

Published: 16 June 2014

Abstract

Background

China may have the largest population of headache sufferers and therefore the most serious burden of disease worldwide. However, the rate of diagnosis for headache disorders is extremely low, possibly due to the relative complexity of headache subtypes and diagnostic criteria. The use of computerized clinical decision support systems (CDSS) seems to be a better choice to solve this problem.

Methods

We developed a headache CDSS based on ICHD-3 beta and validated it in a prospective study that included 543 headache patients from the International Headache Center at the Chinese PLA General hospital, Beijing, China.

Results

We found that the CDSS correctly recognized 159/160 (99.4%) of migraine without aura, 36/36 (100%) of migraine with aura, 20/21 (95.2%) of chronic migraine, and 37/59 (62.7%) of probable migraine. This system also correctly identified 157/180 (87.2%) of patients with tension-type headache (TTH), of which infrequent episodic TTH was diagnosed in 12/13 (92.3%), frequent episodic TTH was diagnosed in 99/101 (98.0%), chronic TTH in 18/20 (90.0%), and probable TTH in 28/46 (60.9%). The correct diagnostic rates of cluster headache and new daily persistent headache (NDPH) were 90.0% and 100%, respectively. In addition, the system recognized 32/32 (100%) of patients with medication overuse headache.

Conclusions

With high diagnostic accuracy for most of the primary and some types of secondary headaches, this system can be expected to help general practitioners at primary hospitals improve diagnostic accuracy and thereby reduce the burden of headache in China.

Keywords

Headache Computer-assisted diagnosis Clinical decision support ICHD-3 beta SAGE

Background

Headache disorder is considered to be one of the most common reasons for medical consultation in both primary care units and neurological clinics [1, 2]. The estimated 1-year prevalence of primary headache in China was 23.8%, including migraine (9.3%) and tension-type headache (TTH) (10.8%). With a population of over 1.3 billion, it is estimated that China may have the largest headache population worldwide. Therefore, the total burden of primary headache disorders in China is very serious, accounting for CNY 672.7 billion (USD 96.9 billion) or 2.24% of GDP [3]. However, the correct rate of diagnosis for headache disorders is extremely low. A recent population-based door-to-door survey conducted in China reported rates of 13.8% for migraine and 2.6% for TTH [4]. A number of nonstandard headache diagnoses such as “vascular headache” and “nervous headache” are still widely applied in clinical practice throughout China [4]. These nonstandard diagnoses may result in inappropriate treatment measures such as analgesic drug abuse, unnecessary auxiliary examinations such as magnetic resonance imaging (MRI), and repeated consultations, which may aggravate the burden of headache disorders.

The International Classification of Headache Disorders, Second Edition (ICHD-II) [5] is available for the diagnosis of headache disorders worldwide. A revised 3rd edition (ICHD-3, beta version) was also published last year [6]. However, the relative complexity of headache subtypes and diagnostic criteria may confuse many general practitioners and even neurological physicians in primary and secondary care who are not familiar with the ICHD criteria.

The clinical decision support system (CDSS) is interactive decision support system (DSS) computer software, which is designed to assist clinicians and other health professionals with decision making tasks. CDSS has been effective in improving outcomes at some healthcare institutions and practice sites by making medical knowledge readily available to users [7]. A substantial body of evidence exists to suggest that DSS can be extremely effective for improving clinical practice, especially the guideline-based CDSS [811]. Thus, the use of CDSS, as an assistant decision-making tool which can help clinicians to solve complex medical problems, seems to be a good choice in the management of headache.

At present, there are few reports utilizing CDSS technology to diagnose headache disorders. Maizels et al. [12] has developed an on-line Computerized Headache Assessment Tool (CHAT). However, the tool was designed for patients not for clinicians. Mainardi et al. [13] also developed a computerized program to assist general practitioners in the diagnosis of the principal forms of primary headaches. However, it is based on the ICHD-II criteria, rather than the latest criteria. Based on ICHD-II criteria, Sarchielli et al. [14] used the Primary Headaches Analyzer 1.0 INT software, to diagnose primary chronic headaches but the headache types covered by the software are limited. To the best of our knowledge, although some methods or systems have been developed, none is based on ICHD-3 and reflects the latest research on headache disorders. In addition, the knowledgebase of most systems is rule-based and lacks a complete computerized clinical guideline model, as well as the integration of electronic medical records. Therefore, it is difficult for these systems to be widely used in the actual clinical setting.

Our previous study completed the preliminary CDSS and validated it retrospectively with the clinical information from outpatients with a confirmed diagnosis of headache [15]. Although the CDSS indicated superior diagnostic accuracy for some types of headaches such as migraine, TTH and cluster headache, the system did not include other subtypes of headaches such as migraine with aura, and new daily persistent headache (NDPH). In addition, it was not based on the new ICHD criteria. Therefore, the purpose of the current study was firstly to revise the CDSS on the basis of ICHD-3 beta and secondly, to prospectively apply this CDSS to the diagnosis of outpatients and evaluate its sensitivity and specificity in our headache clinic. Based on this, we aimed to develop a CDSS to help community doctors, general practitioners and inexperienced physicians to simplify clinical diagnostic procedures and increase diagnostic accuracy, in order to improve the level of diagnosis and lift the burden of headache disorders in China.

Methods

Ethics

The study protocol was approved by the ethics committee of the Chinese PLA General Hospital, Beijing, China.

Knowledgebase development

Building a knowledgebase is an essential step in the development of CDSS. A knowledgebase consists of a series of rules understood and executed by a computer. In our research, these rules were derived from a computerized clinical guideline model. Because a computerized clinical guideline model is totally different from text-based guidelines in format, it is difficult for clinical specialists to translate one format into another. Meanwhile, due to a lack of the necessary medical knowledge, this work is unsuitable to be completed by knowledge engineers even though they are familiar with computerized clinical guidelines. Therefore, the construction of a medical knowledgebase should rely on the joint efforts of clinical specialists and knowledge engineers. Based on this, we proposed a method of constructing a computerized clinical guideline model and medical knowledgebase, as shown in Figure 1.
Figure 1

The development process of headache knowledge.

First of all, headache specialists translated diagnostic knowledge in text-based guideline (ICHD) into a thought process of headache diagnosis. As the ICHD does not include the overall thought process or inference procedure of headache diagnosis, clinical specialists have to summarize it according to their own clinical experience, and then express it in the form of a flowchart. We selected a flowchart as the form of expression because it could be easily understood by both headache specialists and knowledge engineers.

Then, knowledge engineers established the computerized clinical guideline representation model according to these flow charts by referencing the modeling method of SAGE (standards-based sharable active guideline environment) [16]. First, we defined the domain ontology on headache diagnosis. Then, some necessary clinical data modules were set up in the Data Module class of SAGE. Thirdly, we summarized the logical comparison expression in headache diagnosis and defined them in the Expression class of SAGE. The activity graph of SAGE was utilized to express the inference procedure of headache diagnosis. To date, the computerized clinical guideline model has been developed.

Finally, the computerized clinical guideline representation module of headache diagnosis cannot be directly executed by computer although it has been represented. It must be translated into rules that can be executed in the inference engine of CDSS. A rule consists of two parts: LHS (left hand side) is the conditions of a rule and RHS (right hand side) is the reasoning results. Three elements, Decision, Action and Context, are the three indispensable parts of the computerized clinical guideline representation module. Therefore, we mapped the three elements to the two parts of the rule in order to build a knowledge base. The mapping method is shown in Table 1. Triggering events in Contexts and conditions in Decisions were mapped to LHS to choose scenarios and make decisions and Recommendations in action elements via the RHS while the rules were fired. We developed a computer program called the rule generator to complete this mapping work automatically.
Table 1

Map between guideline element and rule

Rule Element

LHS

RHS

Context

Triggering Event

 

Decision

Conditions

 

Action

 

Recommendations

LHS, left hand side; RHS, right hand side.

Validations

We applied the CDSS in a perspective study including 543 headache patients at the International Headache Center of the Chinese PLA General hospital from July 2013 to November 2013. Clinical information for headache disorders such as location, duration, attack frequency, severity, accompanying symptoms, and aura were also input into the CDSS by doctors who were unfamiliar with and blind to the diagnostic criteria of headaches. The CDSS then made a computerized diagnosis. Two different qualified and experienced specialists in headache neurology reviewed this information and made the final diagnosis, which was regarded as the gold standard. Then, for each patient, the specialist’s diagnosis and CDSS’s diagnosis were compared. For the patient who experienced more than one type of headache, we focused solely on the most troubling one.

Statistics

SPSS for Windows (Version 18.0) software was used for statistical analyses. Sensitivity, specificity and positive (PPV) and negative predictive values (NPV) were calculated for the CDSS diagnoses of headache disorders including migraine with (MO) or without aura (MA), probable migraine (PM), chronic migraine (CM), frequent episodic TTH (fETTH), infrequent episodic TTH (iETTH), probable TTH (PTTH), chronic TTH (CTTH), cluster headache (CH), new daily persistent headache (NDPH), medication overuse headache (MOH), and unclassified headaches against the gold standard. Cohen’s kappa (κ) was calculated for the agreement between diagnoses. Guidelines suggest that values of κ above 75% indicate excellent agreement, values between 75 and 40% indicate good to fair agreement and those below 40% show poor agreement [17]. A 5% level of significance and 95% confidence intervals (CI) were utilized.

Results

The CDSS correctly recognized 159/160 (99.4%) of patients with MO, 36/36 (100%) of patients with MA, 20/21 (95.2%) of patients with CM, and 37/59 (62.7%) of patients with PM (Table 2). This system also correctly identified 157/180 (87.2%) patients with TTH, of which iETTH was diagnosed in 12/13 (92.3%), frequent episodic TTH was diagnosed in 99/101 (98.0%), CTTH in 18/20 (90.0%), and probable TTH in 28/46 (60.9%). The correct recognition rate of cluster headache and NDPH were 90.0% and 100%, respectively. In addition, the system recognized 32/32 (100%) patients with MOH.
Table 2

Agreement between clinical decision support system and headache specialist diagnoses

 

Specialist

CDSS

Migraine

TTH

PM + PTTH

CH

NDPH

MOH

Others

Total

MO

MA

CM

PM

iETTH

fETTH

CTTH

PTTH

Migraine

              

MO

159

  

7

 

1

 

4

(4)

    

167

MA

 

36

           

36

CM

  

20

   

2

      

22

PM

   

37

1

1

   

2

  

2

43

TTH

              

iETTH

    

12

    

1

   

13

fETTH

     

99

      

2 (※)

101

CTTH

  

1

   

18

     

2

21

PTTH

       

28

    

10 (#)

38

CH

   

2

     

27

   

29

NDPH

          

14

  

14

MOH

           

32

 

32

Others

1

  

13

   

14

(13)

   

12 (*)

27

Total

160

36

21

59

13

101

20

46

 

30

14

32

28

543

MO: migraine without aura; MA: Migraine with aura; CM: chronic migraine; PM: probable migraine; TTH: tension-type headache; iETTH: infrequent episodic tension-type headache; fETTH: frequent episodic tension-type headache; CTTH: chronic tension-type headache; PTTH: probable tension-type headache; CH: cluster headache; PCH: probable cluster headache; NDPH: new daily-persistent headache; MOH: medication overuse headache ※: 1 patient satisfied 7.6 Headache attributed to epileptic seizure and 1 patient for 14. Other headache disorders; #: 6 patients satisfied 13. Painful cranial neuropathies and other facial pain, 2 patients for 14. Other headache disorders, 1 patient for 4.5 Cold-stimulus headache and 1 patient for 11.2.1 Cervicogenic headache; *: 4 patients satisfied 13. Painful cranial neuropathies and other facial pains, 1 patient for 4.3 Primary headache associated with sexual activity, and 5 patients for 14. Other headache disorders.

The sensitivity, specificity, PPV and NPV of the CDSS for headache disorders were calculated and are presented in Table 3. These results demonstrated that the CDSS was accurate and reliable in diagnosing MO (sensitivity 99.38%, specificity 97.91%, kappa = 0.9606), MA (sensitivity 1, specificity 1, kappa = 1), CM (sensitivity 95.24%, specificity 99.62%, kappa = 0.9274), iETTH (sensitivity 92.31%, specificity 99.81%, kappa = 0.9212), fETTH (sensitivity 98.02%, specificity 99.55%, kappa = 0.9757), CTTH (sensitivity 90%, specificity 99.43%, kappa = 0.8733) and CH (sensitivity 90%, specificity 99.61%, kappa = 0.91). However, sensitivity was relatively low for PM (62.71%) and PTTH (60.87%), despite a very high specificity (98.76% for PM and 97.99% for PTTH). The value of κ indicated fair agreement for PM (0.6978) and PTTH (0.639). With regard to MOH and NDPH, CDSS displayed perfect sensitivity and specificity (1 for both headaches).
Table 3

Statistical indices of clinical decision support system diagnostic performance

 

Migraine

TTH

CH

MOH

NDPH

 

MO

MA

CM

PM

iETTH

fETTH

CTTH

PTTH

   

Sensitivity (%)

99.38

100

95.24

62.71

92.31

98.02

90

60.87

90

100

100

(95% CI)

96.55- 99.89

 

77.33-99.15

49.95-73.92

66.69- 98.63

93.07-99.46

69.9-97.21

46.46-73.61

74.38-96.54

Specificity (%)

97.91

100

99.62

98.76

99.81

99.55

99.43

97.99

99.61

100

100

(95% CI)

95.93- 98.94

 

98.61-99.89

97.32-99.43

98.94-99.97

98.37- 99.88

98.33-99.8

96.34-98.9

98.59-99.89

Coincidence (%)

98.34

100

99.45

94.84

99.63

99.26

99.08

94.84

99.08

100

100

(95% CI)

96.88-99.13

 

98.39- 99.81

92.65-96.41

98.67-99.9

98.12-99.71

97.86-99.61

92.65-96.41

97.86-99.61

False negative rate

0.01

0

0.05

0.37

0.08

0.02

0.10

0.39

0.10

0

0

False positive rate

0.02

0

0.01

0.02

0.01

0.01

0.01

0.02

0.01

0

0

Youden index

0.97

1

0.94

0.61

0.92

0.98

0.89

0.59

0.90

1

1

PPV (%)

95.21

100

90.91

86.05

92.31

98.02

85.71

73.68

93.1

100

100

(95% CI)

90.83-97.55

 

72.18-97.47

72.74- 93.44

66.69-98.63

93.07-99.46

65.36-95.02

57.99-85.03

78.04-98.09

 

NPV (%)

99.73

100

99.81

95.6

99.81

99.55

99.62

96.44

99.42

100

100

(95% CI)

98.51-99.95

 

98.92- 99.97

93.43-97.08

98.94-99.97

98.37-99.88

98.61-99.89

94.44-97.73

98.3-99.8

κ

0.9606

1

0.9274

0.6978

0.9212

0.9757

0.8733

0.639

0.910

1

1

(95% CI)

0.8766-1.045

 

0.8433- 1.011

0.615 - 0.7807

0.8371 - 1.005

0.8916 - 1.06

0.789-0.957

0.555-0.723

0.826-0.995

MO: migraine without aura; MA: Migraine with aura; CM: chronic migraine; PM: probable migraine; TTH: tension-type headache; iETTH: infrequent episodic tension-type headache; fETTH: Frequent episodic tension-type headache; CTTH: chronic tension-type headache; PTTH: probable tension-type headache; CH: cluster headache; MOH: medication overuse headache; CI: confidence interval; PPV: positive predictive value; NPV: negative predictive value; κ: Cohen’s kappa.

Discussion

In our study, CDSS for headache disorders has demonstrated a high degree of accuracy in recognizing MO, MA, CM, iETTH, fETTH, CTTH and cluster headache, and a fair degree of accuracy in distinguishing PM and PTTH. It has also shown perfect recognition of NDPH and MOH. Compared with previous computerized diagnostic tools, the CDSS performed in the current study has several advantages. Firstly, it is based on ICHD-3 beta, the newly published diagnostic tool. For example, based on the CDSS, TTH-like characteristic features are no longer needed for the diagnosis of NDPH. Also, for the diagnosis of MOH, ICHD-3 beta criteria removes the item “headache developed or markedly worsened during medication overuse” making the diagnosis of MOH more relaxed. Secondly, we standardized the computerized guideline model by referencing the modeling method of SAGE, and comparing it with the other headache CDSS. Its data model is based on HL7 V3 RIM [18] and the medical terminology is based on SNOMET CT [19]. The application of these standards allows the headache CDSS to integrate into electronic medical systems more easily, which will ensure greater convenience for clinicians. Thirdly, the proposed approach for knowledgebase development for CDSS provides a feasible method of collaboration between clinical specialists and knowledge engineers. To some extent, it provides a new methodology to solve the bottleneck problems of knowledge acquisition in CDSS fields. Last but not least, with the development of modern medicine, the content of clinical guidelines needs to be continually updated. Therefore, the method of constructing a knowledgebase must be easily maintained and updated. Compared with traditional CDSS (clinical knowledge and inferencing or control knowledge are mixed in the same representation), the rules in our system are separate from the program of headache CDSS, and thus updating of the knowledge base is convenient for system development engineers, they need only modify the rules in the knowledgebase instead of revising and compiling the whole program.

Our previous clinic-based study indicated that of all the headache patients, 78.4% were diagnosed as primary headaches, including migraine, TTH, and cluster headache [20]. Compared with secondary headaches, the diagnostic assessment of primary headaches depends mostly on clinical features such as duration, location, and accompanying symptoms, and thus lacks biochemical and neuroradiological markers. Therefore, we mainly focused CDSS on the primary headache diagnosis. Although secondary headaches accounted for 12.9% of all headaches, MOH was the most common subtype (≈7.4%) [20]. Therefore, we also paid more attention to this type of headache. With regard to other secondary headaches, we performed “red flags”, such as “headache occurred in patients aged more than 50 years old”, “headache which is the most painful ever experienced”, “headache with positive neurologic signs”, “sudden onset headache”, to exclude factors such as tumor, stroke, and infection prior to the diagnosis of primary headaches.

In this study, the reliability of a computerized clinical guideline method was verified in the form of diagnostic accuracy of the headache decision support system. The evaluation of this system has been performed by a comparison of the diagnostic accuracy of the system versus the gold standard identified by headache specialists. For migraine subtypes, we found high sensitivity, specificity, PPV and NPV for MO, MA and CM, with κ = 0.9606, 1, and 0.9274, indicating excellent agreement. However, for PM, κ = 0.6978 indicated only fair to good agreement. With regard to TTH subtypes, excellent agreement was obvious for iETTH (κ = 0.9212), fETTH (κ = 0.9757) and CTTH (κ = 0.8733). However, κ = 0.639 revealed an unsatisfactory diagnostic accordance rate for PTTH. There may be two reasons for the discrepancy. Firstly, the clinical presentations of migraine and TTH can often overlap in individual patients. Specialists can diagnose these patients as PM and/or PTTH based on their clinical experience. However, these ill-defined boundaries for headache features may confuse CDSS and thus inconsistent output may arise. Secondly, some patients experience simultaneous migraine and TTH, doctors who applied CDSS may not pay enough attention to “the troubled headache” the patient was experiencing at presentation. Therefore, it was difficult for the computer to identify mixed headache profiles and which headache was the most troubling headache according to ICHD-II criteria, leading to inconsistent diagnostic results.

In our study, κ for cluster headache was 0.91. The most common reason for the difference between CDSS and the specialist may be due to overlapping of both cluster headache and migraine [2123]. Although the two disorders are distinct clinical entities, patients sometimes present with clinical symptoms having characteristics of both headache types. However, neither disorder fully complies with ICHD criteria. For example, in our series a patient has recurrent onset of migraine headache except for a headache duration of less than 4 hours. Also, the headache attacks in this patient have a seasonal periodicity but lack cranial autonomic symptoms (CAS) such as lacrimation or rhinorrhea. So, it appeared that probable migraine and probable cluster headache were both present. Another patient with cyclic and unmigrainous headache was diagnosed with TTH by CDSS based on ICHD criteria. On the other hand, CAS, which are distinguishing features of trigeminal autonomic cephalalgias, can also occur in patients with migraine. Although the CAS for migraine patients tended to be bilateral, strictly unilateral symptoms may confuse the CDSS, leading to misdiagnosis.

In addition to providing an auxiliary diagnosis, the CDSS also has an educational function by helping general practitioners familiar with ICHD criteria when clinical information for the diagnosis of headache was input and computerized diagnosis was output by the system. The system also has a diagnostic-conformation function: the general practitioners could press the “confirm” button when they agreed with the diagnostic recommendation of the system, if not, the physician could write his own diagnosis. Additionally, this system can provide treatment information based on international clinical guidelines. We have also completed a web-based CDSS for headache disorders and look forward to applying and verifying the system in a multicenter study. Briefly, our final aim is to implement this CDSS in primary care units and community hospitals, to improve the level of diagnosis and treatment for headache and finally reduce the burden of headache in China. Although the CDSS demonstrated satisfying diagnostic accuracy for headache disorders, it needs to be emphasized that this computer assisted diagnosis system does not take the place of the neurologist’s experience.

Conclusion

The study is the first step in developing a computer-based diagnostic tool. The first testing phase in the headache clinic delivered promising results, which revealed a high degree of accuracy in recognizing MO, MA, CM, iETTH, fETTH, CTTH, cluster headache, NDPH, and MOH, and a fair degree of accuracy in distinguishing PM and PTTH. In the next step, the tool should be tested to determine its validity in primary care. The final aim of the study is to disseminate this CDSS and improve the level of diagnosis and treatment for headache in primary care units and community hospitals and thereby reduce the burden of headache in China.

Consent

Written informed consent was obtained from the patient for the publication of this report and any accompanying images.

Notes

Declarations

Acknowledgments

This work was supported by the Clinical and Scientific Supporting Fund of Chinese PLA General Hospital (No. 2012FC-TSYS-3041), Capital Development Scientific Research (No. 2001-5001-04) and National Scientific Research Fund (No. 2011ZX09307-001-07).

Authors’ Affiliations

(1)
International Headache Center, Department of Neurology, Chinese PLA General Hospital
(2)
College of Biomedical Engineering and Instrument Science, Zhejiang University

References

  1. Villaume De Diego E, Lanteri-Minet M: Recognition and management of migraine in primary care: influence of functional impact measured by the headache impact test (HIT). Cephalalgia 2005, 25: 184–190.View ArticleGoogle Scholar
  2. Wang Y, Zhou J, Fan X, Li X, Ran L, Tan G, Chen L, Wang K, Liu B: Classification and clinical features of headache patients: an outpatient clinic study from China. J Headache Pain 2011, 12: 561–567.PubMed CentralView ArticlePubMedGoogle Scholar
  3. Yu S, Liu R, Zhao G, Yang X, Qiao X, Feng J, Fang Y, Cao X, He M, Steiner TJ: The prevalence and burden of primary headaches in China: a population-based door-to-door survey. Headache 2012, 52: 582–591.View ArticlePubMedGoogle Scholar
  4. Liu R, Yu S, He M, Zhao G, Yang X, Qiao X, Feng J, Fang Y, Cao X, Steiner TJ: Health-care utilization for primary headache disorders in China: a population-based door-to-door survey. J Headache Pain 2013, 14: 47.PubMed CentralView ArticlePubMedGoogle Scholar
  5. Headache Classification Subcommittee of the International Headache Society: The international classification of headache disorders: 2nd edition. Cephalalgia 2004, 24(suppl 1):9–160.Google Scholar
  6. Headache Classification Committee of the International Headache Society (IHS): The International Classification of Headache Disorders, 3rd edition (beta version). Cephalalgia 2013, 33: 629–808.View ArticleGoogle Scholar
  7. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE: A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007, 14: 141–145.PubMed CentralView ArticlePubMedGoogle Scholar
  8. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR: Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999, 282: 1458–1465.View ArticlePubMedGoogle Scholar
  9. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005, 293: 1223–1238.View ArticlePubMedGoogle Scholar
  10. Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993, 342: 1317–1322.View ArticlePubMedGoogle Scholar
  11. Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005, 330: 765.PubMed CentralView ArticlePubMedGoogle Scholar
  12. Maizels M, Wolfe WJ: An expert system for headache diagnosis: the Computerized Headache Assessment tool (CHAT). Headache 2008, 48: 72–78.View ArticlePubMedGoogle Scholar
  13. Mainardi F, Maggioni F, Dainese F, Zanchin G: Development of a ICHD-II based computerised system for the general practitioner. J Headache Pain 2005, 6: 211–212.PubMed CentralView ArticlePubMedGoogle Scholar
  14. Sarchielli P, Pedini M, Coppola F, Rossi C, Baldi A, Corbelli I, Mancini ML, Calabresi P: Application of the ICHD-II criteria to the diagnosis of primary chronic headaches via a computerized structured record. Headache 2007, 47: 38–44.View ArticlePubMedGoogle Scholar
  15. Yin Z, Dong Z, Yu S, Lu X, Feng G, Duan H: A guideline-based decision support system for headache diagnosis. Stud Health Technol Inform 2013, 192: 1022.PubMedGoogle Scholar
  16. Tu SW, Campbell JR, Glasgow J, Nyman MA, McClure R, McClay J, Parker C, Hrabak KM, Berg D, Weida T, Mansfield JG, Musen MA, Abarbanel RM: The SAGE Guideline Model: achievements and overview. J Am Med Inform Assoc 2007, 14: 589–598.PubMed CentralView ArticlePubMedGoogle Scholar
  17. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977, 33: 159–174.View ArticlePubMedGoogle Scholar
  18. HL7 Reference Information Model, Health Level Seven, Inc , last access on 2014–2-24 http://www.hl7.org/implement/standards/rim.cfm
  19. Stearns MQ, Price C, Spackman KA, Wang AY: SNOMED clinical terms: overview of the development process and project status. Proc AMIA Symp 2001, 662–666.Google Scholar
  20. Dong Z, Di H, Dai W, Liang J, Pan M, Zhang M, Zhou Z, Li Z, Liu R, Yu S: Application of ICHD-II criteria in a headache clinic of China. PLoS One 2012, 7: e50898.PubMed CentralView ArticlePubMedGoogle Scholar
  21. Applebee AM, Shapiro RE: Cluster-migraine: does it exist? Curr Pain Headache Rep 2007, 11: 154–157.View ArticlePubMedGoogle Scholar
  22. Gelfand AA, Reider AC, Goadsby PJ: Cranial autonomic symptoms in pediatric migraine are the rule, not the exception. Neurology 2013, 81: 431–436.PubMed CentralView ArticlePubMedGoogle Scholar
  23. Lai TH, Fuh JL, Wang SJ: Cranial autonomic symptoms in migraine: characteristics and comparison with cluster headache. J Neurol Neurosurg Psychiatry 2009, 80: 1116–1119.View ArticlePubMedGoogle Scholar

Copyright

© Dong et al.; licensee Springer. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.