The validity of questionnaire-based diagnoses: the third Nord-Trøndelag Health Study 2006–2008
© Springer-Verlag 2009
Received: 4 September 2009
Accepted: 12 November 2009
Published: 28 November 2009
The Nord-Trøndelag Health Study (HUNT 3) performed in 2006–2008 is a replication of the cross-sectional survey from 1995 to 1997 (HUNT 2). The aim of the present study was to assess the sensitivity and specificity of questionnaire-based headache diagnoses using a personal interview by a neurologist as a gold standard. For the questionnaire-based status as headache sufferer, a sensitivity of 88%, a specificity of 86%, and a kappa statistic of 0.70 were found. Chronic headache, chronic tension-type headache (TTH), and medication overuse headache (MOH) were diagnosed with a specificity of ≥99%, and a kappa statistic of ≥0.73. Lower figures were found for the diagnoses of migraine and TTH. For individuals with headache ≥1 day per month, a sensitivity of 58% (migraine) and 96% (TTH), a specificity of 91 and 69%, and a kappa statistic of 0.54 and 0.44 were found, respectively. The specificity for migraine with aura was 95%. In conclusion, the HUNT 3-questionnaire is a valid tool for identifying headache sufferers, and diagnosing patients with chronic headache, including chronic TTH and MOH. The more moderate sensitivity for migraine and TTH makes the questionnaire-based diagnoses of migraine and TTH suboptimal for determining the prevalence. However, the high specificity of the questionnaire-based diagnosis of migraine, in particular for migraine with aura, makes the questionnaire a valid tool for diagnosing patients with migraine for genetic studies.
A careful history taken by a headache specialist is the gold standard for making a valid headache diagnosis. Nevertheless, in large-scale population-based studies, a less time-consuming and costly strategy has to be chosen. A self-administrated questionnaire for the diagnosis of the most common headache disorders ensures standardization , but validation of the diagnostic instrument is required in epidemiological surveys to assess the precision of questionnaire diagnoses [1–3].
The headache part of the third Nord-Trøndelag Health Study (HUNT 3) performed in 2006–2008 is a replication of the cross-sectional survey from 1995 to 1997 (HUNT 2) . In order to validate questionnaire-based headache diagnoses and other information, a clinical interview performed by neurologist was done in a random sample of participants in HUNT 3. The aim of the present study was to assess the sensitivity and specificity of questionnaire-based headache diagnoses using a personal interview by a neurologist as a gold standard.
Materials and methods
The third Nord-Trøndelag Health Survey (HUNT 3)
Headache questions in the second questionnaire (Q2)
17. (a) Have you suffered from headache during the last 12 months?
(b) If yes; what type of headache?
(a) Yes/No (no: go to question 24)
(b) Migraine/other headache
18. State the average number of headache days per month
<1 day/1–6 days/7–14 days/>14 days
19. Usually, what is the pain intensity?
Mild (does not inhibit daily activities)/moderate (inhibiting, but not preventing daily activities)/severe (daily activities suspended)
20. For how long does the headache attack usually last?
<4 h/4 h–1 day/1–3 days/>3 days
21. Is the headache usually accompanied or dominated by: (a) Pulsating pain? (b) Pressing pain? (c) One-sided pain (right or left)? (d) Getting worse by physical activity? (e) Nausea and/or vomiting? (f) Increased sensitivity to light and sound?
22. Prior to or during headache; could you temporary have: (a) Visual disturbance? (flickering lights, spots or lines, loss of vision) (b) Sensory symptoms in one hands or half of the face
23. State the number of days in the past 3 months you missed work or school because of headache?
Study population of the validation study
The method of the present validation study has been described elsewhere . The Nord-Trøndelag County was divided in 25 study areas, and the validation study was performed in two of these (Verdal and Stjørdal). The main survey in Verdal was performed from September to November 2007, and in Stjørdal from December 2007 to April 2008. The participation rates in the HUNT 3 study were 52% of the whole population in Verdal and 50% in Stjørdal. In the present study, a random sample of individuals who had participated in HUNT 3 in the two communities were invited to a face-to-face interview performed by neurologist focusing on four different topics covered in the questionnaire, namely alcohol, sleep, headache and musculoskeletal complaints . The participants’ responses to the questionnaire in HUNT 3 were unknown to the interviewers, and the main objective of the study was to evaluate the validity of questionnaire-based information.
Invitation letters were sent to a random sample on the basis of a list of participants in Verdal and Stjørdal. In HUNT 2, the participation rate was strongly age dependent, with the highest participation in the age group 60–69, and lowest in the age group 20–29 . To ensure acceptably balanced participation in the present study for both genders in all age groups, potential participants were selected from the list of HUNT 3 participants consecutively in the following order: man ≤50 years, man >50 years, woman ≤50 years, and woman >50 years.
Among participants in HUNT 3 living in Verdal and Stjørdal, a random sample of 563 persons got an invitation letter which included general information about the validation study, also informing that they would be contacted on telephone by our research assistants to give further information and to make an appointment for the personal interview. The list of persons who participated in Verdal and Stjørdal were received shortly before the time of the interview. Hence, due to lack of time, in case the research assistants were unable to get in contact despite of a minimum of two attempts they were instructed to call the next person on the list.
For the validation study, a semi-structured face-to-face interview was performed by a total of eight neurologists with special training in headache. When indicated, a clinical examination was performed. Based on the information from the clinical interview, headache disorders were classified in accordance with the ICHD-II criteria , except for medication overuse headache (MOH) for which the revised version was used . Up to three different headache types were diagnosed in each individual. Among the 220 individuals who reported headache during the last year , 117 subjects got two headache diagnoses, whereas 16 got three.
Different sets of criteria for the diagnosis of migraine based on information in the questionnaire
I. Restrictive migraine criteria (definite migraine)
B. Headache attacks lasting ≤72 ha
C. Headache had usually at least two of the following three characteristics:
1. Pulsating quality
2. Unilateral location
3. Moderate or severe pain intensity
4. Aggravation by physical activity
D. During headache, at least one of the following:
1. Nausea and/or vomiting
2. Increased sensitivity to light and sound
II. Migraine with aura
Restrictive migraine criteria, and visualb disturbance prior to or during headache
III. Liberal migraine criteria (definite and probable migraine)
Self-reported migraine, or fulfilled the restrictive criteria
IV. Liberal HUNT 2 migraine criteria (definite and probable migraine)
Self-reported migraine, or fulfilled the criterion B, D, and modified criterion C (headache had usually at least one of the following three characteristics: pulsating quality, unilateral location, or aggravation by physical activity)
The questionnaire-based diagnosis of TTH was based on the ICHD-II criteria . We have previously reported that very few subjects with infrequent TTH consider themselves as headache sufferers , and that a high positive predictive value and high specificity for identifying individuals with TTH ≥1 day/month were found among headache sufferers . As a consequence of these findings, only the validity of TTH >1 day/month was evaluated.
Chronic headache was defined as headache more than 14 days per month. To fulfil the questionnaire-based diagnosis of MOH the participants had to report headache >14 days per month and use of analgesics four times per week or more during the last month.
The present study was an integrated part of the HUNT 3 project which was approved by the Regional Committee for Ethics in Medical Research and the Norwegian Data Inspectorate.
Demographic data were compared between participants and non-participants with independent samples t test for continuous variables and with the chi-squared test for categorical variables. Two-tailed estimations of significance were used, and the level of significance was set at P < 0.05.
Sensitivity, specificity and Cohen’s kappa statistics with 95% CI were calculated for different headache diagnoses based on information in Q2 using neurologists’ headache diagnoses as a gold standard.
Participation rate in the clinical interview
Among the 392 persons who answered on telephone, only 29 stated that they did not want to participate, whereas 66 wanted to participate, but were unable to come because they were out of town, had sick children, were busy in job, or they had forgotten the invitation.
Compared to the 266 non-participants, the 297 participants were older (mean age 52.3 vs. 48.6 years, P = 0.004) and slightly more likely to be men (51 vs. 47%, P = 0.37). No significant difference was found between participants and non-participants with regard to bodily pain lasting ≥6 months (40 vs. 36%, P = 0.30), self-reported good or very good health (75 vs. 79%, P = 0.25), or proportion of working individuals (76 vs. 82%, P = 0.12).
Response rate to the headache questionnaire
The first headache question in Q2 was answered by 543 out of 563 (96%) invited persons (256 non-participants and 287 participants). 108 (37.6%) of the participants suffered from headache. Among these, 105 (97%) answered self-reported type of headache, and 82 (76%) answered all headache questions. When considering agreement between questionnaire-based diagnoses and those made in the clinical interviews, individuals with incomplete information in the questionnaire were excluded from the analyses. These were: three with chronic headache (not reported headache frequency in Q2), two with MOH (not reported use of analgesics in Q2), 14 with migraine, and 25 with TTH.
Prevalence of headache among participants versus non-participants
The prevalence of headache sufferers did not differ significantly between non-participants and participants (overall 39.5 vs. 37.5%, P = 0.72; men 31.4 vs. 32.9%, P = 0.90; women 46.7 vs. 42.6%, P = 0.55). Similarly, no significant difference was found between participants and non-participants regarding prevalence of migraine (12.5 vs. 16.2%, P = 0.20), TTH ≥1 day per month (14.8 vs. 11.3%, P = 0.22), or chronic headache (3.4 vs. 1.9%, P = 0.28).
Reliability of the Q2 diagnoses
The mean interval between answering the Q2 and the validation interview was 50 days (95% CI 48–52 days; median 45 days range, 9–90 days).
Sensitivity, specificity and kappa value of questionnaire-based headache diagnosis
TTH ≥ 1 day/month
Chronic TTH without medication-overuse
Migraine ≥1 day/month (restrictive)
Migraine with visual aura
Migraine ≥1 day/month (liberal)
Migraine (liberal Hunt2)
Overall, the questionnaire-based diagnoses of migraine (MA or MO or both) had a sensitivity of 51%, a specificity of 95%, and a kappa statistics of 0.50. Considering those with migraine ≥1 day per month the figures changed to 58%, 91%, and 0.54, respectively. Correspondingly, the sensitivity, specificity, and kappa statistics of TTH ≥1 day per month were 96%, 69%, and 0.44.
The agreement between the neurologists’ interview and the questionnaire-based diagnoses for headache suffering and chronic headache, including CTTH and MOH, was very good, whereas the agreement for migraine and TTH was more moderate.
Less than half of the invited adults (42%) responded to the headache questions in HUNT 3. It should be emphasised that our results are only valid for these responders. Furthermore, in the present study only 53% of the random sample of invited participants was interviewed. However, selective participation due to headache status seems less likely, because the prevalence of headache, migraine and TTH included were quite similar among participants and non-participants. This finding is of particular interest, because such a comparison between participants and non-participants can rarely be done.
The reported agreement between the neurologists’ interview and the questionnaire-based diagnoses may have been influenced by the number of excluded persons from the analyses because of incomplete information in the questionnaire. However, none of the participants were excluded for the analysis of headache suffering, only three for chronic headache, whereas the number of excluded persons was higher for migraine and TTH (14 and 25, respectively).
The agreement for MOH was good, although it was diagnosed differently in the interview and questionnaire. In the interview, the revised version from 2006 was used , accepting an intake of drugs like tripans, opioids, and combination medications ≥10 days per month on a regular basis of 3 months. In contrast, for the questionnaire-based diagnosis use of analgesics four times per week or more during the last month was accepted.
The fact that there was a mean time interval of 50 days between the questionnaire and the interview may imply that the frequency of headache attacks and the headache characteristics could have changed during this period, which may have reduced the agreement between responses in the questionnaire and clinical interview.
A main problem of studies based on self-administrated questionnaires is to correctly diagnose patients with the co-existence of two or more headaches, usually migraine and tension-type headache. Questions using the term “usually” regarding features of headache may not be ideal if one tries to make the respondent differentiate between different subtypes of headache.
Present and previous studies
The agreement between the questionnaire and clinical interview with respect to being a headache sufferer or not was better in the present study than in the validation study performed in HUNT 2 (kappa value 0.70 vs. 0.57) . Similarly, the agreement was better for chronic headache (0.75 vs. 0.44). A probable reason for the better results this time is that the time span between the Q2 to the validation study was shorter (mean of 50 days) than in the HUNT 2 study (5–9 months) .
As in most previous population-based studies comparing questionnaire-based and interview-based diagnosis [8–13], a moderate agreement rate was found for migraine. A better agreement has been found in validation studies focusing on migraine patients recruited from specialist practice , or on individuals with self-reported migraine . Our moderate agreement rate could in part be explained by the mean time span of 50 days between the questionnaire and interview, because it has previously been shown that key diagnostic features of migraine are not consistently reported or experienced over time. Another potential problem of the questionnaire was to identify patients with co-existence of migraine and tension-type headache, because some respondents with migraine may not keep the different subtypes clear when answering the questions. It may also be that many patients have somewhat atypical migraine (probable migraine), and in these patients the distinction between migraine and TTH will be difficult.
As only a moderate sensitivity was found for the questionnaire-based diagnosis of migraine and TTH, the questionnaire is not optimal for estimating prevalence of these headache types. In particular, the prevalence of migraine may be overestimated when diagnosing migraine by the questionnaire, because headache duration less than 4 h was accepted. However, the high specificity (>90) for the questionnaire-based diagnosis of migraine makes the questionnaire a valid tool to identify a population of individuals with migraine suitable for genetic studies.
The fact that the precision of the headache diagnoses in HUNT 3 seems to be as good as or even better than in HUNT 2 will enable comparisons of the results between the two studies, and will offer an exiting opportunity to explore possible causes and risk factors for headaches in a prospective study of large population.
The HUNT 3-questionnaire is a valid tool for diagnosing patients as headache sufferers, and with chronic headache, including chronic TTH and MOH. The more moderate agreement for migraine and TTH makes the questionnaire-based diagnoses of migraine and TTH suboptimal for determining the prevalence in the population. However, the high specificity of the questionnaire-based diagnosis of migraine, in particular for migraine with aura, makes the questionnaire a valid tool for diagnosing patients with migraine for genetic studies.
The Nord-Trøndelag Health Study (The HUNT study) is a collaboration between The HUNT Research Centre, Faculty of Medicine, The Norwegian University of Science and Technology (NTNU); and the Nord-Trøndelag County Council.
Conflict of interest
- Kirchmann M, Seven E, Björnsson Á, Björnssondóttir G, Gulcher JR, Stefánsson K, Olesen J (2006) Validation of the deCODE migraine questionnaire (DMQ3) for use in genetic studies. Eur J Neurol 13:1239–1244, 10.1111/j.1468-1331.2006.01491.x, 1:STN:280:DC%2BD28ngs1SltQ%3D%3D, 17038039View ArticlePubMedGoogle Scholar
- Russell MB, Aaseth K, Grande RB, Gulbrandsen P, Lundqvist C (2007) Which strategy should be applied? Design of a Norwegian epidemiological survey on chronic headache. Acta Neurol Scand 115(suppl. 187):59–63, 10.1111/j.1600-0404.2007.00849.xView ArticleGoogle Scholar
- Hagen K, Stovner LJ, Zwart JA (2007) Potentials and pitfalls in analytical headache epidemiological studies. Lessons to be learned from the Head-HUNT Study. Cephalalgia 27:403–413, 10.1111/j.1468-2982.2007.01302.x, 1:STN:280:DC%2BD2s3ktVKnug%3D%3D, 17448178View ArticlePubMedGoogle Scholar
- Hagen K, Zwart JA, Vatten L, Stovner LJ, Bovim G (2002) Prevalence of migraine and non-migrainous headache—Head-HUNT, a large population-based study. Cephalalgia 20:900–906, 10.1046/j.1468-2982.2000.00145.xView ArticleGoogle Scholar
- Headache Classification Subcommittee of the International Headache Society (2004) The international classification of headache disorders, 2nd edn. Cephalalgia 24(Suppl 1):1–160Google Scholar
- Hagen K, Zwart JA, Aamodt AH, Nilsen KB, Bråthen G, Helde G, Stjern M, Tronvik EA, Stovner LJ (2008) A face-to-face interview of participants in HUNT 3: The impact of the screening question on headache prevalence. J Headache Pain 9:289–294, 10.1007/s10194-008-0062-6, 18690490PubMed CentralView ArticlePubMedGoogle Scholar
- Headache Classification Committee: Olesen J, Bousser MG, Diener HC, Dodick D, First M, Goadsby PJ, Göbel H, Lainez MJA, Lance JW, Lipton RB, Nappi G, Sakai F, Schoenen J, Silbertstein SD, Steiner TJ (2006) New appendix criteria open for a broader concept of chronic migraine. Cephalalgia 26:742–746Google Scholar
- Hagen K, Zwart JA, Vatten L, Stovner LJ, Bovim G (2000) Head-HUNT: validity and reliability of a headache questionnaire in a large population-based study in Norway. Cephalalgia 20:244–251, 10.1046/j.1468-2982.2000.00049.x, 1:STN:280:DC%2BD3cvkslWjsA%3D%3D, 10999674View ArticlePubMedGoogle Scholar
- Rasmussen BK, Jensen R, Olesen J (1991) Questionnaire versus clinical interview in the diagnosis of headache. Headache 31:290–295, 10.1111/j.1526-4610.1991.hed3105290.x, 1:STN:280:DyaK3MzhslKmsQ%3D%3D, 1860786View ArticlePubMedGoogle Scholar
- Pereira-Monteira J, Maio R, Calheiros JM (1994) Headache diagnosis. Comparison of questionnaire with clinical interview. In: Olesen J (ed) Headache classification and epidemiology. Raven Press, New York, pp 217–220Google Scholar
- Laínez M, Vioque J, Hernández-Aguado I, Titus F (1994) Prevalence of migraine in Spain. An assessment of the questionnaire’s validity by clinical interview. In: Olesen J (ed) Headache classification and epidemiology. Raven Press, New York, pp 221–225Google Scholar
- Gervil M, Ulrich V, Olesen J, Russell MB (1998) Screening for migraine in the general population: validation of a simple questionnaire. Cephalalgia 18:342–348, 10.1046/j.1468-2982.1998.1806342.x, 1:STN:280:DyaK1cvgtVKrug%3D%3D, 9731939View ArticlePubMedGoogle Scholar
- Hayran O, Siva A, Zarifoglu M (1999) Validation of a headache questionnaire based on HIS criteria in the Turkish headache prevalence study. Cephalalgia 19:417Google Scholar
- Russell MB, Rasmussen BK, Thorvaldsen P, Olesen J (1995) Prevalence and sex-ratio of the subtypes of migraine. Int J Epidemiol 24:612–618, 10.1093/ije/24.3.612, 1:STN:280:DyaK2MvgtlSlug%3D%3D, 7672904View ArticlePubMedGoogle Scholar