• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Validation and Refinement of the Questionnaire for Lung Transplant Patients

Posted on: Thursday, 23 December 2004, 03:01 CST

Context-The Questionnaire for Lung Transplant Patients was designed to assess symptoms and activity tolerance in lung transplant recipients during their post-transplant evaluations. The initial psychometric evaluation determined that the questionnaire was clinically useful, reliable, and valid.

Objective-To report the results of further psychometric analyses in a new, expanded sample of lung transplant recipients and to demonstrate the iterative manner by which instruments are refined and tested.

Methods-Internal consistency, test-retest stability, convergent validity, factorial validity, and group differences attributable to age, gender, and transplant type were determined in a pooled sample of 177 lung transplant recipients. Sensitivity to change over time was measured in a subsample (n=51) who provided repeated measures data.

Results-The Questionnaire for Lung Transplant Patients and its subscales were internally consistent (Kuder Richardson reliability of 0.73-0.95). Test-retest stability was high (intraclass correlations >0.70). Symptoms showed a significant curvilinear pattern with a tendency to decrease over time before rising again at the 12-month measurement for the total questionnaire (F=6.8, P=.012) and 2 subscales-Respiratory (F=5.6, P=.022) and Activities of Daily Living (F=19.7, P<.001). Convergent construct validity correlations ranged from 0.29 to 0.53 and were consistent with theoretical expectations. Factorial analysis confirmed 3 domains that coincided with the Respiratory, General, and Activities of Daily Living subscales.

Conclusions-The Questionnaire for Lung Transplant Patients is a reliable and valid measure for assessing physical symptoms and activity intolerance after lung transplantation in individual recipients, recipients in aggregate, and comparison groups, on one occasion and serially over time. These results will guide future refinement and testing of the Questionnaire for Lung Transplant Patients. (Progress in Transplantation. 2004;14:338-345)

The ability to assess symptoms and activity intolerance after lung transplantation is crucial. Relief of symptoms and improvement in activity tolerance are frequently used as outcome measures for determining success after lung transplantation. In addition, symptoms and activity intolerance are harbingers of acute complications after transplantation. However, a recent study found that symptoms of allograft rejection and pulmonary infection, the most frequent acute complications, frequently overlap.1 Therefore, it is important to be able to detect changes in an individual's symptoms and activity intolerance over time. Existing surveys for assessing symptoms after solid-organ transplantation2,3 are limited in their ability to assess the range of symptoms experienced by lung transplant recipients (LTRs), particularly symptoms related to pulmonary impairments and activity intolerance. Thus, research continues about the optimal measure to assess symptoms and activity intolerance in LTRs.

The Questionnaire for Lung Transplant Patients (QLTP) is used by LTRs during their routine post-transplant evaluations to report symptoms and rate their shortness of breath, cough, and activity tolerance. The responses are reviewed by clinicians to assess the stability or deterioration of the patient's condition and to guide practice. In a previous article,4 we explained why instruments developed for clinical use, such as the QLTP, need to be evaluated for reliability and validity if we are to draw correct inferences and conclusions about the phenomena being measured. In the published report, we described the established methods for conducting a psychometric evaluation, and provided evidence to support the QLTP as a reliable and valid measure of symptoms and activity tolerance after lung transplantation. Although the sample size was deemed adequate for initial evaluation of the QLTP, the relatively small sample did not permit advanced assessments of the construct validity of the QLTP, such as factor analysis, nor did it permit an item analysis that would allow us to refine the instrument without losing its psychometric strengths of quality and adequacy. Therefore, the purposes of this article are to report the results of the recent analyses and to demonstrate the iterative manner by which instruments are refined and tested.

Instrument Development

The original version of the QLTP was developed by the pulmonary transplant team at the University of Pittsburgh Medical Center. The self-administered questionnaire permits recipients to check any symptoms that they have had since their previous evaluation, to report difficulties performing activities of daily living, and to rate their shortness of breath and cough. Items of the QLTP were clustered into the following 3 subscales: respiratory symptoms (15 items), general symptoms (28 items), and problems with activities of daily living (ADL; 5 items). The symptom items were dichotomous and each checked symptom was given a value of 1 point. The number of symptoms checked was summed to obtain the 3 subscale scores and a total score. The ranges of possible scores for the respiratory, general, and ADL subscales were 0 to 15, 0 to 28, and 0 to 5, respectively, with a maximum total score of 48. Two additional items asked patients to rate their levels of shortness of breath and cough on scales from 0 to 10.

As with many instruments designed for use in patient care settings, the QLTP was constructed by clinical experts to measure the phenomena of interest, and was initially thought to require only the properties of sensibility, ease, and usefulness. Consequently, procedures to assess whether the QLTP was valid and reliable were not initially performed. However, the fact that clinicians found the instrument useful did not ensure that the QLTP was reliable and valid. Even clinical instruments must be systematically evaluated for their ability to provide scientifically sound information for practice and research. Therefore, the initial psychometric evaluation of the QLTP,4 conducted in a small sample (N = 37) revealed that the QLTP and its subscales were internally consistent (Cronbach α ranged from 0.76-0.96), and reliable (r=0.70 using test-retest methods). Significant correlations were found between the QLTP and all criterion measures (r = 0.50-0.93) and significant correlations in the predicted negative direction were found between the respiratory subscale scores (r = -0.51) and activity tolerance, and between both cough (r = -0.51) and shortness of breath ratings (- 0.68) and activity tolerance.

Several issues prompted further testing, that is, cross- validation of the QLTP in an independent and expanded sample. First, as additional LTRs completed the QLTP, an expanded pool of questionnaires was available to perform more advanced procedures, such as factor analysis and item analyses that require larger sample sizes. Secondly, the larger pool permitted further analysis of convergent validity and an assessment of group differences in symptoms. Lastly, a portion of the expanded pool completed the QLTP over 4 time points, providing data to evaluate the test-retest stability and sensitivity of the QLTP to changes in symptoms and activity tolerance in LTRs over time.

Methods

Sampling

A longitudinal survey design was used to perform the analysis in a pooled sample of 177 independent LTRs who competed the QLTP during routine post-transplant evaluations. Group differences in QLTP scores attributable to gender and type of transplant procedure (single or double), and the relationship between age and the QLTP scores were determined. One third of the sample (n = 51) also provided repeated measures data for up to 12 months after transplantation.

Reliability

Internal Consistency. The internal consistency of the QLTP was determined using the Kuder Richardson formula (KR-20) for the total QLTP and for each subscale. The KR-20 is a special case of the coefficient alpha to determine the consistency of responses to items that are dichotomous.5 The more homogeneous the scale, the higher the internal consistency.6

Test-Retest Stability. To determine whether the QLTP scores rank- ordered patients in a similar way over time, we examined the temporal stability of the QLTP and its 3 subscales using test- retest over a 1-year period in the subset of 51 LTRs. Four repeated measures of the QLTP were performed at 3, 6, 9, and 12 months after transplantation to compute the intraclass correlations (ICCs). The ICC is a special case correlation that measures consistency or agreement of values within cases. The ICC considers 2 or more repeated ratings simultaneously. The higher the ICC, the more stable the QLTP across different occasions.6

Validity

Convergent Construct Validity. Convergent construct validity is concerned with the extent to which a particular measure relates to other measures in a hypothesized direction.7 Convergent construct validity of the QLTP was estimated by correlating total and subscale scores with measures that were expected to be significantly correlated with symptoms-percent decline in forced expiratory volumes in 1 second (FEV^sub 1^) and cough and shortness of breath ratings; and another variable that was expected to be inversely correlated with symptoms-number offlights of stairs that the LTRs could climb.

Factorial Validity. Typically, a concern when evaluating the validity of an instrument is whether the scale measures the unidimensional or multidimensional construct it is intended to measure. Factor analysis refers to the statistical procedures that are used to determine empirically how many dimensions (factors) underlie the items in a scale.8 This is accomplished by identifying groups of items that covary with one another and appear to represent meaningful underlying factors. The criteria to retain factors were an eigenvalue greater than 1 and scree plots that demonstrated a clear "break" in the plot of eigenvalues. The "factor loading" of an item on a factor represents the extent to which the item relates to the hypothetical factor. Items with factor loading values of ≥ 0.4 were considered significantly related to that factor. By inspecting the content of the items with values ≥ 0.4 listed under each factor, it is possible to gain some inkling about the nature of the underlying factor. Each factor is differentiated from the other because each is represented by a different set of items; therefore, each factor can be labeled on the basis of the commonality between the loaded items. In the present case, because the QLTP was developed to be multidimensional with 3 subscales or dimensions, a 3-factor structure was examined using the principal component extraction with varimax rotation method. Before conducting factor analysis, 2 items of the respiratory subscale were removed when it was discovered that they duplicated other items, and 1 item of the general symptoms subscale "jaundice" was eliminated because of lack of discriminating power as no one in the sample reported it.

Sensitivity to Changes Over Time. Sensitivity to change in symptoms and activity tolerance reported by LTRs over time was examined using longitudinal data from the subset of LTRs (n = 51 ) who provided 4 repeated measures of the QLTP over a 12-month period. Generalized linear modeling procedure was used to examine whether the QLTP scores changed over time in the hypothesized direction. It was anticipated that during the first year after transplantation, QLTP scores would tend to decrease as health status improved; however, it would not be unusual for symptom scores to show a tendency to rise over time to reflect the development of later complications.

Table 2 Descriptive statistics for the QLTP scores by gender and transplant type

Table 1 Sample characteristics (n = 177)

Results

Sample Characteristics and QLTP Scores

The QLTP was completed by 177 LTRs on posttransplant days that ranged from 5 to 3028, with a mean and standard deviation of 609.9 (815.7) days. The majority were recipients of single lung transplants (70%), male (54%), with a mean age of 50.4 (12.2) years, able to climb an average of 1.3 (1.4) flights of stairs, and rated their cough and shortness of breath levels at 1.7 (2) and 2.3 (2.4), respectively. Concurrent measures of pulmonary function were available for a subset of 92 LTRs and revealed, on average, a 15.6% (18.9%) decline in FEV^sub 1^ compared to their posttransplant best. The age, gender, and type of transplant were similar to values reported for the transplant program and those reported nationally (Table 1),9 and should therefore be generalizable to other LTRs.

Scores for the QLTP and its subscales overall, by gender, and by transplant type are presented in Table 2. LTRs who received single lung transplant procedures reported significantly more problems with ADL than double lung recipients (P=.05), and women reported significantly more general and fewer ADL symptoms than men (P=.02). These findings are consistent with those of Lanuza and colleagues10 who found that women and single lung recipients reported more frequent symptoms than their counterparts. When the correlations between QLTP scores and age were examined, older patients reported significantly more problems with ADL (P = .002). The total QLTP score was marginally related to age, indicating that older patients tended to report more symptoms overall (P = .051).

Reliability

Internal Consistency. Reliability coefficients were moderate to high (Table 3), with a KR-20 of 0.83 for the total QLTP and 0.73 for the Respiratory, 0.74 for the General, and 0.95 for the problems with ADL subscales, indicating adequate internal consistency for the QLTP and its 3 subscales.

Stability. Test-retest stability of the QLTP in a subsample (n = 51) was high across the 4 time points during the 1 year study period (Table 3) with an ICC of 0.70 for the QLTP total score and ranging between 0.59-0.74 for the subscales.

Validity

Convergent Construct Validity. Pearson correlation coefficients between QLTP scores and other measures are presented in Table 4. The QLTP behaved in a manner consistent with theoretical expectations, demonstrating statistically significant positive correlations between symptom scores, declines in FEV^sub 1^, and cough and shortness of breath ratings. As anticipated, the strongest direct relationships were observed between the respiratory subscale and the convergent measures (correlations ranged from 0.29-0.53), and inverse relationships were found between symptoms and the number of flights of stairs that the LTRs could climb (correlations ranged from -0.18 to -0.35) Although the correlation coefficients were statistically significant, ours is not the first study to fail to find stronger correlations between symptoms and other manifestations of lung disease including pulmonary function and functional ability.11-13

Table 3 Reliability of the QLTP and subscales

Table 4 Correlations between QLTP and convergent construct validity measures

Factor Structure. The factor structure of the QLTP was examined for each of the 3 subscale domains as specified when the QLTP was first developed (Figure 1).

* ADL Domain. For the ADL domain, only 1 factor was extracted for the 5 items in the domain with factor loading values >0.83 (Table 5). A 1-factor model fit the data for all the 5 items in the ADL domain of the QLTP, providing evidence of the unidimensionality of this domain. Principal component factor analysis of the ADL domain revealed the primary factor explained 82.9% (eigenvalue = 4.1 ) of the total variance with all the 5 items loaded significantly (>.40) on the primary ADL factor.

* Respiratory Domain. The respiratory domain showed a multidimensional structure. Principal components analysis results showed that a 4-factor multi-dimensional structure was the most interpretable and explained 53.6% of the total variance among the 12 items in the respiratory domain (Table 6). The 4 factors were labeled: primary, nasal, fever, and sleep, and explained 19.6% (eigenvalue = 2.3), 13.0% (eigenvalue= 1.6), 11.7 % (eigenvalue= 1.4), and 9.4% (eigenvalue= 1.1) of the total variance, respectively. For each of the 4 factors of the respiratory domain, a 1-factor model fit the data, providing evidence of the unidimensionality of each factor in this domain.

* General Domain. Unlike the ADL and respiratory domains, the general domain was specified a priori as a multidimensional structure with 7 factors named for the body systems that manifest the symptom items: neurologic; cardiac; gastrointestinal; musculoskeletal; endocrine; genitourinary; and ear, nose, and throat (ENT). Therefore, analysis for each factor was conducted separately to determine which were unidimensional. As shown in Table 7, for 4 of the 7 factors (neurologic, gastrointestinal, musculoskeletal, and genitourinary) a 1-factor model fit the data providing evidence of their unidimensionality. The primary factor in each of these 4 explained 27.5% (eigenvalue= 1.7) of the total variance in the 6 items of the neurologic; 26.1% (eigenvalue = 2.1) in the 5 items of the gastrointestinal, 66.5% (eigenvalue= 1.3) in the 2 items of the musculoskeletal, and 50.4% (eigenvalue= 1.0) for the 2 items in the genitourinary. For the cardiac, a 2-factor model fit the data, and these 2 subfactors explained 72.7% of the total variance in the 3 items. The 2 subfactors were labeled as cardiac a and cardiac b and each explained 38.5% (eigenvalue= 1.2) and 34.2% (eigenvalue= 1.0) of the total variance, respectively. For the ENT items, a 3-factor model fit the data and explained 53.9% of the total variance in the 8 items in ENT. The 3 subfactors were labeled as ENT a, ENT b, and ENT c and each explained 24.7% (eigenvalue = 2.0), 15.6% (eigenvalue= 1.3), and 13.6% (eigenvalue= 1.1), respectively. Factor analysis was not conducted for the endocrine factor, because there was only 1 item; at least 2 items are required to extract a common factor.

Figure 1 Measurement structure of the Questionnaire for Lung Transplant Patients (QLTP).

Table 5 Principal factor loadings of the items in the Activities of Daily Living subscale

* Relationships Among Domains. Strong, significant correlations (P<.01) were found between the QLTP overall and each of its domains (r = 0.82 for respiratory, r = 0.87 for general, and 0.54 for the ADL). Moderate but significant correlations were found among the 3 domains of the QLTP (Table 8) with (r = 0.58, P<.01) between the respiratory and general domains; (r=0.30, P<.01) between the respiratory and ADL domains, and (r=0.16, P=.04) between general and ADL domains. Correlations among factors are also presented.

Sensitivity to Change. The QLTP data collected across 4 time points (3, 6, 9, and 12 months) were used to examine the sensitivity to change over time. Generalized linear modeling procedure revealed that the QLTP scores changed over time in the hypothesized direction. There was a general pattern of improvement in the QLTP with trends for later declines (Figure 2). A significant linear trend was found among the repeated measures declining over time in the general subscale of the QLTP (F = 7.7, P=.008). Results also showed a sig\nificant curvilinear pattern with a tendency to decrease over time before rising again at the 12-month measurement for the total and respiratory (F = 5.6, P= .022) and ADL (F= 19.7, P<.001) subscales of the QLTP.

Table 6 Principal factor loadings of the items in the Respiratory subscale

Table 7 Principal factor loadings of the items in the General subscale

Table 8 Intercorrelations among the QLTP and subscale scores

Discussion

The present study applied additional statistical techniques to further evaluate the psychometric properties of the QLTP. Reliability estimates confirmed that the QLTP met the criteria for internal consistency and stability (≥ 0.70).14 Correlations between the QLTP and criterion measures were in the hypothesized direction and met the criterion of ≥ 0.25 for convergent validity.5 Factorial analysis provided an empirical basis for testing the validity of the multidimensional instrument and confirmed the presence of 3 domains that coincided with the respiratory, general, and ADL subscales of the QLTP. Item analysis revealed several opportunities for refinement and will be used to guide the selection of items to include in future versions of the scales of the QLTP as described below.

Rationale for Future Revisions

Although the QLTP shows promise for assessing LTRs' physical symptoms and their impact on activity tolerance, the need for ongoing refinement and testing persists. We plan to refine the QLTP by reducing the number of items, dimensionalizing symptom responses to enhance the ability to detect changes over time, and adding symptoms of psychological distress.

Like most clinical instruments, there is a desire to shorten the QLTP to ease completion within the time constraints of an outpatient clinical setting. Several steps will be taken to select the final items to include in the revised version of the QLTP. The first step will be to systematically determine items that were completed by most subjects. For example, the symptom item "jaundice" will be eliminated because no one reported it. The next step will be to apply the decision rule for item retention based on a factor loading value ≥ 0.4. However, exceptions may be made to retain certain items with (<0.4) loadings that are thought to be important clinically. For example, the symptom item "constipation" will be retained under subdomain IV of the general domain (gastrointestinal) for clinical reasons even though its factor loading was <0.4. Also, using principal component analysis, the most informative items from each scale will be reviewed, based on the magnitude of item factor loadings that indicated the discriminating power. For example, because several items within the ADL scale were highly correlated (r = >0.70), items may be eliminated from that subscale to reduce significant redundancy. Also, based on the factor analysis, certain symptom items will be rephrased to improve consistency with the latent variable label. For example, the item "sleep propped on pillows" led to the identification of a subfactor labeled cardiac b. This item will therefore be replaced by the symptom "trouble lying flat to sleep" to reflect its cardiac etiology. Because "increased thirst" was the only item related to "endocrine," at least 3 to 5 other items will be considered for addition because it appears to be generally more difficult to replicate subscales with fewer than 3 salient items for each subscale. Finally, transplant clinicians will be asked to review the revised version to be sure they support elimination of selected items and to suggest any additional items that should be included in the scales.

Figure 2 Scores (and 95% CIs) of the Questionnaire for Lung Transplant Patients at 3, 6, 9, and 12 months (n = 51).

The original QLTP is limited in its ability to assess variations in individual symptom severity over time because the responses to items are dichotomous, allowing the patient to indicate only the presence or absence of a symptom. To improve the ability of the QLTP to detect variation in recurring symptoms over time, the response format will be changed from asking LTRs to endorse whether a symptom is present or absent, to asking them to rate the degree of distress associated with the symptom on a 5-point Likert scale (0 = no distress, 4 = extreme distress). Likert scaling is widely used in instruments measuring opinions, levels or attitudes.5 The scale for rating levels of intensity for cough and shortness of breath will remain on a scale of 0 to 10 because the correlations between those ratings and visual analog scales were previously found to range between 0.87 to 0.93, respectively.4

Lastly, recent evidence suggests that anxiety and depressive symptoms are independent predictors of physical symptoms and activity intolerance after lung transplantation,15 but the original QLTP did not include symptoms related to psychological distress. Therefore, 3 to 5 psychological distress symptom items will be added to assess their presence in individual LTRs and to explore the relationships between symptoms of psychological distress and physical symptoms and activity intolerance.

Conclusion

Overall, these findings provide further support of the reliability and validity of the QLTP for assessing physical symptoms and activity intolerance after lung transplantation in individual recipients, LTRs in aggregate or comparison groups, on one occasion, and serially over time. Like all empirical measures, evaluation and refinement are ongoing. Consistent with the iterative manner in which instruments are designed and tested, after the proposed revisions are made, the psychometric properties, factor structure, and utility of the QLTP will be reevaluated in a new sample of LTRs.

Acknowledgments

This research was supported by US Department of Health and Human Services, National Institutes of Health, NINR F31 NR07425 (ADD) and HL059490 (AI).

References

1. De Vito Dabbs AJ. An Integrated Description of the Symptom Experience of Acute Lung Rejection Using Conceptual Triangulation [dissertation]. Pittsburgh, Pa: University of Pittsburgh; 2003.

2. Lough ME, Lindsey AM, Shinn JA, Stotts NA. Impact of symptom frequency and symptom distress on self-reported quality of life in heart transplant recipients. Heart Lung. 1987;16:193-200.

3. Manzetti JD, Hoffman LA, Sereika SM, Sciurba FC, Griffith BP. Exercise, education, and quality of life in lung transplant recipients. J Heart Lung Transplant. 1994;13:297-305.

4. De Vito Dabbs A, Hoffman LA, Dauber JH, Zullo T, Iacono AT. Evaluating the reliability and validity of the questionaire for lung transplant patients. Prog Transplant. 2002;12:191-200.

5. DeVillis RF. Scale development: theory and applications. Bickman L, Rog DL, eds. Applied Social Research Methods Series. Vol 26. 2nd ed. Thousand Oaks, Calif: Sage Publications; 2003.

6. Anastasi A, Urbina S. Psychological Testing. Upper Saddle River, NJ: Prentice Hall; 1997:98.

7. Carmines EG, Zeller RA. Reliability and Validity Assessment. Newbury Park, Calif: Sage; 1979.

8. Comrey AL, Lee HB. A First Course in Factor Analysis. Hillsdale, Mich: Lawrence Erlbaum Associates; 1992.

9. Organ Procurement Transplantation Network. 2003 Annual Report of the US Registry of Transplant Recipients and Transplant Data 1988- 2003. Richmond, Va: Organ Procurement Transplantation Network; 2003.

10. Lanuza DM, McCabe M, Norton-Rosko M, Corliss JW, Garrity E. Symptom experiences of lung transplant recipients: comparisons across gender. Pretransplantation diagnosis, and type of transplantation. Heart Lung. 1999;28:429-437. 11. de Jong W, van der Schans CP, Mannes GP, van Aalderen WM, Grevink RG, Koeter GH. Relationship between dyspnea, pulmonary function, and exercise capacity in patients with cystic fibrosis. Respir Med. 1997;91:41- 46.

12. Narsavage GL. Applications in research. Research report: the influence of symptoms, lung function, mood, and social support on the level of functioning of patients with COPD. Perspect Respir Num. August 1996;7:6-7.

13. Wolkove N, Dajczman E, Colacone A, Kreisman H. The relationship between pulmonary function and dyspnea in obstructive lung disease. Chest. 1989;96:1247-1251.

14. Nunnally JC, Bernstein IH. Psychometric Theory. New York, NY: McGraw-Hill; 1994:254.

15. De Vito Dabbs A, Dew MA, Stilley CS, et al. Psychosocial vulnerability, physical symptoms, and physical function after cardiothoracic transplantation. J Heart Lung Transplant. 2003;22:1268-1275.

Annette De Vito Dabbs, RN, PhD, Yookyung Kim, PhD, Judith Vensak, BSN, Sean Studer, MD, Aldo Iacono, MD

University of Pittsburgh, Pittsburgh, Pa

To purchase electronic or print reprints, contact:

The InnoVision Group

101 Columbia, Aliso Viejo, CA 92656

Phone (800) 809-2273 (ext 532) or (949) 448-7370 (ext 532)

Fax (949) 362-2049

E-mail reprints@aacn.org

Copyright North American Transplant Coordinators Organization Dec 2004

Source: Progress in Transplantation

More News in this Category