July 30, 2008

Conceptual Development of a Measure to Assess Pharmacists’ Knowledge of Herbal and Dietary Supplements

By Lin, Hsiang-Wen Pickard, A Simon; Mahady, Gail B; Popovich, Nicholas G

Objectives. To describe the conceptual development of a measure for assessing pharmacist knowledge of herbal and dietary supplements. Methods. A standardized approach to constructing a multiple-choice competency examination following 8 pre-specified criteria (eg, specifying the target spectrum of herbal and dietary supplements) was used to create an item bank. The quality of each item was evaluated by 5 herbal and dietary supplement content experts based on specific criteria in 3 rounds of review.

Results. From 122 initial items, 56 items were retained for the item bank representing 4 content areas: efficacy/effectiveness, safety, drug-supplement interactions, and regulation. The experts tended to agree that the constructed items represented a wide range of difficulty.

Conclusion. The initial development of a conceptually based item bank/measure of pharmacist herbal and dietary supplement knowledge lays the groundwork for a large-scale validation study. The measure should be useful as a standalone tool and as a component of a knowledge, attitude, and behavior survey for the assessment of pharmacist traits related to herbal and dietary supplements.

Keywords: knowledge, herbal supplements, dietary supplements, pharmacists, standardized examination


The marketing of herbal and dietary supplements has led to dramatic growth in patient use of these products in the last 2 decades.1-4 As health care professionals whose expertise focuses on pharmacotherapy, pharmacists have an important role in evaluating the safety, efficacy, and effectiveness of herbal and dietary supplement products and guiding patients concerning their appropriate use.5 However, few studies have investigated pharmacists' knowledge level of herbal and dietary supplements and the extent to which pharmacist knowledge and attitudes towards such products contribute to patient counseling.

Researchers in the field of health care have a tradition of developing and conducting knowledge, attitude, and behavior surveys to gather information, evaluate current practice, and effect curricular refinement.6 A review of knowledge, attitude, and behavior studies reported that pharmacists perceived they did not possess sufficient knowledge of herbal and dietary supplements.7 Problematically, there was little evidence to support the availability of conceptually based, validated measures to assess a pharmacist's knowledge concerning herbal and dietary supplements, either as a standalone measure or within the context of a knowledge, attitude, and behavior survey. The purpose of this study was to report the initial development of a measure to assess pharmacist knowledge of herbal and dietary supplements. This represents part of a larger study to develop and validate measures of pharmacists' knowledge, attitude, and counseling on herbal and dietary supplements, which will be used to evaluate the interrelationships between these traits/constructs of interest.

A survey instrument that measures knowledge is typically created to capture respondent knowledge and understanding of scientific and clinical content areas.6 A format, such as multiple-choice or true/ false items, provides a structure for a metric that is used to assess an individual's total score. The score is intended to represent the participant's actual knowledge. Regardless of the type of item structure, there is always a best or most correct answer for each item on a knowledge measure. The advantage of using multiple- choice items over true/false items is that multiple-choice items incur less chance of "blind guessing," which refers to the situation where there is an equal probability of a respondent endorsing each option, where the fewer the options, the higher the likelihood that a respondent's guess will result in the correct answer. Multiple- choice items also demonstrate certain response patterns that are more informative than a dichotomous scoring structure (ie, scoring "1" for choosing the keyed option and scoring "0" for selecting any incorrect distractor options).8 However, the advantage of adopting multiple-choice items for actual knowledge assessment in a survey can be jeopardized when problematic, defective, or biased items and/ or their options (choices) are used. To minimize this phenomenon, wellestablished procedures for item generation and review should be followed. Subsequently, an intensive statistical evaluation of the item responses is then implemented.8


In this study, herbal and dietary supplements refer to those products that follow the definition of Dietary Supplement Health and Education Act of 1994 (DSHEA): ''a product (other than tobacco) that is intended to supplement the diet that bears or contains one or more of the following dietary ingredients: a vitamin, a mineral, an herb or other botanical, an amino acid, a dietary substance for use by man to supplement the diet by increasing the total daily intake or a concentrate, metabolite, constituent, extract, or combinations of these ingredients."9 Knowledge-based items were created to cover the most frequently used and popular products based upon literature searches, existing measures, web sites, and internal expert opinion.

A structured method of constructing knowledge items was used that mirrored Pray and Popovich's study for a standardized competency examination.10An internal review of the items was conducted by 2 experts with herbal and dietary supplement teaching experience in the pharmacy curriculum, postgraduate continuing education, and herbal and dietary supplements counseling with patients and other health care providers. The items were constructed with 8 criteria in mind.

Relevance of the item to the target population. The target population was practicing pharmacists who possess a broad range of knowledge about herbal and dietary supplements used in the United States.

Conformity of the item to the intended purpose of the measure. The herbal and dietary supplements measure was created as a diagnostic assessment, ie, formative assessment, to determine the extent to which pharmacists have mastered herbal and dietary supplement-related knowledge in their formal and continuing professional education.

Pertinence of the item to the target spectrum of dietary supplements. The top 10 herbal products and top 10 supplements, as well as current issues related to herbal and dietary supplements were identified from literature, government web sites, and marketing authorities.11-13 Herbals included St. John's wort, valerian, Ginkgo biloba, chamomile, saw palmetto, ginger, ginseng, garlic, feverfew, and soy products. Co-enzyme Q, creatine, folic acid, melatonin, fish oils/fatty acids, multivitamins, vitamin E, minerals (ie, zinc, calcium/magnesium), glucosamine, and chondroitin constituted the non- herbal dietary supplements. 14-17

Format of the item. The multiple-choice items were designed with the 4-option response structure in mind, labeled as A, B, C,D(ie, 1 best answer as the keyed option and 3 alternative option as distractors) were constructed.

Adherence of the item to the specified themes. Initially, the items were designed to include 6 themes, in no particular ranking of importance: efficacy/effectiveness, safety, drug-supplement interactions, regulation, information retrieval /evaluation, patient counseling.

Contribution of the item toward meeting the stated objectives for the measure. Objectives of the measure were to discern levels of knowledge about herbal and dietary supplements that would enable pharmacists to help patients and consumers make informed decisions about their use. These include the ability to:

* describe common use(s) for the most frequently used herbal and dietary supplements marketed in the United States;

* list the common adverse reactions and/or quality concerns related to herbal and dietary supplements use;

* identify common interactions between herbal and dietary supplements and drugs/diagnostic tests (ie, drug-supplement interactions, drug-test interactions);

* describe the regulatory status of herbal and dietary supplements (ie, DSHEA, labeling/claims of herbal and dietary supplements) in the United States;

* identify appropriate strategies to conduct information/ literature retrieval and evaluation related to herbal and dietary supplements; and

* relate the concepts and communication approaches/ techniques associated with patient counseling for herbal and dietary supplements.

Target literature resources used to construct/ develop the item. In some respects, the application of evidence- based practice can be recognized as an extension of a systematic approach to drug information.18 Originally, the recommended systematic approach for pharmacists to respond to a drug information request included 5 steps: classifying the request, securing background information, performing a systematic search, creating and providing the response, and classifying the information requested.19 Although more limited literature and evidence on herbal and dietary supplements is available compared to conventional medicine, the stepwise approach (ie, consulting tertiary and secondary sources, followed by primary resources) of drug information retrieval and evaluation in conventional medicine was adopted in this study.20 Specifically, the target literature resources included: (1) tertiary resources, eg, the most popular textbooks containing content related to herbal and dietary supplements (ie, Handbook of Nonprescription Drugs, 14th and 15th editions; Nonprescription Product Therapeutics, 2nd edition) 21,22 and books with name recognition23,24; (2) fulltext computer bases, review articles, and secondary resources, which are indexing and abstracting services of the primary literature7,25-36; (3) research studies published in biomedical journals from the primary literature (ie, using English language; 1966-2006; key terms included herbal and dietary supplements [or specific ingredient name], efficacy, adverse reactions, and/ or evidencebased medicine) as identified in PubMed37-39; and (4) of- ficial web sites related to herbal and dietary supplements (eg, Office of Dietary Supplements, National Institutes of Health, and reports posted byAHRQEvidence-based Practice Centers, The Cochrane Library).9,25- 54

Adherence of the item to established writing guidelines. Three cognitive domains adopted from Bloom's taxonomy (ie, knowledge, comprehension, application) for item construction were utilized to categorize items.55 Specifically, the knowledge domain included activities related to one's knowledge, memory, and ability to know the correct answer. The comprehension domain contained activities for translating, interpreting, and extrapolating knowledge. The application domain included the activities of knowing when/why to apply information, especially for a new situation.55

A 31-item checklist for item creation included 4 main categories related to item writing quality: content concern, style concern, the written stem, and item alternatives. 56 For instance, in terms of content, items were intended to reflect specific content, evaluate a single mental behavior as intended by the test specifications, avoid being "opinion-based,' and not perceived as being "picky."56 During the item generation process, this checklist was used as a template to reduce item writing problems.

A 2-way table that specified the content themes to be tested and the desired cognitive domains was created prior to generating the set of items. The expected proportions of herbal and dietary supplements items related to each content theme and cognitive domains were established (Table 1). The marginal proportion of items related to each cognitive domain (ie, knowledge, comprehension, application) were pre-specified at 45%, 33%, and 22%, respectively. The marginal proportion of each content theme was projected as follows: efficacy/effectiveness, 30%; safety, 20%; drug-supplement interactions, 12%; regulation, 15%; information, 12%; and patient counseling, 11%). The intention was to place greater emphasis on the knowledge and comprehension cognitive domains than on the application domain. Three themes (ie, drug interactions, information retrieval and evaluation, patient counseling) constituted approximately 11% of all items in the item bank. Thus, the numbers of items for each theme were initially generated using the guidelines in Table 1. For instance, approximately 7% of all items were created to address the topic of efficacy/effectiveness in the cognitive domain of application.

For the constructed items, 2 separate item reviews were performed. First, draft items were reviewed by lay persons who were not practicing pharmacists. They proofread the items and made suggestions regarding item structure, meaning, and flow. The second content review process was conducted by internal and external experts, including 5 pharmacy-based faculty members and health professional educators who evaluated all items. Two out of 3 external experts (ie, who were not faculty members in the College of Pharmacy) were involved in each round of the "external review," using the same review process and electronic content validation format.

The internal experts reviewed the items based on the following criteria: (1) representativeness of the construct; (2) ability to update information in items; and (3) ability of items to span different levels of difficulty. Based on feedback, more items were created or revised to meet the intended proportion related to the content theme and cognitive domain. The first and second drafts of items in the item bank were distributed to external experts to assess the quality of constructed items. At this stage of the process, the reviewers were asked to provide the keyed response, assign the cognitive domain, assign the corresponding content theme, specify the level of representativeness of the item, specify the level of its difficulty, and provide any editorial suggestions for revision.56 Accordingly, an item was retained in the item bank only if it met all the following criteria for each round of external reviews: at least one expert answered correctly, not all experts guessed, and the mean of representativeness rating was at least 3.5. Upon receiving feedback from external experts, items were retained, revised, or removed for the next draft of items after consultation with the internal experts. This initial development of the measure was approved by the Institution Review Board of the University of Illinois at Chicago.


Figure 1 presents the item development process for the knowledge measure. Originally, 122 items were created for the item bank. Ten items were added to meet the desired proportion of items related to each content theme and cognitive domain in the first draft of items in the item bank. Twenty-one items constructed on the theme of patient counseling appeared to assess the pharmacists' practice experience. Upon re-evaluation, these were deemed out of the scope for a knowledge-specific measure for herbal and dietary supplements. Hence, these items were removed from the first draft of the item bank. After completion of each of the 3 rounds of expert review, there were 111, 98, and 56 items retained, respectively, in the item bank.

When grouping the items based on their cognitive domain and specified content themes (Table 2), the proportion of items created to assess respondents' comprehension domain were consistent (ie, 35.1%, 36.7%, 33.9% in the first, second, and third drafts for the item bank, respectively) with the expected proportion (ie, 33%). The proportion of items developed for the other themes deviated somewhat from the pre-specified proportions in Table 1. In the second draft of items, 13 items (ie, 13.3% of the total item pool) were answered incorrectly by 2 external experts (data not shown); these items were dropped. The remaining 85 items were considered further because at least 1 expert answered the item correctly. Twenty-nine items were removed from the item bank because those items: (1) were answered correctly by experts due to self-reported guessing; (2) were unrepresentative (ie, mean representativeness rating , 3.5); (3) contained out-of-date information; or (4) were too subjective (ie, opinion-based items). For example, questions addressing the regulatory requirement for good manufacturing practices for herbal and dietary supplements were removed because the legal requirement to do so was just put into effect on June 22, 2007.57 None of the items in the theme of information retrieval/evaluation were retained in the third review of the item bank. Of the 56 remaining items, approximately half of the items related to factual knowledge (ie, 50%assigned in knowledge domain). Only 16 % of the retained items measured the application domain (Table 2), which was considered the most complex domain in terms of higher mental processes.

Table 3 presents the results of the "external" expert content validation process at each review step. Of the 56 items in the third draft of the item bank, 44 (79%) were answered correctly without self-reported guessing by 2 external experts. External experts tended to agree that the items were difficult (ie, mean of difficulty = 3.4, on a scale on which 1 = extremely easy to 5 = extremely difficult) but representative (ie, mean of representativeness = approximately 4.0, on a scale on which 1 = not representative and 5 = very representative).


The goal of this study was to apply a systematic approach to develop a measure that assesses pharmacist herbal and dietary supplements-related knowledge. Items were written consistent with guidelines and content themes, producing 56 items for the final item bank that forms the basis for the herbal and dietary supplement knowledge measure.8,10,58 None of the items corresponding to the theme of information retrieval/evaluation were retained in the item bank after compiling the experts' opinion. Upon deliberation, the internal expert panel agreed this theme represented a process that was not specific to herbal and dietary supplements and not an outcome that represented actual herbal and dietary supplements knowledge. Therefore, 4 content themes (efficacy, safety, drug interaction, regulation) concerning herbal and dietary supplements were retained. These 4 themes were consistent with the issues pharmacists confront with regard to common herbal and dietary supplements products according to expert opinion and guidelines proposed by the American Dietetic Association and the American Pharmacist Association (ADA/APhA).5,30,59 The proportion of the total number of items related to each content theme and the cognitive domain of the corresponding draft of items was guided by the proposed proportion of items. Although a smaller proportion of items were retained for the application domain, the domain met the rule of thumb of at least 10 items per cognitive domain.58

When survey design is employed, the purpose of a measure should be clearly delineated, with items formulated to capture the constructs of interest.8,60 To our knowledge, no previous measures of pharmacists' knowledge level of herbal and dietary supplements have followed a structured approach in constructing a standardized competency examination. The results presented in this manuscript represent the first stage of development. In the second stage, the measure will be piloted with a representative group of pharmacists, followed by a large scale validation study. These results will help with the development of a short form derived from the item bank. In the future, the item bank can be reviewed, updated, and/or revised periodically and newshort formmeasures can be developed fromthe pool of items in the herbal and dietary supplement knowledge-based item bank. At the present stage, measure development has relied on expert opinion in the selection and refinement of the items germane to each theme. Psychometric analysis will need to be conducted to further evaluate how well the items were constructed in the item bank. Although fewer items in the application domain were retained in the final item bank, sufficient domain coverage is expected and items can be added to the item bank in the future.


This study reports the initial development of a conceptually based item bank/measure of herbal and dietary supplement knowledge. An item bank of 56 items was developed covering 4 content areas relating to herbal and dietary supplements (ie, efficacy, safety, drug interactions, regulations) and 3 cognitive domains, including knowledge, comprehension, and application. The next phases include large-scale validation, psychometric analysis, and development of a short form version of the measure based on the item bank.


The authors would like to thank the following reviewers and external experts for providing their insights and comments on the draft items in the item bank: Heather Boon, Maria Charisse De Leon, John Hessian, W. Steven Pray, and Matthew Sargent.


1. Eisenberg DM, Davis RB, Ettner SL. Trends in alternative medicine use in the United States, 1990-1997: results of a follow- up national survey. JAMA. 1998;280:1569-75.

2. Wold RS, Lopez ST, Yau CL, et al. Increasing trends in elderly persons' use of nonvitamin, nonmineral dietary supplements and concurrent use of medications. J Am Diet Assoc. 2005;105:54-63.

3. Kaufman DW, Kelly JP, Rosenberg L, et al. Recent patterns of medication use in the ambulatory adult population of the United States: the Slone Survey. JAMA. 2002;287:337-44.

4. Tindle HA, Davis RB, Phillips RS, et al. Trends in use of complementary and alternative medicine by US adults: 1997-2002. Alternative Ther Health Med. 2005;11:42-9.

5. Miller LG, Harris HA, Jackson EA, et al. White paper on herbal products. American College of Clinical Pharmacy. Pharmacotherapy. 2000;20:877-91.

6. Aday LA. Designing and Conducting Health Surveys. 2nd ed. San Francisco, CA: Jossey-Bass; 1996.

7. Kwan D, Hirchkorn K, Boon H. US and Canadian pharmacists' attitudes, knowledge, and professional practice behaviors toward dietary supplements: a systematic review. BMC Complementary Alternative Med. 2006;6:31. Available at: http:// www.biomedcentral.com/1472-6882/6/31. Accessed October 3, 2007.

8. Haladyna TM. Developing and Validating Multiple-choice Test Items. 3rd ed. Mahwah, NJ: Lawrence Erlbaum Associates; 2004.

9. Food Drug Administration. Dietary Supplement Health and Education Act (DSHEA) of 1994. Public Law 103-417,103rd Congress. Available at: http:www.fda.gov/opacom/laws/dshea.html. Accessed November 6, 2006.

10. Pray SW, Popovich NG. The development of a standardized competency examination for doctor of pharmacy students. Am J Pharm Educ. 1985;49:1-9.

11. National Center for Complementary and Alternative Medicine. The use of complementary and alternative medicine in the United States. Available at: www.nccam.nih.gov/news/ camsurvey_fs1.htm#use. Accessed October 6, 2005.

12. American Botanical Council. Common herbs-an introductory guide to herbal healthcare. Available at: http:// www.herbalgram.org/ default.asp?c5common_herbs. Accessed Sepember 6, 2005.

13. Nutraceutical World. The state of dietary supplements. Available at: http://www.nutraceuticalsworld.com/articles/2002/11/ the-stateof- dietary-supplements.php. Accessed Sepember 6, 2005.

14. American Pharmacists Association. ADA/APhA special reporta health professional's guide to evaluate dietary supplements. Available at: http://www.pharmacist.com/pdf/ dietary_supplements.pdf. Accessed August 6, 2005.

15. Barnes P, Powell-Griner E, McFann K NR. CDC advance data report #343. Complementary and alternative medicine use in the United States 2002. Available at: http://nccam.nih.gov/news/ graphics.htm. Accessed December 4, 2006.

16. Marra J. The state of dietary supplements-even slight increases in growth are better than no growth at all. Available at: http:// www.nutraceuticalsworld.com/articles/2002/11/the-state-of- dietarysupplements. php. Accessed December 4, 2004.

17. Clauson KA, McQueen CE, Shield KM, et al. Knowledge and attitudes of pharmacists in Missouri regarding natural products. Am J Pharm Educ. 2003;67(2): Article 41.

18. Moorses KG. Evidence-based clinical practice guidelines. In: Malone PM, Mosdell KM, Kier KL, Stanovich JE, eds. Drug Information: A Guide for Pharmacists. New York: McGraw-Hill Companies; 2001:235- 92.

19. Watanabe AS, McCart G, Shimomura S, et al. Systematic approach to drug information requests. Am J Hosp Pharm. 1975;32:1282- 5.

20. Kier KL, Malone PM, Mosdell KW. Drug information resources. In: Malone PM, Mosdell KM, Kier KL, Stanovich JE, eds. Drug Information; A Guide for Pharmacists. 2nd ed. New York: McGraw- Hill; 2001:53-94.

21. Berardi RR, Kroon LA, McDermott JH, et al. Handbook of Nonprescription Drugs: An Interactive Approach to Self-Care. 15th ed. Washington DC: American Pharmacist Association; 2006.

22. Pray WS. Nonprescription product therapeutics. Philadelphia, PA: Lippincott Williams & Wilkins; 2005.

23. Mahady GB, Fong HHS, Farnsworth NR. Botanical Dietary Supplements: Quality, Safety and Efficacy. The Netherlands: Swets & Zeitlinger; 2001.

24. Mason P. Dietary Supplement. 2nd ed. London: Pharmaceutical Press; 2001.

25. American Academy of Family Physicians. Herbal health products- what you should know. Am Fam Physician.1999;59:4. Available at: www. aafp.org/afp/990301ap/990301e.html. Accessed 2005 January 6.

26. American Academy of Family Physicians. Evaluating perioperative use of herbal medications. Available at: www.aafp.org/ afp/20020501/tips/11.html Accessed December 4, 2006.

27. American Academy of Family Physicians. Herbal therapies: are they good medicine if used properly: American Academy of Family Physicians; 1999. Available at: www.aafp.org/fpr/991000fr/6.html. Accessed December 4, 2006.

28. American Dietetic Association, American Pharmaceutical Association. A healthcare professional? Guide to evaluating dietary supplements. Available at: http://www.pharmacist.com/pdf/ dietary_supplements.pdf. Accessed August 16, 2006.

29. Bondakdar, BA, Guarneri E. Coenzyme Q10. Am Fam Physician. 2005;72:1065-70.

30. Chavis LM. Pharmacy-Based consulting on dietary supplements. J Am Pharm Assoc. 2001;41:181-91.

31. Cupp MJ. Herbal Remedies: adverse effects and drug Interactions. Am Fam Physician. 1999;59. Available at: www.aafp.org/ afp/990301ap/1239.html Accessed March 15.

32. Hadley S, Petry JJ. Valerian. Am Fam Physician. 2003;67:1755- 8.

33. Masley SC. Dietary therapy for preventing and treating coronary artery disease. Am Fam Physician. 1998;57:6. Available at: www.aafp.org/afp/980315ap/masley.html. Accessed March 15, 2005.

34. Morelli V, Naquin C. Alternative therapies for traditional disease states:menopause. Am Fam Physician. 2002;66:129-34.

35. Morris C, Avorn J. Internet marketing of herbal products. JAMA. 2003;290:1505-09.

36. Sierpina VS, Wollschlaeger B, Blumenthal M. Ginkgo biloba. Am Fam Physician. 2003;68:923-6.

37. Gardiner P, Woods C, Kemper K. Dietary supplement use among health care professionals enrolled in an online curriculum on herbs and dietary supplements. BMC Complementary Alternative Med. 2006;6:21. Available at: http://www.pubmedcentral.nih.gov/ picrender.fcgi?artid51526756&blobtype5pdf. Accessed March 5, 2007.

38. Gardner CD, Chatterjee LM, Carlson JJ. The effect of a garlic preparation on plasma lipid levels in moderately hypercholesterolemic adults. Atherosclerosis. 2001;154:213-20.

39. Herrero-Beaumont G, Ivorra JA, Del Carmen Trabado M, et al. Glucosamine sulfate in the treatment of knee osteoarthritis symptoms: a randomized, double-blind, placebo-controlled study using acetaminophen as a side comparator. Arthritis Rheum. 2007;56:555- 67.

40. Evidence-based review program [software]. Version 2005. Available at: http://dietary-supplements.info.nih.gov/Research/ Evidence-Based_Review_Program.aspx#reports. Accessed November 10, 2006.

41. Office of Dietary supplement, National Institutes of Health. Dietary Supplement fact sheet: Vitamin D. Available at: http:// ods.od.nih.gov/factsheets. Accessed December 4, 2004.

42. Office of Dietary supplement, National Institutes of Health. Dietary supplement fact sheet: vitamin B6. Available at: http:// ods.od.nih.gov/factsheets/vitaminb6.asp Accessed December 4, 2004.

43. Office of Dietary supplement, National Institutes of Health. Dietary supplement fact sheet: vitamin B12. Available at: http:// ods.od.nih.gov/factsheets/vitaminb12.asp Accessed December 4, 2004.

44. Office of Dietary supplement, National Institutes of Health. Dietary supplement fact sheet: vitamin E. Available at: http:// ods.od.nih.gov/factsheets/folate.asp Accessed December 4, 2004.

45. Agency for Healthcare Research and Quality. Garlic: effects on cardiovascular risks and disease, protective effects against cancer, and clinical adverse effects, Structured Abstract. Available at: http:// www.ahrq.gov/clinic/tp/garlictp.htm Accessed December 4, 2004. 46. Agency for Healthcare Research and Quality. Effect of supplement antioxidants vitamin C, vitamin E, and Coenzyme Q10 for the prevention and treatment of cardiovascular disease, Structured Abstract. Available at: http://www.ahrq.gov/clinic/tp/aoxcardtp.htm Accessed December 4, 2004.

47. Agency for Healthcare Research and Quality. B Vitamins and Berries and age-related neurodegenerative disorders, Structured Abstract. Available at: http://www.ahrq.gov/clinic/tp/berrytp.htm Accessed December 4, 2004.

48. Agency for Healthcare Research and Quality. Melatonin for treatment of sleep disorders, Structured Abstract. Available at: http:// www.ahrq.gov/clinic/tp/melatntp.htm Accessed December 4, 2004.

49. Agency for Healthcare Research and Quality. Multivitamin/ mineral supplements and prevention of chronic disease. Structured Abstract. Available at: http://www.ahrq.gov/clinic/tp/ multivittp.htm Accessed December 4, 2004.

50. Agency for Healthcare Research and Quality. Health Effects of Omega-3 Fatty Acids on lipids and glycemic control in Type II diabetes and the metabolic syndrome and on inflammatory bowel disease, rheumatoid arthritis, renal disease, systemic lupus erythematosus and osteoporosis, Structured Abstract. Available at: http://www.ahrq.gov/clinic/tp/o3lipidtp.htm Accessed December 6, 2005.

51. Agency for Healthcare Research and Quality. Effects of Soy on health outcomes, Structured Abstract. Available at: http:// www.ahrq.gov/clinic/tp/soytp.htm Accessed December 6, 2006.

52. Agency for Healthcare Research and Quality. EPC Evidence reports-completed reports. Available at: http:// www.ahrq.gov/ clinic/epcindex.htm#dietsup Accessed December 8, 2006.

53. Linde K, Barrett B, Wo?lkart K, et al. Echinacea for preventing and treating the common cold. Cochrane Database of Systematic Reviews. 2006(Issue 1):Art. No.: CD000530. DOI: 10.1002/ 14651858.CD000530.pub2. Available at: http:// www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/ CD000530/ frame.html Accessed December 8, 2006.

54. Palanisamy A, Haller C, Olson KR. Photosensitivity reaction in woman using an herbal supplemnts containg Ginseng, Goldenseal, Bee Pollen. J Toxicol. 2003;41:865-7.

55. Bloom BS. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc; 1956.

56. Haladyna TM, Downing SM, Rodriguez MC. A review of multiple- choice item writing guidelines for classroom assessment. Appl Meas Educ. 2002;15:309-34.

57. Food Drug Administration. FDA issue dietary supplements final rule; 2007. Available at: http://fda.forumsplace.com/ urljump.php?url5http://www.fda.gov/bbs/topics/NEWS/2007/ NEW01657.html. Accessed December 3, 2007.

58. Gronlund NE. Assessment of student achievement. Boston, MA: Publishers' Design and Production Services, Inc; 2003.

59. American Pharmacists Association. APhA special reportmedication and dietary supplement safety, efficacy, and quality: a primer on appropriate product selection 2004.

60. American Psychological Association, American Educational Research Association, National Council on Measurement in Education. Standards for educational and psychological tests. Washington, DC: American Psychological Association; 1999.

Hsiang-Wen Lin, MS, A. Simon Pickard, PhD, Gail B. Mahady, PhD, and Nicholas G. Popovich, PhD

College of Pharmacy, University of Illinois at Chicago

Submitted October 16, 2007; accepted December 9, 2007; published June 15, 2008.

Corresponding Author: A. Simon Pickard, PhD. Assistant Professor, Center for Pharmacoeconomic Research & Depts of Pharmacy Practice and Administration, College of Pharmacy, University of Illinois at Chicago, 833 South Wood St, Rm 164, M/C886, Chicago, IL, USA 60612- 7230, Tel: 312-413-3357. Fax 312-996-0379. E-mail: [email protected]

Copyright American Association of Colleges of Pharmacy 2008

(c) 2008 American Journal of Pharmaceutical Education. Provided by ProQuest Information and Learning. All rights Reserved.