• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Empowerment Evaluation in Redesigning A Public Health Unit Nutrition Program

Posted on: Sunday, 2 April 2006, 06:00 CDT

By Dwyer, John J M; Vavaroutsos, Denise; Lutterman, Ann; Hier, Michelle; Et al

Abstract

This article illustrates how empowerment evaluation was used in Toronto Public Health's (TPH) nutrition programming redesign to consult with staff about how roles, responsibilities, and organisational structure could be changed to improve how nutrition programs are delivered. One of three moderators facilitated the ten two-hour focus group sessions in TPH. TPH staff, namely 71 front- line staff and 13 managers who were responsible for providing community nutrition services, participated in the study. Focus group participants included Public Health Dietitians, Public Health Nutritionists, Public Health Nurses (PHNs), and paraprofessionals (i.e., community nutrition assistants). Participants' preferred roles, responsibilities, and organisational structure in TPH, which they believe would improve nutrition service delivery in the community, were examined. A constant comparison approach was used to develop themes inductively. It was found that participants wanted Dietitians and Nutritionists to provide current nutrition-related information to them. They felt that nutrition programs should be promoted better and made more accessible to the public. They suggested that Dietitians and Nutritionists and other staff should share information with each other better. They suggested that Dietitians and Nutritionists should provide nutrition services directly to the public and provide support to other staff, mainly PHNs, who deliver nutrition programs. In conclusion, this empowerment evaluation produced results that were used to assist in decision making about nutrition programming.

(Can J Diet Prac Res 2006;67:36-40)

Rsum

Cet article dcrit comment l'valuation de l'habilitation a t utilise pour reconcevoir les programmes de nutrition du Dpartement de sant publique (DSP) de Toronto. Avec cette approche, les auteurs ont consult le personnel sur la faon de modifier les rles, les responsabilits et la structure organisationnelle en vue d'amliorer la prestation des programmes de nutrition. Trois animateurs ont dirig chacun un groupe de discussion raison de 10 sessions de deux heures. Le personnel du DSP, soit 71 employs de premire ligne et 13 gestionnaires responsables de la prestation des services de nutrition communautaire, ont particip l'tude. Les participants des groupes de discussion taient des dittistes, des nutritionnistes et des infirmires en sant publique ainsi que des paraprofessionnels (par ex., des assistants en nutrition communautaire). On a examin les rles, responsabilits et lments de structure organisationnelle qui, selon les participants, pourraient amliorer la prestation des services nutritionnels dans la communaut. La mthode de comparaison constante a t utilise pour faire ressortir les thmes par induction. Les participants souhaitaient que les dittistes et les nutritionnistes leur fournissent de l'information de nutrition jour. Ils taient d'avis que les programmes en nutrition devraient faire l'objet d'une meilleure promotion et tre plus accessibles au public. Ils ont suggr que les dittistes et les nutritionnistes et les autres membres du personnel partagent mieux l'information entre eux. Ils ont galement propos que les dittistes et les nutritionnistes fournissent des services nutritionnels directement au public et apportent leur soutien aux autres membres du personnel, notamment les infirmires en sant publique, qui appliquent les programmes de nutrition. En conclusion, cette valuation de l'habilitation a produit des rsultats qui ont clair la prise de dcision sur la programmation en nutrition.

(Rev can prat rech ditt 2006;67:36-40)

INTRODUCTION

In 1998, the amalgamated city of Toronto was formed as a result of provincial legislation that merged seven municipal governments. The strategic and business-planning process at Toronto Public Health (TPH) identified that a restructuring of several TPH programs and services, including nutrition programming (i.e., all nutrition- related program activities), was required. A nutrition programming redesign was necessary for several reasons. First, amalgamation of the six former health units in Metropolitan Toronto revealed variations in the type of nutrition programs and services offered and the method of service delivery. Second, there was an increase in prevention activities and funding opportunities resulting from emerging evidence to support the role of diet in the prevention of chronic diseases of public health significance. Third, the structure of TPH changed during amalgamation such that most Registered Dietitians (RDs) with expertise in population health strategies were placed in Healthy Lifestyles programs with little allocation to Family Health programs. This concentration in one area of TPH raised concerns about the lack of coordination and integration of public health nutrition professionals in planning, implementing, and evaluating nutrition components in two interrelated programs, namely the Healthy Lifestyles and Family Health programs. Fourth, the number of RDs in TPH is limited, so it was important to use these resources effectively to maximise service delivery and to be proactive for future opportunities. In summary, a nutrition programming redesign was necessary to address variation in service delivery, the increase in program activities, and human resource issues.

A comprehensive strategy was used to obtain information for TPH's nutrition programming redesign. The various sources of information included:

1. Identification of TPH's core business in relation to nutrition programs and services

2. Identification of TPH's goals, objectives, and guiding principles for nutrition programs and services (including consideration of the Mandatory Health Programs and Services Guidelines developed by the Ontario Ministry of Health and Long- term Care [1])

3. A community consultation

4. Staff consultation, described in this article

5. Literature reviews on nutrition-related public health issues and effective strategies for nutrition promotion

6. An organisational structure review

A TPH manager requested that a program evaluation specialist (PES) advise on the staff consultation component. The PES (first author) was an experienced in-house evaluation specialist who provided evaluation workshops and consultation to staff on how to evaluate their programs. The PES suggested that Fetterman's empowerment evaluation approach be used (2). "Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination" (2,p.3). It's an evaluation process in which staff evaluate their own programs to promote self-determination. Often, a PES takes the role of facilitator or coach in this collaborative process (2).

Fetterman described three steps to the empowerment evaluation approach: asking staff and participants to a. identify their mission or vision of the program, b. take stock of the program, which involves identifying important program activities and discussing their strengths and weaknesses, and c. identify strategies to achieve program goals and objectives (2). Staff and participants are better able to learn how to evaluate their own programs by taking these steps. These three steps were followed to collect information about TPH's nutrition programming redesign, before the staff consultation described in this article.

Case studies in which PESs facilitated an empowerment evaluation process have been presented in the literature (3-5). For example, Sullins (5) presented a case study of how Fetterman's empowerment evaluation approach (6,7) was adapted when evaluating a mental health drop-in centre. As a PES, the author of that study developed and conducted the evaluation but continually incorporated stakeholders' input. The author contended that the evaluation empowered stakeholders, even though an adapted empowerment approach was used. The current article illustrates how empowerment evaluation was used, with support from a PES, in TPH's nutrition programming redesign to consult with staff about changes to roles, responsibilities, and organisational structure to improve delivery of nutrition programs.

METHOD

Focus group participants

Staff who deliver nutrition services in the community were selected to participate in this study. This included public health dietitians, public health nutritionists, public health nurses (PHNs), and paraprofessionals (i.e., community nutrition assistants). For clarification, public health dietitians and public health nutritionists are registered dietitians (registered with the College of Dietitians of Ontario) but public health nutritionists are required by legislation to have a master's degree in community nutrition. Since the number of staff in some job categories was low, all public health dietitians and managers involved in nutrition- related activities were invited to participate in the study. Staff in other job categories, which are listed below, were randomly selected from TPH's list of employees. Random selection was used to ensure fairness in seeking input from staff. Random selection for the purpose of generalising the results was not appropriate in this type of study because the dynamics of group discussion preclude the collection of independent responses.

A total of 84 TPH staff pa\rticipated in focus groups. Participants were front-line staff (n=71) and managers (n=13) who met two criteria: 1. they had worked in TPH for at least six months and 2. they were responsible for providing nutrition services in the community. The first criterion ensured that participants had sufficient experience in TPH to comment on their roles and responsibilities and the organisational structure. Staff in both Healthy Lifestyles and Family Health teams participated in focus groups. From Healthy Lifestyles, there were ten public health nutritionists, four public health dietitians, nine PHNs, 13 paraprofessionals, and five managers. From Family Health, there were 18 PHNs, eight public health dietitians, nine paraprofessionals, and eight managers.

Focus group interview guide

The focus group interview guide consisted of questions about staffs perceived current and preferred roles and responsibilities, and the organisational structure. This article is limited to staffs preferred roles, responsibilities, and the organisational structure to support future program delivery (see Table 1).

Table 1

Focus group interview guide

Procedure

TPH's research review committee provided ethical approval for the study, which included voluntary informed consent from participants. Upper management supported the study and committed the resources needed to complete it

A nutrition programming redesign project team, consisting of managers and staff in all job categories relevant to nutrition, advised a research team that was a subcommittee of the project team. The research team included a public health nutritionist, a public health dietitian, a PHN from Healthy Lifestyles, a PHN from Family Health, a nutrition manager on an ad hoc basis, and a PES. The PES acted as facilitator or coach to teach and help the staff research team to conceptualise and implement the staff consultation themselves. Some members of the research team were on the redesign project team and the PES attended redesign project team meetings, which ensured that the evaluation was relevant.

With guidance and support from the PES, the research team developed research objectives to direct the evaluation, a. to determine staffs perceived and preferred roles and responsibilities in delivering nutrition programs and b. to obtain their suggestions about how the organisational structure of TPH could be changed to improve the delivery of nutrition programs. The research team agreed on an evaluation method involving focus groups that would provide rich information to the redesign project team. The redesign project team informally pilot tested the interview guide by commenting on whether the questions were relevant, clear, and elicited discussion.

Three non-nutrition staff who were knowledgeable about general principles of conducting focus groups (8,9) and had moderating experience facilitated the ten two-hour sessions during the fall of 2001 and winter of 2002. Between four and 13 staff participated in each session. Steps were taken throughout the focus group study to ensure that the research team conducted a study that produced quality information that was accurate, trustworthy, and timely. For example, the research team decided not to audiotape the sessions because of concerns about inhibiting discussion. To balance methodological rigour and feasibility in the study, the research team decided to have two note-takers at each session to maximise the accuracy of recording. At each session, two from a pool of ten note- takers independently recorded notes and subsequently compared notes to produce a synthesised, accurate account of the discussion.

Research team members used a systematic process of analysing the data. The PES trained the research team members in how to use The Ethnograph software (version 5.0, 1998, Qualis Research Associates, Salt Lake City, Utah). Typed notes from the sessions were imported into The Ethnograph and then printed with numbered lines of text. For expediency, the research team worked in two pairs to analyse these data independently. Each pair analysed data for half of the questions. With support from the PES, the research team used a constant comparison approach to analyse the data. This approach to developing themes involves coding comments by continually referring to previously coded comments for comparison (10,11). The research team and the PES met to discuss the coded comments. Codes for these comments were then transferred to The Ethnograph; comments assigned the same code were retrieved and synthesised to help the research team write up the results.

RESULTS

Only highlights of the results are described in this article. Participants' discussion of preferred roles, responsibilities, and organisational structure in the future yielded several major themes: increasing staff in-services, promoting nutrition programs, increasing accessibility to programs, increasing communication among staff, and expanding and clarifying the roles of TPH public health dietitians and nutritionists.

Increasing staff in-services

Many participants were concerned about the different levels of nutrition knowledge among staff and the potential for dissemination of inaccurate information to staff and the public. As a means of quality assurance, it was suggested that a committee responsible for identifying sound nutrition standards and accurate nutrition information be established. It was also suggested that a Web-based dissemination strategy be used to provide up-to-date nutrition information to staff.

Many participants stated that it would be helpful if TPH public health dietitians and nutritionists provided both formal and informal in-service to staff in Family Health and Healthy Lifestyles teams. They said that informal in-services could feature public health dietitians and nutritionists presenting nutrition updates at staff meetings on various nutritionrelated topics such as breastfeeding, diabetes, and eating disorders. It was deemed important that staff who are not public health dietitians and nutritionists but who deliver nutrition programs have sufficient nutrition knowledge. However, some participants cautioned about providing too much detail during the in-services to avoid unnecessary confusion about the roles, responsibilities, and expected nutrition knowledge of staff. For example, a participant said: "It comes back to the issue of whether nutritionists and dietitians should be training public health nurses to be dietitians." Some participants stated that the multidisciplinary approach to program delivery is an opportunity to recognise all staffs skills and to share accurate nutrition information among staff in different teams such as Healthy Lifestyles, Family Health, Healthy Environments, and Dental Health.

Promoting nutrition programs

On another theme, many participants said that the nutrition programs should be promoted more in the community and to staff in the city. They stated that TPH should work with the media more closely and use the TPH Web site better to disseminate nutrition information to the public. Some participants suggested that TPH should work with Health Canada, which is the federal department that focuses on improving the health of Canadians, and perhaps with industry, to disseminate nutrition information.

Increasing accessibility to programs

Participants reported that greater public accessibility to programs was needed. Many participants perceived that TPH public health dietitians and nutritionists were needed in the community to deliver nutrition services. They said that nutrition services should be extended to clinics, community health centres, and community agencies. Many participants emphasised that services offered by both Healthy Lifestyles and Family Health programs, such as prenatal program drop-in centres and mom-and-tot groups, should be offered at common locations in the community. They felt that this "one-stop shopping" would increase public accessibility to programs, staff referrals, and interaction among staff from various programs. Some participants suggested that nutrition services should also be offered to apartment tenants more frequently. Many participants said that nutrition activities offered as a series of workshops should be more flexible and broader in scope. For example, a participant said, "It could be flexible so that people could come [for] two to three weeks, rather than it being a sixweek program. We have to match with clients and be more flexible instead of asking them to match our needs." Many participants mentioned expanding the delivery of nutrition education and other strategies so that they are offered to people across the lifespan, ranging from prenatal programs to programs for seniors. Some participants suggested that more public health dietitians and nutritionists from various cultures are needed on teams to accommodate the nutrition needs of the multicultural community.

Increasing communication among staff

Another theme was the need for greater communication among staff who deliver nutrition services. For example, participants pointed out that staff who are responsible for delivering certain nutrition programs should be more informed about other community nutrition services provided by their colleagues. This was deemed necessary to ensure that staff have a broader perspective on nutrition service delivery in the community and to improve collaboration and referrals between Healthy Lifestyles and Family Health teams. A nutrition newsletter for staff was suggested. Some participants said that staff should be informed about TPH nutrition resources that are available to them. Some managers suggested that public health dietitians and nutritionists in Healthy Lifestyles teams regularly attend Family Health team meetings to improve communication. Some participants felt that the role of a nutrition program coordinator would also improve internal communication.

Expanding and c\larifying the roles of TPH public health dietitians and nutritionists

Participants also suggested expansion and clarification of roles for public health dietitians and nutritionists, to improve and increase nutrition service delivery in the community. A typical comment was: "Dietitians should be more active and visible in the community... Dietitians should be a vibrant part of community nutrition." To accomplish this, some participants suggested that they belong to fewer work groups to have sufficient time for program delivery. Many participants also wanted public health dietitians and nutritionists to be more accessible to provide support and consultation to Family Health and Healthy Lifestyles teams. Participants in Family Health teams requested that public health dietitians and nutritionists be more accessible to engage in activities such as discussing client cases, participating in joint home visits where there are special nutrition needs, providing menu planning in-services to community agency staff, conducting infant and toddler feeding sessions during parenting classes, and doing one- on-one nutrition counselling. Similarly, many paraprofessionals requested more guidance from public health dietitians and nutritionists. Also, some paraprofessionals felt that they would provide better service if they could consult with public health dietitians and nutritionists and if those professionals would occasionally be available to accompany them on visits to clients.

Considering that nutrition is cross-cutting, some participants suggested that public health dietitians and nutritionists should allocate consulting time and other service time to Healthy Lifestyles programs that are linked to nutrition, such as services related to physical activity, cancer prevention, and tobacco use prevention. While some PHNs felt that they should continue to develop and implement nutrition activities, such as nutrition counselling for prenatal clients and parent education about healthy eating for children, other PHNs said that service delivery would be improved if PHNs focused less on nutrition activities.

TPH has a Health Connection program in which PHNs provide health information and advice on a variety of topics, including nutrition, to members of the community via phone or e-mail. Some PHNs experienced frustration when they tried to contact a public health dietitian or nutritionist with a nutrition question but were not able to access them in a timely manner. Some participants suggested that public health dietitians and nutritionists should be members of the Health Connection team to respond directly to nutrition-related inquiries. Many participants suggested that a decentralised structure in which public health dietitians and nutritionists are located in the various district offices would also allow front-line staff to have greater access to public health dietitians and nutritionists. Also, some participants felt that more multidisciplinary teams, which include public health dietitians and nutritionists and PHNs, are needed to effectively deliver nutrition programs.

DISCUSSION

Participants' comments may reflect the history of their health units before amalgamation when they may have had different roles and fewer public health dietitians and nutritionists, which resulted in these professionals often not being allocated to front-line work. Traditionally, PHNs delivered nutrition programs, designed by public health dietitians and nutritionists, in the community because of the number of PHNs in the health units. There has been an increase in RDs over the years and some TPH programs have received direct funding to deliver nutrition programs in the community. This has led to an increased awareness of the potential role of RDs in front- line service delivery. The participants' struggle with role clarification likely reflects the evolution of the organisation and the limited number of RDs employed.

CONCLUSION

This study shows how a PES can provide evaluation support to a staff research team in a way that fosters the research team's self- determination to obtain data for decisionmaking about programming issues. In the role of coach, a PES can work closely with staff to ensure a methodologically sound evaluation that is relevant to their needs. When a PES encourages staff involvement in evaluations, those staff will more likely understand and feel a sense of ownership of the evaluation process and results which, in turn, should make it more likely that they will use the evaluation findings (12). To maximise use of the results, the research team subsequently shared them with the redesign project team. Also, a nutrition programming redesign newsletter described the evaluation process and highlights of the findings to TPH staff. Data from this staff consultation and other previously mentioned sources were collected, analysed, and interpreted to produce an accurate, comprehensive assessment which, in turn, informed decision-making among upper management. This collective information provided useful direction to establish recommendations for improving the way that nutrition services are delivered to the public.

RELEVANCE TO PRACTICE

Teaching staff to evaluate their own programs and providing assistance during the evaluation can take considerable time. However, conducting empowerment evaluations is time well invested because the process can increase staffs evaluation capacity. Empowerment evaluation applies the following idea to evaluation: "Give someone a fish and you feed her for a day; teach her to fish, and she will feed herself for the rest of her life" (2, p. 35). This evaluation process strengthened skills of staff research team members, such as developing research objectives, developing an evaluation method, and analysing qualitative data, that they can use when evaluating their programs in the future.

This empowerment evaluation approach can be used in other settings beyond public health and to conduct various types of evaluations, such as evaluability assessments to specify how the program is conceptualised, needs assessments, process evaluations, and outcome evaluations.

Acknowledgements

This work was conducted at Toronto Public Health, Toronto, ON. Appreciation is extended to Diana Baxter and Lawrence Keen for their input in planning the consultation process for the nutrition programming redesign and to Carol Paulsen for her assistance with The Ethnograph. Staffs and managers' participation and assistance in the consultation are appreciated.

. . .staff evaluate their own programs to promote self- determination.

References

1. Ontario Ministry of Health and Long-term Care. Mandatory Health Programs and Services Guidelines. Ontario: Queen's Printer for Ontario; 1997. Available at http://www.health.gov.on.ca/english/ providers/ pub/pubhealth/manprog/mhp.pdf; accessed 7 October 2004.

2. Fetterman DM. Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications, Inc.; 2001.

3. Daltuva JA, Williams M, Vazquez L, Robins TG, Fernandez JA. Worker-trainers as evaluators: a case study of a union-based health and safety education program. Health Promot Pract 2004;5:191-8.

4. Secret M, Jordan A, Ford J. Empowerment evaluation as a social work strategy. Health Soc Work 1999;24:120-7.

5. Sullins CD. Adapting the empowerment evaluation model: A mental health drop-in center case example. Amer J Eval 2003;24:387- 98.

6. Fetterman DM. Empowerment evaluation [American Evaluation Association presidential address]. Eval Pract 1994;15:1-15.

7. Fetterman DM. Empowerment evaluation: an introduction to theory and practice. In: Fetterman DM, Kaftarian S, Wandersman A, eds. Empowerment evaluation: knowledge and tools for self- assessment and accountability. Thousand Oaks, CA: Sage Publications, Inc.; 1996, p. 1-46.

8. Krueger RA. Focus groups: A practical guide for applied research (3rd ed.). Newbury Park, CA: Sage Publications, Inc.; 2000.

9. Stewart DW, Shamdasani PN. Focus groups: Theory and practice. Newbury Park, CA: Sage Publications, Inc.; 1990.

10. Flick U. An introduction to qualitative research. Thousand Oaks, CA: Sage Publications, Inc., 1998.

11. Morse JM, Richards L. Readme first for a user's guide to qualitative methods. Thousand Oaks, CA: Sage Publications, Inc.; 2002.

12. Patton MQ. Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.; 1997.

JOHN J. M. DWYER, PhD, Department of Family Relations and Applied Nutrition, College of Social and Applied Human Sciences, University of Guelph, Guelph, ON and Public Health Research, Education and Development Program, Public Health and Community Services, Hamilton, ON; DENISE VAVAROUTSOS, MHSc, RD, ANN LUTTERMAN, MEd, RN, MICHELLE HIER, BASc, RD, MAY HUGHES, BScN, RN, MARY-JO MAKARCHUK, MHSc, RD, Toronto Public Health, Toronto, ON

Copyright Dietitians of Canada Spring 2006

Source: Canadian Journal of Dietetic Practice and Research

More News in this Category