Program Evaluation

Laverne Marri-Perez commented on Group 4: Embodied Learning - Program Evaluation 11/1

Marti Suddarth commented on Group 1 (Stephanie Coleman) - Self-Directed Learning 
          Program Evaluation 11/3

Mandy Lowe commented on Group 2: Transformational Learning - Program Evaluation 11/3

Kim Clist commented on Group 2: Transformation Learning and Group 3 Experiential Learning on 11/6.

Kylie Hartman commented on

Group Member	Roles & Responsibilities for Program Evaluation
Member 1: Kim Clist	Wrote the Introduction and Conclusion, Student Response, Created a Drive in Google Docs, Suggestions for Improvement, created and added links to appendices
Member 2: Mandy Lowe	Prepared Table Summary of Evaluations, Student Response, Suggestions for Improvement
Member 3: Marti Suddarth	Wrote summary of Student Responses, Student Response, Suggestions for Improvement
Member 4: Laverne Marri-Perez	Leader, established, invited, and met with professionals to evaluate our group program. Student response, final review of the paper to post to the Blog
Member 5: Kylie Hartman	Wrote Group Reflection, Student Response, Scribe: Conference Calls, Suggestions for Improvement

Group 5: Program Evaluation of Narrative Learning

Tell Your Story

Ball State University

EDAC The Adult As A Learner

October 30, 2016

Dr. Bo Chang

Introduction

         Creating an educational plan based on research is sound practice. However, it is also imperative that the plan also be evaluated by professionals in the industry, reflected on, and improved as necessary. As a group, we welcomed the experienced critical analysis of two ESL teachers, Bertha L.P. and Kathryn M. What we learned was that we are on the right track, however, there was still work to be done in order to be mindful of and appropriately responsive to various factors the students could introduce to our broad program. As a result, there was a need for refining who the program was appropriate for and how it would be explained and demonstrated. In the end, what we have come up with is a more focused and clearly defined program.

Professional Evaluations

Professional Evaluator #1

What do you like most about the program design?

The crux of this program is narrative learning; a method that uses what is familiar to students as a foundation for new learning.  Its design incorporates a number of key components of Sheltered Instruction Observation Protocol (SIOP) Model that are key to addressing the needs of English Language Learners:

• Building background knowledge
• Linking storytelling to personal experiences and emphasizing key vocabulary
• Contextualizing key vocabulary
• Interactive
• Peer collaboration is important because students can learn from one another and build on what they already know
• Speaking to one another also enables students to practice their speaking skills before they have to share their stories with the class
• Practice and Application
• Including hands on materials (books such as “The Three Little Pigs”)
• Modeling correct English and telling a narrative multiple times before students have to share their own narrative will help them to gain an understanding of language and expectations
• Feedback will help students to reflect on their progress and make adjustments as they work on their narratives; feedback upon completion of the assignment will help student to identify how they succeeded and areas they can improve upon
• Student choice allows students to demonstrate their strengths or practice areas of weakness
• Supplementary Materials
• Using bilingual dictionaries
• Providing a variety of resources (technology) supports student learning and allows them to take more ownership over their work in that they are facilitating their own learning.

What do you think should be improved? Why? How?

• To enhance this program, your group might consider setting clear expectations by using a scoring rubric. Creating/using a rubric would communicate the criteria students need to meet to be successful with the narrative program (i.e.: fluency, pronunciation, vocabulary, etc.).  The rubric would assist students in self-assessing their progress throughout the duration of the project and serve as a jumping point for conversations during conferences.
• Direct vocabulary instruction for basic terms (i.e.: culture, traditions, family, story, memory, childhood, society, memory, narrative, etc.) might be another consideration.  Pre-teaching basic vocabulary will help students to understand the expectations and project.  Consider using terms, definitions, and images to help students understand key words.
• Similar to teaching vocabulary, providing sentence frames throughout the sessions will support students with the language they need in order to respond to discussion questions. Hearing the teacher speaking the sentence frames will also help students with pronunciation and fluency.

Professional Evaluator #2

What do you like most about the program design?

• The emphasis on narrative learning as a method of enabling ESL students to increase their English auditory and verbal skills.
• The use of teaching modeling storytelling: voice modulation, motion, repetition, use of hand and face expressions (importance of “mime” in communication).
• Use of stories, folktales, hopes and dreams taken from students’ backgrounds and experiences.
• 1:1 sessions with students to encourage ESL students; to support ESL students’ in developing American grammar and vocabulary; to evaluate students’ ongoing progress inability to “tell a story” in English, several sessions devoted to practice the narrative.
• Breakout sessions among students where they can learn and help each other with English expressions as well as learn about their different backgrounds (country of origin, types of storytelling themes used in their cultural backgrounds).
• Use of multimedia: video, audio recording device, iPad tablets with drawing apps, computers, and website.

What do you think should be improved? Why? How?

• No mention of the students’ level of proficiency indicated in the program. Assessing the students’ ability to understand basic common everyday expressions is critical to their being able to produce/use the new language. This involves lots of listening and guided/modeled feedback from the students. There are several rubrics available to help assess students in this area.
• Teachers should become familiar with some stories (popular, folktales, generational oral stories) of the students’ ethnic backgrounds to encourage students and contribute to the students’ self-worth in another culture.
• Presentations of the students’ storytelling in the wider community (other ESL classes, libraries) should be arranged to celebrate their cultural diversity and evolving mastery of the English language.

Students’ Responses

We would like to thank Bertha L.P. and Kathryn M. for evaluating our program plan, Tell Your Story.  We know that both are busy educators, and we appreciate the gift of time that each has given us.

We are grateful for the encouragement our evaluators offered.  Both positively noted our use of modeling, as the teacher in our program tells a story or stories to help program participants learn storytelling techniques and program expectations.  Both complimented the use of narrative, familiar stories, and personal experiences as tools to practice oral language skills.  

As each member of the group responded to our evaluators’ feedback, four common themes emerged.

1. We need to establish a minimum level of English-language skill and a way of assessing that skill before the program begins.  While it is obvious that this program is not appropriate for adults with no English language knowledge, our evaluators pointed out, and our members agreed, that we need to:

1. Determine what skills participants need to have before joining the program
2. Create a rubric or other evaluation instrument for determining potential participants’ level of skill
3. Devise a method of communicating those requirements

One of our evaluators mentioned rubrics that are available. Before implementing this program, we would need to determine whether an existing rubric would meet our needs, and if not, then we would partner with an experienced teacher of adult language learners to develop an appropriate pre-program evaluation.

2.   During the program, participants will receive informal, oral feedback from fellow participants and, during one-on-one midweek sessions, from the teacher(s). However, adding more formal, written feedback will increase the effectiveness of the program.   We need to determine whether we will use the same evaluation instrument used as a pre-test or develop a second instrument. Either way, students will have the instrument to guide their preparations, to give feedback, to document growth and learning, and to act as a reference during face-to-face conversations with the teacher.

3. Some of the instructions/plans need to be clarified.  For example:

1. After the teacher tells the class a story, s/he leads a discussion. Will the class discuss the story itself, the teacher’s telling, or both? A list of suggested discussion questions would give the teacher and class guidance. If students have access to the list of questions before observing the storytelling, then they will know what to look and listen for during the presentation.
2. Program participants are offered the choice of using fables/fairy tales originally heard in their first languages or of using personal experience stories from childhood. Guidelines for selecting stories need to be clarified. (i.e. This story will be posted on a website and shared with children, so the story needs to be child-appropriate. The story needs to be less than five minutes in length).
3. Not everyone has experience giving feedback to others, yet peer feedback is an important component of the program. Instructions, written and oral, need to be developed and explained. A written example and rubric or checklist should be developed. Giving feedback should be modeled by the teacher(s) before participants are expected to give feedback.
4. Similar topics, questions, rubrics, etc. need to be developed for the teacher to utilize during one-on-one sessions with participants.

4. Although one of the evaluators suggested that teachers need to become familiar with stories common from participants’ cultures prior to the program’s first session, and while our group members agreed that familiarity would be helpful, we had the following concerns regarding implementation:

1. There are thousands of human languages and cultures and millions of stories. It would be impossible for one human being to be familiar with all of them.  Unless participants are coming from only one or two cultural groups, this requirement seems overwhelming for the teacher.
2. Having participants from a variety of backgrounds increases the richness and depth of the stories. Limiting participants to one or two cultural groups, therefore, seems counterproductive.
3. Likewise, allowing participants to select stories with personal interest or meaning adds to the richness and depth of the program. Limiting participants to just a few tales that are familiar to the teacher is also counterproductive. The teacher(s) will learn from the participants if the stories are new to him/her as well.

For these reasons, we are not incorporating this suggestion into our program.

Reflection

There were many highlights throughout the course of this project which allowed us to grow and become a strong performing group, and to design an educational program utilizing Narrative Learning. Each of us used our own strengths during the design of our narrative learning program. We each allowed and encouraged the other group members to use their creativity and to think outside the box. We continued our weekly phone discussions and divided up assignments. Individual work was posted on Google Drive for other members to proofread and offer suggestions. The wonderful professional feedback we received gave us experienced insight and perspective which we used to modify and strengthen the design.

           During the course of this project we learned many lessons. We learned how important communicating with each other truly is, that no one job is any more important than another, and how lucky we are to be working together. We truly appreciate each other. The lesson that stood out the most for the group was how important it was for each member to actively take part in every step of the project. One person may have been assigned as leader for a certain section but everyone was an active participant in each piece. We all took on the roles of individual work, proofreading, posting, and simply answering questions or sharing ideas. We all supported each other which allowed for our group to flourish. We are looking forward to the next step and the opportunity to finalize our program.

Conclusion

What has come together after this evaluation process is a more focused and clearly defined program. Based on the evaluations of two ESL professionals, we learned that our program was on the right track. However, it was in need of some refining. Rather than the original broad program that could be altered for many situations and students at various levels of language skills, our group narrowed our plan and addressed learning at a specific level. We have created a sound educational program, based on research, evaluated by professionals, and refined in order to be most effectively implementable. The process of research, development, evaluation, reflection, and refinement has been a learning experience for each of us.

Summary of Program Evaluation

Evaluator	Ideas for Improving Program Design	Revisions/Group Responses
Kathryn	Create a rubric to more clearly communicate expectations for student success in the program.	Add written feedback from the instructor to students throughout the sessions. Students can use this for study and during meetings with the instructor. Include videos showing examples of storytelling for students to visualize the final product.
	Clarify instructions regarding class discussion.	We would compile a list of suggested discussion questions for the instructor and students, especially when participating in group discussion. Written guidelines also would be implemented and modeled by the instructor, so that students can give appropriate feedback to classmates.
	Provide sentence frames to help with fluency and pronunciation.	Model and include sentence framing on displays during class sessions. Computers and/or projectors would be used for this purpose as well.
Both Evaluators	Assess the student’s language proficiency prior to the program.	Investigate existing available rubrics and collaborate with ESL teacher to develop an evaluation tool. Implement the evaluation by testing the student prior to beginning the course.
Bertha	Instructor should familiarize himself with popular folktales of student’s backgrounds.	We have decided not to include this as a suggestion for instructors. We feel it is especially important for students to feel open to select their own story, as long as it is appropriate for sharing. This could also be difficult and overwhelming for an instructor.
	Arrange presentations of student work to community.	At the final session, family and friends can be invited for viewing of the student’s final projects. Videos would be shared with the community in promotional materials as well.

Program Evaluation Appendix #1: Professional Profiles

Program Evaluation Appendix #2: Individual Student Responses