Evaluation Models

Chapter 10 of the text makes a strong case for evaluation. As educators, we know that time and money are in limited supply (if not inadequate) in most of our school programs. If we receive grant funds for a particular project, it is even more imperative that we be good stewards of our resources. Poorly planned and implemented programs lead to waste. Program evaluation, although it cannot ensure 100% success, reduces our risk of failure as it helps us to assess the programs value and benefits, use best practices for design and implementation, and follow through so that we may realize goal attainment, or at least better understand our lack thereof.

Several evaluation methods are provided in chapter 10. Along with the CIPP and Kirkpatrick models, I found both Rossi’s and Brinkerhoff’s models to be potentially useful in my future career as a librarian.

I was interested in Rossi’s Five Domain Evaluation Model because it does not take a one-size-fits-all approach, but rather allows for the evaluation to be tailored to better fit the local program. I think this could be effectively employed to evaluate a local library program, service, or resource. For this example I will evaluate the resource of K-12 Online Databases (Encyclopedia Britannica, EBSCO, etc) using Rossi’s 5 steps.

Step 1: There is a need for our school’s students to be able to research reputable sources of information, and the Databases in conjunction with the guidance of the librarian will be able to provide this.

Step 2: We (as evaluators) are able to access data that gives evidence of the learner benefits of being able to research the Databases when the program is implemented by properly trained librarians.

Step 3: We must ask if the librarian (or teachers) have been properly trained in the use of the Databases, and then, if they are able to effectively train students in research skills needed to maximize these resources.

Step 4: After we have employed the Databases for several months, we would evaluate 1) statistics that show how often the Databases are being used, and 2) an assessment of students’ research skills.

Step 5: Now that the Databases are no longer being funded by the state, do the benefits of having the Databases outweigh the high cost of an annual subscription? Could students learn research skills and access this information elsewhere in a way that is more cost-effective?

Brinkerhoff’s Success Case Method would also be an effective evaluation to conduct in order to assess the success of a program that trains students on researching the above mentioned K-12 Databases. I can appreciate that this method seeks to satisfy the needs of stakeholders as well. Because programs and resources such as these require a large investment in both time and money, it is important that students, teachers, administrators, parents, and tax payers feel that the learners and the community are receiving a worthwhile return on their investment in this program. The impact model used during step 2 of this evaluation would be appropriate because it helps to clarify desired skills and knowledge to be gained, takes into consideration environmental factors that may influence the outcome, and takes into account business goals such as cost effectiveness. This method focuses on and studies cases in which this program has experienced greatest success. It also determines what caused that success and what future implications this may have on the program.

What aspects of the program should be evaluated?

It is important to evaluate whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. However, if I were to use any of the evaluation methods addressed above I would also be able to assess other important aspects of the program such as:

Is there a legitimate local need for the program? Was the program designed using best practices? Were implementation goals met? Is the program cost effective? Does it meet the desires of stakeholders? What are environmental factors that influence the program? What are the key factors that contribute to the success of a program like this? These are all important questions.

“Managers do things right, leaders do the right things.” (Bennis, 1994)

Managers are naturally leaders, and leaders naturally fall into management positions. If chosen to develop educational technology training sessions for teachers at my school, I would need to think like both a manager and a leader.

Because I am asked to do this at a time during which our school’s budget is tight, I know that I will have to carefully design the training with thought of maximizing our limited resources.

I would first try to plan what type of technology trainings are most needed, and the scope of the trainings, in light of available resources. I would like to train teachers to use the technology resources already available because I know that a time of economic decline may not be the best time to purchase new technologies, especially if we are not properly trained in using the ones we already have. (I have seen technology resources go to waste too many times in my own district because teachers are not trained sufficiently in their use, or are not equipped with ideas and support in implementing the technology into their classrooms.)

I would next assess my project resources. I know that money is scarce, and that time is also often a scarce resource in school related projects. My most abundant resources may be human resources. Good teachers like to teach and share knowledge, even with other teachers. If I am able to recruit enough teachers to my project team, I may be able to make up for the shortage of time resources so that no one person is stretched too thin. The teachers’ positions on the team would then be delegated.

Subject area/ technology expert teachers would be put in the positions of delivering session instruction and training. It would behoove these teachers in leadership positions to use the 4 phases of Situational Leadership. During phase1 of technology training, instructors will offer detailed instruction and close supervision to the teacher-learner becoming familiar with the technology. During phase 2, instructors will continue to offer detailed instruction, recognize learner achievement, and help learners to build confidence in using the technology. Phase 3 of training will focus on desired outcomes. Perhaps teacher-learners might demonstrate ways in which the technology might be implemented in the classrooms to the benefit of their own students. During phase 4, teacher-learners are now skilled enough to take what they have learned into their actual classrooms. The session leaders might follow up on the training by continuing to monitor the success (or possible struggles) that teachers are having in the implementation of the newly learned technology. The duration of continued support to teacher-learners could be decided upon as part of the scope of the project, but most likely, good teacher-leaders will continue to support their colleagues on an ongoing basis anyway.