Article 1 : MEASURING SUCCESS : EVALUATION STRATEGIES FOR DISTANCE EDUCATION
Online Analytical Response
1. Article 1 : MEASURING SUCCESS : EVALUATION STRATEGIES FOR DISTANCE EDUCATION (Barbara Loockee, Mike Moore, John Burton) http://www.educause.edu/ir/library/pdf/eqm0213.pdf
2. Article 2 : EVALUATION IN DISTANCE LEARNING (Allan Woodley and Adrian Kirkwood)( http://www1.worldbank.org/disted/Management/Benefits/qa-02.html )
latest version
1. Article 1 : MEASURING SUCCESS : EVALUATION STRATEGIES FOR
DISTANCE EDUCATION
by Barbara Lockee, Mike Moore, and John Burton
This article describes the different stages and types of evaluation for distance-delivered experiences. Its also offer some guidance for developing an evaluation plan for distance education programs. Evaluation generally breaks down into two broad categories :
- Formative
- Summative
Formative evaluation serves to improve products, programs, and learning activities by providing information during planning and development. Data collected during the design and development process provides information to designers and developers about what work and what doesn’t an its can be use to improve the system.
Summative evaluation determines if products, programs and learning activities, usually of the need addressed or system goal.
Formative Evaluation the best way to ensure quality in a unit or course before its release. Or we can say that formative evaluation like as internal quality control, and summative evaluation reflects how well the final object works in the real world. It’s also can help identify choices that might not be the most effective, and give the developer an opportunity to revise the course before implementation.
Formative Stages and Methods; generally, formative evaluation can involve several different stages. Here we have six stages:
1. Design review
2. Expert review
3. One-on-one review
4. Small group review
5. Fields trial
6. Ongoing review
Design review: other designer or other familiar with the design strategy to evaluate the elements of the design itself. The review begin very early in the design process and continue at least until the design is ‘set’. This stage aim to improve the project’s instructional design.
Expert review: usually involves a review by content experts or somebody who can judge the appropriateness of the content and learning activity for the target learners. This step avoids developing a product that simply will not work with the intended learner population because of unacceptable content, content bias, missing content or approach to the content.
One on one review: involve a member of the development or design team meeting with the representatives of the target population to go through a piece of the project talking concretely about it.
Small group review: try out fairly finished component in a setting that permits interaction among representatives of the target group. Data are offer collected by survey instruments, group interview, simple test performance and the like.
Field trial: try out the completed package with a samples of target learners in the actual field setting for which the program was designed.
Ongoing review: required because learning system cannot ever be considered complete. No matter how well evaluated the program error will creep in. more importantly content production values and technology, learning orientation and more will change, so making redesign always necessary.
Summative Evaluation is a second approach to determining the success of distance education course or program involve collecting evidence at the conclusion of the instructional event.
The summative evaluation process break down into a few definitive steps. First the general areas of the concern must be delineated, from that, questions regarding these area’s of focus can be developed.
Experience and review of the distance education literature suggest that most categories of interest fall under the three primary, inputs, outcomes, and implementation issues.
For more detail you can look and read table 1 in article. Table 1 provides a tabular representation of the three general areas, their subcomponents and some potential data that could be collected to address each area.
old version
Summary of Article 1
Only by evaluating the effectiveness of DE programs can we justify their use and continue to develop their quality. The many factors involved in the success of distance offerings make the creation of a comprehensive evaluation plan a complex and daunting task. This article describes the different stages and types of evaluation for distance-delivered experiences. It’s also providing some guidance for developing an evaluation plan for distance education programs.
Evaluation generally breaks down into two broad categories; formative and summative. Formative evaluation serves to improve products, programs, and learning activities by providing information during planning and development. Data collected during the design and development process provides information to the designers and developers about what works and what doesn’t, early enough to improve the system while it remains malleable.
Summative evaluation determines if products, programs and learning activities, usually of the need addressed or system goal. Simply, formative and summative evaluations differ in terms of the audience for the information collected, and the intention behind the data collection. Summative evaluation is information provided to audiences external to the design and development team about how the entire package works in a real setting.
Formative Evaluation of DE; is the best way to ensure quality in a unit or course before its release. Or we can say that formative evaluation like as internal quality control, and summative evaluation reflects how well the final object works in the real world.
The development of distance course requires consideration of formative evaluation issues, largely falling into the two primary categories of instructional design issues (such as teaching strategy choices and assessment methods) and interface design issues (Web site navigation, aesthetics, and so forth). For example, online course can fail to produce learning outcomes if students encounter a poorly designed web site. As course development takes place, the designers make choices with regard to each of these categories. Formative evaluation can help identify choices that might not be the most effective, giving the developer an opportunity to revise the course before implementation.
Formative Stages and Methods; generally, formative evaluation can involve several different stages. Here we have six stages:
1. Design review :
2. Expert review :
3. One-on-one review :
4. Small group review :
5. Fields trial :
6. Ongoing review :
Summative Evaluation of DE; is a second approach to determining the success of a DE course or program involves collecting evidence at the conclusion of the instructional event. Summative data can be collected from a variety of sources using an extensive menu of data collection methods. The summative evaluation process breaks down into a few definitive steps. First, the general areas of concern must be delineated; from that, question regarding those areas of focus can be developed. Table 1 provides a tabular representation of the three general areas, their subcomponents, and some potential data that could be collected to address each area. After determining specific evaluation questions, appropriate information gathering techniques can be chosen. Data is collected, analyzed, and reported to constituent audiences. Table 1:
article 2
Article 2 :
Evaluation in Distance Learning
Allan Woodley and Adrian Kirkwood
Evaluation in the sense of "the systematic investigation of the worth or merit of some object" has several meanings within education. Its most common purpose is to grade and classify students, to sort out the successes from the failures. However, here we are concerned with "Programme Evaluation", where the focus is on how well a particular educational programme, curriculum or teaching method works, how it might be improved and how it compares with alternatives.
Innovations in distance education frequently involve new teaching methods, new media, new student groups, new curricula and new institutions. In these contexts "evaluation" tends to cover broad range of activities and employs the full armoury of social science methods of data collection and analysis. Distance education evaluators are aware of the traditional versus the new paradigm debate (see Morgan, 1984) and attempts have been made to overcome the problems of conducting essentially qualitative research with distance students. However, distance education evaluation remains eclectic as to its methods and this seems to be inevitable when we consider what such evaluation consists of.
System Evaluation
- Basic measures of activity
Any system evaluation must begin with certain basic measures. How many courses have been produced? How many students are there? How many applicants had to be turned away? This data is drawn from administrative records and is presented, proudly or otherwise, in annual reports.
- Measures of efficiency
Allied to measures of activity come those of efficiency. How many students successfully complete the courses? What workload do they attempt? What is the throughput of students? Again this data comes from administrative records (McIntosh, Woodley and Morrison, 1980).
The evaluation can move beyond the descriptive to the examination of patterns and causes. For example, postal surveys can be carried out asking students why they dropped out of courses. However, these tend to produce low response rates and answers of dubious validity. A more systematic approach involves a combination of methods including detailed statistical analysis, an understanding of the subjective process of " dropping-out" and an awareness of the different policy options and their likely impact (Woodley and Parlett, 1983).
Other efficiency measures centre on the question of cost-effectiveness. On the face of it distance education is a cheap teaching method, but just how cheap or whether it is cheap at all is still in dispute (Wagner,1977 and Mace, 1978). Disagreements among the economists centre upon difficulties in making comparisons with conventional institutions. How does one allow for the fact that distance students remain economically active while studying? Is it important that mature students have less years of economic activity in which to employ their new knowledge?
- Outcomes
Measures of adequate learning are usually considered to be covered by formal exams and assessment. However, as we note later in this report, there have been some attempts to measure the development of distance students as learners using study inventories and in-depth interviews over a period of several years. In one case Open University economics students were compared with conventional students by administering a standardised test of economic knowledge (Lumsden and Scott, 1980).
On many distance courses there are no formal exams and follow up surveys have to be carried out to see whether there have been appropriate changes in behaviour or attitudes. Such a study has been undertaken at the Open University to see whether people carried out the energy-saving measures in their home that were specifically recommended. Elsewhere the focus has been on health campaigns (Hall, 1978) and changes in agricultural methods (Coleman and Opuku, 1968).
Mail surveys of Open University graduates have been carried out to measure the personal, occupational and educational outcomes of their studies (Swift, 1982). Such studies allow one to measure the subjective benefits experienced by individuals and the extent to which the qualification is recognised by other educational institutions and professional bodies. Methodological problems include those of tracing graduates after a number of years, finding appropriate comparison groups and determining which changes were a direct result of their studies.
The recognition of the qualification can also be approached from the other direction. A survey of employers was carried out to establish the standing of the Open University degree and the acceptability of its graduates (McIntosh and Rigg, 1979).
Distance teaching can have other outputs besides the more obvious ones. These include the use of teaching materials in other institutions (Moss, 1979) (Glaister and Carr, 1986); the passing on of materials to other learners (Stainton Rogers, 1984); and effects on the educational motivation of the children of distance learners (Fenster, 1982).
- Programme aims
Some distance teaching programmes specify their basic aims in terms of what and whom they intend to teach. One of the evaluation needs will be establish the extent to which these aims have been met. The easiest way to measure who the students are is to include detailed questions on the application form and store this data as part of the student record. However, most institutions to not want to make their application forms too daunting or imply that certain information such as ethnic background may influence the chances of gaining a place. It is, therefore, more usual to collect only basic demographic information at this stage and to supplement it by means of special surveys at regular intervals (McIntosh, et al, 1977).
When one of the programme's aims is "open-ness" or an increase in social equity then particular attention must be focussed on formerly disadvantaged groups, be they women, ethnic minorities, working class people or whatever. As well as measuring their representation among the student body and looking at the progress they make on the courses, one can also look at the barriers to greater participation. At the Open University this has taken the form of carrying out surveys of the general public to determine the levels of awareness and knowledge of the institution (Swift, 1980), surveys of people who sent for details but decide not to apply (Woodley and McIntosh 1977), and surveys of applicants who decline the offer of a place (Woodley, 1983).
- Policy Evaluation
Formative evaluation in the policy area can take the form of market research. Surveys of prospective students and employers can be carried out to measure the likely demand for possible new courses. Surveys of current students can also be used to sound out opinion on various policy options. A recent study tested the reactions of Open University students to the possibility of reduced tutorial provision on higher level courses (Thorpe, et al, 1986).
Policy Evaluation can take the form of monitoring. The Open University carries out regular surveys to monitor the financial impact of study on its students, thus gauging the effects of fee increases, changes in local authority assistance, the effects of its own financial assistance schemes, etc (Blacklock, 1982). Other survey data on the ownership of televisions, cassette recorders, home computers, etc, can assist course planning (Grundin, 1983).
Research can evaluate the impact of policy changes. A recent study at the Open University showed that the new policy of "deregistering" non-progressing students actually affected a large number of its own graduates (Heron, et al, 1986).
Finally, evaluation studies make take the form of experiments or pilot schemes. A policy option such as the admission of school-leavers to the Open University can be carried out on a trial basis and the outcome evaluated before making a final decision (Woodley and McIntosh, 1980).
- Organisational evaluation
A distance education programme or institution can be evaluated in terms of its internal organisation and procedures. Just as in any other setting this can involve scrutinising the financial management and general "organisation and methods". More specifically it can involve tasks such as the monitoring of tutors marking patterns and the turn-around time for assignments. Evaluations have also been conducted into the course team approach to distance course writing.
Google+ Web Search
Newest Members
Create a free website at Webs.com