AssessmentProblem

[Working notes, with cloned or borrowed materials that still need to be organized]

The assessment problem

Wikis are not well suited to projects that require specified authorship or protected documents. Wikis are authored by communities, not individuals.

Tracking work created in wiki spaces can become a logistical nightmare, Attribution of individual work can be difficult, and an environment in which students (or even nonstudents) are invited to rework content further complicates matters. Seemingly minor contributions to a collaborative document may have major effects, effects that may be near impossible to assess fairly or even to detect. Brian Lamb

As with the security issue, most of the pedagogic dilemmas presented by wikis can be addressed by “traditional” management approaches. For instance, students may be required to sign or identify any work that they author. Bowdoin College’s Romantic Audience Project asked participating students to avoid making changes to text written by others (poets or fellow students) and to limit themselves to the addition of related links. Instructors may choose to establish categories, topics, and other prompts to direct student participation into more orderly channels.

Difficult evaluation of individual achievement

Participation in the Swiki cannot be expected to be uniform. Scharff, for instance, indicates that in his course “All students participated, but some individuals contributed far more than others. The top three posting students (25% of the class) were responsible for 46% of the total edits.” Scharff

Standard assessment practices tend to focus on individual achievement. We worry about how individuals will fare and (particularly at the collegiate level) think little about how the group works together. Group-level thinking may be even more rare in the process of assessment: we judge our own success and the success of our students largely by how each has performed independently, rather than focusing on how well each one has done within the interactions of the group, or how the group itself has progressed as a whole. This can create problems both for the implementation of the emergent approach and for the assessment process itself, which needs to reflect not only individual measures of achievement but also measures of interpersonal and group function. Emergent Pedagogy

It is important to remind ourselves that individuals benefit in important ways from positive group functioning. Fundamental to the emergent approach is the idea that individuals are altered by their interactions with others. A group that engages in significant interactions increases everyone’s learning opportunities. Functioning productively in a group is also a skill that individuals can carry forward to new educational environments, creating future valuable experiences.

Learning Outcomes that are tricky to evaluate (What is essential is not an outcome but a developmental process)

Conventional assessment also focuses on how well students have mastered (that is, can report back) particular content. The learning objectives of an emergent approach have less to do with content than with process, growth and development. The specifics of what is taught are often secondary to the acquisition of learning methods and processes of inquiry. Emergent Pedagogy

Some of the most exciting outcomes may be particularly tricky to evaluate for another reason. The satisfying paradox of the emergent approach is that it facilitates both independent and collaborative thinking, teaching students to initiate and sustain their own learning through interactions with others who enrich and stimulate their learning environment. But such important and valuable outcomes are difficult to assess, particularly over such short time spans as the duration of a class. They are likely to be most evident across greater periods of time, and may only be apparent in future behavior, rather than in the particular product of any given class session.

For instance, one of the most important and positive outcomes of our Summer Institute came in the form of an increase in positive affect and self-esteem in our participants. Many of them remarked on how important it was to them that they were treated as authors of their own learning. They were impressed that we recognized their contributions to our interaction. Being treated as valuable colleagues may have been for them the most important and long-lasting outcome of the institute. As one teacher remarked,

"When I am working with other professionals, using the resources of a college or university, most importantly the staff of those institutions, when I feel valued as a teacher and as a human being, I am motivated to try a little harder. And the results–those free ideas–are priceless.”

Many of the participants mentioned "collaboration," "connecting linkages," and "discussing our thoughts with our peers" as significant aspects of the institute. Teachers who feel empowered by this form of respectful interaction can return to their own classrooms more able to empower their students to engage in the same form of respectful interaction. As one of them reflected on the last day of the institute,

Ways that assessment can be modified to bring it into greater alignment with the constructivist approach. It seems particularly important not to give up goals or objectives, but rather to see them as flexible and open to constant evaluation.

Assessment in an emergent system should also be multi-dimensional; it is not reducible to a single rubric or axis. Indeed, the evaluator may even have difficulty articulating the standard used to judge progress. (We have all had the experience, as instructors, of "knowing it when we see it.") A chief argument against this tacit approach to grading is that it is somehow unfair or mysterious.

But even our most "objective" way of evaluation entails a tremendous amount of subjectivity. It is our Emergent Pedagogy experience that when this type of subjective evaluation occurs within the context of a rich process of dialogue, trusting interaction and openness to input, both faculty and students are more than satisfied, because they have mutually authored a shared tacit understanding of the work they have done together.

Lack of firm goals to aim for

It is also important to recognize that some students will be uncomfortable without firm goals and a definite plan of action. These students may benefit from the establishment of intermediate goals, and they may be helped by periodic reflections on where they have come from and where they are, even if they cannot know, ahead of time, exactly where they are going. Emergent approaches that emphasize the moment of local interaction may sometimes make it difficult for students to see the bigger picture, which may make it hard for them to mark their progress. Interim reflections can be important moments to highlight progress for both those students and their teachers. Emergent Pedagogy

Difficult evaluation of individual contribution (No notion of authorship)

The use of web statistics is not straightforward. Web statistics record every time a visitor accesses a page - known as a hit. The problem is that if a visitor just goes from page to page they record a number of hits.

Most wikis don’t make it possible for the students’ identity to be known and each user’s contribution can be tracked. Individual contributions cannot be identified with certainty, and it is possible for one user to modify or delete another user’s posting

Peer-review marking.

=from Matt barton It is possible to extend the protection in such a way that no one but the individual student can modify her wiki page. This would successfully offer students protection and a feeling of authorship. However, this is making a particularly poor use of Wikis, as it disables the very features that make wikis exciting in the first place. Other tools would like blogs be more appropriate to host that any content that require restricted authorship authorizations. Therefore, quit trying to make wikis do what you could do under the old paradigms, and try instead to think of ways to use "pure wikis" effectively.

Ones to note pedagogy - "Attribution of individual work can be difficult, and an environment in which students (or even nonstudents) are invited to rework content further complicates matters. Seemingly minor contributions to a collaborative document may have major effects, effects that may be near impossible to assess fairly or even to detect." To truly empower students within collaborative or coconstructed activities requires the teacher to relinquish some degree of control over those activities. The instructor s role shifts to that of establishing contexts or setting up problems to engage students. In a wiki, the instructor may set the stage or initiate interactions, but the medium works most effectively when students can assert meaningful autonomy over the process. It s not that authority can t be imposed on a wiki, but doing so undermines the effectiveness of the tool" Copy and paste

Recommendations

Open final project deliverables. Students were told to create projects which future students could build upon. Final deliverables were not a report or a summary of the project, but a public deliverable of the project work itself. That is, the result was a reusable resource. The kind of resource varied from project to project, but project deliverables were expected to provide sufficient content that other people could understand what they did, and, more importantly, build upon it.

It is not proven that collaborative learning situations are not effective at preparing students for the kinds of tests being used in the current environment. Adapted from Emergent Pedagogy

Protections are always kind of a pain." Authoring software, permissions, or passwords are typically not required. Brian Lamb. Wide Open Spaces: Wikis, Ready or Not

The quantitative evaluation of the Swiki helped provide support for some of the qualitative (and informal) observations about the class. In addition to these usage data, we also relied on several questionnaires (presented to the students throughout the semester), evaluation of project presentations and deliverables, and personal communications with students. Scharff.

Independent research project was a team project where students were expected to research a topic of interest, write a brief report, and present their research during class. The course project was also a team project described in the previous section. The students formed into three teams of 2, 4, and 5 members. Although it was neither necessary nor required, the group project and independent research project teams were the same. Scharff

Do not sacrifices your learning goals for easier assessment

We suspect that there are very real savings to be achieved by creating classroom environments in which students are encouraged to take greater personal responsibility for their own education and the education of those around them. We are quite certain that students learn better when they are encouraged to master material in a context where that material is relevant to their own interests. In short, it might well turn out that emergent pedagogy is the optimal route to better student performance, even in the current realities of economics and assessment.

As this is a group exercise, assess progress at the level of the group

Finally, we recommend adopting goals and assessing progress at the level of the group for both teachers and students. Success or failure may be judged, at least in part, by how well the group interacted and progressed. It also may be useful to invite students to reflect on their participation in the group. They may need help in seeing the value in saying things that are not thought out, of exposure to perspectives different from their own, of relying on others, and of feeling responsibility toward others in turn.

What can be easily evaluated is not material that encourages deep reflection

On-going, in-the-moment reflections on learning may also provide a richer forum for assessment than more traditional end-of-process, product-focused assessment. In the Summer Institute on Emergent pedagogy participants were regularly asked to reflect on what they were learning, where the group was, how they were responding to the process, how they saw the institute affecting their future behavior. Such queries provoked deep reflections and rich dialogues about what was going on. In sharp contrast, when we asked participants to assess the institute at its conclusion, using a traditional rating scale and open-ended questions, we received very high "objective" marks, but very little feedback that was particularly meaningful. Participants observed only that "it was interesting," or "I learned that people have unique viewpoints." Emergent Pedagogy

Evaluation of the developmental process (improvement) rather than assessment of retention of particular content.

Evaluation should include some measures of actual behavior, although the effects of learning on behavior may only become evident with the passage of time. Assessment should not be limited to the time frame of the learning experience itself, but should extend to some reasonable future time. Emergent Pedagogy

This approach is consistent with a current movement known as "dynamic assessment," in which the most accurate evaluation of students occurs through the observation of their learning process. In this type of approach, students are given some sort of baseline measure, and then taught a new skill. Assessment involves an ongoing evaluation of how students respond to instruction and, more importantly, how they apply this newly acquired skill to a new problem (Lidz and Elliott).

More on Dynamic assessment:

• http://www.icelp.org/asp/Dynamic_Cognitive_Assessment.shtm
• Kanal, N. (xxxx). Intelligent tutoring and dynamic assessment for education and training. =on-line:
• Learning through rich assessment and mediation in a university ... =on-line:

Self-evaluation

Throughout the semester, email traffic to the course mailing list and incremental changes to the Swiki were logged. The Swiki kept a complete history of every page created and changed. Anyone could read the Swiki, but students had to log in to make changes, so all pages were tagged by the associated creator / editor. Scharff