maria joão loureiro
DESCRIPTION
The quality of peer assessment in a wiki based online context: a qualitative study in a post-graduation module. Maria João Loureiro. topics context. Coimbra. Oporto. Aveiro. context of the study - Portugal. context (U. Aveiro) created in 1973 10 to 11.000 students - PowerPoint PPT PresentationTRANSCRIPT
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
The quality of peer assessment in a wiki based online context:
a qualitative study in a post-graduation module
Maria João Loureiro
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
topics
context
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
context of the study - Portugal
AveiroOporto
Coimbra
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
context (U. Aveiro)• created in 1973• 10 to 11.000 students• courses (grad and pos-grad)
teachers education sciences, math, …engineering health…
• member of the European Consortium of Innovative Universities
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
contextPhD program in Multimedia in Education module
Distance Education (DE)
blearning regime face-to-face and at a distance group work
module aim - develop research competences related with literature reviewing
search, select, systematize, synthesize and literature related with DE;communication, collaborative work and assessment competences (self- and peer- assessment)
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Context (literature)focus on learner self-directed and collaborative learning -> increase attention to alternative/authentic/integrated assessment methods HE assessment methods are quite narrow – assessment students’ outcomes test based – students are not implicatedpeer assessment (PA) can empower students (critical thinking, reflection)reliability and validity of PA – no consensusliterature on the quality of online (web 2.0) peer assessment scarce
Boud and Falchikov (2007), Joordens et al. (2009), Li et al. (2009, 2010), Peng (2010), Rourke et al. (2008), Sluijsmans et al. (2004), Topping (2000, 2008, 2010),
Zundert et al. (2010)
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Context (literature)PA benefits:
promote higher order thinking, developing students’ critical thinking, communication, lifelong learning, problem solving… increase the amount of feedback, from teachers and from peersenhance students’ sense of ownership, responsibility and students’ motivation, since they find it useful, attractive and enjoyablesupport active and autonomous learning can avoid the involvement of free-riders in group work, i.e, facilitate the identification of individual contributions
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Context (literature)PA pitfalls:
students’ attitudes toward PA may hinder the above mentioned benefits –students’ unconfident towards PA - judging friends – increase students’ anxiety and resistant toward PAPA is time-consuming – training, preparation, and monitoring are neededPA can be subjective – students might be too hard, critical or boasting, or not used to assess their peers
PA advantages increase – formative feedback
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
course organisation and development 1st week online- Diagnosis of students’ perceptions about how to do literature
review and about “Distance Education” (individual).- Familiarisation with the module guide and discussion of the
proposed activities, including assessment (individual).- Individual readings of recommended literature and or other
materials related with theoretical and methodological issues to be taken into account to do a literature review.
- Use of the synthesis sheet – instrument developed to help students to analyse and document their readings (individual).
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
course organisation and development 1st face-to-face session - Teacher, tutor and students’ presentation and definition of work
groups (classroom). - Negotiation of the module activities and schedule and of the
assessment framework (classroom).- Session about online literature search tools (made by a
library technician).- Presentation of tips on how to do a literature review (teach).- Discussion and syntheses of the students’ readings about how to
do a literature review (groups).- Information search (groups).
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
course organisation and development 2nd weekonline- 1st ind.reflection about the ongoing activities and students’
learning- Definition of the literature review themes related with DE and of
the work plan (groups)- Formative assessment of the work plan (teachers)3rd and 4th weeksonline- 2nd ind.reflection about the activities and the developed
competences- Development of the 1st version of the literature review (gr.)- Formative assessment of the ongoing work (teachers and PA) - Revision of the paper and preparation for its presentation
(groups).
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
course organisation and development 2nd face-to-face session- Presentation, discussion and assessment (teachers
and peer summative assessment) of the groups work.- Self and peer assessment of the developed
collaborative competences (intra group) and final reflection concerning students own learning during the module (individual)
- Module assessment (individual).
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
MethodologyQualitative approach - study recent phenomena (like the quality of online PA) in their own contexts Data gathering and analysis techniques:
observation mediated by the wiki (PA was delivered by the students using the wiki) – content analysis (2 researchers, reliability 100%, external validation) inquiry (a questionnaire was applied to evaluate the module in the final face to face session) – descriptive statistics
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Criteria Indicators
Use of the negotiated criteria
Students used the negotiated criteria or words with the same significance explicitly.
Adequacy of the chosen vocabulary
Concepts are used with rigour and differentiated.
Provision of constructive feedback
criticismsPA text includes criticism (both positive and negative), pointing out aspects that could be improved, without making suggestions.
questionsPA text presents questions for reflection concerning, for example, the theme or the development of the literature review.
suggestions for improvement
PA text includes suggestions that can lead to the improvement of the literature review, like, new readings, proposing ways to systematize the corpus of the review....
Fairness of the score the PA includes a score which should be in accordance with the given feedback
Adequacy of the final considerations
A synthesis of the main positive and negative aspects of the review is presented.
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Examplesnegotiated criteria
“(…) the topics and sub-topics that underlie the work are well framed [i.e. the structure is adequate]. Its coherence and rigour are remarkable and the group demonstrates synthesis competences”.
adequacy of the chosen vocabulary “the paper does not present significant gaps in what concerns cohesion, textual coherence or even orthographic or syntactic mistakes.”
constructive feedback “the work does not include the research question that guides the development of the work”
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
%0 20 40 60 80 100
Use of the negotiated criteria
Adequacy of the chosen vocabulary
Provision of constructive feedback(criticisms)
Provision of constructive feedback(questions)
Provision of constructive feedback(suggestions for improvement)
Fairness of the score
Adequacy of the final considerations
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
0 2 4 6 8 10 12 14 16
Col. work should be assessed as a whole with the samemark for all elements
Col. work should be assessed with several components
Col. work should be assessed individually, by teachers andby colleagues
Col. work should be assessed only by teachers
The personal reflections should be published in themodule's site
Self and peer assessment of the work groups should bepublished in the module's site
Self and peer assessment of develloped competences(intra-groups) should be published in the module's site
Number of students
completely disagree disagree agree completely agree don't know
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Final thoughtsPA could have more quality (suggestions, questions, …)Results
In line with the literature – lack of quality can be explained by some of the pitfalls of PA (students’ attitudes towards PA, uncomfortable, lack of confidence, unreliability + fear of being exposed in the social web – in this case while agreement about publishing PA)
PA needs training
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Contributions framework to assess the quality of PA in web 2.0 online contexts, for some of the criteria, quality PA can be achieved reports students’ perceptions and attitudes towards the assessment of collaborative work, including PA
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
Thank you for your attention
questions???
IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
The quality of peer assessment in a wiki based online context
CIDTFF, DE, Universidade de Aveiro
coursesorganization
Ramos (2004)