A five-dimensional framework for the integration of AI in authentic assessment

creativecommons.org

Deed - Attribution-NonCommercial-NoDerivatives 4.0 International - Creative Commons

You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .

Where "are" you at the moment?
  • Assessment with AI: GenAI is a technology to support assessment:
  • To assist on the design of the exam task
  • Rubric creation and helping during the marking process
  • Digital assistant to help students in the exam task
  • Assessment of AI: The assessment, whether explicit or implicit, of the AI competencies/AI literacy required to use Generative AI to support learning and teaching
  • Assessment for AI: Assessment is reconceptualised to build up the AI competencies of students in addition to assess their academic knowledge, competencies and skills
What is AI literacy?
The critical awareness of the possibilities, limitations, socials and ethical challenges that AI models bring to society.
In the context of HE, AI literacy is an 'umbrella term' that encompasses:
  • Technical knowledge and digital competencies/skills (information literacy, data literacy, etc.)
  • High-order cognitive processes (critical and creative thinking, problem formulation, self-reflection, academic argumentation, etc.)
  • Critical discourse around AI ethics and human-centred values, regulatory frameworks, impact on copyright and data protection, explainable AI, etc.
(Garcia Vallejo, 2024)
What are AI Competencies?
AI competencies refer to the skills and abilities necessary to guide educators and students in understanding and using AI ethically, safely, and responsibly (Miao and Cukurova, 2024)

UNESCO's AI competency framework for teachers
What is authentic assessment?
Authentic assessment aims to link what is learned in the classroom with the activities and responsibilities of the professional world, replicating the types of tasks, performance standards, and competencies that students will encounter in their future careers (Villaroel et al., 2018).
It involves learners adopting an active role either as problem-solvers (assessment for learning) and/or co-creators of the assessment task (assessment as learning).
A five-dimensional design framework for integrating GenAI in authentic assessment
1
Context
Professional and social context
Transferable skills and professional competencies
Students' maturity and motivation
Students' assessment literacy
2
Designing the task
What: is the gradable/quantifiable/qualifiable problem/task/project to be undertaken?
What: would be an acceptable solution?
How: the solution should be provided
When: the solution should be provided
Who: should provide the solution?
3
AI integration
Why: AI should be used (intended pedagogical purpose/s)
Which: AI app should be used ?
How: students should be using AI to solve the task
How: students should NOT be using AI to solve the task
4
Evaluation (rubric, criteria, score)
What: is being assessed (soft skills, professional competencies, AI competencies)
How: is being assessed (rubric, assessment criteria, grading scale)
Who: is assessing it (instructor, learner, peer)
When: feedback and grades will be provided to students
5
Providing feedback
What: type of feedback will be provided to students (formative, summative)
How: feedback is designed ("one-off" product, ongoing dialogue)
Which: areas of the agreed solution should receive feedback (evidence of AI competencies, professional standards, academic knowledge)
Who: will provide feedback (assessor, peers)
Step 1: Considering the professional and social context
Professional Context
Understand the industry environment where students will apply their knowledge. Focus on developing transferable skills like teamwork and problem-solving that are valuable in the job market.
Social Context
Consider students' maturity, motivation, and assessment literacy.
There is also a social context in which the task is situated: the assessment should mirror the social processes of an equivalent real-world situation.
Step 2: Designing the Assessment Task
What is Required?
Specify the task type: research project, consultancy report, or prototype development.
Tasks should address real-world problems relevant to students' future careers.
Define a task that is "gradable", quantifiable and/or qualifiable.
AI can assist in designing the assignment.
What outcome/result/solution is expected?
What would be an acceptable solution to the task?
Use transitive verbs with a specific direct object (e.g., "write a report", "produce a video", "record a podcast", etc.).
Specify the format in which the solution should be provided (file format, number of words, software, etc.).
When is it Due?
Provide a detailed schedule with deadlines and intermediate deliverables.
Help students manage time and structure their workload effectively.
Who would provide the outcome/solution?
Determine if assessment is individual or group-based (group projects foster collaboration and help manage assessment workload in large classes)
Step 3: Integrating the "AI dimension"
  • What is the intended purpose for incorporating GenAI in the task?
     - Assessment with AI?
     - Assessment for AI?
     - Assessment of AI?
  • Which AI competencies/skills will be developed through the assessment?
  • Which cognitive domains can AI help to develop?
  • Which AI technologies will be allowed?
  • What instructions, guidance, or recommendations should students be given for an ethical/permitted use of AI to support the task?
AI to support cognitive domains
Step 4: Grading and Evaluation Criteria
  • What is being assessed?
  • Who will be assessing it and/or providing feedback: instructor, peer student, industry sponsor, AI acting as "critical friend"?
  • How is it being assessed? (Human supervision, AI software)
  • When is it being assessed?
Soft/transferable skills
Teamwork, communication, leadership, problem solving, etc.
Discipline-specific competencies
Mastery of technical or professional knowledge and competencies aligned to the academic discipline and professional areas.
AI competencies and literacy
Assessing interactions with AI to support high-order cognitive processes (PAIR framework, EU Digital Innovation Hub AI competencies framework).
Grading criteria
Use well- defined rubrics and discuss/agree assessment criteria with students.
Step 5: Feedback Process
1
How feedback is designed
A fixed product delivered to students
An unfolding dialogue between assessor and student
2
When feedback is provided
"Feed-forward": Timely and relevant
Formative feedback: Focused on improvement, not on grades
3
The channels where feedback is provided
Email, online, in person, generic feedback through the VLE
4
AI as a feedback assistant
AI can help personalising feedback for large classes
Guidance on how to use GenAI to enhance learning and assessment
https://maricruzgarciavallejo.substack.com/

maricruzgarciavallejo@gmail.com
References
  • Garcia Vallejo, M. C. (2024) 'Cómo entrenar tu dragon: developing a module to build up AI literacy of HE lecturers', Journal of Learning Development in Higher Education, (31). https://doi: 10.47408/unknown link31.1278.
  • Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A Five-Dimensional Framework for Authentic Assessment. Educational Technology Research and Development, 52(3), 67–86. https://doi.org/10.1007/BF02504676
  • Messier, N. (2024) Authentic assessments. Available at: https://teaching.uic.edu/cate-teaching-guides/assessment-grading-practices/authentic-assessments/ (Accessed: 12 September 2024).
  • Villarroel, V., Bloxham, S., Bruna, D., Bruna, C. and Herrera-Seda, C. (2018) 'Authentic assessment: creating a blueprint for course design', Assessment & Evaluation in Higher Education, 43(5), pp. 840-854.
Made with