The assessment of the learning outcomes of Italian undergraduates (TEst on COmpetences, TECO)
In 2012, ANVUR launched a project on the assessment of the learning outcomes of Italian undergraduates by means of a test (TECO: TEst on Competences), believing that the analysis of competence levels is an important tool to contribute to the monitoring of the quality of the education process.
TECO results are part of the educational evaluation system (Self-Assessment, Periodic Evaluation, Accreditation - AVA), as useful indicators for assessing the quality of the educational process of individual courses and universities. This activity is regulated by the Presidential Decree establishing the Agency (76/2010, Art. 3) in which a clear reference is made to the quality assessment of the outcomes of training provision as well. In particular, it is clarified that the efficiency and effectiveness of teaching activities must be assessed on the basis of international quality standards, also with reference to the students' learning outcomes and their appropriate integration into the world of work (Art. 3, para. 2(a). More recently, in Annex E in DM 987/2016, it is further clarified that the collection of new data related to transversal and/or disciplinary competences acquired by students will allow for the realisation of indicators that will be used for the periodic evaluation and accreditation of the hubs and courses of study.
For the first two trials, the Agency used the CLA+ test produced by the Council for Aid to Education (CAE) and used internationally to measure generic skills. However, both experiments, conducted in collaboration with several universities (12 in 2013 and 24 in 2015, respectively), revealed some critical issues, such as the partial validity and adaptability of the methodological framework proposed by the EWC and the high costs of the entire procedure. Therefore, the Agency in 2016 redefined the entire research design, including the reference areas, methodological approach and survey instruments for both transversal competences (TECO-T) and disciplinary competences (TECO-D).
Currently, the TECO project aims to construct indicators reflecting the skills acquired by university students in the 1st and 3rd year of three-year or single-cycle degrees at Italian universities.
TECO-T
The transversal competences assessed so far are Literacy and Numeracy. The other survey areas that are being developed are: Civics, Problem Solving and Foreign Language (English). The working hypothesis is that these competences draw on a generalist educational background and, as such, can also be developed during university studies, ensuring comparable information between universities and/or between degree courses.
Literacy tests students' skills in understanding, interpreting and reflecting on a text not directly related to a specific degree course or subject area, using two types of tests: a passage followed by 10 closed questions and a short passage in which 20 words have been deleted (cloze test) which the student has to reinsert, for a total of 30 questions.
Numeracy measures students' levels of ability in understanding and solving logic-quantitative problems, through a short passage with graphs and tables followed by 5 questions, an infographic followed by 5 questions and 15 short logical reasoning questions, for a total of 25 questions.
The Literacy and Numeracy tests were carried out by the Agency with the collaboration of a group of experts, and were validated at the end of 2016 through a Field Trial that took place in five universities (Polytechnic of Turin and the Universities of Messina, Padua, Rome Tor Vergata, and Salento). This first validation phase involved 854 students in the 1st and 3rd year of three-year degrees. The analysis of the validity of the tests showed that the individual items actually measure specific competences, and that the questions represent different levels of difficulty and are able to identify the most competent students. In the spring of 2017, the final version of the Literacy and Numeracy tests was administered to 1,759 students enrolled on the first and third years in five new universities (Universities of Bari, Bologna, Florence, Milan-Bicocca, and Palermo) from five subject areas defined by ANVUR (Biological Sciences, Education and Training Sciences, Psychological Sciences and Techniques, Economics and Health Professions). At the end of the test, the students also answered a Student Questionnaire, providing useful information on family background and personal, university and work experiences.
In both sessions (2016 and 2017), in which students participated on a voluntary basis, the tests were administered by computer via a platform managed remotely by CINECA, within computer classrooms of the universities involved, within time frames predefined by the Agency (approximately two weeks).
Two surveys involving around 700 graduates three years after obtaining their three-year degree are currently under way, conducted in cooperation with AlmaLaurea and INAPP (formerly ISFOL) respectively. The working hypothesis is to test whether, and to what extent, the TECO-T score is able to predict employment outcomes.
TECO-D
Disciplinary competences, on the other hand, are closely linked to the specific training content of the course undertaken by the student, and can therefore only be compared with courses of a similar nature. The survey is coordinated by ANVUR, which uses reference people, chosen by the disciplinary groups themselves, to construct the TECO-D tests. The project presents highly innovative aspects for the academic communities, since
- it stimulates a shared definition of the core subject content and its declination with respect to the five Dublin descriptors;
- it allows the creation of subject tests (TECO-D) whose contents are shared at national level and the results used for the self-assessment of individual degree courses, allowing inter- and intra-university comparisons within their subject groups;
- it guarantees centralised and certified management (CINECA on behalf of ANVUR) for the administration and collection of data.
ASSESSING THE LEARNING OUTCOMES OF UNIVERSITY STUDENTS (TESTS ON COMPETENCIES or TECO)
Assessing the learning outcomes of university students (TEST on Competences, TECO)
ANVUR TESTS THE TRANSVERSAL COMPETENCES OF 15,000 STUDENTS
Anvur tests the trasversal competences of 15,000 students
