My story
When I was an eight-year-old boy in Brazil, my first computer, a TK95, was given by my grandfather. With that computer, I used BASIC to create a simple software: a math test that gave participants a grade in the end, which I submitted to a science fair. I can still remember the long queues of young boys and girls in the fair waiting to solve the problems posed by the test. Little did I know that I would be back to the world of computerized testing in my adult life, but for the assessment of medical students.
Given my preference for Biology at high school, I decided to apply for the nearest medical school. At the beginning of the second year, a very challenging Biochemistry assignment forced me to visit the local Poison Control Center, which had at that time a very compelling slogan on its door: "One information can save lives." By the end of the second year of medical school, I was giving shifts as a volunteer at the Center. I quickly noticed the importance of occupational and environmental neurotoxicology and the critical need of using neurobehavioral test batteries in the early detection of subtle signs of neurotoxicity. It was also where I did my "scientific initiation" with toxicoepidemiological studies on pesticides. Because of my interest in neurotoxicology, I purchased my first book on Psychometrics while still in the clerkship years of medical school. I had to understand the papers that used factor analysis and item response theory to validate the neuropsychological tests used in the clinical evaluation of patients exposed to neurotoxic agents.
I was meant to follow a career in the field of Medical Toxicology. After my graduation in Medicine, I specialized in Occupational Medicine. Two years later, I obtained a master degree in Psychology with the cross-cultural adaptation of a computerized neurobehavioral test battery I intended to use in Brazilian patients. One year later, I was invited to work at a medical school as a tutor for problem-based learning sessions and a Pharmacology lecturer.
As a teacher at this medical school, I realized that the psychometric analyses I saw being applied to the analysis of data from neuropsychological test batteries could be applied in evaluating the tests used in the assessment of students. After detecting problems with the reliability and validity in some of the tests used in that school, I understood the importance of using psychometrics to enhance the quality of the assessment of students. After the analysis of some medical knowledge tests, I became part of the assessment committee and head of the progress testing program at that institution. The idea of measuring long-term retention instead of retrieval of short-term memorized facts with progress tests seemed quite lovely, but I stumbled upon a big problem: reliability was low for students in the early academic years. This finding was observed in other progress tests worldwide, and I started to investigate the potential application of computerized adaptive testing to progress testing. My initial idea was to increase the reliability of test scores, particularly for students in the early academic years.
After an online course given by Nathan Thompson from ASC and a short visit to Cito in the Netherlands, I convinced the Dean of the medical school and the rector of the university to authorize me to make a pilot application of a computerized adaptive progress test to the undergraduate medical students of that institution.
But I was not happy with myself as a medical teacher. I felt students weren't learning as well as they should, and I realized I failed to provide helpful feedback to learners. I had received only a concise training about problem-based learning, and I decided I needed to either seriously professionalize myself or abandon teaching for good. After talking to some senior colleagues who had gone to Maastricht for the Summer Course on problem-based learning, I decided to take the same course.
I fell in love with Maastricht at first sight. After completing the Summer Course, I was able to apply the techniques I had just learned to better communicate with students and become a more effective instructional designer. During this course, while sitting on the bench of a park, I talked to myself: "it would be good if I could work here." Less than three years later, after presenting the findings of the first pilot of a computerized adaptive progress test at the Ottawa Conference held in 2012, Professor Cees van der Vleuten invited me to work in Maastricht, exactly where I wanted, doing what I wanted: working in the application of psychometrics and computerized adaptive testing in the assessment of medical students. I will always be grateful to him for this opportunity and I will always be grateful for the skills I acquired at the School of Health Professions Education and its wonderful team of teachers and researchers, who eventually became my colleagues.
In 2015, Professor van der Vleuten invited me to work as a psychometrician for the European Board of Medical Assessors (EBMA). I kept working 50% of the time for Maastricht University, where, among other tasks, I coordinated the development and implementation of the progress test for their international track of Medicine, together with Mascha Verheggen. The other half of the time I worked for EBMA, helping spread the computerized adaptive progress test to other countries. The first country was Mexico, where Monterrey Tec pioneered the use of computerized adaptive progress testing for a large number of students in all their campi. After the results from Mexico were out, it caught the attention of Jussi Merenmies, from the Faculty of Medicine of the University of Helsinki.
It was in Helsinki that I realized that the use of an adaptive algorithm for progress tests was much more important than just increasing the reliability of test scores. After a lecture at their beautiful Biomedicum building, Ms Auli Pasanen came to me and told me what she liked the most about this technology: by adjusting the difficulty of the test it brought what Vygotsky called "zone of proximal development" i.e., "where learning happens" to the realm of assessment, even if in a somewhat rough manner.
More European institutions started to adopt the computerized adaptive international progress test made available by EBMA. After the adoption of the test in Helsinki, Maastricht decided to pursue its own computerized adaptive progress test, aimed to its international track of Medicine. With the help of David Magis, co-author of the main textbook on computerized adaptive testing, his adaptive testing algorithm was incorporated into the testing system that was being developed in Maastricht at the time. With the help of David Magis, Dario Cecilio-Fernandes and Jeroen Donkers, the large item bank of items from the Dutch interuniversity progress testing consortium could be calibrated using an ingenious approach. David Magis contributed with the customization of his open-source catR package for progress testing specificities. Mr. Cecilio-Fernandes advised for a test equating approach using persons as anchors. Mr. Donkers executed the complex equating and added a correction for the seasonal effects of the four progress tests taken annually. Meanwhile, I was responsible for coordinating the committee responsible to adapt the items and curate the item bank. More than 2500 items were made available for this initiative.
Shortly thereafter, under the auspices of EBMA, eight European medical schools joined forces in a parallel initiative, the Erasmus+ project entitled "Online International Adaptive Progress Test," in which approximately 1500 items were written and reviewed according to a detailed blueprint revised by all partner institutions. Even though the continuation of the project was hindered by the pandemic in 2020 and 2021, the team was capable of delivering the tests to a large number of students with very good results, which should be published soon.
The latest chapter of my story started in 2019, just a few months after I became a naturalized Dutch citizen. Knowing about my expertise in adaptive testing, a colleague from Maastricht University called me on a cold Friday night. He was attending a hackathon, Dutch Hacking Health. In this event, a brave young lady, Ms Irene Gielen, who had already won a previous edition of the event, presented her pitch. She wanted to develop an online tool that could provide adaptive exercises to patients with aphasia. I initially thought that the adaptation of the difficulty of the exercises would be enough. But she corrected me: what she needed was a tool that adapted the exercises to the type of deficit, not the difficulty.
I realized that the adaptive testing algorithm I had been using so far, based on item response theory, would be insufficient. Then I recalled my readings about cognitive diagnostic modelling, and its unique features: it could model the intrinsic multidimensionality of complex items and provide a customizable, flexible test blueprint, a request Prof. van der Vleuten had asked me to keep an open mind about a few years earlier. This fortunate encounter with Ms Gielen also reminded me of my talks with Prof. Lambert Schuwirth, who, together with Prof. van der Vleuten, had always warned me about the reductionism of the hegemonic psychometric approaches, which tend to discard a large parcel of the test score variance as measurement error.
Our team did not win the competition, but second place on the national contest honourable mention by the jury. We decided not to open a start-up, but to create a foundation, the Stichting Aphasia.help. The Stichting started to submit project proposals to grant agencies and is looking for volunteers and partners for the development of the tool envisioned by Ms Gielen, to whom I will always be grateful for her role in motivating me to pursue further studies on cognitive diagnostic modelling.
From September 2012 until November 2022, I held a position as assistant profesor at Maastricht University, where I had many educational duties for graduate and undergraduate students, and where I can do some exciting research, particularly on the validity of several assessment tools and questionnaires, and the evaluation of different item types, such as the increasingly popular Very-Short Answer Questions (VSAQs).
In the Netherlands, it is common to hear "Meten is weten", which can be translated as "Measuring is Knowing." Sometimes I wonder if medical teachers know how well they measure what they do and even if they know what they are genuinely measuring. After learning the limitations of the psychometric rationale, some people advocate for a "post-psychometric era." I insist to disagree with this idea. After learning the boundaries of the psychometric rationale and working to find more appropriate approaches to the use of Psychometrics, I prefer to say "Beter meten is beter weten." In other words, with better measurement, we can make better judgements about the competence of our medical students and doctors.
Now I have decided to take a radical new step in my professional life. After my father, my mother and my sister got sick, it became very difficult for me to honor my commitments to Maastricht University. Despite my love for Maastricht and the fact that I had become a naturalized Dutch citizen, I had to find a way to stay closer to home. Ironically, paradoxically, the best way I found to return to my Brazilian roots was to serve a larger parcel of persons worldwide, in a less restrictive, more inclusive way than what I would ever be able to do in Maastricht. That is why I have joined forces with Bruno Figueiredo Damásio, Wellington Field Costa, and a team of incredible psychometricians and designers to create the short-lived Psychometrics Academy, an initiative designed to make Psychometrics and Quantitative Data Analysis more accessible to academics and non-academics, no matter where they are in the world. The work with the Academy has been very rewarding and motivating, so I can suspect we are on the right track. After a successful start, we decided to part ways, so I can continue this journey independently, with my own personal touch. Stay tuned! This was just the beginning!