Most online study tools use rigid multiple-choice, matching, or short-answer quizzes to provide feedback for which questions you answered “right” or “wrong”. But all those do is measure your ability to recognize a correct answer, rather than actually think of one in your head (like you would need to in the real world).
What if your readiness for a test (or other type of performance) could be better measured using varying degrees of right and wrong for each concept? What if a study tool could let you determine your level of readiness for each exam concept, by simply trusting you to answer the question: “How well did you know this?”
What research tells us about student self-assessment
Such self-assessment is precisely the principle behind Brainscape’s application of the metacognitive concept known as Judgment of Learning (or JOL). Asking learners to rate their JOLs on an item-by-item basis is an extremely valuable instrument in education, as it helps learners more accurately identify their strengths and weaknesses while helping educators focus on areas in which their students are least comfortable. It is also a critical instrument in educational software algorithms that use confidence-based repetition.
Over the past few decades, researchers have used various experimental formats to ask learners to rate their JOLs: either a series of confidence options (e.g. 1-5), a sliding confidence scale (0-100%), or a simple inquiry into whether the learner wishes to study the question again in the future (yes/no).
Researchers have also differed greatly in terms of how such JOLs are incorporated into a question list: either asking for the user’s self-assessment before each answer is revealed, after each answer is revealed, or after all answers in an entire list have been revealed. The important common thread is that each of these types of experiments engages the learner’s metacognitive processing.
How Brainscape uses self-assessment
Brainscape uses a bold application of such metacognitive processing as its JOLs actually replace, rather than supplement, computer-provided feedback.
For example, if Brainscape asks the learner “What is the capital of Canada?”, the user does not actually have to type or select an answer but simply reveals the back side of a virtual flashcard. The true student self-assessment comes when the user rates her own confidence on each flashcard, which Brainscape’s algorithm then uses to determine how soon the flashcard should be displayed again. (Low-confidence items are displayed more frequently until the user reports higher confidence.)
This shift of the assessment burden from the computer to the learner prompts three important questions:
- How do we know that the learner will accurately assess her knowledge?
- How do we know the learner won’t just start revealing answers without genuinely thinking about them beforehand?
- What happens if a learner incorrectly reports high confidence in an item they actually don’t know very well?
Is self-assessment accurate?
In an answer to the first question, the Brainscape team reminds skeptics that adults are actually quite accurate in assessing their JOLs to begin with. Considering that Brainscape exists as a personal study tool, not an external assessment mechanism, Brainscape users have no motivation to “cheat” as they might have on an actual test or graded assignment. Brainscape instead helps users manage their own knowledge and learn to become more honest with themselves.
To the second question, we remind readers that the alternatives to the flashcard approach are sub-optimal for time-pressed learners who are seeking a convenient and effective study experience. Multiple-choice and matching questions are generally considered inferior for learning since they merely test recognition rather than engaging active recall, while short-answer and fill-in-the-blank questions are not conducive to quick study sessions as they involve time-consuming typing effort (particularly on the mobile phone), which is especially bothersome for users who already know the answer confidently.
Brainscape chooses to present questions as simple flip/reveal flashcards in order to increase the number of repetitions that can be achieved in a given span of time. Research shows that the number of memory retrieval attempts, and the optimal spacing between them, are among the most important determinants of the strength of the memory encoding. We believe that the benefit of increased exposures outweighs the costs of potential “zone-out”—especially when the learner is already motivated to concentrate on learning the topic at hand.
What about overconfidence?
Finally, regarding the possibility of overconfident JOLs, we should consider this a benefit rather than a risk of Brainscape’s approach, as the eventual correction of misjudged JOLs can often yield better retention benefits than if the confidence was never misjudged in the first place.
Butterfield and Metcalfe indeed show that people are actually more likely to remember a corrected wrong answer if they had previously exuded high confidence that their submitted answer was correct and subsequently learned that it was wrong.
According to this logic, if a Brainscape user fails to recall a target displaying a previously high confidence ranking, she is likely to devote more mental energies to correcting the error. Barrick and Hall show that such error corrections are even more beneficial when items are spaced out rather than massed together.
In fact, in a spaced or expanding study environment such as Brainscape, even a systematic display of overconfidence is unlikely to hinder the user’s progress. While Meeter and Nelson demonstrate that a systematic confidence bias has no effect on the relative proportions of items in each JOL category, Pashler and colleagues confirm that “the cost of overshooting the right spacing is consistently found to be much smaller than the cost of having very short spacing” (p. 5). Brainscape is therefore rather immune to users’ potentially poor study skills—and can actually improve your metacognitive skills over time.
Whatever software or technique you are using to help manage your study, applying student self-assessment practices will consistently improve your performance on subsequent tests of your knowledge. Remember to trust yourself, because you are the best judge of how well you know something!
[Trying to figure out how to study most effectively? Learn how to develop solid study habits to optimize your learning.]
Sources
Bahrick, H., & Hall, L. (2004). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52(4), 566-577. https://doi.org/10.1016/j.jml.2005.01.012
Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. London: King’s College.
Butterfield, B. & Metcalfe, J (2006). The correction of errors committed with high confidence. Metacognition and Learning, 1(1), 69-84. http://dx.doi.org/10.1007%2Fs11409-006-6894-z
Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of Memory and Language, 33, 545– 565. https://doi.org/10.1006/jmla.1994.1026
Janiszewski, C., Noel, H., & Sawyer, A.G. (2003). A meta-analysis of the spacing effect in verbal learning: Implications for research on advertising repetition and consumer memory. Journal of Consumer Research, 30, 138–149. https://doi.org/10.1086/374692
Lovelace, E. (1984). Metamemory: Monitoring future recallability during study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10(4), 756-766. https://psycnet.apa.org/doi/10.1037/0278-7393.10.4.756
Meeter, M. & Nelson, T. (2003). Multiple study trials and judgments of learning. Acta Psychologica, 13(2), 123-132. https://doi.org/10.1016/S0001-6918(03)00023-4
Nelson, T. O. & Leonesio, R. J. (1988). Allocation of self-paced study time and the “labor-in-vain effect.” Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 676–686. https://psycnet.apa.org/buy/1989-14397-001
Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing instruction and study to improve student learning. Institute for Educational Sciences practice guide, U.S. Department of Education. https://eric.ed.gov/?id=ED498555
Son, L. (2004). Spacing one’s study: Evidence for a metacognitive control strategy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(3), 601-604. https://psycnet.apa.org/doi/10.1037/0278-7393.30.3.601
Squire, L.R. (1992). Memory and the hippocampus: A synthesis from findings with monkeys and humans. Psychological Reviews, 99(2), 195-231. https://psycnet.apa.org/doi/10.1037/0033-295X.99.2.195