Do participants perform better at the beginning of the semester?

Due to the fact that journals mostly publish positive findings, researchers might feel that, when they are unable to replicate an effect, there was something wrong with their own study. The preliminary results might have gone in the right direction, but when data collection was completed the effect was no longer there.

These researchers might feel that the disappearance of the effect during the course of the experiment may have been caused by unmotivated participants in the latter half of the study. These suspicions may be supported with some anecdotal evidence. Everyone who has conducted a couple of studies has at least one story about a participant taking a telephone call during the experiment. A good friend told me once about a participant who thought it was appropriate to bring her pet snake to the study. Moreover, due to the recency effect, researchers are more likely to remember instances of unmotivated participants from the second half of the semester.

Such end-of-semester findings in combination with causal observations of unmotivated participants might lead to the feeling that participants perform better at the beginning of the semester than at the end of the semester. However, is this really the case?

Rolf Zwaan argues on his excellent blog (link) that this feeling might be based on a fallacy. Some observations that participants perform better at the beginning of the semester than participants at the end of the semester might only exist, because a person has examined the results halfway through the experiment.

If the effect is not found after the initial data collection, researchers might decide to terminate the study. This failure to replicate is then unlikely to be attributed to the quality of the participants. However, if the effect has been found or if the preliminary results are going in the right direction, the researchers might decide to continue and collect more results. If the effect is then found consistently throughout the semester, the researchers are unlikely to report in their manuscript when the results were collected. However, if the effect disappears in the second part of the semester, the researchers might attribute this failure to replicate to changes in the quality of the participants.

This fallacy highlights the need to base the decision about the sample size on power analyses before one starts to collect data and to pre-register the experiment and the goals of the experiment. The first two recommendations would prevent decisions to terminate studies prematurely and reports of post-priori findings of end-of-semester effects.

One of the studies that have set out to examine the possible influence of the time of semester is Nicholls et al. (2015).* They examined two factors: Time of semester and reward (credit and paid). There were 80 participants. Half of them participated at the beginning and the other half participated at the end of the semester. Furthermore, half of each group received course credit, whereas the other half of each group were paid for their participation.

Participants completed the Sustained Attention to Response Task. For 360 trials, they had to decide whether a quickly masked digit had been the digit ‘3’. If the digit was not ‘3’ (about 89% of the trials), they had to press a button as quickly as possible. If the digit was ‘3’ (about 11% of the trials), they had to withhold their response altogether. After the sustained-attention task that took about 7 minutes, participants’ intrinsic and extrinsic motivation was measured with the Student Work Preference Inventory.

Despite the sustained-attention task only taking 7 minutes, credit and paid participants’ performance had different trajectories across the semester. Whereas the performance of paid participants slightly improved between the beginning and the end of the semester, the performance of credit participants slightly worsened. Although neither effect was significant, the interaction between time of semester and reward was. Furthermore, it was found that, whereas paid participants did not differ in motivation throughout the semester, credit participants had more intrinsic and extrinsic motivation at the beginning than credit participants at the end of the semester.

Another study was recently conducted by the Many Labs project organized through the Open Science Framework (link). In the study, 20 different laboratories participated. In total, there were 2696 participants, who were presented a series of 6 questionnaires and 10 tests, measuring data quality, individual differences and known experimental effects. The entire study took less than 30 minutes to complete. Whereas the project found no end-of-semester effects on the experimental results, it found weak effects on data quality measures and several individual differences. The results of the project suggest that performance does not but motivation can decrease during the semester.

However, the study of the Many Labs project did not really seem to put participants’ motivation to the test. The consequences of poor motivation may only become apparent when the task is long and repetitive. The study of Nicholls et al. (2015) had a relatively short, but quite repetitive experiment (i.e., 7 minutes), whereas the Many Labs project had much variation (i.e., 16 questionnaires and tests in just 30 minutes). Taken together, the results of these studies therefore seem to suggest that, if your experiment is short or varied, motivation might not really be an issue, but, if your experiment is long and repetitive, motivation might still become an issue later in the semester.

References

Nicholls, M. E. R., Loveless, K. M., Thomas, N. A., Loetscher, T., & Churches, O. (2015). Some participants may be better than others: Sustained attention and motivation are higher early in semester. Quarterly Journal of Experimental Psychology, 68, 10-18. doi: 10.1080/17470218.2014.925481

*Disclaimer: Mike Nicholls and his co-authors are colleagues. They work, like me, at the School of Psychology of Flinders University.

Do participants perform better at the beginning of the semester?