Abstract: Volunteer-based crowdsourcing depend critically on maintaining the engagement of participants. We explore a methodology for extending engagement in citizen science by combining machine learning with intervention design. We first present a platform for using real-time predictions about forthcoming disengagement to guide interventions. Then we discuss a set of experiments with delivering different messages to users based on the proximity to the predicted time of disengagement. The messages address motivational factors that were found in prior studies to influence users’ engagements. We evaluate this approach on Galaxy Zoo, one of the largest citizen science application on the web, where we traced the behavior and contributions of thousands of users who received intervention messages over a period of a few months. We found sensitivity of the amount of user contributions to both the timing and nature of the message. Specifically, we found that a message emphasizing the helpfulness of individual users significantly increased users’ contributions when delivered according to predicted times of disengagement, but not when delivered at random times. The influence of the message on users’ contributions was more pronounced as additional user data was collected and made available to the classifier.

Citation: Avi Segal, Ya’akov Gal, Ece Kamar, Eric Horvitz, Alex Bower, Grant Miller. Intervention Strategies for Increasing Engagement in Volunteer-Based Crowdsourcing. International Joint Conference on Artificial Intelligence (IJCAI), New York, USA, July 2016.

Download: http://bit.ly/2jh9fgG

A preliminary of this work was presented at HAIDM 2014. The 2014 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

This work was the winner of the “Best Student Paper Award” at the Seventh International Conference on Educational Data Mining (EDM 2014).

Abstract: The growing prevalence of e-learning systems and on-line courses has made educational material widely accessible to students of varying abilities, backgrounds and styles. There is thus a growing need to accommodate for individual differences in such e-learning systems. This paper presents a new algorithm for personalizing educational content to students that combines collaborative filtering algorithms with social choice theory. The algorithm constructs a “difficulty” ranking over questions for a target student by aggregating the ranking of similar students, as measured by different aspects of their performance on common past questions, such as grades, number of retries, and time spent solving questions. It infers a difficulty ranking directly over the questions for a target student, rather than ordering them according to predicted performance, which is prone to error. The algorithm was tested on two large real world data sets containing tens of thousands of students and a million records. Its performance was compared to a variety of personalization methods as well as a non-personalized method that relied on a domain expert. It was able to significantly outperform all of these approaches according to standard information retrieval metrics. Our approach can potentially be used to support teachers in tailoring problem sets and exams to individual students and students in informing them about areas they may need to strengthen..

Citation: EduRank: A Collaborative Filtering Approach to Personalization in E-learning Avi Segal, Ziv Katzir, Ya’akov Gal, Guy Shani and Bracha Shapira. EduRank: A Collaborative Filtering Approach to Personalization in E-learning. Seventh International Conference on Educational Data Mining (EDM 2014), London, England, July 2014.

Download: http://bit.ly/1WzaWlY

This work was presented at HAIDM 2015. The 2015 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

Abstract: Despite the prevalence of e-learning systems in schools, most of today’s systems do not personalize educational data to the individual needs of each student. This paper proposes a new algorithm for sequencing questions to students that is empirically shown to lead to better performance and engagement in real schools when compared to a baseline approach. It is based on using knowledge tracing to model students’ skill acquisition over time, and to select questions that advance the student’s learning within the range of the student’s capabilities, as determined by the model. The algorithm is based on a Bayesian Knowledge Tracing (BKT) model that incorporates partial credit scores, reasoning about multiple attempts to solve problems, and integrating item difficulty. This model is shown to outperform other BKT models that do not reason about (or reason about some but not all) of these features. The model was incorporated into a sequencing algorithm and deployed in two classes in different schools where it was compared to a baseline sequencing algorithm that was designed by pedagogical experts. In both classes, students using the BKT sequencing approach solved more difficult questions and attributed higher performance than did students who used the expert-based approach. Students were also more engaged using the BKT approach, as determined by their interaction time and number of log-ins to the system, as well as their reported opinion. We expect our approach to inform the design of better methods for sequencing and personalizing educational content to students that will meet their individual learning needs.

Citation: Yossi Ben David, Avi Segal, and Kobi Gal. Sequencing Educational Content in Classrooms using Bayesian Knowledge Tracing.

Download: http://bit.ly/23I8nDB

This work was presented at HAIDM 2015. The 2015 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

Abstract: The majority of volunteers participating in citizen science projects perform only a few tasks each before leaving the system. We designed an intervention strategy to reduce disengagement in 16 different citizen science projects. Targeted users who had left the system received emails that directly addressed motivational factors that affect their engagement. Results show that participants receiving the emails were significantly more likely to return to productive activity when compared to a control group.

Keywords: Peer production, crowdsourcing, citizen science, intervention strategies.

Citation: Segal, A., Gal, Y.A.K., Simpson, R.J., Victoria Homsy, V., Hartswood, M., Page, K.R. and Jirotka, M., 2015, May. Improving productivity in citizen science through controlled intervention. In Proceedings of the 24th International Conference on World Wide Web Companion (pp. 331-337). International World Wide Web Conferences Steering Committee.

Download: http://bit.ly/1SiqBku