Download: http://bit.ly/2jCx7ZQ

Abstract: Volunteer-based crowdsourcing depend critically on maintaining the engagement of participants. We explore a methodology for extending engagement in citizen science by combining machine learning with intervention design. We first present a platform for using real-time predictions about forthcoming disengagement to guide interventions. Then we discuss a set of experiments with delivering different messages to users based on the proximity to the predicted time of disengagement. The messages address motivational factors that were found in prior studies to influence users’ engagements. We evaluate this approach on Galaxy Zoo, one of the largest citizen science application on the web, where we traced the behavior and contributions of thousands of users who received intervention messages over a period of a few months. We found sensitivity of the amount of user contributions to both the timing and nature of the message. Specifically, we found that a message emphasizing the helpfulness of individual users significantly increased users’ contributions when delivered according to predicted times of disengagement, but not when delivered at random times. The influence of the message on users’ contributions was more pronounced as additional user data was collected and made available to the classifier.

Citation: Avi Segal, Ya’akov Gal, Ece Kamar, Eric Horvitz, Alex Bower, Grant Miller. Intervention Strategies for Increasing Engagement in Volunteer-Based Crowdsourcing. International Joint Conference on Artificial Intelligence (IJCAI), New York, USA, July 2016.

Download: http://bit.ly/2jh9fgG

Abstract: Novel types of crowdsourcing systems require a wider spectrum of incentives for efficient motivation and management of human workers taking part in complex collaborations. Incentive management techniques used in conventional crowdsourcing platforms are not suitable for more intellectually-challenging tasks. Currently, incentives are custom-developed and managed by each particular platform. This prevents incentive portability and cross-platform comparison. In this paper we present PRINGL – a domain-specific language for programming and managing complex incentive strategies for socio-technical platforms in general. It promotes re-use of proven incentive logic and simplifies modeling, adjustment and enactment of complex incentives for socio-technical systems. We demonstrate its applicability and expressiveness on a set of realistic use-cases and discuss its properties.

Citation: Ognjen Scekic, Hong Linh Truong, Schahram Dustdar: PRINGL – A domain-specific language for incentive management in crowdsourcing. Computer Networks 90: 14-33(2015).

Download: http://bit.ly/2iMHw4e

Abstract: Crowdsourcing is generally used for tasks with minimal coordination, providing limited support for dynamic reconfiguration. Modern systems, exemplified by social machines, are subject to continual flux in both the client and development communities and their needs. To support crowd sourcing of open-ended development, systems must dynamically integrate human creativity with machine support. While workflows can be used to handle structured, predictable processes, they are less suitable for social machine development and its attendant uncertainty. We present models and techniques for coordination of human workers in crowd sourced software development environments. We combine the Social Compute Unit—a model of ad-hoc human worker teams—with versatile coordination protocols expressed in the Lightweight Social Calculus. This allows us to combine coordination and quality constraints with dynamic assessments of end-user desires, dynamically discovering and applying development protocols.

Citation: Dave Murray-Rust, Ognjen Scekic, Hong-Linh Truong, Dave Robertson and Schahram Dustdar. A Collaboration Model for Community-Based Software Development with Social Machines. 10th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 22-25 Oct, Miami, FL, USA, 2014.

Download: http://bit.ly/2jaHx55

This work was presented at HAIDM 2015. The 2015 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

Abstract: Crowdsourcing has emerged as a powerful way to provide computer systems with quick and easy access to human intelligence. However, there is a risk that online crowd workers could be directed to perform harmful tasks. To understand the impact of financial incentives on paid crowd workers’ willingness to behave maliciously, we conducted a series of experiments in which we hired crowd workers via one crowdsourcing task (Attack task) to attack a different crowdsourcing task (Target task. We found that roughly one third of all crowd workers were willing to provide the attack task with potentially sensitive information from the target task, and that we could double this number by increasing the payment of the Attack task. Based on exit interviews and community feedback, we discuss some of what workers reported. Our findings reveal a measurable cost to completing malicious work that well-meaning task designers can leverage to protect their systems from attack.

Citation: Walter Lasecki, Jaime Teevan and Ece Kamar. What’s Your Price? The Cost of Asking Crowd Workers to Behave Maliciously.

Download: http://bit.ly/1VB1DTh

This work was presented at HAIDM 2015. The 2015 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

Abstract: It can be hard to automatically identify sensitive content in images or other media because significant context is often necessary to interpret noisy content and complex notions of sensitivity. Online crowds can help computers interpret information that cannot be understood algorithmically. However, systems that use this approach can unwittingly show workers information that should remain private. For instance, images sent to the crowd may accidentally include faces or geographic identifiers in the background, and information pertaining to a task (e.g., the amount of a bill) may appear alongside private information (e.g., an account number). This paper introduces an approach for using crowds to filter information from sensory data that should remain private, while retaining information needed to complete a specified task. The pyramid workflow that we introduce allows crowd workers to identify private information while never having complete access to the (potentially private) information they are filtering. Our approach is flexible, easily configurable, and can protect user information in settings where automated approaches fail. Our experiments with 4685 crowd workers show that it performs significantly better than previous approaches.

Citation: Walter Lasecki, Mitchell Gordon, Jaime Teevan, Ece Kamar and Jeffrey Bigham. CrowdMask: Privacy-Preserving Crowd-Powered Systems.

Download: http://bit.ly/1YF37Ke

This work was presented at HAIDM 2015. The 2015 workshop on Human-Agent Interaction Design and Models was co-organised by SmartSociety.

Abstract: The majority of volunteers participating in citizen science projects perform only a few tasks each before leaving the system. We designed an intervention strategy to reduce disengagement in 16 different citizen science projects. Targeted users who had left the system received emails that directly addressed motivational factors that affect their engagement. Results show that participants receiving the emails were significantly more likely to return to productive activity when compared to a control group.

Keywords: Peer production, crowdsourcing, citizen science, intervention strategies.

Citation: Segal, A., Gal, Y.A.K., Simpson, R.J., Victoria Homsy, V., Hartswood, M., Page, K.R. and Jirotka, M., 2015, May. Improving productivity in citizen science through controlled intervention. In Proceedings of the 24th International Conference on World Wide Web Companion (pp. 331-337). International World Wide Web Conferences Steering Committee.

Download: http://bit.ly/1SiqBku

Abstract. A popular use of crowdsourcing is to collect and aggregate individual worker responses to problems to reach a correct answer. This paper studies the relationship between the computation complexity class of problems, and the ability of a group to agree on a correct solution. We hypothesized that for NP-Complete (NPC) problems, groups would be able to reach a majority-based correct solution once it was suggested by a group member and presented to the other members, due to the “easy to verify” (i.e., verification in polynomial time) characteristic of this complexity class. In contrast, when posed with PSPACE-Complete (PSC) “hard to verify” problems (i.e., verification in exponential time), groups will not necessarily be able to choose a correct solution even if such a solution has been presented. Consequently, increasing the size of the group is expected to facilitate the ability of the group to converge on a correct solution when solving NPC problems, but not when solving PSC problems. To test this hypothesis we conducted preliminary experiments in which we evaluated people’s ability to solve an analytical problem and their ability to recognize a correct solution. In our experiments, participants were significantly more likely to recognize correct and incorrect solutions for NPC problems than for PSC problems, even for problems of similar difficulties (as measured by the percentage of participants who solved the problem). This is a first step towards formalizing a relationship between the computationally complexity of a problem and the crowd’s ability to converge to a correct solution to the problem.

Citation: Ofra Amir, Yuval Shahar, Ya’akov Gal and Litan Ilany. On the Verification Complexity of Group Decision-Making Tasks. Conference on Human Computation and Crowdsourcing, Palm Springs, CA, November 2013.

Download: http://bit.ly/1MBhsqn

Smart Society’s M. Rovatsos has won second prize for best challenge and vision paper in AAMAS 2014!

Abstract: This paper proposes social computation, i.e. large-scale man-machine collaboration mediated by digital interaction media, as a vision for future intelligent systems, and as a new challenge for multiagent systems research. We claim that the study of social computation suggests a re-interpretation of many traditional AI endeavours, has huge potential application benefits, and presents the field of multiagent systems with novel, exciting research questions. We introduce an abstract model of social computation that helps capture some of its core research problems more precisely. We explore the potential contribution of multiagent systems technologies to the solution of these problems by exposing the close relationship between social computation and existing methods in multiagent systems. We describe how these methods could be reused in this novel application context, what methodological implications this has, and argue that the resulting cross-fertilisation will be highly beneficial for both sides.

Keywords: social computation, human-based computation, crowdsourcing, collective intelligence.

doi: http://dl.acm.org/citation.cfm?id=2617388.2617432

Citation: M. Rovatsos. Multiagent Systems for Social Computation (challenge paper), Thirteenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2014), May 5-9, 2014.

Download: http://bit.ly/1W9uFYU

Social computing broadly refers to computing-supported approaches that facilitate interactions among people and organizations. Social computing has emerged as an exciting multidisciplinary area of research, driven by the wealth of easily available information and the success of online social networks and social media. Social computing applications are characterized by high interactivity among users, user-generated content, and in cases such as Wikipedia, more open governance structures. Much of the recent excitement in social computing is driven by data analytics and business models. What is still lacking, however, is a deeper conceptual understanding of social computing — e.g., relating to its conceptual bases, information and abstractions, design principles, and platforms. This event invites papers that take an explicitly multiagent perspective in addressing these gaps and do so in thought-provoking ways.

The First International Workshop on Multiagent Foundations of Social Computing is Co-located with AAMAS 2014

Important Dates

• Submission: January 22, 2014
• Notification: February 19, 2014
• Camera-ready due: March 5, 2014

Collective Intelligence 2014 is an interdisciplinary conference seeking to bring together researchers from a variety of fields relevant to understanding and designing collective intelligence of many types.

The conference will take place at MIT and consist of:

• Invited talks from prominent researchers in different areas related to collective intelligence such as engineering, psychology, management, political science, information science, and sociology
• Oral presentations
• Poster/Demo sessions
• “Ignite” sessions in which practitioners (e.g. policy makers) connect with researchers around collective-intelligence-based solutions to real-world problems

IMPORTANT DATES

• Extended abstract submission deadline:  January 15, 2014
• Notification of acceptance / rejection:  February 15, 2014
• Conference dates:  June 10-12, 2014

Also see Collective Intelligence 2012.

Abstract. Information systems are becoming ever more reliant on different forms of social computing, employing individuals, crowds or assembled teams of professionals. With humans as first-class elements, the success of such systems depends heavily on how well we can motivate people to act in a planned fashion. Incentives are an important part of human resource management, manifesting selective and motivating effects. However, support for defining and executing incentives in today’s information systems is underdeveloped, often being limited to simple, per-task cash rewards. Furthermore, no systematic approach to program incentive functionalities for this type of platforms exists.

In this paper we present fundamental elements of a framework for programmable incentive management in information systems. These elements form the basis necessary to support modeling, programming, and execution of various incentive mechanisms. They can be integrated with different underlying systems, promoting portability and reuse of proven incentive strategies. We carry out a functional design evaluation by illustrating modeling and composing capabilities of a prototype implementation on realistic incentive scenarios.

Keywords: rewards, incentives, social computing, crowdsourcing.

doi: http://dx.doi.org/10.1007/978-3-642-38709-8_44

Citation: Ognjen Scekic, Hong-Linh Truong, Schahram Dustdar, “Programming Incentives in Information Systems”, 25th International Conference on Advanced Information Systems Engineering(CAiSE’13), Springer-Verlag, Valencia, Spain, 17-21 June, 2013.

Download: http://bit.ly/1p8SJOP

Abstract. Conventional incentive mechanisms were designed for business environments involving static business processes and a limited number of actors. They are not easily applicable to crowdsourcing and other social computing platforms, characterized by dynamic collaboration patterns and high numbers of actors, because the effects of incentives in these environments are often unforeseen and more costly than in a well-controlled environment of a traditional company.

In this paper we investigate how to design and calibrate incentive schemes for crowdsourcing processes by simulating joint effects of a combination of different participation and incentive mechanisms applied to a working crowd. More specifically, we present a simulation model of incentive schemes and evaluate it on a relevant real-world scenario. We show how the model is used to simulate different compositions of incentive mechanisms and model parameters, and how these choices influence the costs on the system provider side and the number of malicious workers.

Keywords: rewards, incentives, crowdsourcing, social computing, collective adaptive systems.

doi: http://dx.doi.org/10.1007/978-3-642-41030-7_11

Citation: Ognjen Scekic, Christoph Dorn, Schahram Dustdar, “Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcing Environments”, 21st International Conference on Cooperative Information Systems (CoopIS’13), September 11-13, 2013, Graz, Austria.

Download: http://bit.ly/1SnngGh

Introduction. Incentives and rewards help align the interests of employees and organizations. They first appeared with the division of labor and have since followed the increasing complexity of human labor and organizations. As a single incentive measure always targets a specific behavior and sometimes additionally induces unwanted responses from workers, multiple incentives are usually combined to counteract the dysfunctional behavior and produce  desired results. Numerous studies have shown the effectiveness of different incentive mechanisms and their selective and motivational effects. Their importance is reflected in the fact that most big and mid-size companies employ some kind of incentive measures.

Expansion of social computing will include not only better exploitation of crowdsourcing but also solutions that extend traditional business processes; increasing research interest seems to confirm the trend. Several frameworks aiming to support such new collaboration models are being developed (such as socially enhanced computing). These new forms of social computing are intended to support greater task complexity, more intelligent task division, complex organizational and managerial structures for virtual teams, and virtual “careers.” With envisioned changes, incentives will also gain importance and complexity to address workers’ dysfunctional behavior. This new emphasis calls for automated ways of handling incentives and rewards. However, the social computing market is dominated by flat and short-lived organizational structures, employing a limited number of simple incentive mechanisms. That is why we view the state of the social computing market as an opportunity to add novel ways of handling incentives and rewards.

Here, we analyze incentive mechanisms and suggest how they can be used for next-generation social computing. We start with a classification of incentive mechanisms in the literature and in traditional business organizations, then identify elements that can be used as building blocks for any composite incentive mechanism and show the same elements are also used in social computing, even though the resulting schemes lack the complexity needed to support advanced business processes; we conclude with our vision
for future developments.

doi: http://doi.acm.org/10.1145/2461256.2461275

Citation: Ognjen Scekic, Hong-Linh Truong, Schahram Dustdar, “Incentives and Rewarding in Social Computing”, Communications of the ACM, Vol. 65, No. 6, pp. 72-82.

Download: http://bit.ly/1QfQVcO