The 3rd IEEE Workshop on Human-in-the-loop Methods and Human Machine Collaboration in BigData (HMData 2019)

The 3rd IEEE Workshop on
Human-in-the-loop Methods and
Human Machine Collaboration in BigData
(IEEE HMData 2019)

co-located with IEEE Bigdata 2019
Los Angeles, Dec. 9th (Fixed)

Photo courtesy of wikimedia commons
About Keynotes Dates and Submission Program Student Travel Support Organization HMData 2018

About IEEE HMData 2019


Human power is a key factor to maximize the impact of bigdata technologies. This workshop addresses human-in-the-loop approaches in bigdata lifecycle - in collecting, processing, analyzing, utilizing, archiving and disposing them. The purpose of this workshop is to give excellent opportunities for students, researchers and practitioners to identify important research problems and exchange their ideas on human-in-the-loop in the bigdata context. To make the workshop an attractive place for those people, we solicit practitioner papers as well as research papers, in order to facilitate discussion among researchers who know solutions and practitioners who know problems. We also would like to make the place valuable for young researchers. All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.


This workshop covers a wide range of human-related topics in the bigdata context, such as crowdsourcing, collaborative recommendation, crowdsensing, workflow model for humans and machines, incentives, human-assisted bigdata analysis, bigdata-human interaction, supporting tools for humans in human-in-the-loop systems, security and privacy in human-machine collaboration , human factors and ELSI (ethical, legal and social issues) in human-in-the-loop systems, and human-machine collaboration in real-world problems.


Computation and Organization
Michael Bernstein (Stanford University)
Abstract: Can computation amplify our ability to achieve complex collective goals? Today's techniques in areas such as crowdsourcing often fall short of this vision, in large part because their architecture is based on workflows, which are so inflexible that they can only be used for simple and modular goals. In this talk, I offer an alternative architecture based on computational organizational structures, and demonstrate that this approach enables groups to flexibly collaborate toward complex and open-ended goals such as product design, software engineering, and top-tier research. I will introduce techniques that fluidly assemble flash organizations and continuously adapt their efforts, evolve team structures and membership over time, and coordinate volunteers around the world in pursuing open-ended research. This research argues for a shift away from crowdsourcing as simple microtasks, wiki edits, or competitions, and toward computational systems that proactively aid groups in working together nimbly, reactively, and effectively toward complex goals.

Bio: Michael Bernstein is an Associate Professor of Computer Science at Stanford University, where he is a member of the Human-Computer Interaction group. His research focuses on the design of social computing and crowdsourcing systems. Michael's research has received awards at premier computing venues, and he has been recognized with an NSF CAREER award and an Alfred P. Sloan Fellowship. His Ph.D. students have gone on both to industry (e.g., Adobe Research, Facebook Data Science) and faculty careers (e.g., Carnegie Mellon, UC Berkeley). Michael holds a bachelor's degree in Symbolic Systems from Stanford University, as well as a master's degree and a Ph.D. in Computer Science from MIT.


Labels: [FR] - Full Paper (Research-oriented), [FP] - Full Paper (Practice-oriented), [R] Project-in-Progress Paper (Research-oriented), [P] Project-in-Progress Paper (Practice-oriented)

8:50 Opening (WS Chairs)

8:55-10:10 Session 1 (Chair: Atsuyuki Morishima)

8:55 [FR] Icaro Alzuru, Aditi Malladi, Andrea Matsunaga, Mauricio Tsugawa, and Jose A.B. Fortes. Human-Machine Information Extraction Simulator for Biological Collections
9:20 [FP] Munenari Inoguchi, Keiko Tamura, and Ryota Hamamoto. Establishment of Work-Flow for Roof Damage Detection Utilizing Drones, Human and AI based on Human-in-the-Loop Framework
9:45 [R] Thi Tran, Rohit Valecha, Paul Rad, and H.Raghav Rao. Misinformation Harms During Crises : When The Human and Machine Loops Interact
9:57 [R] Pedoro Cardenas, Georgios Theodoropoulos, Boguslaw Obara, and Ibad Kureshi. Analysing Social Media as a Hybrid Tool to Detect and Interpret likely Radical Behavioural Traits for National Security

10:10-10:30 Coffee Break

10:30-12:25 Session 2 (Chair: Satoshi Oyama)

10:30 [FR] Pei-Yu Hou, Jing Ao, Andrew Rindos, Shivaramu Keelara, Paula J. Fedorka-Cray, and Rada Chirkova. Collaborative Workflow for Analyzing Large-Scale Data for Antimicrobial Resistance: An Experience Report
10:55 [FR] Masafumi Hayashi, Masaki Kobayashi, Masaki Matsubara, Toshiyuki Amagasa, and Atsuyuki Morishima. Incentive Design for Crowdsourced Development of Selective AI for Human and Machine Data Processing: A Case Study
11:20 [FR] Kousuke Uo, Masaki Kobayashi, Masaki Matsubara, Yukino Baba, and Atsuyuki Morishima. Active Learning Strategies for Hierarchical Labeling Microtasks
11:45 [FR] Jordan Hosier, Vijay K.Gurbani, and Neil Milsted. Disambiguation and Error Resolution in Call Transcripts
12:10 [R] Hinako Izumi, Masaki Matsubara, Chiemi Watanabe, and Atsuyuki Morishima. A Microtask Approach to Identifying Incomprehension for Facilitating Peer Learning

12:25-14:00 Lunch

14:00-15:40 Session 3 (Chair: Alex Quinn)

14:00 Keynote by Michael Bernstein (Stanford University)
Computation and Organization

15:00-15:10 Short Break

15:10 [P] Kyriaki Dimitradou, Rahul Manghwani, and Timothy C.Hoad. ClusterClean: a Weak Semi- Supervised Approach for Cleaning Data Labels
15:25 [R] Shigeaki Yuasa, Takumi Nakai, Takanori Maruichi, Manuel Landsmann, Koichi Kise, Masaki Matsubara, and Atsuyuki Morishima. Towards Quality Assessment of Crowdworker Output Based on Behavioral Data

15:40-16:00 Coffee Break

16:00-17:13 Session 4 (Chair: Kyriaki Dimitradou)

16:00 [FR] Ming-Chen Wang, Vahid Golderzahi, and Hsing-Kuo Pao. Extracting Explainable Deep Representation for Machine Tutoring
16:25 [R] Takafumi Suzuki, Satoshi Oyama, and Masahito Kurihara. Explainable Recommendation Using Review Text and a Knowledge Graph
16:37 [P] Mehdi Bahrami and Wei-Peng Chen. WATAPI: Composing Web API Specification from API Documentations through an Intelligent and Interactive Annotation Tool
16:49 [P] Sathish K. Sankarpandi, Spyros Samothrakis, Luca Citi, and Peter Brady. Active learning without unlabeled samples: generating questions and labels using Monte Carlo Tree Search
17:01 [P] Hidehiko Shishido, Hansung Kim, and Itaru Kitahara. Super Long Interval Time-Lapse Image Generation for Proactive Preservation of Cultural Heritage Using Crowdsoursing

17:13 Closing Remarks (WS Chairs)

Important Dates

  • Oct 12 (Sat), 2019: Due date for workshop papers submission Extended
    (Authors have to submit the title and abstract by Oct. 7 (Mon))
  • Nov 1 (Fri), 2019: Notification of paper acceptance to authors
  • Nov 15 (Fri), 2019: Camera-ready of accepted papers
  • Dec 9-12 (Mon-Thu), 2019: Workshops


All submissions must be submitted electorically through the submission page. Please prefix your submission category such as [Research Paper] to the Title of Paper field in the submission page. For example, if you would like to submit a project-in-progress paper "Crowd-centric Approach to Digital Archive Maintenance," you have to put "[project-in-progress paper] Crowd-centric Approach to Digital Archive Maintenance" into the Title of Paper field.
All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.

Submission Categories

  • Research Papers (*) (long presentation): They report significant and original results relevant to the scope of this workshop. We solicit innovative or thought-provoking work but they do not necessarily have to reach the level of completion. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Practitioner papers (*)(long presentation): They present interesting problems that require human-in-the-loop solutions in a variety of application domains, or present the interesting results of applying existing human-in-the-loop solutions to their domains. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Project-in-progress papers (short presentation): They present the goals, challenges, and preliminary results of research or real-world projects in progress. The maximum length is 3 pages.
(*) Some of the papers submitted to the research or practitioner paper categories may be accepted as project-in-progress papers and allotted to short presentation slots.


Papers should be formatted to IEEE Computer Society Proceedings Manuscript Formatting Guidelines in the IEEE Bigdata 2019 CFP page

Student Travel Support

The IEEE BigData 2019 will offer student travel awards to student authors of not only main confernece papers but also workshop papers. Details will be given at IEEE Bigdata 2019 site later.



Senjuti Basu Roy (NJIT)
Alex Quinn (Purdue University)
Atsuyuki Morishima (Univesity of Tsukuba)

Program Committee

Yukino Baba (University of Tsukuba)
Wolf-Tilo Balke (Technische Universitaet Braunschweig)
Adam Bradley (Amazon)
Daniel Barowy (Williams College)
Marco Brambilla (Politecnico di Milano)
Cinzia Cappiello (Politecnico di Milano)
Marina Danilevsky (IBM Research - Almaden)
Ujwal Gadiraju (L3S Research Center)
Munenari Inoguchi (University of Toyama)
Panos Ipeirotis (New York University)
Vana Kalogeraki (Athens University of Economics and Business)
Itaru Kitahara (University of Tsukuba)
Dongwon Lee (Penn State)
Masaki Matsubara (University of Tsukuba)
Satoshi Oyama (Hokkaido University/RIKEN AIP)
Nobuyuki Shimizu (Yahoo!Japan Research)
Elena Simperl (University of Southampton)
Yu Suzuki (Gifu University)
Keishi Tajima (Kyoto University)
Saravanan Thirumuruganathan (QCRI)
Ming Yin (Purdue University)
Demetris Zeinalipour (University of Cyprus)
Jing Zhang (Nanjing U. of Science & Technology)
Yudian Zheng (Twitter)

External Reviewers

Qiong Bu (University of Southampton)
Briony Gray (University of Southampton)