The Second IEEE Workshop on
Human-in-the-loop Methods and
Human Machine Collaboration in BigData
(IEEE HMData2018)

co-located with IEEE Bigdata 2018
Seattle, Dec. 10th

Photo courtesy of wikimedia commons
About Keynotes Dates and Submission Program Student Travel Support Organization HMData 2017

About IEEE HMData 2018


Human power is a key factor to maximize the impact of bigdata technologies. This workshop addresses human-in-the-loop approaches in bigdata lifecycle - in collecting, processing, analyzing, utilizing, archiving and disposing them. The purpose of this workshop is to give excellent opportunities for students, researchers and practitioners to identify important research problems and exchange their ideas on human-in-the-loop in the bigdata context. To make the workshop an attractive place for those people, we solicit practitioner papers as well as research papers, in order to facilitate discussion among researchers who know solutions and practitioners who know problems. We also would like to make the place valuable for young researchers. All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.


This workshop covers a wide range of human-related topics in the bigdata context, such as crowdsourcing, collaborative recommendation, crowdsensing, workflow model for humans and machines, incentives, human-assisted bigdata analysis, bigdata-human interaction, supporting tools for humans in human-in-the-loop systems, security and privacy in human-machine collaboration , human factors and ELSI (ethical, legal and social issues) in human-in-the-loop systems, and human-machine collaboration in real-world problems.


Decision Theoretical Crowdsourcing
Dan Weld (University of Washington)
Abstract: Requestors often complain about the low-quality of crowd work, but what is the underlying cause? We argue that traditional quality-control techniques like majority vote and expectation maximization (EM) miss the point and don’t solve the true, underlying problems: confusing task instructions, poor worker training, and poor allocation of tasks to workers. Instead we advocate three new methods based on decision theory: 1) gated instruction, 2) adaptive testing, 3) micro-argumentation, and 4) self-improving workflows. These methods fuse ideas from HCI with AI methods such as partially-observable Markov decision problems & reinforcement learning. As a bonus, we’ll present recent work on active learning, where the crowd does more than just label examples.

Bio: Daniel S. Weld is Thomas J. Cable / WRF Professor in the Paul G. Allen School of Computer Science & Engineering and Entrepreneurial Faculty Fellow at the University of Washington. After formative education at Phillips Academy, he received bachelor's degrees in both Computer Science and Biochemistry at Yale University in 1982. He landed a Ph.D. from the MIT Artificial Intelligence Lab in 1988, received a Presidential Young Investigator's award in 1989, an Office of Naval Research Young Investigator's award in 1990, was named AAAI Fellow in 1999 and deemed ACM Fellow in 2005. Dan was a founding editor for the Journal of AI Research, was area editor for the Journal of the ACM, guest editor for Computational Intelligence and Artificial Intelligence, and was Program Chair for AAAI-96. Dan has published two books and scads of technical papers. Dan is an active entrepreneur with several patents and technology licenses. He co-founded Netbot Incorporated, creator of Jango Shopping Search (acquired by Excite), AdRelevance, a monitoring service for internet advertising (acquired by Nielsen NetRatings), and data integration company Nimble Technology (acquired by Actuate). Dan is a Venture Partner at the Madrona Venture Group and on the Scientific Advisory Boards of the Allen Institute for Artificial Intelligence and the Madrona Venture Group. Dan teaches many courses, including graduate classes on Artificial Intelligence, Extracting, Managing & Personalizing Web Information and Intelligent User Interfaces, and undergraduate classes on Artficial Intelligence, Advanced Internet Systems, and Machine Learning. In 2012, Dan co-organized a workshop on Crowdsourcing Personalized Online Education. During sabbaticals Dan was a visiting professor at Griffith University in Brisbane, Australia and visited the VIBE group at Microsoft Research.


Labels: [FR] - Full Paper (Research-oriented), [FP] - Full Paper (Practice-oriented), [R] Project-in-Progress Paper (Research-oriented), [P] Project-in-Progress Paper (Practice-oriented)

Participant Survey Questions

We would like to sincerely ask every participant to answer participant survey questions. It takes only one minute to answer. The aggregated values will be open to the participants of the workshop venue to share who we are and what are topics they are interested in

8:45 Opening (WS Chairs)

8:50-10:10 Session 1 (Chair: TBA)

8:50 [FR] Maria Stefan, Jose Gutierrez Lopez, and Rasmus Løvenstein Olsen. Exploring the Potential of Modern Advanced Metering Infrastructure in Low-Voltage Grid Monitoring Systems
9:20 [R] Hidehiko Shishido, Emi Kawasaki, Yutaka Ito, Youhei Kawamura, Toshiya Matsui, and Itaru Kitahara. Time-Lapse Image Generation using Image-Based Modeling by Crowdsourcing
9:30 [R] Yusuke Suzuki, Masaki Matsubara, Keishi Tajima, Toshiyuki Amagasa, and Atsuyuki Morishima. A Cache-based Approach to Dynamic Switching between Different Dataflows in Crowdsourcing
9:40 [P] Flavia Fulco, Munenari Inoguchi, and Tomoya Mikami. Cyber-Physical Disaster Drill: Preliminary Results and Social Challenges of the First Attempts to Unify Human, ICT and AI in Disaster Response
9:50 [R] Tomoya Mikami, Masaki Matsubara, Takashi Harada, and Atsuyuki Morishima. Worker Classification based on Answer Pattern for Finding Typical Mistake Patterns
10:00 [R] Takafumi Suzuki, Satoshi Oyama, and Masahito Kurihara. Toward Explainable Recommendations: Generating Review Text from Multicriteria Evaluation Data

10:10-10:30 Coffee Break

10:30-12:00 Session 2 (Chair: Satoshi Oyama)

10:30 [FP] Keiko Tamura, Munenari Inoguchi, Kei Horie, Ryota Hamamoto, and Haruo Hayashi. Realization of Effective Team Management Collaborating between Cloud-based System and On-site Human Activities -A Case Study of Building Damage Inspection at 2018 Hokkaido Eastern Iburi Earthquake-
11:00 [FR] Masaki Matsubara, Masaki Kobayashi, and Atsuyuki Morishima. A Learning Effect by Presenting Machine Prediction as a Reference Answer in Self-correction
11:30 [R] Huang Li, Shiaofen Fang, Snehasis Mukhopadhyay, Andrew Sakin, and Li Shen. Interactive Machine Learning by Visualization: A Small Data Solution
11:40 [R] Nadeesha Wijerathna, Masaki Matsubara, and Atsuyuki Morishima. Finding Evidences by Crowdsourcing
11:50 [R] Koyo Kobayashi, Hidehiko Shishido, Yoshinari Kameda, and Itaru Kitahara. A Method to Collect Multi-view Images of High Importance Using Disaster Map and Crowdsourcing

12:00-13:30 Lunch

13:30-15:00 Session 3 (Chair: Senjuti Basu Roy)

13:30 Keynote by Dan Weld (University of Washington)
Decision Theoretical Crowdsourcing

14:30 [FR] Yoko Yamakata, Keishi Tajima, and Shinsuke Mori. A Case Study on Start-up of Dataset Construction: In Case of Recipe Named Entity Corpus

15:00-15:40 Coffee Break with Posters

15:40-17:10 Session 4 (Chair: Keishi Tajima)

Note: The order of presentations was changed in the session.

15:40 [FP] Munenari Inoguchi, Keiko Tamura, Kei Horie, Ryota Hamamoto, and Haruo Hayashi. Implementation of Effective Field Survey for Damaged Buildings under Harmonious Collaboration between Human and ICT - A Case Study of 2018 Hokkaido Eastern Iburi Earthquake -
16:10 [FR] Ying Zhong, Masaki Matsubara, and Atsuyuki Morishima. Identification of Important Images for Understanding Web Pages
16:40 [FR] Hirotaka Hashimoto, Masaki Matsubara, Yuhki Shiraishi, Daisuke Wakatsuki, Jianwei Zhang, and Atsuyuki Morishima. A Task Assignment Method Considering Inclusiveness and Activity Degree
17:10 [FR] Nathalia Nascimento, Paulo Alencar, Carlos Lucena, and Donald Cowan. Toward Human-in-the-Loop Collaboration of Software Engineers and Machine Learning Algorithms

17:40 Closing Remarks (WS Chairs)

Important Dates

  • Oct 16, 2018: Due date for full workshop papers submission Extended
    (Authors have to submit the title and abstract by Oct. 13)
  • Nov 1, 2018: Notification of paper acceptance to authors
  • Nov 15, 2018: Camera-ready of accepted papers
  • Dec 10-13, 2018: Workshops


All submissions must be submitted electorically through the submission page. Please prefix your submission category such as [Research Paper] to the Title of Paper field in the submission page. For example, if you would like to submit a project-in-progress paper "Crowd-centric Approach to Digital Archive Maintenance," you have to put "[project-in-progress paper] Crowd-centric Approach to Digital Archive Maintenance" into the Title of Paper field.
All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.

Submission Categories

  • Research Papers (*) (long presentation): They report significant and original results relevant to the scope of this workshop. We solicit innovative or thought-provoking work but they do not necessarily have to reach the level of completion. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Practitioner papers (*)(long presentation): They present interesting problems that require human-in-the-loop solutions in a variety of application domains, or present the interesting results of applying existing human-in-the-loop solutions to their domains. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Project-in-progress papers (short presentation): They present the goals, challenges, and preliminary results of research or real-world projects in progress. The maximum length is 3 pages.
(*) Some of the papers submitted to the research or practitioner paper categories may be accepted as project-in-progress papers and allotted to short presentation slots.


Papers should be formatted to IEEE Computer Society Proceedings Manuscript Formatting Guidelines in the IEEE Bigdata 2018 CFP page

Student Travel Support

The IEEE BigData 2018 will offer student travel awards to student authors of not only main confernece papers but also workshop papers. The application deadline is November 15. Please email the application materials to the Student Travel Award Chair Prof. Feng Chen at Details are given at this page.



Senjuti Basu Roy (NJIT)
Lei Chen (HKUST)
Atsuyuki Morihsima (Univesity of Tsukuba)

Program Committee

Sihem Amer-Yahia (CNRS/LIG)
Yukino Baba (University of Tsukuba)
Wolf-Tilo Balke (Technische Universitat Braunschweig)
Adam Bradley (Amazon)
Marina Danilevsky (IBM Research Almaden )
Munenari Inoguchi (University of Toyama)
Yoshiharu Ishikawa (Nagoya University)
Vana Kalogeraki (Athens University of Economics and Business)
Itaru Kitahara (University of Tsukuba)
Dongwon Lee (Penn State University)
John O\'Donovan (Univerisity of California Santa Barbara)
Danijel Schorlemmer (GFZ German Research Centre for Geosciences)
Satoshi Oyama (Hokkaido University)
Nobuyoshi Shimizu (Yahoo!Japan Research)
Siddharth Suri (Microsoft Research)
Yu Suzuki (NAIST)
Keishi Tajima (Kyoto University)
Saravanan Thirumuruganathan (QCRI)
Gordon Wells (University of Texas, Austin)
Vladimir Zadorozhny (University of Pittsburgh)
Demetris Zeinalipour (Max Planck Institute for Informatics and University of Cyprus)
Jing Zhang (Nanjing U. of Science & Technology)
Yudian Zheng (Twitter)