The 4th IEEE Workshop on
Human-in-the-Loop Methods and Future of Work in BigData
(IEEE HMData 2020)

co-located with IEEE Bigdata 2020
Atlanta, GA, USA, Dec. 10th (Fixed)
Now Taking Place Virtually


Photo courtesy of wikimedia commons
About Keynotes Dates and Submission Program Organization HMData 2019 IEEE Bigdata 2020

About IEEE HMData 2020

Overview

HMData workshop, which originally started as the "Human-Machine collaboration in BigData" workshop, will investigate the opportunities and challenges in human machine collaboration in work with bigdata, which are described by two terms: Human-in-the-Loop Methods and Future of Work. Human-in-the-Loop is a term focusing on the employer's viewpoint while Future of Work focuses more on worker's viewpoint, in both of which the division of labor among humans and machines is a key issue. This area is likely to be heavily AI driven, and we intend to invite papers covering the following aspects, (a) Capturing human capabilities through intelligent models and how to adapt them through changing perceptions, needs, and skills. (2) High level tools that provide the ability for all stakeholders in the new ecosystem, including regulators for policies and AI workers, to specify their requirements. (3) system design and engineering of job platforms for collection, storage, retrieval, and analysis of data deluge about workers, jobs, and their activities. (4) Benchmarking and the development of appropriate metrics to measure system performance as well as human aspects, such as satisfaction, capital advancement, and equity.

We welcome any interesting ideas and results on any relevant topics, but this year, we also encourage submitting papers on new projects inspired by the COVID-19 crisis, such as those on human-in-the-loop solutions in the pandemic, those on re-evaluating how we organize labor and how we share work with machines in the future. To make the workshop an attractive place for those people, we solicit practitioner papers as well as research papers, in order to facilitate discussion among researchers who know solutions and practitioners who know problems. We also would like to make the place valuable for young researchers. All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.

Topics

This workshop covers a wide range of topics of human-machine collaboration in work with bigdata. Keywords include: crowdsourcing, collaborative recommendation, crowdsensing, workflow model for humans and machines, incentives, human-assisted bigdata analysis, bigdata-human interaction, human-machine collaboration in real-world applications (such as natural disaster response, education, and citizen science), and ELSI in Human-in-the-loop systems and Future of Work. We expect submissions to address some of the following issues:
  1. capturing human characteristics and capabilities,
  2. stakeholder requirement specification,
  3. social processes around the human-in-the-loop systems, 
  4. platforms and ecosystems,
  5. computation capabilities, and
  6. benchmarks and metrics for human-in-the-loop systems and Future of Work

Keynotes

Crowd Sleuths: Solving Mysteries with Crowdsourcing, Experts, and AI
Kurt Luther (Virginia Tech)
Abstract: Professional investigators in fields such as journalism, law enforcement, and academia have long sought the public's help in solving mysteries, typically by providing tips. However, as social technologies capture more digital traces of daily life and enable new forms of collaboration, members of the public are increasingly leading their own investigations. These efforts are perhaps best known for high-profile failures characterized by sloppy research and vigilantism, such as the 2013 Boston Marathon Bombing manhunt on Reddit and 4chan. However, other crowdsourced investigations have led to the successful recovery of missing persons and apprehension of violent criminals, suggesting real potential. In this talk, I present three projects from my research group, the Crowd Intelligence Lab, where we build software tools that bring together crowds, experts, and AI to support ethical and effective investigations and solve mysteries. In the CrowdIA project, we adapted the sensemaking loop for intelligence analysts to enable novice crowds to discover a hidden terrorist plot within large quantities of textual evidence documents. In the GroundTruth project, we developed a novel diagramming technique to enable novice crowds to collaborate with expert investigators to geolocate and verify (or debunk) photos and videos shared on social media. In the Photo Sleuth project, we built and launched a free public website with over 10,000 registered users who employ AI-based face recognition to identify unknown soldiers in historical portraits from the American Civil War era. I will conclude the talk by discussing broader opportunities and risks in combining the complementary strengths of human and artificial intelligence for investigation, sensemaking, and other complex and creative tasks.

Bio: Dr. Kurt Luther is an associate professor of computer science and (by courtesy) history at Virginia Tech, based in the Washington, D.C. area. He directs the Crowd Intelligence Lab, creating new ways for experts to leverage the complementary strengths of crowdsourced human intelligence and artificial intelligence (AI) in domains like journalism, national security, and history. His current research focuses on supporting open source intelligence (OSINT) investigations, combating disinformation and misinformation, and identifying unknown people and places in historical and modern photos. Dr. Luther has been honored with the National Science Foundation CAREER Award, the Virginia Tech College of Engineering Outstanding New Assistant Professor Award, and the Purdue Polytechnic Institute Outstanding Technology Alumni Award. His papers have received the ACM CSCW Best Paper Award, the AAAI HCOMP Notable Paper Award, and the ACM IUI Best Paper Award. His software has won the Microsoft Cloud AI Research Challenge Grand Prize and two HCOMP Best Demo Awards. His research has been funded by DOD, Google, NEH, NHPRC, NIH, and NSF; and featured in The Atlantic, CNN, NPR, Smithsonian, and TIME. He is a member of AAAI and a senior member of ACM. Previously, Dr. Luther was a postdoctoral fellow in the Human-Computer Interaction Institute at Carnegie Mellon University. He received his Ph.D. in human-centered computing from Georgia Tech, where he was a James D. Foley Scholar. He received his B.S. in computer graphics technology, with honors and highest distinction, from Purdue University. He also completed internships at IBM Research, Microsoft Research, and YouTube/Google.

Program (PDF Version)

The workshop starts at 1PM on Dec. 10 (Eastern Standard Time)
New York, USA Thu, 10 Dec 2020 at 13:00 EST
Paris, France Thu, 10 Dec 2020 at 19:00 CET
Tokyo, Japan Fri, 11 Dec 2020 at 03:00 JST

Labels: [LR] - Research Paper (Long Presentation), [R] Research Paper (Short Presentation), [W] Project-in-Progress Paper (Short Presentation), [P] Practitioner Paper (Short Presentation)

1:00PM Opening (WS Chairs)

1:05-2:40 Session 1 (Chair: Alex Quinn)

1:05 Keynote by Kurt Luther (Virginia Tech)
Crowd Sleuths: Solving Mysteries with Crowdsourcing, Experts, and AI Kurt Luther

1:55-2:00 Short Break

2:00 [LR] Shubhi Asthana, Shikhar Kwatra, Christine T. Wolf, Pawan Chowdhary, and Taiga Nakamura: Human-in-the-Loop Business Modelling for Emergent External Factors
2:20 [LR] Munenari Inoguchi, Keiko Tamura, Kousuke Uo, and Masaki Kobayashi: Validation of CyborgCrowd Implementation Possibility for Situation Awareness in Urgent Disaster Response -Case Study of International Disaster Response in 2019-

2:40-3:00 Break

3:00-4:16 Session 2 (Chair: Senjuti Basu Roy)

3:00 [LR] Vijaya Kumari Yeruva, Mayanka Chandrashekar, Yugyung Lee, Jeff Rydberg-Cox, Virginia Blanton, and Nathan A Oyler: Interpretation of Sentiment Analysis with Human-in-the-Loop
3:20 [LR] Conor Morgan, Iulia Paun, and Nikos Ntarmos: Exploring Contextual Paradigms in Context-Aware Recommendations
3:40 [W] Seungun Kim, Masaki Matsubara, and Atsuyuki Morishima: Analysis of Hand-drawn Maps of Places in Natural Disaster Pictures
3:52 [W] Hikaru Uchida, Masaki Matsubara, Kei Wakabayashi, and Atsuyuki Morishima: Human-in-the-loop Approach towards Dual Process AI Decisions
4:04 [P] Laura H. Kahn, Onur Savas, Adamma Morrison, Kelsey A. Shaffer, and Lila Zapata: Modelling Hybrid Human-Artificial Intelligence Cooperation: A Call Center Customer Service Case Study

4:16-4:40 Break

4:40-5:50 Session 3 (Chair: Atsuyuki Morishima)

4:40 [LR] Icaro Alzuru, Andréa Matsunaga, Maurício Tsugawa, and José A.B. Fortes: General Self-aware Information Extraction from Labels of Biological Collections
5:00 [LR] Nikolas Dawson, Marian-Andrei Rizoiu, Benjamin Johnston, and Mary-Anne Williams: Predicting Skill Shortages in Labor Markets: A Machine Learning Approach
5:20 [R] Hisatoshi Toriya, Ashraf Dewan, and Itaru Kitahara: Adaptive Image Scaling for Corresponding Points Matching between Images with Differing Spatial Resolutions
5:32 [W] Takafumi Suzuki, Satoshi Oyama, and Masahito Kurihara: A Framework for Recommendation Algorithms Using Knowledge Graph and Random Walk Methods

5:44 Closing Remarks (WS Chairs)

Important Dates

  • Oct 11 (Sun), 2020: Due date for workshop papers submission Extended
    (Authors have to submit the title and abstract by Oct. 5 (Mon))
  • Nov 2 (Mon), 2020: Notification of paper acceptance to authors
  • Nov 15 (Sun), 2020: Camera-ready of accepted papers
  • Dec 10-13 (Thu-Sun), 2020: Workshops

Submission

All submissions must be submitted electorically through the submission page. Please prefix your submission category such as [Research Paper] to the Title of Paper field in the submission page. For example, if you would like to submit a project-in-progress paper "Crowd-centric Approach to Digital Archive Maintenance," you have to put "[project-in-progress paper] Crowd-centric Approach to Digital Archive Maintenance" into the Title of Paper field.
All papers accepted for the workshop will be included in the Workshop Proceedings published by the IEEE Computer Society Press, made available at the Conference.

Submission Categories

  • Research Papers (*) (long presentation): They report significant and original results relevant to the scope of this workshop. We solicit innovative or thought-provoking work but they do not necessarily have to reach the level of completion. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Practitioner papers (*)(long presentation): They present interesting problems that require human-in-the-loop solutions in a variety of application domains, or present the interesting results of applying existing human-in-the-loop solutions to their domains. The expected length is between 4 and 6 pages. The maximum length is 10 pages, though the paper should be commensurate with the size of the contribution.
  • Project-in-progress papers (short presentation): They present the goals, challenges, and preliminary results of research or real-world projects in progress. The maximum length is 3 pages.
(*) Some of the papers submitted to the research or practitioner paper categories may be accepted as project-in-progress papers and allotted to short presentation slots.

Format

Papers should be formatted to IEEE Computer Society Proceedings Manuscript Formatting Guidelines in the IEEE Bigdata 2020 CFP page

Organization

Chairs

Senjuti Basu Roy (NJIT)
Alex Quinn (Pardue University)
Atsuyuki Morihsima (Univesity of Tsukuba)

Program Committee

  • Yukino Baba (University of Tsukuba)
  • Wolf-Tilo Balke (Technische Universitaet Braunschweig)
  • Daniel Barowy (Williams College)
  • Ria Mae Borromeo (University of the Philippines Open University)
  • Pierre Bourhis (CNRS, CRIStAL)
  • Francois Charoy (University of Lorraine, Inria, CNRS)
  • Marina Danilevsky (IBM Research - Almaden)
  • Gianluca Demartini (University of Queensland)
  • Kyriaki Dimitradou (Amazon)
  • Shady Elbassuoni (American University of Beirut)
  • Ujwal Gadiraju (Delft University of Technology)
  • David Gross Amblard (Rennes 1 University / IRISA Lab)
  • Munenari Inoguchi (University of Toyama)
  • Vana Kalogeraki (Athens University of Economics and Business)
  • Itaru Kitahara (University of Tsukuba)
  • Dongwon Lee (Pennsylvania State University)
  • Masaki Matsubara (University of Tsukuba)
  • Satoshi Oyama (Hokkaido University)
  • Raghav Rao (University of Texas at San Antonio)
  • Naoki Sakai (National Research Institute for Earth Science and Disaster Resilience)
  • Nobuyuki Shimizu (Yahoo!Japan Research)
  • Yu Suzuki (Gifu University)
  • Keishi Tajima (Kyoto University)
  • Saravanan Thirumuruganathan (QCRI)
  • Demetris Zeinalipour (University of Cyprus)
  • Jing Zhang Nanjing (U. of Science ＆ Technology)

Contact

hmdata.chairs@gmail.com