A Five-Star Guide for Achieving Replicability and Reproducibility When Working with GIS Software and Algorithms

Reference: John P. Wilson, Kevin Butler, Song Gao, Yingje Hu, Wenwen Li & Dawn J. Wright (2020) A Five-Star Guide for Achieving Replicability and Reproducibility When Working with GIS Software and AlgorithmsAnnals of the American Association of Geographers, DOI: 10.1080/24694452.2020.1806026

Abstract: The availability and use of geographic information technologies and data for describing the patterns and processes operating on or near the Earth’s surface have grown substantially during the past fifty years. The number of geographic information systems software packages and algorithms has also grown quickly during this period, fueled by rapid advances in computing and the explosive growth in the availability of digital data describing specific phenomena. Geographic information scientists therefore increasingly find themselves choosing between multiple software suites and algorithms to execute specific analysis, modeling, and visualization tasks in environmental applications today. This is a major challenge because it is often difficult to assess the efficacy of the candidate software platforms and algorithms when used in specific applications and study areas, which often generate different results. The subtleties and issues that characterize the field of geomorphometry are used here to document the need for (1) theoretically based software and algorithms; (2) new methods for the collection of provenance information about the data and code along with application context knowledge; and (3) new protocols for distributing this information and knowledge along with the data and code. This article discusses the progress and enduring challenges connected with these outcomes.

New Protocols for Distributing the Data and Code of Geospatial Research

Here, we propose a five-star practical guide for sharing data and code in geospatial research, modeled after the five-star system offered by Berners-Lee (2009) for publishing linked open data on the Web. Instead of asking researchers to share all pieces of data and code, this five-star guide encourages a simple start of data and code sharing, and researchers can move to a higher level when time and other resources allow.

See more papers on the Forum on Reproducibility and Replicability in Geography.

Prof. Gao receives a new geospatial data science research grant

The American Family Insurance Data Science Institute (AFIDSI) is honored to announce the results of the new round of the American Family Funding Initiative, a research competition for data science projects. American Family Insurance has partnered with UW–Madison through the Institute to offer “mini grants” of $75k-to-150k per year for data science research. This is the second installation of a $10 million research agreement.

The goal of the American Family Funding Initiative is to stimulate and support highly innovative research. The successful projects, reviewed by faculty and staff from across UW-Madison campus, were evaluated based on their potential contributions to the field of data science, practical use and the novelty of their approaches.

AFIDSI brings people together to launch new research in data science and apply findings to solve problems. In collaboration with researchers across campus and beyond, AFIDSI focuses on the fundamentals of data science research and on translating that research into practice.

New projects funded in the second round of the American Family Funding Initiative include:

A Deep Learning Approach to User Location Privacy Protection
Principal Investigator: Song Gao, Assistant Professor of Geography.
Co-Principal Investigator: Jerry Zhu, Computer Sciences.

Location information is among the most sensitive data being collected by mobile apps, and users increasingly raise privacy concerns. The proposed research aims to develop a deep learning architecture that will protect users’ location privacy while keeping the capability for location-based business recommendations such as usage-based insurance (UBI).

Machine Learning Approaches for Metadata Standardization
Principal investigator: Colin Dewey, Professor of Biostatistics and Medical Informatics.
Co-Principal Investigator: Mark Craven, Biostatistics and Medical Informatics.

The need to manually standardize metadata describing records in large data sets, compiled from many sources, is a major bottleneck in both research and business. This project will develop machine learning approaches for automating metadata standardization and identifying records that would most benefit from expert human input.

Adaptive Operations Research and Data Modeling for Insurance Applications
Principal Investigator: Michael Ferris, Professor of Computer Sciences.

Insurance claims applications must be operated efficiently under normal conditions and allow for rapid reconfiguration in crisis situations. The proposed work will develop optimization models, data and solution processes to schedule resources over time, servicing normal workloads, while creating resilience to abrupt changes from random disturbances.

GAN-mixup: A New Approach to Improve Generalization in Machine Learning
Principal Investigator: Kangwook Lee, Assistant Professor of Electrical and Computer Engineering.
Co-Principal Investigator: Dimitris Papailiopoulos, Electrical and Computer Engineering.

Recent machine learning successes rely on predictive models that adapt to previously unseen data. This research will provide a new approach to improve such generalization, with provable performance guarantees.

Integer Programming for Mixture Matrix Completion
Principal Investigator: Jeff Linderoth, Professor of Industrial and Systems Engineering.
Co-Principal Investigators: Jim Luedtke, Industrial and Systems Engineering; Daniel Pimentel-Alarcon, Biostatistics and Medical Informatics.

Matrix completion, or filling in the unknown entities in a matrix, is used in applications such as recommender systems and systems for analyzing visual images. This project will apply integer programming techniques to develop algorithms for solving a mixture matrix completion problem, paving the way towards applying this method to large-scale data science problems.

Developing a State-of-the-Science Regional Weather Forecasting System
Principal Investigator: Michael Morgan, Professor of Atmospheric and Oceanic Sciences.
Co-Principal Investigator: Brett Hoover, Space Science and Engineering Center.

This project will develop a weather prediction system for American Family Insurance, run entirely in cloud computing infrastructure, that will improve the accuracy of forecasting hazards such as hail and hurricanes. The probabilistic system will also estimate the uncertainty associated with the predictability of hazardous weather.

Model Recycling: Accelerating Machine Learning by Re-using Past Completions
Principal Investigator: Shivaram Venkataraman, Assistant Professor of Computer Sciences.
Co-Principal Investigator: Dimitris Papailiopoulos, Electrical and Computer Engineering.

Training machine learning models that are used in a wide range of domains, from drug discovery to recommendation engines, takes significant time and resources. This project will automate and accelerate this process of fine-tuning by reusing and sharing past computations from prior training jobs, using a technique called model recycling.

Additionally, two projects from the first round received continued funding:

Question Asking with Differing Knowledge and Goals
Principal investigator: Joe Austerweil, Assistant Professor of Psychology.

Despite tremendous progress in machine learning, automated answers to questions are still inferior to answers from humans. This project investigates whether incorporating psycholinguistic factors that influence how people respond to language can improve automated question-answering methods.

Lightweight Natural Language and Vision Algorithms for Data Analysis
Principal investigator: Vikas Singh, Professor of Biostatistics and Medical Informatics. Collaborators: Zhanpeng Zeng, Computer Sciences; Shailesh Acharya and Glenn Fung, American Family Insurance.

Natural language processing is a form of artificial intelligence that helps computers read and understand human language. The overarching goal of this project is to accelerate the time it takes to train and test efficient, accurate natural language processing models.

National Fellowships Engage Geospatial Research And Education On COVID-19

Projects address human mobility patterns, access to health care and food systems, racial and disability disparities during the pandemic.

The Geospatial Software Institute (GSI) Conceptualization Project has announced 16 fellowships to researchers at 13 institutions to tackle COVID-19 challenges using geospatial software and advanced capabilities in cyberinfrastructure and data science. Prof. Song Gao was selected as one of the geospatial fellows. A full list of the fellows, with biographies and project information, is at https://gsi.cigi.illinois.edu/geospatial-fellows-members/.

The GSI Conceptualization Project is supported by the National Science Foundation (NSF), and carried out in partnership with the American Association of Geographers (AAG), Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI), the National Opinion Research Center (NORC) at the University of Chicago, Open Geospatial Consortium (OGC), and University Consortium for Geographic Information Science (UCGIS). Technical and cyberinfrastructure support are provided by the CyberGIS Center for Advanced Digital and Spatial Studies (CyberGIS Center)  at the University of Illinois at Urbana-Champaign. 

“The COVID-19 crisis has shown how critical it is to have cutting-edge geospatial software and cyberinfrastructure to tackle the pandemic’s many challenges,” said Shaowen Wang, the principal investigator of the NSF project and founding director of the CyberGIS Center. “We are extremely grateful for NSF’s support to fund this talented group of researchers, whose work is so diverse yet complementary.”

Michael Goodchild, chair of the NSF project advisory committee and professor emeritus in geography at UC-Santa Barbara, agreed. “Geospatial data and tools have enormous potential for helping us address the challenges of COVID-19, and these 16 Fellows have exactly the right qualifications and experience. I’m very excited to see what they are able to achieve.”

The Fellows come from varied professional, cultural, and institutional backgrounds, representing many disciplinary areas, including public health, food access, emergency management, housing and neighborhood change, and community-based mapping. The fellowship projects represent frontiers of emerging geospatial data science, including for example geospatial AI and deep learning, geovisualization, and advanced approaches to gathering and analyzing geospatial data.

Pioneered by multi-million dollar research funded by NSF, cyberGIS (i.e., cyber geographic information science and systems based on advanced computing and cyberinfrastructure) has emerged as a new generation of GIS, comprising a seamless integration of advanced cyberinfrastructure, GIS, and spatial analysis and modeling capabilities while leading to widespread research advances and broad societal impacts. Built on the progress made by cyberGIS-related communities, the GSI conceptualization project is charged with developing a strategic plan for a long-term hub of excellence in geospatial software infrastructure, one that can better address emergent issues of food security, ecology, emergency management, environmental research and stewardship, national security, public health, and more.

The Geospatial Fellows program will enable diverse researchers and educators to harness geospatial software and data at scale, in reproducible and transparent ways; and will contribute to the nation’s workforce capability and capacity to utilize geospatial big data and software for knowledge discovery. With a particular focus on COVID-19, the combined research findings of the Fellows will offer insight on how to make geospatial research computationally reproducible and transparent, while also developing novel methods, including analysis, simulation, and modeling, to study the spread and impacts of the virus. The Fellows’ research will substantially add to public understanding of the societal impacts of COVID-19 on different communities, assessing the social and spatial disparities of COVID-19 among vulnerable populations.

“I look forward to seeing the results of these projects, particularly as FAIR and open datasets, software, and models that others can then build on,” says Daniel S. Katz, Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications (NCSA), the University of Illinois.

For more information about the GSI conceptualization project, see their website: https://gsi.cigi.illinois.edu/.

For a list of Geospatial Fellows and their projects, visit https://gsi.cigi.illinois.edu/geospatial-fellows-members/.