Search USGS⌕ Search
Home/Archive/USGS

USGS · 70199929

Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor

Richard A. Erickson·Michael N. Fienen·S. Grace McCalla·Emily L. Weiser·Melvin L. Bower·Jonathan M. Knudson·Greg Thain
View original source ↗Export metadata ↓

Abstract

Biologists and environmental scientists now routinely solve computational problems that were unimaginable a generation ago. Examples include processing geospatial data, analyzing -omics data, and running large-scale simulations. Conventional desktop computing cannot handle these tasks when they are large, and high-performance computing is not always available nor the most appropriate solution for all computationally intense problems. High-throughput computing (HTC) is one method for handling computationally intense research. In contrast to high-performance computing, which uses a single "supercomputer," HTC can distribute tasks over many computers (e.g., idle desktop computers, dedicated servers, or cloud-based resources). HTC facilities exist at many academic and government institutes and are relatively easy to create from commodity hardware. Additionally, consortia such as Open Science Grid facilitate HTC, and commercial entities sell cloud-based solutions for researchers who lack HTC at their institution. We provide an introduction to HTC for biologists and environmental scientists. Our examples from biology and the environmental sciences use HTCondor, an open source HTC system.

Explore related subjects

PLOS Computational BiologyArticle

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Richard A. Erickson, Michael N. Fienen, S. Grace McCalla, Emily L. Weiser, Melvin L. Bower, Jonathan M. Knudson, Greg Thain. 2018-10-03. Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor. https://doi.org/10.1371/journal.pcbi.1006468

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related USGS reports

USGS2024

Epidemiological modeling of SARS-CoV-2 in white-tailed deer (Odocoileus virginianus) reveals conditions for introduction and widespread transmission

Emerging infectious diseases with zoonotic potential often have complex socioecological dynamics and limited ecological data, requiring integration of epidemiological modeling with surveillance. Although our understanding of SARS-CoV-2 has advanced considerably since its detection in late 2019, the factors influencing its introduction and transmission in wildlife hosts, particularly white-tailed deer ( Odocoileus virginianus ), remain poorly understood. We use a Susceptible-Infected-Recovered-Susceptible epidemiological model to investigate the spillover risk and transmission dynamics of SARS-CoV-2 in wild and captive white-tailed deer populations across various simulated scenarios. We found that captive scenarios pose a higher risk of SARS-CoV-2 introduction from humans into deer herds and subsequent transmission among deer, compared to wild herds. However, even in wild herds, the transmission risk is often substantial enough to sustain infections. Furthermore, we demonstrate that the strength of introduction from humans influences outbreak characteristics only to a certain extent. Transmission among deer was frequently sufficient for widespread outbreaks in deer populations, regardless of the initial level of introduction. We also explore the potential for fence line interactions between captive and wild deer to elevate outbreak metrics in wild herds that have the lowest risk of introduction and sustained transmission. Our results indicate that SARS-CoV-2 could be introduced and maintained in deer herds across a range of circumstances based on testing a range of introduction and transmission risks in various captive and wild scenarios. Our approach and findings will aid One Health strategies that mitigate persistent SARS-CoV-2 outbreaks in white-tailed deer populations and potential spillback to humans.

PLOS Computational Biology↗
USGS2023

Quantifying ecosystem states and state transitions of the Upper Mississippi River System using topological data analysis

Aquatic systems worldwide can exist in multiple ecosystem states (i.e., a recurring collection of biological and chemical attributes), and effectively characterizing multidimensionality will aid protection of desirable states and guide rehabilitation. The Upper Mississippi River System is composed of a large floodplain river system spanning 2200 km and multiple federal, state, tribal and local governmental units. Multiple ecosystem states may occur within the system, and characterization of the variables that define these ecosystem states could guide river rehabilitation. We coupled a long-term (30-year) highly dimensional water quality monitoring dataset with multiple topological data analysis (TDA) techniques to classify ecosystem states, identify state variables, and detect state transitions over 30 years in the river to guide conservation. Across the entire system, TDA identified five ecosystem states. State 1 was characterized by exceptionally clear, clean, and cold-water conditions typical of winter (i.e., a clear-water state); State 2 had the greatest range of environmental conditions and contained most the data (i.e., a status-quo state); and States 3, 4, and 5 had extremely high concentrations of suspended solids (i.e., turbid states, with State 5 as the most turbid). The TDA mapped clear patterns of the ecosystem states across several riverine navigation reaches and seasons that furthered ecological understanding. State variables were identified as suspended solids, chlorophyll a , and total phosphorus, which are also state variables of shallow lakes worldwide. The TDA change detection function showed short-term state transitions based on seasonality and episodic events, and provided evidence of gradual, long-term changes due to water quality improvements over three decades. These results can inform decision making and guide actions for regulatory and restoration agencies by assessing the status and trends of this important river and provide quantitative targets for state variables. The TDA change detection function may serve as a new tool for predicting the vulnerability to undesirable state transitions in this system and other ecosystems with sufficient data. Coupling ecosystem state concepts and TDA tools can be transferred to any ecosystem with large data to help classify states and understand their vulnerability to state transitions.

PLOS Computational Biology↗