Scientists increasingly depend on ‘black-box’ tools they cannot control or fully understand  - University of Exeter News

Skip to content

University home

Study

Research

Business

Alumni and supporters

Our departments

About us

Contact Press Office

Home<br>Research news<br>Scientists increasingly depend on ‘black-box’ tools they cannot control or fully understand

Scientists are increasingly relying on powerful data sources and tools that they often cannot fully understand, inspect or verify, according to a new study.

State-of-the-art tools and data like artificial intelligence (AI), satellite imagery, online data and digital sensors are revolutionising the way scientists study the natural world.

But such systems effectively operate as scientific “black boxes” that can increasingly challenge the trust in science.

The new study, by an international team of scientists and available here, addresses the problems of reproducibility, trust and the future of scientific research in an era when critical technologies can shape science and influence knowledge without being fully open to scrutiny.

These technologies can process enormous amounts of information, monitor biodiversity and threats across continents, and reveal patterns that would once have been out of reach.

“However, many of these tools represent true black boxes, by keeping the processes behind those results largely hidden,” said Ivan Jarić, researcher from the University of Paris-Saclay, and lead author of the study.

“They are often owned by private companies that intentionally limit access to information about how their systems operate or process data, guided by proprietary constraints and commercial aims”.

The paper identifies several types of black boxes that are becoming widely used in ecology and conservation.

One of the most prominent examples are large language models and other AI technologies, increasingly used to analyse massive datasets, interpret satellite imagery, and model ecosystems.

However, researchers often have little or no access to the data used to train these systems, the underlying algorithms, direct system testing, or understanding how and why they generate particular outputs.

As AI becomes more capable and autonomous, this lack of transparency will make scientific findings harder to interpret and verify.

This issue extends beyond AI. Many remote sensing products rely on proprietary processing that researchers cannot fully access and verify, while some wildlife tracking devices provide only processed animal locations, while withholding the underlying raw data.

Online platforms such as search engines and social media, which have become valuable sources for studying biodiversity and human interactions with nature, are based on hidden algorithms and changing policies that can introduce unknown biases in such data.

Similar problems are also affecting social surveys. Scientists are increasingly relying on private companies to recruit participants and manage surveys, with often limited information about how respondents are selected, how data quality is maintained, or whether responses may have been affected by AI agent interference.

“This problem is not simply due to commercial and proprietary issues,” said Professor Karen Anderson, from the University of Exeter, another author of the study.

“Modern scientific tools are also becoming so technically complex that users, and in some cases even their developers, may struggle to fully scrutinise and understand how they operate.”

The growing dependence on black-box technologies is further strengthened by a publish-or-perish culture, a growing pressure on scientists to increase productivity and remain competitive, but also by the need to more effectively cope with growing datasets and urgent environmental crises.

Beside the risk of monopoly, impaired efforts towards open science, and susceptibility to manipulation, the researchers caution that this trend could critically undermine overall reproducibility of science.

If key analytical steps cannot be inspected or repeated, confidence in scientific findings may gradually erode.

The authors recommend a number of solutions for making black-box technologies more transparent and accountable.

This includes prioritising open-source software and hardware whenever possible, benchmarking proprietary tools against transparent datasets, comparing results across multiple methods, carefully documenting the training data, pipelines, versions, settings, and especially tool limitations, and ultimately systematic efforts towards a wider awareness and recognition of this problem.

“Human oversight should remain central throughout the research process, especially since it is the study authors who must take responsibility for any errors and uncertainties produced by the use of black-box tools in their work,” said Michael Bertram from the Swedish University of Agricultural Sciences and Stockholm University, another author of the study.

“It is...