Labelled multi-dimensional arrays for representing research data | djmannion.net

Research data are often represented in tabular form as a set of rows and columns, similar to a spreadsheet, and stored in a comma-separated value (CSV) file.<br>However, a multi-dimensional structure can also provide a powerful and intuitive data representation for many research contexts.<br>In the Python ecosystem, numpy (Harris et al., 2020) provides a wonderful way of computing with multi-dimensional arrays and the xarray package (Hoyer & Hamman, 2017) builds on numpy to allow multi-dimensional arrays to have labelled dimensions and additional metadata.

Here, I aim to walk through my process of converting an example dataset from a tabular CSV file into a multi-dimensional xarray data representation.<br>I will use the data from the recent study by Graham & Hobaiter (2023) titled &ldquo;Towards a great ape dictionary: Inexperienced humans understand common nonhuman ape gestures&rdquo;.<br>I chose this dataset because its associated article was in the media when I was preparing this post, it is open-access, it is a relatively large dataset with a few interesting features and quirks, it is a bit of a different sort of perception study, and I thought it might be fun to work through a Bayesian analysis in a future post.

It is unfortunately still not as common as it should be for the raw data to be available with a published article—thanks to the authors for doing so here!

Briefly, the experiment involved asking human participants to identify which of four potential &lsquo;meanings&rsquo; were associated with a video of an ape making a gesture.<br>There were 10 different gesture types and 2 different species (chimpanzee and bonobo), and each participant completed the 20 trials corresponding to all combinations of these two conditions.<br>Participants either saw just the video or the video with an additional line of contextual information, in a between-subjects manipulation.<br>The key experimental question of interest concerned the accuracy with which participants could correctly identify the meaning that was associated with each gesture.

Accessing the raw data

The authors have made the data available in a repository on zenodo, so we could just download it from there and put it in a known location on our file system.<br>Here I am going to instead experiment with using the pooch package (Uieda et al., 2020) to download and manage the data.<br>The advantage of this approach is that we make it explicit in the code where the raw data are coming from, and the hash-based integrity verification in pooch means that we only download the file as necessary and we can be confident that it matches our expectations.

We can access the raw data in this fashion via:

import pooch

def get_raw_data_path():

url = (<br>"https://zenodo.org/record/7352326/files/"<br>+ "Wild-Minds/GreatApeDictionary-v1.1.zip"

raw_data_path = pooch.retrieve(<br>url=url, known_hash="md5:8c40d1bce25619548f4daa16d63f36a3"

return raw_data_path

Ideally, we would use the functionality in pooch to download using a DOI rather than a HTTPS URL.<br>However, there is a problem with the current version (when writing this post) of pooch with accessing files in some repositories; I&rsquo;ve opened an issue.

The zip file containing the raw data will then be downloaded to a cache directory (if not already present), and its location returned in the raw_data_path variable.

Reading the raw tabular data

Now, we will define a get_rows function that reads each row of the CSV file that is contained within the raw data archive.<br>We define it as a generator function, which means that it contains a yield statement within it.<br>This makes it behave as an iterator, returning one row at a time to its caller when iterated over—which is really neat for this use case.<br>The row that is returned has been parsed by the csv built-in module and is in the form of a dictionary where the keys are the column names and the values are the associated column values.

Most of the code in the get_rows function handles the parsing of the zip file that is necessary to get to the CSV file.<br>Well, zip files plural, since the CSV file is within a zip file within the zip file.

See my previous post Benefits of storing and accessing research data in zip files for more on processing zip files in Python.

import zipfile<br>import csv<br>import io

def get_rows():

raw_data_path = get_raw_data_path()

with zipfile.ZipFile(file=raw_data_path) as outer_zip:

# path of the zip file containing the CSV, within the zip file<br>csv_zip_path = "Wild-Minds-GreatApeDictionary-4ed8d77/GAD_FullDataset.csv.zip"

with zipfile.ZipFile(file=outer_zip.open(name=csv_zip_path)) as csv_zip:

# name of the CSV file within the zip file<br>csv_path = "GAD_FullDataset.csv"

with csv_zip.open(csv_path) as csv_handle:

reader = csv.DictReader(io.TextIOWrapper(csv_handle))

for row in reader:<br>yield row

We can have a look at its representation of the first data row by running next(get_rows()) in...