Complete API

Reader objects

High-level classes to open and read DarwinCore Archive.

class dwca.read.DwCAReader(path: str, extensions_to_ignore: List[str] = None, tmp_dir: str = None, skip_metadata: bool = False)

Bases: object

This class is used to represent a Darwin Core Archive as a whole.

It gives read access to the contained data, to the scientific metadata, … It supports archives with or without Metafile, such as described on page 2 of the Reference Guide to the XML Descriptor.

Parameters:

path (str) – path to the Darwin Core Archive (either a zip/tgz file or a directory) to open.
extensions_to_ignore (list) – path (relative to the archive root) of extension data files to ignore. This will improve speed and memory usage for large archives. Missing files are silently ignored.
tmp_dir (str) – temporary directory to use to uncompress the archive (if needed). If not provided, Python default will be used.
skip_metadata (bool) – if True, the archive’s scientific metadata file is not parsed and the metadata attribute stays None. This does not affect source_metadata, which is still populated either way. Use this to avoid the parsing cost when only the data rows are needed.

Raises:

dwca.exceptions.InvalidArchive

Raises:

dwca.exceptions.InvalidSimpleArchive

Usage:

from dwca.read import DwCAReader

dwca = DwCAReader('my_archive.zip')

# Iterating on core rows is easy:
for core_row in dwca:
    # core_row is an instance of dwca.rows.CoreRow
    print(core_row)

# Scientific metadata (EML) is available as an ElementTree.Element object
print(dwca.metadata)

# Close the archive to free resources
dwca.close()

The archive can also be opened using the with statement. This is recommended, since it ensures resources will be properly cleaned after usage:

from dwca.read import DwCAReader

with DwCAReader('my-archive.zip') as dwca:
    pass  # Do what you want

# When leaving the block, resources are automatically freed.

absolute_temporary_path(relative_path: str) → str

Return the absolute path of a file located within the archive.

This method allows raw access to the files contained in the archive. It can be useful to open additional, non-standard files embedded in the archive, or to open a standard file with another library.

Parameters:: relative_path (str) – the path (relative to the archive root) of the file.
Returns:: the absolute path to the file.

Usage:

dwca.absolute_temporary_path('occurrence.txt')  # => /tmp/afdfsec7/occurrence.txt

Warning

If the archive is contained in a zip or tgz file, the returned path will point to a temporary file that will be removed when closing the dwca.read.DwCAReader instance.

Note

File existence is not tested.

archive_path: str: The path to the Darwin Core Archive file, as passed to the constructor.

close() → None

Close the Darwin Core Archive and remove temporary/working files.

Note

Alternatively, DwCAReader can be instanciated using the with statement. (see example above).

core_contains_term(term_url: str) → bool: Return True if the Core file of the archive contains the term_url term.

core_file: CSVDataFile: An instance of dwca.files.CSVDataFile for the core data file.

property core_file_location: str

The (relative) path to the core data file.

Example: ‘occurrence.txt’

descriptor: ArchiveDescriptor | None: An descriptors.ArchiveDescriptor instance giving access to the archive descriptor/metafile (meta.xml)

extension_files: List[CSVDataFile]: A list of dwca.files.CSVDataFile, one entry for each extension data file , sorted by order of appearance in the Metafile (or an empty list if the archive doesn’t use extensions).

get_corerow_by_id(row_id: str) → CoreRow

Return the (core) row whose ID is row_id.

Parameters:: row_id (str) – ID of the core row you want
Returns:: dwca.rows.CoreRow – the matching row.
Raises:: dwca.exceptions.RowNotFound

Warning

It is rarely a good idea to rely on the row ID, because: 1) Not all Darwin Core Archives specifies row IDs. 2) Nothing guarantees that the ID will actually be unique within the archive (depends of the data publisher). In that case, this method don’t guarantee which one will be returned. get_corerow_by_position() may be more appropriate in this case.

get_corerow_by_position(position: int) → CoreRow

Return a core row according to its position/index in core file.

Parameters:: position (int) – the position (starting at 0) of the row you want in the core file.
Returns:: dwca.rows.CoreRow – the matching row.
Raises:: dwca.exceptions.RowNotFound

Note

The position is often an appropriate way to unambiguously identify a core row in a DwCA.

get_descriptor_for(relative_path: str) → DataFileDescriptor

Return a descriptor for the data file located at relative_path.

Parameters:: relative_path (str) – the path (relative to the archive root) to the data file you want info about.
Returns:: dwca.descriptors.DataFileDescriptor
Raises:: dwca.exceptions.NotADataFile if relative_path doesn’t reference a valid data file.

Examples:

dwca.get_descriptor_for('occurrence.txt')
dwca.get_descriptor_for('verbatim.txt')

iter_terms(terms: List[str]) → Iterator[Tuple[str, ...]]

Yield one tuple of values per core row, holding terms in the order given.

A faster alternative to iterating over the reader when only a few terms are needed. See dwca.files.CSVDataFile.iter_terms().

Usage:

for identifier, latitude, longitude in dwca.iter_terms(
        [qn('occurrenceID'), qn('decimalLatitude'), qn('decimalLongitude')]):
    pass

The special name “id” requests the core file’s id column - the same name used by dwca.descriptors.DataFileDescriptor.headers. It resolves even when the Metafile declares no <field> for that column, which is the common case. If the core file declares an actual term named “id” (possible in metafile-less archives, where terms are raw CSV header names), that declared term takes precedence, which matches what CoreRow.data[‘id’] already returns.

Parameters:: terms – a list of full term identifiers.
Raises:: ValueError – if any of terms is not present in the core data file.

metadata: Element | None: A xml.etree.ElementTree.Element instance containing the (scientific) metadata of the archive, or None if the archive has no metadata or if the skip_metadata parameter is True.

next() → CoreRow: Return the next core row.

Deprecated since version Iterate: over the reader instead. This method keeps its own implicit iterator, which is independent of any for row in reader: loop: interleaving the two makes each of them scan the archive separately. Once this iterator is exhausted it raises StopIteration, and the call after that starts a fresh pass from the first row rather than raising again.

open_included_file(relative_path: str, *args: Any, **kwargs: Any) → IO

Simple wrapper around Python’s build-in open function.

To be used only for reading.

Warning

Don’t forget to close the files after usage. This is especially important on Windows because temporary (extracted) files won’t be cleanable if not closed.

orphaned_extension_rows() → Dict[str, Dict[str, List[int]]]

Return a dict of the orphaned extension rows.

Orphaned extension rows are extension rows who reference non-existing core rows. This methods returns a dict such as:

{'description.txt': {u'5': [3, 4], u'6': [5]},
 'vernacularname.txt': {u'7': [4]}}

Meaning:

in description.txt, rows at position 3 and 4 reference a core row whose ID is ‘5’, but such a core row doesn’t exists. Row at position 5 references an imaginary core row with ID ‘6’

in vernacularname.txt, the row at position 4 references an imaginary core row with ID ‘7’

pd_read(relative_path, **kwargs)

Return a Pandas DataFrame for the data file located at relative_path.

This method wraps pandas.read_csv() and accept the same keyword arguments. The following arguments will be ignored (because they are set appropriately for the data file): delimiter, skiprows, header and names.

Parameters:: relative_path (str) – path to the data file (relative to the archive root).
Raises:: ImportError if Pandas is not installed.
Raises:: dwca.exceptions.NotADataFile if relative_path doesn’t designate a valid data file in the archive.

Warning

You’ll need to install Pandas before using this method.

Note

Default values of Darwin Core Archive are supported: A column will be added to the DataFrame if a term has a default value in the Metafile (but no corresponding column in the CSV Data File). This is unfortunately not supported in case the value returned by pandas.read_csv() is a TextFileReader (e.g. when using chunksize or iterator=True).

property rows: List[CoreRow]

A list of rows.CoreRow objects representing the content of the archive.

The list is in order of appearance in the core data file, the same order produced by iterating the reader.

Warning

All rows will be loaded in memory. In case of a large Darwin Core Archive, you may prefer using a for loop.

source_metadata: Dict[str, Element]

If the archive contains source-level metadata (typically, GBIF downloads), this is a dict such as:

{'dataset1_UUID': <dataset1 EML> (xml.etree.ElementTree.Element object),
 'dataset2_UUID': <dataset2 EML> (xml.etree.ElementTree.Element object), ...}

See The GBIF Occurrence download format for more details.

property use_extensions: bool: True if the archive makes use of extensions.

Row objects

Objects that represents data rows coming from DarwinCore Archives.

class dwca.rows.CoreRow(csv_line: str, position: int, datafile_descriptor: DataFileDescriptor)

Bases: Row

This class is used to represent a row/line from a Darwin Core Archive core data file.

You probably won’t instantiate it manually but rather obtain it via dwca.read.DwCAReader.get_corerow_by_position(), dwca.read.DwCAReader.get_corerow_by_id() or simply by looping over a dwca.read.DwCAReader object.

property extensions: A list of ExtensionRow instances that relates to this Core row.

class dwca.rows.ExtensionRow(csv_line: str, position: int, datafile_descriptor: DataFileDescriptor)

Bases: Row

This class is used to represent a row/line from a Darwin Core Archive extension data file.

Most of the time, you won’t instantiate it manually but rather obtain it trough the extensions attribute of CoreRow.

class dwca.rows.Row(csv_line: str, position: int, datafile_descriptor: DataFileDescriptor)

Bases: object

This class is used to represent a row/line in a Darwin Core Archive.

This class is intended to be subclassed rather than used directly.

classmethod from_fields(raw_fields: List[str], position: int, datafile_descriptor: DataFileDescriptor) → Row

Build a Row from an already-split data row.

This is the constructor used by the streaming engine. __init__(), which takes a raw CSV line, is kept for backwards compatibility.

dwca.rows.csv_line_to_fields(csv_line, line_ending, field_ending, fields_enclosed_by)

Split a line from a CSV file.

Return a list of fields. Content is not trimmed.

Descriptor objects

Classes to represents descriptors of a DwC-A.

ArchiveDescriptor represents the full archive descriptor, initialized from the metafile content.
DataFileDescriptor describes characteristics of a given data file in the archive. It’s either created from a subsection of the ArchiveDescriptor describing the data file, either by introspecting the CSV data file (useful for Archives without metafile).

class dwca.descriptors.ArchiveDescriptor(metaxml_content: str, files_to_ignore: List[str] = None)

Bases: object

Class used to encapsulate the whole Metafile (meta.xml).

extensions: List[DataFileDescriptor]: A list of dwca.descriptors.DataFileDescriptor instances describing each of the archive’s extension data files.

extensions_type

A list of extension (types) in use in the archive.

Example:

["http://rs.gbif.org/terms/1.0/VernacularName",
 "http://rs.gbif.org/terms/1.0/Description"]

metadata_filename: The path (relative to archive root) of the (scientific) metadata of the archive.

raw_element: Element: A xml.etree.ElementTree.Element instance containing the complete Archive Descriptor.

class dwca.descriptors.DataFileDescriptor(created_from_file: bool, raw_element: Element, represents_corefile: bool, datafile_type: str | None, file_location: str, file_encoding: str, id_index: int, coreid_index: int, fields: List[Dict], lines_terminated_by: str, fields_enclosed_by: str, fields_terminated_by: str)

Bases: object

Those objects describe a data file fom the archive.

They’re generally not instanciated manually, but rather by calling:

make_from_metafile_section() (if the archive contains a metafile)

make_from_file() (created by analyzing the data file)

coreid_index: If the section represents an extension data file, the index/position of the core_id column in that file. The core_id in an extension is the foreign key to the “extended” core row.

created_from_file: True if this descriptor was created by analyzing the data file.

property field_plan: FieldPlan: A cached FieldPlan turning a split data row into a term -> value dict.

fields

A list of dicts where each entry represent a data field in use.

Each dict contains:

The term identifier
(Possibly) a default value
The column index/position in the CSV file (except if we use a default value instead)

Example:

[{'term': 'http://rs.tdwg.org/dwc/terms/scientificName',
  'index': '1',
  'default': None},

 {'term': 'http://rs.tdwg.org/dwc/terms/locality',
  'index': '2',
  'default': ''},

 # The data for `country` is a the default value 'Belgium' for all rows, so there's
 # no column in CSV file.

 {'term': 'http://rs.tdwg.org/dwc/terms/country',
  'index': None,
  'default': 'Belgium'}]

fields_enclosed_by: The string or character used to enclose fields in the data file.

fields_terminated_by: The string or character used as a field separator in the data file. Example: “\t”.

file_encoding: The encoding of the data file. Example: “utf-8”.

file_location: str: The data file location, relative to the archive root.

property headers: List[str]

A list of (ordered) column names that can be used to create a header line for the data file.

Example:

['id', 'http://rs.tdwg.org/dwc/terms/scientificName', 'http://rs.tdwg.org/dwc/terms/basisOfRecord',
'http://rs.tdwg.org/dwc/terms/family', 'http://rs.tdwg.org/dwc/terms/locality']

See also short_headers if you prefer less verbose headers.

id_index: If the section represents a core data file, the index/position of the id column in that file.

lines_terminated_by: The string or character used as a line separator in the data file. Example: “\n”.

property lines_to_ignore: int: Return the number of header lines/lines to ignore in the data file.

classmethod make_from_file(datafile_path)

Create and return a DataFileDescriptor by analyzing the file at datafile_path.

Parameters:: datafile_path (str) – Relative path to a data file to analyze in order to instantiate the descriptor.

classmethod make_from_metafile_section(section_tag)

Create and return a DataFileDescriptor from a metafile <section> tag.

Parameters:: section_tag (xml.etree.ElementTree.Element) – The XML Element section containing details about the data file.

raw_element: The <section> element describing the data file, from the metafile. None if the archive contains no metafile.

represents_corefile: True if this descriptor is used to represent the core file an archive.

represents_extension: True if this descriptor is used to represent an extension file in an archive.

property short_headers: List[str]

A list of (ordered) column names (short version) that can be used to create a header line for the data file.

Example:

['id', 'scientificName', 'basisOfRecord', 'family', 'locality']

File objects

File-related classes and functions.

class dwca.files.CSVDataFile(work_directory: str, file_descriptor: DataFileDescriptor)

Object used to access a DwCA-enclosed CSV data file.

Parameters:

work_directory – absolute path to the target directory (archive content, previously extracted if necessary).
file_descriptor – an instance of dwca.descriptors.DataFileDescriptor describing the data file.

The file content can be accessed:

By iterating on this object: a str is returned, including separators.
With get_row_by_position() (A dwca.rows.CoreRow or dwca.rows.ExtensionRow object is returned)
For an extension data file, with get_all_rows_by_coreid() (A dwca.rows.CoreRow or dwca.rows.ExtensionRow object is returned)

An index of line offsets is built on first random access (get_row_by_position() or get_all_rows_by_coreid()). This may take time for large files, but makes further random access faster.

close() → None

Close the file.

The content of the file will not be accessible in any way afterwards.

property coreid_index: Dict[str, array]

An index of the core rows referenced by this data file.

It is a Python dict such as:

{
core_id1: [1],    # Row at position 1 references a Core Row whose ID is core_id1
core_id2: [8, 10] # Rows at position 8 and 10 references a Core Row whose ID is core_id2
}

Warning

for performance reasons, dictionary values are arrays(‘L’) instead of regular python lists

Warning

Creating this index can be time and memory consuming for large archives, so it’s created on the fly at first access.

file_descriptor: DataFileDescriptor: An instance of dwca.descriptors.DataFileDescriptor, as given to the constructor.

get_all_rows_by_coreid(core_id: int) → List[Row]

Return the rows whose linking id matches core_id.

For an extension file, this is the row’s coreid field. For a core file, it is the row’s own id, since coreid_index then maps each row’s id to its position. The return type is List[Row] to cover both dwca.rows.CoreRow (core file) and dwca.rows.ExtensionRow (extension file). Returns an empty list if core_id is not found.

get_row_by_position(position: int) → CoreRow | ExtensionRow

Return the row at position in the file.

Header lines are ignored.

Raises:: IndexError if there’s no line at position.

iter_rows() → Iterator[CoreRow | ExtensionRow]

Yield every data row of the file, in order of appearance.

Unlike repeated get_row_by_position() calls this is a single forward pass and never touches the line offset index.

iter_terms(terms: List[str]) → Iterator[Tuple[str, ...]]

Yield one tuple of values per data row, holding terms in the order given.

This is a faster alternative to iterating over rows for consumers that only need a few of the file’s columns: neither a dwca.rows.Row object nor its term to value dict is built. A single term still yields a one-element tuple.

Usage:

for locality, latitude in data_file.iter_terms([qn('locality'),
                                                qn('decimalLatitude')]):
    pass

The special names “id” and “coreid” request the file’s key column - the same names used by dwca.descriptors.DataFileDescriptor.headers. “id” resolves for a core file, “coreid” for an extension file; the other one raises, as does either name when the file has no such column (metafile-less archives have neither). If the file declares an actual term of the same name, that declared term takes precedence, which matches what Row.data[‘id’] already returns.

Parameters:: terms – a list of full term identifiers.
Raises:: ValueError – if any of terms is not present in this data file.

lines_to_ignore: int: Number of lines to ignore (header lines) in the CSV file.

next() → str

Return the next raw line of the data file, including its line terminator.

Header lines are skipped. Raises StopIteration once the file is exhausted. This is the iterator protocol behind for line in data_file, described in the class docstring.

Star record objects

class dwca.star_record.StarRecordIterator(files_to_join: List[CSVDataFile], how: Literal['inner', 'outer'] = 'inner')

Object used to iterate over multiple DWCA-files joined on the coreid

Parameters:

files_to_join – a list of the dwca.files.CSVDataFile objects to join. May or may not include the core file (the core is not treated in a special way)
how – indicates the type of join. “inner” and “outer” correspond vaguely to inner and full joins. The outer join includes rows that don’t match on all files, however, it doesn’t create null fields to fill in when rows are missing in files. Attempts to conform to pandas.DataFrame.merge API.

Helpers

This module contains small helpers to make life easier.

dwca.darwincore.utils.qualname(short_term)

Takes a darwin core term (short form) and returns the corresponding qualname.

Note

It is generally used to make data access less verbose (see example below).

Raises:: StopIteration if short_term is not found.

Typical real-world example:

from dwca.darwincore.utils import qualname as qn

qn("Occurrence")  # => "http://rs.tdwg.org/dwc/terms/Occurrence"

# To access data row:
myrow.data[qn('scientificName')]  # => u"Tetraodon fluviatilis"

# Instead of the verbose:
myrow.data['http://rs.tdwg.org/dwc/terms/scientificName']  # => u"Tetraodon fluviatilis"

Exceptions

Exceptions for the whole package.

exception dwca.exceptions.InvalidArchive: The archive appears to be invalid.

exception dwca.exceptions.InvalidSimpleArchive: The simple archive appears to be invalid.

exception dwca.exceptions.NotADataFile: The file doesn’t exist or is not a data file.

exception dwca.exceptions.RowNotFound: The DwC-A Row cannot be found.