pasteur.dataset.TabularDataset

class pasteur.dataset.TabularDataset(**_)

Boilerplate for a tabular dataset. Assumes the dataset contains one table named table, the index of which is the keys.

By default, assumes one raw table raw@table, exists. If the data is concatenated from multiple sources, deps can be modified to reflect this.

Attributes

bootstrap	An optional function that is used for one-time tasks (such as extraction).
catalog	A kedro catalog that represents the dataset's sources.
deps	Defines the Tables of the dataset and their dependencies, ex.:
folder_name	Specifies the name of the folder in the raw directory that will be used for the dataset's raw sources.
key_deps	Provides the table dependencies (Table, not raw) that are used to create the keys of the dataset.
name
raw_sources	A raw source that can be used to download the dataset.
raw_tables	Returns the raw dependency names of the dataset.
tables	Returns the table names of the dataset.

Methods

ingest(name, **tables)	Creates the table <name> using the tables provided based on the dependencies.
keys(**tables)	Returns a DataFrame containing only the index column of table "table".

bootstrap: Optional[Callable[[str, str], None]] = None

An optional function that is used for one-time tasks (such as extraction). Can be run with pasteur bootstrap <dataset_name>.

Is provided with 2 paths: the raw directory of the dataset and another directory dedicated to the dataset named bootstrap. If the dataset has any archives, extract them from the raw directory to bootstrap and then use the bootstrap directory as a base in the catalog.

catalog: dict[str, Any] | str | None = None

A kedro catalog that represents the dataset’s sources. Can be provided as a dictionary to be used as is, or as a filepath, in which case the path will be loaded and processed, by replacing the paths with appropriate ones based on the raw directory and folder name.

deps: dict[str, list[str]] = {'table': ['table']}

Defines the Tables of the dataset and their dependencies, ex.:

`python {"table1": ["raw1", "raw2"], "table2": ["raw3", "raw4"]} `

folder_name: str | None = None

Specifies the name of the folder in the raw directory that will be used for the dataset’s raw sources. If the folder does not exist, the dataset is disabled (used for packaging).

ingest(name, **tables)

Creates the table <name> using the tables provided based on the dependencies.

The dependencies may be anything and should be defined in the catalog. The raw tables of a dataset are the only kedro datasets explicitly defined by the user.

Can return a dataframe, callable which produces a dataframe, or dict of callables, dataframes. If it’s a dict, the table will be partitioned using the dict keys.

@warning: all partitioned tables should have the same partitions. Some tables may not be partitioned.

Tip: use a match statement to fork based on table name to per-table functions.

key_deps: list[str] = ['table']

Provides the table dependencies (Table, not raw) that are used to create the keys of the dataset.

keys(**tables)

Returns a DataFrame containing only the index column of table “table”.

Return type:

DataFrame

name: str = ''

raw_sources: dict[str, RawSource] | RawSource | None = None

A raw source that can be used to download the dataset.

Optionally, multiple sources can be supplied and downloaded with pasteur download <name>.

property raw_tables

Returns the raw dependency names of the dataset.

property tables

Returns the table names of the dataset.