pasteur.dataset.TypedDataset

class pasteur.dataset.TypedDataset(**_)

Extend from to create an intermediary step in ingestion, where the table is loaded from <dataset>.raw@<table> to a parquet one `<dataset>.typed.<table>.

Useful for multiple reads to raw tables. You can also override the type() function to make minor changes to the dataset. By default it’s the identity.

Since parquet files don’t support chunked loading it’s unused.

Attributes

bootstrap	An optional function that is used for one-time tasks (such as extraction).
catalog	A kedro catalog that represents the dataset's sources.
deps	Defines the Tables of the dataset and their dependencies, ex.:
folder_name	Specifies the name of the folder in the raw directory that will be used for the dataset's raw sources.
key_deps	Provides the table dependencies (Table, not raw) that are used to create the keys of the dataset.
name
raw_sources	A raw source that can be used to download the dataset.
raw_tables	Returns the raw dependency names of the dataset.
tables	Returns the table names of the dataset.

Methods

ingest(name, **tables)	Creates the table <name> using the tables provided based on the dependencies.
keys(**tables)	Returns a set of keys which split the current dataset.
type(table)

bootstrap: Optional[Callable[[str, str], None]] = None

An optional function that is used for one-time tasks (such as extraction). Can be run with pasteur bootstrap <dataset_name>.

Is provided with 2 paths: the raw directory of the dataset and another directory dedicated to the dataset named bootstrap. If the dataset has any archives, extract them from the raw directory to bootstrap and then use the bootstrap directory as a base in the catalog.

catalog: dict[str, Any] | str | None = None

A kedro catalog that represents the dataset’s sources. Can be provided as a dictionary to be used as is, or as a filepath, in which case the path will be loaded and processed, by replacing the paths with appropriate ones based on the raw directory and folder name.

deps: dict[str, list[str]] = {}

Defines the Tables of the dataset and their dependencies, ex.:

`python {"table1": ["raw1", "raw2"], "table2": ["raw3", "raw4"]} `

folder_name: str | None = None

Specifies the name of the folder in the raw directory that will be used for the dataset’s raw sources. If the folder does not exist, the dataset is disabled (used for packaging).

ingest(name, **tables)

Creates the table <name> using the tables provided based on the dependencies.

The dependencies may be anything and should be defined in the catalog. The raw tables of a dataset are the only kedro datasets explicitly defined by the user.

Can return a dataframe, callable which produces a dataframe, or dict of callables, dataframes. If it’s a dict, the table will be partitioned using the dict keys.

@warning: all partitioned tables should have the same partitions. Some tables may not be partitioned.

Tip: use a match statement to fork based on table name to per-table functions.

Return type:

Union[LazyDataset[DataFrame], DataFrame]

key_deps: list[str] = []

Provides the table dependencies (Table, not raw) that are used to create the keys of the dataset.

keys(**tables)

Returns a set of keys which split the current dataset.

Keys do not need to be unique per partition, since splitting will also be partition based. Gets a set of table partitions based on key_deps.

Use the to_chunked operator to handle partitions.

Return type:

Union[LazyDataset[DataFrame], DataFrame]

name: str = ''

raw_sources: dict[str, RawSource] | RawSource | None = None

A raw source that can be used to download the dataset.

Optionally, multiple sources can be supplied and downloaded with pasteur download <name>.

property raw_tables

Returns the raw dependency names of the dataset.

property tables

Returns the table names of the dataset.

type(table)