aws

AWS Cloud Utility API

This module provides utility functions to interact with AWS services.

Functions

Name	Description
`download_s3_file`	Download a file from S3
`download_s3_object`	Download an object from S3
`download_s3_prefix`	Download all objects under an S3 prefix into a local directory
`download_s3_objects`	Download all objects in a S3 bucket with a given prefix (deprecated)

Machine-readable model

loggerattribute
download_s3_filefunctionDownload a file from S3
download_s3_objectfunctionDownload an object from S3
download_s3_objectsfunctionDownload all objects in a S3 bucket with a given prefix.
download_s3_prefixfunctionDownload all objects under an S3 prefix into a local directory.

logger = setup_logger(__name__)

download_s3_file(
    key: str,
    dst: Path,
    bucket: str,
    client: boto3.client = None,
    checksum: str = 'size',
    config: Config | None = Config(signature_version=UNSIGNED),
) -> bool

Download a file from S3

Parameters

Parameters of download_s3_file
Name	Type	Default	Description
`key`	`str`	Required	Object key
`dst`	`Path`	Required	Destination path
`bucket`	`str`	Required	Bucket name
`client`	`boto3.client`	None	S3 client
`checksum`	`str`	'size'	Checksum type. Defaults to "size".
`config`	`Config`	Config(signature_version=UNSIGNED)	Boto3 config. Defaults to Config(signature_version=UNSIGNED).

Returns

Returns of download_s3_file
Value	Type	Description
`bool`	`bool`	True if file was downloaded, False if already exists

download_s3_object(
    item: dict[str, str],
    dst: Path,
    bucket: str,
    client: boto3.client = None,
    checksum: str = 'size',
    config: Config | None = Config(signature_version=UNSIGNED),
) -> bool

Download an object from S3

Parameters

Parameters of download_s3_object
Name	Type	Default	Description
`item`	`dict[str, str]`	Required	Object metadata
`dst`	`Path`	Required	Destination path
`bucket`	`str`	Required	Bucket name
`client`	`boto3.client`	None	S3 client
`checksum`	`str`	'size'	Checksum type. Defaults to "size".
`config`	`Config`	Config(signature_version=UNSIGNED)	Boto3 config. Defaults to Config(signature_version=UNSIGNED).

Returns

Returns of download_s3_object
Value	Type	Description
`bool`	`bool`	True if file was downloaded, False if already exists

download_s3_objects(
    bucket: str,
    prefix: str,
    dst: Path,
    checksum: str = 'size',
    progress: bool = True,
    num_workers: int | None = None,
    config: Config | None = Config(signature_version=UNSIGNED),
)

Download all objects in a S3 bucket with a given prefix.

.. deprecated:: Use :func:download_s3_prefix instead. This function preserves the full S3 key (including the prefix) when building local paths, which causes files to be nested one level too deep when dst already contains the prefix directory. The replacement strips the prefix so that dst is always the root of the downloaded tree.

Parameters

Parameters of download_s3_objects
Name	Type	Default	Description
`bucket`	`str`	Required	Bucket name
`prefix`	`str`	Required	Prefix to filter objects
`dst`	`Path`	Required	Destination directory
`checksum`	`str`	'size'	Checksum type. Defaults to "size".
`progress`	`bool`	True	Show progress bar. Defaults to True.
`num_workers`	`int \| None`	None	Number of workers. Defaults to None.
`config`	`Config \| None`	Config(signature_version=UNSIGNED)	Boto3 config. Defaults to Config(signature_version=UNSIGNED).

download_s3_prefix(
    bucket: str,
    prefix: str,
    dst: Path,
    checksum: str = 'size',
    progress: bool = True,
    num_workers: int | None = None,
    config: Config | None = Config(signature_version=UNSIGNED),
) -> int

Download all objects under an S3 prefix into a local directory.

Unlike :func:download_s3_objects, this function strips the prefix from each object key before joining it with dst, so that dst becomes the root of the downloaded tree.

Example::

S3 objects: s3://my-bucket/datasets/ptbxl/00001.h5

s3://my-bucket/datasets/ptbxl/00002.h5

download_s3_prefix( bucket=“my-bucket”, prefix=“datasets/ptbxl”, dst=Path(“./data/ptbxl”), )

Results in: ./data/ptbxl/00001.h5

./data/ptbxl/00002.h5

Parameters

Parameters of download_s3_prefix
Name	Type	Default	Description
`bucket`	`str`	Required	Bucket name.
`prefix`	`str`	Required	Key prefix to filter objects. A trailing ``/`` is added automatically if missing.
`dst`	`Path`	Required	Local directory that will mirror the contents found under prefix.
`checksum`	`str`	'size'	Checksum strategy (``"size"`` or ``"md5"``). Defaults to ``"size"``.
`progress`	`bool`	True	Show a ``tqdm`` progress bar. Defaults to ``True``.
`num_workers`	`int \| None`	None	Thread-pool size. ``None`` uses the :class:`~concurrent.futures.ThreadPoolExecutor` default.
`config`	`Config \| None`	Config(signature_version=UNSIGNED)	Boto3 client config. Defaults to unsigned requests.

Returns

Returns of download_s3_prefix
Value	Type	Description
`int`	`int`	Number of objects downloaded (excludes skipped / up-to-date).