`s3_tools`

S3 utilities for reading and writing data to AWS S3.

This module provides utility functions for interacting with S3 buckets, including JSON file operations, file/folder management, and data transfer.

Requires: - databricks-sdk - dbutils (Databricks utilities) - boto3 (for direct S3 operations) - pandas (for some operations)

Author: Gino F. Fazzi, gino.franco.fazzi@audienceproject.com

`s3_tools.dbutils_walk(path)`

Recursively walk a DBFS or S3 directory, like os.walk for Databricks.

Author: Gino F. Fazzi, gino.franco.fazzi@audienceproject.com Date: 2026-06-23

Parameters:

Name	Type	Description	Default
`path`		Directory to walk, for example `s3://bucket/prefix/`.	required

Yields:

Type	Description
`tuple`	`(path, dirs, files)` for each visited directory, where `dirs` and `files` hold fully qualified child paths. Directories are visited top-down, so the caller sees a parent before its children.

Notes

Uses dbutils.fs.ls, so it requires an active Databricks environment and issues one listing call per directory.

`s3_tools.read_json_from_s3(path)`

Read a JSON file from S3 and return as a Python dictionary.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path to the JSON file on S3. Must start with 's3://'.	required

Returns:

Name	Type	Description
`dict`		Dictionary containing the data from the JSON file.

Raises:

Type	Description
`ValueError`	If path doesn't start with 's3://' or end with '.json'.
`FileNotFoundError`	If file not found at path.
`ValueError`	If path points to a folder instead of a file.

Note

dbutils.fs.head() has size limitations, so this function reads the full file size first before loading to ensure complete data.

Created

2023-11-30

Last Modified: 2024-08-15

Example

data = read_json_from_s3("s3://my-bucket/data.json") print(data)

`s3_tools.path_exists_in_s3(s3_path)`

Return whether a file or directory exists in S3, like os.path.exists.

Parameters:

Name	Type	Description	Default
`s3_path`		Path to test, for example `s3://bucket/prefix/file.json`.	required

Returns:

Type	Description
`bool`	`True` if `dbutils.fs.ls` can list the path.

Notes

Any listing failure is reported as False, so a permissions error is indistinguishable from a missing path.

`s3_tools.write_json_to_s3(data, save_path, overwrite=False, indent=4)`

Write a JSON-like object to S3 as a JSON file.

Parameters:

Name	Type	Description	Default
`data`	`dict`	Dictionary object to save as JSON.	required
`save_path`	`str`	S3 path where to save the file. Must start with 's3://'. The '.json' extension is automatically added if not present.	required
`overwrite`	`bool`	Whether to overwrite existing file. Defaults to False.	`False`
`indent`	`int`	Indentation width passed to `json.dumps`. Defaults to 4.	`4`

Raises:

Type	Description
`ValueError`	If data is not a dict, path format is invalid, or file exists and overwrite is False.
`FileExistsError`	If file already exists at save_path and overwrite is False.

Note

Input must be a dict (not nested list or other JSON types)
The '.json' extension is automatically added if missing

Created

2023-03-20

Last Modified: 2024-08-15

Example

write_json_to_s3( {"key": "value", "nested": {"data": 123}}, "s3://my-bucket/output" )

`s3_tools.read_delta_from_s3(s3_uri)`

Load a Delta table stored at an S3 location.

Parameters:

Name	Type	Description	Default
`s3_uri`	`str`	Delta table location in `s3://bucket/prefix` form.	required

Returns:

Type	Description
`DataFrame`	Spark DataFrame representing the Delta table.

Raises:

Type	Description
`ValueError`	If the URI is not a bucket and prefix location.
`AnalysisException`	If the location is not a readable Delta table.

`s3_tools.write_spark_df_to_single_json(df, s3_uri, overwrite=False)`

Write a Spark DataFrame to an explicitly named JSON S3 location.

Spark writes JSON as a directory containing part files. This helper stages a single partition in a temporary directory, copies its JSON part to the requested S3 object, and removes the temporary directory. The resulting file is newline-delimited JSON, with one DataFrame row per line.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	Spark DataFrame to write.	required
`s3_uri`	`str`	S3 URI ending in a JSON filename, for example `s3://bucket/exports/responses.json`.	required
`overwrite`	`bool`	Whether to overwrite the file if it already exists.	`False`

Raises:

Type	Description
`ValueError`	If the S3 URI is invalid or does not contain a JSON filename.
`FileExistsError`	If the destination already exists and `overwrite` is false.

`s3_tools.export_delta_to_single_json(delta_s3_uri, json_s3_uri, overwrite=False)`

Export a Delta table from S3 as one newline-delimited JSON file in S3.

Parameters:

Name	Type	Description	Default
`delta_s3_uri`	`str`	Source Delta table location in `s3://bucket/prefix` form.	required
`json_s3_uri`	`str`	Destination JSON object URI ending in `.json`.	required
`overwrite`	`bool`	Whether to replace an existing destination file.	`False`

Raises:

Type	Description
`ValueError`	If either path is invalid for its operation.
`FileExistsError`	If the destination exists and `overwrite` is false.