basic_tools

Basic tools utilities, with multi-purpose functions.

This module provides multi-purpose utility functions and classes for resuable operations across various projects.

Requires: - databricks-sdk - dbutils (Databricks utilities) - pandas (for some operations)

Author: Gino F. Fazzi, gino.franco.fazzi@audienceproject.com

basic_tools.BasicLogger

Persist timestamped notebook logs locally and optionally back them up.

Parameters:

Name	Type	Description	Default
instance_id		Identifier used as the log filename. When omitted, an ID is generated from the current timestamp and Databricks notebook name.	None
log_directory		Destination directory used by commit_logs.	's3://ap-analyst/Gino/_system_logs_'
local_temp_dir		Local directory used for the working tab-separated log.	'./temp_logs'
backup_s3		If True, attempt to load an existing log from log_directory; otherwise load it from local_temp_dir.	False

Notes

Construction creates local_temp_dir when needed and immediately calls load_logs. This class expects active Databricks spark and dbutils objects.

basic_tools.BasicLogger.__init__(instance_id=None, log_directory='s3://ap-analyst/Gino/_system_logs_', local_temp_dir='./temp_logs', backup_s3=False)

Initialize a log session and load its existing or new log records.

Parameters:

Name	Type	Description	Default
instance_id		Identifier for this log session. When omitted, one is generated from the current timestamp and the running notebook's name.	None
log_directory		S3 prefix holding persisted log records.	's3://ap-analyst/Gino/_system_logs_'
local_temp_dir		Local staging directory for log files. Created if it does not exist.	'./temp_logs'
backup_s3		Also write a backup copy of the logs to S3.	False

Notes

Calls :meth:load_logs during construction, so instantiating this class reads from log_directory.

basic_tools.BasicLogger.load_logs()

Load an existing log or initialize a new one.

Returns:

Type	Description
	None. The loaded or newly created pandas DataFrame is stored in
	self.logdf.

Notes

When backup_s3 is enabled, a local uncommitted log with the same ID triggers an interactive prompt before it is committed and replaced.

basic_tools.BasicLogger.log(message, category=None, process=None, process_level=None, verbose=True)

Append a message to the working log and persist it locally.

Parameters:

Name	Type	Description	Default
message	str | Series	Message text or pandas Series of messages to append.	required
category		Optional category label. INFO and WARNING use the corresponding colored console helpers.	None
process		Optional process label prepended to string messages.	None
process_level		Optional integer rendered as leading hyphens.	None
verbose		Retained for backward compatibility; console output is always emitted by the current implementation.	True

Returns:

Type	Description
	None.

Raises:

Type	Description
Exception	If message is neither a string nor a pandas Series.

basic_tools.BasicLogger.commit_logs(clean_temp=True)

Write the accumulated log to its destination and optionally clean up.

Parameters:

Name	Type	Description	Default
clean_temp		If True, remove the local log file after the commit attempt and remove the temporary directory when it is empty.	True

Returns:

Type	Description
	None.

Notes

Commit failures are reported as warnings rather than re-raised. Cleanup still runs when clean_temp is True.

basic_tools.send_slack_notification(webhook_url, payload, timeout_seconds=10)

Send a JSON payload to a Slack-compatible incoming webhook.

Parameters:

Name	Type	Description	Default
webhook_url		Non-empty incoming webhook URL.	required
payload		JSON-serializable request body accepted by the webhook.	required
timeout_seconds		Maximum number of seconds to wait for the HTTP request.	10

Raises:

Type	Description
TypeError	If webhook_url is not a string, payload is not a dictionary, or timeout_seconds is not numeric.
ValueError	If webhook_url is empty or timeout_seconds is not positive.
RuntimeError	If the webhook request fails or returns an unsuccessful HTTP status.

Notes

Delivery failures are also reported, when possible, to the RoboGino Logs webhook from the dame-automations secret scope. Failure logging does not suppress the original delivery error.

basic_tools.check_country_code_validity(country_code, criteria='synpop')

Verify a country code is valid according to the selected criteria.

Parameters:

Name	Type	Description	Default
country_code		ISO 3166-1 alpha-2 country code to check.	required
criteria		Reference list to check against. "synpop" uses the countries available in the synthetic population; "all" uses every country in s3://ap-analyst/DAME/_system_/resources/countries-table.csv.	'synpop'

Returns:

Type	Description
bool	True when country_code is present in the selected list.

Raises:

Type	Description
ValueError	If country_code is not in the selected list.
UnboundLocalError	If criteria is neither "synpop" nor "all"; no reference list is built for other values.

basic_tools.get_country_attribute(country_code, attribute)

Get a specific attribute for a given country code.

Parameters:

Name	Type	Description	Default
country_code	str	The ISO2 or ISO3 country code (e.g., "DK", "SE").	required
attribute	str	The attribute to retrieve. Available attributes are: "name", "demonym", "currency", "census_agency".	required

Returns:

Type	Description
str	The value of the requested attribute for the specified country code.

Raises:

Type	Description
ValueError	If the country code or attribute is not found.

basic_tools.cleanhtml(raw_html)

Remove HTML tags from a string.

Parameters:

Name	Type	Description	Default
raw_html	str	The string containing HTML tags.	required

Returns:

Type	Description
str	The input with every <...> tag removed. Tag contents are dropped but the surrounding text is preserved; HTML entities are left as-is.

basic_tools.exit_process(exception_msg, send_slack=True)

Abort the current process, optionally notifying Slack first.

Parameters:

Name	Type	Description	Default
exception_msg		Message used both as the Slack notification body and the exception text.	required
send_slack		Post the message to the Synpop Pipeline Alerts webhook before raising.	True

Raises:

Type	Description
Exception	Always. This function exists to terminate the caller.

Notes

Reads the webhooks secret from the Databricks dame scope even when send_slack is false, so it requires secret-scope access either way.