synpop.survey_qa

synpop.survey_qa.RowCheck dataclass

Describe one row-level survey validation rule.

Parameters:

Name	Type	Description	Default
check_id	str	Stable identifier used in temporary columns and QA logs.	required
expr	Column	Spark boolean expression that is true when a row passes.	required
severity	str	Check category, normally error, warn, or exception.	'error'
field	Optional[str]	Optional source field or field pattern associated with the rule.	None
message	Optional[str]	Optional human-readable failure description.	None
reconciliation_code	Optional[int]	Optional vendor reconciliation reason code.	None
weight	int	Relative rule weight retained for downstream consumers.	1

synpop.survey_qa.apply_row_checks(df, checks, id_cols, max_error_fails=1, dump_if_failed_check_ids=None)

Apply row checks and split survey responses into accepted and dumped sets.

Parameters:

Name	Type	Description	Default
df	DataFrame	Survey response Spark DataFrame. Must contain every column referenced by checks and by id_cols, and must not already contain columns prefixed with __qa_; those names are used internally and dropped before returning.	required
checks	List[RowCheck]	Validation rules to evaluate for every row. Rules with severity error count toward max_error_fails; all other severities are recorded in the log without affecting the dump decision.	required
id_cols	List[str]	Columns copied into the failure log. The log projection currently requires id, vendor, and uid to be present, so passing a different set raises an AnalysisException that is caught and reported as (None, None, None).	required
max_error_fails	int	A row is dumped when its error count is strictly greater than this threshold, so the default of 1 tolerates a single error-level failure.	1
dump_if_failed_check_ids	Optional[List[str]]	Check IDs that always dump a row when failed, regardless of max_error_fails.	None

Returns:

Type	Description
tuple	(good_df, dump_df, logs_df). good_df and dump_df retain the input schema. logs_df holds one row per failed check with columns id, vendor, uid, check_id, error_fail_count, is_dump, severity, field, message, and reconciliation_code. Returns (None, None, None) if any step fails; the error is printed rather than raised.

Examples:

>>> good_df, dump_df, logs_df = apply_row_checks(
...     responses_df,
...     row_checks_list(responses_df),
...     id_cols=["id", "vendor", "uid"],
... )

synpop.survey_qa.make_not_null_checks(cols, *, severity='warn')

Create checks requiring selected columns to be non-null.

Parameters:

Name	Type	Description	Default
cols		Column names to validate.	required
severity		Severity assigned to each generated rule.	'warn'

Returns:

Type	Description
	One RowCheck per column, or an empty list if construction fails.

synpop.survey_qa._try_cast(col_name, spark_type)

Return a Column that safely casts col_name to spark_type.

Parameters:

Name	Type	Description	Default
col_name	str	Source column name. Wrapped in backticks, so names containing spaces or special characters are accepted.	required
spark_type	str	Target Spark SQL type name, for example int or timestamp.	required

Returns:

Type	Description
Column	Expression yielding the cast value, or NULL where the value cannot be cast. Returns a literal NULL column if the expression cannot be built.

synpop.survey_qa._any_column_is_not_null(cols)

Return a type-neutral expression checking if any column has a value.

Parameters:

Name	Type	Description	Default
cols		Column names to test. Each is cast to string first so columns of differing types can be coalesced together.	required

Returns:

Type	Description
Column	Boolean expression that is true when at least one of cols is non-NULL.

synpop.survey_qa.make_nullable_int_checks(cols, *, severity='warn')

Create checks accepting null values or values castable to integers.

Parameters:

Name	Type	Description	Default
cols		Column names to validate.	required
severity		Severity assigned to each generated rule.	'warn'

Returns:

Type	Description
	One RowCheck per column, or an empty list if construction fails.

synpop.survey_qa.make_nullable_float_checks(cols, *, severity='warn')

Create checks accepting null values or values castable to doubles.

Parameters:

Name	Type	Description	Default
cols		Column names to validate.	required
severity		Severity assigned to each generated rule.	'warn'

Returns:

Type	Description
	One RowCheck per column, or an empty list if construction fails.

synpop.survey_qa.make_nullable_timestamp_checks(cols, *, severity='warn')

Create checks accepting null values or values castable to timestamps.

Parameters:

Name	Type	Description	Default
cols		Column names to validate.	required
severity		Severity assigned to each generated rule.	'warn'

Returns:

Type	Description
	One RowCheck per column, or an empty list if construction fails.

synpop.survey_qa.make_nullable_string_checks(cols, *, severity='warn', allow_empty=False)

Create checks accepting null values or valid string values.

Because Spark can cast most values to strings, validation focuses on whether non-null values may be empty after trimming.

Parameters:

Name	Type	Description	Default
cols		Column names to validate.	required
severity		Severity assigned to each generated rule.	'warn'
allow_empty		Whether empty or whitespace-only strings pass.	False

Returns:

Type	Description
	One RowCheck per column, or an empty list if construction fails.

synpop.survey_qa.expand_columns(df, specs)

Expand column specifications that may include wildcard prefix patterns.

Parameters:

Name	Type	Description	Default
df		Source Spark DataFrame whose columns are matched.	required
specs		Exact column names or prefix patterns ending with *.	required

Returns:

Type	Description
	Sorted, deduplicated matching column names. Returns an empty list if
	expansion fails.

Example

expand_columns(df, ["id", "ownership_*"]) includes id and every DataFrame column whose name begins with ownership_.

synpop.survey_qa.row_checks_list(df)

Build the complete list of survey-specific RowCheck rules.

Assembles sanity, attention, consistency, plausibility, value-domain, and data-type checks for the synthetic-population survey DataFrame.

Parameters:

Name	Type	Description	Default
df		Survey Spark DataFrame whose columns are inspected to expand wildcard-based rule groups.	required

Returns:

Type	Description
	Complete list of checks ready for apply_row_checks.

synpop.survey_qa.synpop_qa_survey_test(df, country_code)

Run survey QA and export its artifacts.

The pipeline creates rules, separates accepted and dumped records, writes dump and log CSV files, creates a Cint reconciliation file, and uploads an HTML report to the configured S3 locations.

Parameters:

Name	Type	Description	Default
df		Raw survey response Spark DataFrame.	required
country_code		Country code used in artifact paths and report content.	required

Returns:

Type	Description
	Spark DataFrame containing accepted responses, or None if the
	outer pipeline fails. Individual export failures are logged and do not
	prevent the accepted DataFrame from being returned.