synpop.population_qa

Population validation and review-report helpers for SynPop datasets.

synpop.population_qa.BasePop

Read the latest synthetic or demographic population artifact.

Notes

Construction reads the latest metadata to discover geographic columns, so instantiating this class performs a Spark read. The artifact DataFrame itself is loaded lazily and cached on the object by :meth:population_df.

synpop.population_qa.BasePop.__init__(country_code, version, folder, population_path='s3://ap-synthetic-population/prod/review')

Configure the latest dated population artifact for loading.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition to load, for example "DE".	required
version	str	Population version partition to load.	required
folder	str	Artifact folder below the latest dated partition, for example "artifact=synthetic_population".	required
population_path	str	Root location containing the data and metadata prefixes.	's3://ap-synthetic-population/prod/review'

Raises:

Type	Description
FileNotFoundError	If no complete dated partition exists below the resolved country and version path, propagated from :func:helpers.find_latest_date_partition.

synpop.population_qa.BasePop.population_df()

Load and cache the latest population artifact as a Spark DataFrame.

Returns:

Type	Description
	Spark DataFrame read from the artifact folder configured by the
	subclass.

synpop.population_qa.BasePop.grouped(column_list=None)

Count population records overall or by selected dimensions.

Parameters:

Name	Type	Description	Default
column_list		Optional grouping columns. The special value geo expands to the geographic columns discovered from population metadata. When omitted, a single total row is returned.	None

Returns:

Type	Description
DataFrame	Spark DataFrame containing the grouping columns and integer N counts. Age and gender helper columns are added by :func:demographic_tools.with_age_gender_columns before grouping, so age_group and gender_label are also valid grouping keys.

synpop.population_qa.BasePop.total_population()

Return the total number of records in the population artifact.

Returns:

Type	Description
	Integer population count, or zero when the aggregate is null.

synpop.population_qa.BasePop.online_population_percentage()

Return the percentage of records where online penetration is observed.

Returns:

Type	Description
	Float percentage, or zero when the aggregate is null.

synpop.population_qa.DemoPop

Bases: BasePop

Read the latest demographic-population artifact.

synpop.population_qa.DemoPop.__init__(country_code, version)

Configure access to the demographic-population artifact.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition to load, for example "DE".	required
version	str	Population version partition to load.	required

Raises:

Type	Description
FileNotFoundError	If no complete dated partition exists for the country and version.

synpop.population_qa.SynPop

Bases: BasePop

Read the latest synthetic-population artifact.

synpop.population_qa.SynPop.__init__(country_code, version)

Configure access to the synthetic-population artifact.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition to load, for example "DE".	required
version	str	Population version partition to load.	required

Raises:

Type	Description
FileNotFoundError	If no complete dated partition exists for the country and version.

synpop.population_qa.OnlinePop

Look up the latest internet-usage percentage for a country.

Notes

The source uses ISO alpha-3 country codes. This class resolves the supplied alpha-2 code through :func:basic_tools.get_country_attribute and returns the newest non-null annual value from the source row.

synpop.population_qa.OnlinePop.__init__(country_code, source_path=DEFAULT_SOURCE_PATH)

Configure the ITU source and country whose usage rate is requested.

Parameters:

Name	Type	Description	Default
country_code	str	ISO 3166-1 alpha-2 country code, for example "DE". Upper-cased before use.	required
source_path	str	S3 location of the ITU internet-usage Parquet dataset. The dataset must expose a Country Code column holding alpha-3 codes and one column per year.	DEFAULT_SOURCE_PATH

synpop.population_qa.OnlinePop.online_population_percentage()

Return the latest available internet-usage percentage.

Returns:

Type	Description
float	Latest non-null annual percentage of individuals using the
float	internet for the configured country.

Raises:

Type	Description
ValueError	If the source schema lacks a country-code or annual column, or if no value is available for the country.

synpop.population_qa.CensusPop

Read current-year UN World Population Prospects population totals.

Notes

The WPP file contains one row per age and sex for each country and period. :meth:population aggregates those rows for the current calendar year. Values retain the units used by the WPP source (thousands of people).

synpop.population_qa.CensusPop.__init__(country_code, source_path=DEFAULT_SOURCE_PATH)

Configure the WPP source and country whose population is requested.

Parameters:

Name	Type	Description	Default
country_code	str	ISO 3166-1 alpha-2 country code, for example "DE". Upper-cased before use and matched against the source ISO2_code column.	required
source_path	str	S3 location of the compressed WPP single-age dataset. The dataset must contain the columns listed in :attr:REQUIRED_COLUMNS.	DEFAULT_SOURCE_PATH

synpop.population_qa.CensusPop.population()

Return current-year total, male, and female population values.

Returns:

Type	Description
Dict[str, Union[int, float]]	Dictionary with period, total_population,
Dict[str, Union[int, float]]	male_population, and female_population. Population values
Dict[str, Union[int, float]]	are expressed in thousands, matching the WPP source.

Raises:

Type	Description
ValueError	If required source columns or current-year data for the requested country are unavailable.

synpop.population_qa.ColumnExpectation dataclass

Describe schema and value constraints for a population column.

Parameters:

Name	Type	Description	Default
dtype	Union[DataType, str]	Expected Spark data type or normalized type name.	required
nullable	Optional[bool]	Whether null values are allowed. None disables the schema-nullability comparison.	False
required	bool	Whether absence of the column is an error.	True
valid_values	Optional[Union[Tuple[Any, Any], List[Any], Set[Any]]]	Optional allowed set/list or inclusive (min, max).	None

synpop.population_qa.validate_syn_vs_demo(demo_grouped, syn_grouped, internal_log, geo_columns, assert_flag=False)

Compare demographic and synthetic totals and grouped counts.

Parameters:

Name	Type	Description	Default
demo_grouped		Demographic Spark DataFrame with dimensions and N.	required
syn_grouped		Synthetic Spark DataFrame with dimensions and N.	required
internal_log		Existing human-readable validation log.	required
geo_columns		Geographic dimensions included in the comparison key.	required
assert_flag		Raise assertions instead of only appending failures.	False

Returns:

Type	Description
	Tuple containing the combined pass status and updated log text.

Raises:

Type	Description
AssertionError	If a population mismatch exists and assert_flag is enabled.

synpop.population_qa.freq_columns(dtype=DoubleType(), nullable=False, required=True, platforms=PLATFORMS)

Build expectations for freq_<platform> columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	DoubleType()
nullable		Whether null values are allowed.	False
required		Whether each column must exist.	True
platforms		Platform names used to generate column names.	PLATFORMS

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.children_freq_columns(dtype=DoubleType(), nullable=False, required=True, platforms=PLATFORMS)

Build expectations for child platform-frequency columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	DoubleType()
nullable		Whether null values are allowed.	False
required		Whether each column must exist.	True
platforms		Platform names used to generate column names; aggregate meta and amazon entries are excluded.	PLATFORMS

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.ownership_columns(dtype=IntegerType(), nullable=False, required=True, devices=DEVICES)

Build expectations for ownership_<device> columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	IntegerType()
nullable		Whether null values are allowed.	False
required		Whether each column must exist.	True
devices		Device names used to generate column names.	DEVICES

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.coview_columns(dtype=IntegerType(), nullable=True, required=True, platforms=PLATFORMS)

Build expectations for platform co-viewing columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	IntegerType()
nullable		Whether null values are allowed.	True
required		Whether each column must exist.	True
platforms		Platform names used to generate column names; aggregate meta and amazon entries are excluded.	PLATFORMS

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.device_media_columns(dtype=IntegerType(), nullable=True, required=True, devices=DEVICES, platforms=PLATFORMS)

Build expectations for device-by-platform usage columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	IntegerType()
nullable		Whether null values are allowed.	True
required		Whether each column must exist.	True
devices		Device names; other and tv are excluded.	DEVICES
platforms		Platform names; aggregate meta and amazon entries are excluded.	PLATFORMS

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.ads_columns(dtype=IntegerType(), nullable=True, required=True, platforms=PLATFORMS)

Build expectations for platform advertising columns.

Parameters:

Name	Type	Description	Default
dtype		Expected Spark data type.	IntegerType()
nullable		Whether null values are allowed.	True
required		Whether each column must exist.	True
platforms		Platform names used to generate column names; aggregate meta and amazon entries are excluded.	PLATFORMS

Returns:

Type	Description
	Mapping from generated column names to expectations.

synpop.population_qa.geo_columns(geo_columns, dtype=StringType(), nullable=False, required=True)

Build expectations for configured geography columns.

Parameters:

Name	Type	Description	Default
geo_columns		Geography column names.	required
dtype		Expected Spark data type.	StringType()
nullable		Whether null values are allowed.	False
required		Whether each column must exist.	True

Returns:

Type	Description
	Mapping from geography column names to expectations.

synpop.population_qa.create_columns_dict(country_code, version, geo_columns_list)

Build the expected synthetic-population schema.

country_code and version are retained for backward compatibility and future country/version-specific expectations.

Parameters:

Name	Type	Description	Default
country_code		Reserved country identifier.	required
version		Reserved population version.	required
geo_columns_list		Geography columns required for this population.	required

Returns:

Type	Description
	Mapping of every expected public column to its validation constraints.

synpop.population_qa._normalize_type_name(dtype)

Normalize Spark data types and accepted aliases for schema comparison.

Parameters:

Name	Type	Description	Default
dtype	Union[DataType, str]	Either a Spark :class:~pyspark.sql.types.DataType instance or a type name. Names may use SQL aliases such as int, bigint, or string, and are matched case-insensitively.	required

Returns:

Type	Description
str	Lower-cased canonical type name such as integertype. Unrecognized names are returned stripped and lower-cased rather than rejected, so comparison falls back to a literal match.

synpop.population_qa.validate_schema(df, expectations, *, check_actual_nulls=True, enforce_nullable_flag=False)

Validate DataFrame fields, types, nulls, and allowed values.

Parameters:

Name	Type	Description	Default
df	DataFrame	Spark DataFrame to inspect.	required
expectations	Dict[str, ColumnExpectation]	Expected constraints keyed by column name.	required
check_actual_nulls	bool	Count null values in required non-null columns.	True
enforce_nullable_flag	bool	Compare Spark schema nullability with each expectation in addition to inspecting actual values.	False

Returns:

Type	Description
Dict[str, Any]	Report containing pass status, error and warning details, null counts,
Dict[str, Any]	and invalid-value counts.

synpop.population_qa.assert_schema(report)

Assert that a schema-validation report has no errors.

Parameters:

Name	Type	Description	Default
report	Dict[str, Any]	Result produced by validate_schema.	required

Raises:

Type	Description
AssertionError	If the report is not successful. The message lists each failed check with its expected and actual values.

synpop.population_qa._with_gender_labels(synpop_df)

Add standard male and female labels to a synthetic-population frame.

Parameters:

Name	Type	Description	Default
synpop_df		Synthetic-population Spark DataFrame containing a numeric gender column where 1 is male and 2 is female.	required

Returns:

Type	Description
DataFrame	Input DataFrame with an added gender_label string column holding "M", "F", or NULL for any other value.

synpop.population_qa._load_enriched_synpop(country_code, version)

Load a synthetic population with normalized age and gender columns.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition to load, for example "DE".	required
version	str	Synthetic-population version partition to load.	required

Returns:

Type	Description
DataFrame	Latest synthetic-population artifact with an added gender_label column plus the age_group and gender_label normalization applied by :func:demographic_tools.with_age_gender_columns.

Raises:

Type	Description
FileNotFoundError	If no complete dated partition exists for the country and version.

synpop.population_qa._frequency_columns(synpop_df)

Return synthetic-population columns that contain platform frequencies.

Parameters:

Name	Type	Description	Default
synpop_df		Synthetic-population Spark DataFrame.	required

Returns:

Type	Description
list of str	Column names prefixed with freq_, in DataFrame column order. Empty if the artifact carries no frequency columns.

synpop.population_qa._platform_agg_exprs(freq_cols)

Build per-platform Spark expressions that count positive frequencies.

Parameters:

Name	Type	Description	Default
freq_cols	list[str]	Frequency column names to aggregate, normally from :func:_frequency_columns.	required

Returns:

Type	Description
list of Column	One aggregate expression per column, each counting rows whose value is greater than zero and aliased back to the original column name. NULL frequencies count as zero.

synpop.population_qa._build_platform_aggregates(synpop_df, freq_cols)

Aggregate total and age-level platform usage for report visualizations.

Parameters:

Name	Type	Description	Default
synpop_df		Synthetic-population Spark DataFrame containing an age column and every column named in freq_cols.	required
freq_cols	list[str]	Frequency column names to aggregate.	required

Returns:

Type	Description
dict	Mapping with three unevaluated Spark DataFrames: totals_by_age (record count N per age), totals_by_platform (one row of per-platform user counts), and usage_by_age (per-platform user counts per age).

synpop.population_qa._build_age_gender_section(synpop_df, country_code)

Build synthetic and reference age-gender comparison report blocks.

Parameters:

Name	Type	Description	Default
synpop_df		Synthetic-population Spark DataFrame containing age, gender_label, open_web, and gender columns.	required
country_code	str	ISO 3166-1 alpha-2 country code used for the reference pyramid and the external-source comparison.	required

Returns:

Type	Description
list of str	Three HTML blocks: the external-source comparison table, a side-by-side synthetic-versus-reference population pyramid, and the pyramid source note.

Raises:

Type	Description
ValueError	If synpop_df lacks open_web or gender, or if external ITU or UN population data is unavailable for the country.

Notes

Collects the age-gender aggregate to the driver with toPandas for plotting. The aggregate has one row per age and gender, so it stays small.

synpop.population_qa._build_online_population_comparison(synpop_df, country_code)

Build ITU and UN population comparisons for a synthetic population.

Parameters:

Name	Type	Description	Default
synpop_df		Synthetic-population Spark DataFrame with open_web and gender columns.	required
country_code	str	ISO 3166-1 alpha-2 country code.	required

Returns:

Type	Description
str	Styled HTML table containing the ITU online-penetration comparison and
str	total, female, and male population comparisons against UN World
str	Population Prospects data.

Raises:

Type	Description
ValueError	If the synthetic-population artifact lacks required columns, or external population data is unavailable.

synpop.population_qa._build_platform_summary_section(platform_totals_df, freq_cols)

Build the overall platform-usage visualization block.

Parameters:

Name	Type	Description	Default
platform_totals_df		Single-row Spark DataFrame of per-platform user counts, normally the totals_by_platform entry from :func:_build_platform_aggregates.	required
freq_cols	list[str]	Frequency column names present in platform_totals_df. Must be non-empty; an empty list produces an invalid stack expression.	required

Returns:

Type	Description
str	HTML block containing the platform-usage bar chart. Platform labels have the freq_ prefix removed.

synpop.population_qa._build_platform_usage_by_age_section(totals_by_age_df, usage_by_age_df, country_code)

Build age-level usage visualization blocks for every platform.

Parameters:

Name	Type	Description	Default
totals_by_age_df		Spark DataFrame with age and record count N.	required
usage_by_age_df		Spark DataFrame with age and one user-count column per platform.	required
country_code	str	ISO 3166-1 alpha-2 country code shown in the chart titles.	required

Returns:

Type	Description
list of str	One HTML block per platform. Ages present in only one input are dropped by the inner join on age.

synpop.population_qa._build_population_report_blocks(synpop_df, country_code, *, subtitle=None, include_intro=True)

Assemble all visual blocks for a synthetic-population QA report.

Parameters:

Name	Type	Description	Default
synpop_df		Enriched synthetic-population Spark DataFrame, normally from :func:_load_enriched_synpop. Must contain age, gender, gender_label, open_web, and the freq_ platform columns.	required
country_code	str	ISO 3166-1 alpha-2 country code used in titles and external lookups.	required
subtitle	Optional[str]	Optional intro subtitle. Defaults to country_code when omitted.	None
include_intro	bool	Prepend the report intro heading block.	True

Returns:

Type	Description
list of str	Ordered HTML blocks: the optional intro, the age-gender section, the platform-usage summary, and one block per platform by age.

Raises:

Type	Description
ValueError	If synpop_df lacks columns required by the age-gender section, or if external ITU or UN population data is unavailable.

synpop.population_qa.synpop_qa_population_test(country_code, version='prod', assert_flag=False, display_report=False)

Validate a synthetic population against demographic data and its schema.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition to validate.	required
version	str	Synthetic-population version partition.	'prod'
assert_flag	bool	Raise assertions for population or schema failures.	False
display_report	bool	Display or print the schema report.	False

Returns:

Type	Description
	Tuple containing the overall pass status and human-readable failure log.

Raises:

Type	Description
AssertionError	If validation fails while assert_flag is enabled.

synpop.population_qa.s3_uri_to_console_url(s3_uri, region=DEFAULT_S3_REGION)

Convert an S3 object URI to its AWS console URL.

Parameters:

Name	Type	Description	Default
s3_uri	str	URI in s3://bucket/key form.	required
region	str	AWS region used by the console URL.	DEFAULT_S3_REGION

Returns:

Type	Description
str	URL that opens the object in the AWS S3 console.

Raises:

Type	Description
ValueError	If the URI does not contain both a bucket and object key.

synpop.population_qa.synpop_qa_review_report(country_code, version, synpop_id)

Build and upload a population review report.

Parameters:

Name	Type	Description	Default
country_code	str	Country partition loaded into the report.	required
version	str	Synthetic-population version partition.	required
synpop_id	str	Identifier used as the output HTML filename and subtitle.	required

Returns:

Type	Description
	AWS console URL for the uploaded HTML report.

Notes

The report is written with dbutils.fs.put beneath the production SynPop review-report prefix.