visual_tools

Reusable visual helpers for SynPop QA and related reports.

visual_tools._figure_to_inline_svg(fig)

Convert a matplotlib figure to inline SVG and close the figure.

Parameters:

Name	Type	Description	Default
fig		Matplotlib figure to render. Closed before returning, so it must not be reused afterwards.	required

Returns:

Type	Description
str	SVG markup, rendered on the report theme's page background and cropped to the figure's tight bounding box.

visual_tools._set_white_plot_defaults()

Keep matplotlib defaults aligned with the report theme.

visual_tools._format_compact_number(value)

Format a numeric value using compact K, M, or B suffixes.

Parameters:

Name	Type	Description	Default
value	float	Numeric value to format.	required

Returns:

Type	Description
str	Value scaled to one decimal place with a K, M, or B suffix. Values below 1000 are returned as thousands-separated integers.

visual_tools._format_axis_ticklabels(values)

Format numeric axis values as compact labels.

Parameters:

Name	Type	Description	Default
values	Iterable[float]	Numeric tick values to format.	required

Returns:

Type	Description
list of str	One compact label per value, in input order.

visual_tools._age_sort_key(label)

Return the leading numeric value used to order age labels.

Parameters:

Name	Type	Description	Default
label		Age label such as "25-34" or "75+".	required

Returns:

Type	Description
int	First integer found in the label, or 999 when it contains no digits so unrecognized labels sort last.

visual_tools._normalize_age_labels(df, age_col='age')

Sort a pandas DataFrame by normalized numeric age labels.

Parameters:

Name	Type	Description	Default
df	DataFrame	Input pandas DataFrame. Not modified; a copy is returned.	required
age_col	str	Name of the age label column, cast to string before sorting.	'age'

Returns:

Type	Description
DataFrame	Copy sorted by numeric age then label, with a reset index and an added _age_sort helper column.

visual_tools._platform_label(platform_name)

Convert a machine-readable platform name into a display label.

Parameters:

Name	Type	Description	Default
platform_name	str	Platform key such as "open_web".	required

Returns:

Type	Description
str	Title-cased label with underscores replaced by spaces, for example "Open Web".

visual_tools._build_report_assets(include_plotly=False)

Build shared HTML, CSS, and optional Plotly assets for reports.

Parameters:

Name	Type	Description	Default
include_plotly	bool	Include the Plotly script tag. Required for reports containing interactive Plotly charts.	False

Returns:

Type	Description
str	HTML fragment with the report stylesheet and, when requested, the Plotly script tag.

Notes

Plotly is loaded from a CDN, so reports built with include_plotly need network access when opened.

visual_tools.html_intro(title, synpop_id)

Build the header section for a SynPop HTML report.

Parameters:

Name	Type	Description	Default
title	str	Report title displayed prominently.	required
synpop_id	str	Synthetic-population identifier displayed as metadata.	required

Returns:

Type	Description
str	HTML section string for the report header.

visual_tools.svg_block(svg_str, title)

Wrap an inline SVG in a styled report section.

Parameters:

Name	Type	Description	Default
svg_str	str	Inline SVG markup.	required
title	str	Optional visualization title. An empty value omits the heading.	required

Returns:

Type	Description
str	HTML visualization block.

visual_tools.info_block(message_html, title='Note')

Build a styled informational block from trusted HTML content.

Parameters:

Name	Type	Description	Default
message_html	str	HTML inserted into the block body without escaping.	required
title	str	Label displayed above the message.	'Note'

Returns:

Type	Description
str	HTML information block.

visual_tools.population_pyramid_reference_note()

Return the standard PopulationPyramid.net reference disclaimer.

Returns:

Type	Description
str	HTML note explaining that the demographic pyramid is a visual
str	benchmark and is not used for SynPop generation or calibration.

visual_tools.side_by_side_blocks(left_block, right_block, left_title='Synthetic Population', right_title='Demographic Population', height_px=760, header_px=44, footer_px=28)

Lay out synthetic and demographic visualizations side by side.

Parameters:

Name	Type	Description	Default
left_block	str	HTML for the synthetic-population panel.	required
right_block	str	HTML for the demographic reference panel.	required
left_title	str	Left panel heading.	'Synthetic Population'
right_title	str	Right panel heading.	'Demographic Population'
height_px	int	Total height of each panel in pixels.	760
header_px	int	Reserved panel-header height in pixels.	44
footer_px	int	Reserved panel-footer height in pixels.	28

Returns:

Type	Description
str	Responsive two-panel HTML layout.

visual_tools.wrap_html(blocks, page_title='Report', include_plotly=False)

Assemble report blocks into a complete standalone HTML document.

Parameters:

Name	Type	Description	Default
blocks	list[str]	Ordered HTML fragments to place in the report container.	required
page_title	str	Browser-document title.	'Report'
include_plotly	bool	If True, include Plotly assets in the report head.	False

Returns:

Type	Description
str	Complete HTML document string with shared report styles and assets.

visual_tools._add_response_time_minutes(df)

Add valid survey response durations in minutes to a Spark DataFrame.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing date_started and date_submitted columns parseable as timestamps.	required

Returns:

Type	Description
DataFrame	Input with added date_started_ts, date_submitted_ts, and response_time_minutes columns, filtered to rows whose duration is non-null and non-negative. Rows with unparseable or reversed timestamps are dropped.

visual_tools._add_age_gender_columns(df, age_col='age', gender_col='gender')

Add visual-report age/gender labels and sort keys.

This private wrapper keeps existing visual helper behavior while the shared demographic normalization logic lives in demographics.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing the age and gender source columns.	required
age_col	str	Name of the input age column.	'age'
gender_col	str	Name of the input gender column.	'gender'

Returns:

Type	Description
DataFrame	Input with age_group, gender_label, and a sort_key column added, using the bucket and label conventions the visual reports expect.

visual_tools.compute_median_and_threshold(df)

Calculate the median response time and the fast-response threshold.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing response_time_minutes.	required

Returns:

Type	Description
	Pair (median_minutes, threshold) where the threshold is 40% of the
	approximate median.

visual_tools.extract_response_time_array(df)

Collect non-null response times into a NumPy array.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing response_time_minutes.	required

Returns:

Type	Description
	One-dimensional NumPy array collected through pandas.

Notes

This operation collects the selected column to the driver.

visual_tools.remove_right_tail_outliers(x)

Remove values above the upper Tukey fence from an array.

Parameters:

Name	Type	Description	Default
x		Numeric NumPy-compatible array.	required

Returns:

Type	Description
	Values less than or equal to Q3 + 1.5 * IQR. Low-tail values are
	retained.

visual_tools.compute_histogram_and_smooth_density(x, bins=80, sigma=2.0)

Build a density histogram and Gaussian-smoothed density series.

Parameters:

Name	Type	Description	Default
x		Numeric sample array.	required
bins		Histogram bin count or other value accepted by numpy.histogram.	80
sigma		Gaussian-kernel standard deviation in grid units.	2.0

Returns:

Type	Description
	Tuple (density, bin_edges, bin_centers, smooth_density) of NumPy
	arrays.

visual_tools.build_plot_figure(x, bin_centers, smooth_density, threshold, median_minutes, bins=80)

Create the response-time histogram and threshold figure.

Parameters:

Name	Type	Description	Default
x		Response-time sample array.	required
bin_centers		X coordinates for the smoothed density line.	required
smooth_density		Smoothed density values.	required
threshold		Fast-response threshold in minutes.	required
median_minutes		Median response time shown in the annotation.	required
bins		Histogram bin count.	80

Returns:

Type	Description
	Matplotlib Figure containing the histogram, density line, threshold,
	median annotation, and legend.

visual_tools.response_time_svg_html(df, bins=80, sigma=2.0)

Render a survey response-time distribution as inline SVG.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing date_started and date_submitted.	required
bins		Histogram bin count.	80
sigma		Gaussian smoothing strength.	2.0

Returns:

Type	Description
	Inline SVG string containing the response-time chart.

Notes

Responses starting before 2026-01-01, negative durations, null durations, and upper-tail Tukey outliers are excluded.

visual_tools._aggregate_to_pandas(df_age_gender)

Aggregate age-gender counts and collect the small plotting table.

Parameters:

Name	Type	Description	Default
df_age_gender		Spark DataFrame containing age_group, sort_key, and gender_label columns.	required

Returns:

Type	Description
DataFrame	One row per age group with integer Male and Female counts, ordered by sort_key. Missing combinations become zero.

Notes

Collects to the driver with toPandas. The pivoted result has one row per age group, so it stays small regardless of input size. Only M and F labels are pivoted; other gender values are dropped.

visual_tools._build_pyramid_svg(pyramid_pd, title='Age-Gender Distribution Pyramid')

Render an aggregated age-gender population pyramid as SVG.

Parameters:

Name	Type	Description	Default
pyramid_pd		Pandas DataFrame with age_group, Male, and Female columns, normally from :func:_aggregate_to_pandas.	required
title		Chart title.	'Age-Gender Distribution Pyramid'

Returns:

Type	Description
str	Inline SVG markup for the pyramid. Legend labels include each gender's total and its share of the population.

visual_tools.age_gender_pyramid_svg_html(df, age_col='age', gender_col='gender', title='Age-Gender Distribution Pyramid (after dump)')

Render an age-and-gender population pyramid as inline SVG.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing age and gender values.	required
age_col	str	Source age-column name.	'age'
gender_col	str	Source gender-column name.	'gender'
title		Figure title.	'Age-Gender Distribution Pyramid (after dump)'

Returns:

Type	Description
	Inline SVG string with male counts on the left and female counts on the
	right across standardized age groups.

Notes

Aggregated counts are collected to pandas for plotting.

visual_tools._to_age_band(age)

Map an integer age to a five-year display band.

Parameters:

Name	Type	Description	Default
age	int	Age in years.	required

Returns:

Type	Description
str	Band label such as "25-29", or "100+" for ages of 100 and above.

visual_tools._age_band_sort_key(label)

Return the lower bound used to order an age-band label.

Parameters:

Name	Type	Description	Default
label	str	Band label produced by :func:_to_age_band.	required

Returns:

Type	Description
int	Lower bound of the band, and 100 for "100+".

visual_tools._resolve_gender_sides(gender_labels)

Choose the left and right labels for an age-gender pyramid.

Parameters:

Name	Type	Description	Default
gender_labels	list[str]	Exactly two gender category labels.	required

Returns:

Type	Description
tuple	(left_label, right_label). Recognizes M/F and Male/Female and orders them consistently; any other pair is returned in its original order.

Raises:

Type	Description
ValueError	If gender_labels does not contain exactly two entries.

visual_tools._prepare_age_gender_pivot(df, age_group_col, gender_label_col, n_col)

Validate, age-band, and pivot population counts for pyramid plotting.

Parameters:

Name	Type	Description	Default
df	DataFrame	Pandas DataFrame of population counts.	required
age_group_col	str	Column holding a numeric age, bucketed into five-year bands.	required
gender_label_col	str	Column holding the gender label, pivoted into one column per label.	required
n_col	str	Column holding the numeric count to sum.	required

Returns:

Type	Description
DataFrame	Counts pivoted to age band by gender label, ordered by age band with missing combinations filled with zero.

Raises:

Type	Description
ValueError	If a required column is missing, or if no rows remain after dropping nulls and non-numeric ages and counts.

visual_tools.age_gender_pyramid_html(df, age_group_col, gender_label_col, n_col, include_plotlyjs='cdn')

Build an interactive age-gender pyramid and return it as HTML.

Parameters:

Name	Type	Description	Default
df	DataFrame	Pandas DataFrame of population counts.	required
age_group_col	str	Column holding a numeric age, bucketed into five-year bands.	required
gender_label_col	str	Column holding the gender label. Must contain exactly two labels.	required
n_col	str	Column holding the numeric count to sum.	required
include_plotlyjs	str | bool	How to include the Plotly library, passed through to Plotly. "cdn" links it, True inlines it, and False omits it.	'cdn'

Returns:

Type	Description
str	HTML for the interactive pyramid. Bars show each band's share of the grand total, with the horizontal axis fixed at plus or minus 10 percent.

Raises:

Type	Description
ValueError	If a required column is missing, if no usable rows remain, if gender_label_col does not hold exactly two labels, or if the counts sum to zero.

visual_tools._usage_rate_expr(colname, *, null_is_valid_no=False)

Calculate the share of valid yes/no responses where the value is yes.

Survey checkbox columns use 1 for selected/used/owned. Depending on the source path, unselected valid responses may be encoded as 0 or -2. Other values are treated as missing/skipped and excluded from the denominator unless null_is_valid_no is set.

Parameters:

Name	Type	Description	Default
colname	str	Survey column to aggregate. The result is aliased back to this name.	required
null_is_valid_no	bool	Count NULL as a valid "no" response rather than excluding it from the denominator.	False

Returns:

Type	Description
Column	Aggregate expression yielding the share of valid responses equal to 1, or NULL when the denominator is zero rather than dividing by zero.

visual_tools._mean_score_expr(colname)

Build a Spark mean expression that excludes the missing-value sentinel.

Parameters:

Name	Type	Description	Default
colname		Numeric survey column to average. The result is aliased back to this name.	required

Returns:

Type	Description
Column	Mean expression ignoring the -2 "not answered" sentinel, so skipped responses do not pull the average down.

visual_tools.age_gender_heatmap_svg_html(df, col_prefix, agg='share_used', value_fmt='{:.2f}', figsize=(14, 6))

Render prefixed metrics by age and gender as an inline SVG heatmap.

Parameters:

Name	Type	Description	Default
df		Spark DataFrame containing age, gender, and metric columns.	required
col_prefix	str	Prefix selecting metric columns.	required
agg	str	Aggregation mode: share_used, mean, or sum.	'share_used'
value_fmt	str	Format string for cell annotations, or None to omit values.	'{:.2f}'
figsize		Matplotlib figure size.	(14, 6)

Returns:

Type	Description
str	Inline SVG heatmap. If every aggregated value is null, a small
str	No data to plot SVG is returned.

Raises:

Type	Description
ValueError	If no columns match col_prefix or agg is unsupported.

Notes

share_used treats 1 as yes and 0/-2 as valid no values. Null ownership values are also treated as valid no responses.

visual_tools._prepare_platform_usage_dataframe(df)

Collect platform usage data when necessary and order it by age.

Parameters:

Name	Type	Description	Default
df		Pandas or Spark DataFrame containing an age column.	required

Returns:

Type	Description
DataFrame	Copy ordered by numeric age with an added _age_sort column.

Notes

Spark input is collected to the driver with toPandas. The input is expected to be a per-age aggregate rather than row-level data.

visual_tools._build_platform_usage_series(pdf, platform_name)

Build positive and remaining population series for one platform.

Parameters:

Name	Type	Description	Default
pdf	DataFrame	Pandas DataFrame with age, N, _age_sort, and a freq_{platform_name} column.	required
platform_name	str	Platform whose frequency column supplies the positive series.	required

Returns:

Type	Description
DataFrame	Frame with positive users and the zero remainder (N - positive), ready to stack into a bar chart.

visual_tools._style_axis_as_compact_numbers(ax)

Replace an axis's y-axis labels with compact numeric labels.

Parameters:

Name	Type	Description	Default
ax		Matplotlib axis to restyle in place.	required

Notes

Fixes the current tick positions before relabeling, so the labels stay aligned with the ticks.

visual_tools._render_platform_usage_chart(platform_df, platform_name, country_code)

Render one age-based platform-usage chart as an HTML/SVG block.

Parameters:

Name	Type	Description	Default
platform_df	DataFrame	Frame from :func:_build_platform_usage_series with age, positive, and zero columns.	required
platform_name	str	Platform being charted. Must be a key of PLATFORM_COLORS.	required
country_code	str	Country code included in the block title.	required

Returns:

Type	Description
str	HTML block containing the stacked platform-usage chart, with users shown against the total population per age.

visual_tools.platform_usage_per_age(df, country_code)

Create per-platform age-distribution visualization blocks.

Parameters:

Name	Type	Description	Default
df		Pandas or Spark DataFrame containing age, N, and optional	required
country_code	str	Country code included in chart titles.	required

Returns:

Type	Description
list[str]	List of HTML/SVG blocks, one for each configured platform column found
list[str]	in the input.

Notes

Spark input is collected to pandas before plotting.

visual_tools.plot_bar_chart_from_columns(df)

Render labeled values as a descending inline SVG bar chart.

Parameters:

Name	Type	Description	Default
df	DataFrame	Pandas DataFrame containing label and value columns.	required

Returns:

Type	Description
str	Inline SVG bar chart using configured platform colors and compact
str	numeric labels.

visual_tools.embed_from_population_pyramid_net(country_code, year=2026, size_px=None)

Build a responsive PopulationPyramid.net iframe embed.

Parameters:

Name	Type	Description	Default
country_code	str	Two-letter country code. Common codes are mapped to the	required
year	int	Population-pyramid year.	2026
size_px	int | None	Optional square chart size. Defaults to the report panel's	None

Returns:

Type	Description
str	HTML wrapper containing a lazy-loading iframe.

Raises:

Type	Description
ValueError	If country_code is empty or not two characters long.

visual_tools._as_pandas(df)

Return a pandas copy from either a pandas or Spark DataFrame.

Parameters:

Name	Type	Description	Default
df		Pandas or Spark DataFrame.	required

Returns:

Type	Description
DataFrame	Independent copy, so callers can mutate it without affecting the input.

Notes

Spark input is collected to the driver, so this is only safe for aggregates.

visual_tools._build_error_rate_by_vendor_quarter_html(logs_pdf, all_responses_df)

Build an HTML table of failed-check rates by vendor and quarter.

Parameters:

Name	Type	Description	Default
logs_pdf	DataFrame	Pandas DataFrame of QA failure log rows. Must share a respondent identifier column with all_responses_df.	required
all_responses_df		Pandas or Spark DataFrame of all responses. Must contain vendor, date_started, and a respondent identifier among uid, id, row_id, rowId, or rowID.	required

Returns:

Type	Description
str	HTML table of error rates per vendor and quarter, or an explanatory HTML message when all_responses_df is None or lacks the required columns. Missing inputs degrade to a message rather than raising.

visual_tools._build_answer_count_by_vendor_quarter_html(all_responses_df)

Build an HTML table of response counts by vendor and quarter.

Parameters:

Name	Type	Description	Default
all_responses_df		Pandas or Spark DataFrame of all responses. Must contain vendor and date_started. A respondent identifier among uid, id, row_id, rowId, or rowID is used when present.	required

Returns:

Type	Description
str	HTML table of response counts per vendor and quarter, or an explanatory HTML message when all_responses_df is None or lacks vendor and date_started. Missing inputs degrade to a message rather than raising.

visual_tools.build_logs_report_html(df, country_code, good_records, dumped_records, svg_list, all_responses_df=None)

Build a standalone survey QA report from validation logs.

Parameters:

Name	Type	Description	Default
df	DataFrame	Pandas or Spark log DataFrame. Common fields include uid,	required
country_code	str	Country displayed in the report title.	required
good_records	int	Number of responses retained by QA.	required
dumped_records	int	Number of responses rejected by QA.	required
svg_list	list	Ordered visualization strings for response time, age/gender,	required
all_responses_df		Optional full response DataFrame used to calculate	None

Returns:

Type	Description
str	Complete standalone HTML report containing KPIs, distributions, error
str	tables, visualizations, and a sample of failed rows.

Notes

Spark inputs are collected to pandas. Text values included in tables are HTML-escaped and truncated to 500 characters.