Skip to content

panelbook_tools

panelbook_tools._load_panel_book_template(template_name)

Load a bundled Panel Book template.

Parameters:

Name Type Description Default
template_name str

Filename of a template bundled in panelbook_tools_utils.

 required

Returns:

Type Description
Document

Loaded Word document.

panelbook_tools.get_N_most_recent_panel_dates(start_yy, start_mm, start_dd, N, panel_type='online', include_latest=True)

Return the N most recent panel dates for a panel type, working backwards.

Written by Jacob Hansen for the ad-hoc Panel Book (legacy version).

Parameters:

Name Type Description Default
start_yy

Start year.

 required
start_mm

Start month.

 required
start_dd

Start day.

 required
N

Number of panel dates to return.

 required
panel_type

Panel type to search, for example "online".

 'online'
include_latest

Include a panel published on the start date itself. Path lookup excludes the start date, so one day is added back when this is true.

 True

Returns:

Type Description
list of tuple

N (year, month, day) tuples in descending date order.
Notes

Issues one path lookup per date, so cost grows linearly with N.

panelbook_tools.CountryPanelBook

Build the per-country source tables behind a generated panel book.

Notes

Age ranges differ by country: GB treats 0-17 as out of scope, every other country treats 0-15 as out of scope. Those ranges are excluded from the tables and from the ratios computed over them.

panelbook_tools.CountryPanelBook.__init__(country_code, run_date, population_stage='active')

Load country panel data and create all panelbook source tables.

Parameters:

Name Type Description Default
country_code

Country to build tables for. Validated against the SynPop country list.

 required
run_date

Inclusive upper bound used when resolving the population and panel snapshots.

 required
population_stage

Lifecycle stage of the population to load, for example active.

 'active'

Raises:

Type Description
ValueError

If country_code is not a supported SynPop country.
Exception

If no population matches the country, stage, and run_date.
Notes

Construction reads the population and builds every source table, so instantiating this class is expensive and triggers Spark work.

panelbook_tools.CountryPanelBook.generate_base_population_table()

Return age- and gender-level counts and shares for the population.

Returns:

Type Description
DataFrame

One row per age range and gender with record counts and their share of the population. Out-of-scope age ranges are excluded, so shares are computed over the in-scope population only.

panelbook_tools.CountryPanelBook.generate_online_table()

Return online-population counts and penetration rates by age and gender.

Returns:

Type Description
DataFrame

One row per age range and gender with online record counts and the corresponding penetration rate. Out-of-scope age ranges are excluded.

panelbook_tools.CountryPanelBook.generate_relative_panel_table()

Return the online panel's age and gender composition.

Returns:

Type Description
DataFrame

One row per age range with Male_ratio, Female_ratio, and Total_ratio shares of the panel, ordered by age. Panelists with a null age are excluded.

panelbook_tools.CountryPanelBook.generate_relative_online_population_table()

Return the online population's age and gender composition.

Returns:

Type Description
DataFrame

One row per age range with ratio and count columns for male, female, and total, ordered by age. Also cached on self.online_relative_population_table for reuse by :meth:generate_comparison_table.

panelbook_tools.CountryPanelBook.generate_comparison_table()

Return panel-to-population composition ratios and derived weights.

Returns:

Type Description
DataFrame

One row per age range comparing the population composition against the panel composition, with the weight needed to align the panel to the population.

panelbook_tools.CountryPanelBook.generate_churn_table(panel_source, N_weeks=6, dates=None, include_latest=True)

Return churn and survival metrics for a sequence of panel snapshots.

Parameters:

Name Type Description Default
panel_source

Panel type to load snapshots from.

 required
N_weeks

Number of weekly snapshots to compare.

 6
dates

Optional explicit snapshot dates. When omitted, the most recent N_weeks dates are discovered automatically.

 None
include_latest

Include a panel published on the run date itself.

 True

Returns:

Type Description
tuple

(churn_data, churn_N_data) holding churn rates and the corresponding panelist counts across the compared snapshots.
Notes

Reads one panel snapshot per date, so cost grows with N_weeks.

panelbook_tools.CountryPanelBook.generate_panelist_recognition_graph(panel_source='user_report', save_path='s3://ap-analyst/DAME/Panel Books/Panel Recognition Curves/')

Build the curve of panelists recognized across prior panel snapshots.

Parameters:

Name Type Description Default
panel_source

Panel type to load snapshots from.

 'user_report'
save_path

S3 prefix the rendered recognition curve is written to.

 's3://ap-analyst/DAME/Panel Books/Panel Recognition Curves/'

Returns:

Type Description
Any

The panelist recognition curve for this country.
Notes

Writes the rendered curve to save_path as a side effect.

panelbook_tools.CountryPanelBook.get_plot_curves()

Render and save the panelist-recognition curve for the panelbook.

panelbook_tools.PanelBookGenerator

Class to generate panel books. Arguments: - run_date: string format date (%Y-%m-%d) for the run. - countries: list of countries to generate panel books from. Countries must have synthetic populations created. - generate_master_book: boolean indicating if the "Master PanelBook" should be created. - population_stage: string [approved', 'ingested', 'active']. Indicates which populations will be used for the panel book. By default, and strictly recommended to set to 'active'.

panelbook_tools.PanelBookGenerator.__init__(run_date, countries=None, load_tables_from_files=False, generate_master_book=True, population_stage='active', verbose=False)

Configure a multi-country panelbook generation run.

Parameters:

Name Type Description Default
run_date

Run date as a (year, month, day) sequence. Used to resolve populations and to stamp the generated documents.

 required
countries

Countries to generate books for. When omitted, every country with an available prod population at population_stage is used.

 None
load_tables_from_files

Load previously saved source tables instead of recomputing them, which also skips country validation.

 False
generate_master_book

Also generate the combined "Master PanelBook".

 True
population_stage

Population lifecycle stage to use: approved, ingested, or active. active is strongly recommended.

 'active'
verbose

Print progress information while generating.

 False

Raises:

Type Description
Exception

If population_stage is not one of the three accepted values.
ValueError

If any requested country has no available population at that stage.

panelbook_tools.PanelBookGenerator.run_process(logger=None)

Generate configured country and master panelbook documents.

Parameters:

Name Type Description Default
logger

Optional logger. Progress and failures are logged only when one is supplied.

 None

Raises:

Type Description
Exception

If a configured country is not supported. Country validation is skipped when load_tables_from_files is set.
Notes

Writes the generated documents to PANEL_BOOK_OUTPUT_DIRECTORY, creating it when absent.

panelbook_tools.PanelBookGenerator.generate_panel_book(replacements, template_path=None, save_filename=os.path.join(PANEL_BOOK_OUTPUT_DIRECTORY, 'panel_book.docx'), template_name=COUNTRY_PANEL_BOOK_TEMPLATE)

Generate a Panel Book document from a Word template.

Parameters:

Name Type Description Default
replacements list

Replacement definitions for strings, tables, and images in the template.

 required
template_path Optional[Union[str, PathLike, BinaryIO]]

Optional path or binary file object for a Word template. When omitted, the bundled template selected by template_name is used.

 None
save_filename Union[str, PathLike]

Path where the generated Word document is saved. By default, the document is saved in PANEL_BOOK_OUTPUT_DIRECTORY.

 join(PANEL_BOOK_OUTPUT_DIRECTORY, 'panel_book.docx')
template_name str

Filename of a template bundled in panelbook_tools_utils. This is ignored when template_path is provided.

 COUNTRY_PANEL_BOOK_TEMPLATE

panelbook_tools.PanelBookGenerator.__fill_table_from_df(doc, df, table_index=0, skip_header=True, horiz_align=WD_ALIGN_PARAGRAPH.CENTER, vert_align=WD_ALIGN_VERTICAL.CENTER)

Copy DataFrame values into a Word table and format its final row.

Parameters:

Name Type Description Default
doc

Word document containing the target table.

 required
df

Values to write. Must fit within the template table's dimensions.

 required
table_index

Index of the table within doc.

 0
skip_header

Treat the template's first row as a header and start writing at the second row.

 True
horiz_align

Horizontal paragraph alignment applied to each written cell.

 CENTER
vert_align

Vertical alignment applied to each written cell.

 CENTER

Raises:

Type Description
ValueError

If df has more rows or columns than the template table.
Notes

The last DataFrame row is bolded, as panelbook templates use it for totals.

panelbook_tools.PanelBookGenerator.__replace_string_in_table(doc, find, replace, ha='left', table_index=None)

Replace placeholder text in the XML text nodes of a Word table.

Parameters:

Name Type Description Default
doc

Word document containing the target table.

 required
find str

Placeholder text to search for.

 required
replace str

Replacement text.

 required
ha

Horizontal alignment applied to modified paragraphs.

 'left'
table_index

Index of the table within doc. Defaults to the first table.

 None
Notes

Operates on the underlying XML text nodes, so run-level formatting on the placeholder is preserved.

panelbook_tools.PanelBookGenerator.__replace_string_in_doc(doc, find, replace)

Replace placeholder text in Word document paragraph runs.

Parameters:

Name Type Description Default
doc

Word document to modify in place.

 required
find str

Placeholder text to search for.

 required
replace str

Replacement text.

 required
Notes

Only body paragraphs are scanned. Placeholders inside tables need :meth:__replace_string_in_table.

panelbook_tools.PanelBookGenerator.__insert_image_at_placeholder(doc, placeholder, image_path, width=None)

Replace one placeholder paragraph with an image in a Word document.

Parameters:

Name Type Description Default
doc

Word document to modify in place.

 required
placeholder

Text identifying the paragraph to replace. The first paragraph containing it is used.

 required
image_path

Local path to the image to insert.

 required
width

Optional image width. When omitted, the image's native size is used.

 None
Notes

Inserts at most one image and returns as soon as the first match is handled, so later occurrences of placeholder are left in place.

panelbook_tools.PanelBookGenerator.upload_documents_to_drive(notify_channel=True)

Archive the prior panelbooks and upload the newly generated documents.

Parameters:

Name Type Description Default
notify_channel

Post a Slack notification once the upload completes.

 True

Returns:

Type Description
None

Documents are uploaded to Google Drive rather than returned.
Notes

Authenticates with the google_oauth_token secret from the dame-automations scope and writes to the Team Data Science & Delivery shared drive. Existing books are moved to the archive folder before the new ones are uploaded.

panelbook_tools.PanelBookGenerator.__notify_users()

Send the configured Slack notification for newly uploaded panelbooks.

panelbook_tools.PanelBookGenerator.notify_users()

Notify panelbook users that the latest documents are available.

panelbook_tools.load_most_recent_sp(country_code, max_date=None, use_legacy_population=False, version='prod', return_path=False, verbose=True, rename_channel_cols=False, stage='approved')

Load the most recent synthetic population for a country.

Parameters:

Name Type Description Default
country_code str

Country whose population should be loaded.

 required
max_date str | list[str] | tuple[str] | int | list[int] | tuple[int] | date | datetime | None

Optional inclusive upper bound on the population date. Accepts a date, datetime, YYYY-MM-DD string, or a (year, month, day) sequence.

 None
use_legacy_population bool

Read from the legacy data/populations layout instead of the current approved prefix. Ignored when stage is active.

 False
version str

Population version to resolve, for example prod.

 'prod'
return_path bool

Also return the resolved S3 path alongside the DataFrame.

 False
verbose bool

Print progress information while resolving the population.

 True
rename_channel_cols

Rename channel columns to their canonical names after loading.

 False
stage str

Population lifecycle stage: approved, ingested, or active.

 'approved'

Returns:

Type Description
DataFrame or tuple

The population DataFrame, or (DataFrame, path) when return_path is true.

Raises:

Type Description
Exception

If stage is not one of the three accepted values, or if no population matches the country, version, and date bound.
Notes

This is DAME's temporary replacement for the dsr_tools implementation and prints a notice saying so on every call.