# Column Summaries While previewing a table with [preview()](../../reference/preview.md#pointblank.preview) is undoubtedly a good thing to do, sometimes you need more. This is where summarizing a table comes in. When you view a summary of a table, the column-by-column info can quickly increase your understanding of a dataset. Plus, it allows you to quickly catch anomalies in your data (e.g., the maximum value of a column could be far outside the realm of possibility). Pointblank provides a function to make it extremely easy to view column-level summaries in a single table. That function is called [col_summary_tbl()](../../reference/col_summary_tbl.md#pointblank.col_summary_tbl) and, just like [preview()](../../reference/preview.md#pointblank.preview) does, it supports the use of any table that Pointblank can use for validation. And no matter what the input data is, the resultant reporting table is consistent in its design and construction. # Trying out [col_summary_tbl()](../../reference/col_summary_tbl.md#pointblank.col_summary_tbl) The function only requires a table. Let's use the `small_table` dataset (a very simple table) to start us off: ``` python import pointblank as pb small_table = pb.load_dataset(dataset="small_table", tbl_type="polars") pb.col_summary_tbl(small_table) ```
PolarsRows13Columns8
Column NA UQ Mean SD Min P5 Q1 Med Q3 P95 Max IQR
date_time Datetime(time_unit='us', time_zone=None)
0
0		 12
0.92		 - - 2016
01
04 00:32:00		 - - - - - 2016
01
30 11:23:00		 -
date Date
0
0		 11
0.85		 - - 2016
01
04		 - - - - - 2016
01
30		 -
a Int64
0
0		 7
0.54		 3.77 2.09 1 1.06 2 3 4 7.4 8 2
b String
0
0		 12
0.92		 9 0 9 9 9 9 9 9 9 0
c Int64
2
0.15		 7
0.54		 5.73 2.72 2 2.05 3 7 8 9 9 5
d Float64
0
0		 12
0.92		 2,304.7 2,631.36 108.34 118.88 837.93 1,035.64 3,291.03 6,335.44 9999.99 2,453.1
e Boolean
0
0		 T0.62
F0.38		 - - - - - - - - - -
f String
0
0		 3
0.23		 3.46 0.52 3 3 3 3 4 4 4 1
String columns statistics regard the string's length.
The header provides the type of table we're looking at (`POLARS`, since this is a Polars DataFrame) and the table dimensions. The rest of the table focuses on the column-level summaries. As such, each row represents a summary of a column in the `small_table` dataset. There's a lot of information in this summary table to digest. Some of it is intuitive since this sort of table summarization isn't all that uncommon, but other aspects of it could also give some pause. So we'll carefully wade through how to interpret this report. # Data Categories in the Column Summary Table On the left side of the table are icons of different colors. These represent categories that the columns fall into. There are only five categories and columns can only be of one type. The categories (and their letter marks) are: - `N`: numeric - `S`: string-based - `D`: date/datetime - `T/F`: boolean - `O`: object The numeric category (`N`) takes data types such as floats and integers. The `S` category is for string-based columns. Date or datetime values are lumped into the `D` category. Boolean columns (`T/F`) have their own category and are *not* considered numeric (e.g., `0`/`1`). The `O` category is a catchall for all other types of columns. Given the disparity of these categories and that we want them in the same table, some statistical measures will be sensible for certain column categories but not for others. Given that, we'll explain how each category is represented in the column summary table. # Numeric Data Three columns in `small_table` are numeric: `a` (`Int64`), `c` (`Int64`), and `d` (`Float64`). The common measures of the missing count/proportion (`NA`) and the unique value count/proportion (`UQ`) are provided for the numeric data type. For these two measures, the top number is the absolute count of missing values and the count of unique values. The bottom number is a proportion of the absolute count divided by the row count; this makes each proportion a value between `0` and `1` (bounds included). The next two columns represent the mean (`Mean`) and the standard deviation (`SD`). The minumum (`Min`), maximum, (`Max`) and a set of quantiles occupy the next few columns (includes `P5`, `Q1`, `Med` for median, `Q3`, and `P95`). Finally, the interquartile range (`IQR`: `Q3` - `Q1`) is the last measure provided. # String Data String data is present in `small_table`, being in columns `b` and `f`. The missing value (`NA`) and uniqueness (`UQ`) measures are accounted for here. The statistical measures are all based on string lengths, so what happens is that all strings in a column are converted to those numeric values and a subset of stats values is presented. To avoid some understandable confusion when reading the table, the stats values in each of the cells with values are annotated with the text `"SL"`. It makes less sense to provide a full suite of quantile values so only the minimum (`Min`), median (`Med`), and maximum (`Max`) are provided. # Date/Datetime Data and Boolean Data We see that in the first two rows of our summary table there are summaries of the `date_time` and `date` columns. The summaries we provide for a date/datetime category (notice the green `D` to the left of the column names) are: 1. the missing count/proportion (`NA`) 2. the unique value count/proportion (`UQ`) 3. the minimum and maximum dates/datetimes One column, `e`, is of the `Boolean` type. Because columns of this type could only have `True`, `False`, or missing values, we provide summary data for missingness (under `NA`) and proportions of `True` and `False` values (under `UQ`). # Using [DataScan](../../reference/DataScan.md#pointblank.DataScan) Directly The [col_summary_tbl()](../../reference/col_summary_tbl.md#pointblank.col_summary_tbl) function is a convenience wrapper around the DataScan class. When you need more than a visual report (for example, to save the profile for later comparison), you can work with [DataScan](../../reference/DataScan.md#pointblank.DataScan) directly: ``` python small_table = pb.load_dataset(dataset="small_table", tbl_type="polars") scan = pb.DataScan(data=small_table, tbl_name="small_table") ``` The [DataScan](../../reference/DataScan.md#pointblank.DataScan) object computes the same column-level statistics shown in the summary table. You can access the profile as a dictionary with [to_dict()](../../reference/Step.md#pointblank.Step.to_dict), export it as JSON with [to_json()](../../reference/DataScan.to_json.md#pointblank.DataScan.to_json), or render the same tabular report with [get_tabular_report()](../../reference/Validate.get_tabular_report.md#pointblank.Validate.get_tabular_report). # Saving and Loading Profiles A [DataScan](../../reference/DataScan.md#pointblank.DataScan) profile can be saved to disk and loaded back later. This is useful for establishing baselines that you compare against in the future. ``` python # Save the profile to a JSON file scan.save_to_json("small_table_profile.json") # Later, load it back without needing the original data loaded = pb.DataScan.load_from_json("small_table_profile.json") ``` The [from_json()](../../reference/DataScan.from_json.md#pointblank.DataScan.from_json) classmethod works with JSON strings directly, and [from_dict()](../../reference/Step.md#pointblank.Step.from_dict) accepts the dictionary format produced by [to_dict()](../../reference/Step.md#pointblank.Step.to_dict). All three approaches produce a fully restored [DataScan](../../reference/DataScan.md#pointblank.DataScan) that retains the column names, types, statistics, and sample data from the original scan. # Comparing Profiles for Drift When your data changes over time, you can compare two [DataScan](../../reference/DataScan.md#pointblank.DataScan) profiles to detect drift. The [compare()](../../reference/DataScan.compare.md#pointblank.DataScan.compare) method identifies schema changes (columns added, removed, or with changed types) and statistical shifts in common columns. ``` python import polars as pl # Simulate two versions of a dataset orders_v1 = pl.DataFrame({ "order_id": [1, 2, 3, 4, 5], "amount": [10.0, 25.0, 15.0, 30.0, 20.0], "status": ["paid", "paid", "refund", "paid", "paid"], }) orders_v2 = pl.DataFrame({ "order_id": [1, 2, 3, 4, 5, 6, 7, 8], "amount": [10.0, 25.0, 15.0, 30.0, 20.0, 150.0, 200.0, 175.0], "status": ["paid", "paid", "refund", "paid", "paid", "paid", "paid", "paid"], "region": ["US", "EU", "US", "EU", "US", "US", "EU", "EU"], }) baseline = pb.DataScan(data=orders_v1, tbl_name="orders_v1") current = pb.DataScan(data=orders_v2, tbl_name="orders_v2") diff = current.compare(baseline) ``` The returned [`DataScanDiff`](%60pointblank.DataScanDiff%60) object provides programmatic access to the changes: ``` python print("Has changes:", diff.has_changes) print("Columns added:", diff.columns_added) print("Row count (baseline vs current):", diff.row_count_diff) ``` Has changes: True Columns added: ['region'] Row count (baseline vs current): (5, 8) You can also get the full comparison as a dictionary with [to_dict()](../../reference/Step.md#pointblank.Step.to_dict), or view it as a styled report: ``` python diff.get_tabular_report() ```
Profile Comparison: orders_v1 vs orders_v2
Row count: 5 (baseline) vs 8 (current)
Column Status Type (Baseline) Type (Current) Changed Statistics
order_id Stats Changed Int64 Int64 n_unique: 5 -> 8; mean: 3 -> 4.5; median: 3 -> 4.5; std: 1.581 -> 2.449; max: 5 -> 8; p05: 1.02 -> 1.035; q_1: 2 -> 2.75; q_3: 4 -> 6.25; p95: 4.8 -> 7.65; iqr: 2 -> 3.5
amount Stats Changed Float64 Float64 n_unique: 5 -> 8; mean: 20 -> 78.12; median: 20 -> 27.5; std: 7.906 -> 81.54; max: 30 -> 200; p05: 10.1 -> 10.18; q_1: 15 -> 18.75; q_3: 25 -> 156.2; p95: 29 -> 191.2; iqr: 10 -> 137.5
status Stats Changed String String mean: 4.4 -> 4.25; std: 0.8944 -> 0.7071; p95: 5.6 -> 5.3
region Added String
The report shows each column's status (OK, Added, Removed, Stats Changed, or Type Changed) along with any statistics that shifted between the baseline and current profiles. This makes it straightforward to spot when your data's shape or distribution has changed in ways that might affect downstream analyses or validation rules. A typical workflow is to save a baseline profile after your initial data quality checks pass, then compare new data against that baseline on each pipeline run: ``` python # On the first run: establish the baseline baseline = pb.DataScan(data=production_table, tbl_name="orders") baseline.save_to_json("orders_baseline.json") # On subsequent runs: compare against the baseline baseline = pb.DataScan.load_from_json("orders_baseline.json") current = pb.DataScan(data=production_table, tbl_name="orders") diff = current.compare(baseline) if diff.has_changes: print("Data drift detected!") print(diff.to_dict()) ```