Columns (pipe and map_columns())

def cap(c:pl.Expr, ceil:int = 5) -> pl.Expr: return pl.when( c > ceil).then(ceil).otherwise( c )

df.with_columns( pl.col('x').pipe(cap))

This is also works when applying a transformation to mutliple columns with selectors. However, the pipe can result in a conflict in which all variables have the same name (unlike native chaining). This is fixed by appending .name.keep() which access and reapplies the name of the initial column being mapped.

df.with_columns( cs.numeric().pipe(cap).name.keep() )

If you have no other transformations you wish to do simultaneously, map_columns() is a slightly more concise alternative which accepts a column selector and a single transformation to be applied to all passed columns.

df.map_columns( cs.numeric(), cap)

Data Frames (pipe)

Alternatively, you may wish to encapsulate logic operating at the level of the entire DataFrame versus an individual column.

def calc_diffs(df:pl.DataFrame, threshhold:int = 5) -> pl.DataFrame:

    df_out = (
        df
        .with_columns(
            abs = (pl.col('x') - pl.col('y')).abs(),
            abs_gt_t = (pl.col('x') - pl.col('y')).abs() > threshhold,
        )
    )
    return df_out 

This too can be chained onto a DataFrame using the pipe():

df.pipe(calc_diffs)

Values of other arguments to your function can be passed with kwargs³

df.pipe(calc_diffs, threshhold = 3)

This also allows us to write DataFrame-level functions that operate on different variables by passing the variables as parameters.

def calc_diffs(df:pl.DataFrame, var1:str = 'x', var2:str = 'y', threshhold:int = 5) -> pl.DataFrame:

    df_out = (
        df
        .with_columns(
            abs = (pl.col(var1) - pl.col(var2)).abs(),
            abs_gt_t = (pl.col(var1) - pl.col(var2)).abs() > threshhold,
        )
    )
    return df_out 

df.pipe(calc_diffs, var1 = 'y', var2 = 'x', threshhold = 3)

Map Batches

Imagine we want to simulate draws from a binomial distribution, based on the sample size x and probability p in the dataset above.

In the simplest case in which our function receives 1 input, we can chain map_batches() onto that expression. Here, we simply provide the function of interest and option fields to confirm that our return value is a scalar (the result of a single coin flip) of type integer:

# one column in, one value out
df.with_columns(
    coin_flip = pl.col('p').map_batches(function = lambda p: binomial(n = 1, p = p), returns_scalar = True, return_dtype = pl.UInt16)
)

However, if our function requires multiple expressions as inputs, we must either create a struct or internally pass the names of those expressions to exprs (which I find cleaner). The function you are mapping must similar assume it is receiving an input containing those expressions in the same order and, thus, accessing them through indexing.

# two column in, one value out - with structs
df.with_columns(
    coin_flip = pl.struct('x','p').map_batches(
                               function = lambda z: binomial(n = z.struct['x'], p = z.struct['p']), 
                               returns_scalar = True, return_dtype = pl.UInt16)
)

# two columns in, one value out - with exprs
df.with_columns(
    coin_flip = pl.map_batches(exprs = ['x', 'p'],
                               function = lambda z: binomial(n = z[0], p = z[1]), 
                               returns_scalar = True, return_dtype = pl.UInt16)
)

Multiple Outputs

Finally, you can also return multiple outputs. Suppose we want to simulate 100 draws not just 1. Our internal function can instead return an array. Afterward, we can calculate the average outcome versus the expected value to see that this worked as intended.

# many columns out
df.with_columns(
    coin_flip = pl.struct('x','p').map_batches(
                               function = lambda z: binomial(n = z.struct['x'], 
                                                             p = z.struct['p'],
                                                             size = (100,z.shape[0])
                                                             ).transpose(), 
                                return_dtype = pl.Array(pl.UInt16, 100) 
                                )
).with_columns( 
    avg_outcome = pl.col('coin_flip').arr.mean(),
    exp_value = pl.col('x') * pl.col('p')
)

Extension libraries

Recall that native Rust and polars implementations will generally be faster than the techniques shown. Thus, another good option is to familiarize yourself with the burgeoning ecosystem of polars extensions to see if one suits your needs. Awesome Polars maintains a growing list of such packages.

For example, the polars-ds package can natively handle the AUROC use case above as it provides many useful evaluation functions:

df.group_by('group').agg (
    auroc = pds.query_roc_auc('truth', 'mod_bst')
)

The Generator Trick

While pds.query_roc_auc() can calculate AUROC out of the box, it expect string column names as inputs – not expressions. That means we cannot benefit from polars’s selectors and expression expansion to calculate multiple combinations of columns with one line of code (e.g. calculation AUROC for each combination of truth and varying model scores).

To apply an aggregation to multiple column subsets, the Polars docs recommend a pattern like this:

• write a wrapper function that handles the iteration and acts as a generator yielding the expression of interest
• obtain relevant selectors to pass into the function (here, you can use the cs.expand_selectors() helpers or any raw parsing of the column names)
• pass the generator into the standard df.group_by(...).agg(...) flow

def auroc_expressions(models):
    for m in models:
        yield pds.query_roc_auc( 'truth', m).alias(m)

mods = cs.expand_selector(df, cs.starts_with('mod_')) # could also do: [c for c in df.columns if c[:4] == 'mod_']
df.group_by('group').agg( auroc_expressions( mods ))

Complex Object Types

polars DataFrames can hold arbitrary objects (of datatype pl.Object) – not just scalars and vectors. This means, if we so choose, we can do complex multi-step tasks without leaving the DataFrame⁴

Consider one final sample dataset:

data_dict = {
  'group': ['a']*4 + ['b']*4,
  'x': [0.99,0.75,0.25,0.01]*2,
  'y': [0.99,0.75,0.25,0.01]+[0.5]*3+[0.6]
}
df = pl.DataFrame(data_dict)
df.glimpse()

Rows: 8
Columns: 3
$ group <str> 'a', 'a', 'a', 'a', 'b', 'b', 'b', 'b'
$ x     <f64> 0.99, 0.75, 0.25, 0.01, 0.99, 0.75, 0.25, 0.01
$ y     <f64> 0.99, 0.75, 0.25, 0.01, 0.5, 0.5, 0.5, 0.6

If we wish, we can even use map_groups() to create a column that represents complex objects like models and then map_elements() to extract information from these models.

(
df.group_by('group').agg (
    mod = pl.map_groups(
    exprs = ['x', 'y'],
    function = lambda x: sm.OLS( x[0].to_numpy(), sm.add_constant( x[1] )).fit() ,
    return_dtype = pl.Object,
    returns_scalar = True
    )
)
.with_columns(
    params = pl.col('mod').map_elements(lambda x: x.params, return_dtype = pl.List(pl.Float64)),
    r_sq  = pl.col('mod').map_elements(lambda x: x.rsquared, return_dtype = pl.Float64)
)
)

This pattern can be very useful if you are doing something like, for example, bootstrap aggregation. It might not be the most efficient computationally (not parallelized), but for small problems where speed is not a gamechanger, it can make for concise and readable analysis.

Partitions

However, just because you can keep everything in a DataFrame does not mean you should. The above pattern is useful if your end goal is to extract a singular quantity like a coefficient back into the DataFrame. However, if you ultimately want to go do other things with the objects you are generating, it may make for cleaner code to go ahead and break the DataFrame abstraction.

A final pattern I find particularly pleasant and effective is using the partition_by() method. This splits a DataFrame into separate frames based on grouping columns and organizes them in either a list (by default) or as a dictionary (when as_dict = True) indexed with a tuple containing the values of the grouping variable(s).

dfs = df.partition_by('group', as_dict = True, include_key = True)
for k,v in dfs.items():
    print(f"{k}  : {v}")

('a',)  : shape: (4, 3)
┌───────┬──────┬──────┐
│ group ┆ x    ┆ y    │
│ ---   ┆ ---  ┆ ---  │
│ str   ┆ f64  ┆ f64  │
╞═══════╪══════╪══════╡
│ a     ┆ 0.99 ┆ 0.99 │
│ a     ┆ 0.75 ┆ 0.75 │
│ a     ┆ 0.25 ┆ 0.25 │
│ a     ┆ 0.01 ┆ 0.01 │
└───────┴──────┴──────┘
('b',)  : shape: (4, 3)
┌───────┬──────┬─────┐
│ group ┆ x    ┆ y   │
│ ---   ┆ ---  ┆ --- │
│ str   ┆ f64  ┆ f64 │
╞═══════╪══════╪═════╡
│ b     ┆ 0.99 ┆ 0.5 │
│ b     ┆ 0.75 ┆ 0.5 │
│ b     ┆ 0.25 ┆ 0.5 │
│ b     ┆ 0.01 ┆ 0.6 │
└───────┴──────┴─────┘

This allows up to break up the data in the way we wish to process it, and then do processing in more python-native syntax such as a list comprehension. I find this makes highly concise and readable code and is ultimately a better strategy when further data wrangling is not needed.

dfs = df.partition_by('group', as_dict = True, include_key = True)
grps = [ k[0] for k in dfs.keys() ] # turn tuple to scalar bcs only one grouping var in key
mods = [ sm.OLS( d['x'].to_numpy(), 
                 sm.add_constant( d['y'].to_numpy() )
                ).fit() for k,d in dfs.items()]
coef = [m.params[1] for m in mods]
dict(zip( grps, coef))

{'a': np.float64(1.0000000000000004), 'b': np.float64(-6.533333333333337)}

Key Features and Design Choices

If you want the TLDR, I’ll briefly explain the key design choices for this pipeline:

• Initial Data Preparation
  • Set up dbt tests to test sources before joining your feature table. This can better catch dropout from failed joins, newly emerging encoded categories, etc. Consider what additional filters you want to put in downstream tables (Better to “alert and allow” and “block until fixed”?)
  • Prepare feature separately (for normalization) but join different features in database to take advantage of database processing
  • Consider adding random number in training table for reproducible test/train split (this has to be linked to hash or something about the entities your randomizing to ensure reproducibility without regard to ordering of data samples)
• Feature Engineering
  • Create separate scikit-learn pipelines and/or tidymodels workflows for the feature engineering and training steps so you can render these as separate queries. This can enable better data testing and make queries more efficient so orbital does not repeat the feature transformation logic
  • Use test-driven development to update dbt data tests as you develop. For example, encoding a categorical? Immediately add an upstream test to check for previously unseen values.
• Preparing orbital SQL (supported by sqlglot)
  • Add back your identifier column to the query so predictions are joinable
  • Add a model version field into the query for better context to users
  • Change placeholder table to a dbt ref()
  • Rename columns to remove .s so you do not have to always quote in queries
  • Output nicely formatted version for readability
• Deploying as a model
  • Consider carefully whether to make a table, view, or macro depending on your specific database, query latency, and desire to score bespoke populations
• Observability, logging, and error handling
  • Use dbt snapshots to save timestamped past predictions and feature values if these can change over time. This improves auditability and future analysis
  • Execute tests to --store-failures to detect changes in your data that might require model retraining or additional error handling
  • Check out dbt packages like elementary to log more aspects of the model run process

Set-Up

The sample IBM data is provided as “one big table”, so I break things up to look a bit more like normalized database data representing subscription information, billing information, demographics, and churn targets. I also add a few columns to simulate different months, censor data I want to pretend is in the future, and add a few data errors for fun.

Here’s a preview of the resulting tables, connected by a customer_id primary key:

import duckdb
import polars as pl

with duckdb.connect('dev.duckdb') as con:

    df_serv = con.table('serv').pl()
    df_bill = con.table('bill').pl()
    df_demo = con.table('demo').pl()
    df_chrn = con.table('chrn').pl()

df_serv.glimpse()

Rows: 7043
Columns: 12
$ customer_id        <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ tenure             <i32> 1, 34, 2, 45, 2, 8, 22, 10, 28, 62
$ phone_service      <str> 'No', 'Yes', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'Yes'
$ multiple_lines     <str> 'No phone service', 'No', 'No', 'No phone service', 'No', 'Yes', 'Yes', 'No phone service', 'Yes', 'No'
$ internet_service   <str> 'DSL', 'DSL', 'DSL', 'DSL', 'Fiber optic', 'Fiber optic', 'Fiber optic', 'DSL', 'Fiber optic', 'DSL'
$ online_security    <str> 'No', 'Yes', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'No', 'Yes'
$ online_backup      <str> 'Yes', 'No', 'Yes', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes'
$ device_protection  <str> 'No', 'Yes', 'No', 'Yes', 'No', 'Yes', 'No', 'No', 'Yes', 'No'
$ tech_support       <str> 'No', 'No', 'No', 'Yes', 'No', 'No', 'No', 'No', 'Yes', 'No'
$ streaming_tv       <str> 'No', 'No', 'No', 'No', 'No', 'Yes', 'Yes', 'No', 'Yes', 'No'
$ streaming_movies   <str> 'No', 'No', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes', 'No'
$ dt_renewal        <date> 2025-07-01, 2025-07-01, 2025-07-01, 2025-07-01, 2025-07-01, 2025-07-01, 2025-08-01, 2025-07-01, 2025-07-01, 2025-07-01

df_bill.glimpse()

Rows: 7043
Columns: 6
$ customer_id       <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ contract          <str> 'Month-to-month', 'One year', 'Month-to-month', 'One year', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'One year'
$ paperless_billing <str> 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No'
$ payment_method    <str> 'Electronic check', 'Mailed check', 'Mailed check', 'Bank transfer (automatic)', 'Electronic check', 'Electronic check', 'Credit card (automatic)', 'Mailed check', 'Electronic check', 'Bank transfer (automatic)'
$ monthly_charges   <f64> 29.85, 56.95, 53.85, 42.3, 70.7, 99.65, 89.1, 29.75, 104.8, 56.15
$ total_charges     <f64> 29.85, 1889.5, 108.15, 1840.75, 151.65, 820.5, 1949.4, 301.9, 3046.05, 3487.95

df_demo.glimpse()

Rows: 7043
Columns: 5
$ customer_id    <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ gender         <str> 'Female', 'Male', 'Male', 'Male', 'Female', 'Female', 'Male', 'Female', 'Female', 'Male'
$ senior_citizen <i32> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
$ partner        <str> 'Yes', 'No', 'No', 'No', 'No', 'No', 'No', 'No', 'Yes', 'No'
$ dependents     <str> 'No', 'No', 'No', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes'

df_chrn.glimpse()

Rows: 7043
Columns: 2
$ customer_id <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ churn       <str> 'No', 'No', 'Yes', 'No', 'Yes', 'Yes', 'No', 'No', 'Yes', 'No'

Ultimately, these are saved as seeds in the dbt project as a lightweight way to ingest small CSVs; in reality, they would be my sources flowing into my data warehouse from other production sources.

Features & Training

Feature preparation and training are the heart of where orbital fits into our pipelines. I recommend doing these steps one-at-a-time and explain them similarly. However, since the code is closely coupled, I’ll provide it at once for reference. The combination of feature engineering and model training steps look like this:

# build pipeline(s)

## feature pipeline does OneHotEncoding on all string columns (all are low/known cardinality)
## orbital can create some very verbose variable names (for uniqueness) so we clean those up some
cols_str = X.select( cs.string() ).columns
onho_enc = ('oh', OneHotEncoder(sparse_output = False), cols_str)
ppl_feat = Pipeline([
  ("encoder", ColumnTransformer([onho_enc], remainder='passthrough'))
]).set_output(transform="polars")
X_tran = ppl_feat.fit_transform(X, y)
X_tran.columns = [c.replace(' ','_').replace('-','_').replace('(','').replace(')','') for c in X_tran.columns]

## training pipeline fits actual random forest model
ppl_pred = Pipeline([
  ("prep", ColumnTransformer([], remainder='passthrough')),
  ("pred", RandomForestClassifier(max_depth=3, n_estimators=100, random_state=123))
])
ppl_pred.fit(X_tran, y)

# convert to orbital

tbl = "TBL_REF" # placeholder replaced in cleaning

## creating mapping of source data types to orbital types 
type_map = {
    pl.String:orbital.types.StringColumnType(),
    pl.Int32:orbital.types.Int32ColumnType(),
    pl.Float64:orbital.types.DoubleColumnType()
}
dict_feat = {e: type_map.get(t) for e, t in zip(X.columns, X.dtypes)}
dict_pred = {e: type_map.get(t) for e, t in zip(X_tran.columns, X_tran.dtypes)}

## features
orb_ppl_feat = orbital.parse_pipeline(ppl_feat, features=dict_feat)
sql_raw_feat = orbital.export_sql(tbl, orb_ppl_feat, dialect="duckdb")

## scoring
orb_ppl_pred = orbital.parse_pipeline(ppl_pred, features=dict_pred)
sql_raw_pred = orbital.export_sql(tbl, orb_ppl_pred, dialect="duckdb")

SQL Cleanup

While orbital does a lot of heavy lifting, the SQL it produces is not perfect:

• It does not SELECT any metadata or identifier columns, rendering your predictions impossible to join to other data sources. Inserting this column requires care because sometimes the upstream data is being queried within the main query and other times it is queried in a CTE
• Its hard to get orbital to query from a ref() that plays nice with dbt’s Jinja because orbital is rigorous about quoting table and column names. So, it’s easier to put a placeholder table name and edit it in post-processing.
• It uses somewhat long and bulky variable names that reflect scikit-learn internals, including .s in column names which can reduce readability and requires quoting since . usually means something different in SQL
• It includes positive predictions, negative predictions, and labels which may be excessive. I’ve never wanted anything more than the positive predictions
• It’s not formatted which shouldn’t matter but will wrankle anyone who has ever worked with SQL

To mitigate these multiple issues, sqlglot makes it easy to further parse the query. sqlglot is a package that allows you to turn any SQL script into an AST for ease of programatic modification. I defined a helper function with sqlglot to fix all of the above.

import sqlglot
from sqlglot import parse_one, exp 

def clean_sql(sql_raw: str, 
              tbl_ref: str, 
              model_version: str = None,
              col_id: str = 'id', 
              cols_renm: dict[str, str] = {'output_probability.1':'pred', 
                                           'output_probability.0': '0', 
                                           'output_label': '0'},
              ) -> str:

    """Opinionated clean-up of SQL returned by orbital

    This function executes the following transformations:
    - Rename desired columns such as the prediction column (per result of cols_renm)
    - Remove unwanted variables (those being "renamed" to "0")
    - Add back ID variable for joining predictions to other datasets 
    - Fix table reference from default TBL_REF to a specific dbt model reference
    - Reformats SQL for improved readability

    Parameters
    ----------
    sql_raw: SQL string provided by `orbital`
    tbl_ref: Name of dbt model to be referenced in query's FROM clause
    model_version: Version number of model to be added as own column. Defaults to None to add no column
    col_id: Name of the column representing the unique identifier of entities to be predicted
    cols_renm: Dictionary of {default_name: desired_name} to rename fields

    Returns
    -------
    str
        A formatted and updated SQL query
    """


    ast = parse_one(sql_raw)
    
    for e in ast.expressions:
        # rename prediction column
        if cols_renm.get(e.alias) == '0':
            e.set(arg_key='this',value=None)
            e.set(arg_key='alias',value=None)
        if e.alias in cols_renm.keys():
            e.set(arg_key='alias',value=cols_renm.get(e.alias))
    
    # add back a variable for reference (typically like an ID for joining to other tables)
    # this is tricky because sometimes orbital uses CTEs and other times it doesn't;
    # generally, we need to get the identifier inside the CTE if it exists
    col = exp.Column(this=exp.to_identifier(col_id))
    if ast.find(exp.CTE) is not None:
        cte_select = ast.find(exp.CTE).this
        cte_select.expressions.append(col)
    ast = ast.select(col_id)

    # add model version to outer query if desired
    if model_version is not None:

        col_version = exp.Alias(
            this=exp.Literal.string(model_version), 
            alias="model_version")
        ast.find(exp.Select).expressions.append(col_version)
    
    # pretty print
    sql_fmt = sqlglot.transpile(ast.sql(), 
                                write="duckdb", 
                                identify=True, 
                                pretty=True)[0]
    
    # change out table to dbt reference
    ref_str = f"{{{{ ref('{tbl_ref}')}}}}"
    sql_fnl = sql_fmt.replace('"TBL_REF"', ref_str) 
  
    return sql_fnl

I run the SQL generated from both ppl_feat and ppl_rafo through this function before writing them to models/churn_model/prep_feat.sql and models/churn_model/pred_churn.sql in my dbt models/ directory.

This establishes our core model deployment pipeline:

Scoring, Preserving, and Activating Predictions

We now have a table in our database that has our churn model predictions! Here is where we can begin to utilize the full benefit of the data management tools that dbt has built in.

Before scoring, we can run our dbt test to ensure that our features are stable and valid.

For scoring, depending on our use case we can set the table materialization to be a table (rebuilt on a schedule) or a view (generated on the fly for a specific population).

For archiving past scores, we can update our dbt-project.yml to include snapshotting our predictions table. This means even if we publish our tables as a view, we could schedule a call to dbt snapshot on a regular basis to record a timestamped record of what our scores were at any given point in time. This could be useful for model monitoring or auditiability. For example, if we are using our churn model to segment a marketing campaign, we might need these scores later to determine who got what treatment in the campaign.

For staging analysis, we can use dbt analyses to render the scripts that might be needed to conduct model monitoring (e.g. merging past scores with observed targets.)

We can see examples of these different artifacts branching off of our DAG:

Datamart Preview

Below, we can tour the resulting datasets:

import duckdb
import polars as pl

with duckdb.connect('dev.duckdb') as con:

    df_feat = con.table('raw_feat').pl()
    df_prep = con.table('prep_feat').pl()
    df_pred = con.table('pred_churn').pl()
    df_snap = con.table('main_snapshots.pred_churn_snapshot').pl()
    df_fail = con.table('main_audit.accepted_values_serv_internet_service__DSL__Fiber_optic__No').pl()

df_feat.glimpse()

Rows: 6944
Columns: 21
$ customer_id       <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ cat_train_test    <str> 'Train', 'Train', 'Train', 'Train', 'Train', 'Train', 'Train', 'Train', 'Train', 'Train'
$ tenure            <i32> 1, 34, 2, 45, 2, 8, 22, 10, 28, 62
$ phone_service     <str> 'No', 'Yes', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'Yes'
$ multiple_lines    <str> 'No phone service', 'No', 'No', 'No phone service', 'No', 'Yes', 'Yes', 'No phone service', 'Yes', 'No'
$ internet_service  <str> 'DSL', 'DSL', 'DSL', 'DSL', 'Fiber optic', 'Fiber optic', 'Fiber optic', 'DSL', 'Fiber optic', 'DSL'
$ online_security   <str> 'No', 'Yes', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'No', 'Yes'
$ online_backup     <str> 'Yes', 'No', 'Yes', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes'
$ device_protection <str> 'No', 'Yes', 'No', 'Yes', 'No', 'Yes', 'No', 'No', 'Yes', 'No'
$ tech_support      <str> 'No', 'No', 'No', 'Yes', 'No', 'No', 'No', 'No', 'Yes', 'No'
$ streaming_tv      <str> 'No', 'No', 'No', 'No', 'No', 'Yes', 'Yes', 'No', 'Yes', 'No'
$ streaming_movies  <str> 'No', 'No', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes', 'No'
$ gender            <str> 'Female', 'Male', 'Male', 'Male', 'Female', 'Female', 'Male', 'Female', 'Female', 'Male'
$ senior_citizen    <i32> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
$ partner           <str> 'Yes', 'No', 'No', 'No', 'No', 'No', 'No', 'No', 'Yes', 'No'
$ dependents        <str> 'No', 'No', 'No', 'No', 'No', 'No', 'Yes', 'No', 'No', 'Yes'
$ contract          <str> 'Month-to-month', 'One year', 'Month-to-month', 'One year', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'Month-to-month', 'One year'
$ paperless_billing <str> 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No'
$ payment_method    <str> 'Electronic check', 'Mailed check', 'Mailed check', 'Bank transfer (automatic)', 'Electronic check', 'Electronic check', 'Credit card (automatic)', 'Mailed check', 'Electronic check', 'Bank transfer (automatic)'
$ monthly_charges   <f64> 29.85, 56.95, 53.85, 42.3, 70.7, 99.65, 89.1, 29.75, 104.8, 56.15
$ total_charges     <f64> 29.85, 1889.5, 108.15, 1840.75, 151.65, 820.5, 1949.4, 301.9, 3046.05, 3487.95

df_prep.glimpse()

Rows: 6944
Columns: 47
$ oh__phone_service_No                       <f64> 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0
$ oh__phone_service_Yes                      <f64> 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0
$ oh__multiple_lines_No                      <f64> 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0
$ oh__multiple_lines_No_phone_service        <f64> 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0
$ oh__multiple_lines_Yes                     <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0
$ oh__internet_service_DSL                   <f64> 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0
$ oh__internet_service_Fiber_optic           <f64> 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0
$ oh__internet_service_No                    <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__online_security_No                     <f64> 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0
$ oh__online_security_No_internet_service    <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__online_security_Yes                    <f64> 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0
$ oh__online_backup_No                       <f64> 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0
$ oh__online_backup_No_internet_service      <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__online_backup_Yes                      <f64> 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0
$ oh__device_protection_No                   <f64> 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0
$ oh__device_protection_No_internet_service  <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__device_protection_Yes                  <f64> 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0
$ oh__tech_support_No                        <f64> 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0
$ oh__tech_support_No_internet_service       <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__tech_support_Yes                       <f64> 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0
$ oh__streaming_tv_No                        <f64> 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0
$ oh__streaming_tv_No_internet_service       <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__streaming_tv_Yes                       <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0
$ oh__streaming_movies_No                    <f64> 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0
$ oh__streaming_movies_No_internet_service   <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__streaming_movies_Yes                   <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0
$ oh__gender_Female                          <f64> 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0
$ oh__gender_Male                            <f64> 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0
$ oh__partner_No                             <f64> 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0
$ oh__partner_Yes                            <f64> 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0
$ oh__dependents_No                          <f64> 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0
$ oh__dependents_Yes                         <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0
$ oh__contract_Month_to_month                <f64> 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0
$ oh__contract_One_year                      <f64> 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0
$ oh__contract_Two_year                      <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ oh__paperless_billing_No                   <f64> 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0
$ oh__paperless_billing_Yes                  <f64> 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0
$ oh__payment_method_Bank_transfer_automatic <f64> 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0
$ oh__payment_method_Credit_card_automatic   <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0
$ oh__payment_method_Electronic_check        <f64> 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0
$ oh__payment_method_Mailed_check            <f64> 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0
$ remainder__tenure                          <f64> 1.0, 34.0, 2.0, 45.0, 2.0, 8.0, 22.0, 10.0, 28.0, 62.0
$ remainder__senior_citizen                  <f64> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
$ remainder__monthly_charges                 <f64> 29.85, 56.95, 53.85, 42.3, 70.7, 99.65, 89.1, 29.75, 104.8, 56.15
$ remainder__total_charges                   <f64> 29.85, 1889.5, 108.15, 1840.75, 151.65, 820.5, 1949.4, 301.9, 3046.05, 3487.95
$ customer_id                                <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ model_version                              <str> '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0'

df_pred.glimpse()

Rows: 6944
Columns: 3
$ pred          <f64> 0.4350639304611832, 0.14068829294410534, 0.34994459204608575, 0.10898763570003211, 0.5811184463091195, 0.5483232741244137, 0.4043196897255257, 0.311830934981117, 0.3962726652389392, 0.1372128768125549
$ customer_id   <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ model_version <str> '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0'

df_snap.glimpse()

Rows: 6944
Columns: 7
$ pred                    <f64> 0.4350639304611832, 0.14068829294410534, 0.34994459204608575, 0.10898763570003211, 0.5811184463091195, 0.5483232741244137, 0.4043196897255257, 0.311830934981117, 0.3962726652389392, 0.1372128768125549
$ customer_id             <str> '7590-VHVEG', '5575-GNVDE', '3668-QPYBK', '7795-CFOCW', '9237-HQITU', '9305-CDSKC', '1452-KIOVK', '6713-OKOMC', '7892-POOKP', '6388-TABGU'
$ model_version           <str> '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0', '1.0'
$ dbt_scd_id              <str> 'c4671964ba707c90a41d74f6f2ef75b7', '7dc40efa71bcee4795c7f54b3b5bc783', 'b05d4425f5d07106f1f2f2e782461f44', '3b919e27eb23ba54e200462af172e7da', 'eb6117ba3156a771b0e02e5e7bc644ab', 'ddae31e6abdabdd771ea4bbd1072fe55', 'aa7fe49fcbb5a937b44f7ac589b3ff34', 'da7eb2655934862105e8782e40ca5eb5', '882f945d0e265290e5976d4c8d04679e', '72f44f68e12a53baaf1d9ddd2469a616'
$ dbt_updated_at <datetime[μs]> 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000
$ dbt_valid_from <datetime[μs]> 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000, 2025-08-15 19:22:45.830000
$ dbt_valid_to   <datetime[μs]> 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00, 9999-12-31 00:00:00

df_fail.glimpse()

Rows: 1
Columns: 2
$ value_field <str> 'Fiber'
$ n_records   <i64> 48

Dreaming bigger

This demo shows just orbital + dbt, but that’s just the beginning. Treating the whole MLOps process just like data processing means you can benefit from a wide range of integrated tools and capabilities, e.g.:

• data ingestion
  • retrieve training data for APIs with dlt
  • ingest features from flatfiles on blob sources via the dbt external-tables package
• better testing with dbt packages such as dbt-expectatons (from Great Expectations)
• logging and observability
  • snapshot features table as well as predictions table
  • use dbt packages like elementary to write more run metadata to your warehouse
• orchestration with Dagster
  • unfurl your local dbt DAG into a broader pipeline
  • trigger more model-adjacent tasks from refitting, monitoring, etc.
• documentions with dbt docs (which can be enhanced with Quarto)
• reverse ETL with tools like HighTouch or Census to easily sync analytical data models into production systems like CRMs

Communication in education

The majority of technical communication consumed and produced by early career professionals happened during their education. However, academic¹ communication has a fundamentally different goal, so it does not provide an effective model.

Academic communcation leans towards exhaustive knowledge sharing of all the details – either because the target audience needs to know them or the audience needs to know that the communicator knows them.

When students are communicating (completing problemsets or assignments), they have the goal of proving their mastery. Their audience (professors, TAs) can be assumed to have deeper knowledge of the topic than the presenter, and communication is intended to demonstrate comprehensiveness of knowledge – or at least to “show their work” for partial credit.

When students are consuming communication (from an instructor or textbook), they experience communication with the goal of exhaustive knowledge transfer. Instructors or textbooks aim to make the audience know what they know and to be able to execute that information independently.

Communication in industry

These are decidedly not the objective of professional communication. We are given a job because we are judged to have the mastery of a topic specifically that no one else has the time, energy, or desire to think about in enough detail. The goal is not to show what you know (or, how much work you did along the way) or to get the audience to your intimacy of understanding.²

Instead, the goal is to be an effective abstraction layer between the minute details and what is actually needed to act. Communication is an act of curating the minimal spanning set of relevant facts, context, and supporting evidence or caveats.³

Story structuring

Roughly speaking, this means we are used to talking about data science work in the procedural order:

1. I wondered...
2. I prepared my data...
3. I ran this analysis...
4. This gave me another question...
5. That {did/not} work...
6. So finally I ended up with this...
7. I learned...

However, for effective communication, it may be more useful to structure our work with progressive disclosure:

1. I wondered...
7. I ultimately found...
6. This is based on trying this...
(3-5). We also considered other options...
2. And this is all based on this data, details, etc.

In short, we want to tell the story of why others should care about our results – not the story of how we got the result.⁴ Then, it helps turn a presentation or written document into a “conversation” where they can selectively partake of the details instead of waiting for the main point to be revealed as in a murder mystery.

Quarto embed overview

The embed shortcode lets us reference the output of another .qmd or ipynb Quarto document in a different Quarto file. This requires two steps:

First, in the original notebook we label the top of the chunk whose output we wish to target, e.g. in our notebook analysis.ipynb:⁵

#| label: my-calc

1+1

Then in our main document we can pull in the output (and optionally the code) of that calculation, e.g. in a final Quarto document final-writeup.qmd we could add:

{{< embed analysis.ipynb#my-calc >}}

This works with any sort of cell output including raw print() statement output, plots, tables, etc.

Usage patterns

Why are embeds a game-changer for data science communication? Because writing is thinking and storytelling is curation. Embeds can help us switch our mindset away from “showing our work” and towards providing persuasive evidence that supports our narrative.

The workflow I recommend is:

• Still use good practices to modularizes steps like data pulls, separate modules for reusable functions, etc.
• Do the analysis and last-mile transformation you would do in a Jupyter notebook, leaving the commentary that you would along the way
• After you’re done, think about what is important. What does your audience need to see and in what order?
• Then take a step back and write your actual story in a Quarto notebook
• Selectively embed compelling evidence at the right points in your narrative

This is illustrated in the figure below:

This simple shortcode unblocks us from critical storytelling and workflow challenges:

• We can generate content in rerunnable linear notebooks
• We can start writing from a blank slate to ensure that we are focused on substance and not just sheer volume of content
• We can then selectively curate output worthy of inclusion in a final document
• We can insert these in the order that makes sense for the story versus for generation
• This can exist without deleting or modifying our notebooks that capture the full thought process
• As a bonus, our final document need not be a gnarly .ipynb but a plaintext .qmd to make version control, editing, and collaborating with noncoding contributors easier

It’s not just code output that can be imported either. Perhaps you already wrote up an introduction framed as an experimental design or a proposal? Other full markdown files can similarly be inclued with the includes shortcode. (includes adds the unrendered text of another .qmd file, including any code chunks to be executed, to your main .qmd; whereas, for embeds, we are referencing the output of another file without rerendering.)

Demo

To give a quick demo, I’ll made a separate notebook that just pulls some data from an API, cleans it up, and makes a few aggregations and plots. But suppose I doubt you’re interested in any of that. If you’ve read this long, you seem to trust me to do some amount of data stuff correctly.

So instead, I just put the following line in the .qmd file that is creating this post:

{{< embed raw-analysis.ipynb#tbl-pit-eo >}}

That produces this:

Perhaps I thought it was better to show you a simple table first. But, then I want to show you a more complex plot. You don’t have to see that in my original notebook I actually made the plot first. (Actually, I made two plots, but only one seemed important to show.) So, I write:

{{< embed raw-analysis.ipynb#fig-cum-eo >}}

That produces this:

If I had done a particularly good job of summarizing my thoughts immediately after seeing this plot, I might have already written them in a markdown cell over there. Embeds technically also work to embed markdown cells, so the following line you see is embedded from my other notebook also:

However, I don’t advocate for embedding text. I think using a final qmd file as a single, self-contained spot to document your analysis has a lot of benefits.

And then I could go on to add my relevant thoughts and analysis of specific to that plot. But, in this case, another part of professional communication is staying on topic.

Converting xgboost model to an XGBClassifier pipeline

xgboost provides two interfaces: a native learning API and a scikit-learn compatible API. The learning API is sometimes favored for performance advantages. However, since orbital can only work with scikit-learn pipelines, it’s necessary to move to a compatible API.

The strategy here is to fit an xgboost model (assuming that’s what you wanted to do in the first place), initialize a XGBClassifier, and set its attributes. Then, we can directly put our trained XGBClassifier into the a pipeline.

# train with xgb learning api
## keeping parameters super simple so it trains fast and easy to compare
## important: this only works for now with base_score=0.5 
## this is the default assumed by orbital's logic, and I haven't figured out how to convince it otherwise
dtrain = xgb.DMatrix(X_train, y_train, feature_names = nm_cols)
params = {'max_depth':2, 
          'objective':'binary:logistic', 
          'base_score':0.5, 
          'seed':504}
model = xgb.train(params, num_boost_round = 1, dtrain = dtrain)
preds_xgb = model.predict(xgb.DMatrix(X_train, feature_names = nm_cols))

# convert back to skl interface & rebuild needed metadata
clf = xgb.XGBClassifier()
clf._Booster = model
clf.n_classes_ = 2
clf.base_score = 0.5
preds_skl = clf.predict_proba(X_train)[:,-1]

# validate that the results are the same
print(f"xgb and skl match: {np.all(np.isclose(preds_xgb, preds_skl))}")

# add to skl pipeline
ppl = Pipeline([("gbm", clf)])
preds_ppl = ppl.predict_proba(X_train)[:,-1]

# validate that the results are the same
print(f"xgb and ppl match: {np.all(np.isclose(preds_xgb, preds_ppl))}")

xgb and skl match: True
xgb and ppl match: True
xgb and skl match: True
xgb and ppl match: True

We see all three approaches produce the same predictions.

Unfortunately, things aren’t quite that simple.

Below, I remake the pipeline with a column transformer, ask it to apply to an empty list of variables, and request the rest (i.e. all of them) be passed through untouched.

# now we actually make a slightly more complicated pipeline
# orbital seems unhappy if there isn't at least one preprocessing step,
# so we make one that processes no variables and passes through the rest
pipeline = Pipeline([
    ("pre", ColumnTransformer([], remainder="passthrough")),
])
pipeline.fit(X_train)
pipeline.steps.append(("gbm", clf))
preds_ppl2 = pipeline.predict_proba(X_train)[:,1]
print(f"xgb and ppl2 matches: {np.all(np.isclose(preds_xgb, preds_ppl2))}")

xgb and ppl2 matches: True
xgb and ppl2 matches: True

Again, we see this “null” step does not change our predictions.

Enabling onnxmltools for XGBClassifier conversion

orbital depends on skl2onnx which implements a smaller set of model types. onnxmltools offers many additional model converters. However, for skl2onnx to correctly find and apply these converters, they must be registered.

# `options` copied straight from `onnxmltools` docs
update_registered_converter(
    XGBClassifier,
    "XGBoostXGBClassifier",
    calculate_linear_classifier_output_shapes,
    convert_xgboost,
    options={"nocl": [True, False], 
             "zipmap": [True, False, "columns"], 
            },
)

However, there’s another nuance here. We all know the challenges of python package versioning, but both skl2onnx and onnxmltools also require coordinating on a version of the ONNX spec’s version as a universal way to represent model objects. The skl2onnx function that allows us to request a version is wrapped in orbital without the ability to pass in parameters. So, we must override that function.

def parse_pipeline_local(
    pipeline: Pipeline, features: orbital.types.FeaturesTypes
) -> orbital.ast.ParsedPipeline:

    onnx_model = skl2onnx.to_onnx(
        pipeline,
        initial_types=[
            (fname, ftype._to_onnxtype())
            for fname, ftype in features.items()
            if not ftype.is_passthrough
        ],
        target_opset={"": 15,'ai.onnx.ml':3}
    )
    return orbital.ast.ParsedPipeline._from_onnx_model(onnx_model, features)

Run orbital!

If you’ve made it this far, you’ll be happy to know the next step is straightforward. We can now run orbital to generate the SQL representation of our model prediction logic.

# translate into an Orbital Pipeline
orbital_pipeline = parse_pipeline_local(pipeline, features=feat_dict)
sql_raw = orbital.export_sql("DATA_TABLE", orbital_pipeline, dialect="duckdb")

Validate results

So, after all that, did we get the right result? One way we can confirm (especially because we kept the initial xgboost model very simple) is to compare the visual of our tree with the resulting SQL.

Here’s the tree grown by xgboost:

Here’s the SQL developed by orbital:

SELECT
  1 / (
    EXP(
      -CASE
        WHEN "t0"."f4" < -0.04800000041723251
        THEN CASE
          WHEN "t0"."f4" < -0.8119999766349792
          THEN -0.5087512135505676
          ELSE -0.21405750513076782
        END
        ELSE CASE
          WHEN "t0"."f18" < -0.4269999861717224
          THEN -0.3149999976158142
          ELSE 0.5008015036582947
        END
      END
    ) + 1
  ) AS "pred",
  "f1"
FROM "DATA_TABLE" AS "t0"
SELECT
  1 / (
    EXP(
      -CASE
        WHEN "t0"."f4" < -0.04800000041723251
        THEN CASE
          WHEN "t0"."f4" < -0.8119999766349792
          THEN -0.5087512135505676
          ELSE -0.21405750513076782
        END
        ELSE CASE
          WHEN "t0"."f18" < -0.4269999861717224
          THEN -0.3149999976158142
          ELSE 0.5008015036582947
        END
      END
    ) + 1
  ) AS "pred",
  "f1"
FROM "DATA_TABLE" AS "t0"

These appear to match!

However, if we go to use the results, we find that there are some non-equal predictions.

DATA_TABLE = pd.DataFrame(X_train, columns = nm_cols)
db_preds = duckdb.sql(sql_mod).df()
preds_orb = db_preds['pred']
print(f"xgb and orb match: {np.all(np.isclose(preds_xgb, preds_orb))}")

xgb and orb match: False
xgb and orb match: False

# isolate and size misses
misses = np.where(~np.isclose(preds_xgb, preds_orb))
print(f'Different predictions (N): {len(misses[0])}')
print(f'Different predictions (P): {len(misses[0]) / len(X_train):.4f}')

# pull out f4 and f18; notice that all discrepancies lie exactly at the splitting points
X_train[misses][:,[4,18]]

Different predictions (N): 5
Different predictions (P): 0.0005
Different predictions (N): 5
Different predictions (P): 0.0005

array([[-0.812, -0.515],
       [-0.812, -0.739],
       [ 1.715, -0.427],
       [ 0.025, -0.427],
       [ 2.119, -0.427]])

For setting up

The first hurdle is often getting started – both in terms of installing the tools you’ll need and getting into a comfortable IDE to run them.

• Installation: R keeps installation simple; there’s one way to do it so you do and it’s done³. But before python converts can print("hello world"), they face a range of options (system Python, Python installer UI, Anaconda, Miniconda, etc.) each with its own kinks. These decisions are made harder in Python since projects tend to have stronger dependencies of the language, requiring one to switch between versions. Fortunately, uv now makes this task easy with many different commands for:
  • Installing one or more specific versions: uv python install <version, constraints, etc.>
  • Listing all available installations: uv python list
  • Returning path of python executables: uv python find
  • Spinning up a quick REPL with a temporary python version and packages: e.g. uv run --python 3.12 --with pandas python
• Integrated Development Environment: Once R is install, R users are typically off to the races with the intuitive RStudio IDE which helps them get immediately hands-on with the REPL. With the UI divided into quadrants, users can write an R script, run it to see results in the console, conceptualize what the program “knows” with the variable explorer, and navigate files through a file explorer. Once again, python is not lacking in IDE options, but users are confronted with yet another decision point before they even get started. Pycharm, Sublime, Spyder, Eclipse, Atom, Neovim, oh my! For python, I’d recommend either VS Code or Positron, which are both extensions of Code OSS.
  • VS Code is an industry standard tool for software development. This means it has a rich set of features for coding, debugging, navigating large projects, etc. It’s rich extension ecosystem also means that most major tools (e.g. Quarto, git, linters and stylers, etc.) have nice add-ons so, like RStudio, you can customize your platform to perform many side-tasks in plaintext or with the support of extra UI components.⁴
  • Positron is a newer entrant from Posit PBC (formerly RStudio). It streamlines the offerings of VS Code to center the features most useful for data analysis. Positron may feel easier to go from zero-to-one. It does a great job finding and consistently using the right versions of R, python, Quarto, etc. and prioritizes many of the IDE elements that make RStudio wonderful for working with data (e.g. object preview pane). Additionally, most VS Code extensions will work in Positron; however, Positron cannot use extensions that rely on Microsoft’s PyLance meaning some realtime linting and error detection tools like ErrorLens do not work out-of-the-box. Ultimately, your comfort navigating VS Code and your mix of dev versus data work may determine which is best for you.

For data analysis

As data practitioners know, we’ll spend most of our time on cleaning and wrangling. As such, R users may struggle particularly to abandon their favorite tools for exploratory data analysis like dplyr and ggplot2. Fans of those packages often appreciate how their functional paradigm helps achieve a “flow state”. Precise syntax may differ, but new developments in the python wrangling stack provide increasingly close analogs to some of these beloved Rgonomics.

• Data Wrangling: (See my separate post on polars)Although pandas is undoubtedly the best-known wrangling tool in the python space, I believe the growing polars project offers the best experience for a transitioning developer (along with other nice-to-have benefits like being dependency free and blazingly fast). polars may feel more natural and less error-prone to R users for may reasons:
  • it has more internal consistent (and similar to dplyr) syntax such as select, filter, etc. and has demonstrated that the project values a clean API (e.g. recently renaming groupby to group_by)
  • it does not rely on the distinction between columns and indexes which can feel unintuitive and introduces a new set of concepts to learn
  • it consistently returns copies of dataframes (while pandas sometimes alters in-place) so code is more idempotent and avoids a whole class of failure modes for new users
  • it enables many of the same “advanced” wrangling workflows in dplyr with high-level, semantic code like making the transformation of multiple variables at once fast with column selectors, concisely expressing window functions, and working with nested data (or what dplyr calls “list columns”) with lists and structs
  • supporting users working with increasingly large data. Similar to dplyr’s many backends (e.g. dbplyr), polars can be used to write lazily-evaluated, optimized transformations and it’s syntax is reminiscent of pyspark should users ever need to switch between
• Visualization: Even some of R’s critics will acknowledge the strength of ggplot2 for visualization, both in terms of it’s intuitive and incremental API and the stunning graphics it can produce. seaborn’s object interface seems to strike a great balance between offering a similar workflow (which cites ggplot2 as an inspiration) while bringing all the benefits of using an industry-standard tool

For communication

Historically, one possible dividing line between R and python has been framed as “python is good at working with computers, R is good at working with people”. While that is partially inspired by reductive takes that R is not production-grade, it is not without truth that the R’s academic roots spurred it to overinvest in a rich “communication stack” and translating analytical outputs into human-readable, publishable outputs. Here, too, the gaps have begun to close.

• Tables: R has no shortage of packages for creating nicely formatted tables, an area that has historically lacked a bit in python both in workflow and outcomes. Barring strong competition from the native python space, the one “port” I am bullish about is the recently announced Great Tables package. This is a pythonic clone of R’s gt package. I’m more comfortable recommending this since it’s maintained by the same developer as the R version (to support long-term feature parity), backed by an institution not just an individual (to ensure it’s not a short-lived hobby project), and the design feels like it does a good job balancing R inspiration with pythonic practices
• Computational notebooks: Jupyter Notebooks are widely used, widely critiqued parts of many python workflows. While the ability to mix markdown and code chunks. However, notebooks can introduce new types of bugs for the uninitiated; for example, they are hard to version control and easy to execute in the wrong environment. For those coming from the world of R Markdown, plaintext computational notebooks like Quarto may provide a more transparent development experience. While Quarto allows users to write in .qmd files which are more like their .rmd predecessors, its renderer can also handle Jupyter notebooks to enable collaboration across team members with different preferences

Miscellaneous

A few more tools may be helpful and familiar to some R users who tend towards the more “developer” versus “analyst” side of the spectrum. These, in my mind, have even more varied pros and cons, but I’ll leave them for consideration:

• Environment Management: There’s a truly overwhelming number of ways⁵ to manage project-level dependencies in python. As a consequence, there’s also a lot of outdated advice weighing pros and cons of feature sets that have since evolved. Here again, uv takes the cake as a swiss army knife tool. It features fast installation, auto-updating of the pyproject.toml and uv.lock files (so you don’t need to remember to pip freeze), separate trakcing of primary dependencies from the fully resolved environment (so you can cleanly and completely remove dependencies-of-dependencies you no longer need), and so much more. uv can operate as a drop in replacement for pip and generate a requirements.txt if needed for compatability; however, given it’s explosive popularity and ergonomic design, I doubt you’ll have trouble convincing collaborators to adopt the same.
• Developer Tools: ruff (another Astral project) provides a range of linting and styling options (think R’s lintr and styler) and provides a one-stop-shop over what can be an overwhelming number of atomic tools in this space (isort, black, flake8, etc.). ruff is super fast, has a nice VS Code extension, and, while this class of tools is generally considered more advanced, I think linters can be a fantastic “coach” for new users about best practices

Configure Role Authentiation

Netlify maintains an Identity widget that handles recognizing authenticated users and their roles from their JWTs. To inject this Javascript snippet into every page, open the _quarto.yml file and add the Javascript snippet to the include-in-header: key under the HTML format, e.g.:

format:
  html: 
    include-in-header: 
      text: |
        <script type="text/javascript" src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
        <script>
        window.netlifyIdentity.on('login', (user) => {
        window.netlifyIdentity.refresh(true).then(() => {
          console.log(user);
        });
        });
        window.netlifyIdentity.on('logout', (user) => {
        window.location.href = '/login';
        });
        window.netlifyIdentity.init({ container: '#netlify' });
        </script>

Note, the official widget is injected using the src field of the first script tag.

Configure Site Redirects

Next, create a _redirects file at the top level of your project (or open the existing file) and add the following lines:

/login /
/*  /:splat  200!  Role=admin
/site_libs/* /site_libs/:splat 200!
/   /        200!
/*  /  401!

Syntax for the _redirects file is described here, but basically each line defines a rule with the structure:

<what was requested> <where to go> <the result to give> <conditional on role>

And, like a case when statement, the first “matching” rule dominates.

So, the example above can roughly be read in English as:

If users go to the /login page, take them back to home
If users try to go anywhere else on my site and they have role admin, let them do that 
If users try to go to the hompage of my site (regardless of their role), let them do that
If users otherwise try to go to other parts of the site (and they don't have admin), give an error

Of course, this could be further customized to set different rules for different subdirectories.

Create User Interface

To create the user interface for the login screen, I added code to inject a Netlify-maintained login widget to my site’s index.qmd, e.g.:

---
date: last-modified
---

# Home {.unnumbered}

<div data-netlify-identity-menu></div>

Welcome! Please sign in to view the dashboard. 

If you are a first time user, please create a login and email [emilyriederer@gmail.com](mailto:emilyriederer@gmail.com?subject=Dashboard%20Access%20Request) to elevate your access.

User Onboarding

After the changes above to your actual Quarto site, the rest of the work lies in the Netlify admin panel. For a small number of users, you can manually change their role in the user interface.

However, to work at any scale, you may need a more automated solution. For that, Netlify’s docs explain how to configure initial role assignment via lambda functions. However, out-of-the box functionality that I found to be lacking was assigning default roles for new users or the ability to configure basic logic such as assigning the same role to any new users onboarding from a certain email domain.

1. What columns are in the table?

Without a good data catalog, analysts will first need to check what fields exist in a table. While there may be lower cost ways to do this like looking at a pre-rendered preview (ala BigQuery), using a DESCRIBE statement (ala Spark), or limiting their query to the first few rows, some analysts may default to requesting all the data.

select *
from invoices;

2. Is the table still live and updating?

After establishing that a table has potentially useful information, analysts should next wonder if the data is still live and updating. First they might check a date field to see if the table seems “fresh”.

select max(order_date) 
from invoices;

But, of course, tables often have multiple date fields. For example, an e-commerce invoice table might have fields for both the date an order was placed and the date the record was last modified. So, analysts may guess-and-check a few of these fields to determine table freshness.

select max(updated_date) 
from invoices;

After identifying the correct field, there’s still a question of refresh cadence. Are records added hourly? Daily? Monthly? Lacking system-level metrics and metadata on the upstream table freshness, analysts are still left in the dark. So, once again, they can check empirically by looking at the frequency of the date field.

select max(updated_date), count(1) as n
from invoices
group by 1;

3. What is the grain of the table?

Now that the table is confirmed to be usable, the question becomes how to use it. Specifically, to credibly query and join the table, analysts next must determine its grain. Often, they start with a guess informed by the business context and data modeling conventions, such as assuming an invoice table is unique by order_id.

select count(1) as n, count(distinct order_id)
from invoices;

‍However, if they learn that order_id has a different cardinality then the number of records, they must ask why. So, once again, they scan the full table to find examples of records with shared order_id values.

select *
from invoices
qualify count(1) over (partition by order_id) > 1
order by order_id
limit 10;

Eyeballing the results of this query, the analysts might notice that the same order_id value can coincide with different ship_id values, as a separate invoice is generated for each part of an order when a subset of items is shipped. With this new hypothesis, the analyst iterates on the validation of the grain.

select count(1) as n, count(distinct order_id, ship_id)
from invoices;

4. What values can categorical variables take?

The prior questions all involved table structure. Only now can an analyst finally begin to investigate the table’s content. A first step might be to understand the valid values for categorical variables. For example, if our analyst wanted to ensure only completed orders were queried, they might inspect the potential values of the order_status_id field to determine which values to include in a filter.

select distinct order_status_id
from invoices;

They’ll likely repeat this process for many categorical variables of interest. Since our analyst is interested in shipments specifically to Ohio, they might also inspect the cardinality of the ship_state field to ensure they correctly format the identifier.

select distinct ship_state
from invoices;

5. Do numeric columns have nulls or ‘sentinel’ values to encode nulls?

Similarly, analysts may wish to audit other variables for null handling or sentinel values by inspecting column-level statistics.

select distinct ship_state
from invoices;

6. Is the data stored with partitioning or clustering keys?

Inefficient queries aren’t only a symptom of ad hoc data validation. More complex and reused logic may also be written wastefully when table metadata like partitioning and clustering keys is not available to analysts. For example, an analyst might be able to construct a reasonable query filtering either on a shipment date or an order date, but if only one of these is a partitioning or clustering key, different queries could have substantial performance differences.

1. What columns are in the table? And do we need a table?

Comprehensive search across all of an organization’s assets can help users quickly identify the right resources based on table names, field names, or data descriptions. Even better, search can incorporate observed tribal knowledge of table popularity and common querying patterns to prioritize the most relevant results. Moreover, when search also includes downstream data products like pre-built reports and dashboards, analysts might sometimes find an answer to their question exists off the shelf.

2. Is the table still live and updating? And are its own sources current?

Data is not a static artifact so metadata should not be either. After analysts identify a candidate table, they should have access to real-time operational information like table usage, table size, refresh date, and upstream dependencies to help confirm whether the table is a reliable resource.

Ideally, analysts can interrogate not just the freshness of a final table but also its dependencies by exploring the table’s data lineage.

3. What is the grain of the table? And how does it relate to others?

Table grain should be clearly documented at the table level and emphasized in the data dictionary via references to primary and foreign keys. Beyond basic documentation, entity-relationship (ER) diagrams will help analysts gain a richer mental model of grains of how they can use these primary-foreign key relationships to link tables to craft information with the desired grain and fields. Alternatively, they can glean this information from the wisdom of the crowds if they have access to how others have queried and joined the data previously.

4. What values can categorical variables take? Do numeric columns have nulls or ‘sentinel’ values to encode nulls?

Information about proper expectations and handling of categorical and null values may be published as field definitions, pointed to lookup tables, implied in data tests, or illustrated in past queries. To drive consistency and offload redundant work from data producers, such field definitions can be propagated from upstream tables.

‍5. Is the data stored with partitioning or clustering keys?

Analysts cannot write efficient code if they don’t know where the efficiency gains lie. Table-level documentation should clearly highlight the use of clustering or partitioning files so analysts can use the most impactful variables in filters and joins. Here, consistency of documentation is paramount; analysts may not always be incented to care about query efficiency, so if this information is hard to find or rarely available, they can be easily dissuaded from looking.

Beyond a poor user experience, poor data discoverability creates inefficiency and added cost. Even if you don’t have large scale historical data or broad data user communities today, slow queries and tedious work still detract from data team productivity while introducing context-switching and chaos. By focusing on improving data discoverability, you can streamline workflows and enhance the overall efficiency of your data operations.

Application: Generating File Names

Combining what we’ve discussed about datetime and f-strings, here’s a pattern I use frequently. If I am logging results from a run of some script, I might save the results in a file suffixed with the run timestamp. We can generate this easily.

dt_stub = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
file_name = f"output-{dt_stub}.csv"
print(file_name)

output-20240120_071517.csv

Application: Simulation

As a (slightly) more realistic(ish) example, let’s consider how list comprehensions might help us conduct a numerical simulation or sensitivity analysis.

Suppose we want to simulate 100 draws from a Bernoulli distribution with different success probabilites and see how close our empirically calculated rate is to the true rate.

We can define the probabilites we want to simulate in a list and use a list comprehension to run the simulations.

import numpy as np
import numpy.random as rnd

probs = [0.1, 0.25, 0.5, 0.75, 0.9]
coin_flips = [ np.mean(np.random.binomial(1, p, 100)) for p in probs ]
coin_flips

[0.05, 0.3, 0.48, 0.77, 0.87]

Alternatively, instead of returning a list of the same length, our resulting list could include whatever we want – like a list of lists! If we wanted to keep the raw simulation results, we could. The following code returns a list of 5 lists - one with the raw simulation results.

coin_flips = [ list(np.random.binomial(1, p, 100)) for p in probs ]
print(f"""
  coin_flips has {len(coin_flips)} elements
  Each element is itself a {type(coin_flips[0])}
  Each element is of length {len(coin_flips[0])}
  """)

  coin_flips has 5 elements
  Each element is itself a <class 'list'>
  Each element is of length 100
  

If one wished, they could then put these into a polars dataframe and pivot those list-of-lists (going from a 5-row dataset to a 500-row dataset)to conduct whatever sort of analysis with want with all the replicates.

import polars as pl

df_flips = pl.DataFrame({'prob': probs, 'flip': coin_flips})
df_flips.explode('flip').glimpse()

Rows: 500
Columns: 2
$ prob <f64> 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1
$ flip <i32> 0, 0, 0, 0, 1, 0, 1, 1, 0, 0

We’ll return to list comprehensions in the next section.

Fake Datasets

For the simplest of datasets, we can manually write a few entries as with data.frame() in R. Here, we define series in a named dictionary where each dictionary key turns into a column name.

import polars as pl

pl.DataFrame({
  'a': [1,2,3],
  'b': ['x','y','z']
})

If we need longer datasets, we can use helper functions in packages like numpy to generate the series. Methods like arange and linspace work similarly to R’s seq().

import polars as pl
import numpy as np

pl.DataFrame({
  'a': np.arange(stop = 3),
  'b': np.linspace(start = 9, stop = 24, num = 3)
})

If we need groups in our sample data, we can use np.repeat() which works like R’s rep(each = TRUE).

pl.DataFrame({
  'a': np.repeat(np.arange(stop = 3), 2),
  'b': np.linspace(start = 3, stop = 27, num = 6)
})

Alternatively, for more control and succinct typing, we can created a nested dataset in polars and explode it out.

(
  pl.DataFrame({
    'a': [1, 2, 3],
    'b': ["a b c", "d e f", "g h i"]
  })
  .with_columns(pl.col('b').str.split(" "))
  .explode('b')
)

Similarly, we could use what we’ve learned about polars list columns and list comprehensions.

a = [1, 2, 3]
b = [ [q*i for q in [1, 2, 3]] for i in a]
pl.DataFrame({'a':a,'b':b}).explode('b')

In fact, multidimensional list comprehensions can be used to mimic R’s expand.grid() function.

pl.DataFrame(
  [(x, y) for x in range(3) for y in range(3)],
  schema = ['x','y']
  )

Built-In Data

R has a number of canonical datasets like iris built in to the core language. This can be easy to quickly grab for experimentation⁴. While base python doesn’t include such capabilities, many of the exact same or similar datasets can be found in seaborn.

seaborn.get_dataset_names() provides the list of available options. Below, we load the Palmers Penguins data and, if you wish, convert it from pandas to polars.

import seaborn as sns
import polars as pl

df_pd = sns.load_dataset('penguins')
df = pl.from_pandas(df_pd)
df.glimpse()

Rows: 344
Columns: 7
$ species           <str> 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie', 'Adelie'
$ island            <str> 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen', 'Torgersen'
$ bill_length_mm    <f64> 39.1, 39.5, 40.3, None, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0
$ bill_depth_mm     <f64> 18.7, 17.4, 18.0, None, 19.3, 20.6, 17.8, 19.6, 18.1, 20.2
$ flipper_length_mm <f64> 181.0, 186.0, 195.0, None, 193.0, 190.0, 181.0, 195.0, 193.0, 190.0
$ body_mass_g       <f64> 3750.0, 3800.0, 3250.0, None, 3450.0, 3650.0, 3625.0, 4675.0, 3475.0, 4250.0
$ sex               <str> 'Male', 'Female', 'Female', None, 'Female', 'Male', 'Female', 'Male', None, None

Main Verbs

dplyr and polars offer the same foundational functionality for manipulating dataframes. Their APIs for these operations are substantially similar.

For a single dataset:

• Column selection: select() -> select() + drop()
• Creating or altering columns: mutate() -> with_columns()
• Subsetting rows: filter() -> filter()
• Ordering rows: arrange() -> sort()
• Computing group-level summary metrics: group_by() + summarize() -> group_by() + agg()

For multiple datasets:

• Merging on a shared key: *_join() -> join(strategy = '*')
• Stacking datasets of the same structure: union() -> concat()
• Transforming rows and columns: pivot_{longer/wider}()² -> pivot()

Main Verb Design

Beyond the similarity in naming, dplyr and polars top-level functions are substantially similar in their deeper design choices which impact the ergonomics of use:

• Referencing columns: Both make it easy to concisely references columns in a dataset without the repeated and redundant references to said dataset (as sometimes occurs in base R or python’s pandas). dplyr does this through nonstandard evaluation wherein a dataframe’s coumns can be reference directly within a data transformation function as if they were top-level variables; in polars, column names are wrapped in pl.col()
• Optional argument: Both tend to have a wide array of nice-to-have optional arguments. For example the joining capabilities in both libraries offer optional join validation³ and column renaming by appended suffix
• Consistent dataframe-in -> dataframe-out design: dplyr functions take a dataframe as their first argument and return a dataframe. Similarly, polars methods are called on a dataframe and return a dataframe which enables the chaining workflow discussed next

Chaining (Piping)

These methods are applied to polars dataframes by chaining which should feel very familiar to R dplyr fans.

In dplyr and the broad tidyverse, most functions take a dataframe as their first argument and return a dataframe, enabling the piping of functions. This makes it easy to write more human-readable scripts where functions are written in the order of execution and whitespace can easily be added between lines. The following lines would all be equivalent.

transformation2(transformation1(df))

df |> transformation1() |> transformation2()

df |>
  transformation1() |>
  transformation2()

Similarly, polars’s main transfomration methods offer a consistent dataframe-in dataframe-out design which allows method chaining. Here, we similarly can write commands in order where the . beginning the next method call serves the same purpose as R’s pipe. And for python broadly, to achieve the same affordance for whitespace, we can wrap the entire command in parentheses.

(
  df
  .transformation1()
  .transformation2()
)

One could even say that polars dedication to chaining goes even deeper than dplyr. In dplyr, while core dataframe-level functions are piped, functions on specific columns are still often written in a nested fashion⁴

df %>% mutate(z = g(f(a)))

In contrast, most of polars column-level transformation methods also make it ergonomic to keep the same literate left-to-right chaining within column-level definitions with the same benefits to readability as for dataframe-level operations.

df.with_columns(z = pl.col('a').f().g())

Explicit API for row-wise operations

While row-wise operations are relatively easy to write ad-hoc, it can still be nice semantically to have readable and stylistically consistent code for such transformations.

dplyr’s rowwise() eliminates ambiguity in whether subsequent functions should be applied element-wise or collectively. Similiarly, polars has explicit *_horizontal() functions.

df.with_columns(
  b_plus_c = pl.sum_horizontal(pl.col('b'), pl.col('c')) 
)

Column Selectors

dplyr’s column selectors dynamically determine a set of columns based on pattern-matching their names (e.g. starts_with(), ends_with()), data types, or other features. I’ve previously written and spoken at length about how transformative this functionality can be when paired with

polars has a similar set of column selectors. We’ll import them and see a few examples.

import polars.selectors as cs

To make things more interesting, we’ll also turn one of our columns into a different data type.

df = df.with_columns(pl.col('a').cast(pl.Utf8))

df.select(cs.starts_with('b') | cs.string())

df.select(~cs.string())

df.with_columns(
  cs.integer().add(1).name.suffix("_plus1")
)

df.with_columns(
  row_total = pl.sum_horizontal(cs.integer())
)

df.group_by(cs.string()).agg(cs.integer().sum())

Consistent API for Window Functions

Window functions are another incredibly important tool in any data wrangling language but seem criminally undertaught in introductory analysis classes. Window functions allows you to apply aggregation logic over subgroups of data while preserving the original grain of the data (e.g. in a table of all customers and orders and a column for the max purchase account by customer).

dplyr make window functions trivially easy with the group_by() + mutate() pattern, invoking users’ pre-existing understanding of how to write aggregation logic and how to invoke transformations that preserve a table’s grain.

polars takes a slightly different but elegant approach. Similarly, it reuses the core with_columns() method for window functions. However, it uses a more SQL-reminiscent specification of the “window” in the column definition versus a separate grouping statement. This has the added advantage of allowing one to use multiple window functions with different windows in the same with_columns() call if you should so choose.

A simple window function tranformation can be done by calling with_columns(), chaining an aggregation method onto a column, and following with the over() method to define the window of interest.

df.with_columns(
  min_b = pl.col('b').min().over('a')
)

The chaining over and aggregate and over() can follow any other arbitrarily complex logic. Here, it follows a basic “case when”-type statement that creates an indicator for whether column b is null.

df.with_columns(
  n_b_odd = pl.when( (pl.col('b') % 2) == 0)
              .then(1)
              .otherwise(0)
              .sum().over('a')
)

List Columns and Nested Frames

While the R tidyverse’s raison d’etre was originally around the design of heavily normalize tidy data, modern data and analysis sometimes benefits from more complex and hierarchical data structures. Sometimes data comes to us in nested forms, like from an API⁶, and other times nesting data can help us perform analysis more effectively⁷ Recognizing these use cases, tidyr provides many capability for the creation and manipulation of nested data in which a single cell contains values from multiple columns or sometimes even a whoel miniature dataframe.

polars makes these operations similarly easy with its own version of structs (list columns) and arrays (nested dataframes).

df.with_columns(list_col = pl.struct( cs.integer() ))

df.group_by('a').agg(list_col = pl.struct( cs.integer() ) )

cols = ['a']
(df
  .group_by(cs.by_name(cols))
  .agg(list_col = pl.struct(~cs.by_name(cols)))
)

df_nested = df.group_by('a').agg(list_col = pl.struct( cs.integer() ) )

df_nested.explode('list_col')

df_nested.explode('list_col').unnest('list_col')

For setting up

The first hurdle is often getting started – both in terms of installing the tools you’ll need and getting into a comfortable IDE to run them.

• Installation: R keeps installation simple; there’s one way to do it* so you do and it’s done. But before python converts can print("hello world"), they face a range of options (system Python, Python installer UI, Anaconda, Miniconda, etc.) each with its own kinks. These decisions are made harder in Python since projects tend to have stronger dependencies of the language, requiring one to switch between versions. For both of these reasons, I favor the pyenv (or pyenv-win for those on Windows) for easily managing python installation(s) from the command line. While the installation process of pyenv may be technically different, it’s similar in that it “just works” with just a few commands. In fact, the workflow is so slick that things seem to have gone 180 degrees with pyenv inspiring similar project called rig to manage R installations. This may sound intimidating, but the learning curve is actually quite shallow:
  • pyenv install --list: To see what python versions are available to install
  • pyenv install <version number>: To install a specific version
  • pyenv versions: To see what python versions are installed on your system
  • pyenv global <version number>: The set one python version as a global default
  • pyenv local <version number>: The set a python version to be used within a specific directory/project
• Integrated Development Environment: Once R is install, R users are typically off to the races with the intuitive RStudio IDE which helps them get immediately hands-on with the REPL. With the UI divided into quadrants, users can write an R script, run it to see results in the console, conceptualize what the program “knows” with the variable explorer, and navigate files through a file explorer. Once again, python is not lacking in IDE options, but users are confronted with yet another decision point before they even get started. Pycharm, Sublime, Spyder, Eclipse, Atom, Neovim, oh my! I find that VS Code offers the best functionality. It’s rich extension ecosystem also means that most major tools (e.g. Quarto, git, linters and stylers, etc.) have nice add-ons so, like RStudio, you can customize your platform to perform many side-tasks in plaintext or with the support of extra UI components.³

For data analysis

As data practitioners know, we’ll spend most of our time on cleaning and wrangling. As such, R users may struggle particularly to abandon their favorite tools for exploratory data analysis like dplyr and ggplot2. Fans of those packages often appreciate how their functional paradigm helps achieve a “flow state”. Precise syntax may differ, but new developments in the python wrangling stack provide increasingly close analogs to some of these beloved Rgonomics.

• Data Wrangling: Although pandas is undoubtedly the best-known wrangling tool in the python space, I believe the growing polars project offers the best experience for a transitioning developer (along with other nice-to-have benefits like being dependency free and blazingly fast). polars may feel more natural and less error-prone to R users for may reasons:
  • it has more internal consistent (and similar to dplyr) syntax such as select, filter, etc. and has demonstrated that the project values a clean API (e.g. recently renaming groupby to group_by)
  • it does not rely on the distinction between columns and indexes which can feel unintuitive and introduces a new set of concepts to learn
  • it consistently returns copies of dataframes (while pandas sometimes alters in-place) so code is more idempotent and avoids a whole class of failure modes for new users
  • it enables many of the same “advanced” wrangling workflows in dplyr with high-level, semantic code like making the transformation of multiple variables at once fast with column selectors, concisely expressing window functions, and working with nested data (or what dplyr calls “list columns”) with lists and structs
  • supporting users working with increasingly large data. Similar to dplyr’s many backends (e.g. dbplyr), polars can be used to write lazily-evaluated, optimized transformations and it’s syntax is reminiscent of pyspark should users ever need to switch between
• Visualization: Even some of R’s critics will acknowledge the strength of ggplot2 for visualization, both in terms of it’s intuitive and incremental API and the stunning graphics it can produce. seaborn’s object interface seems to strike a great balance between offering a similar workflow (which cites ggplot2 as an inspiration) while bringing all the benefits of using an industry-standard tool

For communication

Historically, one possible dividing line between R and python has been framed as “python is good at working with computers, R is good at working with people”. While that is partially inspired by reductive takes that R is not production-grade, it is not without truth that the R’s academic roots spurred it to overinvest in a rich “communication stack” and translating analytical outputs into human-readable, publishable outputs. Here, too, the gaps have begun to close.

• Tables: R has no shortage of packages for creating nicely formatted tables, an area that has historically lacked a bit in python both in workflow and outcomes. Barring strong competition from the native python space, the one “port” I am bullish about is the recently announced Great Tables package. This is a pythonic clone of R’s gt package. I’m more comfortable recommending this since it’s maintained by the same developer as the R version (to support long-term feature parity), backed by an institution not just an individual (to ensure it’s not a short-lived hobby project), and the design feels like it does a good job balancing R inspiration with pythonic practices
• Computational notebooks: Jupyter Notebooks are widely used, widely critiqued parts of many python workflows. While the ability to mix markdown and code chunks. However, notebooks can introduce new types of bugs for the uninitiated; for example, they are hard to version control and easy to execute in the wrong environment. For those coming from the world of R Markdown, plaintext computational notebooks like Quarto may provide a more transparent development experience. While Quarto allows users to write in .qmd files which are more like their .rmd predecessors, its renderer can also handle Jupyter notebooks to enable collaboration across team members with different preferences

Miscellaneous

A few more tools may be helpful and familiar to some R users who tend towards the more “developer” versus “analyst” side of the spectrum. These, in my mind, have even more varied pros and cons, but I’ll leave them for consideration:

• Environment Management: Joining the python world means never having to settle on an environment management tool for installing packages. There’s a truly overwhelming number of ways to manage project-level dependencies (virtualenv, conda, piptools, pipenv, poetry, and that doesn’t even scratch the surface) with different pros and cons and phenomenal amount of ink/pixels have been spilled over litigating these trade-offs. Putting all that aside, lately, I’ve been favoring pdm because it prioritizes features I care most about (auto-updating pyproject.toml, isolating dependencies from dependencies-of-dependencies, active development and error handling, mostly just works pretty undramatically)
• Developer Tools: ruff provides a range of linting and styling options (think R’s lintr and styler) and provides a one-stop-shop over what can be an overwhelming number of atomic tools in this space (isort, black, flake8, etc.). ruff is super fast, has a nice VS Code extension, and, while this class of tools is generally considered more advanced, I think linters can be a fantastic “coach” for new users about best practices