Python Polars: The Definitive Cheatsheet :: Posit Open Source

posit::conf is September 14-16 in Houston, TX! Register now to attend in person or virtually.

Resources

Great Tables

plotnine

Python

Python Polars: The Definitive Cheatsheet

Quick reference guide for transforming, analyzing, and visualizing data with Python Polars

Read more...

Jeroen Janssens,Thijs Nieuwdorp

Download PDF

Polars is a library for transforming, analyzing, and visualizing data with a fast<br>and expressive DataFrame API.<br>It was first released by Ritchie Vink in 2020.

Install Polars with all of its optional dependencies from the terminal:

uv pip install "polars[all]"

Import Polars in Python, and confirm which versions of Polars and its<br>dependencies you have installed:

import polars as pl

pl.show_versions()

Polars queries typically read data, transform it, and write the result back out.<br>A complete query is often a single chain of method calls:

fruit = pl.read_csv("fruit.csv")

fruit.filter(<br>(pl.col("weight") > 1000) & pl.col("is_round")<br>).write_parquet("fruit.parquet")

Throughout this cheatsheet, df is a DataFrame, lf is a LazyFrame, o is a<br>second DataFrame to combine with df, and e stands for any expression.<br>So e.abs() means &ldquo;call .abs() on an expression&rdquo;, as in pl.col("x").abs().

Data Structures#

Polars stores all of its data in either a Series or a DataFrame.

Structure<br>Description

Series<br>One-dimensional. Holds a sequence of values of the same data type.

DataFrame<br>Two-dimensional. Has rows and columns. One or more Series, all of the same length.

LazyFrame<br>Resembles a DataFrame but holds no data. A blueprint for generating a DataFrame.

Unlike pandas, Polars DataFrames do not have a row index, and the API favors<br>immutability and method chaining over in-place modifications.

Create a Series by passing a name and a sequence of values:

series = pl.Series("sales", [150.00, 300.00, 250.00])

Create a DataFrame from a dictionary of columns, where each value is a Series<br>or a plain Python sequence.<br>You can also use any of the pl.read_*() functions to create one from a file:

df = pl.DataFrame({<br>"sales": series,<br>"id": [41, 42, 43]<br>})

Because there is no row index, add one explicitly as a column when you need it:

df.with_row_index("id")

Turn a DataFrame into a LazyFrame.<br>Alternatively, start from a LazyFrame directly with any of the pl.scan_*()<br>functions:

lf = df.lazy()

Eager and Lazy APIs#

The eager API executes immediately, whereas the lazy API builds an optimized query<br>plan first.<br>The optimizer automatically applies predicate pushdown (filtering as early as<br>possible) and projection pushdown (dropping columns that are never used).

You move between the two representations with .lazy() and .collect(): .lazy()<br>turns a DataFrame into a LazyFrame, and .collect() executes a LazyFrame and gives<br>you a DataFrame back.

Turn a DataFrame into a LazyFrame, and execute a LazyFrame to get a DataFrame:

lf = df.lazy()<br>df = lf.collect()

Use the streaming engine to process data out-of-core, so that datasets larger<br>than memory can still be handled:

lf.collect(engine="streaming")

Print the optimized query plan as text, or visualize it as a graph, to see what<br>the optimizer decided to do:

lf.explain()<br>lf.show_graph()

Execute the query and return per-node timings, which tells you where the time<br>actually goes:

lf.profile()

Data Types#

Polars implements most of the Apache Arrow memory specification, which is an<br>efficient columnar format for flat and hierarchical data.

Group<br>Type<br>Notes

Numeric<br>Decimal<br>128 bits, precision, scale

Float32<br>Ranges ±3.4×10³⁸

Float64<br>Ranges ±1.8×10³⁰⁸

Int8<br>Ranges ±128

Int16<br>Ranges ±32,768

Int32<br>Ranges ±2.1×10⁹

Int64<br>Ranges ±9.2×10¹⁸

Int128<br>Ranges ±3.4×10³⁸

UInt8<br>Ranges 0–255

UInt16<br>Ranges 0–65,535

UInt32<br>Ranges 0–4.3×10⁹

UInt64<br>Ranges 0–1.8×10¹⁹

Temporal<br>Date<br>Days since Unix epoch

Datetime<br>Microseconds since epoch

Duration<br>Time duration / delta

Time<br>Time of day

Nested<br>Array<br>Fixed-length sequence

List<br>Variable-length sequence

Struct<br>Multiple fields with names

String<br>String<br>UTF-8 text, variable length

Categorical<br>Dict of Strings

Enum<br>Fixed dict of Strings

Other<br>Boolean<br>True / False

Binary<br>Raw bytes

Null<br>Represents Null / None

Inspecting Types#

Get a dictionary of column names and data types, or just the list of data types:

df.schema<br>df.dtypes

Print one row per column, including data types, which is useful for wide<br>DataFrames where printing the DataFrame itself is unreadable:

df.glimpse()

Compute per-column summary statistics, including the number of nulls:

df.describe()

Report the in-memory size of the DataFrame in the unit you ask for:

df.estimated_size("mb")

Casting#

Cast a column to another data type.<br>By default the cast is strict, so a value that does not fit raises an error:

df.select(pl.col("id").cast(pl.UInt64))

Pass strict=False to cast without raising.<br>Values that overflow the target type become nulls...