Spy: language semantics for a statically-typed compiled variant of Python

Inside SPy, part 2: Language semantics - Antonio Cuni's blog

Initializing search

Related links

Inside SPy 🥸, part 2: Language semantics¶

This is the second post of the Inside SPy series. The first post was mostly about motivations and goals of SPy. This post will cover in more detail the semantics of SPy, including the parts which make it different from CPython.

We will talk about phases of execution, colors, redshifting, the very peculiar way SPy implements static typing, and we will start to dive into metaprogramming.

Before diving in, I want to express my gratitude to my employer, Anaconda, for giving me the opportunity to dedicate 100% of my time to this open-source project.

Motivation and goals, recap¶

Shameless plug: give SPy a star ⭐

I admit I never cared much about GitHub stars, but it looks like nowadays it's what you need to be considered important. We are at 678 stars at the moment of writing, let's try to get to 5000!

Part 1 describes motivations and goals in great detail, but let's do a quick recap.

The main motivation is to make Python faster; by "faster" I mean comparable to C, Rust and Go. After spending 20 years in this problem space, I am convinced that it's impossible to achieve such a goal without breaking compatibility.

The second motivation is that static typing is playing a more and more important role in the Python community, but Python is not a language designed for that, which leads to a suboptimal experience.

There are two possible definitions of SPy; both of them are accurate, although from very different perspectives:

SPy is an interpreter and a compiler for a statically typed variant of Python, with focus on performance.

and:

SPy is a thought experiment to determine how much dynamicity we can remove from Python while still feeling Pythonic.

The part about "interpreter and compiler" is fundamental: the interpreter is needed for ease of development and debugging, the compiler is needed for speed. The job of SPy is to ensure that the two pieces have the exact same semantics so that the compilation step is just a transparent speedup.

100% compatibility with Python is explicitly not a goal . The Zen of SPy contains the goals and design guidelines of SPy. This is a shortened version, see the link for full details:

Easy to use and implement.

We have an interpreter.

We have a compiler.

Static typing.

Performance matters.

Predictable performance.

Rich metaprogramming capabilities.

Zero cost abstractions.

Opt-in dynamism.

One language, two levels.

SPy version and playground

At the moment of writing, SPy is still changing very rapidly and it's very likely that some of the examples will break in the future. We don't have any official release yet, but all the following examples have been tried on SPy commit 229235b8.

All the examples have a Try it yourself button which opens the code snippet in the SPy Playground, a PyScript app to try SPy directly in the browser. The official SPy playground tracks the latest git main, while this blog post uses a custom version pinned to this exact commit.

Phases of execution and compilation pipeline¶

From the point of view of the user, SPy code runs in three distinct execution phases :

Import time : this is when we run all the module-level code, including global variable initializers, decorators, metaclasses, etc.. After this phase, all the globals are frozen .

Redshift : during this phase we apply partial evaluation to all expressions that are safe to be evaluated eagerly. This is an optional phase which happens only during compilation or when explicitly requested. The presence/absence of redshift should not have any visible effects on the behavior of the program.

Runtime : the actual execution of the program, starting from a main function.

The following is a diagram representing the compilation pipeline in the simplified case of a single .spy file:

graph TD

subgraph FRONTEND["Import time"] SRC["*.spy source"]:::node AST["Untyped AST"]:::node SYMAST["Untyped AST + symtable"]:::node IMPORTLABEL(["import"]):::label

SRC -- parse --> AST AST -- ScopeAnalyzer --> SYMAST SYMAST --- IMPORTLABEL end

SPyVM["SPyVM"]:::node IMPORTLABEL --> SPyVM

subgraph RS[" "] RSLABEL(["redshift"]):::label REDSHIFTED["Typed AST"]:::node end

C["C Source (.c)"]:::node EXE_NAT["Native exe"]:::node EXE_WASI["WASI exe"]:::node EXE_EM["Emscripten exe"]:::node

subgraph RT["Runtime"] INTERP(["interp"]):::label DOPPLER(["interp(doppler)"]):::label EXECUTE_NAT(["execute"]):::label EXECUTE_WASI(["execute"]):::label EXECUTE_EM(["execute"]):::label end

OUT["Output"]:::node SPyVM --- INTERP --> OUT

SPyVM --- RSLABEL --> REDSHIFTED REDSHIFTED -- cwrite --> C REDSHIFTED --- DOPPLER --> OUT

C -- cc --> EXE_NAT --- EXECUTE_NAT --> OUT C -- cc --> EXE_WASI --- EXECUTE_WASI --> OUT C -- cc --> EXE_EM --- EXECUTE_EM --> OUT

style FRONTEND fill:#fafaff,stroke:#9090cc,stroke-dasharray:5 5 style RS...

Spy: language semantics for a statically-typed compiled variant of Python

Related Articles

Amazon, Facebook, FBI have access to a private intelligence-sharing network

SpaceX not the behemoth everyone thought

The Mirror Is Part of the Machine

Elevated error rates on requests to multiple models

Donald Trump and sons to be 'forever' exempt from tax audits