Why PartiQL?

PartiQL is a SQL-compatible query language that gives you one way to query relational, semi-structured, and nested data — wherever it lives. It is backwards compatible with SQL-92, so standard SQL-92 queries are valid PartiQL, and it extends SQL with a small number of carefully chosen additions: path navigation into nested values, unnesting collections in the FROM clause, and well-defined behavior for missing attributes and schemaless data. What makes PartiQL unique is that it separates the syntax and semantics of a query from the underlying data format and data store. The same query can run over tables, JSON or Amazon Ion documents, or data whose shape is only known at runtime, and the language is defined by a formal specification.

Gradually typed

PartiQL is a gradually typed query language: it uses as much type information as you give it, and no more.

  • Without a schema, PartiQL operates on completely schemaless data. Queries run over heterogeneous documents whose attributes may be present, absent, or of different types from one value to the next. In permissive mode, missing attributes and type mismatches evaluate to MISSING under well-defined semantics rather than failing the query.

  • With table definitions, PartiQL uses them to type-check your queries. In strict mode, you get the compile-time errors (for example, referencing a column that does not exist or applying an operator to incompatible types) and the execution errors that you would expect from other SQL dialects.

This means the same language serves both exploratory queries over raw, semi-structured data and production workloads over well-defined tables, and you can add schema incrementally as your data matures.

A modular set of libraries

PartiQL is more than a language — it is an open-source toolkit for building query systems. Each stage of query processing is available as its own library, so you can take exactly the pieces you need:

  • AST — a typed abstract syntax tree for representing and transforming PartiQL queries.

  • Parser — turns PartiQL text into the AST, for validation, linting, rewriting, or translation.

  • Logical plan — a representation of a query’s semantics for analysis and optimization.

  • Planner — converts the AST into a resolved, typed logical plan using your catalogs and functions.

  • Compiler and evaluator — compiles a plan into an executable and runs it.

Use all of it to build a new database or query engine, or use just one piece — say, the parser to power an editor or a query-validation service, or the planner to analyze queries before handing them to an existing engine. You plug in your own data through catalogs, tables, and user-defined functions, and PartiQL takes care of the language.

This is how PartiQL is used in practice: different databases and services at Amazon adopt different libraries to fit their needs, from just the parser up to the full query stack.

See Maven Packages for the JVM libraries, or the partiql-lang-rust repository for the equivalent Rust crates.

The JVM and Rust

PartiQL has two open-source implementations, one for each of the ecosystems where large systems are built.

The JVM

Java has been the language of large-scale systems for decades. For JVM users, partiql-lang-kotlin provides PartiQL’s libraries, written in Kotlin and usable from Java, Kotlin, or any JVM language. Its 1.0 release introduced a plan-based evaluator, an extendable compiler, a comprehensive error-handling framework, and a stable, forward-compatible API. Start with the partiql-lang-kotlin documentation.

On the JVM you also get Scribe, a PartiQL transpiler built on partiql-lang-kotlin. Scribe semantically compiles your query — parsing it and resolving and type-checking it against your catalog into a logical plan — and then outputs that plan in the SQL dialect of your choice, such as Spark SQL, Trino, or Redshift. Because it translates the resolved, typed plan rather than rewriting query text, the output is based on what the query means, not just how it is spelled. Scribe is experimental and under active development.

Rust

Rust has become the de facto language for building fast, efficient large-scale systems today. partiql-lang-rust has made great strides in both performance and correctness. The current release, v0.15.0, provides the parser, AST, logical planner, and evaluator as separate crates, and is used by services today. See Getting Started for links to the crates and API documentation.

Rust (Upcoming)

The next generation of the Rust implementation is a new, register-based bytecode virtual machine (PartiQLVM), designed for zero-copy data access, control over memory layout, reusable thread-safe compiled programs, and projection pushdown. It is still a work in progress.

An early preview shipped in the v0.15.0-alpha.1 pre-release; it is not part of the v0.15.0 release described above. In the words of the release notes, "Preliminary performance benchmarks (via Criterion) have indicated significant latency improvements (~20x faster for in-memory inputs, compared to partiql-lang-rust 0.14.0), coupled with CPU and memory savings." That preview supports only a small subset of the language (SELECT, SELECT VALUE, FROM, WHERE, joins, and some scalar expressions).

For teams that have already integrated PartiQL on the JVM, the same pre-release published partiql-jni, a zero-dependency Java library that wraps the Rust virtual machine. The release notes report that its preliminary Java benchmarks (via JMH, with cacheable physical plans) showed a 2-3x latency improvement compared to partiql-lang-kotlin.

Example use cases

Analyze logs without defining a schema

Application logs are usually streams of JSON records whose fields vary from one line to the next. PartiQL queries them as they are — no schema, no flattening, no ETL step. Nested fields are a path away, and in permissive mode (the default) records that lack a field simply don’t match.

SELECT path, COUNT(*) AS errors
FROM app_logs AS l
WHERE l.severity = 'ERROR' AND l.response.status >= 500
GROUP BY l.request.path AS path

Join logs with your database

Because PartiQL treats schemaless documents and typed tables as values in the same data model, one query can join raw log records with the structured tables in a real database. For example, count failed requests by the tier of the customer that made them:

SELECT tier, COUNT(*) AS failed_requests
FROM app_logs AS l
  JOIN customers AS c ON l.request.customer_id = c.id
WHERE l.response.status >= 500
GROUP BY c.tier AS tier

When customers comes from a catalog with a table definition, references to it are type-checked, while app_logs stays schemaless (in permissive mode).

A query front end for your own compute

If you already run your own execution engine or managed compute service, you don’t need PartiQL’s evaluator. Use just the parser and AST as your front end: accept PartiQL from your users, validate it, analyze or rewrite the AST, and translate it into whatever your engine executes. Add the planner when you want name resolution and type checking against your catalog too.

Write once, run elsewhere

With Scribe, your queries are written in one language, PartiQL, but executed in another. Scribe semantically compiles each query and outputs it in the dialect of the engine you choose, so you can run the same query on a distributed engine for scale, on a warehouse for speed, or move between supported engines with fewer rewrites.

Next steps