Untitled :: PartiQL

v0.14 to v1 Types Migration

If there are inaccuracies discovered with this documentation, please submit a GitHub issue.

This guide is intended to cover the differences between the type interfaces used in partiql-lang-kotlin 0.14 and partiql-lang-kotlin v1.

Background on v0.14’s `StaticType`

StaticType was introduced back in 2020 in partiql-lang-kotlin v0.1 to cover the typing needs of the evaluator. Since that time, the PartiQL team has gradually evolved the API to be used more broadly in other parts of the code including the plan, planner, and catalog interfaces. By the time v0.14.9 was released, the PartiQL team has found a number of issues with the modeling of StaticType.

1. Modeling was not backwards compatible.

StaticType extensively used Kotlin’s data classes which provide many convenience methods such as copy and destructuring declarations. However, the convenience of data classes makes it hard to add new fields without breaking public APIs. The guidance from the Kotlin documentation is to not use data classes in the public API.

A similar issue applies to sealed classes/interfaces and enums which we cannot easily add new variants without breaking existing customers.

2. Burdensome modeling of nullable and missable types

Due to how we modeled types and union of types, all types in 0.14 are non-null and non-missing by default. In many typing scenarios, we often deal with nullable and missable types. A nullable type was represented in 0.14 with a unionOf(<type>, null). This modeling was complex and prone to error as users would have to remember to omit a null or missing type from a union before working with the type.

3. Coupled type modeling to Java/Kotlin

StaticType 's modeling as a set of classes and interfaces was tightly coupled to Java/Kotlin’s class semantics. This made certain operations such as extracting constraints complicated and involved a lot of additional casts to users' code.

4. Contradictory constraints for `StructType`

StructType, the representation of a struct in StaticType, had a messy modeling with potentially contradictory constraints and unused fields. StructType defined a contentClosed field as well as a separate TupleConstraint.Open constraint which represented the same thing. This required users of StaticType working with structs, to ensure both the field and the constraint are properly set and do not conflict. There was also a primaryKeyFields that was not used in the library, adding further overhead to the public API.

5. Type unions requiring `.flatten()` calls

0.14 allows users to define unions of arbitrary types using the AnyOfType. However, this required users of StaticType to constantly ensure the types are properly normalized using the .flatten() method.

Background on `PType`

PType was introduced in PLK v1 to address the prior pain points of StaticType. It provides a "fat" interface to allow for easier access to relevant methods and to decouple its reliance on Java’s type semantics. PType was also set up in such a way to allow for better backwards compatibility as we add new types and constraints in the future.

To use the PType, we’ve provided more examples within the partiql-lang-kotlin library’s migration guide

v0.14 examples
v1 examples

Included in the code examples are how to

Collecting All Types from Rex Expressions

In v0.14, you could get all types under a Rex by calling StaticType.flatten() on the Rex’s StaticType. PLK v1 doesn’t have similar support like flatten() and uses visitors for traversal instead, without the allTypes property.

To achieve equivalent functionality, users need to implement the logic themselves using the visitor pattern. Here’s a utility class that provides the same functionality for reference:

/**
 * import org.partiql.plan.Operator
 * import org.partiql.plan.OperatorVisitor
 * import org.partiql.plan.rex.*
 * import org.partiql.spi.types.PType
 */

/**
 * Utility for collecting all PTypes from Rex expressions and PType hierarchies.
 * Provides functionality equivalent to PLK v0.14's StaticType.flatten() and StaticType.allTypes.
 * Warning: This code snippet is only for reference and guaranteed to be compatible with PLK v1.2.2.
 */
public object TypeCollector {

    /**
     * Collects all PTypes from a Rex expression tree, equivalent to old PLKv0.14's StaticType.flatten()
     */
    public fun getAllTypes(rex: Rex): Set<PType> {
        val collector = RexTypeCollector()
        rex.accept(collector, Unit)
        return collector.types
    }

    /**
     * Flattens a PType to get all nested types, equivalent to old PLK v0.14's StaticType.allTypes
     */
    public fun flattenPType(pType: PType): Set<PType> {
        val types = mutableSetOf<PType>()
        collectPTypeRecursively(pType, types)
        return types
    }

    internal fun collectPTypeRecursively(pType: PType, collector: MutableSet<PType>) {
        collector.add(pType)

        when (pType.code()) {
            PType.ARRAY, PType.BAG -> {
                collectPTypeRecursively(pType.typeParameter, collector)
            }
            PType.ROW -> {
                pType.fields.forEach { field ->
                    collectPTypeRecursively(field.type, collector)
                }
            }
            // STRUCT is open, no nested types to traverse
        }
    }
}

internal class RexTypeCollector : OperatorVisitor<Unit, Unit> {
    internal val types = mutableSetOf<PType>()

    override fun visitStruct(rex: RexStruct, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.fields.forEach { field ->
            field.key.accept(this, ctx)
            field.value.accept(this, ctx)
        }
    }

    override fun visitArray(rex: RexArray, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.values.forEach { it.accept(this, ctx) }
    }

    override fun visitBag(rex: RexBag, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.values.forEach { it.accept(this, ctx) }
    }

    override fun visitCall(rex: RexCall, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.args.forEach { it.accept(this, ctx) }
    }

    override fun visitCase(rex: RexCase, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.match?.accept(this, ctx)
        rex.branches.forEach { branch ->
            branch.condition.accept(this, ctx)
            branch.result.accept(this, ctx)
        }
        rex.default?.accept(this, ctx)
    }

    override fun visitCast(rex: RexCast, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.operand.accept(this, ctx)
    }

    override fun visitCoalesce(rex: RexCoalesce, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.args.forEach { it.accept(this, ctx) }
    }

    override fun visitLit(rex: RexLit, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
    }

    override fun visitNullIf(rex: RexNullIf, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.v1.accept(this, ctx)
        rex.v2.accept(this, ctx)
    }

    override fun visitPathIndex(rex: RexPathIndex, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.operand.accept(this, ctx)
        rex.index.accept(this, ctx)
    }

    override fun visitPathKey(rex: RexPathKey, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.operand.accept(this, ctx)
        rex.key.accept(this, ctx)
    }

    override fun visitPathSymbol(rex: RexPathSymbol, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.operand.accept(this, ctx)
    }

    override fun visitPivot(rex: RexPivot, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
        rex.key.accept(this, ctx)
        rex.value.accept(this, ctx)
    }

    override fun visitSelect(rex: RexSelect, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
        rex.constructor.accept(this, ctx)
    }

    override fun visitSpread(rex: RexSpread, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.args.forEach { it.accept(this, ctx) }
    }

    override fun visitSubquery(rex: RexSubquery, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
    }

    override fun visitSubqueryComp(rex: RexSubqueryComp, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
    }

    override fun visitSubqueryIn(rex: RexSubqueryIn, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
        rex.args.forEach { it.accept(this, ctx) }
    }

    override fun visitSubqueryTest(rex: RexSubqueryTest, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.input.accept(this, ctx)
    }

    override fun visitTable(rex: RexTable, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
    }

    override fun visitVar(rex: RexVar, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
    }

    override fun visitError(rex: RexError, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
    }

    override fun visitDispatch(rex: RexDispatch, ctx: Unit) {
        types.addAll(TypeCollector.flattenPType(rex.type.pType))
        rex.args.forEach { it.accept(this, ctx) }
    }

    override fun defaultReturn(operator: Operator, ctx: Unit) {
        // Handle any Rex types not explicitly covered
        if (operator is Rex) {
            types.addAll(TypeCollector.flattenPType(operator.type.pType))
        }
    }
}

Usage:

Use TypeCollector.getAllTypes(rex) to get all types from a Rex expression tree (equivalent to PLK v0.14’s StaticType.flatten())
Use TypeCollector.flattenPType(pType) to flatten a PType and get all nested types (equivalent to PLK v0.14’s StaticType.allTypes)

Other Differences

v0.14 StaticType is pulled in from partiql-types. v1 PType is pulled in from partiql-spi. We made this change since the SPI package already heavily relies on PType and our prior usages of partiql-spi also used PType.
v1 defines a dynamic type for better modeling the concept of dynamic typing. This effectively replaces v0.14’s AnyOfType (i.e. union of types). There were limited usages where actually knowing the types in the union was helpful, so that capability has been removed in v1. Since v1 allows for backwards compatibility in its "fat" interface, we can always add back that functionality.
v1 defines a ROW type which is a closed, ordered struct most similar to SQL’s ROW. This was introduced to make it easier to define computation involving such closed, ordered structs.
v1 has removed arbitrary sized decimals, strings, and datetime types since there were limited use cases in the rest of the v1 library and applications. We may add it back if a use-case arises.