Vantage Framework

Vantage is a data entity persistence and abstraction framework for Rust.

Vantage changes the way you think about your data. Instead of writing queries and loading rows, you work with sets of records — “unpaid invoices older than 30 days”, “customers who ordered today” — narrowing them, combining them, traversing from one to another, and acting on them where they live.

Vantage offers two ways to work with your data. In transactional mode there is no state to manage: load a set, act on it, done — each operation goes straight to the backend. This is the natural shape of a REST endpoint: handle the request, read or update the data, respond. (The name describes that request/response shape, not database transactions — BEGIN/COMMIT is the backend’s business, and Vantage doesn’t wrap your operations in one.) In live mode Vantage maintains a local representation of the data segment you’re working with: you operate on your local copy, and it reconciles with the source over time. That’s how a user interface behaves — any kind. Live mode is built on top of transactional mode, so everything you learn about the first carries into the second.

This documentation tracks the current 0.6 release line.

Ethos

A few principles run through every layer of the framework:

• Let the backend do the work. If a database or API can filter, join, aggregate, or paginate, Vantage pushes the work there — data is narrowed at the source, not in your process.
• Fill the gaps client-side, honestly. Where a backend can’t (a CSV file can’t sort; two databases can’t join), the layer above provides it — and every handle advertises exactly what it supports, so a missing capability is an explicit error, never a silent guess.
• Respect native types. Each backend keeps its own type system end to end — decimals stay precise, dates stay dates, nothing is funnelled through JSON.
• Business logic lives on the entity. Validation, audit, soft-delete, and domain methods attach to your model once and apply everywhere the entity is used.
• Fail loudly, retry safely. No panics, no silent zeros or match-alls; operations are idempotent wherever possible, so retrying is always an option.
• Be aware of observers. Data knows who is watching: changes stream to subscribers, and edits reconcile instead of clobbering. (This is live mode — the second half of the guide builds it up.)

Three ways to work with data

Vantage lets you choose how strongly typed your data access is — per component, not per project:

1. Entity mode — records are plain Rust structs (Table<SqliteDB, Product>). Columns and conditions are compiler-checked, and business logic attaches to the entity: full type safety, the natural choice for hand-written Rust code.
2. Record mode — each record is an arbitrary structure of named values (Record<V>), with schema introspectable at runtime. No entity struct required — built for generic code: data grids, admin tools, import/export.
3. Rhai scripts — more generic still: schema declared in YAML, custom expressions and logic in Rhai. Defined, loaded, and sealed at runtime with no recompiling — made for configuration-driven and agent-driven tooling.

The three interoperate: a Rhai-declared table serves records, records deserialize into entities, and all of it sits behind the same capability-checked handles.

What makes Vantage different

Vantage is a framework, not a library. A library solves one concern and hands the rest back to you; Vantage takes over the data layer entirely — modelling, querying, types, caching, reactivity — and gives you well-defined places to plug in what’s yours. The implementation spans 10+ crates (query builders, backend drivers, the entity layer, the reactive stack), all built on the same cohesive, extensible principles: what you learn in one crate applies in the next, and every extension point looks the same wherever you meet it.

Vantage also doesn’t mimic frameworks from other languages. No reflection, no runtime magic, no inheritance hierarchies. It leans on what Rust is uniquely good at — traits for composition, ownership for safe sharing, generics that compile away, async throughout — so the framework feels native rather than translated.

The four layers

Everything above maps onto four layers. Each builds on the one below, and you climb only as far as your application needs:

Table<SqliteDB, Product>   entity mode — typed, compiler-checked, transactional
Vista                      record mode — schema known at runtime, capabilities honest
Dio                        live mode begins — local cache, write queue, change events
Scenery                    reactive views over a Dio — what a UI binds to and watches

A Table is where your model and business logic live. Wrap it into a Vista when generic code needs to consume it (record mode, above). Bind the Vista to a Dio when you want a live local representation — caching, write routing, reconciliation. Open a Scenery over the Dio for an ordered table, a single record, or an aggregate that updates as the data changes. Layers never leak upward: a Table doesn’t know it’s being cached, and a Scenery consumer can’t tell which backend is underneath.

Vantage and Vantage UI

Vantage UI is a native admin console built directly on these crates — point it at your databases, APIs, and tools, and an AI agent configures tables, forms, and dashboards over them. It is a closed-source product (free download), and this framework is its open foundation: Vantage is open-sourced so that you can build your own services, CLIs, and UIs on the same data layer — and extend it, adding persistences and capabilities that custom builds can carry further than the stock app does.

If you want the finished tool, start with Vantage UI. If you want the foundation, read on.

Getting Started

Vantage covers a lot of ground — multiple databases, type systems, entity frameworks, UI adapters — but none of that matters until you’ve seen it do something useful.

This guide introduces Vantage concepts one at a time, each building on the last. We start with something you already know — SQL — and build a shared foundation: a small product catalog that begins as a CLI, becomes an HTTP API, and switches databases without touching its handlers, then erases into a runtime-generic data handle.

From there the guide forks by the shape of your problem. The reactive stack that sits on top — caching, change events, watchable views — is the same whichever backend is underneath, so you follow it down whichever path is yours:

• A facade over an API you don’t control — a slow, read-only cloud API that can’t sort, search, or paginate. Diorama caches it and fills the gaps.
• A live view in front of your own database — a relational database you own and write to. Diorama gives it a cached, watchable facade, and the path ends by moving the same app from SQLite to PostgreSQL with a single switch.

Same abstractions, opposite backends — that contrast is the point.

The foundation

1. SQLite and the Query Builder — connect to a database, build and execute typed queries, map rows to structs.
2. Tables and Typed Data Access — define entities and tables, narrow sets with conditions, traverse relationships, add computed fields; CRUD becomes one-liners.
3. A Standalone Axum Server — put the model behind HTTP with one generic CRUD handler for every entity, then migrate the whole server from SQLite to MongoDB by editing only the model.
4. Vista — the Universal Data Handle — erase the entity and backend into a schema-bearing runtime handle, with explicit capability contracts.

Choose your path

With a Vista in hand, pick the path that matches your data — a facade over an API you don’t control, or a live view over your own relational database. Both build the same caching, reactive, watch-streaming stack; they differ only in the backend beneath it.

You’ll need basic Rust experience (structs, traits, async/await, cargo). No prior Vantage knowledge required.

Start here: SQLite and the Query Builder

Beyond the guide

The rest of the book is reference material — read it when the guide points at it, or jump straight to what you need:

• Expressions & Queries and the per-backend chapters (SQL, SurrealDB) — the query-building layer in depth.
• Records: Traversal, Invariants & Hooks — the write pipeline: audit stamps, validation, soft-delete.
• Model-Driven Architecture — how Vantage expects you to structure business software.
• Config-Driven Vistas: YAML & Rhai — define and reshape data handles from configuration, sealed at runtime without recompiling.
• Adding a New Persistence — nine incremental steps to connect Vantage to a backend it doesn’t know yet; the main path for extending the framework.
• Augmentation and the Type System — enriching rows from a second source, and teaching backends your own types.
• Historical Timeline — how the framework got here, release by release.

SQLite and the Query Builder

Vantage is a big framework. It covers SQL databases, SurrealDB, MongoDB, CSV files, REST APIs — and ties them all together with a shared type system, expression engine, and data abstraction layer.

We’ll get to all of that. But right now, let’s start with something familiar: building SQL queries.

For this chapter we’ll use SQLite. It’s lightweight, needs no server, and works with a plain file on disk.

Set up

Create a new project:

cargo init learn-1 && cd learn-1
cargo add vantage-sql --features sqlite
cargo add vantage-expressions
cargo add tokio --features full

Three dependencies — vantage-sql gives us the SQLite query builder and connection pool, vantage-expressions is needed by the sqlite_expr! macro, and tokio provides the async runtime because all database operations are async.

Create and populate a database

We’ll make a small product catalog from scratch. Create seed.sql in your project root:

CREATE TABLE product (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    price INTEGER NOT NULL,
    category_id INTEGER,
    is_deleted BOOLEAN NOT NULL DEFAULT 0
);

INSERT INTO product VALUES (1, 'Cupcake',           120, 1, 0);
INSERT INTO product VALUES (2, 'Doughnut',          135, 1, 0);
INSERT INTO product VALUES (3, 'Tart',              220, 2, 0);
INSERT INTO product VALUES (4, 'Pie',               299, 2, 0);
INSERT INTO product VALUES (5, 'Cookies',           199, 1, 0);
INSERT INTO product VALUES (6, 'A Stale Cake',       80, 1, 1);
INSERT INTO product VALUES (7, 'Sourdough Loaf',    350, 3, 0);

CREATE TABLE category (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL
);

INSERT INTO category VALUES (1, 'Sweet Treats');
INSERT INTO category VALUES (2, 'Pastries');
INSERT INTO category VALUES (3, 'Breads');

Run it:

sqlite3 products.db < seed.sql

You now have products.db — 7 products (6 active, 1 deleted) across 3 categories. Quick check:

sqlite3 products.db "SELECT name, price FROM product WHERE is_deleted = 0"

Start with an async main

All database operations in Vantage are async, so we need a Tokio runtime. Replace src/main.rs with:

use vantage_sql::prelude::*;

#[tokio::main]
async fn main() {
    if let Err(e) = run().await {
        e.report();
    }
}

async fn run() -> VantageResult<()> {
    println!("Ready!");
    Ok(())
}

A few things going on here:

• use vantage_sql::prelude::* brings in everything we need for this chapter — SqliteDB, SqliteSelect, the sqlite_expr! macro, error types, and the traits that make builder and execution methods work.
• VantageResult<()> is Vantage’s own Result type. It uses VantageError, which tracks context and error chains for readable diagnostics.
• e.report() prints the error in a structured format. We call it from main() because Rust’s default Result-returning main uses Debug formatting, which is ugly. This pattern gives us clean error output instead.

Run cargo run to make sure it compiles.

Connect to SQLite

Add this inside run():

#![allow(unused)]
fn main() {
let db = SqliteDB::connect("sqlite:products.db?mode=ro")
    .await
    .context("Failed to connect to products.db")?;
}

SqliteDB wraps an sqlx connection pool. The connection string is an sqlx URL — ?mode=ro opens read-only, which is all we need for now.

#![allow(unused)]
fn main() {
let db = SqliteDB::new(existing_pool);
}

Error: Failed to connect to products.db
│
╰─▶ error returned from database: (code: 14) unable to open database file

Build a SELECT

SqliteSelect is the query builder for SQLite. Other persistences have their own — PostgresSelect, MongoSelect — and they all implement the Selectable trait, so the interface is identical apart from vendor-specific extensions. None of them need a database connection — they’re just structs that accumulate query parts. You build them with a chain of .with_*() calls:

#![allow(unused)]
fn main() {
let select = SqliteSelect::new()
    .with_source("product")
    .with_field("name")
    .with_field("price");

println!("{}", select.preview());
// SELECT "name", "price" FROM "product"
}

.preview() renders the final SQL as a string — handy for debugging, but never used for execution.

#![allow(unused)]
fn main() {
// Builder style
let select = SqliteSelect::new()
    .with_source("product")
    .with_field("name");

// Mutable style
let mut select = SqliteSelect::new();
select.add_source("product", None);
select.add_field("name");
}

Execute it

#![allow(unused)]
fn main() {
let result = db.execute(&select.expr()).await?;
println!("{:?}", result);
}

Two steps here: .expr() turns the builder into an Expression — Vantage’s internal representation that keeps parameters separate from the SQL template. Then db.execute() sends it to the database.

The result is AnySqliteType — a type-tagged wrapper around whatever came back. The Debug output isn’t pretty, but you should see all 6 product rows in there.

Adding conditions

Our database has a soft-delete flag — “A Stale Cake” has is_deleted = 1. Let’s filter it out:

#![allow(unused)]
fn main() {
let condition = sqlite_expr!("\"is_deleted\" = {}", false);

let select = SqliteSelect::new()
    .with_source("product")
    .with_field("name")
    .with_field("price")
    .with_condition(condition);

println!("{}", select.preview());
// SELECT "name", "price" FROM "product" WHERE "is_deleted" = 0
}

sqlite_expr! creates an Expression — a SQL template with typed, bound parameters. That {} is not Rust’s format!: the value false is stored separately and bound through sqlx’s parameterized query interface at execution time — no injection risk, ever. The preview shows it inline for readability.

In SQL persistences, a condition is just an expression. When you pass it to .with_condition(), it’s nested inside the select’s own expression tree. At execution time the whole tree is flattened into a single template + parameter list that the database driver can bind safely.

Run it — you should see 5 rows, with “A Stale Cake” filtered out.

Typed columns and operators

Writing \"is_deleted\" in a raw expression works, but there’s a cleaner way. Column<T> creates a typed column reference, then chain an SqliteOperation like .eq() to build the condition. (Column lives in the vantage-table crate, but the vantage_sql prelude re-exports it — no new dependency needed.)

#![allow(unused)]
fn main() {
let is_deleted = Column::<bool>::new("is_deleted");
let condition = is_deleted.eq(false);
}

Same result, but the type parameter <bool> ensures you can only compare against matching types. Try is_deleted.eq(42) — it won’t compile. Other operators — .gt(), .lt(), .ne(), .in_() — enforce the same type safety.

The result is a SqliteCondition — the backend’s native condition type — ready to be passed directly to .with_condition().

Multiple conditions combine with AND:

#![allow(unused)]
fn main() {
let is_deleted = Column::<bool>::new("is_deleted");
let price = Column::<i64>::new("price");

let select = SqliteSelect::new()
    .with_source("product")
    .with_field("name")
    .with_condition(is_deleted.eq(false))
    .with_condition(price.gt(150));
// ... WHERE "is_deleted" = 0 AND "price" > 150
}

#![allow(unused)]
fn main() {
use vantage_sql::sqlite::sqlite_ident as ident;

let condition = ident("is_deleted").eq(false);
}

Primitives, query builders, Column, conditions — even native values like i64 and bool — all implement Expressive<T>, where T is your database’s any-type (AnySqliteType for SQLite, AnyMongoType for MongoDB). That shared trait is what makes any of them usable as a parameter inside any expression.

Working with Any-types

Record<V> is an ordered map (IndexMap<String, V>) — one record per row, with column names as keys. It lives in the vantage-types crate, so add that dependency and import its prelude:

cargo add vantage-types --features serde

#![allow(unused)]
fn main() {
use vantage_types::prelude::*;
}

So far we’ve been printing raw AnySqliteType with Debug. That works for verifying queries, but it’s useless for real work. When db.execute() returns a multi-row result, the AnySqliteType holds an array of maps internally. Convert it to Vec<Record<AnySqliteType>> to work with individual rows:

#![allow(unused)]
fn main() {
let raw = db.execute(&select.expr()).await?;

let records = Vec::<Record<AnySqliteType>>::try_from(raw)
    .context("Failed to convert to records")?;

for rec in &records {
    let name: String = rec["name"].try_get::<String>().unwrap();
    let price: i64 = rec["price"].try_get::<i64>().unwrap();
    println!("{} — {} cents", name, price);
}
// Cupcake — 120 cents
// Doughnut — 135 cents
// ...
}

Access fields by column name with rec["name"]. Each value is still an AnySqliteType, so you call .try_get::<T>() to extract a typed Rust value. If the type doesn’t match (say you call .try_get::<i64>() on a text column), you get None — no panics, no garbage.

#![allow(unused)]
fn main() {
let json_rec: Record<serde_json::Value> = rec.into_record();
}

Under the hood, each persistence has its own type system for storing values. SQLite uses CBOR — a compact binary format that preserves types like Decimal, NaiveDate, and NaiveDateTime through tagged values. MongoDB uses BSON natively. The AnySqliteType / AnyMongoType wrappers hide these details — you interact with .try_get::<T>() regardless of which persistence you’re using. If you ever need to inspect the raw representation, .value() gives you the underlying CBOR value:

#![allow(unused)]
fn main() {
let price_cbor = rec["price"].value();
println!("{:?}", price_cbor);
// Integer(Integer(120))
}

See Persistence-aligned Type System for the full picture.

Mapping rows to structs

Calling .try_get::<T>() on every field gets tedious. The #[entity] macro generates TryFromRecord<AnySqliteType> for your struct, so conversion happens in one call with no type information lost:

#![allow(unused)]
fn main() {
#[entity(SqliteType)]
struct Product {
    name: String,
    price: i64,
}

let raw = db.execute(&select.expr()).await?;
let records = Vec::<Record<AnySqliteType>>::try_from(raw)?;

for rec in records {
    let product = Product::from_record(rec)?;
    println!("{} — {} cents", product.name, product.price);
}
}

The macro needs vantage-core as a direct dependency (it’s already a transitive dep through vantage-sql, but the generated code references it in your crate):

cargo add vantage-core

The macro also supports multiple type systems in one attribute — #[entity(SqliteType, PostgresType, MongoType)] generates a separate TryFromRecord impl for each persistence. One struct, all backends.

Putting it together

Here’s the complete src/main.rs — connect, query, convert to entities, print:

use vantage_sql::prelude::*;
use vantage_types::prelude::*;

#[entity(SqliteType)]
struct Product {
    name: String,
    price: i64,
}

#[tokio::main]
async fn main() {
    if let Err(e) = run().await {
        e.report();
    }
}

async fn run() -> VantageResult<()> {
    let db = SqliteDB::connect("sqlite:products.db?mode=ro")
        .await
        .context("Failed to connect to products.db")?;

    let select = SqliteSelect::new()
        .with_source("product")
        .with_field("name")
        .with_field("price")
        .with_condition(Column::<bool>::new("is_deleted").eq(false));

    let raw = db.execute(&select.expr()).await?;
    let records = Vec::<Record<AnySqliteType>>::try_from(raw)?;

    for rec in records {
        let p = Product::from_record(rec)?;
        println!("{:<12} {:>3} cents", p.name, p.price);
    }

    Ok(())
}

cargo run
# Cupcake      120 cents
# Doughnut     135 cents
# Tart         220 cents
# Pie          299 cents
# Cookies      199 cents

What we covered

Tables and Typed Data Access

A table is a structure representing a collection of records in a database:

#![allow(unused)]
fn main() {
let products = Product::table(db);

for (id, product) in products.list().await? {
    println!("{} — {} cents", product.name, product.price);
}
}

Think of Table<SqliteDB, Product> as a Vec<Product> — except the records aren’t in memory. They live somewhere else (a database, a file, an API), and you don’t know how many there are or what they contain until you list them.

If you’ve used an ORM before, you might expect Vantage to work with individual records — load one, change it, save it back. Vantage works differently: you always operate on a set of records. A set might contain one record, no records, or millions — you don’t pull them into memory to find out.

Operations like counting, filtering, and updating happen on the database side — Vantage builds the right query and sends it over. Even traversal operations and sub-queries are done by building queries and executing remotely — if the database supports it, of course.

When you .clone() a table, you don’t clone the data — you clone the definition. From there you can narrow it down by adding conditions, turning it into a subset. Some examples of sets you might work with:

• all user records except those with a soft-delete flag
• orders placed today
• orders of paid customers
• customers who have an unpaid invoice older than 30 days
• products that sold at least 5 items today

Each of these is a set — defined by conditions, not a list of IDs — and one set can serve as the condition of another (“orders of paid customers”). Acting on a set is a single call: notify those customers, archive those orders, apply that discount. Put together, even complex operations are easy to express in code:

#![allow(unused)]
fn main() {
// "notify customers who have an unpaid invoice older than 30 days"
let mut overdue = Invoice::table(db);
overdue.add_condition(overdue["status"].eq("unpaid"));
overdue.add_condition(overdue["issued_at"].lt(days_ago(30)));

overdue.ref_customer().send_reminder().await;
}

#![allow(unused)]
fn main() {
// "apply a discount to customers who spent more than $500 this month"
let mut customers = Customer::table(db);
customers.add_expression("spent_this_month", |c| {
    c.subquery_orders().only_this_month().field_sum("total")
});
customers.add_expression("discount", |c| {
    primitives::ternary(c["spent_this_month"].gt(500), 10, 0)
});
}

Don’t worry about the exact syntax — we’ll get to all of it. The point is that sets compose naturally: define one, use it to narrow another, then act on the result.

Defining a Table

The syntax for defining a table looks similar to query building from chapter 1, but there are important differences:

A Table is typically defined in its own file alongside the entity. Create src/product.rs:

#![allow(unused)]
fn main() {
// src/product.rs
use vantage_sql::prelude::*;
use vantage_types::prelude::*;

#[entity(SqliteType)]
#[derive(Clone, Default)]
pub struct Product {
    pub name: String,
    pub price: i64,
}

impl Product {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Product> {
        let is_deleted = Column::<bool>::new("is_deleted");
        Table::new("product", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<i64>("price")
            .with_column_of::<bool>("is_deleted")
            .with_condition(is_deleted.eq(false))
    }
}
}

#![allow(unused)]
fn main() {
impl Product {
    fn table() -> Table<SqliteDB, Product> {
        Table::new("product", get_sqlite_db())
            // ...columns...
    }
}
}

This defines the product table once. You can now generate queries from it:

#![allow(unused)]
fn main() {
let table = Product::table(db);

// Full SELECT with all columns and conditions applied
let select = table.select();
println!("{}", select.preview());
// SELECT "id", "name", "price", "is_deleted" FROM "product"

// Count query (does not execute — just builds the expression)
let count_query = table.get_count_query();
// SELECT COUNT(*) FROM "product"

// Sum query for a specific column
let sum_query = table.get_sum_query(&table["price"]);
// SELECT SUM("price") FROM "product"
}

These return the same SqliteSelect and Expression types from chapter 1 — the table just assembles them for you. But most of the time you won’t need the raw query at all.

CRUD operations

With the table defined, all four operations — create, read, update, delete — are one-liners. Each comes from a trait in the prelude:

#![allow(unused)]
fn main() {
let table = Product::table(db);

// Read — list all, get one by ID
let all = table.list().await?;             // IndexMap<String, Product>
let pie = table.get("4").await?;           // Option<Product> — id 4 is Pie

// Create — insert with a known ID
let muffin = Product { name: "Muffin".into(), price: 175 };
table.insert(&"8".to_string(), &muffin).await?;

// Update — replace the entire record
let updated = Product { name: "Blueberry Muffin".into(), price: 195 };
table.replace(&"8".to_string(), &updated).await?;

// Delete
table.delete(&"8".to_string()).await?;
}

That’s it. list() returns an IndexMap<Id, Product> — ordered and keyed by ID. get() returns Option<Product> — None when the ID doesn’t exist, so you can pattern-match or .ok_or(...) into your own not-found error. There’s also get_some() which returns an (Id, Product) pair (still Option-wrapped) for sampling, and insert_return_id() for when you want the database to generate the ID.

Listing any table

Table<SqliteDB, Product> is great when you know the types at compile time. Sometimes you want a function that lists any table — products, orders, customers — without naming the entity type.

With generics you keep full type safety, limited to Rust callers:

#![allow(unused)]
fn main() {
async fn list_table<E: Entity + std::fmt::Debug>(
    table: &impl ReadableDataSet<E>,
) -> VantageResult<()> {
    for (id, entity) in table.list().await? {
        println!("  {}: {:?}", id, entity);
    }
    Ok(())
}
}

To go fully type-erased — iterate columns by name, get runtime values, and expose tables outside Rust (a web admin UI, FFI, scripting) — wrap the table in a Vista, the universal schema-bearing data handle. That’s the subject of its own chapter; Vista replaced the older AnyTable carrier in 0.5.

Relationships

Our database has a category table that we haven’t used yet. Let’s define it and connect it to products.

Create src/category.rs:

#![allow(unused)]
fn main() {
// src/category.rs
use vantage_sql::prelude::*;
use vantage_types::prelude::*;

#[entity(SqliteType)]
#[derive(Debug, Clone, Default)]
pub struct Category {
    pub name: String,
}

impl Category {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Category> {
        Table::new("category", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_many("products", "category_id", Product::table)
    }
}
}

The important line is .with_many("products", "category_id", Product::table). This declares:

• “products” — the name of the relationship (you’ll use this to traverse it)
• “category_id” — the foreign key column on the product table
• Product::table — a function that builds the target table (same one we defined earlier)

No changes to Product are needed — the relationship is declared on the side that “owns” it. Category has many products; the foreign key lives on product.

Given a set of categories, get_ref_as traverses the relationship and returns matching products:

#![allow(unused)]
fn main() {
let categories = Category::table(db.clone());
let products = categories.get_ref_as::<Product>("products")?;
}

The result is a Table<SqliteDB, Product> with an extra condition — only products whose category_id matches one of the category IDs. Narrow the categories first and the subquery narrows too. Let’s use this in a CLI that accepts an optional search filter. The .with_search() method adds a LIKE condition across all columns of the table — handy for quick filtering.

Update src/main.rs:

mod category;
mod product;

use category::Category;
use product::Product;
use vantage_sql::prelude::*;

async fn list_products(table: &Table<SqliteDB, Product>) -> VantageResult<()> {
    for (id, p) in table.list().await? {
        println!("  {:<4} {:<20} {:>3} cents", id, p.name, p.price);
    }
    Ok(())
}

#[tokio::main]
async fn main() {
    if let Err(e) = run().await {
        e.report();
    }
}

async fn run() -> VantageResult<()> {
    let db = SqliteDB::connect("sqlite:products.db")
        .await
        .context("Failed to connect to products.db")?;

    let filter = std::env::args().nth(1);

    let products = match &filter {
        Some(search) => Category::table(db.clone())
            .with_search(search)
            .get_ref_as::<Product>("products")?,
        None => Product::table(db),
    };

    list_products(&products).await?;

    Ok(())
}

Try it:

cargo run                # all active products
cargo run "Sweet"        # Cupcake, Doughnut, Cookies
cargo run "Pastries"     # Tart, Pie
cargo run "t"            # matches "Sweet Treats" AND "Pastries" — 5 products

The search "t" matches two categories (“Sweet Treats” and “Pastries”), so products from both are returned.

SELECT ... FROM "product"
WHERE "category_id" IN (SELECT "id" FROM "category" WHERE "name" LIKE '%t%')
AND "is_deleted" = 0

Computed fields with expressions

So far, list_products only shows name and price. It would be nice to show the category name too — but Product doesn’t have a category field, and the category name lives in a different table.

Vantage solves this with expressions — computed fields that are evaluated as part of the SELECT query. You define them on the table, and they appear alongside regular columns.

First, add a with_one relationship on Product (the reverse of with_many on Category):

#![allow(unused)]
fn main() {
.with_one("category", "category_id", Category::table)
}

This says: each product has one category, linked through category_id. Now you can use get_subquery_as inside with_expression to build a correlated subquery:

#![allow(unused)]
fn main() {
.with_expression("category", |t| {
    t.get_subquery_as::<Category>("category")
        .unwrap()
        .select_column("name")
})
}

select_column("name") builds a subquery like SELECT "name" FROM "category" WHERE "id" = "product"."category_id" — a correlated subquery that fetches the category name for each product row.

Here’s the updated src/product.rs:

#![allow(unused)]
fn main() {
use vantage_sql::prelude::*;
use vantage_types::prelude::*;

use crate::category::Category;

#[entity(SqliteType)]
#[derive(Debug, Clone, Default)]
pub struct Product {
    pub name: String,
    pub price: i64,
    pub category: Option<String>,
}

impl Product {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Product> {
        let is_deleted = Column::<bool>::new("is_deleted");
        Table::new("product", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<i64>("price")
            .with_column_of::<bool>("is_deleted")
            .with_condition(is_deleted.eq(false))
            .with_one("category", "category_id", Category::table)
            .with_expression("category", |t| {
                t.get_subquery_as::<Category>("category")
                    .unwrap()
                    .select_column("name")
            })
    }
}
}

A few things to note:

• category: Option<String> — the field is Option because a product might not have a category (NULL in the database). The expression result is deserialized into this field automatically.
• No with_column_of for “category” — the expression replaces what would normally be a column. Vantage adds it to the SELECT as a computed field.
• with_one + with_expression — the relationship defines how to traverse; the expression defines what to fetch. They work together but serve different purposes.

Update list_products to show the category:

#![allow(unused)]
fn main() {
async fn list_products(table: &Table<SqliteDB, Product>) -> VantageResult<()> {
    for (id, p) in table.list().await? {
        let cat = p.category.as_deref().unwrap_or("-");
        println!("  {:<4} {:<20} {:>3} cents  [{}]", id, p.name, p.price, cat);
    }
    Ok(())
}
}

Run it:

cargo run

  1    Cupcake              120 cents  [Sweet Treats]
  2    Doughnut             135 cents  [Sweet Treats]
  3    Tart                 220 cents  [Pastries]
  4    Pie                  299 cents  [Pastries]
  5    Cookies              199 cents  [Sweet Treats]
  7    Sourdough Loaf       350 cents  [Breads]

The category name comes from a subquery — no JOIN, no extra round trip. And the category filter still works:

cargo run "t"

  1    Cupcake              120 cents  [Sweet Treats]
  2    Doughnut             135 cents  [Sweet Treats]
  3    Tart                 220 cents  [Pastries]
  4    Pie                  299 cents  [Pastries]
  5    Cookies              199 cents  [Sweet Treats]

#![allow(unused)]
fn main() {
// In Category::table()
.with_expression("product_count", |t| {
    t.get_subquery_as::<Product>("products")
        .unwrap()
        .get_count_query()
})
.with_expression("title", |t| {
    let name = t.get_column_expr("name").unwrap();
    let count = t.get_column_expr("product_count").unwrap();
    concat_!(name, " (", count, ")").expr()
})
}

#![allow(unused)]
fn main() {
.with_expression("category", |t| {
    t.get_subquery_as::<Category>("category")
        .unwrap()
        .select_column("title")
})
}

  1    Cupcake              120 cents  [Sweet Treats (3)]
  2    Doughnut             135 cents  [Sweet Treats (3)]
  3    Tart                 220 cents  [Pastries (2)]
  4    Pie                  299 cents  [Pastries (2)]
  5    Cookies              199 cents  [Sweet Treats (3)]
  7    Sourdough Loaf       350 cents  [Breads (1)]

Extension traits

Throughout this chapter you’ve seen calls like categories.get_ref_as::<Product>("products"). The turbofish ::<Product> and the string "products" are repetitive and error-prone — get the string wrong and you get a runtime error.

Rust’s extension traits solve this. Define a trait on Table<SqliteDB, Category> that wraps the traversal:

#![allow(unused)]
fn main() {
pub trait CategoryTable {
    fn ref_products(&self) -> Table<SqliteDB, Product>;
}

impl CategoryTable for Table<SqliteDB, Category> {
    fn ref_products(&self) -> Table<SqliteDB, Product> {
        self.get_ref_as("products").unwrap()
    }
}
}

The trait declares the vocabulary; the impl block supplies the bodies. Nothing is auto-implemented — ref_products() appears on Table<SqliteDB, Category> the moment you write this impl, and on nothing else. The bodies must live impl-side, too: methods like get_ref_as are inherent to Table, so they’re only reachable where self is concretely that table — inside the impl, not inside a trait default.

The unwrap() is safe because we know “products” is defined on every CategoryTable. If the string were wrong, it would panic immediately during development, not silently fail at runtime.

Now callers write categories.ref_products() — no turbofish, no string, no ?, and the compiler catches typos.

The same pattern works for typed column access:

#![allow(unused)]
fn main() {
pub trait ProductTable {
    fn price(&self) -> Column<i64>;
}

impl ProductTable for Table<SqliteDB, Product> {
    fn price(&self) -> Column<i64> {
        self.get_column("price").unwrap()
    }
}
}

With both traits in place, code reads naturally:

#![allow(unused)]
fn main() {
let products = categories.ref_products();
let expensive = products.with_condition(products.price().gt(200));
}

#![allow(unused)]
fn main() {
pub trait ProductTable {
    fn price(&self) -> Column<i64>;
    async fn print(&self) -> VantageResult<()>;
}

impl ProductTable for Table<SqliteDB, Product> {
    fn price(&self) -> Column<i64> {
        self.get_column("price").unwrap()
    }

    async fn print(&self) -> VantageResult<()> {
        vantage_cli_util::print_table(self).await
    }
}
}

+----+----------------+-------+-------------------+
| id | name           | price | category          |
+----+----------------+-------+-------------------+
| 1  | Cupcake        | 120   | Sweet Treats (3)  |
| 2  | Doughnut       | 135   | Sweet Treats (3)  |
| 3  | Tart           | 220   | Pastries (2)      |
| ...                                              |
+----+----------------+-------+-------------------+

Persistence Abstraction

The files you have written so far — Cargo.toml, product.rs, category.rs, main.rs — are the makings of an extensible business software architecture: scalable, maintainable, testable, and portable across databases without rewriting a single line of business logic.

What you have is four distinct components, each with a clear job:

flowchart TD
    A["<b>Business code</b><br/><code>main.rs</code><br/>minimal, domain-focused"]
    B["<b>Model definitions</b><br/><code>product.rs</code>, <code>category.rs</code><br/>entities, tables, relationships, domain methods"]
    C["<b>Persistence crate</b><br/><code>vantage-sql</code> (or surrealdb, mongodb, csv…)<br/>query builder, CRUD, type system"]
    D["<b>Vantage framework</b><br/><code>vantage-table</code>, <code>vantage-expressions</code>, <code>vantage-types</code><br/>the extensible mechanism that ties it together"]
    A --> B
    B --> C
    C --> D

The majority of your business code can work with data without any knowledge of where it’s stored or how. It operates on typed entities, relationships, and domain methods — nothing else.

The model definitions focus on describing where and how data is stored, but they don’t micro-manage the process. If storage requirements change — a new backend, a schema migration, a cached layer in front — the model is where you make the tweaks. Business code remains untouched.

This separation gives you the ergonomics scripting-language ORMs are known for — concise, domain-focused code — with Rust’s type safety and performance underneath: the abstraction compiles down to direct calls. When storage requirements shift, the change stays contained in the model layer instead of rippling through business code, and the whole architecture travels as a single static binary — server, desktop, or embedded.

A Standalone Axum Server

Chapter 2 ended with a CLI that prints products to stdout. In this chapter we put the same model behind an HTTP API:

curl "http://localhost:3001/categories"

[{ "name": "Sweet Treats" }, { "name": "Pastries" }, { "name": "Breads" }]

# Products in a specific category
curl "http://localhost:3001/categories/1/products"

# Writes
curl -X POST "http://localhost:3001/categories" \
  -H 'content-type: application/json' -d '{"name":"Gluten-Free"}'

A few things about the shape of this API:

• /categories/{id}/products is a nested route — products narrowed by the relationship we defined in chapter 2. The handler for it is the same generic list as /categories, just applied to a scoped table.
• /products and /categories start out on SQLite, same as chapter 2. Toward the end of the chapter we migrate them to MongoDB. That migration is a change to the model file; the handlers and routes don’t know the difference.

The handler functions are each written once and mounted against any Table<Backend, Entity> the router has on hand — one list, one get, one post, one patch, one delete, parameterised over the backend and entity types. Adding another entity is a route registration, not a new handler.

The minimum Axum skeleton

Two pieces: a small adaptation to chapter 2’s entities so they round-trip through HTTP, then the server itself. One new dependency:

cargo add axum

Tokio and serde are already pulled in from earlier chapters. axum is the HTTP framework; it uses the existing serde to encode response bodies as JSON.

Entities. Chapter 2’s Product carried a computed category field, and the Expressions compose callout showed Category gaining a computed title. Both were assembled from subqueries — great for display, but they don’t round-trip cleanly through a POST body because the JSON would try to write columns that don’t exist. For a writable API we replace the computed fields with the plain FK column that sits under them. (See with_expression for the chapter-2 computed-field setup we’re walking away from here.)

src/category.rs:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};

#[entity(SqliteType)]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Category {
    pub name: String,
}

impl Category {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Category> {
        Table::new("category", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_many("products", "category_id", Product::table)
    }
}
}

src/product.rs:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};

#[entity(SqliteType)]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Product {
    pub name: String,
    pub price: i64,
    #[serde(skip_serializing_if = "Option::is_none", default)]
    pub category_id: Option<String>,
    #[serde(default)]
    pub is_deleted: bool,
}

impl Product {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Product> {
        let is_deleted = Column::<bool>::new("is_deleted");
        Table::new("product", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<i64>("price")
            .with_column_of::<String>("category_id")
            .with_column_of::<bool>("is_deleted")
            .with_condition(is_deleted.eq(false))
            .with_one("category", "category_id", Category::table)
    }
}
}

Two things to notice:

• category_id is now a real column in both the struct and the table, not a computed expression. Same information as before, but clients can read it from GET and supply it on POST.
• is_deleted picks up #[serde(default)] so POST /products bodies don’t have to carry it; new products default to live.

The soft-delete condition from chapter 2 stays — GET /products will still hide rows with is_deleted = true.

Now the server. Replace src/main.rs with:

mod category;
mod product;

use std::sync::OnceLock;

use axum::{routing::get, Json, Router};
use category::Category;
use vantage_sql::prelude::*;

static DB: OnceLock<SqliteDB> = OnceLock::new();

fn db() -> SqliteDB {
    DB.get().expect("database not initialised").clone()
}

async fn list_categories() -> Json<Vec<Category>> {
    let rows = Category::table(db()).list().await.unwrap();
    Json(rows.into_values().collect())
}

#[tokio::main]
async fn main() -> VantageResult<()> {
    let conn = SqliteDB::connect("sqlite:products.db")
        .await
        .context("Failed to connect to products.db")?;
    DB.set(conn).ok();

    let app = Router::new().route("/categories", get(list_categories));

    let listener = tokio::net::TcpListener::bind("0.0.0.0:3001").await.unwrap();
    axum::serve(listener, app).await.unwrap();

    Ok(())
}

A few notes on what this is doing:

• static DB: OnceLock<SqliteDB>. The handler needs a database handle but we don’t want to open a new connection per request. A program-wide OnceLock is the simplest possible holder — set once in main, read from anywhere.
• Each request calls Category::table(db()) — building the table (columns, relationships) and then lists it.
• The handler returns Json<Vec<Category>> directly. No DTO layer — the entity’s fields become the JSON fields.

Run it:

cargo run

In another terminal:

curl http://localhost:3001/categories

[
  { "name": "Sweet Treats" },
  { "name": "Pastries" },
  { "name": "Breads" }
]

Caching table definitions

We can cache each table definition so it’s built once and handed out by reference on every subsequent call. In src/category.rs, add use std::sync::OnceLock; at the top and wrap the body of Category::table:

#![allow(unused)]
fn main() {
impl Category {
    pub fn table(db: SqliteDB) -> &'static Table<SqliteDB, Category> {
        static CACHE: OnceLock<Table<SqliteDB, Category>> = OnceLock::new();
        CACHE.get_or_init(|| {
            Table::new("category", db)
                // ...columns unchanged...
                .with_many("products", "category_id", |db| Product::table(db).clone())
        })
    }
}
}

Three changes from chapter 2’s version:

• Return type is &'static Table<...> instead of Table<...>. The cache owns the definition; callers get a shared reference. Most table operations (list, get, insert, replace, delete, get_ref_as) take &self, so they work on the reference without any clone.
• static CACHE: OnceLock<...> lives inside the function. A function-local static is still a program-wide single instance — Rust allows this and it keeps the cache private to the table that owns it.
• The with_many callback is a closure: |db| Product::table(db).clone(). The framework still hands us a db and expects an owned Table<SqliteDB, Product> back, so we call the cached accessor and clone the reference. On the first call this triggers Product’s cache to build; every subsequent call just clones a pre-built definition.

src/product.rs follows the same pattern — wrap the body in OnceLock::get_or_init, change the return type to &'static Table<SqliteDB, Product>, and rewrite with_one as |db| Category::table(db).clone().

The handler in main.rs doesn’t change at all — Category::table(db()) used to return an owned Table, and now it returns a &'static Table that auto-derefs through the same .list() call. Restart the server and hit /categories — the response is identical, but now the table definitions are built on the first request and reused forever after.

#![allow(unused)]
fn main() {
let narrowed = Category::table(db())
    .clone()                      // owned copy of the cached definition
    .with_condition(...)           // now we can narrow it
}

Products of a category

The with_many relationship we kept from chapter 2 lets us serve a nested route — /categories/{id}/products. Before writing the handler, give the relationship a proper name with an extension trait on Table<SqliteDB, Category>. Add this to src/category.rs:

#![allow(unused)]
fn main() {
pub trait CategoryTable {
    fn ref_products(&self) -> Table<SqliteDB, Product>;
}

impl CategoryTable for Table<SqliteDB, Category> {
    fn ref_products(&self) -> Table<SqliteDB, Product> {
        self.get_ref_as("products").unwrap()
    }
}
}

Chapter 2 introduced this pattern — a typed, discoverable name instead of the string-and-turbofish call. The .unwrap() is safe here because we’re the ones who registered "products"; a typo surfaces immediately at startup.

Now the handler. Take the cached category table, narrow it to a single id, and traverse the relationship:

#![allow(unused)]
fn main() {
use axum::extract::Path;
use category::CategoryTable;
use product::Product;

async fn list_category_products(Path(id): Path<i64>) -> Json<Vec<Product>> {
    let id_col = Column::<i64>::new("id");
    let products = Category::table(db())
        .clone()
        .with_condition(id_col.eq(id))
        .ref_products();
    let rows = products.list().await.unwrap();
    Json(rows.into_values().collect())
}
}

Three things going on:

• Category::table(db()).clone() — we need an owned Table to chain with_condition onto, so we clone the cached definition — the shape, not any rows.
• with_condition(id_col.eq(id)) — narrows the category table to one row: the one we’re asking for. Nothing hits the database yet.
• ref_products() — traverses the with_many relationship we registered on Category in chapter 2, via get_ref_as. The returned table is Table<SqliteDB, Product>, already scoped to products whose category_id matches the narrowed category set. Chapter 2 walked through what the emitted SQL looks like; it’s the same here.

Register the route in main.rs:

#![allow(unused)]
fn main() {
let app = Router::new()
    .route("/categories", get(list_categories))
    .route("/categories/{id}/products", get(list_category_products));
}

Hit it:

curl http://localhost:3001/categories/1/products

[
  { "name": "Cupcake", "price": 120, "category_id": "1", "is_deleted": false },
  { "name": "Doughnut", "price": 135, "category_id": "1", "is_deleted": false },
  { "name": "Cookies", "price": 199, "category_id": "1", "is_deleted": false }
]

curl http://localhost:3001/categories/2/products

[
  { "name": "Tart", "price": 220, "category_id": "2", "is_deleted": false },
  { "name": "Pie", "price": 299, "category_id": "2", "is_deleted": false }
]

Swap the id in the URL and the products narrow accordingly. The relationship was declared once, back in chapter 2, and we haven’t touched it since — every new nested route reuses the same declaration.

Notice "category_id": "1" in the responses — a string, even though the SQLite column is INTEGER. This is chapter 2’s id convention at work: ids travel as strings through Vantage, so the entity declares category_id: Option<String> and the value converts on its way out of the database.

A generic crud helper

Two handlers so far — list_categories and list_category_products — and they’re already doing the same thing: take a narrowed table, call .list(), return JSON. Adding POST /categories, PATCH /categories/{id}, and DELETE /categories/{id} would mean four more near-identical handlers per entity. That’s not how you scale a codebase.

The five HTTP methods of CRUD all map onto one of two operation shapes:

• List-level at /collection: GET (list all) and POST (create one).
• Item-level at /collection/{id}: GET (read one), PATCH (update), DELETE (remove).

If we can describe the set of rows this endpoint operates on with one closure, the same handler bodies serve every entity. That closure is Fn(SqliteDB, &Params) -> Table<SqliteDB, E> — given the database and whatever path params axum extracted, return the Table to act on. Put it in main.rs:

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use std::sync::Arc;

type Params = HashMap<String, String>;

fn crud<E, F>(make_table: F) -> Router
where
    F: Fn(SqliteDB, &Params) -> Table<SqliteDB, E> + Send + Sync + 'static,
    E: Entity<AnySqliteType> + Serialize + DeserializeOwned + Send + Sync + 'static,
{
    let f = Arc::new(make_table);
    Router::new()
        .route(
            "/",
            get({
                let f = f.clone();
                move |p: Option<Path<Params>>| async move {
                    let params = p.map(|Path(p)| p).unwrap_or_default();
                    let rows = f(db(), &params).list().await.unwrap();
                    Json::<Vec<E>>(rows.into_values().collect())
                }
            })
            .post({
                let f = f.clone();
                move |p: Option<Path<Params>>, Json(entity): Json<E>| async move {
                    let params = p.map(|Path(p)| p).unwrap_or_default();
                    let id = f(db(), &params).insert_return_id(&entity).await.unwrap();
                    Json(serde_json::json!({ "id": id }))
                }
            }),
        )
        .route(
            "/{id}",
            get({
                let f = f.clone();
                move |Path(params): Path<Params>| async move {
                    let id = params["id"].clone();
                    let entity = f(db(), &params).get(&id).await.unwrap().unwrap();
                    Json(entity)
                }
            })
            .patch({
                let f = f.clone();
                move |Path(params): Path<Params>, Json(partial): Json<E>| async move {
                    let id = params["id"].clone();
                    let updated = f(db(), &params).patch(&id, &partial).await.unwrap();
                    Json(updated)
                }
            })
            .delete({
                let f = f;
                move |Path(params): Path<Params>| async move {
                    let id = params["id"].clone();
                    WritableDataSet::<E>::delete(&f(db(), &params), &id).await.unwrap();
                    StatusCode::NO_CONTENT
                }
            }),
        )
}
}

Five handler bodies, each tiny, all generic over the entity type. Mount crud(...) under a prefix with .nest(...) and every route below it gets the full CRUD verb set for free.

/categories becomes a one-liner:

#![allow(unused)]
fn main() {
.nest("/categories", crud(|db, _| Category::table(db).clone()))
}

That gives us:

The nested /categories/{cat_id}/products route uses the same helper. The closure reads cat_id out of the params map to narrow the set:

#![allow(unused)]
fn main() {
.nest(
    "/categories/{cat_id}/products",
    crud(|db, p| {
        let cat_id: i64 = p["cat_id"].parse().unwrap();
        let mut c = Category::table(db).clone();
        c.add_condition(c.id().eq(cat_id));
        c.ref_products()
    }),
)
}

c.id() comes from extending the CategoryTable trait we already set up — add it next to ref_products:

#![allow(unused)]
fn main() {
pub trait CategoryTable {
    fn id(&self) -> Column<i64>;
    fn ref_products(&self) -> Table<SqliteDB, Product>;
}

impl CategoryTable for Table<SqliteDB, Category> {
    fn id(&self) -> Column<i64> {
        self.get_column("id").unwrap()
    }
    fn ref_products(&self) -> Table<SqliteDB, Product> {
        self.get_ref_as("products").unwrap()
    }
}
}

Nesting crud under /categories/{cat_id}/products gives a full CRUD surface for products belonging to that category:

Try it:

curl -X POST http://localhost:3001/categories \
  -H 'content-type: application/json' -d '{"name":"Gluten-Free"}'
# {"id":"4"}

curl http://localhost:3001/categories/1/products/1
# {"name":"Cupcake","price":120,"category_id":"1","is_deleted":false}

curl -X PATCH http://localhost:3001/categories/1 \
  -H 'content-type: application/json' -d '{"name":"Sweet Things"}'
# {"name":"Sweet Things"}

curl -X DELETE http://localhost:3001/categories/4
# HTTP/1.1 204 No Content

The inline list_categories and list_category_products functions can be deleted — crud covers them both.

Error handling

Every handler in crud still ends in .unwrap(). The happy path has been fine to demo, but an API that panics on the slightest database hiccup isn’t usable. The worst part isn’t the 500 — it’s what happens on the wire when axum’s request task panics: the connection is dropped and the client sees an empty reply, with no status code and no body to explain.

A proper REST API needs three things:

• Missing resources return 404, not 500.
• Bad JSON bodies return 400 with a useful message.
• Everything else returns 500 but with a structured JSON body, not silence.

Axum already gives us the middle one for free — it rejects malformed Json<E> bodies with 400. The other two come down to converting VantageError into an HTTP response. Add an ApiError type to main.rs:

#![allow(unused)]
fn main() {
use axum::response::{IntoResponse, Response};

struct ApiError {
    status: StatusCode,
    message: String,
}

impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        (
            self.status,
            Json(serde_json::json!({ "error": self.message })),
        )
            .into_response()
    }
}

impl From<VantageError> for ApiError {
    fn from(e: VantageError) -> Self {
        eprintln!("API error: {:?}", e);
        Self {
            status: StatusCode::INTERNAL_SERVER_ERROR,
            message: e.to_string(),
        }
    }
}

fn not_found(id: &str) -> ApiError {
    ApiError {
        status: StatusCode::NOT_FOUND,
        message: format!("not found: {}", id),
    }
}

type ApiResult<T> = Result<T, ApiError>;
}

Three pieces matter here:

• IntoResponse makes ApiError returnable from a handler; axum calls into_response() to assemble status, headers, and body.
• From<VantageError> lets us use ? inside a handler — every .await? short-circuits to an ApiError mapped to a 500, which axum will render for us.
• not_found(id) is how we build a 404 explicitly. Vantage’s get returns Option<E>, so we .ok_or_else(|| not_found(&id))? the missing case straight into a 404 — no error-message string-matching, no brittleness.

With that in place, the unwraps inside crud turn into ?, and each handler returns ApiResult<T>:

#![allow(unused)]
fn main() {
get({
    let f = f.clone();
    move |p: Option<Path<Params>>| async move {
        let params = p.map(|Path(p)| p).unwrap_or_default();
        let rows = f(db(), &params).list().await?;
        ApiResult::Ok(Json::<Vec<E>>(rows.into_values().collect()))
    }
})
}

For the GET /{id} handler, convert the Option into a 404 at the handler:

#![allow(unused)]
fn main() {
get({
    let f = f.clone();
    move |Path(params): Path<Params>| async move {
        let id = params["id"].clone();
        let entity = f(db(), &params)
            .get(id.clone())
            .await?
            .ok_or_else(|| not_found(&id))?;
        ApiResult::Ok(Json(entity))
    }
})
}

Do the same shape for post, patch, delete — drop the .unwrap(), add ?, wrap the happy path in ApiResult::Ok(...). Handlers that don’t read by id just need the ?.

Try a few error cases:

curl -w "\nstatus=%{http_code}\n" http://localhost:3001/categories/999
# {"error":"not found: 999"}
# status=404

curl -w "\nstatus=%{http_code}\n" -X PATCH http://localhost:3001/categories/999 \
  -H 'content-type: application/json' -d '{"name":"Ghost"}'
# {"error":"patch_table_value: no row found (id: \"999\")"}
# status=500

curl -w "\nstatus=%{http_code}\n" -X POST http://localhost:3001/categories \
  -H 'content-type: application/json' -d '{not-json}'
# Failed to parse the request body as JSON: ...
# status=400

curl -w "\nstatus=%{http_code}\n" -X DELETE http://localhost:3001/categories/999
# status=204

Missing ids produce 404s with a clean JSON body. The malformed body gets axum’s built-in 400 for free. DELETE on a missing id still returns 204 — vantage’s delete is idempotent, and “the resource is gone” is true whether or not it was ever there. PATCH on a missing id still 500s for now — patching doesn’t go through get, so there’s no Option to intercept; adding a pre-flight get-and-ok_or(not_found) before the patch call would give you 404 there too.

Pagination and search

Three categories is fine for dev; three thousand would crush any client that naively calls GET /categories and tries to render everything. Real APIs page through long lists and let callers filter by a search term — two query-string features that belong inside crud so every entity gets them.

Axum parses query strings for us via the Query<T> extractor. Add a small struct for the parameters, and a second extractor on the list handler:

#![allow(unused)]
fn main() {
use axum::extract::Query;
use vantage_table::pagination::Pagination;

#[derive(Deserialize, Default)]
struct ListQuery {
    page: Option<i64>,
    per_page: Option<i64>,
    q: Option<String>,
}
}

Inside crud, only the GET / handler changes. It takes both extractors, mutates the narrowed table with whatever the caller asked for, and lists:

#![allow(unused)]
fn main() {
get({
    let f = f.clone();
    move |p: Option<Path<Params>>, Query(q): Query<ListQuery>| async move {
        let params = p.map(|Path(p)| p).unwrap_or_default();
        let mut t = f(db(), &params);
        if q.page.is_some() || q.per_page.is_some() {
            t.set_pagination(Some(Pagination::new(
                q.page.unwrap_or(1),
                q.per_page.unwrap_or(50),
            )));
        }
        if let Some(term) = q.q.as_deref() {
            t.add_search(term);
        }
        let rows = t.list().await?;
        ApiResult::Ok(Json::<Vec<E>>(rows.into_values().collect()))
    }
})
}

Two things happen:

• set_pagination(Some(Pagination::new(page, per_page))) takes a page number and a page size. Vantage applies these as LIMIT … OFFSET … on the SELECT. Missing params fall back to the defaults (page 1, 50 per page) — and if neither is supplied we don’t touch pagination at all, so the unfiltered list still hits the whole set.
• add_search(term) is the .with_search we used in chapter 2 to add a LIKE filter across all columns. Both end up as extra WHERE clauses on the query that vantage already compiles for us.

Try it:

curl "http://localhost:3001/categories?page=1&per_page=2"
# [{"name":"Sweet Treats"},{"name":"Pastries"}]

curl "http://localhost:3001/categories?page=2&per_page=2"
# [{"name":"Breads"}]

curl "http://localhost:3001/categories?q=Pastries"
# [{"name":"Pastries"}]

curl "http://localhost:3001/categories?q=e"
# [{"name":"Sweet Treats"},{"name":"Pastries"},{"name":"Breads"}]

The nested route gets these for free — crud is the same function. per_page works on /categories/{cat_id}/products out of the box:

curl "http://localhost:3001/categories/1/products?per_page=2"
# [
#   {"name":"Cupcake","price":120,"category_id":"1","is_deleted":false},
#   {"name":"Doughnut","price":135,"category_id":"1","is_deleted":false}
# ]

Because the closure for the nested mount narrows the table with with_condition before crud applies its own pagination/search, the filters compose cleanly: the category scope stays in effect, and pagination just counts rows within it.

Migrating to MongoDB

Chapter 2 closed on a claim: the model layer isolates business code from storage, so swapping databases is a change to the model, not to the routes, handlers, or business logic. Now we cash the check. /categories, /categories/{id}, and /categories/{cat_id}/products keep their exact URL shape, response bodies, and behaviour — but the data lives in MongoDB instead of SqliteDB.

Start Mongo — Docker is the easiest way:

docker run -d --name mongo-learn -p 27017:27017 mongo:7

Cargo.toml

Drop vantage-sql, add vantage-mongodb:

[dependencies]
axum = "0.8.9"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["full"] }
vantage-core = "0.6"
vantage-dataset = "0.6"
vantage-expressions = "0.6"
vantage-mongodb = "0.6"
vantage-table = "0.6"
vantage-types = { version = "0.6", features = ["serde"] }

Entities

Swap the #[entity] type tag from SqliteType to MongoType and the id column name from id to the MongoDB-idiomatic _id. Imports collapse to a single prelude use-line.

src/category.rs:

#![allow(unused)]
fn main() {
use std::sync::OnceLock;

use vantage_mongodb::prelude::*;

use crate::product::Product;

#[entity(MongoType)]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Category {
    pub name: String,
}

impl Category {
    pub fn table(db: MongoDB) -> &'static Table<MongoDB, Category> {
        static CACHE: OnceLock<Table<MongoDB, Category>> = OnceLock::new();
        CACHE.get_or_init(|| {
            Table::new("category", db)
                .with_id_column("_id")
                .with_column_of::<String>("name")
                .with_many("products", "category_id", |db| Product::table(db).clone())
        })
    }
}
}

src/product.rs gets the symmetric changes:

#![allow(unused)]
fn main() {
use std::sync::OnceLock;

use vantage_mongodb::prelude::*;

use crate::category::Category;

#[entity(MongoType)]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Product {
    pub name: String,
    pub price: i64,
    #[serde(skip_serializing_if = "Option::is_none", default)]
    pub category_id: Option<String>,
    #[serde(default)]
    pub is_deleted: bool,
}

impl Product {
    pub fn table(db: MongoDB) -> &'static Table<MongoDB, Product> {
        static CACHE: OnceLock<Table<MongoDB, Product>> = OnceLock::new();
        CACHE.get_or_init(|| {
            let is_deleted = Column::<bool>::new("is_deleted");
            Table::new("product", db)
                .with_id_column("_id")
                .with_column_of::<String>("name")
                .with_column_of::<i64>("price")
                .with_column_of::<String>("category_id")
                .with_column_of::<bool>("is_deleted")
                .with_condition(is_deleted.eq(false))
                .with_one("category", "category_id", |db| Category::table(db).clone())
        })
    }
}
}

Nothing about the struct shape changed — category_id and is_deleted have been real columns since § 2. The with_* calls line up one-for-one with the SQLite version; only the id column name and the entity’s type tag shift.

The CategoryTable trait from § 3 also updates — MongoDB’s _id is a string, not an integer:

#![allow(unused)]
fn main() {
impl CategoryTable for Table<MongoDB, Category> {
    fn id(&self) -> Column<String> {
        self.get_column("_id").unwrap()
    }
    fn ref_products(&self) -> Table<MongoDB, Product> {
        self.get_ref_as("products").unwrap()
    }
}
}

The nested route’s closure gets a little simpler as a result — no parse step, just narrow by the URL’s cat_id string directly:

#![allow(unused)]
fn main() {
crud(|db, p| {
    let mut c = Category::table(db).clone();
    c.add_condition(c.id().eq(p["cat_id"].as_str()));
    c.ref_products()
})
}

.eq(&str) works here because vantage-mongodb’s From<&str> for AnyMongoType auto-promotes 24-character hex strings to ObjectId and leaves everything else as String. The comparison fires with the right BSON type whichever _id convention the collection uses.

main.rs

The crud function’s generic bounds shift from Entity<AnySqliteType> to Entity<AnyMongoType> and its Fn(SqliteDB, ...) becomes Fn(MongoDB, ...). Body unchanged.

#![allow(unused)]
fn main() {
fn crud<E, F>(make_table: F) -> Router
where
    F: Fn(MongoDB, &Params) -> Table<MongoDB, E> + Send + Sync + 'static,
    E: Entity<AnyMongoType> + Serialize + DeserializeOwned + Send + Sync + 'static,
}

Connection and id handling are the only handler-level changes:

#![allow(unused)]
fn main() {
// db() returns MongoDB; connect with URL + database name.
let conn = MongoDB::connect("mongodb://localhost:27017", "learn3")
    .await
    .context("Failed to connect to MongoDB")?;

// item-level ids are MongoId. String → MongoId dispatches to ObjectId when the
// string is a 24-char hex, otherwise stays a plain String, via a `From<String>`
// smart-parse in vantage-mongodb.
let id: MongoId = params["id"].clone().into();
}

The 404 path needs no change at all. Mongo’s get returns Result<Option<E>> just like SQLite’s, so the same .ok_or_else(|| not_found(&id))? maps a missing document to 404 — the promise of the Why not match on the error message? box holds across backends.

Running it

cargo run

The collection is empty, so the first request returns an empty array:

curl http://localhost:3001/categories
# []

POST a few categories — responses carry the auto-generated MongoDB ObjectId as the new id:

SWEETS=$(curl -s -X POST http://localhost:3001/categories \
  -H 'content-type: application/json' -d '{"name":"Sweet Treats"}' \
  | jq -r .id)
echo "$SWEETS"
# 69e2b9101a552c206f5f8468

Create a product in that category by POSTing to the nested route, including the parent id as category_id in the body:

curl -X POST "http://localhost:3001/categories/$SWEETS/products" \
  -H 'content-type: application/json' \
  -d "{\"name\":\"Cupcake\",\"price\":120,\"category_id\":\"$SWEETS\"}"
# {"id":"69e2b9101a552c206f5f846a"}

curl "http://localhost:3001/categories/$SWEETS/products"
# [{"name":"Cupcake","price":120,"category_id":"69e2b9101a552c206f5f8468","is_deleted":false}]

Same URL shape as the SQLite version. Same JSON. Same error codes. Handlers, routing, pagination, filtering — nothing in the request path learned that storage moved from a local file to a document database.

Scaling up: CRUD as a one-liner

The crud function, ApiError, Params, and ListQuery aren’t really tied to this app — they’re generic over any Table<MongoDB, E>. Move them into their own module and main.rs collapses to exactly what it’s about: connecting the database and registering routes. Three files do the work:

src/vantage_axum.rs (one file, ~120 lines) — everything HTTP: ApiError, ListQuery, and the crud<E, F> helper. Accepts any entity E that implements Entity<AnyMongoType> plus the usual serde bounds.

src/db.rs (17 lines) — the static DB: OnceLock<MongoDB>, an init(url, db) helper that connects and stores the handle, and a pub fn db() -> MongoDB accessor that hands out cheap MongoDB clones.

src/main.rs — now down to ~30 lines:

mod category;
mod db;
mod product;
mod vantage_axum;

use axum::Router;
use category::{Category, CategoryTable};
use vantage_axum::crud;
use vantage_mongodb::prelude::*;

#[tokio::main]
async fn main() -> VantageResult<()> {
    db::init("mongodb://localhost:27017", "learn3").await?;

    let app = Router::new()
        .nest("/categories", crud(|db, _| Category::table(db).clone()))
        .nest(
            "/categories/{cat_id}/products",
            crud(|db, p| {
                let mut c = Category::table(db).clone();
                c.add_condition(c.id().eq(p["cat_id"].as_str()));
                c.ref_products()
            }),
        );

    let listener = tokio::net::TcpListener::bind("0.0.0.0:3001").await.unwrap();
    axum::serve(listener, app).await.unwrap();

    Ok(())
}

Adding a new entity to this server is now two things:

1. Write a Table::new(...) constructor for it — declarative, one function, same shape we learned in chapter 2.
2. Mount it.

#![allow(unused)]
fn main() {
.nest("/widgets", crud(|db, _| Widget::table(db).clone()))
}

The full surface: GET list, POST create, GET /{id}, PATCH /{id}, DELETE /{id}, pagination via ?page=&per_page=, full-text search via ?q=, 404s for missing ids, 400s for malformed bodies, structured JSON errors. No new handler code. No per-entity error mapping. No per-entity query-param struct. One line per entity.

Every route plays by the same rules because every route is served by the same crud — the single description of “what this route does” lives in the entity’s Table definition, and the HTTP boundary just routes to it. That is the “one description, many operations” principle from chapter 2’s Table carried all the way to the wire, unbroken.

The vantage_axum module is generic enough to lift directly into a larger codebase — it has no knowledge of Category, Product, or your particular routes. Drop it into your own binary, give it a db() accessor, write entity files, mount routes.

One thing it deliberately doesn’t do is authentication — every endpoint here is anonymous. Auth belongs to the HTTP layer: wrap the Router in your usual tower middleware; Vantage stays out of it.

Vista — the Universal Data Handle

Chapters 1–3 built a typed data layer: Table<SqliteDB, Product>, conditions, relationships, CRUD. That layer is great when you know the entity at compile time. But step 3’s generic crud() helper already showed the limitation — it could only exist because we erased the entity type with Serialize + DeserializeOwned and went through JSON.

A CLI that lists “any table from any backend”, a web admin that draws forms from a YAML schema, a UI data grid that works with whatever you point it at — none of these know your Product struct. They need a handle that carries its own schema, speaks a single value type, and works regardless of which database sits underneath.

That handle is Vista.

What Vista actually is

A Vista wraps a typed Table<DB, E> and erases both the backend and the entity. All data flows through Record<CborValue> — an ordered map of string keys to CBOR values. All schema lives on the Vista itself: columns (with types and flags), references, id column, and a set of capability flags.

Think of the progression:

Table<SqliteDB, Product>   — typed entity, typed backend, compile-time safe
Vista                      — fully erased: schema-bearing, CborValue, no generics

Vista trades away compile-time knowledge for runtime flexibility: everything is a string key and a CBOR value, and the schema travels with the handle rather than with your types. Wrapping a typed Table — this chapter’s path — is also not the only way to get one: every driver ships a VistaFactory that can materialize a Vista from a declarative YAML spec, with Rhai scripts for the expressions YAML can’t state. That path has its own guide — Config-Driven Vistas — and this chapter stays on the typed one.

Wrapping a typed Table

Each backend ships a factory that turns a typed table into a Vista. For SQLite:

#![allow(unused)]
fn main() {
use vantage_sql::prelude::*;
use vantage_vista::Vista;

let table = Product::table(db.clone());
let vista = SqliteVistaFactory::new(db).from_table(table)?;
}

That’s it. The factory harvests columns, id field, title fields, and references from the typed table definition you already built in chapter 2. No extra mapping code.

For MongoDB it would be MongoVistaFactory, for AWS it would be AwsVistaFactory — same shape, different import. The resulting Vista is identical regardless of which factory produced it.

Reading schema

Once you have a Vista, everything is runtime introspection:

#![allow(unused)]
fn main() {
// Columns — name, original type, flags
for name in vista.get_column_names() {
    let col = vista.get_column(name).unwrap();
    println!("{}: {}", col.name, col.original_type);
}
// name: String
// price: i64
// is_deleted: bool

// ID column
if let Some(id_col) = vista.get_id_column() {
    println!("id column: {}", id_col);
}

// Title columns — the human-readable ones
for title in vista.get_title_columns() {
    println!("title: {}", title);
}

// References — relationships declared on the typed table
for (name, kind) in vista.list_references() {
    println!("ref: {} ({:?})", name, kind);
}
// ref: products (HasMany)
}

Column carries an original_type string (preserved from the typed column definition) and a set of flags. The standard flag vocabulary:

Flags are open — drivers and consumers can add their own.

Adding conditions

Vista doesn’t carry its own condition type. Instead, it delegates to a TableShell — the per-driver executor every Vista wraps, and the piece each factory actually builds — which translates the value into whatever the backend speaks:

#![allow(unused)]
fn main() {
let mut v = vista.clone();
v.add_condition_eq("category_id", 1.into())?;

let rows = v.list_values().await?;
// Only products with category_id == 1
}

.into() converts the i64 into a CborValue. The driver’s shell translates that into a native condition — Expression for SQL, bson::Document for MongoDB, AwsCondition::Eq for AWS — and pushes it onto the wrapped table.

Narrowing by id

A common pattern — “I have an id and want the row”:

#![allow(unused)]
fn main() {
let mut v = vista.clone();
v.with_id("7")?;
let row = v.get_some_value().await?;
}

with_id reads the id column name from the schema and calls add_condition_eq for you. Returns &mut Self so you can chain.

Search and ordering

#![allow(unused)]
fn main() {
let mut v = vista.clone();

// Quicksearch — fans across columns flagged SEARCHABLE
v.add_search("tart")?;

// Sort by column
v.add_order("price", SortDirection::Descending)?;

let rows = v.list_values().await?;
}

Both are replace semantics — calling again drops the previous filter/order. Both return an error if the driver doesn’t support them. Check capabilities first (or just try and handle the error).

#![allow(unused)]
fn main() {
let caps = vista.capabilities();
if caps.can_search {
    v.add_search("query")?;
}
if caps.can_order {
    v.add_order("name", SortDirection::Ascending)?;
}
}

Pagination

Vista exposes two pagination primitives. Not every driver supports both — capabilities tell you which to use.

Offset pagination: fetch_page

Random-access — jump to any page by number:

#![allow(unused)]
fn main() {
let mut v = vista.clone();
v.set_page_size(25)?;

let page1 = v.fetch_page(1).await?;   // first 25 rows
let page2 = v.fetch_page(2).await?;   // next 25 rows
}

Works for SQL databases, MongoDB, anything with LIMIT … OFFSET. Requires can_fetch_page.

Cursor pagination: fetch_next

Forward-only — each call returns a token for the next:

#![allow(unused)]
fn main() {
let v = vista.clone();

let (rows, token) = v.fetch_next(None).await?;        // first page
let (rows, token) = v.fetch_next(token).await?;      // second page
// token is None when exhausted
}

The token is opaque — its shape is driver-private (a DynamoDB LastEvaluatedKey, a REST nextToken, an offset counter). Just round-trip it. Requires can_fetch_next.

Traversing references

Same-persistence references (the with_many / with_one from chapter 2) come through automatically. Given a parent row, get_ref resolves the relationship:

#![allow(unused)]
fn main() {
// Given a category row, traverse to its products
let products = vista.get_ref("products", &category_row)?;
for (id, row) in products.list_values().await? {
    println!("  {} — {:?}", id, row["name"]);
}
}

get_ref needs a Record<CborValue> — the parent row. It reads the join field (e.g. category_id) out of that row to build the eq-condition on the target. You can’t traverse from a Vista alone; you need a row first. Typical flow: fetch the parent with get_some_value() or list_values(), then traverse from each row.

get_ref routes in order:

1. Contained relations — records embedded in a column of the parent row (an order’s lines array) resolve first, as an editable sub-Vista.
2. Foreign-key references — the shell forwards to the with_one / with_many relationships declared on the typed table, and re-wraps the narrowed target as a fresh Vista.

Either way the result is another Vista, so traversal chains without ever leaving the universal surface.

Cross-backend references

What if your categories live in PostgreSQL and your products in MongoDB? That is deliberately not a Vista’s job — a Vista honestly describes one backend, and there is no engine that could execute a join across two. Cross-persistence traversal lives one layer up, in vantage-vista-factory’s VistaCatalog: register a loader per model name, declare the relation, and catalog.traverse_from("category", "products", &row) resolves the target in its backend and narrows it by values read from the parent row. The consumer still just receives a Vista — see Config-Driven Vistas for the catalog in action.

Capabilities — the explicit contract

VistaCapabilities is a struct of booleans. The driver sets each flag to reflect what it actually implements:

#![allow(unused)]
fn main() {
let caps = vista.capabilities();
println!("can_count: {}", caps.can_count);
println!("can_insert: {}", caps.can_insert);
println!("can_update: {}", caps.can_update);
println!("can_delete: {}", caps.can_delete);
println!("can_order: {}", caps.can_order);
println!("can_search: {}", caps.can_search);
println!("can_fetch_page: {}", caps.can_fetch_page);
println!("can_fetch_next: {}", caps.can_fetch_next);
}

A CSV file sets can_count and nothing else — it’s read-only. A SQL database sets everything. AWS DynamoDB sets can_count and can_fetch_next (cursor-only) but not can_fetch_page (no random access).

Putting it together

Here’s a small function that takes any Vista and prints a summary — works with any backend:

#![allow(unused)]
fn main() {
use vantage_vista::Vista;

async fn print_vista(vista: &Vista) -> VantageResult<()> {
    let columns = vista.get_column_names();
    let caps = vista.capabilities();

    // Header
    print!("  {:>12} ", vista.get_id_column().unwrap_or("id"));
    for col in &columns {
        print!("{:>16} ", col);
    }
    println!();

    // Count
    if caps.can_count {
        println!("  ({} rows)", vista.get_count().await?);
    }

    // Rows
    let rows = vista.list_values().await?;
    for (id, record) in &rows {
        print!("  {:>12} ", id);
        for col in &columns {
            let val = record
                .get(col)
                .map(|v| format!("{:?}", v))
                .unwrap_or_default();
            print!("{:>16} ", val);
        }
        println!();
    }
    Ok(())
}
}

No generics. No entity type. No backend knowledge. print_vista works with a SQLite Vista, a MongoDB Vista, an AWS Vista — anything the framework can produce.

What we covered

Choose Your Path

You now have a Vista — a runtime handle over your data that works the same no matter what’s behind it. Everything up to here was the foundation. Everything past here is the reactive stack: a local cache, an event bus, and watchable views that update as the data changes.

That stack is backend-agnostic, and the rest of the guide proves it by forking here into two paths. Both build the same layers — Dio, Lens, Augmentation, Scenery, and a watch-streaming HTTP server — and differ only in the backend underneath. Pick the one shaped like your problem; the code you write is nearly identical either way.

Which one is yours?

Why the guide leads with S3

If you own a SQL database, an S3 bucket may look like a strange place to start teaching caching and reactivity. That is deliberate. A slow, read-only, capability-poor backend makes the value of the layer above visible: every millisecond the cache saves, and every sort the backend cannot do but Diorama can, is obvious. A capable SQL database would hide the very problem the reactive stack solves — so Path A teaches it where it shows, and Path B proves it was never about S3.

Both paths converge on the same idea: once a Dio wraps a Vista, nothing above it knows or cares what the backend is. Read whichever path is yours — or both, and watch the code stay the same while the backend changes underneath it.

Dio & Lens — Caching and Events

Chapter 4 gave you Vista — a universal handle that works with any backend. But every call still hits the backend. That was easy to forgive with a local SQLite file; this chapter picks a data source where it hurts: a real cloud API, hundreds of milliseconds away, read-only, and unable to sort or search. We’ll build a small CLI around it — an inventory of a public S3 bucket, with a cache.

Diorama (vantage-diorama) is the layer that makes this pleasant. It sits between your Vista and whatever consumes it, and does three things:

1. Transparent caching. Keep a local copy of your data. Reads come from cache, not from the master — a listing that costs seconds over the network costs microseconds after the first fetch.
2. Capability injection. A Vista backed by S3 can’t sort, search, or paginate a listing. Diorama caches the dataset locally and answers those queries from cache — the consumer sees a richer handle than the backend actually offers.
3. Custom write routing. Writes don’t have to go to the master — useful when the master can’t take them at all. You decide what a write means: a queue, a different store, an API call.

Table, Vista, Dio

This is the third handle to the same records — and like the query-vs-table comparison back in chapter 2, each one trades something away for something new:

Caching

Diorama caches at two levels. Page segments hold windows of an ordered query result — they’re what lets a viewport scroll a huge listing without re-asking the master, and they power the loading strategies you’ll meet in chapter 7. Beneath them sits the key/value record store — one entry per record id, plain and dumb. We keep it simple for now: everything in this chapter runs on the record store alone.

A Dio owns exactly the two blocks above: a master Vista — the source of truth, however far away — and a cache, the local copy. What it deliberately does not decide is policy: when the cache fills, when it goes stale, what a write means. There are too many valid answers for one default — seed once and keep forever, re-fetch on a timer, reconcile from a push stream, write through to the master, write somewhere else entirely. That’s what the Lens is for: it describes how the cache is used. The simplest possible Lens pumps the master into the cache once, when the Dio loads:

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_at("cache.redb")
        .on_start(|dio| {
            let dio = dio.clone();
            async move {
                // Pump: read everything from the master, write it to the cache.
                let rows = dio.master().list_values().await?;
                dio.cache().insert_values(rows).await
            }
        })
        .build()?,
);

let dio = lens.make_dio(vista).await?;   // runs on_start, returns a warm Dio
}

With the Dio in place, there are two ways to consume it. The first is the facade Vista — dio.vista() — for proactive querying: you ask, it answers, exactly like the backend’s own Vista from chapter 4. Same interface, same records — except the answers come from the cache, and the capability set can be wider than the master’s, because the Lens decides what to add: a read-only master gains writes when the Lens routes them somewhere, and counting is always on the menu because the cache can count. The second way in is the Scenery — a standing, reactive view that keeps itself current and tells you when it changed. UIs bind to sceneries; chapter 7 lives there.

The facade Vista

Hand dio.vista() to chapter 4’s print_vista and it just works — it never learns a cache is underneath. What makes the facade interesting is its capability set. Chapter 4’s contract was honest but rigid: whatever the backend can’t do, your application can’t have. The facade solves that — it carries the master’s capabilities, and the Lens can manipulate and extend them. Our read-only S3 listing gains can_insert the moment an on_write callback gives writes somewhere to go. The honesty contract still holds; the facade just advertises what the pipeline can do, not what the backend alone can.

The project: a weather-station inventory

NOAA publishes its daily climate archive — GHCN, one CSV per weather station — as a public S3 bucket in the AWS Open Data registry. It’s a perfect Diorama subject: listing it is slow, every listing request is paid again on every run, and the API can’t sort or search. We’ll grow one small tool across three chapters:

• This chapter — a CLI that lists the station files from a persistent local cache.
• Chapter 6 — augmentation: each file gains columns computed from its contents (how many readings, how recent).
• Chapter 7 — a live terminal UI that scrolls the whole archive and fetches per-file data for exactly the rows on screen.

Set up the project:

cargo new learn-4 && cd learn-4
cargo add serde --features derive
cargo add tokio --features full
cargo add vantage-aws vantage-diorama vantage-vista

vantage-aws is the S3/DynamoDB/IAM driver — it signs (or deliberately doesn’t sign) requests itself, so there’s no AWS SDK in the tree. vantage-diorama is what this chapter is about. The code splits into two files: files.rs holds the table definition, main.rs uses it.

files.rs — the listing as a table

One row per file in the bucket. This is chapter 2’s pattern — an entity, and a table() constructor that describes the source — pointed at a cloud API instead of a database:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use vantage_aws::prelude::*;

/// One file in the bucket. Field names match S3's wire XML
/// (`<Contents><Key/><Size/></Contents>`).
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct File {
    #[serde(rename = "Key")]
    pub filename: String,
    #[serde(rename = "Size")]
    pub size: String,
}
}

The serde renames map S3’s wire names onto the fields we actually want to write in Rust. Size is a String because that’s what the XML carries — no silent coercion.

#![allow(unused)]
fn main() {
impl File {
    /// `ListObjectsV2` narrowed to one bucket and prefix. S3 sends at
    /// most `max-keys` keys per response; the `@continuation-token`
    /// cursor tells the framework to keep requesting pages until the
    /// listing is complete.
    pub fn table(aws: AwsAccount, bucket: &str, prefix: &str) -> Table<AwsAccount, File> {
        Table::new(
            "restxml/Contents@continuation-token=NextContinuationToken:s3/GET /{Bucket}?list-type=2",
            aws,
        )
        .with_id_column("Key")
        .with_column_of::<String>("Size")
        .with_condition(eq("Bucket", bucket))
        .with_condition(eq("prefix", prefix))
        .with_condition(eq("max-keys", 100))
    }
}
}

The table name is doing a lot of work here — for vantage-aws, it’s the wire protocol spelled out: restxml is the protocol S3 speaks; Contents is the response element holding the rows; s3/GET /{Bucket}?list-type=2 is the service and request. Conditions complete the request the same way chapter 2’s conditions completed a WHERE clause: Bucket fills the path placeholder, and anything else (prefix, max-keys) becomes a query parameter.

The @continuation-token=NextContinuationToken part is auto-pagination. S3 answers at most max-keys files per response, plus a continuation token when more exist; the cursor declaration tells the driver to keep re-issuing the request — token folded back in — until the listing is complete. One list() call, as many HTTP requests as it takes.

{protocol}/{array_key}[@cursor|@request=response]:{service}/{METHOD} {path}?{query}

A first listing

main.rs, shortest possible version — build the table, list it, print:

mod files;

use files::File;
use vantage_aws::prelude::*;

const BUCKET: &str = "noaa-ghcn-pds";
const PREFIX: &str = "csv/by_station/GM";

#[tokio::main]
async fn main() {
    if let Err(e) = run().await {
        e.report();
    }
}

async fn run() -> VantageResult<()> {
    let aws = AwsAccount::public("us-east-1");
    let files = File::table(aws, BUCKET, PREFIX);

    for (filename, file) in files.list().await? {
        println!("{:>10}  {filename}", file.size);
    }
    Ok(())
}

The GM prefix narrows the archive to Germany’s stations — 1122 files, small enough to look at, big enough to feel:

   4796352  csv/by_station/GM000001153.csv
   7057976  csv/by_station/GM000001474.csv
   2978238  csv/by_station/GM000002277.csv
   ...
1122 files in 2.3s

2.3 seconds: twelve HTTPS round-trips (1122 files at 100 per page), paid on every run, forever, because nothing remembers the answer. This is the itch the rest of the chapter scratches.

The master Vista, and what it can’t do

The Dio wants a Vista for its master, so we erase the typed table the same way chapter 4 did:

#![allow(unused)]
fn main() {
let master = aws
    .vista_factory()
    .from_table(File::table(aws.clone(), BUCKET, PREFIX))?;
}

Ask this Vista what it can do, and you’ll see why this backend needs a Diorama:

#![allow(unused)]
fn main() {
let caps = master.capabilities();
// can_count: true    can_fetch_next: true — and that's it.
// can_search: false   can_order: false   can_insert: false
}

can_fetch_next is worth pausing on — it’s the one capability this chapter leans on. fetch_next(token) is Vista’s cursor-style pagination from chapter 4: pass None for the first page, pass back the returned token for the next one. For S3 the driver defines the token as the last key of the previous page, because S3 lists keys in order and accepts any key as a starting point (start-after). That has a consequence the generic contract doesn’t promise: this cursor survives process restarts. Any key you already hold — say, the last key in a cache — resumes the listing right after it.

The Lens: pump pages, resume where you left off

Now the Lens. One callback — on start, synchronize the cache with the bucket:

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_at("cache.redb")
        .on_start(|dio| {
            let dio = dio.clone();
            async move { sync(&dio).await }
        })
        .build()
        .context("Failed to build lens")?,
);
let dio = lens.make_dio(master).await?;
}

• .cache_at("cache.redb") opens a redb file on disk. This is why the cache survives process restarts. An in-memory alternative, .cache_in_memory(), is there for when persistence isn’t wanted.
• .on_start(...) fires once, inside make_dio. Callbacks receive &Dio and clone it to hold across .await — a cheap Arc bump; every Lens callback follows this shape.
• make_dio does the rest in one call: opens a cache table named after the master, spawns the write worker, runs on_start, and hands back the Dio.

sync is where the cursor pays off. Instead of one big list_values(), it pulls the listing one page at a time and seeds the cursor from the cache:

#![allow(unused)]
fn main() {
/// Pump the master listing into the cache, one page per request. S3's
/// paging cursor is simply "the last key seen" — so the last key already
/// in the cache resumes the listing, and pages loaded by an earlier run
/// (even one that was interrupted) are never fetched again.
async fn sync(dio: &Dio) -> VantageResult<()> {
    let mut token: Option<CborValue> = dio
        .cache()
        .list_values()
        .await?
        .keys()
        .last()
        .cloned()
        .map(Into::into);
    loop {
        let start = Instant::now();
        let (page, next) = dio.master().fetch_next(token).await?;
        let count = page.len();
        dio.cache().insert_values(page.into_iter().collect()).await?;
        println!("fetched {count} files in {:?}", start.elapsed());
        if next.is_none() {
            return Ok(());
        }
        token = next;
    }
}
}

Read the first statement again: the initial token is the last filename already cached. On a cold cache that’s None — start from the top. On a warm cache it’s the end of what we have — S3 continues from there. Kill the process halfway through a sync and run it again: the pages you already paid for are never fetched twice. The cache isn’t just an answer store; it’s the resume point.

Reads come from the cache

The listing loop stops asking S3 and starts asking the Dio — through the facade Vista from the first half of the chapter. Notice who talks to what: sync, being Lens plumbing, addresses the Dio’s two sides directly (dio.master() to fetch, dio.cache() to store and resume); a consumer asks dio.vista() and never learns what’s underneath. The records come back as Record<CborValue> — the erased form from chapter 4 — so field access goes through the CborValueExt helpers the prelude brings in:

#![allow(unused)]
fn main() {
let start = Instant::now();
let listing = dio.vista().list_values().await?;
for (filename, file) in &listing {
    let size = file.get("Size").and_then(|v| v.as_str()).unwrap_or("");
    println!("{size:>10}  {filename}");
}
println!("{} files from cache in {:?}", listing.len(), start.elapsed());
}

First run — cold cache, the sync pump narrating each page:

fetched 100 files in 215ms
fetched 100 files in 217ms
...
fetched 22 files in 114ms
   4796352  csv/by_station/GM000001153.csv
   7057976  csv/by_station/GM000001474.csv
   ...
1122 files from cache in 18ms

Second run — same command, new process:

fetched 0 files in 370ms
1122 files from cache in 17ms

One round-trip — the resume request from the last cached key, confirming nothing new exists — and then the full listing in 17 milliseconds. New files that do appear under the prefix arrive on exactly that request, without re-fetching the thousand we already hold.

Invalidating

A resuming cache has one blind spot: it only ever looks past what it holds, so files deleted from the bucket linger locally. The honest fix is to start over — wipe and re-pump:

#![allow(unused)]
fn main() {
if std::env::args().any(|a| a == "--invalidate") {
    dio.cache().clear().await?;
    sync(&dio).await?;
}
}

$ cargo run -- --invalidate
fetched 0 files in 370ms
fetched 100 files in 215ms
...
1122 files from cache in 18ms

The CLI is complete: two seconds once, milliseconds forever after, resumable mid-sync, and an escape hatch back to the truth.

The event bus

One more thing make_dio set up, invisibly: every Dio carries a broadcast event bus. Anything that changes data announces it there as a DioEvent — DatasetChanged when the set of records was rewritten wholesale, RecordChanged / RecordInserted / RecordRemoved for row-level changes, Refreshing when a reconcile starts, WriteFailed when a queued write fails. Nothing in this chapter listens yet; chapter 7’s sceneries subscribe to this bus to know when to recompute, and chapter 6’s hydration uses it to report progress.

One rule is worth forming as a habit now: direct cache writes are silent. Our sync calls dio.cache().insert_values(...), which stores rows and announces nothing — harmless while nothing listens, invisible data the moment something does. When observers exist, either use the Dio’s row-level helpers that write and announce in one motion — patched(id, record), removed(id) — or follow a bulk cache write with notify_dataset_changed(), the “re-read everything” announcement. Chapter 7’s live view does exactly that.

Callback summary

What we covered

Augmentation — Enriching Rows

Chapter 5’s inventory knows every station file’s name and size — the two things an S3 listing gives away for free. Everything interesting is inside the files: how many readings a station has recorded, and how recently. Getting that costs one download per file, which is exactly the kind of expense you want to pay once and remember.

Augmentation is Diorama’s answer: enrich the master’s rows, one row at a time, from a detail source — and let the Dio’s cache hold the result. The listing stays the cheap, fast spine it was in chapter 5; each row gains columns it never had, hydrated on demand and persisted alongside the row. This chapter adds two such columns to the inventory:

      SIZE     ROWS     LATEST  FILENAME
   4796352   140629   19911231  csv/by_station/GM000001153.csv

ROWS — how many readings the file holds; LATEST — the date of the most recent one. Neither exists anywhere in S3’s listing; both are computed from the file’s contents.

We build on chapter 5’s crate unchanged — learn-5 starts as a copy of learn-4 (files.rs and main.rs exactly as they were) and this chapter only adds. One new file, readings.rs, describes the detail side; a handful of lines in main.rs wire it up.

The detail side: readings.rs

Every station file is a CSV of date-ordered readings:

ID,DATE,ELEMENT,DATA_VALUE,M_FLAG,Q_FLAG,S_FLAG,OBS_TIME
GM000001153,18910101,TMAX,-20,,,I,
GM000001153,18910102,TMAX,-40,,,I,
...

So the two columns we’re after are cheap derivations: rows is the line count minus the header, latest is the DATE field of the last line. The expensive part is getting the contents at all.

The detail source is a table — the same ListObjectsV2 listing as the master, in fact, with two differences: it declares no prefix (the augmentation will narrow it to a single file per fetch), and it carries the derived columns. Start readings.rs with its entity:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use vantage_aws::prelude::*;

/// One file, seen through the augmenter: the listing row plus columns
/// computed from the file's contents.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Readings {
    #[serde(rename = "Key")]
    pub filename: String,
    pub rows: i64,
    pub latest: String,
}
}

Lazy expressions

Where do rows and latest come from? Chapter 2’s with_expression won’t do it — those expressions lower into the backend’s query (a SQL subselect), and S3 can’t compute anything server-side. What we need runs on our side, after the row comes back.

That’s a lazy expression: a column computed in Rust, on the returned record. Lazy expressions apply in declaration order, and each callback borrows the record as built so far — including the columns earlier lazy expressions added. That ordering rule is the whole trick:

#![allow(unused)]
fn main() {
impl Readings {
    pub fn table(aws: AwsAccount, bucket: &str) -> Table<AwsAccount, Readings> {
        let bucket = bucket.to_string();
        Table::new("restxml/Contents:s3/GET /{Bucket}?list-type=2", aws.clone())
            .with_id_column("Key")
            .with_condition(eq("Bucket", bucket.clone()))
            .with_lazy_expression("contents", move |row| {
                let aws = aws.clone();
                let bucket = bucket.clone();
                let key = row.get("Key").and_then(|v| v.as_str()).unwrap_or_default().to_string();
                async move { Ok(s3::get_object(&aws, &bucket, &key).await?.into()) }
            })
            .with_lazy_expression("rows", |row| {
                // Every line after the CSV header is one reading.
                let contents = row.get("contents").and_then(|v| v.as_str()).unwrap_or_default();
                let rows = contents.lines().count().saturating_sub(1) as i64;
                async move { Ok(rows.into()) }
            })
            .with_lazy_expression("latest", |row| {
                // Readings are date-ordered; take the last line's DATE column.
                let contents = row.get("contents").and_then(|v| v.as_str()).unwrap_or_default();
                let latest = contents
                    .lines()
                    .last()
                    .and_then(|line| line.split(',').nth(1))
                    .unwrap_or_default()
                    .to_string();
                async move { Ok(latest.into()) }
            })
    }
}
}

Read it as a pipeline over one record. The listing returns {Key, Size, …}; then:

1. contents reads Key off the record and downloads the file — s3::get_object is the driver’s raw fetch, unsigned here just like the listing. Its return value is inserted into the record under contents. This is the expensive step, and it happens once.
2. rows never touches the network: it reads row.get("contents") — the column the previous expression just added — and counts lines.
3. latest reads the same contents and takes the last line’s date.

One download feeds every derived column declared after it. Each callback clones what it needs out of the borrowed record before going async, and each expression’s name is also registered as a column on the table, so the derived fields are part of its schema like any other.

Why not just list this table?

It’s tempting to stop here — the augmenter table already produces every column we want, so why not make it the Dio’s master? Because lazy expressions run when data is fetched. A list() on this table triggers the whole pipeline for every row it returns: a thousand files means a thousand downloads inside one blocking call, and nothing comes back until the last one lands. That’s not a listing — it’s a batch job wearing a listing’s interface.

What we’re after is different. The list of files should reach the user instantly — it’s chapter 5’s cached listing, it costs milliseconds. The expensive columns should then be fetched one row at a time, each result written into the Dio and announced on the event bus, so anything watching updates progressively. Loading details for every station takes a long time no matter what; the win is in never blocking the listing on it, and in choosing the order — the next chapter puts a viewport in charge, so the rows the user is currently looking at are hydrated first, and rows nobody scrolls to are never fetched at all.

That split — a cheap master everyone lists, an expensive detail source consulted per row — is what augmentation declares.

Wiring the augmentation

Back in main.rs. The augmenter becomes a Vista like any table, and the Dio is told how to use it — this is the only structural addition to chapter 5’s run():

#![allow(unused)]
fn main() {
let augmenter = aws
    .vista_factory()
    .from_table(Readings::table(aws.clone(), BUCKET))?;

let dio = lens.make_dio(master).await?.augment(
    Arc::new(VistaCatalog::new()),
    vec![Augmentation {
        detail: Detail::Fixed(Arc::new(augmenter)),
        source: Source::Column {
            from: "Key".into(),
            to: Some("prefix".into()),
        },
        fetch: Fetch::PerRow,
        merge: MergeRule {
            columns: vec!["rows".into(), "latest".into()],
        },
    }],
);
}

An Augmentation answers four questions:

• detail — where do detail records come from? Detail::Fixed holds our augmenter Vista directly. (Detail::Catalog(name) resolves one by name instead — the config-driven form, which is also why augment takes a VistaCatalog. Building an empty catalog just to satisfy the signature is admittedly awkward; the parameter earns its place only in the catalog form.)
• source — how does a master row select its detail record? Source::Column maps the master’s Key onto the detail table’s prefix condition. A full filename used as an S3 prefix matches exactly one object — so each fetch narrows the augmenter to a one-row listing, and the lazy expressions run for precisely that file.
• fetch — Fetch::PerRow: one detail fetch per master row.
• merge — which detail columns land on the master row? Just rows and latest. contents is deliberately absent: it exists only inside the detail fetch, feeds the derived columns, and is never cached. The megabytes stay out of the Dio; the two numbers stay in.

Reads hydrate

Nothing has fetched anything yet — declaring an augmentation is free. The work happens on read: rows a facade read returns come back hydrated — any of them still missing its augment columns runs the detail fetch first, and the result is written back to the cache as complete.

Not every read, though. The listing must stay what chapter 5 made it — instant — so list_values through the facade returns the cheap rows untouched. Hydration belongs to bounded reads: get_value for one record, fetch_window for a range. The rows you ask for are the rows that pay:

#![allow(unused)]
fn main() {
// The listing stays instant — cheap rows, no downloads.
let listing = dio.vista().list_values().await?;
println!("{} files (listed in {:?})", listing.len(), start.elapsed());

// Details are paid for by the rows you ask for: a window of ten.
let window = dio.vista().fetch_window(0, 10).await?;
for (filename, file) in &window {
    let size = file.get("Size").and_then(|v| v.as_str()).unwrap_or("");
    let rows = file.get("rows").and_then(|v| v.as_i64()).unwrap_or(0);
    let latest = file.get("latest").and_then(|v| v.as_str()).unwrap_or("");
    println!("{size:>10} {rows:>8} {latest:>10}  {filename}");
}
}

Ten downloads is still a wait worth narrating, and the Dio announces it: one DioEvent::Hydrating with the pending count before the first fetch, then a RecordChanged per row as each lands. A dozen lines of event plumbing become the progress display:

#![allow(unused)]
fn main() {
let mut events = dio.subscribe_events();
tokio::spawn(async move {
    while let Ok(event) = events.recv().await {
        match event {
            DioEvent::Hydrating { pending } => println!("hydrating {pending} files…"),
            DioEvent::RecordChanged { id } => println!("  {id}"),
            _ => {}
        }
    }
});
}

First run — chapter 5’s sync, the instant listing, then the window doing its downloads:

fetched 100 files in 215ms
...
1122 files (listed in 16ms)
hydrating 10 files…
  csv/by_station/GM000001153.csv
  csv/by_station/GM000001474.csv
  ...
   4796352   140629   19911231  csv/by_station/GM000001153.csv
   7057976   206874   20260531  csv/by_station/GM000001474.csv
   2978238    87207   19990131  csv/by_station/GM000002277.csv
   ...
10 files detailed in 20.2s

Real data, and readable at a glance: station GM000001474 has 206,874 readings and is still reporting (May 2026); its neighbour GM000001153 went silent at the end of 1991.

Second run — new process, warm cache:

fetched 0 files in 340ms
1122 files (listed in 16ms)
   4796352   140629   19911231  csv/by_station/GM000001153.csv
   ...
10 files detailed in 16ms

No hydrating line at all: those ten rows already carry their augment columns, so the window finds no gaps and never touches the network. Twenty seconds became sixteen milliseconds; the derived numbers live in cache.redb with the rest of the row. Ask for a different window — fetch_window(500, 10) — and only its gaps download. (--invalidate still clears everything, derived columns included — derived data is data.)

What we covered

Scenery — Reactive Views

Chapter 6’s CLI asks for one window of details and exits — the asking is still on the programmer. A real interface is a standing view: the user scrolls, and the window should follow; details land one by one, and the rows should repaint; the bucket changes, and the listing should notice. And it has to hold up at scale — the full GHCN archive is about 122,000 station files, of which a screen shows forty. What a UI actually needs is an ordered row set it can read by index, hydration that follows the user’s attention, and a signal whenever anything on screen changes.

Scenery is that layer. Each Scenery is a reactive view onto a Dio, exposing one access pattern:

• TableScenery — ordered rows by index, with a viewport
• RecordScenery — a single record by id
• ValueScenery — a computed scalar (count, sum, custom)

All three share one reactivity mechanism: a Generation counter that bumps whenever the view’s state changes. Consumers subscribe() to a watch channel and redraw on each bump — the channel only ever holds the latest generation, so a burst of changes costs one repaint, not one per change.

A Scenery is not another handle to your data — it hands you no records to keep. It maintains one particular view of the Dio’s copy, precomputed and ready for a widget to read:

That last cell is where this chapter ends up. The consumer — a terminal table here, a desktop grid or a remote client such as a React page elsewhere — drives its scenery in one repeating loop:

1. The consumer declares what it shows: set_viewport(40..80) — nothing more than “these rows are on screen.”
2. The scenery turns that into work for the Dio: list pages for the spine, detail fetches for the rows in view.
3. Results land in the Dio’s cache, and the event bus announces each one — RecordChanged, row by row.
4. The scenery reacts to the bus: it updates the affected slot and bumps its Generation.
5. The consumer’s watch channel ticks; it re-reads row(idx) for what’s visible and repaints.

The code that runs this loop for one particular consumer is the adapter — the same handful of lines whether it paints a terminal, a desktop toolkit, or a wire to the browser.

Back to the inventory

Chapter 6 left the CLI listing one prefix and detailing ten fixed rows. What we actually want from the inventory is the tool you’d reach for daily — and that sets the requirements:

• the whole archive, not a prefix — all 122,000 station files, scrollable end to end;
• instant open, even on the first run — an empty table that fills as the listing arrives, never a frozen prompt;
• details where the user is looking — the rows on screen sprout ROWS and LATEST as their files are read, and the cursor drags that attention with it;
• standing freshness — the bucket changes, the table follows, plus a running total that ticks upward as data lands.

Measure chapter 6’s ending against that. It read from the Dio with dio.vista().fetch_window(0, 10) — and it works fine, for a CLI: the read is bounded to ten records, and it resolves once all ten have their details. Nothing blocks — it’s an async wait, the runtime stays free — but the user waits all the same: twenty seconds of silence on a cold cache, and every widening of the window widens the wait. For a better experience we need a more responsive UI:

1. display results as soon as we have them — don’t hold rows hostage to their details;
2. show data immediately and fix it reactively as better values arrive;
3. register what the user is currently looking at, reducing what is fetched and transmitted.

The Scenery implements exactly this:

• rows are served the moment it opens — whatever the cache already holds, with … standing in for details still pending;
• every landed detail updates its own row and bumps the Generation — one tick, one repaint;
• set_viewport(range) registers the user’s attention, and only those rows hydrate.

For the consumer, this chapter reaches for ratatui — a terminal-UI library — which Vantage updates in real time through a ready-made adapter. The dataset-ui-adapters crate ships such adapters for several UI frameworks (egui, Slint, GPUI, Cursive, Tauri among them), and they all speak the same scenery loop. Here is ours a few seconds into a cold start — the listing still streaming in and the first two visible rows already hydrated:

FILENAME                              SIZE        ROWS      LATEST
csv/by_station/ACW00011604.csv        41673       1231      19490814
csv/by_station/ACW00011647.csv        481021      14355     20260710
csv/by_station/AE000041196.csv        1998959     …         …
…
 8000 rows · 2 augmented · total rows 15586 · ↑/↓ PgUp/PgDn scroll · r refresh · q quit

learn-6 starts as a copy of learn-5’s data side — files.rs, readings.rs, and the whole Dio setup carry over. Three things change: the prefix widens to the entire archive, the Lens learns to serve a UI, and the printing loop is replaced by a scenery-bound terminal table.

cargo add dataset-ui-adapters --features ratatui

Widening the scope is two touches. The prefix loses its country filter:

#![allow(unused)]
fn main() {
const PREFIX: &str = "csv/by_station/";
}

And the listing pages grow. Every page is one HTTP round-trip: chapter 5 fetched 100 keys at a time, a comfortable twelve requests for its 1,122 files — but at 122,000 files the same setting means 1,220 round-trips of mostly latency. S3 caps a single response at 1,000 keys, so files.rs asks for the cap and the whole archive syncs in about 122 requests:

#![allow(unused)]
fn main() {
        // S3's per-response maximum — we're listing the whole station set.
        .with_condition(eq("max-keys", 1000))
}

A Lens that serves a UI

Chapter 5’s Lens had one job: warm the cache before anyone reads. A UI inverts the priorities — the screen must appear immediately, on whatever is known so far, and data streams in behind it. Three additions to the builder:

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_at("cache.redb")
        .on_start(|dio| {
            let dio = dio.clone();
            async move { sync(&dio).await }
        })
        // Don't wait for the sync: the UI opens on whatever the cache
        // holds (nothing, on a first run) and rows stream in behind it.
        .on_start_blocking(false)
        // The scenery's list pages come straight from the warmed cache —
        // zero network. The master is only contacted by the refresh
        // reconcile and the per-row detail fetches.
        .on_list_page(|dio, q| {
            let dio = dio.clone();
            async move {
                Ok(dio
                    .cache()
                    .list_values()
                    .await?
                    .into_iter()
                    .skip(q.offset)
                    .take(q.limit)
                    .collect())
            }
        })
        // Reconcile against the bucket once a minute: new files appear
        // as un-hydrated rows, vanished ones drop out, changed ones are
        // demoted for re-hydration.
        .refresh_every(Duration::from_secs(60))
        .build()
        .context("Failed to build lens")?,
);
}

on_start_blocking(false) detaches the sync: make_dio returns at once and on_start runs as a background task. The UI’s first frame shows an empty table; a few minutes later it shows 122,000 rows — filling in page by page in between, a thousand at a time. For that to be visible, the sync itself gains one line — after each landed page it announces the change on the event bus, and every open view re-reads:

#![allow(unused)]
fn main() {
        dio.cache().insert_values(page.into_iter().collect()).await?;
        dio.notify_dataset_changed();
}

(The println! narration from chapter 5 is gone — a terminal UI owns the screen now.)

on_list_page needs the most context, because it names how a TableScenery loads. An augmented Dio drives two-pass loading: a list pass fetches cheap rows and establishes the view’s order — its spine — and a detail pass hydrates individual rows’ augment columns as the viewport reaches them. The passes are separately pluggable. The detail pass is already defined — it’s chapter 6’s augmentation. The list pass, left alone, would ask the master — a full 122-request S3 walk every time a view (re)builds its spine. But chapter 5 built something better: a cache that is the listing. Registering on_list_page overrides the list pass; the QueryDescriptor argument carries the window (offset, limit — plus conditions, sort, and search when the view has them), and our implementation is a window over the cache. The scenery’s spine now costs zero network.

refresh_every completes the freshness story. Once a minute the Dio reconciles its cache against the master (this does walk the listing — 122 requests, in the background): new files appear as un-hydrated rows, deleted ones drop out, and a file whose listing entry changed keeps showing its old numbers but is marked stale — the next time it’s on screen, the detail pass re-fetches it. Chapter 5 needed --invalidate to notice deletions; a standing view just notices.

Creating the reactive UI

Our UI needs two things from the Dio: the rows, and a running total. Each one is a scenery. Let’s open them one at a time.

The table

#![allow(unused)]
fn main() {
// One list page covers the whole cached listing (~122k station files);
// the viewport's detail pass hydrates whatever is on screen first.
let scenery = dio.table_scenery().page_size(200_000).open().await?;
}

.open() seeds the scenery from the cache, spawns its reactor (the task that watches the Dio’s event bus), and hands back Arc<dyn TableScenery>. What you get is deliberately small and synchronous:

• row_count() and row(idx) — an EnrichedRecord: the record plus a per-row status;
• set_viewport(range) — more on this in a moment;
• subscribe() — the generation channel from the start of the chapter.

The builder can also chain sort(col, dir), search(text), and where_eq(col, value) — every one of them served locally, on a backend that can do none of it. We don’t use them here — the table shows the archive as-is.

A UI rarely commits to one order at open time, though — and it doesn’t have to reopen. The handle mutates in place: set_sort(col, dir) and set_search(text), bound to a key or a header click, re-point the scenery at the ordered index for the new variant, swap the visible rows in one atomic step (the grid never blanks mid-reorder), and restart hydration for whatever is on screen. Sorting back reuses the already-built index — zero list calls. Conditions are the exception: where_eq defines what the view is, so it’s set at open, not toggled on a live scenery.

One thing probably caught your eye: page_size(200_000)?! That’s the list pass — how many spine rows one on_list_page call returns. It’s a separate axis from the viewport, which drives only the detail pass over rows the spine already holds. Left at the default, the spine would stop at the first hundred files and nothing in this UI asks for page two. Set past the archive size, the first list call builds the entire spine — one cheap read of the local cache.

The viewport

The viewport is the scenery’s answer to “which rows is the user actually looking at?” — a plain range of row indexes, like 40..80. It’s declared through set_viewport(range), the method on the handle we just opened. You rarely call it yourself: whoever renders the table calls it as the user scrolls — later in this chapter, that’s the ratatui adapter’s job.

It is the load-bearing call, because the viewport drives hydration. The rows a consumer declares as its viewport are the rows the detail pass works on. Framework code calls this the demand gate: no row is detailed unless some live view demands it. Everything else stays a cheap list row — and rows never observed never cost a download.

Recognize it? This is chapter 6’s bounded read with the asking automated: there the programmer chose the window; here the viewport is the window, and it moves with the user.

The running total

#![allow(unused)]
fn main() {
// Grand total of the ROWS column — recomputes reactively as files
// hydrate, so the status bar counts up while data arrives.
let totals = dio.value_scenery().sum("rows").open().await?;
}

Hold on — how can it sum rows when most rows aren’t augmented yet? It doesn’t wait for them:

• it recomputes over the cache whenever the Dio announces a change;
• rows that have no rows column yet are skipped — the sum covers what has been observed so far;
• each detail fetch that merges rows into the cache fires RecordChanged; the scenery recomputes and bumps its generation only when the value actually moved.

So the number in the status bar is honest about coverage: it starts at zero and climbs with hydration, one landed file at a time.

(There is a third kind, RecordScenery — dio.record_scenery(id) — one record under the same contract: read it, subscribe, redraw on bump. A detail pane beside the table would use one; our table doesn’t need it.)

Open freely, drop when done

Sceneries are inexpensive. A page opens as many as it needs — a grid, a running total, a detail pane — and identical opens share one instance under the hood (the sharing key is the scenery’s query — sort, search, conditions — plus the columns it demands; two views asking the same question get the same scenery). The other half of that contract: release them when the page closes. Dropping the handle stops the scenery’s tasks and withdraws its demand, so a closed page stops pulling data. Chapter 8 meets the one case where sharing is wrong — two remote viewers asking the same question but scrolling different pages — and opts out per scenery.

Binding to a terminal

What remains is rendering — and none of it is specific to this app. Scrolling a virtualized table, forwarding the visible range as the viewport, redrawing on generation bumps: that’s a reusable binding, and dataset-ui-adapters ships it for ratatui in its ratatui_dio module. The entire UI:

#![allow(unused)]
fn main() {
    ratatui_dio::SceneryTable::new(scenery)
        .with_column("FILENAME", "Key", 0)
        .with_column("SIZE", "Size", 10)
        .with_column("ROWS", "rows", 8)
        .with_column("LATEST", "latest", 10)
        .with_status_value("total rows", totals)
        .run()
        .await
        .context("terminal UI failed")
}

Line by line:

• ratatui_dio::SceneryTable::new(scenery) — hands the scenery to the adapter. The full module path marks the boundary: everything before this line was framework, everything after is the ratatui binding.
• .with_column(header, field, width) — one table column: the header text, the record field it reads, and a width in characters (0 = flexible fill). A field the row doesn’t have yet renders as … — which is exactly how un-hydrated rows look.
• .with_status_value(label, scenery) — pins a ValueScenery into the status bar; it repaints whenever that scenery’s generation bumps, so our total rows sum ticks live.
• .run() — takes over the terminal until q. It draws only the visible rows (at 122,000 you don’t build widgets for the rest), keeps the scenery’s viewport on a ten-row band around the cursor — details load for the record the user is on and its neighbours, not the whole screen. Band-not-screen is the adapter’s policy, not the scenery’s: each detail fetch here is a multi-second download, and hydrating all forty visible rows would waste most of that work every time the user scrolls on. It listens to every subscribed generation for repaints, and renders the status bar: row count, an augmented counter (how many rows on hand are fully hydrated), your pinned values, and the key legend. r runs the Dio’s reconcile on demand, ahead of the timer.

Running it

cargo run on a cold cache is the whole system visible at once. The table appears instantly — empty. Within a second the first thousand filenames arrive; the row counter keeps climbing as the background sync streams pages, passing 122,000 a few minutes in. Meanwhile the rows around the cursor sprout numbers, one file at a time, as the detail pass works through the band — … becoming 14355 20260710 — and the augmented counter and total rows sum tick upward with each one. Move the cursor, and the band follows; jump to End, and the last stations of the alphabet get their turn. Quit, run again: everything already observed is back instantly, warm from cache.redb, and hydration resumes wherever you look next.

Notice what the application never wrote: a render loop, a fetch, an event match. It reads scenery.row(idx) and totals.value() through a binding that repaints when a generation channel says so. Nothing more passes between data and display — and it’s precisely how a real UI binds to Vantage.

What we covered

Serving Scenery — Axum & Watch Streams

Chapter 7’s client of the Dio was a terminal driving two sceneries — one showing the rows, the other aggregating a running sum. Sceneries scale well beyond that: the framework can drive many active viewers at once, and nothing requires them to live in the same process. This chapter sends them across the network — every connected browser tab becomes its own viewer, looking at its own page of the archive, with details streaming in as they land.

The traditional plumbing for live updates over HTTP is a WebSocket, server-sent events, or the style the Kubernetes API made standard — the watch: a plain HTTP response that simply never ends, delivering one JSON line per change over chunked transfer encoding. We take the Kubernetes shape: every endpoint answers a plain GET with a snapshot, and the same URL with ?watch=true keeps the connection open and streams changes for as long as the client stays. A watch is not a polling loop — it is a Scenery on the far end of an HTTP connection.

This chapter re-uses chapter 3’s Axum server, wired to the Dio through a new adapter crate — and finishes with a small React app browsing the bucket the way the AWS S3 Explorer does. Worth pausing on what the frontend gets for free: the page renders instantly from the cache, cells fill themselves in as augmentation lands, and every open tab stays current — the responsiveness chapter 7 built for the terminal, now delivered over a wire.

One flight per row

Before any of that can be safe, the framework needs an answer to a question chapter 7 never had to ask: what happens when several views drive augmentation at once?

Until now, each scenery ran its own detail fetches, inline, for its own viewport. One terminal, one viewport — fine. But two browser tabs watching overlapping pages would each download the same CSVs; two tabs on disjoint pages would race each other with no ordering at all; and a tab opened mid-download had no way to say “me next”. The fetches were nobody’s job to coordinate.

They are now the Dio’s. Every consumer that wants rows hydrated — a scenery’s viewport, a facade read blocking on its window — registers a queue with the Dio’s augment scheduler and enqueues row ids into it. A worker pool drains the queues:

• Round-robin across consumers. The worker takes one id from each queue in turn, so two views with disjoint pages interleave — neither starves behind the other’s backlog.
• One flight per row. An id already being fetched is never fetched again; every queue waiting on it is notified by the same completion. Overlapping views cost one download, not one per view.
• Closing a view withdraws its work. The queue registration is owned by the scenery; when the last handle drops, its queued ids vanish. A fetch already in the air completes and lands in the cache — paid-for work is kept.
• Workers are configurable. The default single worker keeps fetch order deterministic; Lens::new().augment_workers(4) hydrates four rows at a time when the detail source can take it.

Nothing in the example code changes for this — the scheduler sits under the same set_viewport and fetch_window calls the previous chapters used. What changes is what you can now safely do: open as many sceneries as you have HTTP connections.

One builder flag is new. Sceneries de-duplicate: two identical opens share one instance, which is right for two widgets showing the same grid — but wrong for two clients watching different pages of the same query, where each connection must keep its own viewport. The adapter opens its sceneries with .exclusive(): never shared, still counted in the demand union — the Dio hydrates the union of the columns every open scenery demands — and still released on drop.

The adapter: DioRouter

Everything HTTP lives in a new adapter crate, vantage-api-adapters — the server-side sibling of the dataset-ui-adapters crate that provided chapter 7’s ratatui binding. Where a UI adapter binds a scenery to a widget, an API adapter binds it to a route:

#![allow(unused)]
fn main() {
use vantage_api_adapters::axum_dio::DioRouter;

let api = DioRouter::new(dio.clone())
    .with_column("filename", "Key")
    .with_column("size", "Size")
    .with_column("rows", "rows")
    .with_column("latest", "latest")
    .with_page_size(50)
    .into_router();
}

Each .with_column(name, field) maps a record field to a JSON key — and doubles as the watch sceneries’ demand: naming rows and latest here is exactly what makes a watch connection drive their hydration (chapter 7’s demand gate, now per connection). into_router() yields a plain axum::Router with two routes, each in two modes:

The split embodies the demand philosophy: a plain GET is not a standing view, so it serves the Dio’s current knowledge instantly — augmented columns appear once some view has paid for them. A watch is the standing view: it declares its page as the viewport, hydration follows, and every change streams back as a Kubernetes-style NDJSON line:

{"type":"ADDED","object":{"index":3,"filename":"…","size":"4880965","rows":null,"latest":null}}
{"type":"MODIFIED","object":{"index":3,"filename":"…","size":"4880965","rows":143676,"latest":"20260531"}}

The stream diffs against what it already sent — a generation bump that changed nothing on this page costs nothing on the wire. And the scenery is owned by the response stream: when the client disconnects, the stream drops, the scenery’s guard aborts its tasks, its queued fetches are withdrawn, and its demand drains. A closed tab stops pulling — the same lifecycle rule as chapter 7’s closing grid, now enforced by HTTP.

One honesty note against the Kubernetes original: there is no resume token (no resourceVersion). A client that reconnects gets a fresh snapshot of ADDED lines and a new watch — not a replay of what it missed while away.

The server

learn-7 is learn-6 with the terminal swapped for the router — and the prefix narrowed back to chapter 5’s GM (1,122 files): a server should boot in seconds, and everything below works the same way on the full archive. The data plumbing is otherwise identical — files.rs, the augmenter, the Lens — with two differences. First, the cache is opened by hand, because we want a second table in the same redb file:

#![allow(unused)]
fn main() {
let cache = Arc::new(RedbCache::open("cache.redb").context("Failed to open cache")?);
let contents = ContentsCache::new(cache.open_table("contents").await?);
}

A Lens normally opens cache.redb itself (cache_at); cache_source(cache) hands it ours instead. The Dio claims one named table for the listing, and open_table("contents") claims another for the cache we’re about to meet.

Second, on_start stays blocking (the default — learn-6 turned it off to open its UI on an empty table). A server should answer its first request from a warm cache, so make_dio runs the chapter-5 sync to completion before axum::serve ever binds the port. On a restart the sync resumes from redb and confirms with a single request:

fetched 1000 files in 831.454208ms
fetched 122 files in 194.010875ms
serving on http://localhost:3007

The rest of main.rs is mounting:

#![allow(unused)]
fn main() {
let app = axum::Router::new()
    .nest("/api/files", api)
    .fallback_service(ServeDir::new("frontend/dist"));

axum::serve(TcpListener::bind("0.0.0.0:3007").await?, app).await
}

Concurrency needs no further code. Dio is a cheap clone over shared state, redb reads run concurrently, and no lock is held across a network await anywhere in the read path — every simultaneous request simply proceeds.

Neither does authentication — because there is none. The endpoints are anonymous, as in chapter 3: wrap the router in your tower auth middleware before exposing it; Vantage deliberately stays out of authn.

A cache that earns its keep

The detail endpoint returns everything about one file, augment columns included — which means the first GET /{id} downloads the CSV. Chapter 6 deliberately kept contents out of the cache: at 122,000 stations, storing every casually-viewed file would grow cache.redb by gigabytes. But a server sees repeat traffic — the same popular stations, again and again — and re-downloading those is just as wasteful.

ContentsCache splits the difference with lazy admission: a file must be requested twice before its contents earn a slot. It is application code, not framework — the framework contributes open_table (a named key-value table in the Dio’s own redb file); the admission policy is entirely ours to write.

#![allow(unused)]
fn main() {
pub struct ContentsCache {
    table: Arc<dyn CacheTable>,
    /// Keys downloaded at least once — the admission ledger.
    seen: Mutex<HashSet<String>>,
}
}

Its one operation is cache-first with the admission rule on the miss path:

#![allow(unused)]
fn main() {
pub async fn get_or_fetch<F, Fut>(&self, key: &str, fetch: F) -> VantageResult<String>
}

A hit is served from redb. A miss runs fetch either way — but only a key already in seen (a repeat request) gets its body written to the contents table. One-off requests leave nothing behind but their key. (The ledger itself is never trimmed — bare keys, fine at 122,000 files; a bigger keyspace would want an eviction rule, and the policy being application code makes that your call.) The augmenter’s contents lazy expression routes its download through it, and everything downstream — rows, latest, the detail endpoint — is unchanged:

#![allow(unused)]
fn main() {
.with_lazy_expression("contents", move |row| {
    // …
    async move {
        let body = contents
            .get_or_fetch(&key, || async move {
                s3::get_object(&aws, &bucket, &fetch_key).await
            })
            .await?;
        Ok(body.into())
    }
})
}

Watching it work

A plain GET is the cache, instantly — rows and latest are null because nothing has demanded them yet:

$ curl 'localhost:3007/api/files?offset=0&limit=3'
{"total":1122,"offset":0,"limit":3,"items":[
  {"index":0,"filename":"csv/by_station/GM000001153.csv","size":"4796352","rows":null,"latest":null},
  …

A detail GET blocks on the bounded facade read — 1.8 s to download and digest 4.8 MB — and the same request again is a 21 ms cache hit:

$ curl 'localhost:3007/api/files/csv%2Fby_station%2FGM000001153.csv'
{"Key":"csv/by_station/GM000001153.csv", …"Size":"4796352", …"rows":140629,"latest":"19911231"}

Now the standing view. curl -N holds the connection; the page arrives as ADDED rows, then each CSV lands as a MODIFIED line the moment its download completes:

$ curl -N 'localhost:3007/api/files?offset=3&limit=3&watch=true'
{"type":"ADDED","object":{"index":3,"filename":"csv/by_station/GM000002288.csv","size":"4880965","rows":null,"latest":null}}
{"type":"ADDED","object":{"index":4,"filename":"csv/by_station/GM000002698.csv","size":"6351302","rows":null,"latest":null}}
{"type":"ADDED","object":{"index":5,"filename":"csv/by_station/GM000002716.csv","size":"5042384","rows":null,"latest":null}}
{"type":"MODIFIED","object":{"index":3,…"rows":143676,"latest":"20260531"}}
{"type":"MODIFIED","object":{"index":4,…"rows":186088,"latest":"20081031"}}
{"type":"MODIFIED","object":{"index":5,…"rows":147011,"latest":"20250824"}}

And the scheduler is visible from outside. Two watches on disjoint pages, side by side: their MODIFIED lines alternate — B, A, B, A — the single worker taking one row from each view’s queue in turn:

19:35:35 [B] {"type":"MODIFIED","object":{"index":42,…"rows":201382,…}}
19:35:36 [A] {"type":"MODIFIED","object":{"index":40,…"rows":77595,…}}
19:35:38 [B] {"type":"MODIFIED","object":{"index":43,…"rows":132579,…}}
19:35:40 [A] {"type":"MODIFIED","object":{"index":41,…"rows":128335,…}}

Two watches on the same page: each row’s MODIFIED reaches both connections in the same second — one download, fanned out by the event bus to every scenery holding the row:

19:36:18 [B] {"type":"MODIFIED","object":{"index":50,…"rows":135802,…}}
19:36:18 [A] {"type":"MODIFIED","object":{"index":50,…"rows":135802,…}}

Run the first GET again afterwards: the page the watches paid for now answers with its numbers filled in. Current knowledge grew.

A tiny React client

The frontend is one component. It fetches its page with watch=true and reads the NDJSON stream directly off fetch — no client library, twenty lines:

const res = await fetch(`/api/files?offset=${offset}&limit=${LIMIT}&watch=true`, {
  signal: ctl.signal,
})
const reader = res.body.getReader()
// …accumulate chunks, split on '\n'…
const event = JSON.parse(line)
setRows(rs => ({ ...rs, [event.object.index]: event.object }))

ADDED and MODIFIED merge the same way — the object lands at its index — so the table renders the listing immediately and the Rows / Latest cells flip from … to numbers as the watch delivers them, exactly like chapter 7’s terminal cells did. Paging aborts the fetch via the AbortController, which closes the connection, which drops the scenery server-side. Clicking a row hits the detail endpoint into a side panel. npm run build in learn-7/frontend emits dist/, the ServeDir fallback serves it, and http://localhost:3007 is a bucket explorer: title, file count, Prev/Next, and a table filling itself in.

What we covered

Dio & Lens over SQL

This path builds a live, cached, watchable view in front of a SQL database you own. We start from a Vista over a SQLite product table — a runtime handle that can already sort, search, paginate, and write. It is a fully capable backend, which raises a fair question: if the database already does all that, what is a caching-and-events layer even for?

For the two things a capable backend doesn’t hand you on its own. A Dio keeps a live local copy of a data segment — reads answered from memory, not round-trips — and it announces every change on an event bus, so a view can watch the data instead of re-polling it. (A slow, read-only API needs a third thing from Diorama — the injection of capabilities the backend lacks, like sorting a listing it cannot sort. A SQL database needs none of that; here the live cache and the change stream are the whole story.)

We’ll build a bar’s shelf: a handful of products, each with a stock count, behind a cached, watchable API. A background “till” will sell items so the data actually moves. This chapter builds the cache; the next serves it as a live watch; the last moves the whole thing to PostgreSQL without touching a line of the model.

The full crate is learn-8.

The model

The Introduction’s Product, with one column added — stock, the units on the shelf:

#![allow(unused)]
fn main() {
// src/product.rs
use vantage_sql::prelude::*;
use vantage_types::prelude::*;

#[entity(SqliteType)]
#[derive(Debug, Clone, Default)]
pub struct Product {
    pub name: String,
    pub price: i64,
    pub stock: i64,
}

impl Product {
    pub fn table(db: SqliteDB) -> Table<SqliteDB, Product> {
        Table::new("product", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<i64>("price")
            .with_column_of::<i64>("stock")
    }
}
}

Nothing here knows it is about to be cached, watched, or served from Postgres. That is the whole bet of the four-layer model — the entity is written once and the layers above it decide what to do with it.

The master

The master is the product table as a Vista — built exactly the way the Introduction built one: hand the table to its datasource’s Vista factory.

#![allow(unused)]
fn main() {
let db = SqliteDB::connect(&url).await?;
let master = db.vista_factory().from_table(Product::table(db.clone()))?;
}

Ask this Vista what it can do and it answers yes to everything — can_order, can_fetch_window, can_insert, can_update, can_delete. Contrast that with the S3 path, where the Vista admits it cannot sort or search. There is no gap to fill here, so there is no capability injection in this path. What we still want is the cache.

The lens: eager and reactive

A slow, paginated backend like S3 is loaded a page at a time, resuming from a cursor. A SQL table needs none of that ceremony — it hands back the whole set in one call — so this path uses the eager shape of a Lens: load everything on start, then reconcile on a timer.

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_in_memory()
        .on_start(|dio| {
            let dio = dio.clone();
            async move {
                let rows = dio.master().list_values().await?;
                dio.cache().insert_values(rows).await?;
                Ok(())
            }
        })
        .on_refresh(|dio| {
            let dio = dio.clone();
            async move {
                let rows = dio.master().list_values().await?;
                dio.cache().clear().await?;
                dio.cache().insert_values(rows).await?;
                Ok(())
            }
        })
        .refresh_every(Duration::from_secs(1))
        .build()?,
);
let dio = lens.make_dio(master).await?;
}

Three builder calls carry the whole behaviour:

• cache_in_memory() — the cache is ephemeral; a restart re-reads the table. (The S3 path persists its cache with cache_at("cache.redb") because re-listing a remote bucket is expensive. Re-reading a local table is not.)
• on_start — before the server answers a single request, list_values() pulls the entire table into the cache. make_dio blocks on this, so the first read is already warm.
• on_refresh + refresh_every — once a second, the cache is rebuilt from the master. This is what makes the copy live: a sale the till commits, or an edit you type into sqlite3, is reconciled on the next tick, and a DatasetChanged event goes out to every watcher. The clear-then-reload is deliberate — a product that has sold out is simply absent from the master’s answer, so clearing first drops it cleanly.

Reads come from the cache

From here on, reads never touch SQLite. make_dio returned a Dio, and everything a consumer asks of it — a listing, a page, a single record — is answered from the in-memory cache the on_start warmed. The database is consulted only by the once-a-second reconcile, never by a reader. The next chapter puts a DioRouter over this Dio and you’ll see a plain GET return the shelf instantly, without a query reaching the table.

That is the inversion the whole guide has been climbing toward: the backend is the source of truth, but the handle you read is a local, live mirror of it.

Next: serve this Dio as a Kubernetes-style watch, and turn on the till so the shelf moves while you look at it.

Serving & Watching Live Stock

Last chapter left a Dio holding a live copy of the product table. Now we put it behind HTTP — using the same adapter the S3 path uses. DioRouter neither knows nor cares that its Dio is backed by SQL instead of S3; it binds a Dio to a pair of routes, GET and watch, and that is all it needs.

#![allow(unused)]
fn main() {
use vantage_api_adapters::axum_dio::DioRouter;

let api = DioRouter::new(dio.clone())
    .with_column("name", "name")
    .with_column("price", "price")
    .with_column("stock", "stock")
    .with_page_size(50)
    .into_router();

let app = axum::Router::new().nest("/api/products", api);
axum::serve(listener, app).await?;
}

A plain GET is the cache, answered instantly — never a query against SQLite:

$ curl 'localhost:3008/api/products?offset=0&limit=10'
{"total":5,"offset":0,"limit":10,"items":[
  {"index":0,"name":"Espresso","price":280,"stock":12},
  {"index":1,"name":"Cappuccino","price":340,"stock":8},
  {"index":2,"name":"Cold Brew","price":420,"stock":5},
  {"index":3,"name":"Croissant","price":260,"stock":6},
  {"index":4,"name":"Cheesecake","price":520,"stock":2}]}

A till to make it move

A cache over a table nobody writes is just a slow constant. To see the reactive stack work, the data has to change — so the app runs a till: a background task that sells one unit of a random product every 800ms, removes anything that sells out, and every so often takes a delivery. These are ordinary SQL writes, on the same connection the master reads from:

#![allow(unused)]
fn main() {
// src/sim.rs — the interesting line
sqlx::query(
    "UPDATE product SET stock = stock - 1
     WHERE id = (SELECT id FROM product WHERE stock > 0 ORDER BY RANDOM() LIMIT 1)",
)
.execute(db.pool())
.await?;
}

The server itself never calls the till and the till never calls the server. They meet only through the database and the once-a-second reconcile — which is precisely the point: any writer, in any process, moves the shelf. The sqlite3 CLI will do just as well, as we’ll see.

The watch

curl -N holds the connection open. The page arrives as ADDED lines — the current shelf — and then every sale streams back as a MODIFIED line the moment the reconcile picks it up:

$ curl -N 'localhost:3008/api/products?watch=true'
{"type":"ADDED","object":{"index":0,"name":"Espresso","price":280,"stock":12}}
{"type":"ADDED","object":{"index":1,"name":"Cappuccino","price":340,"stock":8}}
{"type":"ADDED","object":{"index":2,"name":"Cold Brew","price":420,"stock":5}}
{"type":"ADDED","object":{"index":3,"name":"Croissant","price":260,"stock":6}}
{"type":"ADDED","object":{"index":4,"name":"Cheesecake","price":520,"stock":2}}
{"type":"MODIFIED","object":{"index":4,"name":"Cheesecake","price":520,"stock":1}}
{"type":"MODIFIED","object":{"index":2,"name":"Cold Brew","price":420,"stock":4}}
{"type":"MODIFIED","object":{"index":0,"name":"Espresso","price":280,"stock":9}}

That is the reactive stack running over SQL, end to end: a SQL write, reconciled into the cache, a DatasetChanged event, a scenery re-derive, a diff against what this connection last saw, one NDJSON line on the wire. Nothing in that chain is specific to SQL — it is the same machinery the S3 path serves the NOAA bucket with.

It really is the database

The strongest proof that this isn’t an in-process illusion: change the table from outside the app entirely. With a watch open, in another terminal:

$ sqlite3 learn-8/products.db "UPDATE product SET price=999 WHERE id='p1'"

and the open watch stream reflects it within the second:

{"type":"MODIFIED","object":{"index":0,"name":"Espresso","price":999,"stock":11}}

No API call, no shared memory — a separate process wrote a row, the reconcile read it, and the change fanned out to every watcher. The Dio is a mirror of the database, not a copy that happens to agree with it.

Next: the same app, the same shelf, the same watch — served from PostgreSQL instead of a file, changed by flipping one compile flag.

Moving to PostgreSQL

A file on disk is a fine place for a bar to start. When it outgrows one — concurrent writers, a second instance, a backup story — you reach for a real database server. In most stacks that is a migration project. Here it is a refactor that touches one new file and two attributes, and then a compile flag decides which backend the binary speaks.

The finished crate is learn-9 — the same app as learn-8, made backend-parametric.

Name the backend once

learn-8 wrote SqliteDB in a handful of places. The refactor pulls every one of them into a single alias, so the concrete backend is named in exactly one file:

#![allow(unused)]
fn main() {
// src/db.rs — the only file that names a database
use vantage_sql::prelude::*;

#[cfg(not(feature = "pg"))]
pub type Db = vantage_sql::sqlite::SqliteDB;
#[cfg(feature = "pg")]
pub type Db = vantage_sql::postgres::PostgresDB;

pub async fn connect() -> VantageResult<Db> {
    #[cfg(not(feature = "pg"))]
    let url = format!("sqlite:{}?mode=rwc", concat!(env!("CARGO_MANIFEST_DIR"), "/products.db"));
    #[cfg(feature = "pg")]
    let url = std::env::var("DATABASE_URL")
        .map_err(|_| vantage_core::error!("DATABASE_URL must be set for the `pg` build"))?;

    Db::connect(&url).await.context("connect db")
}
}

PostgresDB::connect, db.pool(), and db.vista_factory().from_table(...) are the same calls as their SQLite twins — the two backends implement one interface — so connect() is the whole of the connection difference: a file path on one side, a DATABASE_URL on the other.

Two attributes on the model

The entity marker names whichever backend is compiled in; the table builder now takes the Db alias instead of SqliteDB. That is the entire change to the model:

#![allow(unused)]
fn main() {
// src/product.rs
#[cfg_attr(not(feature = "pg"), entity(SqliteType))]
#[cfg_attr(feature = "pg",       entity(PostgresType))]
#[derive(Debug, Clone, Default)]
pub struct Product {
    pub name: String,
    pub price: i64,
    pub stock: i64,
}

impl Product {
    pub fn table(db: Db) -> Table<Db, Product> {
        Table::new("product", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<i64>("price")
            .with_column_of::<i64>("stock")
    }
}
}

The builder body — the columns, the id — does not move. #[entity(...)] is a list of the backends a struct serves, not a choice of one; listing PostgresType alongside SqliteType just generates a second set of conversions. The same Product could serve six backends at once, as bakery_model3 does.

Everything else — main.rs, sim.rs, the lens, the Dio, the DioRouter — is unchanged. It was written against Db and the portable data-set methods, so it never named a backend to begin with. The reconcile still calls master().list_values(); the till still runs the same UPDATE … ORDER BY RANDOM(); the watch still streams ADDED/MODIFIED.

Run it on Postgres

Start a local Postgres (the repo ships the script), point DATABASE_URL at it, and build with the flag:

$ vantage-sql/scripts/postgres/start.sh          # docker: postgres:17-alpine on :5433
$ export DATABASE_URL='postgres://vantage:vantage@localhost:5433/vantage'
$ cargo run -p learn-9 --no-default-features --features pg
serving on http://localhost:3009 — try: curl -N 'localhost:3009/api/products?watch=true'

The shelf serves and streams exactly as it did on SQLite — same routes, same NDJSON, same till:

$ curl -N 'localhost:3009/api/products?watch=true'
{"type":"ADDED","object":{"index":0,"name":"Espresso","price":280,"stock":12}}
{"type":"ADDED","object":{"index":1,"name":"Cappuccino","price":340,"stock":8}}
...
{"type":"MODIFIED","object":{"index":4,"name":"Cheesecake","price":520,"stock":1}}

And the it-really-is-the-database proof works over Postgres too — edit a row with psql, watch it land:

$ psql "$DATABASE_URL" -c "UPDATE product SET price=777 WHERE id='p2'"
{"type":"MODIFIED","object":{"index":1,"name":"Cappuccino","price":777,"stock":6}}

What this proves

The reactive stack — cache, events, sceneries, watch server — ran unchanged over a file-backed SQLite database and a networked PostgreSQL server, and the code that moved between them was one alias and two attributes. This is not a toy path: the repo’s tests/postgres/6_vista.rs exercises a Vista over a real Postgres, and the launch-control example deploys the identical model to AWS Lambda against Aurora Postgres while defaulting to SQLite locally.

There is still one thing left on the table. The app reconciles on a one-second timer — it re-reads the table whether or not anything changed, and it is always up to a beat behind. A file on disk can’t do better; a server can. Next, we commit to Postgres and trade that poll for real-time push: the database tells us the moment a row changes.

Real-Time Push with LISTEN/NOTIFY

The app so far reconciles on a timer — Lens::refresh_every(Duration::from_secs(1)). That is honest polling: once a second the Dio re-reads the master and diffs. It works, but it is a beat behind every change and it re-queries even when nothing moved. A file-backed SQLite database has no better option — it can’t call you back. A PostgreSQL server can: it can tell you the instant a row changes, and then the poll disappears.

This chapter does exactly that. Having proven the app is portable, we commit to Postgres — drop the feature flag, delete sim.rs, and split the writer into its own process — and replace the refresh timer with a single dio.watch(). The finished crate is learn-10.

Teach the database to announce changes

Postgres LISTEN/NOTIFY is a publish/subscribe channel built into the server. A trigger fires pg_notify on every write; anyone LISTENing on that channel wakes up. So the only schema we add beyond the table is a trigger:

#![allow(unused)]
fn main() {
// src/db.rs — after CREATE TABLE product (...)
sqlx::query(
    "CREATE OR REPLACE FUNCTION product_notify() RETURNS trigger AS $$
     BEGIN PERFORM pg_notify('product_changed', ''); RETURN NULL; END;
     $$ LANGUAGE plpgsql",
).execute(db.pool()).await?;

sqlx::query(
    "CREATE TRIGGER product_notify_trg
     AFTER INSERT OR UPDATE OR DELETE ON product
     FOR EACH STATEMENT EXECUTE FUNCTION product_notify()",
).execute(db.pool()).await?;
}

FOR EACH STATEMENT fires the notify once per statement, not per row — a single “something changed” ping is all the Dio needs to reconcile. The payload is empty on purpose (more on that below).

Tell the Vista the trigger exists

Postgres offers no way to ask “is anything going to notify me on this channel?” — LISTEN on a channel nobody feeds succeeds and then waits forever. A Vista that assumed it could push just because the table is writable would advertise a feed that may never arrive, and no consumer could tell the difference between that and a quiet table. So the application, which is the only party that knows, says so:

#![allow(unused)]
fn main() {
let mut master = db
    .vista_factory()
    .with_notify(true)   // we installed product_notify_trg above
    .from_table(Product::table(db.clone()))?;
}

That one call is what makes can_watch() answer true in the next section. Leave it off and dio.watch() becomes a no-op — which matters here, because this chapter is about to delete the refresh timer. Push and a refresh_every poll are the two ways the cache learns about a write; an app that declines both never updates.

One line replaces the poll

The server builds the Dio exactly as before — an eager in-memory cache, an on_start that loads it and an on_refresh that reloads it — but with no refresh_every. Instead:

#![allow(unused)]
fn main() {
let dio = lens.make_dio(master).await?;

// The transparent live feed. The master Vista advertises `can_watch`, so this
// subscribes over LISTEN/NOTIFY and reconciles the instant a write lands — no
// polling timer.
dio.watch().await?;
}

That is the whole change. Dio::watch() asks the master Vista whether it can push (Vista::can_watch()); because we opted in above, the Postgres Vista answers yes and opens a PgListener on the product_changed channel. Each notification drives one reconcile. If the backend couldn’t push, watch() would be a no-op and a refresh_every timer would carry the load instead — the same call is correct either way, which is the point of the capability.

The till becomes its own process

To drive it, learn-10 ships a second binary — the mutator — run alongside the server:

$ cargo run -p learn-10                 # the reactive server, port 3010
$ cargo run -p learn-10 --bin mutator   # a separate writer process

The mutator writes through Vantage’s active-entity API — no hand-written SQL. list_entities() hands back drinks that each carry their own id and datasource, so selling one is just:

#![allow(unused)]
fn main() {
let mut shelf = table.list_entities().await?;
let drink = shelf.choose_mut(&mut rand::thread_rng()).unwrap();
if drink.stock <= 1 {
    drink.delete().await?;        // last unit — off the shelf
} else {
    drink.stock -= 1;
    drink.save().await?;          // one sale
}
}

Two independent processes that share nothing but the database and the notify channel: the strongest possible proof that the reactive stack tracks the data, not some in-process shortcut.

Removals, and a UI to see them

This app also keys its watch by identity, so a sold-out drink is reported as a real removal rather than a silent list-shrink:

#![allow(unused)]
fn main() {
let api = DioRouter::new(dio.clone())
    .with_column("id", "id")
    .with_column("name", "name")
    .with_column("price", "price")
    .with_column("stock", "stock")
    .key_by("id")            // identity-keyed: emits DELETED when a row leaves
    .with_page_size(50)
    .into_router();
}

With key_by("id"), the watch stream gains a third event — DELETED — alongside ADDED and MODIFIED. A small React page (frontend/index.html, served by the same axum app) consumes the stream and animates each drink: arriving with a flash, ticking down as it sells, and disposing with a “sold out” stamp when a DELETED lands. It is plain React over the watch endpoint — not part of Vantage — and, notably, the exact same file the next chapter reuses unchanged.

Watch it on the wire:

$ curl -N 'localhost:3010/api/products?watch=true'
{"type":"ADDED","object":{"index":0,"id":"d51...","name":"Mojito","price":1000,"stock":11}}
{"type":"ADDED","object":{"index":1,"id":"d59...","name":"Boulevardier","price":1400,"stock":12}}
{"type":"MODIFIED","object":{"index":0,"id":"d51...","name":"Mojito","price":1000,"stock":10}}
{"type":"DELETED","object":{"index":0,"id":"d59...","name":"Boulevardier","price":1400,"stock":1}}

Every line is a real database write, announced by Postgres, reconciled into the cache, diffed by the scenery, and fanned out — with no timer anywhere in the loop.

Next: the same app, the same frontend, a different database — SurrealDB, whose native live queries push the actual change rather than a ping.

Switching to SurrealDB

Postgres was a migration within SQL — a different server, the same dialect family. SurrealDB is a different database entirely: multi-model, schemaless by default, and reached over a WebSocket rather than a pooled SQL connection. Adopting it is the real test of the claim the whole path has been making. The goal here is to change as little as possible — and then to cash in the one thing SurrealDB does better than a SQL server: it can push the actual change, not just a ping.

The finished crate is learn-11. Set it next to learn-10 and the diff is small and boring — which is the result we want.

What changes

Three touch-points name the backend, and nothing else does.

The entity marker — swap the backend in the attribute list:

#![allow(unused)]
fn main() {
#[entity(SurrealType)]           // was: #[entity(PostgresType)]
#[derive(Debug, Clone, Default)]
pub struct Product {
    pub name: String,
    pub price: i64,
    pub stock: i64,
    pub created: i64,
}
}

The table builder — the same columns, built against SurrealDB; SurrealDB’s native id is a Thing (product:⟨key⟩):

#![allow(unused)]
fn main() {
pub fn surreal_table(db: SurrealDB) -> Table<SurrealDB, Product> {
    Table::new("product", db)
        .with_id_column("id")
        .with_column_of::<String>("name")
        .with_column_of::<i64>("price")
        .with_column_of::<i64>("stock")
        .with_column_of::<i64>("created")
}
}

Connect and setup — a DSN instead of a DATABASE_URL, and — notably — no trigger. Where Postgres needed a pg_notify function and an AFTER … FOR EACH STATEMENT trigger, SurrealDB emits change frames on its own. Setup is just making sure the table exists:

#![allow(unused)]
fn main() {
let client = SurrealConnection::dsn(&dsn)?.connect().await?;
// The one bit of schema. SurrealDB streams changes natively via LIVE SELECT,
// so there is nothing else to wire.
client.query("DEFINE TABLE IF NOT EXISTS product SCHEMALESS", None).await?;
let db = SurrealDB::new(client);
}

The till (the mutator) is the same active-entity code as before; only the id it hands new_entity is a Thing:

#![allow(unused)]
fn main() {
table.new_entity(
    Thing::new("product", key),          // was: a String id
    Product { name, price, stock, created },
).save().await?;
}

What doesn’t change

The server’s reactive core is untouched. The Vista comes from SurrealVistaFactory instead of db.vista_factory(), and then every line is identical to the Postgres chapter — including the one that matters:

#![allow(unused)]
fn main() {
let mut master = SurrealVistaFactory::new(db.clone())
    .from_table(Product::surreal_table(db.clone()))?;
master.add_order("created", SortDirection::Ascending)?;

let dio = lens.make_dio(master).await?;   // same Lens, same cache, same on_start/on_refresh
dio.watch().await?;                        // the same call — now backed by LIVE SELECT
}

And the frontend is byte-for-byte the same file as learn-10. The React page consumes ?watch=true and animates the shelf; it never learns which database is behind the stream. Diff the two frontend/index.html files and you get nothing — the clearest statement the guide can make that the reactive layer, up to and including the UI, does not depend on the backend.

The payoff: precise push, no re-query

dio.watch() looks the same, but underneath it is doing something better than it could on Postgres. SurrealDB’s LIVE SELECT delivers a typed, per-row notification — the action (CREATE / UPDATE / DELETE) and the affected record — for every change. So the Dio applies each change directly to its cache: a create inserts one row, an update repaints one row, a delete removes one row. There is no “something changed, re-read everything” step, because the change already arrived in full.

$ curl -N 'localhost:3011/api/products?watch=true'
{"type":"ADDED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d51..."]]},"name":"Mojito","price":1000,"stock":11}}
{"type":"MODIFIED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d51..."]]},"name":"Mojito","price":1000,"stock":10}}
{"type":"DELETED","object":{"index":0,"id":{"@@TAGGED@@":[8,["product","d59..."]]},"name":"Boulevardier","price":1400,"stock":1}}

(The id arrives as a SurrealDB Thing rather than a bare string — the frontend’s idOf helper normalises either shape to a stable key, which is why the same UI serves both backends.)

From here the reference half of the book takes over — SurrealDB for the driver details, SQL for the dialects, Config-Driven Vistas for declaring all of this from YAML, Live Data for what every backend does about freshness, and Adding a New Persistence when your backend isn’t one Vantage already speaks.

The Vantage Journey

Vantage didn’t start as a multi-backend entity framework. It started as a weekend experiment to see if Rust could build SQL queries without feeling like Rust.

This page walks you through each release — what changed, why it mattered, and how the API evolved from raw Postgres queries to a universal persistence layer.

• 0.0 — “DORM” (April–November 2024)
  • Milestones
• 0.1 — Vantage is born (December 2024)
• 0.2 — Entity Framework & MDA (February 2025)
• 0.3 — The Great Separation (July–October 2025)
• 0.4 — The Type System Rewrite (November 2025–April 2026)
• 0.5 — The Vista Era (May–June 2026)
  • Capabilities — the explicit contract
  • Config-driven: YAML specs and Rhai scripting
  • Contained relations and nested writes
  • Diorama — caching, events, and reactive views
  • More backends, more reach
• 0.6 — Hardening (June 2026)
• The bigger picture

graph LR
    D[0.0 DORM] -->|rename| V1[0.1 Vantage]
    V1 -->|entity framework| V2[0.2 MDA]
    V2 -->|crate split| V3[0.3 UI Adapters]
    V3 -->|type rewrite| V4[0.4 Type System]
    V4 -->|universal handle| V5[0.5 Vista]
    V5 -->|hardening| V6[0.6 Robustness]
    style D fill:#555,color:#fff
    style V1 fill:#4a7c59,color:#fff
    style V2 fill:#2d6a8f,color:#fff
    style V3 fill:#8f5a2d,color:#fff
    style V4 fill:#7c2d8f,color:#fff
    style V5 fill:#2d8f6a,color:#fff
    style V6 fill:#8f2d4a,color:#fff

0.0 — “DORM” (April–November 2024)

The project was originally called DORM — the Dry ORM. It was Postgres-only, monolithic, and proudly opinionated. Everything lived in one crate.

The core idea was already there: Data Sets. Instead of loading records eagerly, you describe what you want and let the framework figure out the query.

#![allow(unused)]
fn main() {
let clients = Client::table();           // Table<Postgres, Client>
let paying = clients.with_condition(
    clients.is_paying_client().eq(&true)
);
let orders = paying.ref_orders();         // Table<Postgres, Order>

for order in orders.get().await? {
    println!("#{} total: ${:.2}", order.id, order.total as f64 / 100.0);
}
}

Behind this innocent-looking code, DORM generated a single SQL query with subqueries, joins, and soft-delete filters — all derived from model definitions:

SELECT id,
    (SELECT name FROM client WHERE client.id = ord.client_id) AS client_name,
    (SELECT SUM((SELECT price FROM product WHERE id = product_id) * quantity)
     FROM order_line WHERE order_line.order_id = ord.id) AS total
FROM ord
WHERE client_id IN (SELECT id FROM client WHERE is_paying_client = true)
  AND is_deleted = false

No hand-written SQL. No query strings. The framework composed everything from relationship definitions and table extensions like SoftDelete.

Milestones

0.1 — Vantage is born (December 2024)

On December 12, the framework was renamed from DORM to Vantage and published to crates.io for the first time. The API stayed the same — this was a branding milestone, not an architectural one.

The same month brought the first Axum integration (bakery_api), proving that data sets could drive REST endpoints naturally:

#![allow(unused)]
fn main() {
async fn list_orders(
    client: axum::extract::Query<OrderRequest>,
    pager: axum::extract::Query<Pagination>,
) -> impl IntoResponse {
    let orders = Client::table()
        .with_id(client.client_id.into())
        .ref_orders();

    let mut query = orders.query();
    query.add_limit(Some(pager.per_page));
    Json(query.get().await.unwrap())
}
}

Tags v0.1.0 and v0.1.1 were published the same day. SQLx data source landed 11 days later.

0.2 — Entity Framework & MDA (February 2025)

Version 0.2 repositioned Vantage as a full Entity Framework with Model-Driven Architecture. The README doubled in size. The vision expanded from “clever query builder” to “how enterprises should structure business logic.”

The key insight: entities aren’t just database rows. They’re business objects that might live in SQL, NoSQL, a REST API, or a message queue — and your code shouldn’t care which.

#![allow(unused)]
fn main() {
impl Client {
    fn table() -> Table<Client, Oracle> { /* ... */ }
    fn registration_queue() -> impl Insertable<Client> { /* Kafka */ }
    fn admin_api() -> impl DataSet<Client> { /* REST */ }
    fn read_csv(file: String) -> impl ReadableDataSet<Client> { /* CSV */ }
}
}

This release also introduced the idea of struct projection — using different Rust types against the same data set to control which fields get queried:

#![allow(unused)]
fn main() {
struct MiniClient { name: String }
struct FullClient { name: String, email: String, balance: Decimal }

// Only fetches `name` from the database
let name = clients.get_id_as::<MiniClient>(42).await?.name;
}

The monolith was getting heavy, though. Everything still lived in one crate, and adding a new database meant touching core code.

0.3 — The Great Separation (July–October 2025)

Version 0.3 broke the monolith into dedicated crates and bet heavily on SurrealDB as the primary backend. The trait-based architecture that defines Vantage today was born here.

Table definitions moved from static initialization to a builder pattern:

#![allow(unused)]
fn main() {
// 0.2 — static, Postgres-only
Table::new_with_entity("bakery", postgres())
    .with_id_column("id")
    .with_column("name")
    .with_many("clients", "bakery_id", || Box::new(Client::table()))

// 0.3 — builder, any datasource
Table::<SurrealDB, Client>::new("client", ds.clone())
    .with_id_column("id")
    .with_column("name")
    .with_column("email")
    .with_many("orders", "client_id", || Client::order_table())
}

Field accessors now return Expression instead of column objects — making them composable across query builders:

#![allow(unused)]
fn main() {
// Build conditions from expressions
let active = clients.is_paying_client().eq(true);
let big_spenders = clients.balance().gt(1000);
let query = clients.with_condition(active).with_condition(big_spenders);
}

The AnyTable type-erasure system arrived, enabling generic code that works with any datasource:

#![allow(unused)]
fn main() {
let tables: Vec<AnyTable> = vec![
    AnyTable::new(Client::table()),   // SurrealDB
    AnyTable::new(Product::table()),  // SQLite
];

for table in &tables {
    println!("{}: {} records", table.name(), table.count().await?);
}
}

This release culminated with UI adapters for six frameworks — egui, GPUI, Slint, Tauri, Ratatui, and Cursive — all driven by the same AnyTable interface.

0.4 — The Type System Rewrite (November 2025–April 2026)

0.5 — The Vista Era (May–June 2026)

Version 0.4 made the type system precise. Version 0.5 asks the next question: how does code that doesn’t know your entity type talk to your data? A CLI that lists any table, a web admin that draws forms from a YAML schema, a UI data grid pointed at whatever you give it — none of them know your Product struct.

0.4’s answer was AnyTable: type-erasure that funnelled everything through JSON. 0.5 replaces it with Vista — a universal, schema-bearing data handle. A Vista wraps any typed Table<DB, E>, erases both backend and entity, and carries its own schema (columns, references, capabilities) while delegating execution to a per-driver TableShell.

#![allow(unused)]
fn main() {
// Wrap any typed table — SQLite, MongoDB, AWS, all the same shape
let vista = SqliteVistaFactory::new(db).from_table(Product::table(db.clone()))?;

// Everything is now runtime introspection — no generics, no entity type
for name in vista.get_column_names() {
    let col = vista.get_column(name).unwrap();
    println!("{}: {}", col.name, col.original_type);
}

let mut v = vista.clone();
v.add_condition_eq("category_id", 1.into())?;     // driver translates to native condition
v.add_search("tart")?;                            // fans across SEARCHABLE columns
v.add_order("price", SortDirection::Descending)?;
let rows = v.fetch_page(1).await?;
}

The AnyTable carrier was deleted outright in vantage-table 0.5.2 — type erasure now lives one layer up, in Vista. That decommission is what the 0.5 version bump marks.

Capabilities — the explicit contract

Not every backend can do everything. A CSV file can’t sort server-side; DynamoDB can only order by its sort key; a REST API may paginate by cursor but not by page number. Vista makes this explicit with a struct of capability flags, and each driver declares exactly what it supports:

#![allow(unused)]
fn main() {
let caps = vista.capabilities();
if caps.can_search { v.add_search("query")?; }
if caps.can_fetch_page { /* random-access pager */ } else { /* load-more button */ }
}

Calling a method the driver doesn’t advertise returns an Unsupported error — never a silent match-all, never a panic. UI adapters branch on these flags to decide which controls to render.

Config-driven: YAML specs and Rhai scripting

A Vista no longer needs a hand-written Rust definition. Tables, columns, and relations can be declared in a YAML spec (VistaSpec) and loaded by the driver’s factory. For anything YAML can’t express — vendor-specific expressions, derived sources, scripted reference traversal — there’s an optional Rhai DSL that compiles to native queries:

#![allow(unused)]
fn main() {
let users = table("users").alias("u");
select()
    .from(users)
    .expression(users["name"])
    .where(users["age"] >= 18)
    .order_by(users["name"], "asc")
}

The same script renders dialect-correct SQL for SQLite, Postgres, and MySQL — automatic identifier quoting, date_format() mapping to strftime/TO_CHAR/DATE_FORMAT, group_concat() becoming GROUP_CONCAT/STRING_AGG. SurrealDB gets its own vocabulary (graph traversal, RELATE, record ids). YAML stays the primary format; Rhai is the targeted, serializable escape hatch.

Contained relations and nested writes

Embedded objects and arrays — an order’s lines, a row’s JSON column — now surface as a fully editable sub-Vista. SurrealDB backs them with native nested objects; SQL backends store them as JSON columns and patch the host column on writeback. And insert_value learned to walk relations: hand it a record whose keys name a relation and it sequences the writes, stamping foreign keys automatically.

Diorama — caching, events, and reactive views

The biggest new subsystem is vantage-diorama — a layer that sits between a Vista and whatever consumes it. Three concepts:

• Lens — long-lived shared infrastructure: a cache backend, lifecycle callbacks, refresh policy. Built once per application.
• Dio — a Vista bound to a Lens. Owns the cache table, a write queue, an event bus, and a refresh task.
• Scenery — a reactive view onto a Dio (ordered tables, individual records, aggregates) that a UI binds to.

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_at("./cache.redb")
        .on_start(|dio| { let dio = dio.clone(); async move {
            let rows = dio.master().list_values().await?;
            dio.cache().insert_values(rows).await?;
            Ok(())
        }})
        .refresh_every(Duration::from_secs(300))
        .build()?,
);
let products = lens.make_dio(products_vista).await?;
let mut v = products.vista();          // facade: reads from cache, writes through the queue
}

More backends, more reach

0.5 also widened the backend roster around the Vista abstraction: redb (embedded key-value store), AWS (DynamoDB and friends, all returning Vistas directly), a Cmd source that turns a shell script into a queryable table, and an API pool. Cross-persistence reference traversal — categories in Postgres, products in MongoDB — moved out of Vista and into vantage-vista-factory’s VistaCatalog, keeping each Vista strictly single-backend.

0.6 — Hardening (June 2026)

If 0.5 was about reach, 0.6 is about trust. The release is a coordinated, workspace-wide sweep with a single theme: no panics, no silent failures. Every path that could abort the process deep inside a write, or quietly return a wrong answer, was made fallible and explicit.

• Serialization is fallible. A new TryIntoRecord trait replaces the infallible blanket: a failing Serialize (non-string map keys, an out-of-range number) now becomes a recoverable error instead of a process abort mid-write. Entity write paths (insert / replace / patch) propagate it.
• No silent zeros. get_count unwraps the common [{"count": N}] shape and surfaces unrecognized result shapes as errors instead of returning 0.
• No silent match-all. Search on a backend that can’t search server-side (CSV, redb) returns an Unsupported error — never a quietly unfiltered result set.
• Expression safety. f64::IntoValue degrades NaN/Infinity to Null instead of panicking; Expression::preview() uses single-pass interleaving so a {} inside a rendered value can’t corrupt the output. select_column became fallible.
• Clearer error classification. vantage-core’s error-kind annotators were renamed to mark_unsupported / mark_unimplemented (the is_* names are now real predicates), and trace emission was decoupled — mark_* no longer logs implicitly; chain .traced() when you want the tracing::error! event.
• Security: SurrealDB identifier injection. Identifier escaping was unified into surreal-client’s single escape_identifier authority and fixed for SurrealDB 3.x — a crafted ⟩ inside a column name could previously break out of ⟨…⟩ quoting and inject arbitrary SurrealQL. similarity() and time_group() literals are now bound as parameters rather than interpolated, closing the same hole on the Rhai-exposed search path.

Alongside the hardening, 0.6 folded the standalone vantage-data-script crate into vantage-vista’s rhai feature, added random-access windowing (fetch_window) with REST lazy-loading, and introduced ColumnFlag::Label for status-tag display hints.

The bigger picture

Looking at the trajectory:

What started as 16 commits in April 2024 is now 800+ commits, 300+ pull requests, and a framework that can drive the same business logic across a dozen backends and six UI frameworks — typed when you know the entity, universal (via Vista) when you don’t.

The destination hasn’t changed since day one: describe your data once, use it everywhere. Each version just made “everywhere” a little bigger.

What’s New in Vantage 0.4

Version 0.4 is a ground-up rewrite of the type system and a massive expansion of persistence support. If 0.3 proved the architecture, 0.4 makes it production-ready.

• Progressive model — implement only what you need
• Persistence-specific Type Systems
• Table — the interface you get for free
• ActiveEntity and ActiveRecord
• Record<V> — persistence-native value bags
• CBOR everywhere
• New persistences
• Unified error handling
• AnyTable goes universal
• What’s still coming

Progressive model — implement only what you need

In earlier versions, adding a new backend meant implementing everything. Version 0.4 flips this: capabilities unlock progressively based on which traits your persistence (backend) implements. There are two parallel trait hierarchies — one for typed entities, one for schema-less records.

DataSet works with your Rust structs:

#![allow(unused)]
fn main() {
// A CSV file is read-only — and that's fine
impl ReadableDataSet<Product> for CsvFile { /* ... */ }

// A message queue can only append
impl InsertableDataSet<Event> for KafkaTopic { /* ... */ }

// A full database gets the complete toolkit
impl ReadableDataSet<Client> for SurrealDB { /* ... */ }
impl WritableDataSet<Client> for SurrealDB { /* ... */ }
// ReadableDataSet + WritableDataSet automatically unlocks ActiveEntitySet!
}

ValueSet is the schema-less counterpart — same hierarchy, but operates on Record<Value> instead of entities. Useful for dynamic tables, config-driven schemas, or when you don’t have (or want) a struct:

#![allow(unused)]
fn main() {
// Read raw records — no struct needed
impl ReadableValueSet for CsvFile { /* returns Record<CsvType> */ }

// Full CRUD on raw values
impl WritableValueSet for SurrealDB { /* accepts Record<AnySurrealType> */ }
// ReadableValueSet + WritableValueSet automatically unlocks ActiveRecordSet!
}

Each layer is opt-in. A CSV persistence never pretends it can build select queries.

The DataSource layer is progressive too. These traits aren’t a strict hierarchy — they combine independently, and each persistence implements only what its engine supports:

DataSource                        ← marker, all backends
 ├── TableSource                  ← CRUD, columns, conditions
 ├── ExprDataSource               ← execute expressions, defer()
 └── SelectableDataSource         ← query builder (Selectable)

TableExprSource                   ← count/sum/max/min as composable expressions
 requires: TableSource + ExprDataSource

TableQuerySource                  ← table definition → full query
 requires: TableSource + SelectableDataSource

Here’s what each persistence implements:

CSV
 └── TableSource (read-only, in-memory conditions)

REST API  (vantage-api-client)
 ├── TableSource (read-only, HTTP GET)
 └── ExprDataSource (basic resolve)

MongoDB
 ├── TableSource (full CRUD, native bson::Document conditions)
 └── SelectableDataSource (field projection, $match pipeline)
     └── aggregates via native $sum/$max/$min pipeline

Postgres / MySQL / SQLite  (via sqlx)
 ├── TableSource (full CRUD)
 ├── ExprDataSource (parametric SQL execution)
 ├── SelectableDataSource (JOINs, CTEs, window functions)
 └── TableQuerySource

SurrealDB
 ├── TableSource (full CRUD)
 ├── ExprDataSource (CBOR execution)
 ├── SelectableDataSource (SurrealQL builder)
 ├── TableExprSource (composable aggregates)
 ├── TableQuerySource
 └── vendor extensions (graph queries, RELATE, live queries)

There’s also MockTableSource in vantage-table and ImDataSource in vantage-dataset — a fully in-memory persistence backed by HashMap<String, IndexMap<String, Record<Value>>>. Great for tests and prototyping without any database.

Persistence-specific Type Systems

The vantage-types crate is the foundation of 0.4. Each persistence defines its own type universe via the vantage_type_system! macro — no more funnelling everything through JSON:

#![allow(unused)]
fn main() {
vantage_type_system!(AnySurrealType, SurrealDB);   // CBOR-native types
vantage_type_system!(AnyMongoType, MongoDB);        // BSON-native types
vantage_type_system!(AnySqliteType, SqliteDB);      // CBOR with integer/real/text markers
vantage_type_system!(AnyPostgresType, PostgresDB);  // CBOR with full PG type coverage
vantage_type_system!(AnyMysqlType, MysqlDB);        // CBOR with MySQL type coverage
}

All SQL backends (SQLite, Postgres, MySQL) now use CBOR as their internal value representation — not JSON. This preserves type fidelity that JSON loses (integer vs float, binary blobs, precise decimals) while keeping the same vantage_type_system! ergonomics.

Strict conversions replace silent casting — try_into() is explicit and fallible:

#![allow(unused)]
fn main() {
let val: i64 = surreal_value.try_into()?;   // fails if it's actually a string
let val: String = mongo_value.try_into()?;   // fails if it's actually a number
}

These types flow through the entire framework, not just storage. Columns carry their type as a generic parameter (Column<i64>, Column<String>) — surviving type-erasure via original_type so UI adapters can inspect what a column actually holds. AssociatedExpression<'a, DS, T, R> is parameterised by the value type T, so a count query on SurrealDB returns AnySurrealType while the same query on Postgres returns AnyPostgresType. AssociatedQueryable<R> then converts that into your expected Rust type — all type-checked at compile time.

#![allow(unused)]
fn main() {
// Column preserves type through erasure
let price = Column::<i64>::new("price").with_flag(ColumnFlag::Indexed);
let erased = Column::<AnyType>::from_column(price);
assert_eq!(erased.get_type(), "i64");  // original type survives

// AssociatedExpression carries the persistence's value type
let count: AssociatedExpression<'_, SurrealDB, AnySurrealType, usize> =
    table.get_expr_count();
let n: usize = count.get().await?;  // executes, converts AnySurrealType → usize
}

This means the full range of your persistence’s native types is preserved end-to-end — from column definition through query building to result extraction — without narrowing down to JSON variants.

Table — the interface you get for free

Once a persistence implements TableSource, the framework hands you Table<DB, Entity> — a fully-featured abstraction over your data with columns, conditions, ordering, pagination, relationships, and aggregates. You don’t build any of this yourself; it comes from vantage-table.

Table<DB, E> auto-implements a wide range of traits from the DataSet and ValueSet hierarchies: ReadableDataSet<E>, WritableDataSet<E>, InsertableDataSet<E>, ActiveEntitySet<E>, ReadableValueSet, WritableValueSet, InsertableValueSet, and ActiveRecordSet. All of these come for free once your persistence implements TableSource.

#![allow(unused)]
fn main() {
let products = Product::table(db)
    .with_condition(products["is_deleted"].eq(false))
    .with_order(products["price"].desc())
    .with_pagination(Pagination::ipp(25));

// ReadableDataSet, WritableDataSet, ActiveEntitySet — all auto-implemented
for (id, product) in products.list().await? { /* ... */ }
let count = products.get_count().await?;
}

Additionally, table.select() yields a vendor-specific query builder — SurrealSelect, SqliteSelect, PostgresSelect — giving you full access to the persistence’s native query capabilities when you need to go beyond what the generic Table API offers.

This dramatically simplifies implementing new persistences. You implement TableSource methods (read, write, aggregate) and get the entire Table API for free — conditions compose automatically, columns carry types, references traverse between tables, and AnyTable type-erasure just works.

graph LR
    P[Select Persistence]
    M[Create Model Layer]
    B[Build Business Logic]
    E[Export via FFI or API]
    P ==> M ==> B ==> E
    style P fill:#4a7c59,color:#fff
    style M fill:#2d6a8f,color:#fff
    style B fill:#8f5a2d,color:#fff
    style E fill:#7c2d8f,color:#fff

Pick your persistence (SurrealDB, Postgres, CSV — or several). Define entities and tables in a shared model crate. Build business logic against Table and DataSet traits — persistence-agnostic and testable with mocks. Finally, expose your model to other languages via FFI (C ABI, PyO3, UniFFI, WASM) or through API layers (Axum, gRPC). The model stays in Rust; consumers don’t need to know. Read more in Model-Driven Architecture and Three Paths for Developers.

ActiveEntity and ActiveRecord

The vantage-dataset crate introduces two flavours of the active record pattern:

ActiveEntity<D, E> — wraps a typed entity. Derefs straight to your struct, tracks the ID, and saves back to any WritableDataSet:

#![allow(unused)]
fn main() {
let mut user = users.get_entity(&id).await?.unwrap();
user.email = "new@example.com".to_string();  // modify via DerefMut
user.save().await?;                           // persists the change
}

ActiveRecord<D> — same idea, but schema-less. Works with Record<Value> instead of a concrete struct, perfect for dynamic tables or config-driven entities:

#![allow(unused)]
fn main() {
let mut rec = table.get_value_record(&id).await?;
rec["status"] = json!("active");
rec.save().await?;
}

Both auto-unlock via blanket impls — if your datasource implements ReadableDataSet + WritableDataSet, you get ActiveEntitySet for free. No extra code.

#![allow(unused)]
fn main() {
let mut user = users.get_entity(&id).await?
    .unwrap_or_else(|| users.new_entity(id, User::default()));
user.name = "Alice".into();
user.save().await?;
}

Record<V> — persistence-native value bags

Record<V> is an IndexMap<String, V> wrapper — but V is not serde_json::Value. It’s your persistence’s own type. A SurrealDB table returns Record<AnySurrealType>, Postgres returns Record<AnyPostgresType>, MongoDB returns Record<AnyMongoType>. This ensures the full range of your persistence’s native types is preserved — not narrowed down to what JSON can represent.

#![allow(unused)]
fn main() {
// Each persistence speaks its native types
let record: Record<AnySurrealType> = surreal_table.get_value(&id).await?;
let record: Record<AnyPostgresType> = pg_table.get_value(&id).await?;
let record: Record<AnyMongoType> = mongo_table.get_value(&id).await?;

// Type-safe extraction — respects the persistence's type boundaries
let name: String = record["name"].try_get::<String>().unwrap();
}

Record is the common currency across the framework — ReadableValueSet, WritableValueSet, ActiveRecord, and entity conversion all work through it. JSON conversion only happens at the boundary when you need AnyTable type-erasure (via from_table()).

CBOR everywhere

The SurrealDB client switched from JSON to binary CBOR, improving serialization performance and type fidelity. All backends followed — Postgres, MySQL, SQLite.

New persistences

MongoDB, CSV, and REST API persistences joined SurrealDB and SQLite. CSV evaluates conditions in-memory. MongoDB uses native BSON filters — no expression translation needed:

#![allow(unused)]
fn main() {
// Same handle_commands function — different persistences
handle_commands(SurrealDB::table("product")).await?;
handle_commands(CsvFile::table("products.csv")).await?;
handle_commands(MongoDB::table("product")).await?;
}

Unified error handling

vantage-core introduced VantageError with structured context — replacing the patchwork of Box<dyn Error> and .unwrap() calls:

#![allow(unused)]
fn main() {
use vantage_core::{error, util::error::Context, Result};

connection.connect()
    .with_context(|| error!("Failed to connect", dsn = &dsn, timeout = 30))?;
}

AnyTable goes universal

AnyTable::from_table() now wraps any datasource whose types convert to/from JSON, using an internal JsonAdapter for on-the-fly conversion. This means your generic code works with persistences that don’t even share a value type:

#![allow(unused)]
fn main() {
// Wrap a MongoDB table for use in generic code
let any: AnyTable = AnyTable::from_table(mongo_products);
let any: AnyTable = AnyTable::from_table(csv_products);
// Both work through the same interface
}

What’s still coming

What’s New in Vantage 0.5

Version 0.4 made the type system precise — every backend carries its own types, and a persistence implements only what its engine supports. Version 0.5 answers the question that follows: how does code that doesn’t know your entity type talk to your data?

A CLI that lists any table, a web admin that draws forms from a YAML schema, a UI data grid pointed at whatever you hand it — none of them know your Product struct. 0.4’s answer was AnyTable: type-erasure funnelled through JSON. 0.5 replaces it with Vista, a universal, schema-bearing data handle, and builds an entire reactive stack on top of it.

• Vista — the universal data handle
• CBOR, not JSON
• Capabilities — the explicit contract
• Config-driven: YAML specs and Rhai scripting
• Contained relations and nested writes
• Diorama — caching, events, and reactive views
• Cross-persistence traversal: VistaCatalog
• More backends
• Migrating off AnyTable
• What’s still coming

Vista — the universal data handle

A Vista wraps any typed Table<DB, E> and erases both the backend and the entity. All data flows through Record<CborValue> — an ordered map of string keys to CBOR values. All schema lives on the Vista itself: columns (with types and flags), references, the id column, and a set of capability flags. Execution is delegated to a per-driver TableShell.

Table<SqliteDB, Product>   — typed entity, typed backend, compile-time safe
Vista                      — fully erased: schema-bearing, CborValue, no generics

Each backend ships a factory that turns a typed table into a Vista. The factory harvests columns, id field, title fields, and references from the table definition you already built — no extra mapping code:

#![allow(unused)]
fn main() {
use vantage_sql::prelude::*;
use vantage_vista::Vista;

let table = Product::table(db.clone());
let vista = SqliteVistaFactory::new(db).from_table(table)?;
}

For MongoDB it’s MongoVistaFactory, for AWS AwsVistaFactory — same shape, different import. The resulting Vista is identical regardless of which factory produced it. From there, everything is runtime introspection:

#![allow(unused)]
fn main() {
// Schema — no entity type required
for name in vista.get_column_names() {
    let col = vista.get_column(name).unwrap();
    println!("{}: {}", col.name, col.original_type);  // name: String, price: i64, ...
}
for (name, kind) in vista.list_references() {
    println!("ref: {} ({:?})", name, kind);           // ref: products (HasMany)
}

// Query — Vista delegates the value to the driver's native condition type
let mut v = vista.clone();
v.add_condition_eq("category_id", 1.into())?;
v.add_search("tart")?;                                 // fans across SEARCHABLE columns
v.add_order("price", SortDirection::Descending)?;
let rows = v.fetch_page(1).await?;
}

CBOR, not JSON

Where AnyTable narrowed every value to serde_json::Value, Vista carries ciborium::Value end to end. CBOR preserves the type fidelity JSON loses — integer vs float, binary blobs, precise decimals — so a Record<CborValue> round-trips through the Vista layer without lossy hops. JSON conversion happens only at the boundary, when you actually need it (an HTTP response, for example), and it’s a one-liner.

This is the same carrier the backends already use internally (Surreal and SQL store CBOR; Mongo stores BSON), so wrapping a typed table into a Vista no longer crosses a JSON funnel.

Capabilities — the explicit contract

Not every backend can do everything. A CSV file can’t sort or search server-side; DynamoDB orders only by its declared sort key; a token-paginated REST API offers a forward cursor but no page numbers. Vista makes this explicit with VistaCapabilities — a struct of booleans where each driver declares exactly what it supports:

#![allow(unused)]
fn main() {
let caps = vista.capabilities();
if caps.can_search { v.add_search("query")?; }
if caps.can_fetch_page {
    let page = v.fetch_page(2).await?;          // random access
} else if caps.can_fetch_next {
    let (rows, token) = v.fetch_next(None).await?;  // forward cursor
}
}

UI adapters branch on these flags directly: a data grid checks can_fetch_page to decide between a scrollbar (random access) and a “load more” button (cursor-based).

Config-driven: YAML specs and Rhai scripting

A Vista no longer needs a hand-written Rust definition. Tables, columns, and relations can be declared in a YAML spec (VistaSpec) and materialized by the driver’s factory:

table: product
columns:
  - { name: name, flags: [title, searchable] }
  - { name: price }
references:
  - { name: orders, kind: has_many, foreign_key: product_id }

For everything YAML can’t express — vendor-specific expressions, derived sources, scripted reference traversal — there’s an optional Rhai DSL that compiles to native queries. The same script renders dialect-correct SQL across all three SQL backends:

#![allow(unused)]
fn main() {
let users = table("users").alias("u");
select()
    .from(users)
    .expression(users["name"])
    .where(users["age"] >= 18)
    .order_by(users["name"], "asc")
}

• Automatic identifier quoting (backticks for MySQL, double quotes for Postgres/SQLite)
• Dialect-aware primitives: date_format() → strftime() / TO_CHAR() / DATE_FORMAT()
• group_concat() → GROUP_CONCAT (SQLite/MySQL) / STRING_AGG (Postgres)

SurrealDB gets its own Rhai vocabulary — graph traversal (graph()/recurse()), record ids (thing), $parent references, and SurrealDB-namespaced aggregates. A per-reference Rhai script can even override the default foreign-key traversal, evaluated lazily with the parent row in scope.

Contained relations and nested writes

Embedded objects and arrays — an order’s lines array, a row’s JSON column — now surface as a fully editable sub-Vista:

#![allow(unused)]
fn main() {
let order = Order::table(db)
    .with_contained_many("lines", |line| {
        line.with_column("product_id").with_column("quantity")
    });
}

SurrealDB backs contained relations with native nested objects and arrays; the SQL backends store them as JSON columns and patch the host column on writeback. Reads project the column into records; writes re-serialize the whole collection and patch the parent row. Contained records can even traverse out to real tables (line.product).

Insert learned to walk relations, too. Hand insert_value a record whose keys name a relation instead of a column, and Vista sequences the writes so foreign keys populate automatically: a has-one child is inserted first and its id stamped into the parent’s FK; has-many children are inserted after the parent with the parent’s id stamped into each. Arbitrary depth, same-persistence relations only.

Diorama — caching, events, and reactive views

The largest new subsystem is vantage-diorama, a layer that sits between a Vista and whatever consumes it. It does three things: caches transparently, injects capabilities the backend lacks, and routes writes wherever you want.

graph LR
    M[Master Vista] --> D[Dio]
    L[Lens<br/>cache + callbacks] --> D
    D --> F[Facade Vista]
    D --> S[Scenery]
    D -. cache + events .-> D
    style M fill:#4a7c59,color:#fff
    style L fill:#2d6a8f,color:#fff
    style D fill:#8f5a2d,color:#fff
    style S fill:#7c2d8f,color:#fff
    style F fill:#7c2d8f,color:#fff

Four words you’ll see throughout:

• Vista — a single-backend data source (the master).
• Lens — long-lived shared infrastructure: cache backend, lifecycle callbacks, refresh policy. Built once per application.
• Dio — a Vista bound to a Lens. Owns the cache table, a write queue, an event bus, and a refresh task. Produced by lens.make_dio(vista).
• Scenery — a reactive view onto a Dio (ordered tables, individual records, aggregates). The UI binds here.

#![allow(unused)]
fn main() {
let lens = Arc::new(
    Lens::new()
        .cache_at("./cache.redb")
        .on_start(|dio| { let dio = dio.clone(); async move {
            let rows = dio.master().list_values().await?;   // seed cache from master
            dio.cache().insert_values(rows).await?;
            Ok(())
        }})
        .refresh_every(Duration::from_secs(300))
        .build()?,
);

let products = lens.make_dio(products_vista).await?;
let mut v = products.vista();          // facade: reads from cache, writes through the queue
let rows = v.list_values().await?;     // cache hit — never touches the master
}

Writes go through a queue as WriteOps; the cache updates immediately and persistence happens asynchronously. A broadcast event bus publishes DioEvents (record changed, inserted, removed, invalidated) that Sceneries subscribe to. Upstream changes — another user’s edit, a database trigger, a webhook — feed in as ChangeEvents through an on_event callback that reconciles them into the cache.

Two-pass progressive loading handles slow sources: a cheap list pass renders immediately, and expensive per-row detail hydrates lazily as rows scroll into view, keyed per query so filter/sort variants share the detail store without blocking the UI.

Cross-persistence traversal: VistaCatalog

A reference can cross a backend boundary — categories in Postgres, products in MongoDB. In 0.5 that’s no longer a Vista concern (a Vista is strictly single-backend). It moved up into vantage-vista-factory’s VistaCatalog: register a model loader per table name, then build_vista(name) materializes a Vista and traverse(relation, parent_row) resolves and narrows the related Vista regardless of which persistence backs it.

#![allow(unused)]
fn main() {
let catalog = VistaCatalog::new();
catalog.register("category", |name| postgres_factory.build(name));
catalog.register("product",  |name| mongo_factory.build(name));

let category = catalog.build_vista("category").await?;
let products = catalog.traverse("products", &category_row).await?;  // Postgres → MongoDB
}

More backends

0.5 widened the roster around the Vista abstraction:

Each implements TableShell and slots into the same Vista surface, the same YAML/Rhai config, and the same Diorama caching — nothing downstream changes.

Migrating off AnyTable

AnyTable is gone. The carrier, AnyRecord, the CborAdapter, Table::get_ref (the AnyTable-returning one), Reference::resolve_as_any, and the model_cli runner were all deleted across the 0.5.x line.

The CLI runner story collapsed to one path: vista_cli (in vantage-cli-util), which has carried the full token grammar — operators, selectors, search, aggregates — since 0.4.5 and is what every in-tree consumer already uses.

What’s still coming

Expressions & Queries

Vantage builds queries without string concatenation. The vantage-expressions crate provides a type-safe, composable expression system that works across all persistence backends — SQL, SurrealDB, MongoDB, CSV, and anything you add yourself.

• The core idea
• Vendor macros
• Composing expressions
• Identifier quoting
• ExprDataSource — executing expressions
• Deferred expressions — cross-database values
• Selectable — the query builder interface
• SelectableDataSource — wiring it up
• Expressive trait

The core idea

An Expression<T> is a template string with typed parameters:

#![allow(unused)]
fn main() {
let expr = sqlite_expr!("SELECT {} FROM {} WHERE {} > {}",
    (ident("name")), (ident("product")), (ident("price")), 100i64);
// → SELECT "name" FROM "product" WHERE "price" > ?1  (with 100 bound as i64)
}

Parameters are never interpolated into the string. They’re carried separately, each tagged with a type marker from your persistence’s type system. The bind layer uses these markers to call the right driver method — bind_i64, bind_str, bind_bool — no guessing, no silent coercion.

Three kinds of parameters:

• Scalar — a typed value: 42i64, "hello", true
• Nested — another expression, composed into the template
• Deferred — a closure that executes later (cross-database resolution)

Vendor macros

Each persistence provides a convenience macro that produces Expression<AnyType> with the correct type wrapping:

#![allow(unused)]
fn main() {
let e = sqlite_expr!("SELECT * FROM product WHERE price > {}", 100i64);
let e = surreal_expr!("SELECT * FROM product WHERE price > {}", 100i64);
let e = postgres_expr!("SELECT * FROM product WHERE price > {}", 100i64);
let e = mysql_expr!("SELECT * FROM product WHERE price > {}", 100i64);
}

Same syntax, different type universes. The compiler ensures you can’t accidentally mix a Expression<AnySqliteType> into a SurrealDB query.

Composing expressions

Expressions nest naturally. Parenthesised arguments call .expr() automatically:

#![allow(unused)]
fn main() {
let condition = sqlite_expr!("{} > {}", (ident("price")), 100i64);
let query = sqlite_expr!("SELECT {} FROM {} WHERE {}",
    (ident("name")), (ident("product")), (condition));
}

The ExpressionFlattener collapses all nesting into a single flat template with positional parameters — each one still carrying its type marker.

For building lists (e.g. multi-row INSERT), use Expression::from_vec:

#![allow(unused)]
fn main() {
let row1 = sqlite_expr!("({}, {})", "tart", 220i64);
let row2 = sqlite_expr!("({}, {})", "pie", 299i64);
let rows = Expression::from_vec(vec![row1, row2], ", ");
}

Identifier quoting

SQL identifiers need quoting — and each database uses different quote characters. The Identifier struct handles this by implementing Expressive<T> for each backend type:

#![allow(unused)]
fn main() {
// Quoting adapts to the expression's type context
sqlite_expr!("SELECT {}", (ident("name")));      // → SELECT "name"
mysql_expr!("SELECT {}", (ident("name")));        // → SELECT `name`

// Qualified identifiers
sqlite_expr!("SELECT {}", (ident("name").dot_of("u")));  // → SELECT "u"."name"

// Aliases
mysql_expr!("SELECT {}", (ident("name").with_alias("n"))); // → SELECT `name` AS `n`
}

ExprDataSource — executing expressions

The ExprDataSource<T> trait connects expressions to a live database:

#![allow(unused)]
fn main() {
// Execute directly
let result: AnySqliteType = db.execute(&expr).await?;

// Associate with an expected return type
let count: i64 = db.associate::<i64>(sqlite_expr!("SELECT COUNT(*) FROM product"))
    .get().await?;
}

AssociatedExpression<'a, DS, T, R> carries both the expression and a reference to the datasource. Call .get() to execute and convert in one step. The return type R is checked at compile time.

Deferred expressions — cross-database values

Sometimes a query on one database needs a value from another. defer() wraps a query as a closure that resolves at execution time:

#![allow(unused)]
fn main() {
// Query config_db for a threshold — but don't execute yet
let threshold = config_db.defer(
    sqlite_expr!("SELECT value FROM config WHERE key = {}", "min_price")
);

// Use the deferred value in a query against shop_db
let expensive = Expression::<AnySqliteType>::new(
    "SELECT name FROM product WHERE price >= {}",
    vec![ExpressiveEnum::Deferred(threshold)],
);
let result = shop_db.execute(&expensive).await?;
// 1. Resolves deferred → calls config_db, gets 150
// 2. Binds 150 as a scalar parameter
// 3. Executes against shop_db
}

This is not a subquery — the deferred query runs first, produces a concrete value, and that value gets bound as a regular parameter.

Selectable — the query builder interface

The Selectable<T> trait is the standard interface for building SELECT queries. Each persistence provides its own SELECT struct (SqliteSelect, SurrealSelect, PostgresSelect) implementing this trait:

#![allow(unused)]
fn main() {
let select = SqliteSelect::new()
    .with_source("product")
    .with_field("name")
    .with_field("price")
    .with_condition(sqlite_expr!("{} = {}", (ident("is_deleted")), false))
    .with_order(sqlite_expr!("{}", (ident("price"))), false)
    .with_limit(Some(10), None);
}

Builder methods come free from the trait — with_source, with_field, with_condition, with_order, with_limit. You only implement the mutating methods (add_field, add_where_condition, etc.).

Aggregate shortcuts clone the query and replace fields:

#![allow(unused)]
fn main() {
let count = select.as_count();                              // SELECT COUNT(*) FROM ...
let total = select.as_sum(sqlite_expr!("{}", (ident("price")))); // SELECT SUM("price") FROM ...
}

SelectableDataSource — wiring it up

SelectableDataSource<T> connects the query builder to execution:

#![allow(unused)]
fn main() {
impl SelectableDataSource<AnySqliteType> for SqliteDB {
    type Select = SqliteSelect;

    fn select(&self) -> Self::Select { SqliteSelect::new() }
    async fn execute_select(&self, select: &Self::Select) -> Result<Vec<AnySqliteType>> {
        self.execute(&select.expr()).await
    }
}
}

Once implemented, table.select() returns your vendor-specific builder pre-populated with the table’s columns, conditions, and ordering — ready for execution or further customisation.

Expressive trait

Anything that implements Expressive<T> can be used inside an expression. This includes:

• Columns — table["price"]
• Operations — table["price"].gt(100)
• Identifiers — ident("name")
• Query builders — select.expr()
• Sort orders — table["name"].desc()
• Scalar values — 42i64, "hello", true
• Closures — that’s what defer() returns

You can implement Expressive<T> for your own types to make them composable into the expression system.

Because conditions are themselves Expressive, operations chain across type boundaries: price.gt(10) returns a SqliteCondition, and .eq(false) works on it — reading “(price > 10) = 0”. Type safety is enforced on the first operation, at the column level; once you hold a condition, any value compatible with the backend’s any-type is accepted (price.gt(10).eq("foobar") compiles by design). Columns of the same type compare directly (price.eq(price.clone())), and operations take ownership of their arguments — clone a column you plan to reuse across conditions.

Records: Traversal, Invariants & Hooks

Once a backend implements TableSource (see Adding a New Persistence), vantage-table gives every Table<Db, Entity> a uniform write pipeline. On each write the record flows through three stages before it reaches the datasource:

record  →  lifecycle hooks (before)  →  set invariants  →  datasource write  →  hooks (after)

This page covers the consumer-facing features built on that pipeline: traversing a relation from a loaded record, the foreign-key invariants that traversal sets up, and the lifecycle hooks you can attach for audit, validation, soft-delete, and after-effects. They are backend-agnostic — everything here works the same on SQLite, Postgres, MySQL, Mongo, etc.

Traversing from a loaded record: get_ref

Table::get_ref_as / get_ref_from_row traverse a relation from a table (both covered in Relations & Traversal). The GetRefExt trait adds the record-level equivalent: traverse straight from a loaded ActiveEntity (typed) or ActiveRecord (untyped).

#![allow(unused)]
fn main() {
use vantage_table::prelude::GetRefExt;

let launch = launches.get_entity(id).await?.expect("launch");

// A child set scoped to this launch — and carrying its foreign key (see invariants below).
let crew = launch.get_ref::<LaunchCrew>("launch_crew")?;
crew.insert_return_id(&LaunchCrew { astronaut_id: Some(a), role: Some("Pilot".into()), ..Default::default() }).await?;
}

get_ref::<E2>(relation) returns a Table<T, E2> scoped to the parent. For a typed ActiveEntity the entity’s id is injected into the row before traversal (so has-many relations resolve); an untyped ActiveRecord already holds the raw row and forwards directly. The same method exists on both handles — GetRefExt carries blanket implementations for each, generic over any TableSource (the handles themselves live in vantage-dataset and know nothing about Table, which is why this arrives as an extension trait).

Set invariants

A table narrowed by a literal column = value is a set, and every row written into it must conform to that definition. Vantage records the pair as an invariant and enforces it on write. The most common source is relationship traversal: launch.get_ref::<LaunchCrew>("launch_crew") narrows the child set by launch_id = <this launch>, so an inserted crew row’s launch_id is filled automatically.

Invariants are registered automatically wherever the scope is a plain column = value:

• Table::with_id(id) — narrows by the id column.
• has-many / has-one traversal (Reference::resolve_from_row) — narrows by the foreign key.

Expression scopes never register an invariant. You can also set one explicitly with Table::with_invariant(column, value) / add_invariant.

On every insert / replace / patch, each invariant column is resolved by a four-way rule:

So a child row inserted through a relation needs no foreign key (it’s filled), may state the matching one (kept), but cannot smuggle in a different one (error — it doesn’t belong to this set).

#![allow(unused)]
fn main() {
let crew = launch.get_ref::<LaunchCrew>("launch_crew")?;

// launch_id absent → filled from the set:
crew.insert_return_id(&LaunchCrew { astronaut_id: Some(a), role: Some("Pilot".into()), ..Default::default() }).await?;

// launch_id set to a *different* launch → Err:
assert!(crew.insert_return_id(&LaunchCrew { launch_id: Some("other".into()), ..Default::default() }).await.is_err());
}

Lifecycle hooks

Attach async callbacks around writes with Table::with_hook(Hook::…). Hooks are how audit stamps, normalization, validation, soft-delete, and after-effects are expressed — generically, on any backend.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use vantage_table::prelude::{Hook, Phase};

let launches = Launch::table(db).with_hook(Hook::BeforeInsert(Phase::Populate, Arc::new(stamp_created)));
}

The Hook variants

One enum carries a placement-specific closure, so each hook receives exactly what’s available at its stage. Before-write hooks get the record (mutable) and the entity-erased table; the delete hook gets the id and the row’s former contents:

The &Table handed to a hook is entity-erased, so it can traverse relations (get_ref) and reach the datasource — enough for cross-row validation and after-effects.

Ordering and control flow

• Before-write hooks run ahead of invariant enforcement, ordered by Phase: Normalize → Populate (the default) → Validate, then registration order within a phase. So normalize inputs, then derive/stamp fields, then validate the final record.
• Returning Err from any before-hook cancels the operation before anything is written.
• BeforeDelete returning HookReturn::Handled skips the real delete and reports success — this is how soft-delete works (patch a deleted marker, return Handled). A delete with hooks loads the row once so before/after hooks see its contents; delete_all fires no hooks.
• After-hooks run for side-effects only. Vantage favours idempotence over transactions: an after-hook failure surfaces an error but does not roll back the committed write — design after-effects to be safe to retry.

Writing a hook

Hooks are boxed async closures. The reliable construction is a free fn returning a boxed future whose lifetime ties to the arguments, then Arc::new(it):

#![allow(unused)]
fn main() {
use std::future::Future;
use std::pin::Pin;
use vantage_core::Result;
use vantage_types::Record;
use vantage_sql::sqlite::{AnySqliteType, SqliteDB};
use vantage_table::table::Table;

// before-insert: stamp an audit field
fn stamp_created<'a>(
    rec: &'a mut Record<AnySqliteType>,
    _t: &'a Table<SqliteDB, vantage_types::EmptyEntity>,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'a>> {
    Box::pin(async move {
        rec.insert("created".into(), AnySqliteType::new(now_iso8601()));
        Ok(())
    })
}

// before-delete: soft-delete instead of removing the row
fn soft_delete<'a>(
    id: &'a String,
    _former: &'a Record<AnySqliteType>,
    table: &'a Table<SqliteDB, vantage_types::EmptyEntity>,
) -> Pin<Box<dyn Future<Output = Result<vantage_table::prelude::HookReturn>> + Send + 'a>> {
    Box::pin(async move {
        let mut patch = Record::new();
        patch.insert("deleted".into(), AnySqliteType::new(now_iso8601()));
        table.patch_value(id.clone(), &patch).await?;
        Ok(vantage_table::prelude::HookReturn::Handled)
    })
}
}

To pull in outside context (e.g. an updated_by actor), have the hook capture an Arc or read a task-local — the closure runs in async context. A capturing closure works too, but needs its boxed return type annotated explicitly so it coerces to the hook type.

Relations & Traversal

Vantage treats a relation as something you traverse, not something you load. Declared once on the model, a relation gives you:

1. Set-to-set traversal — narrow “clients” down to the paying ones, traverse, and you hold “orders of paying clients” as a new set. One query when it executes; no loop over IDs.
2. Related values inside queries — a client row carrying its order_count, an order carrying its client.name — computed by the database through correlated subqueries or native link paths.
3. The same vocabulary on every backend — relations are declared on the model, not read from foreign-key constraints, so CSV files, REST APIs, and document stores get the same treatment as SQL.
4. Relations that survive the stack — the declaration travels from the typed Table through the type-erased Vista into the cached Dio; where two datasources meet, the Dio stitches the join client-side.

If you come from an ORM, the shape is different enough to spell out:

Our running example is bakery_model3 — an example model crate that ships in the Vantage repository. It defines one set of entities (bakeries, clients, orders, products) with table constructors for several datasources — SQLite, PostgreSQL, SurrealDB, MongoDB, CSV, DynamoDB — and doubles as a test fixture for the framework itself. Model-Driven Architecture walks through its anatomy; here we just borrow it as the example.

Three of its tables belong together: a Bakery has many Clients, and each Client has many Orders. In SQL terms, client.bakery_id points at a bakery and client_order.client_id points at a client:

erDiagram
    bakery ||--o{ client : "client.bakery_id"
    client ||--o{ client_order : "client_order.client_id"

Almost every real question you’d ask of this data crosses one of those links — “orders of paying clients”, “which bakery does this order belong to” — so this guide is about how Vantage models those links and how you cross them.

You met relations briefly in the Introduction, where a product catalog picked up with_many and get_ref_as; this guide covers the same machinery in depth, with a fuller model. We work SQL-first here (SQLite), with SurrealDB and other backends in notes along the way. The chapters after this one dig into traversal forms, subquery expressions, implicit references, and how relations survive into the type-erased Vista and cached Dio layers — this page sets up the vocabulary they all share.

One relation, four layers

An ORM or a data mapper typically has exactly one place where references live: on the mapped object. client.orders() loads objects, and that is the whole story. In Vantage the same declared relation is traversable at (approximately) four layers. Each layer traverses with what it has in hand and gives you back a handle native to that layer — you never drop down a level to cross a link:

The traversal method is called get_ref at every layer — what changes is what you hold when you call it, and when you’d want to:

Why is there a Dio layer at all, if the catalog already crosses datasources? Because catalog traversal navigates — it hands you the related set, and every read of it is a live round-trip. The Dio owns what navigation lacks: a cache, a viewport, and change events. It caches each traversed detail set, and it hosts augmentation — not navigation to related rows but a per-row join into the master’s rows, which needs exactly that cache and viewport. Relations and Dio works through both.

The rest of this page covers the first vocabulary — declaring the relation. The child chapters then walk the layers: tables, vistas, and dio.

What a relation is

A relation in Vantage is a declared, named link between two table definitions. It lives on the table definition — the model — not on the entity struct, and not in the database. Vantage never introspects foreign-key constraints; whether your SQLite schema actually declares FOREIGN KEY (client_id) REFERENCES client(id) is invisible to it. A relation is exactly three things:

1. A name — the string you traverse by, like "orders".
2. A foreign-key field — the column that carries the link.
3. A target-table constructor — a function that builds the table on the other end.

Two properties follow from this shape. References are one-sided: a declaration gives this table a way to reach the target — nothing is declared on the target, and no back-reference is required. If you want to traverse the other direction, declare that separately (the bakery model does: orders on Client, client on Order).

And because traversal produces sets, the same pair of tables can be linked by any number of references — same target, same foreign key, different conditions. The constructor argument is what makes this natural: pass a closure that narrows the target, and the relation names a meaningful subset, not just a foreign key:

#![allow(unused)]
fn main() {
.with_many("paid_invoices", "client_id", |db| {
    let mut t = Invoice::table(db);
    t.add_condition(t["status"].eq("paid"));
    t
})
.with_many("due_invoices", "client_id", |db| {
    let mut t = Invoice::table(db);
    t.add_condition(t["status"].eq("due"));
    t
})
}

With the extension-trait convention (below), application code reads client.ref_paid_invoices() / client.ref_due_invoices() — one invoice table underneath, two named ways to reach it.

Why not read the foreign keys from the database?

Because most of the backends Vantage talks to don’t have any. CSV files, REST APIs, and MongoDB have no foreign-key concept to introspect, yet their data is just as related. Declaring the relation on the model means the same declaration works everywhere — the link between clients and orders exists whether the rows live in Postgres or in a spreadsheet export.

Two cardinalities

There are two declaration methods, and the difference between them comes down to one question: which table holds the foreign key?

• with_one("name", "fk_field", constructor) — the FK lives on this table. Traversal yields at most one row: many-to-one. An order has one client.
• with_many("name", "fk_field", constructor) — the FK lives on the target table. Traversal yields a set: one-to-many. A client has many orders.

Both declarations describe the same underlying link — one foreign-key column, viewed from either end:

graph LR
    order["client_order<br/>(carries client_id)"]
    client["client<br/>(has id)"]
    order -- "with_one('client')" --> client
    client -- "with_many('orders')" --> order

The two are more similar than the names suggest. Both take the same closure and use it the same way: build the target table, then pin it with a single eq-condition. The only difference is the condition’s orientation — with_one reads the foreign key out of the source row and pins the target’s id; with_many reads the source row’s id and pins the target’s foreign-key column. Same closure, same eq-condition, read from opposite ends of the link.

Internally these become HasOne and HasMany implementations of the Reference trait (vantage-table/src/references/), and the cardinality is surfaced as ReferenceKind::HasOne / ReferenceKind::HasMany for anything that needs to reason about relations generically — the vista layer will, later.

You could write it yourself

with_one, with_many, and get_ref are convenience. A traversal is just “build a table, add a condition from what I’m holding” — and nothing stops you from writing exactly that as an extension method on a loaded record, without registering any reference at all. It doesn’t even have to stay in the same datasource:

#![allow(unused)]
fn main() {
pub trait ClientMailLog {
    fn ref_mail_log(&self) -> Table<Mailchimp, MailLogEntry>;
}

impl<'a> ClientMailLog for ActiveEntity<'a, Table<SqliteDB, Client>, Client> {
    fn ref_mail_log(&self) -> Table<Mailchimp, MailLogEntry> {
        let mut log = MailLogEntry::table(mailchimp());
        log.add_condition(log["email"].eq(self.email.clone()));
        log
    }
}
}

(self.email works because ActiveEntity derefs to the entity struct — the loaded Client’s fields are right there. An untyped ActiveRecord derefs to the raw Record instead, where the same value is self.get("email").)

From the caller’s side client.ref_mail_log() is indistinguishable from a declared relation — but there is no foreign key here, no registry entry, just a Mailchimp-backed table conditioned by the client’s email. What the declared forms add over hand-rolling is everything that needs the registry: string-addressable traversal for the generic layers (vistas, implicit references), and the foreign-key invariant on inserts. Reach for a hand-rolled method when the link doesn’t fit an FK eq-match; declare a relation when you want the machinery.

Coercion — pinning get_ref’s type

A relation is addressed by string, and a string carries no type. get_ref therefore cannot infer what entity comes back — it is generic, and the caller chooses: get_ref::<Order>("orders"). The turbofish is an assertion, not a check: the registry knows which table the relation builds, but the entity type you name is simply coerced onto the result. Name the wrong one and you find out at runtime, when rows fail to deserialize — not from the compiler.

This uncertainty is exactly why the model wraps every relation once, in an extension trait next to the table definition:

#![allow(unused)]
fn main() {
pub trait ClientTable {
    fn ref_orders(&self) -> Table<SqliteDB, Order>;
}

impl ClientTable for Table<SqliteDB, Client> {
    fn ref_orders(&self) -> Table<SqliteDB, Order> {
        self.get_ref_as("orders").unwrap()
    }
}
}

And the same on the other side — Order gets ref_client():

#![allow(unused)]
fn main() {
pub trait OrderTable {
    fn ref_client(&self) -> Table<SqliteDB, Client>;
}

impl OrderTable for Table<SqliteDB, Order> {
    fn ref_client(&self) -> Table<SqliteDB, Client> {
        self.get_ref_as("client").unwrap()
    }
}
}

The coercion happens in one place — where the right answer is obvious — and the unwrap() is safe there too: "orders" is declared on every Client table the model produces, so a typo panics in development, not silently at runtime. This is the convention to follow throughout a model crate: every declared relation surfaces as a typed ref_<name>() method, callers write clients.ref_orders() / orders.ref_client(), and the strings and turbofish exist in exactly one place with no opportunity to name the wrong entity. The Introduction used the same pattern, and Model-Driven Architecture covers where these traits live in a model crate.

Declaring relations on the model

Here is Client from bakery_model3/src/client.rs — the entity struct and its SQLite table constructor:

#![allow(unused)]
fn main() {
#[entity(CsvType, SurrealType, SqliteType, PostgresType, MongoType, DynamoType)]
#[derive(Debug, Clone, PartialEq, Default)]
pub struct Client {
    pub name: String,
    pub email: String,
    pub contact_details: String,
    pub is_paying_client: bool,
    pub bakery_id: Option<String>,
}

impl Client {
    pub fn sqlite_table(db: SqliteDB) -> Table<SqliteDB, Client> {
        Table::new("client", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<String>("email")
            .with_column_of::<String>("contact_details")
            .with_column_of::<bool>("is_paying_client")
            .with_column_of::<String>("bakery_id")
            .with_one("bakery", "bakery_id", Bakery::sqlite_table)
            .with_many("orders", "client_id", Order::sqlite_table)
    }
}
}

Notice that the struct knows nothing about relations. bakery_id is just an Option<String> field like any other; the links are declared on the table definition, where the columns and conditions already live.

And here is the other end, from order.rs:

#![allow(unused)]
fn main() {
impl Order {
    pub fn sqlite_table(db: SqliteDB) -> Table<SqliteDB, Order> {
        Table::new("client_order", db)
            .with_id_column("id")
            .with_column_of::<String>("client_id")
            .with_column_of::<bool>("is_deleted")
            .with_one("client", "client_id", Client::sqlite_table)
    }
}
}

The same underlying relation appears from both ends: Client declares with_many("orders", "client_id", …) and Order declares with_one("client", "client_id", …). Each side is declared independently and each is usable independently — you could declare only one of them if you never traverse the other direction. And both name the same column: "client_id" on the client_order table. For with_many that’s the FK on the target table; for with_one it’s the FK on the source — here Order happens to be the FK-carrying side both times.

The constructor argument

The third argument — Order::sqlite_table — is not a table, it’s a function. It’s the very same constructor the model already exposes; you’re just passing it by name instead of calling it. That buys two things. First, declaring a relation costs nothing up front: the target table is only constructed when the relation is actually traversed. Second, and more importantly, when the target is constructed, it’s the real model definition — its own columns, its own conditions, its own relations. If Order::sqlite_table filtered out soft-deleted rows, every traversal that reaches orders through a relation would inherit that filter. There is no second, weaker definition of “orders” that relations quietly use.

A first traversal

Traversal is set-to-set: narrow the source set, traverse, and the target set arrives already narrowed by the relationship. Here’s “orders of paying clients”:

#![allow(unused)]
fn main() {
let mut paying = Client::sqlite_table(db.clone());
paying.add_condition(paying["is_paying_client"].eq(true));

// Table<SqliteDB, Order>, narrowed to orders of paying clients
let orders = paying.get_ref_as::<Order>("orders")?;
}

No query has run yet — orders is a Table<SqliteDB, Order> like any other, and you can add conditions, select columns, or traverse further. When it does execute, the narrowing happens via a subquery — not a JOIN, and not a round-trip to fetch client IDs first. The shape (illustrative):

SELECT ... FROM "client_order"
WHERE "client_id" IN (SELECT "id" FROM "client" WHERE "is_paying_client" = 1)

This is the payoff of set-to-set thinking: “orders of paying clients” is one narrowing plus one traversal, executed as a single query — never a loop over IDs.

In application code, the string and the turbofish disappear behind the wrapper from the coercion section above:

#![allow(unused)]
fn main() {
let orders = paying.ref_orders();   // Table<SqliteDB, Order> — type inferred, no string
}

The next chapter covers the full range of traversal forms; this is just the first taste.

SurrealDB: record links instead of foreign keys

SurrealDB stores relations as record links — the FK column holds a typed record id (a Thing, rendered table:key) rather than a scalar. The declaration shape is identical; only the field differs. From the same bakery_model3/src/client.rs:

#![allow(unused)]
fn main() {
pub fn surreal_table(db: SurrealDB) -> Table<SurrealDB, Client> {
    Table::new("client", db)
        .with_id_column("id")
        .with_column_of::<String>("name")
        .with_column_of::<String>("email")
        .with_column_of::<String>("contact_details")
        .with_column_of::<bool>("is_paying_client")
        .with_one("bakery", "bakery", Bakery::surreal_table)
        .with_many("orders", "client", Order::surreal_table)
}
}

What changes is the column’s content, not its name — that is a database-design choice. Any column can hold a record link: you could equally keep calling it bakery_id and store a Thing in it, and the declaration would name that column instead. bakery_model3 happens to follow the common SurrealDB convention of naming the link column after the target ("bakery"), which is why its declarations differ from the SQLite ones only in that argument. Either way, the relation name is free to differ from the link field; nothing couples them.

Writing into a traversed set

A traversed set is not read-only. Insert into it, and the new record belongs to the set you traversed to:

#![allow(unused)]
fn main() {
client.ref_orders().insert_return_id(&order).await?;
}

The order’s client_id is filled in automatically — you don’t set it, and if you set a conflicting one the write is rejected. How that enforcement works (set invariants, lifecycle hooks) is covered in Records: Traversal, Invariants & Hooks.

Where this guide goes

You now have the vocabulary: a relation is a named (name, fk_field, constructor) triple on the table definition, in one of two cardinalities, traversed set-to-set. The child chapters each take one thread further:

• Traversing Sets and Records — the full range of traversal forms: set-to-set (fetching and embedding), row-in-hand, record-level, and contained.
• Expressions & Subqueries — correlated lookups, counts, and how relation-derived expressions compose.
• Implicit References — dotted column names, the declarative way to pull a field across a relation.
• Vista, YAML and Rhai — vista factories, traversal after type erasure, capability checks, and declaring relations in YAML and Rhai.
• Relations and Dio — where cross-datasource enrichment lives.

Traversing Sets and Records

The previous chapter declared the bakery model: Bakery → Clients (client.bakery_id) → Orders (client_order.client_id), with the relations "orders", "client", and "bakery" registered via with_one / with_many. Declaring a relation stores the join recipe. This chapter is about using it: turning “the orders of these clients” into an actual query.

There are four traversal forms. They differ in what you have in hand when you traverse — a set (conditions, no data), a loaded row, or a loaded record object — and, between the two set forms, in how the condition is applied to the target. Each form exists because each starting point lets the backend do something different.

Form 1 — set-to-set: get_ref_as

You have a set — a table narrowed by conditions, possibly matching many rows, none of them loaded. Table::get_ref_as::<E2>(relation) traverses from the whole set at once:

#![allow(unused)]
fn main() {
let mut paying = Client::sqlite_table(db.clone());
paying.add_condition(paying["is_paying_client"].eq(true));

let orders = paying.get_ref_as::<Order>("orders")?;
// SELECT ... FROM "client_order"
// WHERE "client_id" IN (SELECT "id" FROM "client" WHERE "is_paying_client" = 1)
}

No client rows are fetched. The traversal generates an IN (subquery) condition on the target, and the subquery is whatever conditions the source set carries. Narrow paying further and the subquery narrows with it — the traversal call never changes. That composability is the point: you build the set that describes which parents you mean, and the child set inherits that description.

In application code these calls hide behind the model’s extension traits — paying.ref_orders(), orders.ref_client() — so the string and the turbofish live once, next to the table definition (Coercion — pinning get_ref’s type). This chapter spells out the underlying calls so you can see what each form generates.

How the IN condition is built is per-backend — each backend implements TableSource::related_in_condition:

• SQL builds the subquery you see above: "client_id" IN (SELECT "id" FROM "client" WHERE …).
• MongoDB has no subqueries, so it builds a deferred { field: { "$in": [...] } } document — the parent ids are resolved when the child query runs.
• CSV fetches the join values in memory and builds an IN condition from them.

Use this form when the parent is a set — filtered, unfiltered, one row or a thousand.

Form 2 — set-to-set, for embedding: get_subquery_as

Technically set-to-set as well — same starting point, same declared relation — but the condition is applied differently. Where get_ref_as narrows the target with IN (subquery) so the related rows can be fetched on their own, Table::get_subquery_as::<E2>(relation) attaches a correlated condition — target column against source column, row by row:

#![allow(unused)]
fn main() {
let orders = clients.get_ref_as::<Order>("orders")?;      // fetch:  WHERE client_id IN (SELECT id FROM client …)
let orders = clients.get_subquery_as::<Order>("orders")?; // embed:  WHERE client_id = client.id
}

A correlated table is useless to fetch standalone — its condition references the source’s rows — but it is exactly right to embed as a scalar subquery inside the source’s own SELECT: a client’s order_count, an order’s client.name. The embedding recipe (select_column, aggregates, composition) is the subject of the next chapter.

Form 3 — row-in-hand: get_ref_from_row

You already hold a loaded row — a Record<T::Value> that came back from the database. There is nothing to compute: the join value is sitting in the row. Table::get_ref_from_row::<E2>(relation, &row) reads it out and applies it as a single eq-condition on the target:

#![allow(unused)]
fn main() {
let orders = clients.get_ref_from_row::<Order>("orders", &row)?;
// WHERE "client_id" = <the id read out of row> — one eq-condition
}

Which field gets read depends on the relation’s direction: a HasOne relation reads its stored foreign-key column from the row; a HasMany relation reads the source’s id field. No subquery, no deferred fetch — row already carries the value.

Form 4 — record-level: GetRefExt::get_ref

The same traversal as form 3, called from a loaded handle instead of a raw row: launch.get_ref::<LaunchCrew>("launch_crew") works on an ActiveEntity (typed — its id is injected into the row first, so has-many relations resolve) and on an ActiveRecord (untyped — the raw row forwards directly). The method comes from the GetRefExt extension trait in vantage-table, with blanket implementations for both handles over any TableSource. The worked example (loading a record, traversing, inserting a child) lives in Records: Traversal, Invariants & Hooks.

The bare target: get_ref_target

Sometimes you want the relation’s target table with no condition at all — Table::get_ref_target::<E2>(relation) builds exactly that. Where the traversal forms narrow the target to related rows, get_ref_target hands you the table you’d insert a new related row into before any join value exists (this is what Vista’s nested insert uses).

Contained relations

Some backends embed related data inside the row instead of linking to another table. SurrealDB’s Order carries an embedded array<object> of order lines — declared as a column (so it is selected) and as a contains-many relation whose record schema is built by a closure. From bakery_model3/src/order.rs:

#![allow(unused)]
fn main() {
pub fn surreal_table(db: SurrealDB) -> Table<SurrealDB, Order> {
    Table::new("order", db)
        .with_id_column("id")
        .with_column_of::<Thing>("client")
        .with_column_of::<bool>("is_deleted")
        // `lines` is an embedded `array<object>` of `{ product, quantity,
        // price }` — declared as a column so it's selected, and as a
        // contains-many relation whose record schema is built by the
        // closure (like `with_many`).
        .with_column_of::<AnySurrealType>("lines")
        .with_one("client", "client", Client::surreal_table)
        .with_contained_many(
            "lines",
            "lines",
            |db| {
                Table::new("lines", db)
                    .with_column_of::<Thing>("product")
                    .with_column_of::<i64>("quantity")
                    .with_column_of::<i64>("price")
                    // a line traverses out to the real product table
                    .with_one("product", "product", Product::surreal_table)
            },
            None,
        )
}
}

Contained relations are a K/V-and-document-store shape: the value already holds the related data, so “traversal” is descent into the row, not a query. Note that a contained record can still hold references back out — a line’s product field is a with_one to the real product table, so from an embedded line you traverse out with the same forms as anywhere else.

Conclusion

You can now:

1. Traverse a filtered set with get_ref_as — conditions compose into an IN (subquery) on the target.
2. Correlate instead of fetch with get_subquery_as — same relation, a per-row condition, built for embedding in the source’s SELECT.
3. Traverse from a loaded row or record with get_ref_from_row / get_ref — one eq-condition, supported by every backend.
4. Obtain a bare insert target with get_ref_target — the relation’s target with no condition.
5. Declare embedded/contained relations on document stores with with_contained_many, where traversal is descent rather than a query.

Expressions & Subqueries

The previous chapter used relations to fetch related rows — traverse from a client to their orders, get the orders back as a set. This chapter uses relations inside a query: a client row that carries its own order count, computed by the database, delivered as just another column.

The recipe has three parts:

1. Table::with_expression(name, |t| …) — adds a computed field to the table. It’s evaluated as part of the SELECT, alongside the physical columns.
2. Table::get_subquery_as::<E2>(relation) — builds the relation’s target table with a correlated condition. For the clients → orders relation that condition is client_order.client_id = client.id — the direction is always target.<id> = source.<fk>, supplied by the backend through TableSource::related_correlated_condition. This table is explicitly designed for embedding as a scalar subquery.
3. select_column("field") — projects a single column from that subquery, so it can sit where a scalar value is expected.

Here’s the shape, straight from the select_column rustdoc:

#![allow(unused)]
fn main() {
.with_expression("category", |t| {
    t.get_subquery_as::<Category>("category").unwrap()
        .select_column("name")
        .expect("Category has a 'name' column")
})
}

The key difference from the previous chapter: get_ref_as narrows with IN (subquery) to fetch related rows; get_subquery_as correlates to embed a related value per row. Same relation declaration, two different query shapes.

select_column returns Option<Expression<T::Value>> — None when the field isn’t a column on the target table. When the column is hardcoded and known to exist, .expect(...) is the honest choice: it panics in development, where you can fix the typo, rather than silently at runtime.

Why not a JOIN?

Vantage composes sets. A correlated scalar subquery keeps the row shape of the source table — one row per client, one extra column — with no join-key bookkeeping and no row multiplication when a client has ten orders. It also nests arbitrarily, which joins don’t do gracefully. And you’re not paying for the nesting: the database’s optimizer flattens a correlated subquery where a join would be equivalent.

Aggregates over a relation

get_subquery_as returns a full Table, so the aggregate query-builders work on it: get_count_query() and get_sum_query(&col). Both wrap their output in parentheses, so they nest safely inside the outer SELECT.

Here’s the real thing — verbatim from bakery_model3/src/client.rs, a client table whose order_count is computed by the database:

#![allow(unused)]
fn main() {
pub fn surreal_table(db: SurrealDB) -> Table<SurrealDB, Client> {
    Table::new("client", db)
        .with_id_column("id")
        .with_column_of::<String>("name")
        .with_column_of::<String>("email")
        .with_column_of::<String>("contact_details")
        .with_column_of::<bool>("is_paying_client")
        .with_one("bakery", "bakery", Bakery::surreal_table)
        .with_many("orders", "client", Order::surreal_table)
        .with_expression("order_count", |t| {
            let orders = t.get_subquery_as::<Order>("orders").unwrap();
            orders.get_count_query()
        })
}
}

The generated query has this shape (illustrative):

SELECT *, (SELECT COUNT(*) FROM order WHERE order.client = client.id) AS order_count
FROM client

The expression field shows up in ReadableValueSet results alongside the physical columns. To get it onto your entity struct, add a matching Option field — it deserializes like any other column.

Composing expressions

Expressions can reference other expressions. get_column_expr(name) resolves a name to either a real column or a computed expression, so one expression can build on another — a title composed of name and a count subquery via concat_!, for instance. All of it renders into a single query. The nesting looks redundant on paper, but SQL engines optimize it away; there is no extra round trip.

The Table chapter walks through a concrete title-composition example step by step — see Working with Tables rather than repeating it here.

Nesting arbitrary expressions — select_expression

select_column is sugar over a more general form. Table::select_expression(expr) wraps any expression as a single-column subquery over the table’s source and conditions — SELECT <expr> FROM table WHERE … — clearing fields and ordering, since a scalar subquery has exactly one output and no meaningful sort.

This is what lets one correlated subquery nest inside another: a two-hop lookup is outer.select_expression( (inner_subquery) ). It is also the machinery under the next chapter’s implicit references, which automate exactly this recipe.

SurrealDB notes

SurrealDB’s record links give it a second way to express a related field: a native idiom path. client.name on an order traverses the link in place — no subquery at all, and often cheaper. The implicit references chapter uses exactly that lowering automatically.

Correlated subqueries still work too. The order_count example above runs on SurrealDB as written — order.client = client.id correlates through the record link.

Conclusion

At this point you should be able to:

1. Embed a related field as a computed column — with_expression + get_subquery_as::<E2>(relation) + select_column("field").
2. Add relation aggregates — get_count_query() and get_sum_query(&col) on the subquery table.
3. Compose expressions — get_column_expr(name) resolves columns and expressions alike, so expressions can build on each other inside one query.
4. Nest subqueries — select_expression(expr) wraps any expression as a scalar subquery, one inside another.

Next: Implicit References — the declarative form of everything in this chapter’s first half.

Implicit References

The previous chapter (Expressions & Subqueries) closed with a recipe: take a relation, call get_subquery_as to build a correlated subquery for one field, and register it with with_expression so it projects as a column. That recipe works, but surfacing a related row’s field is common enough to deserve a declarative form.

That form is a dotted name in Table::with_active_columns. Write "client.name" and the table traverses the declared has_one relation "client", imports the target’s name field, and projects it as a read-only column aliased under the literal dotted name — no hand-written expression. This feature is also called implicit references.

The first segment addresses the relation by its registry name — whatever you named it in with_one — not by the foreign-key column. bakery_model3 declares with_one("client", "client_id", …), so the path is client.name on SQL and SurrealDB alike; a model that names its relations after the FK columns would write client_id.name instead. Either convention works — the segment just has to match the declared name.

with_active_columns does two jobs at once:

1. Plain names restrict projection. Only the listed columns are selected — useful on its own for narrowing wide tables. The id column is always projected regardless.
2. Dotted names traverse relations. Each dot is one has_one hop from the current table to its target.

One hop and two hops

Using the bakery model from this guide — Orders relate to Client via client_order.client_id, and Client relates to Bakery via client.bakery_id, declared as relations "client" and "bakery" with with_one:

#![allow(unused)]
fn main() {
// Plain names restrict projection; dotted names traverse has_one relations.
let orders = Order::sqlite_table(db).with_active_columns(&[
    "id",
    "client_id",
    "client.name",        // one hop:  client_order -> client
    "client.bakery.name", // two hops: client_order -> client -> bakery
])?;
}

client.name is one hop. client.bakery.name recurses through two — traversal supports an arbitrary depth of has_one hops out of the box. On a SQL backend this lowers to nested correlated scalar subqueries:

SELECT id,
  (SELECT name FROM client WHERE client.id = client_order.client_id) AS "client.name",
  (SELECT (SELECT name FROM bakery WHERE bakery.id = client.bakery_id)
     FROM client WHERE client.id = client_order.client_id)           AS "client.bakery.name"
FROM client_order

The alias is the dotted name itself, so rows come back with a flat key equal to the dotted name:

#![allow(unused)]
fn main() {
for row in orders.list_values().await?.values() {
    let get = |k: &str| row.get(k).map(|v| format!("{v}")).unwrap_or_default();
    println!(
        "  order {}: client.name={}, client.bakery.name={}",
        get("id"),
        get("client.name"),
        get("client.bakery.name"),
    );
}
}

There is no nested object to unwrap — row.get("client.bakery.name") is a plain lookup on a flat record. The full example lives in bakery_model3/examples/implicit-references.rs; run it with cargo run -p bakery_model3 --example implicit-references.

Validation at table construction

Dotted names are ordinary strings, so the compiler cannot vet them. Instead, with_active_columns returns Result and validates everything as it runs — when the table definition is constructed, typically in model or startup code, before any query is made. A mistake surfaces there, never as a silently empty column at fetch time:

• an unknown column or unknown relation in a dotted name;
• a has_many hop — traversal is has_one-only, because a to-many field is a set, not a value (for an aggregate over the set, a count or a sum, use the previous chapter’s get_count_query);
• a backend that can lower neither a correlated subquery nor a native path (MongoDB, CSV, REST) — its TableSource::supports_traversal is false, and dotted names are refused up front. Traversal is also same-datasource only; enriching rows with fields from a different datasource belongs to Dio augmentation (Dio).

Why not let a bad dotted name fail at fetch time?

Because a fetch-time failure in a projection is invisible. The query still runs, the column comes back empty, and nothing tells you whether the relation was misspelled or the data is genuinely absent. Constructing the table is the moment you have full knowledge — the relation registry, the target’s columns, the backend’s capabilities — so that’s where every check runs.

Read-only semantics

An imported column is a computed projection. No backend can honestly store client.name on the client_order table — the value lives in a different row of a different table. So imported columns carry write-path enforcement:

• They are flagged calculated in vista metadata, so downstream consumers know the column is derived (see Vistas).
• They are stripped from full-record write payloads — insert, replace, generated-id insert. A read-modify-save round-trip never persists client.name as a real field.
• They are rejected outright in a patch. A partial payload naming a read-only column is explicit intent, and silently dropping it would turn the patch into a successful no-op.

The key difference: strip vs reject. A full-record payload naturally contains everything you read — including imported columns — so stripping them is the correct interpretation. A patch payload contains only what you chose to send, so a read-only column there is a mistake worth an error.

Imported columns are also excluded from quicksearch and (on SurrealDB) not orderable — the dotted name is a projection alias, not a physical field.

Per-backend lowering

SQL backends lower dotted names through the generic chain: nested correlated scalar subqueries built on the same get_subquery_as/select_expression machinery from the previous chapter. A backend that already supports that recipe gets implicit references with nothing backend-specific to implement.

SurrealDB overrides TableSource::traversal_path_expr and emits a native idiom path instead — client.name — with each segment escaped separately. Escaping the joined path as one identifier would produce a literal ⟨client.name⟩ field lookup: a dead column that matches nothing. The idiom descends each hop’s link field, not the relation’s registry name — a relation declared with_one("owner", "client", …) still lowers to client.name. Multi-hop comes for free in the idiom.

Why not just use with_expression?

You still can, and sometimes you should. The key difference: reach for with_expression + get_subquery_as when you need an arbitrary expression — arithmetic, aggregates, raw escapes. Reach for a dotted column when you just want a related field. The dotted form adds what the manual recipe can’t:

• Proper errors — the manual recipe’s checks are Option/unwrap panics (select_column returns None for a misspelled column); the dotted form returns Result, with an error naming the bad relation, column, cardinality, or backend.
• A typed imported column — the definition is cloned from the target’s column, so type metadata travels with it.
• Enforced read-only write semantics — a with_expression column has no write-path story at all.

Gotchas

• Table::derive_from does not inherit implicit references. A derived table starts with no active set, and listing an imported dotted column in its column inheritance copies only a bare definition. Re-declare the dotted names on the derived table.
• Columns added after with_active_columns are not in the active set and won’t project — declare the active set last.
• Expression-only columns (registered via with_expression with no column definition) may be named in the active set; they stay projected.
• Same-datasource only, by design — see Dio for the cross-datasource alternative.

Conclusion

At this point you should be able to:

1. Restrict projection with plain names in with_active_columns (the id column always projects).
2. Import related fields with dotted names — one hop or many, over declared has_one relations.
3. Read them back via flat keys equal to the dotted name.
4. Predict every construction-time error — unknown column, unknown relation, has_many hop, backend without traversal support.
5. Explain the write-path semantics — stripped from full-record writes, rejected in a patch.
6. Choose the right tool — dotted column for a related field, with_expression for an arbitrary expression.

The next page, Vista, YAML and Rhai, covers how the same declarations work from YAML specs, and how the calculated flag and the can_traverse_in_columns capability surface at the erased layer.

Vista, YAML and Rhai

Everything so far assumed you could write Table<SqliteDB, Order> in your source code. Generic consumers can’t. A UI grid, an admin panel, a scripting surface, a config-driven tool — none of them know your model types at compile time. What they hold is a Vista: the schema-bearing runtime handle introduced in the Vista chapter.

The question this page answers: what happens to relations when the type is erased? The answer is that they survive in three forms:

1. Metadata — the relation names, targets, and cardinalities are introspectable, so a consumer can discover them at runtime.
2. Traversal — get_ref still works. Hand it a loaded row, get back a new Vista narrowed to the related rows.
3. Capabilities — an honest contract about which traversal forms this backend actually serves, so a consumer knows what to offer before trying.

The factory

A Vista is never constructed by hand — construction goes through a vista factory (the VistaFactory trait, covered in Vista integration). The factory’s defining ability is working by name: where typed code passes User::table(db), a generic consumer asks for "users" and gets a Vista back. How the name resolves is the factory’s business — typically it loads table specs from files — and that indirection is what lets config-driven tools, scripts, and by-name traversal address models they have no Rust types for.

Behind a name, a vista is built one of two ways:

1. From a typed table — from_table wraps the table your model already builds:

   #![allow(unused)]
   fn main() {
   let vistas = vec![
       db.vista_factory().from_table(Client::surreal_table(db.clone()))?,
       sqlite.vista_factory().from_table(Product::sqlite_table(sqlite.clone()))?,
       csv.vista_factory().from_table(Order::csv_table(csv.clone()))?,
   ];
   }

2. From a YAML spec — no Rust model at all: columns, relations, and computed fields declared as data (Config-Driven Vistas). This path is how the declarations in the last section of this page become live, traversable relations. A spec’s source doesn’t have to be a physical table, either — a rhai: block builds it from a query (a query-sourced vista), and a base: block derives it from another spec.

Whichever route constructs it, the factory folds the table’s relations into VistaMetadata — name, target, cardinality, foreign key. The erased handle carries enough to introspect the relations and to traverse them. Nothing about the relation is lost in the erasure; only the compile-time names are.

Metadata: introspecting relations

Vista exposes the relation metadata directly:

• get_references() -> Vec<String> — the relation names.
• list_references() -> Vec<(String, ReferenceKind)> — names paired with cardinality (ReferenceKind::HasOne / HasMany).
• get_reference(name) -> Option<&Reference> — the full reference entry.
• list_contained() — contained (embedded) relations with their kind.

This is what lets a UI render relation tabs or context-menu entries without knowing the model. list_references tells it “orders (has_many), country (has_one)” — enough to label a tab and decide whether the target is a single row or a set — and it learned that from the handle, not from your source code.

Traversal: get_ref on a loaded row

The erased twin of the typed get_ref_from_row:

#![allow(unused)]
fn main() {
pub fn get_ref(&self, relation: &str, row: &Record<CborValue>) -> Result<Vista>
}

Hand it a loaded row — Record<CborValue> is the erased record type — and get back a new Vista narrowed to the related rows. The relation is named by string because that’s all a generic consumer has; the metadata folded in by the factory supplies the target and foreign key.

There is also get_ref_target(relation) -> Result<Vista> — the bare, unconditioned target. This is the insert destination for nested creates: when a UI wants to add a related row, it needs the target table without any narrowing applied.

Both return Vista, so traversal chains and the consumer never leaves the erased world. A grid that drills from clients into orders into line items is calling get_ref three times, each time holding nothing more specific than a Vista and a row.

The scripting surface

The same traversal is exposed to Rhai data scripts (the vantage-vista rhai feature). A script holding a row can hop a relation:

let t = table("tag");
let row = t.get_some();
let course = t.get_ref("golf_course", row);

There’s no separate scripting implementation here — the erased Vista is what the script engine wraps, and get_ref there follows the same metadata.

Capabilities: the honest contract

Not every backend serves every traversal form — the traversal chapter already showed that set-level traversal needs subqueries. At the typed layer, you knew that when you wrote the code. At the erased layer, the consumer has to ask. That’s what VistaCapabilities is for. The flags relevant to relations:

• can_traverse_to_record — “Record-level reference traversal via get_ref(relation, row) — read the join value out of a known row and narrow the target with a plain eq-condition. Every backend that can filter by equality supports this (SQL, CSV, Mongo, Surreal, REST/GraphQL).”
• can_traverse_to_set — “Set-level reference traversal — narrow the target with an IN (subquery) derived from the parent’s own conditions (the get_ref_as / reports path). Requires the backend to support subqueries; SQL and SurrealDB do, CSV/Mongo/REST do not.”
• can_build_ref_via_script — “Per-reference Rhai-scripted traversal — a reference carrying a build_script resolves through the script engine … rather than the fixed FK eq-condition path.”
• can_traverse_in_columns — whether the backend can lower a dotted active column (country.name) into its own query; true for SQL and SurrealDB shells, false for CSV/Mongo/REST.

Consumers branch on these before offering the corresponding affordance: a report builder checks can_traverse_to_set before offering set-level aggregation; a grid checks can_traverse_in_columns before letting the user add a dotted column. The flags are also readable by name (capability_flag("can_traverse_to_set")) and appear in the rhai capabilities map, so scripts and config-driven tools can branch the same way.

Across datasources: the VistaCatalog

Everything above stays inside one datasource — a Vista’s references forward to the wrapped table, and a single Vista deliberately knows nothing about other datasources. Cross-persistence traversal lives one layer up, in the VistaCatalog (vantage-vista-factory): a name → Vista catalog spanning many datasources, plus reference traversal between the models it holds.

You register models by name — each as a ModelLoader closure that builds a fresh, unconditioned Vista, from whichever backend backs it — and then register relations between catalog models:

#![allow(unused)]
fn main() {
let mut cat = VistaCatalog::new();
cat.register("client", Arc::new(|| /* build the client Vista — one datasource */));
cat.register("bakery", Arc::new(|| /* build the bakery Vista — possibly another */));

cat.register_relation(
    "client",
    Relation::single_key("bakery", "bakery", ReferenceKind::HasOne, "id", "bakery_id"),
);

// From a loaded client row, traverse into the (possibly foreign) bakery model:
let bakery = cat.traverse(&cat.relations_for("client")[0], &client_row)?;
}

A Relation is a single-key (target.foreign_key == parent_row[narrow_via]) or multi-key join description; traverse builds the target by name and pushes one eq-condition per key. How each condition is honoured — SQL WHERE, in-memory filter, REST path/query param — is the target driver’s concern at fetch time.

traverse_from is the unified entry point: it prefers the parent Vista’s own same-persistence reference when the shell declares it and advertises can_traverse_to_record (that path stays entirely inside one driver), and otherwise falls back to a registered cross-persistence Relation. The catalog is also what the Dio layer’s augmentation uses to resolve its detail sources — the next page picks that up.

Declaring relations in YAML and Rhai

Config-driven vistas (full chapter: Config-driven vistas) declare relations in the table spec — either as column-level sugar or a top-level map. The factory reads the spec, registers the relations, and the resulting Vista traverses them exactly like one built from_table:

columns:
  - name: batch
    type: string
    references: batch        # sugar: has_one to table "batch", FK = this column
  - name: batch.name         # implicit reference through that relation
    type: string
    optional: true
references:
  tags:
    table: tag
    kind: has_many
    foreign_key: batch

The column-level references: batch sugar names the relation after the target table, which is the common case. The full form (references: { table, kind, foreign_key, name }) can name it differently — you need that when two relations point at the same table.

When a relation needs more than a plain foreign-key match — extra conditions, ordering, a search — give the reference a rhai: build script:

references:
  recent_orders:
    table: order
    kind: has_many
    foreign_key: client
    rhai: |
      table("order").add_condition_eq("client", row.id).add_order("created_at", "desc")

The script runs lazily when the relation is traversed, with the parent record in scope as row, and must return a Vista — start it with table("<name>") and chain the conventional verbs (add_condition_eq, add_order, add_search, set_page_size, with_id). Without rhai:, the relation falls back to the plain foreign_key match. This is the can_build_ref_via_script path from the capabilities list — the reference resolves through the script engine instead of the fixed eq-condition.

Imported dotted columns — like batch.name above, the YAML form of implicit references — go through the same traversal import as the Rust API, with the same construction-time validation: a bad dotted column fails the spec load, not the first fetch. They arrive in metadata flagged calculated. That flag means read-only for consumers: the value comes from a traversal, not from a column you can write. This is how a UI knows not to offer editing on them.

The key difference: at the typed layer you know which traversal forms are safe — you wrote the code against a backend you chose. At the erased layer, the capabilities say so. Same relations, discovered instead of known.

Conclusion

At this point you should be able to:

1. Enumerate a Vista’s relations — get_references, list_references with cardinality, get_reference for the full entry, list_contained for embedded relations.
2. Traverse from a loaded erased row — get_ref(relation, row) returns a narrowed Vista; get_ref_target(relation) returns the bare target for nested creates.
3. Branch on the four traversal capabilities — can_traverse_to_record, can_traverse_to_set, can_build_ref_via_script, can_traverse_in_columns — before offering an affordance.
4. Traverse across datasources — register models and Relations on a VistaCatalog; traverse_from prefers the same-persistence reference and falls back to the catalog join.
5. Declare relations in YAML and Rhai — column-level references: sugar, the full top-level form, implicit dotted columns through a relation, and rhai: build scripts for traversals that need more than an FK match — all consumed by the same vista factory that wraps typed tables.
6. Explain why calculated columns are read-only in generic UIs — their values come from traversal, not from a writable column.

Next: what happens above the Vista — combining erased handles across backends.

Relations and Dio

Everything in this guide so far has executed at or below the Vista: typed relations on the Table, traversal narrowing, subquery expressions, implicit references. This final chapter climbs one layer higher. A Dio keeps a live local copy of a data segment (a cache), reads through a Lens, announces changes on an event bus, and serves reactive views (Scenery). You built one in the intro paths (Dio over SQL, Scenery).

Two questions matter here: what happens to the relation machinery you already know once a Dio sits on top — and what relation shape exists only at the Dio layer, because nothing below it can express one.

Same-persistence relations resolve beneath the Dio

Traversal, subquery expressions, and implicit references all happen in the Table or Vista under the Dio. The Dio caches the resulting rows like any others. If your table declares with_active_columns(&["id", "client.name"]), the client.name column is already part of each row when it enters the cache — nothing at the Dio layer knows or cares that it was traversed. From the cache’s point of view, a traversed column and a plain column are indistinguishable.

The practical consequence is the master/detail pattern in a UI: traverse at the Vista, cache at the Dio. Take a row from the cached master, call Vista::get_ref(relation, row) to build the narrowed detail Vista — the same erased traversal from the Vistas chapter, or VistaCatalog::traverse_from when the detail may live in another datasource — and put a Dio in front of that for the detail pane. Each narrowed detail set gets its own cache entry, so switching between master rows switches between already-cached detail sets rather than re-narrowing one shared cache.

The join the layers below cannot express

The VistaCatalog can already navigate across datasources — hand it a parent row and it returns the related set from another backend. What nothing below the Dio can do is merge: show master rows whose columns come from two sources at once. That is a join, and joins are governed by the capability contract — a SQL Vista can join its own tables, a REST Vista cannot, and two different backends can never push a join down to either engine. A Vista also has nowhere to put stitched rows: no cache, no viewport, no notion of “only visible rows pay”.

Where the backend can’t, the layer above fills the gap — that is the whole job of the Dio. Augmentation wires two Vistas into one Dio: a master that is listed, and a detail source loaded one row at a time and merged on top. The detail is resolved by name through the VistaCatalog, so it is persistence-agnostic — a REST master enriched by a cmd detail, or a SQL master enriched from another database.

It runs in two passes:

1. List pass — the master is listed cheaply; each row enters the cache marked Incomplete.
2. Detail pass — viewport-driven: for each visible incomplete row, the augmentation resolves its detail Vista, fetches the matching record, merges the chosen columns, and the row flips to Fresh.

Rows off-screen are never fetched; hydrated rows are never re-fetched. A failed detail fetch marks only that row.

Declaring an augmentation

Augmentations are configured on the Lens. The catalog is what makes the detail side resolvable by name; supplying at least one augmentation is what engages the two-pass behavior:

#![allow(unused)]
fn main() {
let lens = Lens::new()
    .cache_at(cache_path)
    .catalog(catalog)                       // resolves `table:` names
    .augment(vec![augmentation])            // ≥1 engages two-pass
    .build()?;
}

Each augmentation answers four questions — which detail Vista, how a master row selects its detail, how it is fetched, and which columns to lift:

#![allow(unused)]
fn main() {
Augmentation {
    table:  "tfstate_detail".into(),  // catalog name of the detail Vista
    source: Source::Column { from: "key".into(), to: None },
    fetch:  Fetch::PerRow,
    merge:  MergeRule { columns: vec!["resources".into(), "serial".into()] },
}
}

• source picks how a master row selects its detail: Source::Id matches master.id → detail.id; Source::Column { from, to } matches master[from] → detail[to], or detail.id when to is None; Source::Build(closure) performs arbitrary narrowing from the whole row, and is per-row only.
• fetch picks how the detail is read: Fetch::PerRow today; Batched is planned.
• merge.columns lists the detail columns to lift. Empty means all; on a name clash the detail wins.

The same declaration works from YAML:

augment:
  - table: tfstate_detail
    source: { kind: column, from: key }
    fetch:  { kind: per_row }
    merge:  [resources, serial, outputs]

To see the two passes run, try the example — two in-memory Vistas, list pass then detail pass:

cargo run -p vantage-diorama --example augmentation

The full treatment of augmentation — merge rules, capability reasoning, the planned Batched fetch and keyed caching — is in the Augmentation chapter; the tutorial form is intro step 6.

Why not model the detail as a foreign-key relation?

A relation resolves within one persistence and can be pushed down as a join. The detail here may be a different backend entirely; there is nothing to join on and no engine that could honour it. Augmentation is the cross-Vista form, stitched by the Dio.

Choosing between implicit references and augmentation

Both produce the same user-visible result: a row enriched with related fields. Which one to use depends on where the data lives and who should pay for the fetch.

Path A: implicit references (same datasource, in-query)

One query; the backend does the work. The value is present on every row, including off-screen ones, it is validated when the table is constructed, and it carries read-only column semantics. Choose this when both tables live in one datasource that supports traversal (SQL, SurrealDB) — see Implicit References.

Path B: augmentation (any two datasources, per-row client-side join)

Two passes; expensive work follows the user’s attention — only viewport rows pay, and only once. It works across arbitrary backends, and failed details degrade per-row, not per-page. Choose this when the sources differ, or when the detail is expensive and you want viewport-driven hydration even within one backend.

The key difference: implicit references push the join down into the engine; augmentation lifts it up into the cache. Same user-visible result — a row enriched with related fields — chosen by where the data lives and who should pay for the fetch.

Guide conclusion

This closes the Relations & Traversal guide. At this point you should be able to:

1. Say where each relation form executes — traversal and implicit references in the engine, erased traversal through the Vista handle, augmentation in the Dio’s cache.
2. Build master/detail — Vista::get_ref(relation, row) for the narrowed detail Vista, with a Dio (and its own cache entry) per detail set.
3. Declare an augmentation — catalog + source + fetch + merge — in Rust or YAML.
4. Choose deliberately between implicit references and augmentation based on where the data lives and who should pay for the fetch.

Relations in Vantage are declared once on the model and then travel — into queries, through erasure, and up into the cache — each layer serving the form the one below can’t.

Model-Driven Architecture

Vantage is opinionated about how you structure business software. The framework prescribes a Model-Driven Architecture (MDA) where entities, relationships, and business rules live in a shared model crate — decoupled from any specific persistence, UI, or API layer.

• The idea
• Anatomy of a model crate
• Defining entities
• Table constructors
• Relationships
• Computed fields
• Implicit references — dotted columns
• Connection management
• Using the model
• Type-erased access
• The layered architecture

The idea

Most Rust projects scatter database queries across handlers, services, and utilities. When the schema changes or a new backend is needed, you’re hunting through dozens of files.

Vantage inverts this. You define your entities once in a model crate, then every consumer — REST API, CLI, desktop UI, background worker — uses the same Table and DataSet interfaces. The model is the source of truth.

graph TD
    M[Model Crate] --> API[Axum API]
    M --> CLI[CLI Tool]
    M --> UI[Desktop UI]
    M --> Worker[Background Worker]
    M --> Test[Mock Tests]
    style M fill:#4a7c59,color:#fff

Anatomy of a model crate

The bakery_model3 crate in the Vantage repo demonstrates the pattern. Here’s how it’s structured:

bakery_model3/
 ├── src/
 │    ├── lib.rs          ← re-exports, DB connection helpers
 │    ├── bakery.rs       ← Bakery entity + table constructors
 │    ├── client.rs       ← Client entity + table constructors
 │    ├── order.rs        ← Order entity + table constructors
 │    └── product.rs      ← Product entity + table constructors
 └── examples/
      ├── cli.rs          ← multi-source CLI using Vista
      └── 0-intro.rs      ← direct SurrealDB queries

Each entity file follows the same pattern: struct → trait impls → table constructors.

Defining entities

An entity is a plain Rust struct. The #[entity(...)] macro generates Record conversions for each persistence’s type system:

#![allow(unused)]
fn main() {
#[entity(CsvType, SurrealType, SqliteType, PostgresType, MongoType)]
#[derive(Debug, Clone, PartialEq, Default)]
pub struct Client {
    pub name: String,
    pub email: String,
    pub contact_details: String,
    pub is_paying_client: bool,
    pub bakery_id: Option<String>,
}
}

One struct, five type systems. The same Client works with CSV files, SurrealDB, SQLite, Postgres, and MongoDB — each using its own native Record<AnyType> representation.

Table constructors

Each entity provides table constructors per persistence. These are plain functions that return Table<DB, Entity> with columns, relationships, and computed fields pre-configured:

#![allow(unused)]
fn main() {
impl Client {
    pub fn csv_table(csv: Csv) -> Table<Csv, Client> {
        Table::new("client", csv)
            .with_column_of::<String>("name")
            .with_column_of::<String>("email")
            .with_column_of::<bool>("is_paying_client")
            .with_column_of::<String>("bakery_id")
            .with_one("bakery", "bakery_id", Bakery::csv_table)
            .with_many("orders", "client_id", Order::csv_table)
    }

    pub fn surreal_table(db: SurrealDB) -> Table<SurrealDB, Client> {
        Table::new("client", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<String>("email")
            .with_column_of::<bool>("is_paying_client")
            .with_one("bakery", "bakery", Bakery::surreal_table)
            .with_many("orders", "client", Order::surreal_table)
            .with_expression("order_count", |t| {
                let orders = t.get_subquery_as::<Order>("orders").unwrap();
                orders.get_count_query()
            })
    }

    pub fn sqlite_table(db: SqliteDB) -> Table<SqliteDB, Client> {
        Table::new("client", db)
            .with_id_column("id")
            .with_column_of::<String>("name")
            .with_column_of::<String>("email")
            .with_column_of::<bool>("is_paying_client")
            .with_column_of::<String>("bakery_id")
            .with_one("bakery", "bakery_id", Bakery::sqlite_table)
            .with_many("orders", "client_id", Order::sqlite_table)
    }
}
}

Notice how the shape is consistent but details differ — SurrealDB uses embedded document references ("bakery" not "bakery_id"), and only SurrealDB gets the computed order_count expression field (which requires correlated subquery support).

Relationships

Relationships are declared on the table, not the entity. Two methods:

• with_one("name", "fk_field", constructor) — foreign key to parent (many-to-one)
• with_many("name", "fk_field", constructor) — parent to children (one-to-many)

Traversal is synchronous and returns a new Table with conditions applied:

#![allow(unused)]
fn main() {
let paying = Client::surreal_table(db)
    .with_condition(clients["is_paying_client"].eq(true));

// Traverse — returns Table<SurrealDB, Order> with subquery condition
let orders = paying.get_ref_as::<SurrealDB, Order>("orders").unwrap();

// The generated query filters orders by paying clients automatically
for (id, order) in orders.list().await? {
    println!("{}: {}", id, order.total);
}
}

Computed fields

with_expression adds fields that don’t exist in the database — they’re computed via correlated subqueries:

#![allow(unused)]
fn main() {
.with_expression("order_count", |t| {
    let orders = t.get_subquery_as::<Order>("orders").unwrap();
    orders.get_count_query()
})
// SELECT *, (SELECT COUNT(*) FROM order WHERE order.client = client.id)
//   AS order_count FROM client
}

These fields appear in ReadableValueSet results alongside physical columns.

Implicit references — dotted columns

Surfacing a related row’s field is common enough that it has a declarative form. A dotted name in with_active_columns traverses declared has_one relations and imports the target’s field as a read-only column, aliased under the literal dotted name — no hand-written expression:

#![allow(unused)]
fn main() {
let orders = Order::sqlite_table(db)
    .with_active_columns(&["id", "client.name", "client.bakery.name"])?;
// SELECT id,
//   (SELECT name FROM client WHERE client.id = client_order.client_id) AS "client.name",
//   (SELECT (SELECT name FROM bakery WHERE bakery.id = client.bakery_id)
//      FROM client WHERE client.id = client_order.client_id)           AS "client.bakery.name"
// FROM client_order
}

client.name is one hop; client.bakery.name recurses through two. Each backend lowers the traversal into its own query — SQL nests correlated scalar subqueries, SurrealDB emits a native idiom path (client.name, each segment escaped separately). The row comes back with a flat key equal to the dotted name (row.get("client.bakery.name")).

A plain (non-dotted) entry simply restricts projection to that column — with_active_columns is also how you narrow a wide table to the columns you actually read. Everything is validated when the table is built, so mistakes surface immediately rather than at fetch time:

• an unknown column or relation is a build-time error;
• a has_many hop is rejected (traversal is has_one-only — a to-many field is a set, not a value);
• a backend that can lower neither a subquery nor an idiom path (e.g. MongoDB, CSV, REST) refuses dotted names up front; same-datasource only.

Imported columns are read-only: they are flagged calculated for consumers, stripped from full-record write payloads (insert, replace, generated-id insert) so a read-modify-save round-trip never tries to persist client.name as a real field, and rejected outright in a patch — a partial payload naming a read-only column is explicit intent, and silently dropping it would turn the patch into a successful no-op. This is the declarative counterpart to the manual with_expression + get_subquery_as recipe above — reach for with_expression when you need an arbitrary expression, and a dotted column when you just want a related field. The full treatment — traversal forms, validation, write semantics, per-backend lowering, and how relations surface on Vistas and Dio — is the Relations & Traversal guide.

Connection management

The model crate owns database connections. bakery_model3 uses a OnceLock<SurrealDB> pattern for global access, with a DSN-based connection function:

#![allow(unused)]
fn main() {
pub async fn connect_surrealdb() -> Result<()> {
    let dsn = std::env::var("SURREALDB_URL")
        .unwrap_or_else(|_| "cbor://root:root@localhost:8000/bakery/v2".into());

    let client = SurrealConnection::dsn(&dsn)?
        .connect().await?;

    set_surrealdb(SurrealDB::new(client))
}
}

Consumers call connect_surrealdb() once at startup, then use Client::surreal_table(surrealdb()) anywhere.

Using the model

Once the model crate exists, consumers are simple. They don’t know or care about SQL, SurrealQL, or BSON — they work with entities and tables:

#![allow(unused)]
fn main() {
// In an Axum handler
async fn list_clients() -> Json<Vec<Client>> {
    let clients = Client::surreal_table(surrealdb())
        .with_condition(clients["is_paying_client"].eq(true));
    Json(clients.list().await.unwrap().into_values().collect())
}

// In a CLI tool
let table = Client::sqlite_table(db);
println!("{} clients", table.get_count().await?);

// In a test — no database needed
let mock = MockTableSource::new()
    .with_data("client", test_data).await;
let table = Table::<MockTableSource, Client>::new("client", mock);
assert_eq!(table.get_count().await?, 3);
}

Type-erased access

For truly generic code (UI grids, admin panels, config-driven tools), wrap tables in a Vista — the schema-bearing handle that replaced the removed AnyTable in 0.5:

#![allow(unused)]
fn main() {
let vistas = vec![
    db.vista_factory().from_table(Client::surreal_table(db.clone()))?,
    sqlite.vista_factory().from_table(Product::sqlite_table(sqlite.clone()))?,
    csv.vista_factory().from_table(Order::csv_table(csv.clone()))?,
];

// Same code handles all three — different databases, same interface
for v in &vistas {
    println!("{} records", v.list_values().await?.len());
}
}

See also: bakery_model3/examples/cli.rs for a complete multi-source CLI built on this pattern.

The layered architecture

Putting it all together, Vantage prescribes four layers:

┌─────────────────────────────────────────────────┐
│  4. Consumers                                   │
│     Axum API · CLI · egui · Tauri · gRPC        │
├─────────────────────────────────────────────────┤
│  3. Business Logic                              │
│     Traits on Table · custom methods · rules    │
├─────────────────────────────────────────────────┤
│  2. Model Crate                                 │
│     Entities · Table constructors · Relations   │
├─────────────────────────────────────────────────┤
│  1. Persistence                                 │
│     SurrealDB · Postgres · SQLite · CSV · API   │
└─────────────────────────────────────────────────┘

Layer 1 is implemented once per database (or use an existing Vantage crate). Layer 2 is your model crate — entity definitions, relationships, computed fields. Layer 3 adds business-specific traits and methods on top of Table. Layer 4 is any number of consumers that import the model and don’t think about persistence.

Three Paths for Developers

Vantage gives you three ways to work with data. Each path builds on the previous one, and all three converge into a unified data abstraction layer.

• ① Entity Framework
• ② Query Building
  • SQLite — CASE, UNION, window functions
  • PostgreSQL — DISTINCT ON, LATERAL JOIN, array ops
• ③ Custom Persistences
• Convergence

graph TD
    EF["① Entity Framework<br/>CRUD over business objects"]
    QB["② Query Building<br/>Vendor-specific SQL power"]
    CP["③ Custom Persistences<br/>APIs, caches, reactive data"]
    EF --> DAL["Universal Data Abstraction Layer"]
    QB --> DAL
    CP --> DAL
    DAL --> E["Export: FFI · API · UI · SDK"]
    style EF fill:#4a7c59,color:#fff
    style QB fill:#2d6a8f,color:#fff
    style CP fill:#8f5a2d,color:#fff
    style DAL fill:#7c2d8f,color:#fff
    style E fill:#555,color:#fff

① Entity Framework

The most common path. Define entities, build tables, use DataSet and ActiveEntity for CRUD — your code never touches SQL or any query language.

#![allow(unused)]
fn main() {
// Define once
#[entity(SurrealType, SqliteType, PostgresType)]
struct Product {
    name: String,
    price: i64,
    is_deleted: bool,
}

// Use everywhere
let table = Product::surreal_table(db);
let expensive = table.with_condition(table["price"].gt(200));
let count = expensive.get_count().await?;

let mut item = table.get_entity(&id).await?.unwrap();
item.price = 350;
item.save().await?;
}

This is the bread and butter of enterprise development — hundreds of entities, each with relationships, computed fields, and business rules, all persistence-agnostic and testable with mocks.

② Query Building

When you need the full power of your database — JOINs, CTEs, window functions, JSONB operators, array aggregation, recursive queries — drop into the vendor-specific query builder.

SQLite — CASE, UNION, window functions

#![allow(unused)]
fn main() {
use vantage_sql::primitives::case::Case;

let select = SqliteSelect::new()
    .with_source("users")
    .with_field("name")
    .with_field("salary")
    .with_expression(
        Case::new()
            .when(sqlite_expr!("{} >= {}", (ident("salary")), 100000.0f64),
                  sqlite_expr!("{}", "senior"))
            .when(sqlite_expr!("{} >= {}", (ident("salary")), 60000.0f64),
                  sqlite_expr!("{}", "mid"))
            .else_(sqlite_expr!("{}", "junior"))
            .as_alias("band"),
    )
    .with_order(ident("salary"), Order::Desc);
}

PostgreSQL — DISTINCT ON, LATERAL JOIN, array ops

#![allow(unused)]
fn main() {
let select = PostgresSelect::new()
    .with_distinct_on(ident("user_id").dot_of("o"))
    .with_source_as("orders", "o")
    .with_expression(ident("name").dot_of("u"))
    .with_expression(ident("id").dot_of("o").with_alias("order_id"))
    .with_expression(ident("total").dot_of("o"))
    .with_join(PostgresSelectJoin::inner(
        "users", "u",
        ident("id").dot_of("u").eq(ident("user_id").dot_of("o")),
    ))
    .with_order(ident("user_id").dot_of("o"), Order::Asc)
    .with_order(ident("created_at").dot_of("o"), Order::Desc);
}

Both examples are type-safe — parameters are bound with the correct CBOR type markers, identifiers are quoted for the target dialect, and results deserialize into typed structs. See the full test suites: SQLite complex queries, PostgreSQL complex queries.

③ Custom Persistences

Vantage isn’t limited to databases. Implement the persistence traits for anything that stores or produces data:

• REST APIs — vantage-api-client wraps any paginated JSON API as a read-only TableSource
• API pools — vantage-api-pool adds connection pooling, prefetching, and rate limiting
• CSV files — vantage-csv reads structured files with in-memory conditions
• Local caches — vantage-live syncs fast cache (ImTable, ReDB) with slow backend (Postgres, SurrealDB) for responsive UIs
• Message queues — implement InsertableDataSet for append-only sources like Kafka topics
• Mixed sources — read from SQL, write through a queue, cache in memory — same Table interface

#![allow(unused)]
fn main() {
// Read from a REST API
let api_products = Product::api_table(RestApi::new("https://api.example.com/products"));

// Cache locally for fast UI
let live = LiveTable::new(
    Product::surreal_table(db),   // permanent backend
    ImTable::new(&cache, "products"),  // fast cache
);

// Both implement the same traits — your UI code doesn't care
let products: Vec<Product> = live.list().await?.into_values().collect();
}

Convergence

All three paths converge into the same data abstraction layer. Entity framework tables, raw query results, and custom persistence data all flow through Table, DataSet, and Vista — giving you one interface for the entire organisation’s data.

┌──────────────────────────────────────────────────────────────┐
│                  Data Abstraction Layer                       │
│                                                              │
│   Table<SurrealDB, Client>    ← entity framework             │
│   Table<PostgresDB, Report>   ← complex queries              │
│   Table<RestApi, ExternalData> ← custom persistence          │
│   Table<LiveTable, Product>   ← cached + reactive            │
│                                                              │
│   All implement: DataSet · ValueSet · ActiveEntitySet         │
│   All wrap into: Vista (type-erased, generic code)          │
└──────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
          Axum API        egui/Tauri       FFI / SDK

This is where Rust’s type system pays off at scale. Every Table is checked at compile time. Every persistence has strict type boundaries. Every consumer works against abstract traits. Add a new database, API, or cache — nothing else changes. Add a new UI framework or language binding — the model stays the same.

Config-Driven Vistas: YAML & Rhai

The Vista chapter built Vistas by wrapping typed tables — Rust code, compiled in. This page covers the other construction path: declare the Vista in YAML, load it at runtime, and seal it behind the same honest handle. Change the YAML, rebuild the Vista, and the consumer sees the new shape — no recompiling.

This is the path configuration-driven tooling takes: admin panels reading a folder of model files, AI agents writing schema on a user’s behalf, and any application (Vantage UI among them) whose data layer is user-editable data rather than user-compiled code. For everything YAML can’t express — vendor expressions, derived queries, scripted traversal — there’s an optional Rhai layer that compiles to native queries.

• One page of YAML, one working Vista
• Anatomy of a spec
  • Column types
  • Typos fail at parse time
  • Column mapping per driver
• References
  • The spec resolver
• The Rhai layer
  • 1. Query-sourced vistas — rhai:
  • 2. Derived vistas — base: + inherit + transform
  • 3. Post-build tweaks — modify: (SurrealDB)
  • 4. Scripted reference traversal — reference-level rhai: (SurrealDB)
• Contained relations
• Cross-persistence: the VistaCatalog
• Sealed at runtime: data scripts
• Choosing a path

One page of YAML, one working Vista

Every driver’s factory implements VistaFactory, whose from_yaml parses a spec and lowers it:

#![allow(unused)]
fn main() {
let yaml = r#"
name: product_view
columns:
  id:
    type: string
    flags: [id]
  name:
    type: string
    flags: [title, searchable]
  price:
    type: int
sqlite:
  table: product
"#;

let vista = db.vista_factory().from_yaml(yaml)?;

assert_eq!(vista.name(), "product_view");
assert_eq!(vista.get_id_column(), Some("id"));
let rows = vista.list_values().await?;   // reads the `product` table
}

The factory builds a typed Table under the hood — each type: becomes a real Column<T> — and wraps it exactly as from_table would. From here on, nothing downstream can tell how the Vista was made: same schema introspection, same conditions, same capability contract.

Two implicit rules worth knowing:

• The Vista’s name is the spec’s name, not the storage name. product_view is what a UI tab or a catalog key sees; sqlite: { table: product } is where the rows live. Omit the block and the spec name doubles as the table name.
• The id column resolves in a fixed order: explicit id_column: at the top level wins, then the first column flagged id, then a driver default ("id" for SQL, "_id" for MongoDB). The explicit key exists so a YAML author can correct a wrong flag without touching anything else.

Anatomy of a spec

A VistaSpec has a uniform core that every driver understands, plus one driver-named block at each level:

Column types

type: names lower to real typed columns; unknown names are a parse error, not a fallback:

flags: is the same open vocabulary the Vista chapter introduced — id, title, searchable, orderable, hidden, mandatory — and drives the same consumer behaviour (title columns label rows, searchable feeds quicksearch, hidden drops out of default views).

Typos fail at parse time

Every driver block sets deny_unknown_fields, so sqlite: { tabel: product } is rejected when the YAML is parsed — with the offending key in the message — instead of silently falling back to a default and failing later at query time. Driver-block validation follows the same rule: an empty nested_path, a a..b path, mutually-exclusive options — all surface as parse errors naming the column.

Column mapping per driver

When the spec column name differs from the storage name, the column’s driver block maps it:

CSV is also the one driver whose table block is mandatory — csv: { path: data/products.csv } — because without a path there is no file.

References

A relation to another vista, in the same uniform vocabulary the typed with_one/with_many uses:

name: category
columns:
  id: { type: string, flags: [id] }
  name: { type: string, flags: [title] }
references:
  products:
    table: product
    kind: has_many
    foreign_key: category_id

Three shapes are available:

• Full form (above): table, kind (has_one / has_many), foreign_key.

• Column sugar — when the foreign key is the column, declare it inline:

  columns:
    category_id:
      type: string
      references: category      # has_one, foreign_key = the column itself
  

• Multi-key joins — keys: lists { to, from } pairs when the child is narrowed by more than one parent field (a deployment matched on both product_id and version_id).

The spec resolver

A reference names its target by spec name — so the factory needs a way to find the target’s current spec at traversal time. That’s the spec resolver, attached once:

#![allow(unused)]
fn main() {
let resolver: SqliteSpecResolver = Arc::new(move |name| specs.get(name).cloned());
let factory = db.vista_factory().with_resolver(resolver);
let category = factory.from_yaml(category_yaml)?;

// Traversal rebuilds `product` from its live spec:
let row = /* a category row */;
let products = category.get_ref("products", &row)?;
}

The resolver is a plain closure — back it with an in-memory map, a folder of .yaml files, or a hot-reloading inventory. Because targets are re-resolved on every traversal, editing a spec changes what the next traversal builds. Without a resolver, a traversal falls back to a column-less target and the next query fails loudly.

The Rhai layer

YAML declares structure. For anything with an expression in it, specs escalate to Rhai — a small, embeddable scripting language whose Vantage vocabulary compiles to native queries. The expression primitives are shared across backends where the concept overlaps (count, avg, coalesce, case_when, date_format …) — see SQL Primitives and SurrealDB Primitives for the full vocabularies. Rhai appears in a spec in four places, each with a distinct job.

1. Query-sourced vistas — rhai:

Replace the physical table with a script-built SELECT:

name: expensive_products
columns:
  id: { type: string, flags: [id] }
  name: { type: string }
sqlite:
  rhai: |
    select().from("product").field("id").field("name").where(expr("price > 150"))

The script runs once at build time and its SELECT becomes the vista’s source. A query-sourced vista is read-only — the factory clears can_insert / can_update / can_delete, because there’s no single table a write could honestly land in. Capability honesty is preserved automatically; consumers find out by checking, same as ever.

2. Derived vistas — base: + inherit + transform

Derive one vista from another, inheriting schema and transforming the query. The base’s select() is seeded into the script’s scope as base:

name: category_totals
id_column: category_id
columns:
  total_price: { type: int }
sqlite:
  base: product
  inherit:
    columns: [category_id]
  rhai: |
    base.clear_fields().field("category_id")
        .expression(expr("SUM(price) AS total_price"))
        .group_by(expr("category_id"))

This is the aggregate pattern: group the base by a key, declare the aggregate outputs as the derived vista’s own columns:, and re-key with id_column. base: resolves eagerly through the same spec resolver as references. Derived vistas are query-sourced, hence read-only.

3. Post-build tweaks — modify: (SurrealDB)

A script applied to the finished vista, exposed as self, using the builder verbs plus vendor expressions YAML keys can’t state:

name: active_products
columns:
  id: { type: thing, flags: [id] }
  name: { type: string, flags: [title] }
surreal:
  table: product
  modify: |
    self.with_condition(ident("is_deleted") == false)
        .add_order("name", "asc")

Unlike a query source, modify: narrows a real table — so the vista stays writable. It runs last, composing with table:, rhai:, or base:.

4. Scripted reference traversal — reference-level rhai: (SurrealDB)

When a relation can’t be expressed as a foreign-key equality — a graph edge, a computed join — the reference carries its own build script, evaluated lazily at traversal time with the parent row in scope:

references:
  products:
    table: product
    kind: has_many
    foreign_key: category
    surreal:
      rhai: |
        table("product").add_condition_eq("category", row.id)

The script returns the narrowed target vista; foreign_key remains as metadata for consumers that introspect the relation.

Contained relations

Embedded objects and arrays — an order’s lines, a JSON column — declare as a contained: section and surface as editable sub-vistas:

name: order
columns:
  id: { type: string, flags: [id] }
  lines: { type: string }          # the host column — declare it so it's selected
contained:
  lines:
    host_column: lines
    kind: contains_many
    id_column: line_id             # omit for positional ids
    columns:
      product: { type: string }
      quantity: { type: int }

Reads project the embedded collection into records; writes patch it back into the host column. The mechanics — and the sharp edges — are covered in Contained Relations.

Cross-persistence: the VistaCatalog

A single Vista is strictly single-backend. When a system spans several — categories in Postgres, products behind a REST API — VistaCatalog sits one layer up: it holds a loader per model name and traverses relations whose target lives in a different persistence:

#![allow(unused)]
fn main() {
let mut catalog = VistaCatalog::new();
catalog.register("category", Arc::new(move || pg_factory.from_yaml(&category_yaml)));
catalog.register("product",  Arc::new(move || api_factory.from_yaml(&product_yaml)));
catalog.register_relation("category", Relation::single_key(
    "products", "product", ReferenceKind::HasMany,
    "category_id",   // target column to constrain
    "id",            // parent-row field whose value narrows it
));

let category = catalog.build_vista("category")?;
let row = /* a category row */;
let products = catalog.traverse_from("category", "products", &row)?;  // Postgres → REST
}

Loaders return a fresh, unconditioned Vista on every call, so the catalog composes with hot-reloading inventories the same way spec resolvers do. The catalog is also what Augmentation uses to resolve its detail sources — the augment: block in a table’s configuration is this same machinery pointed at row enrichment.

Sealed at runtime: data scripts

Once vistas are config-defined, the last step of the story is consuming them from config too. vantage-vista’s rhai feature ships run_script — a sandboxed evaluator where table(name) resolves through a catalog-style resolver and a handful of read verbs fetch data:

let o = table("orders").add_condition_eq("status", "unpaid").get_some();
if o != () {
    table("orders").get_ref("client", o).get_some()
}

The vocabulary is deliberately small: builder verbs (add_condition_eq, add_order, get_ref …) plus terminals — list, get_some, count, capabilities, columns, references. Every list() is capped (50 rows hard ceiling) — this is an inspection and automation surface, not a bulk reader. It’s the surface an AI agent or an MCP tool drives: the schema it sees, the capabilities it must respect, and the rows it reads all come from the same sealed handles this page built.

Choosing a path

Both construction paths produce the same Vista — the choice is about who edits the definition:

• Typed from_table when the model is code: business logic, compile-time safety, entity structs, with_expression closures. This is the path the introduction guide walks.
• YAML (+ Rhai) when the model is data: inventories on disk, user- or agent-edited schema, hot-reload, no recompiling. Structure in YAML; expressions in Rhai; capabilities sealed either way.

They mix freely — a catalog can hold typed-table loaders next to YAML loaders, and a YAML spec’s reference can resolve to either. If you’re implementing this machinery for a new backend, the driver-side walkthrough is Adding a New Persistence, Step 8.

Live Data: Push, Poll & Freshness

A cache only stays correct if it finds out when the underlying data changes. There are two ways it can: it asks (poll), or the database tells it (push). dio.watch() is the same call either way — it asks the master Vista whether it can push, subscribes if so, and quietly does nothing if not.

That last part is the bit worth knowing before you pick a backend. watch() returning Ok(()) does not mean you have a live feed; it means nothing went wrong. This page says exactly what each shipped driver delivers today, and what — if anything — you have to do to get it.

• Three freshness models
• What each backend delivers today
• SpacetimeDB — filtered fine push, no setup
• SurrealDB — fine push, no setup
• PostgreSQL — coarse push, and why you must ask for it
• What a consumer must assume
• Adding push to a driver

Three freshness models

Every backend answers two questions, and the pair decides what you get:

1. Can it tell you something changed at all? A file on disk cannot. A database server can.
2. When it does, does it hand you the row, or only a signal?

Fine push is the only model where a change costs you no query. Coarse push still beats polling: you re-read because something happened, not on the off-chance.

There is a fourth position, currently held by one backend. Fine push still leaves a question open — does this row belong in my filtered set? — because a driver that watches the whole table will hand you rows your conditions exclude. Filtered fine push closes it: the conditions go into the subscription itself and the database maintains the result incrementally, so a row that changes out of the set arrives as a delete. Membership stops being something anyone has to reconcile.

What each backend delivers today