mirror of
https://github.com/storytold/artcraft.git
synced 2026-10-20 17:02:53 -04:00
rename CLAUDE.md to AGENTS.md (#1940)
Co-authored-by: Hanashi <shared2@storyteller.ai>
This commit is contained in:
Brandon Thomas
co-authored by
Hanashi
Hanashi
parent
ed57c6de47
commit
7fb81500a7
@@ -0,0 +1,97 @@
|
||||
# AGENTS.md
|
||||
|
||||
This file provides guidance for coding agents when working with the Artcraft monorepo.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Artcraft is a web and desktop application for generating AI image and video. It is written in
|
||||
Rust and TypeScript and contains desktop, server, and frontend components.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
artcraft/
|
||||
├── _database/ # Schema definitions and migrations (MySQL, SQLite, Elasticsearch, etc.)
|
||||
│ ├── elasticsearch/ # Elasticsearch schema and queries
|
||||
│ └── sql/ # MySQL and SQLite schema definitions and migrations
|
||||
│ ├── artcraft_migrations/ # ArtCraft desktop app SQLite migrations
|
||||
│ ├── migrations/ # Server MySQL migrations
|
||||
│ └── migrations_squashed/ # Fully materialized MySQL schema definitions for most tables
|
||||
├── _tools/ # Various 3rd party tool integrations and configurations
|
||||
│ └── postman/ # Postman configs for test HTTP requests against development and production
|
||||
├── build/ # Dockerfile build instructions for server components
|
||||
├── crates/ # Rust workspace
|
||||
│ ├── api_clients/ # HTTP clients for calling internal and 3rd party services
|
||||
│ ├── cli/ # Command line tools
|
||||
│ ├── desktop/ # Desktop (Tauri) apps
|
||||
│ │ └── artcraft/ # (Important) ArtCraft, the desktop app. This is one of our main pieces of software
|
||||
│ ├── lib/ # Various utility libraries for servers, CLI tools, desktop, etc.
|
||||
│ ├── schema/ # Data definition layer: MySQL, SQlite, Redis, S3/R2 buckets, etc.
|
||||
│ │ ├── buckets/ # Declares R2 cloud bucket topology
|
||||
│ │ ├── database/ # MySQL, SQLite, Elasticsearch, Redis, etc.
|
||||
│ │ │ ├── elasticsearch_schema/ # Elasticsearch
|
||||
│ │ │ ├── migration/ # (deprecated) Online schema adapters for MySQL
|
||||
│ │ │ ├── mysql_queries/ # Sqlx MySQL queries for our backend monolith `storyteller-web`, jobs, etc.
|
||||
│ │ │ ├── redis_common/ # Redis support
|
||||
│ │ │ ├── redis_schema/ # Redis key and HKEY topology
|
||||
│ │ │ ├── sqlite_queries/ # (deprecated) Sqlite queries
|
||||
│ │ │ └── sqlite_tasks/ # Queries for the ArtCraft desktop app's "tasks" database.
|
||||
│ │ ├── public/ # Token identifier and enum variant definitions
|
||||
│ │ │ ├── composite_identifier/ # MySQL composite key system
|
||||
│ │ │ ├── enums/ # MySQL "enums" stored in VARCHAR fields.
|
||||
│ │ │ └── tokens/ # Primary database identifiers with Stripe-like ID prefixes, eg. "user_{entropy}"
|
||||
│ │ └── service/ # Backend HTTP services and jobs
|
||||
│ │ ├── job/ # Backend jobs.
|
||||
│ │ │ └── video_thumbnail_job/ # Render video thumbnails
|
||||
│ │ ├── plugins/ # Collections of reusable Actix-Web HTTP functions (user and billing systems)
|
||||
│ │ └── web/ # HTTP web servers
|
||||
│ │ └── storyteller_web/ # (Important) Our main HTTP API monolith and backend.
|
||||
│ └── frontend/ # Nx typescript monorepo for our websites and Tauri desktop apps
|
||||
│ ├── apps/ # Websites and Tauri desktop apps
|
||||
│ │ ├── artcraft/ # ArtCraft the Tauri app's frontend. Used in conjunction with `artcraft` the Rust crate.
|
||||
│ │ └── artcraft-website/ # The website for https://getartcraft.com
|
||||
│ └── libs/ # Support libraries, reusable React components, etc.
|
||||
└── Cargo.toml # Rust monorepo workspace
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- Rust with no minimum supported version
|
||||
- Actix-web for HTTP services
|
||||
- SQLx for MySQL and SQLite; prefer `sqlx::query!` / `sqlx::query_as!` compile-time checked macros over runtime `sqlx::query()` whenever possible
|
||||
- MySQL queries belong in the `mysql_queries` crate, not in application/service crates like `storyteller-web` — call those query functions instead of embedding `sqlx::query!` / `sqlx::query()` in handlers. (Query-writing conventions live in `crates/schema/database/mysql_queries/AGENTS.md`.)
|
||||
- A mix of wreq and reqwest for Rust HTTP clients
|
||||
- Never use `println!` or `eprintln!` outside of tests; use `log` crate macros instead
|
||||
- When two crates export the same type name, alias imports with a suffix: `use foo::Bar as BarFoo;`
|
||||
- Prefer `use` imports over inline fully-qualified paths; only qualify inline for true one-offs or std collisions
|
||||
- TypeScript with Nx, React, Vite, Zustand, and Three.js
|
||||
- Use two spaces for indentation
|
||||
|
||||
### File Layout
|
||||
|
||||
Organize for top-to-bottom reading. Important things first, details later.
|
||||
|
||||
- **Constants** at the top (after imports)
|
||||
- **Structs/enums** next; outer structs above inner sub-structs
|
||||
- **API types** in order: Request, Response, Error
|
||||
- **In impl blocks**: constructors first, then public methods, then private helpers
|
||||
- Private helpers go *below* the methods that call them
|
||||
- Among helpers: meatier logic above leaf-level formatters
|
||||
- **In test modules**: constants first, then test cases (grouped into sub-modules when 2+), then helper functions last
|
||||
|
||||
## Markdown
|
||||
|
||||
- **Tables must be space-padded so columns align in plain text.** Markdown
|
||||
tables are read raw (terminals, diffs, editors) at least as often as
|
||||
rendered, and condensed tables are unreadable there. Pad every cell to its
|
||||
column width:
|
||||
|
||||
```markdown
|
||||
| Model | Configuration | Credits | Speed | Score |
|
||||
|--------------|---------------|--------------------|---------|
|
||||
| Meshy 6 | text or image | 104 | 80.0 | +24 |
|
||||
| Rodin 2.5 | text or image | 13 | 10.0 | +3 |
|
||||
```
|
||||
|
||||
Not: `| Model | Configuration | Credits | Speed | Score |` packed tight
|
||||
with varying widths per row.
|
||||
@@ -1,96 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance for Claude Code when working with the Artcraft monorepo.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Artcraft is a web and desktop application for generating AI image and video. It is written in
|
||||
Rust and TypeScript and contains desktop, server, and frontend components.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
artcraft/
|
||||
├── _database/ # Schema definitions and migrations (MySQL, SQLite, Elasticsearch, etc.)
|
||||
│ ├── elasticsearch/ # Elasticsearch schema and queries
|
||||
│ └── sql/ # MySQL and SQLite schema definitions and migrations
|
||||
│ ├── artcraft_migrations/ # ArtCraft desktop app SQLite migrations
|
||||
│ ├── migrations/ # Server MySQL migrations
|
||||
│ └── migrations_squashed/ # Fully materialized MySQL schema definitions for most tables
|
||||
├── _tools/ # Various 3rd party tool integrations and configurations
|
||||
│ └── postman/ # Postman configs for test HTTP requests against development and production
|
||||
├── build/ # Dockerfile build instructions for server components
|
||||
├── crates/ # Rust workspace
|
||||
│ ├── api_clients/ # HTTP clients for calling internal and 3rd party services
|
||||
│ ├── cli/ # Command line tools
|
||||
│ ├── desktop/ # Desktop (Tauri) apps
|
||||
│ │ └── artcraft/ # (Important) ArtCraft, the desktop app. This is one of our main pieces of software
|
||||
│ ├── lib/ # Various utility libraries for servers, CLI tools, desktop, etc.
|
||||
│ ├── schema/ # Data definition layer: MySQL, SQlite, Redis, S3/R2 buckets, etc.
|
||||
│ │ ├── buckets/ # Declares R2 cloud bucket topology
|
||||
│ │ ├── database/ # MySQL, SQLite, Elasticsearch, Redis, etc.
|
||||
│ │ │ ├── elasticsearch_schema/ # Elasticsearch
|
||||
│ │ │ ├── migration/ # (deprecated) Online schema adapters for MySQL
|
||||
│ │ │ ├── mysql_queries/ # Sqlx MySQL queries for our backend monolith `storyteller-web`, jobs, etc.
|
||||
│ │ │ ├── redis_common/ # Redis support
|
||||
│ │ │ ├── redis_schema/ # Redis key and HKEY topology
|
||||
│ │ │ ├── sqlite_queries/ # (deprecated) Sqlite queries
|
||||
│ │ │ └── sqlite_tasks/ # Queries for the ArtCraft desktop app's "tasks" database.
|
||||
│ │ ├── public/ # Token identifier and enum variant definitions
|
||||
│ │ │ ├── composite_identifier/ # MySQL composite key system
|
||||
│ │ │ ├── enums/ # MySQL "enums" stored in VARCHAR fields.
|
||||
│ │ │ └── tokens/ # Primary database identifiers with Stripe-like ID prefixes, eg. "user_{entropy}"
|
||||
│ │ └── service/ # Backend HTTP services and jobs
|
||||
│ │ ├── job/ # Backend jobs.
|
||||
│ │ │ └── video_thumbnail_job/ # Render video thumbnails
|
||||
│ │ ├── plugins/ # Collections of reusable Actix-Web HTTP functions (user and billing systems)
|
||||
│ │ └── web/ # HTTP web servers
|
||||
│ │ └── storyteller_web/ # (Important) Our main HTTP API monolith and backend.
|
||||
│ └── frontend/ # Nx typescript monorepo for our websites and Tauri desktop apps
|
||||
│ ├── apps/ # Websites and Tauri desktop apps
|
||||
│ │ ├── artcraft/ # ArtCraft the Tauri app's frontend. Used in conjunction with `artcraft` the Rust crate.
|
||||
│ │ └── artcraft-website/ # The website for https://getartcraft.com
|
||||
│ └── libs/ # Support libraries, reusable React components, etc.
|
||||
└── Cargo.toml # Rust monorepo workspace
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- Rust with no minimum supported version
|
||||
- Actix-web for HTTP services
|
||||
- SQLx for MySQL and SQLite. MySQL queries belong in the `mysql_queries` crate, not in application/service crates like `storyteller-web` — call those query functions instead of embedding `sqlx::query!` / `sqlx::query()` in handlers. (Query-writing conventions live in `crates/schema/database/mysql_queries/CLAUDE.md`.)
|
||||
- A mix of wreq and reqwest for Rust HTTP clients
|
||||
- Never use `println!` or `eprintln!` outside of tests; use `log` crate macros instead
|
||||
- When two crates export the same type name, alias imports with a suffix: `use foo::Bar as BarFoo;`
|
||||
- Prefer `use` imports over inline fully-qualified paths; only qualify inline for true one-offs or std collisions
|
||||
- TypeScript with Nx, React, Vite, Zustand, and Three.js
|
||||
- Use two spaces for indentation
|
||||
|
||||
### File Layout
|
||||
|
||||
Organize for top-to-bottom reading. Important things first, details later.
|
||||
|
||||
- **Constants** at the top (after imports)
|
||||
- **Structs/enums** next; outer structs above inner sub-structs
|
||||
- **API types** in order: Request, Response, Error
|
||||
- **In impl blocks**: constructors first, then public methods, then private helpers
|
||||
- Private helpers go *below* the methods that call them
|
||||
- Among helpers: meatier logic above leaf-level formatters
|
||||
- **In test modules**: constants first, then test cases (grouped into sub-modules when 2+), then helper functions last
|
||||
|
||||
## Markdown
|
||||
|
||||
- **Tables must be space-padded so columns align in plain text.** Markdown
|
||||
tables are read raw (terminals, diffs, editors) at least as often as
|
||||
rendered, and condensed tables are unreadable there. Pad every cell to its
|
||||
column width:
|
||||
|
||||
```markdown
|
||||
| Model | Configuration | Credits | Speed | Score |
|
||||
|--------------|---------------|--------------------|---------|
|
||||
| Meshy 6 | text or image | 104 | 80.0 | +24 |
|
||||
| Rodin 2.5 | text or image | 13 | 10.0 | +3 |
|
||||
```
|
||||
|
||||
Not: `| Model | Configuration | Credits | Speed | Score |` packed tight
|
||||
with varying widths per row.
|
||||
@@ -0,0 +1,38 @@
|
||||
# SQL Migrations
|
||||
|
||||
Incremental schema changes applied in order by timestamp prefix.
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Migration directories follow the format:
|
||||
|
||||
```
|
||||
YYYY-MM-DD-HHMMSS-NNNN_description_goes_here/
|
||||
up.sql
|
||||
down.sql
|
||||
```
|
||||
|
||||
These are created with the `diesel` tool, and in particular:
|
||||
|
||||
> `diesel migration generate description_goes_here`
|
||||
|
||||
- `NNNN` is a zero-padded sequence number for ordering multiple migrations on the same timestamp.
|
||||
- `up.sql` applies the migration forward.
|
||||
- `down.sql` reverses it.
|
||||
|
||||
## Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
- All SQL files should start with the standard IntelliJ noinspection comments:
|
||||
```sql
|
||||
-- noinspection SqlDialectInspectionForFile
|
||||
-- noinspection SqlNoDataSourceInspectionForFile
|
||||
-- noinspection SqlResolveForFile
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- Every `up.sql` must have a corresponding `down.sql` that cleanly reverses the migration.
|
||||
- For `CREATE TABLE` migrations, the `down.sql` is typically `DROP TABLE IF EXISTS table_name;`.
|
||||
- For `ALTER TABLE` migrations, the `down.sql` reverses the alteration (e.g. `DROP COLUMN`).
|
||||
@@ -1,38 +0,0 @@
|
||||
# SQL Migrations
|
||||
|
||||
Incremental schema changes applied in order by timestamp prefix.
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Migration directories follow the format:
|
||||
|
||||
```
|
||||
YYYY-MM-DD-HHMMSS-NNNN_description_goes_here/
|
||||
up.sql
|
||||
down.sql
|
||||
```
|
||||
|
||||
These are created with the `diesel` tool, and in particular:
|
||||
|
||||
> `diesel migration generate description_goes_here`
|
||||
|
||||
- `NNNN` is a zero-padded sequence number for ordering multiple migrations on the same timestamp.
|
||||
- `up.sql` applies the migration forward.
|
||||
- `down.sql` reverses it.
|
||||
|
||||
## Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
- All SQL files should start with the standard IntelliJ noinspection comments:
|
||||
```sql
|
||||
-- noinspection SqlDialectInspectionForFile
|
||||
-- noinspection SqlNoDataSourceInspectionForFile
|
||||
-- noinspection SqlResolveForFile
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- Every `up.sql` must have a corresponding `down.sql` that cleanly reverses the migration.
|
||||
- For `CREATE TABLE` migrations, the `down.sql` is typically `DROP TABLE IF EXISTS table_name;`.
|
||||
- For `ALTER TABLE` migrations, the `down.sql` reverses the alteration (e.g. `DROP COLUMN`).
|
||||
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,26 @@
|
||||
# Squashed / Materialized Schemas
|
||||
|
||||
Fully materialized table definitions representing the current state of each table
|
||||
after all migrations have been applied. These are not executed — they exist as
|
||||
human-readable documentation of the current schema.
|
||||
|
||||
## When to Update
|
||||
|
||||
Update the corresponding squashed file whenever a migration changes a table's schema
|
||||
(adding columns, changing types, dropping and recreating, etc.).
|
||||
|
||||
## Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
- All SQL files should start with the standard IntelliJ noinspection comments:
|
||||
```sql
|
||||
-- noinspection SqlDialectInspectionForFile
|
||||
-- noinspection SqlNoDataSourceInspectionForFile
|
||||
-- noinspection SqlResolveForFile
|
||||
```
|
||||
|
||||
## File Organization
|
||||
|
||||
Files are named after their table (e.g. `users.sql`, `media_files.sql`).
|
||||
Some tables are grouped into subdirectories (e.g. `user_tables/`).
|
||||
@@ -1,26 +0,0 @@
|
||||
# Squashed / Materialized Schemas
|
||||
|
||||
Fully materialized table definitions representing the current state of each table
|
||||
after all migrations have been applied. These are not executed — they exist as
|
||||
human-readable documentation of the current schema.
|
||||
|
||||
## When to Update
|
||||
|
||||
Update the corresponding squashed file whenever a migration changes a table's schema
|
||||
(adding columns, changing types, dropping and recreating, etc.).
|
||||
|
||||
## Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
- All SQL files should start with the standard IntelliJ noinspection comments:
|
||||
```sql
|
||||
-- noinspection SqlDialectInspectionForFile
|
||||
-- noinspection SqlNoDataSourceInspectionForFile
|
||||
-- noinspection SqlResolveForFile
|
||||
```
|
||||
|
||||
## File Organization
|
||||
|
||||
Files are named after their table (e.g. `users.sql`, `media_files.sql`).
|
||||
Some tables are grouped into subdirectories (e.g. `user_tables/`).
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,59 @@
|
||||
# Rust Workspace
|
||||
|
||||
All Rust code lives under `crates/`. The workspace is defined in `Cargo.toml` at the repo root.
|
||||
|
||||
## Code Style
|
||||
|
||||
- Two spaces for indentation (not four)
|
||||
- No minimum supported Rust version; use latest stable features freely
|
||||
- Never use `println!` or `eprintln!` outside of tests; use `log` crate macros instead
|
||||
- Use `anyhow::Result` / `AnyhowResult` for fallible functions in application code
|
||||
- Fields that are optional use the `maybe_` prefix: `maybe_creator_user_token`
|
||||
- When two crates export the same type name, alias with a suffix: `use foo::Bar as BarFoo;`
|
||||
- Prefer `use` imports over inline fully-qualified paths; only qualify inline for true one-offs or std prelude collisions (`Result`, `Option`, `Error`)
|
||||
|
||||
## File Layout
|
||||
|
||||
Organize for top-to-bottom reading. Important things first, details later.
|
||||
|
||||
1. **Constants** at the top (after imports)
|
||||
2. **Primary public type** (the main struct/enum the file defines)
|
||||
3. **Supporting types** below the primary type
|
||||
4. **API types** in order: Request, Response, Error
|
||||
5. **`impl` blocks** after their type definitions
|
||||
- Constructors first, then public methods, then private helpers
|
||||
- Callers above callees — follow the call chain downward
|
||||
|
||||
## Test Layout
|
||||
|
||||
Within `#[cfg(test)] mod tests { ... }`:
|
||||
|
||||
1. **Imports**
|
||||
2. **Constants** (test IDs, URLs, fixture data) — small context that tests reference
|
||||
3. **Test functions or sub-modules** grouped by category
|
||||
4. **Helper functions** (builders, pipeline runners, assertions)
|
||||
|
||||
Constants provide small context up front. Test cases come next so engineers see *what* is
|
||||
tested before *how* the helpers work.
|
||||
|
||||
Group related tests into sub-modules (`mod pricing_tests { ... }`) when a category has 2+
|
||||
tests. Don't wrap a single test in its own module. Keep nesting to two levels max. Shared
|
||||
helpers go in the parent `mod tests` so sub-modules can `use super::*`.
|
||||
|
||||
## Building
|
||||
|
||||
- Most crates: `cargo check -p {crate_name}` or `cargo test -p {crate_name}`
|
||||
- Crates using SQLx: `SQLX_OFFLINE=true cargo check -p {crate_name}`
|
||||
- `storyteller-web` and `mysql_queries` require `SQLX_OFFLINE=true`
|
||||
- `storyteller-web` has ~370 pre-existing warnings — check the last few lines for actual errors
|
||||
|
||||
## Key Crates
|
||||
|
||||
- `storyteller_web` — main HTTP API monolith (actix-web)
|
||||
- `artcraft` (desktop) — Tauri desktop app
|
||||
- `artcraft_router` — provider routing for image/video generation (Artcraft, Fal, KinoviWeb, Muapi)
|
||||
- `artcraft_api_defs` — shared HTTP API type definitions
|
||||
- `kinovi_web_client` — HTTP client for the Kinovi/KinoviWeb video generation service
|
||||
- `enums` — database-backed enums stored as VARCHAR fields
|
||||
- `tokens` — primary key identifiers with Stripe-like prefixes (e.g. `user_`, `mf_`)
|
||||
- `mysql_queries` — SQLx MySQL queries for the backend
|
||||
@@ -1,59 +0,0 @@
|
||||
# Rust Workspace
|
||||
|
||||
All Rust code lives under `crates/`. The workspace is defined in `Cargo.toml` at the repo root.
|
||||
|
||||
## Code Style
|
||||
|
||||
- Two spaces for indentation (not four)
|
||||
- No minimum supported Rust version; use latest stable features freely
|
||||
- Never use `println!` or `eprintln!` outside of tests; use `log` crate macros instead
|
||||
- Use `anyhow::Result` / `AnyhowResult` for fallible functions in application code
|
||||
- Fields that are optional use the `maybe_` prefix: `maybe_creator_user_token`
|
||||
- When two crates export the same type name, alias with a suffix: `use foo::Bar as BarFoo;`
|
||||
- Prefer `use` imports over inline fully-qualified paths; only qualify inline for true one-offs or std prelude collisions (`Result`, `Option`, `Error`)
|
||||
|
||||
## File Layout
|
||||
|
||||
Organize for top-to-bottom reading. Important things first, details later.
|
||||
|
||||
1. **Constants** at the top (after imports)
|
||||
2. **Primary public type** (the main struct/enum the file defines)
|
||||
3. **Supporting types** below the primary type
|
||||
4. **API types** in order: Request, Response, Error
|
||||
5. **`impl` blocks** after their type definitions
|
||||
- Constructors first, then public methods, then private helpers
|
||||
- Callers above callees — follow the call chain downward
|
||||
|
||||
## Test Layout
|
||||
|
||||
Within `#[cfg(test)] mod tests { ... }`:
|
||||
|
||||
1. **Imports**
|
||||
2. **Constants** (test IDs, URLs, fixture data) — small context that tests reference
|
||||
3. **Test functions or sub-modules** grouped by category
|
||||
4. **Helper functions** (builders, pipeline runners, assertions)
|
||||
|
||||
Constants provide small context up front. Test cases come next so engineers see *what* is
|
||||
tested before *how* the helpers work.
|
||||
|
||||
Group related tests into sub-modules (`mod pricing_tests { ... }`) when a category has 2+
|
||||
tests. Don't wrap a single test in its own module. Keep nesting to two levels max. Shared
|
||||
helpers go in the parent `mod tests` so sub-modules can `use super::*`.
|
||||
|
||||
## Building
|
||||
|
||||
- Most crates: `cargo check -p {crate_name}` or `cargo test -p {crate_name}`
|
||||
- Crates using SQLx: `SQLX_OFFLINE=true cargo check -p {crate_name}`
|
||||
- `storyteller-web` and `mysql_queries` require `SQLX_OFFLINE=true`
|
||||
- `storyteller-web` has ~370 pre-existing warnings — check the last few lines for actual errors
|
||||
|
||||
## Key Crates
|
||||
|
||||
- `storyteller_web` — main HTTP API monolith (actix-web)
|
||||
- `artcraft` (desktop) — Tauri desktop app
|
||||
- `artcraft_router` — provider routing for image/video generation (Artcraft, Fal, KinoviWeb, Muapi)
|
||||
- `artcraft_api_defs` — shared HTTP API type definitions
|
||||
- `kinovi_web_client` — HTTP client for the Kinovi/KinoviWeb video generation service
|
||||
- `enums` — database-backed enums stored as VARCHAR fields
|
||||
- `tokens` — primary key identifiers with Stripe-like prefixes (e.g. `user_`, `mf_`)
|
||||
- `mysql_queries` — SQLx MySQL queries for the backend
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,22 @@
|
||||
# artcraft_api_defs
|
||||
|
||||
Shared HTTP API type definitions used by both the server (`storyteller_web`) and clients.
|
||||
|
||||
## Structure
|
||||
|
||||
Organized by feature: `users/`, `media_file/`, `jobs/`, `moderation/`, etc.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Request types: `#[derive(Deserialize, ToSchema)]`
|
||||
- Response types: `#[derive(Serialize, ToSchema)]`
|
||||
- Path params: `#[derive(Deserialize, ToSchema)]` with `PathInfo` suffix
|
||||
- Query params: `#[derive(Deserialize, IntoParams)]`
|
||||
|
||||
## Orphan Rule
|
||||
|
||||
Types that need `ResponseError` impl (actix-web) CANNOT live here because of Rust's orphan rule. Keep error response types in `storyteller_web` and only put data types here.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Only `serde`, `serde_derive`, `utoipa`, `chrono`, `url`, and internal schema crates (`enums`, `tokens`). No actix-web dependency.
|
||||
@@ -1,22 +0,0 @@
|
||||
# artcraft_api_defs
|
||||
|
||||
Shared HTTP API type definitions used by both the server (`storyteller_web`) and clients.
|
||||
|
||||
## Structure
|
||||
|
||||
Organized by feature: `users/`, `media_file/`, `jobs/`, `moderation/`, etc.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Request types: `#[derive(Deserialize, ToSchema)]`
|
||||
- Response types: `#[derive(Serialize, ToSchema)]`
|
||||
- Path params: `#[derive(Deserialize, ToSchema)]` with `PathInfo` suffix
|
||||
- Query params: `#[derive(Deserialize, IntoParams)]`
|
||||
|
||||
## Orphan Rule
|
||||
|
||||
Types that need `ResponseError` impl (actix-web) CANNOT live here because of Rust's orphan rule. Keep error response types in `storyteller_web` and only put data types here.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Only `serde`, `serde_derive`, `utoipa`, `chrono`, `url`, and internal schema crates (`enums`, `tokens`). No actix-web dependency.
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,127 @@
|
||||
# generate_video
|
||||
|
||||
This module implements the v2 video generation pipeline. It replaces the v1 system
|
||||
(in `generate_video/`) with a two-phase approach that separates request planning from
|
||||
execution, allowing cost estimation and media uploads to happen at the right time.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
```
|
||||
GenerateVideoRequestBuilder
|
||||
│
|
||||
▼ build2()
|
||||
VideoGenerationDraftOrRequest
|
||||
│
|
||||
├─ Draft(VideoGenerationDraftRequest)
|
||||
│ │
|
||||
│ ▼ estimate_cost() ← cost estimate from draft state
|
||||
│ │
|
||||
│ ▼ finalize(context) ← uploads media, resolves tokens
|
||||
│ │
|
||||
│ ▼
|
||||
│ VideoGenerationRequest
|
||||
│ │
|
||||
│ ▼ estimate_cost() ← cost estimate from finalized request
|
||||
│ │
|
||||
│ ▼ send_request() ← fires the API call
|
||||
│ │
|
||||
│ ▼
|
||||
│ GenerateVideoResponse
|
||||
│
|
||||
└─ Request(VideoGenerationRequest) ← skips draft phase (used by Artcraft provider)
|
||||
```
|
||||
|
||||
### Key types
|
||||
|
||||
- **`GenerateVideoRequestBuilder`** — generic builder with provider/model/prompt/resolution/etc.
|
||||
Located in `generate_video/generate_video_request_builder.rs`. The `use_new_builder()` method
|
||||
gates which provider+model combos use the v2 path. `build2()` dispatches to the correct
|
||||
provider's build function.
|
||||
|
||||
- **`VideoGenerationDraftOrRequest`** — the output of `build2()`. Either `Draft` (for providers
|
||||
that need media upload/resolution, like Kinovi) or `Request` (for providers that work with
|
||||
media file tokens directly, like Artcraft).
|
||||
|
||||
- **`VideoGenerationDraftRequest`** — enum with one variant per provider+model. Holds
|
||||
materialized settings (resolution, duration, batch count) plus unresolved media references.
|
||||
Can estimate cost without network calls.
|
||||
|
||||
- **`VideoGenerationDraftContext`** — context needed during `finalize()`: the provider client,
|
||||
media file token-to-URL map, and character token-to-ID map.
|
||||
|
||||
- **`VideoGenerationRequest`** — enum with one variant per provider+model. Holds a fully
|
||||
materialized request ready to send. All media URLs are resolved and uploaded.
|
||||
|
||||
### Cost estimation
|
||||
|
||||
Cost can be estimated at two stages:
|
||||
1. **From draft** — before `finalize()`. Uses the planned resolution/duration/batch count.
|
||||
Good for showing the user a price before committing.
|
||||
2. **From request** — after `finalize()`. Uses the final materialized request. Should match
|
||||
the draft estimate for the same parameters.
|
||||
|
||||
## Module layout
|
||||
|
||||
```
|
||||
generate_video/
|
||||
├── mod.rs
|
||||
├── video_generation_draft.rs ← VideoGenerationDraftRequest enum
|
||||
├── video_generation_draft_context.rs ← context for finalize()
|
||||
├── video_generation_draft_or_request.rs
|
||||
├── video_generation_request.rs ← VideoGenerationRequest enum
|
||||
└── providers/
|
||||
├── artcraft/ ← Artcraft provider (uses media file tokens directly)
|
||||
│ ├── seedance_2p0/ ← Seedance 2.0 Pro model (skips draft, returns Request directly)
|
||||
│ │ ├── mod.rs
|
||||
│ │ ├── build.rs ← builder → Request conversion + plan helpers
|
||||
│ │ ├── cost.rs ← standalone cost estimation (independent of kinovi_web_client)
|
||||
│ │ └── request.rs ← request state + send() via Artcraft multi-function API
|
||||
│ └── seedance_2p0_fast/ ← Seedance 2.0 Fast model (skips draft, 480p/720p only)
|
||||
│ ├── mod.rs
|
||||
│ ├── build.rs ← builder → Request conversion + plan helpers (no 1080p)
|
||||
│ ├── cost.rs ← standalone cost estimation with Fast-specific pricing
|
||||
│ └── request.rs ← request state + send() via Artcraft omni-gen API
|
||||
└── kinovi/ ← Kinovi/KinoviWeb provider
|
||||
├── mod.rs
|
||||
├── resolve.rs ← shared: media token resolution, upload helpers
|
||||
├── upload.rs ← shared: download + re-upload to KinoviWeb CDN
|
||||
├── seedance_2p0/ ← Seedance 2.0 Pro model
|
||||
│ ├── mod.rs
|
||||
│ ├── build.rs ← builder → draft conversion + plan helpers
|
||||
│ ├── cost.rs ← cost estimation state
|
||||
│ ├── draft.rs ← draft state + finalize (to_request)
|
||||
│ └── request.rs ← request state + send()
|
||||
└── seedance_2p0_fast/ ← Seedance 2.0 Fast model
|
||||
├── mod.rs
|
||||
├── build.rs
|
||||
├── cost.rs
|
||||
├── draft.rs
|
||||
└── request.rs
|
||||
```
|
||||
|
||||
## Adding a new provider or model
|
||||
|
||||
1. **Create the module** under `providers/{provider}/{model}/` with these files:
|
||||
- `mod.rs` — declares sub-modules
|
||||
- `build.rs` — `build_{provider}_{model}(builder) -> VideoGenerationDraftOrRequest`.
|
||||
Contains `plan_*` helpers for aspect ratio, resolution, batch count, duration.
|
||||
- `draft.rs` — draft state struct with `to_request()` async method for finalization.
|
||||
**Not needed** if the provider uses media tokens directly (no upload/resolution needed);
|
||||
in that case `build.rs` returns `VideoGenerationDraftOrRequest::Request(...)` directly.
|
||||
- `cost.rs` — cost state with `from_request()` and `estimate_cost()`. Add `from_draft()`
|
||||
if the provider uses the draft phase.
|
||||
- `request.rs` — request state with `send()` async method.
|
||||
|
||||
2. **Register the module** in the parent provider's `mod.rs`.
|
||||
|
||||
3. **Add enum variants** in:
|
||||
- `video_generation_draft.rs` — `VideoGenerationDraftRequest` enum + `estimate_cost` + `finalize`
|
||||
(skip if provider has no draft phase)
|
||||
- `video_generation_request.rs` — `VideoGenerationRequest` enum + `estimate_cost` + `send_request`
|
||||
|
||||
4. **Wire into the builder** in `generate_video_request_builder.rs`:
|
||||
- Add the `(Provider, Model)` pair to `use_new_builder()` returning `true`
|
||||
- Add the dispatch arm in `build2()`
|
||||
|
||||
5. **Tests**: each `build.rs` and `cost.rs` should have comprehensive tests. `request.rs`
|
||||
should have `#[ignore]` live API tests for manual verification.
|
||||
@@ -1,127 +0,0 @@
|
||||
# generate_video
|
||||
|
||||
This module implements the v2 video generation pipeline. It replaces the v1 system
|
||||
(in `generate_video/`) with a two-phase approach that separates request planning from
|
||||
execution, allowing cost estimation and media uploads to happen at the right time.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
```
|
||||
GenerateVideoRequestBuilder
|
||||
│
|
||||
▼ build2()
|
||||
VideoGenerationDraftOrRequest
|
||||
│
|
||||
├─ Draft(VideoGenerationDraftRequest)
|
||||
│ │
|
||||
│ ▼ estimate_cost() ← cost estimate from draft state
|
||||
│ │
|
||||
│ ▼ finalize(context) ← uploads media, resolves tokens
|
||||
│ │
|
||||
│ ▼
|
||||
│ VideoGenerationRequest
|
||||
│ │
|
||||
│ ▼ estimate_cost() ← cost estimate from finalized request
|
||||
│ │
|
||||
│ ▼ send_request() ← fires the API call
|
||||
│ │
|
||||
│ ▼
|
||||
│ GenerateVideoResponse
|
||||
│
|
||||
└─ Request(VideoGenerationRequest) ← skips draft phase (used by Artcraft provider)
|
||||
```
|
||||
|
||||
### Key types
|
||||
|
||||
- **`GenerateVideoRequestBuilder`** — generic builder with provider/model/prompt/resolution/etc.
|
||||
Located in `generate_video/generate_video_request_builder.rs`. The `use_new_builder()` method
|
||||
gates which provider+model combos use the v2 path. `build2()` dispatches to the correct
|
||||
provider's build function.
|
||||
|
||||
- **`VideoGenerationDraftOrRequest`** — the output of `build2()`. Either `Draft` (for providers
|
||||
that need media upload/resolution, like Kinovi) or `Request` (for providers that work with
|
||||
media file tokens directly, like Artcraft).
|
||||
|
||||
- **`VideoGenerationDraftRequest`** — enum with one variant per provider+model. Holds
|
||||
materialized settings (resolution, duration, batch count) plus unresolved media references.
|
||||
Can estimate cost without network calls.
|
||||
|
||||
- **`VideoGenerationDraftContext`** — context needed during `finalize()`: the provider client,
|
||||
media file token-to-URL map, and character token-to-ID map.
|
||||
|
||||
- **`VideoGenerationRequest`** — enum with one variant per provider+model. Holds a fully
|
||||
materialized request ready to send. All media URLs are resolved and uploaded.
|
||||
|
||||
### Cost estimation
|
||||
|
||||
Cost can be estimated at two stages:
|
||||
1. **From draft** — before `finalize()`. Uses the planned resolution/duration/batch count.
|
||||
Good for showing the user a price before committing.
|
||||
2. **From request** — after `finalize()`. Uses the final materialized request. Should match
|
||||
the draft estimate for the same parameters.
|
||||
|
||||
## Module layout
|
||||
|
||||
```
|
||||
generate_video/
|
||||
├── mod.rs
|
||||
├── video_generation_draft.rs ← VideoGenerationDraftRequest enum
|
||||
├── video_generation_draft_context.rs ← context for finalize()
|
||||
├── video_generation_draft_or_request.rs
|
||||
├── video_generation_request.rs ← VideoGenerationRequest enum
|
||||
└── providers/
|
||||
├── artcraft/ ← Artcraft provider (uses media file tokens directly)
|
||||
│ ├── seedance_2p0/ ← Seedance 2.0 Pro model (skips draft, returns Request directly)
|
||||
│ │ ├── mod.rs
|
||||
│ │ ├── build.rs ← builder → Request conversion + plan helpers
|
||||
│ │ ├── cost.rs ← standalone cost estimation (independent of kinovi_web_client)
|
||||
│ │ └── request.rs ← request state + send() via Artcraft multi-function API
|
||||
│ └── seedance_2p0_fast/ ← Seedance 2.0 Fast model (skips draft, 480p/720p only)
|
||||
│ ├── mod.rs
|
||||
│ ├── build.rs ← builder → Request conversion + plan helpers (no 1080p)
|
||||
│ ├── cost.rs ← standalone cost estimation with Fast-specific pricing
|
||||
│ └── request.rs ← request state + send() via Artcraft omni-gen API
|
||||
└── kinovi/ ← Kinovi/KinoviWeb provider
|
||||
├── mod.rs
|
||||
├── resolve.rs ← shared: media token resolution, upload helpers
|
||||
├── upload.rs ← shared: download + re-upload to KinoviWeb CDN
|
||||
├── seedance_2p0/ ← Seedance 2.0 Pro model
|
||||
│ ├── mod.rs
|
||||
│ ├── build.rs ← builder → draft conversion + plan helpers
|
||||
│ ├── cost.rs ← cost estimation state
|
||||
│ ├── draft.rs ← draft state + finalize (to_request)
|
||||
│ └── request.rs ← request state + send()
|
||||
└── seedance_2p0_fast/ ← Seedance 2.0 Fast model
|
||||
├── mod.rs
|
||||
├── build.rs
|
||||
├── cost.rs
|
||||
├── draft.rs
|
||||
└── request.rs
|
||||
```
|
||||
|
||||
## Adding a new provider or model
|
||||
|
||||
1. **Create the module** under `providers/{provider}/{model}/` with these files:
|
||||
- `mod.rs` — declares sub-modules
|
||||
- `build.rs` — `build_{provider}_{model}(builder) -> VideoGenerationDraftOrRequest`.
|
||||
Contains `plan_*` helpers for aspect ratio, resolution, batch count, duration.
|
||||
- `draft.rs` — draft state struct with `to_request()` async method for finalization.
|
||||
**Not needed** if the provider uses media tokens directly (no upload/resolution needed);
|
||||
in that case `build.rs` returns `VideoGenerationDraftOrRequest::Request(...)` directly.
|
||||
- `cost.rs` — cost state with `from_request()` and `estimate_cost()`. Add `from_draft()`
|
||||
if the provider uses the draft phase.
|
||||
- `request.rs` — request state with `send()` async method.
|
||||
|
||||
2. **Register the module** in the parent provider's `mod.rs`.
|
||||
|
||||
3. **Add enum variants** in:
|
||||
- `video_generation_draft.rs` — `VideoGenerationDraftRequest` enum + `estimate_cost` + `finalize`
|
||||
(skip if provider has no draft phase)
|
||||
- `video_generation_request.rs` — `VideoGenerationRequest` enum + `estimate_cost` + `send_request`
|
||||
|
||||
4. **Wire into the builder** in `generate_video_request_builder.rs`:
|
||||
- Add the `(Provider, Model)` pair to `use_new_builder()` returning `true`
|
||||
- Add the dispatch arm in `build2()`
|
||||
|
||||
5. **Tests**: each `build.rs` and `cost.rs` should have comprehensive tests. `request.rs`
|
||||
should have `#[ignore]` live API tests for manual verification.
|
||||
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,24 @@
|
||||
# beeble_client
|
||||
|
||||
HTTP client for the Beeble SwitchX compositing API.
|
||||
|
||||
## API Reference
|
||||
|
||||
- Docs: https://developer.beeble.ai/docs
|
||||
- OpenAPI: https://api.beeble.ai/developer-api-docs/openapi.json
|
||||
- Auth: `x-api-key` header
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `create_upload_url` — POST /v1/uploads — get a presigned PUT URL for uploading media
|
||||
- `start_generation` — POST /v1/switchx/generations — start an image or video compositing job
|
||||
- `get_job_status` — GET /v1/switchx/generations/{job_id} — poll job status until completed/failed
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
cargo test -p beeble_client # run non-ignored tests (API shape only)
|
||||
cargo test -p beeble_client -- --ignored # run live API tests (requires key, incurs costs)
|
||||
```
|
||||
|
||||
Live tests require: `/Users/bt/Artcraft/credentials/beeble_api_key.txt`
|
||||
@@ -1,24 +0,0 @@
|
||||
# beeble_client
|
||||
|
||||
HTTP client for the Beeble SwitchX compositing API.
|
||||
|
||||
## API Reference
|
||||
|
||||
- Docs: https://developer.beeble.ai/docs
|
||||
- OpenAPI: https://api.beeble.ai/developer-api-docs/openapi.json
|
||||
- Auth: `x-api-key` header
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `create_upload_url` — POST /v1/uploads — get a presigned PUT URL for uploading media
|
||||
- `start_generation` — POST /v1/switchx/generations — start an image or video compositing job
|
||||
- `get_job_status` — GET /v1/switchx/generations/{job_id} — poll job status until completed/failed
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
cargo test -p beeble_client # run non-ignored tests (API shape only)
|
||||
cargo test -p beeble_client -- --ignored # run live API tests (requires key, incurs costs)
|
||||
```
|
||||
|
||||
Live tests require: `/Users/bt/Artcraft/credentials/beeble_api_key.txt`
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,58 @@
|
||||
# support-tool
|
||||
|
||||
Production support CLI for debugging and manual operations against external services.
|
||||
|
||||
## Building & Running
|
||||
|
||||
```bash
|
||||
cargo build -p support-tool
|
||||
cargo run -p support-tool -- kinovi_web generate_video --prompt "A corgi at the lake"
|
||||
```
|
||||
|
||||
## Environment
|
||||
|
||||
Requires a `.env-support-tool-secrets` file (or env vars) with:
|
||||
|
||||
- `SEEDANCE2PRO_COOKIES` — session cookies for Kinovi/KinoviWeb API auth
|
||||
- `ARTCRAFT_COOKIES` — session cookies for ArtCraft API auth (format: `session=...; visitor=...`)
|
||||
|
||||
## Architecture
|
||||
|
||||
- Entry point: `src/main.rs` → normalizes args (underscores optional) → dispatches
|
||||
- Top-level commands in `src/commands/run.rs` (`TopLevelCommand` enum)
|
||||
- Each top-level command has its own module with `dispatch.rs`, `state.rs`, `subcommands/`
|
||||
- Subcommands use `#[derive(Args)]` for their argument structs
|
||||
- All subcommand `run()` functions are `async fn` returning `anyhow::Result<()>`
|
||||
|
||||
## Adding a New Subcommand (to kinovi_web)
|
||||
|
||||
1. Create `src/commands/kinovi_web/subcommands/my_command.rs`
|
||||
2. Add `pub mod my_command;` to `subcommands/mod.rs`
|
||||
3. Add the name to `SUBCOMMAND_NAMES` in `dispatch.rs`
|
||||
4. Add a variant to `KinoviWebCommand` enum in `dispatch.rs`
|
||||
5. Add a match arm in `dispatch.rs::run()`
|
||||
|
||||
## Utilities
|
||||
|
||||
- `src/utils/parse_video_model.rs` — shared parser for loose video model names → `CommonVideoModel`
|
||||
- `src/utils/normalize_subcommands.rs` — underscore-insensitive arg normalizer
|
||||
|
||||
## Conventions
|
||||
|
||||
- Use `log::info!()` for status output, never `println!` (except for final results)
|
||||
- Use `anyhow::anyhow!()` for ad-hoc errors
|
||||
- Get `KinoviWebSession` from `state.cookies` via `from_cookies_string()`
|
||||
- Get `StorytellerCredentialSet` from `state.creds` (parsed from ARTCRAFT_COOKIES)
|
||||
- External crates used: `artcraft_client` (media file download, omni API), `kinovi_web_client` (Kinovi direct), `artcraft_api_defs` (request types)
|
||||
|
||||
## Subcommands
|
||||
|
||||
### kinovi_web (direct Kinovi API)
|
||||
|
||||
- `kinovi_web find_job --token <order_id>` — search for a job across all pages
|
||||
- `kinovi_web failed_job_histogram` — histogram of failure reasons
|
||||
- `kinovi_web generate_video --prompt <text_or_file> [--model seedance2p0] [--start_frame_media_token mf_xxx] [--localhost] [--download_path /tmp/media_files]`
|
||||
|
||||
### artcraft (omni API)
|
||||
|
||||
- `artcraft generate_video --prompt <text_or_file> [--model seedance2p0] [--start_frame_media_token mf_xxx] [--duration 5] [--localhost]`
|
||||
@@ -1,58 +0,0 @@
|
||||
# support-tool
|
||||
|
||||
Production support CLI for debugging and manual operations against external services.
|
||||
|
||||
## Building & Running
|
||||
|
||||
```bash
|
||||
cargo build -p support-tool
|
||||
cargo run -p support-tool -- kinovi_web generate_video --prompt "A corgi at the lake"
|
||||
```
|
||||
|
||||
## Environment
|
||||
|
||||
Requires a `.env-support-tool-secrets` file (or env vars) with:
|
||||
|
||||
- `SEEDANCE2PRO_COOKIES` — session cookies for Kinovi/KinoviWeb API auth
|
||||
- `ARTCRAFT_COOKIES` — session cookies for ArtCraft API auth (format: `session=...; visitor=...`)
|
||||
|
||||
## Architecture
|
||||
|
||||
- Entry point: `src/main.rs` → normalizes args (underscores optional) → dispatches
|
||||
- Top-level commands in `src/commands/run.rs` (`TopLevelCommand` enum)
|
||||
- Each top-level command has its own module with `dispatch.rs`, `state.rs`, `subcommands/`
|
||||
- Subcommands use `#[derive(Args)]` for their argument structs
|
||||
- All subcommand `run()` functions are `async fn` returning `anyhow::Result<()>`
|
||||
|
||||
## Adding a New Subcommand (to kinovi_web)
|
||||
|
||||
1. Create `src/commands/kinovi_web/subcommands/my_command.rs`
|
||||
2. Add `pub mod my_command;` to `subcommands/mod.rs`
|
||||
3. Add the name to `SUBCOMMAND_NAMES` in `dispatch.rs`
|
||||
4. Add a variant to `KinoviWebCommand` enum in `dispatch.rs`
|
||||
5. Add a match arm in `dispatch.rs::run()`
|
||||
|
||||
## Utilities
|
||||
|
||||
- `src/utils/parse_video_model.rs` — shared parser for loose video model names → `CommonVideoModel`
|
||||
- `src/utils/normalize_subcommands.rs` — underscore-insensitive arg normalizer
|
||||
|
||||
## Conventions
|
||||
|
||||
- Use `log::info!()` for status output, never `println!` (except for final results)
|
||||
- Use `anyhow::anyhow!()` for ad-hoc errors
|
||||
- Get `KinoviWebSession` from `state.cookies` via `from_cookies_string()`
|
||||
- Get `StorytellerCredentialSet` from `state.creds` (parsed from ARTCRAFT_COOKIES)
|
||||
- External crates used: `artcraft_client` (media file download, omni API), `kinovi_web_client` (Kinovi direct), `artcraft_api_defs` (request types)
|
||||
|
||||
## Subcommands
|
||||
|
||||
### kinovi_web (direct Kinovi API)
|
||||
|
||||
- `kinovi_web find_job --token <order_id>` — search for a job across all pages
|
||||
- `kinovi_web failed_job_histogram` — histogram of failure reasons
|
||||
- `kinovi_web generate_video --prompt <text_or_file> [--model seedance2p0] [--start_frame_media_token mf_xxx] [--localhost] [--download_path /tmp/media_files]`
|
||||
|
||||
### artcraft (omni API)
|
||||
|
||||
- `artcraft generate_video --prompt <text_or_file> [--model seedance2p0] [--start_frame_media_token mf_xxx] [--duration 5] [--localhost]`
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,40 @@
|
||||
# Pager Crate
|
||||
|
||||
Alerting/paging system that sends notifications via Rootly.
|
||||
|
||||
## Key Types
|
||||
|
||||
- `Pager` - Main client, cloneable. Has `enqueue_page()` and `shutdown_worker()`
|
||||
- `PagerBuilder` - Constructs a Pager with optional background worker
|
||||
- `NotificationDetails` - Alert payload with title, description, urgency, HTTP context, entity tokens
|
||||
- `NotificationDetailsBuilder` - Fluent builder for constructing notifications
|
||||
- `NotificationUrgency` - High, Medium, Low
|
||||
|
||||
## Building Notifications
|
||||
|
||||
```rust
|
||||
let notification = NotificationDetailsBuilder::from_title("Something happened".to_string())
|
||||
.set_description(Some("Details...".to_string()))
|
||||
.set_urgency(Some(NotificationUrgency::High))
|
||||
.set_http_method(Some("POST".to_string()))
|
||||
.set_http_path(Some("/v1/endpoint".to_string()))
|
||||
.set_inference_job_token(Some(job_token.to_string()))
|
||||
.build();
|
||||
```
|
||||
|
||||
For errors, use `from_error()` which auto-populates title and description from the error:
|
||||
|
||||
```rust
|
||||
NotificationDetailsBuilder::from_error(&err)
|
||||
.set_title("Custom title".to_string())
|
||||
.set_urgency(Some(NotificationUrgency::Medium))
|
||||
.build()
|
||||
```
|
||||
|
||||
## Deduplication
|
||||
|
||||
`NotificationDetails::to_deduplication_key()` generates a SHA-256 hash for dedup. Same title + HTTP context + hour = same key.
|
||||
|
||||
## Worker Thread
|
||||
|
||||
The pager worker uses `Condvar::wait()` (blocking). It MUST run on a dedicated OS thread, not a tokio task.
|
||||
@@ -1,40 +0,0 @@
|
||||
# Pager Crate
|
||||
|
||||
Alerting/paging system that sends notifications via Rootly.
|
||||
|
||||
## Key Types
|
||||
|
||||
- `Pager` - Main client, cloneable. Has `enqueue_page()` and `shutdown_worker()`
|
||||
- `PagerBuilder` - Constructs a Pager with optional background worker
|
||||
- `NotificationDetails` - Alert payload with title, description, urgency, HTTP context, entity tokens
|
||||
- `NotificationDetailsBuilder` - Fluent builder for constructing notifications
|
||||
- `NotificationUrgency` - High, Medium, Low
|
||||
|
||||
## Building Notifications
|
||||
|
||||
```rust
|
||||
let notification = NotificationDetailsBuilder::from_title("Something happened".to_string())
|
||||
.set_description(Some("Details...".to_string()))
|
||||
.set_urgency(Some(NotificationUrgency::High))
|
||||
.set_http_method(Some("POST".to_string()))
|
||||
.set_http_path(Some("/v1/endpoint".to_string()))
|
||||
.set_inference_job_token(Some(job_token.to_string()))
|
||||
.build();
|
||||
```
|
||||
|
||||
For errors, use `from_error()` which auto-populates title and description from the error:
|
||||
|
||||
```rust
|
||||
NotificationDetailsBuilder::from_error(&err)
|
||||
.set_title("Custom title".to_string())
|
||||
.set_urgency(Some(NotificationUrgency::Medium))
|
||||
.build()
|
||||
```
|
||||
|
||||
## Deduplication
|
||||
|
||||
`NotificationDetails::to_deduplication_key()` generates a SHA-256 hash for dedup. Same title + HTTP context + hour = same key.
|
||||
|
||||
## Worker Thread
|
||||
|
||||
The pager worker uses `Condvar::wait()` (blocking). It MUST run on a dedicated OS thread, not a tokio task.
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,79 @@
|
||||
# mysql_queries
|
||||
|
||||
SQLx-based MySQL queries for the storyteller-web backend and background jobs.
|
||||
|
||||
## Building
|
||||
|
||||
```
|
||||
SQLX_OFFLINE=true cargo check -p mysql_queries
|
||||
```
|
||||
|
||||
The `SQLX_OFFLINE=true` env var is **required** for compilation. This crate uses the sqlx
|
||||
offline cache (`.sqlx/` directory at the workspace root) to validate queries at compile time
|
||||
without a live database connection.
|
||||
|
||||
## Query Style
|
||||
|
||||
### Prefer `sqlx::query!` and `sqlx::query_as!` macros
|
||||
|
||||
Always use the compile-time checked `sqlx::query!` and `sqlx::query_as!` macros over the
|
||||
runtime `sqlx::query()` and `sqlx::query_as()` functions. The macros validate SQL syntax,
|
||||
column names, and parameter types against the offline cache at compile time.
|
||||
|
||||
```rust
|
||||
// PREFERRED: compile-time checked
|
||||
sqlx::query!(
|
||||
r#"INSERT INTO my_table SET token = ?, name = ?"#,
|
||||
token,
|
||||
name,
|
||||
)
|
||||
.execute(args.mysql_executor)
|
||||
.await?;
|
||||
|
||||
// AVOID: runtime only, no compile-time checking
|
||||
sqlx::query(r#"INSERT INTO my_table SET token = ?, name = ?"#)
|
||||
.bind(token)
|
||||
.bind(name)
|
||||
.execute(args.mysql_executor)
|
||||
.await?;
|
||||
```
|
||||
|
||||
Use runtime `sqlx::query()` only when:
|
||||
- The query involves a table not yet in the offline cache and you cannot run `cargo sqlx prepare`
|
||||
- The query is dynamically constructed (rare)
|
||||
- SELECT queries for new tables where column type metadata isn't cached yet
|
||||
|
||||
### Offline Cache
|
||||
|
||||
When adding new queries for **existing tables**, the `.sqlx/` cache should already have the
|
||||
schema metadata. For **new tables**, you may need to:
|
||||
|
||||
1. Run the migration against a real database
|
||||
2. Run `cargo sqlx prepare` to regenerate the cache
|
||||
3. Or manually create cache entries for simple INSERT/UPDATE queries (they only need parameter count)
|
||||
|
||||
### Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
|
||||
### Error Types
|
||||
|
||||
New queries should return `Result<T, sqlx::Error>` instead of the legacy `AnyhowResult<T>`.
|
||||
The `AnyhowResult` pattern is being phased out.
|
||||
|
||||
### Transaction Support
|
||||
|
||||
Use the executor-generic `Args` pattern for queries that need to work inside transactions:
|
||||
|
||||
```rust
|
||||
pub struct MyQueryArgs<'a, 'c: 'a, E>
|
||||
where E: 'a + Executor<'c, Database = MySql>
|
||||
{
|
||||
pub field: &'a str,
|
||||
pub mysql_executor: E,
|
||||
pub phantom: PhantomData<&'c E>,
|
||||
}
|
||||
```
|
||||
|
||||
This allows the same query function to accept both `&MySqlPool` and `&mut Transaction`.
|
||||
@@ -1,79 +0,0 @@
|
||||
# mysql_queries
|
||||
|
||||
SQLx-based MySQL queries for the storyteller-web backend and background jobs.
|
||||
|
||||
## Building
|
||||
|
||||
```
|
||||
SQLX_OFFLINE=true cargo check -p mysql_queries
|
||||
```
|
||||
|
||||
The `SQLX_OFFLINE=true` env var is **required** for compilation. This crate uses the sqlx
|
||||
offline cache (`.sqlx/` directory at the workspace root) to validate queries at compile time
|
||||
without a live database connection.
|
||||
|
||||
## Query Style
|
||||
|
||||
### Prefer `sqlx::query!` and `sqlx::query_as!` macros
|
||||
|
||||
Always use the compile-time checked `sqlx::query!` and `sqlx::query_as!` macros over the
|
||||
runtime `sqlx::query()` and `sqlx::query_as()` functions. The macros validate SQL syntax,
|
||||
column names, and parameter types against the offline cache at compile time.
|
||||
|
||||
```rust
|
||||
// PREFERRED: compile-time checked
|
||||
sqlx::query!(
|
||||
r#"INSERT INTO my_table SET token = ?, name = ?"#,
|
||||
token,
|
||||
name,
|
||||
)
|
||||
.execute(args.mysql_executor)
|
||||
.await?;
|
||||
|
||||
// AVOID: runtime only, no compile-time checking
|
||||
sqlx::query(r#"INSERT INTO my_table SET token = ?, name = ?"#)
|
||||
.bind(token)
|
||||
.bind(name)
|
||||
.execute(args.mysql_executor)
|
||||
.await?;
|
||||
```
|
||||
|
||||
Use runtime `sqlx::query()` only when:
|
||||
- The query involves a table not yet in the offline cache and you cannot run `cargo sqlx prepare`
|
||||
- The query is dynamically constructed (rare)
|
||||
- SELECT queries for new tables where column type metadata isn't cached yet
|
||||
|
||||
### Offline Cache
|
||||
|
||||
When adding new queries for **existing tables**, the `.sqlx/` cache should already have the
|
||||
schema metadata. For **new tables**, you may need to:
|
||||
|
||||
1. Run the migration against a real database
|
||||
2. Run `cargo sqlx prepare` to regenerate the cache
|
||||
3. Or manually create cache entries for simple INSERT/UPDATE queries (they only need parameter count)
|
||||
|
||||
### Table Conventions
|
||||
|
||||
- Primary key `id` columns must be `BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT`. Do not use signed IDs.
|
||||
- Always create tables with `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`.
|
||||
|
||||
### Error Types
|
||||
|
||||
New queries should return `Result<T, sqlx::Error>` instead of the legacy `AnyhowResult<T>`.
|
||||
The `AnyhowResult` pattern is being phased out.
|
||||
|
||||
### Transaction Support
|
||||
|
||||
Use the executor-generic `Args` pattern for queries that need to work inside transactions:
|
||||
|
||||
```rust
|
||||
pub struct MyQueryArgs<'a, 'c: 'a, E>
|
||||
where E: 'a + Executor<'c, Database = MySql>
|
||||
{
|
||||
pub field: &'a str,
|
||||
pub mysql_executor: E,
|
||||
pub phantom: PhantomData<&'c E>,
|
||||
}
|
||||
```
|
||||
|
||||
This allows the same query function to accept both `&MySqlPool` and `&mut Transaction`.
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,26 @@
|
||||
# Enums Crate
|
||||
|
||||
Database-backed enums stored as VARCHAR fields in MySQL.
|
||||
|
||||
## Adding a New Enum
|
||||
|
||||
1. Create file: `src/by_table/{table_name}/{field_name}.rs`
|
||||
2. Define the enum with required derives:
|
||||
```rust
|
||||
#[cfg_attr(test, derive(EnumIter, EnumCount))]
|
||||
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
|
||||
pub enum MyEnum { ... }
|
||||
```
|
||||
3. Implement `to_str()` and `from_str()` manually
|
||||
4. Add macros: `impl_enum_display_and_debug_using_to_str!(MyEnum);` and `impl_mysql_enum_coders!(MyEnum);`
|
||||
5. Add to parent `mod.rs`
|
||||
6. Write tests: serialization, round-trip, `all_variants()` count
|
||||
|
||||
## Rules
|
||||
|
||||
- **NEVER change existing variant string values** - they are stored in the database
|
||||
- New variants CAN be added freely - but be very careful about the Tauri API callers
|
||||
ingesting new values. Make sure to pair the client with variants with an "Unknown(String)"
|
||||
catch-all / future-proof bucket.
|
||||
- All string values must be snake_case
|
||||
- Use `#[serde(rename = "snake_case_value")]` on each variant
|
||||
@@ -1,26 +0,0 @@
|
||||
# Enums Crate
|
||||
|
||||
Database-backed enums stored as VARCHAR fields in MySQL.
|
||||
|
||||
## Adding a New Enum
|
||||
|
||||
1. Create file: `src/by_table/{table_name}/{field_name}.rs`
|
||||
2. Define the enum with required derives:
|
||||
```rust
|
||||
#[cfg_attr(test, derive(EnumIter, EnumCount))]
|
||||
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
|
||||
pub enum MyEnum { ... }
|
||||
```
|
||||
3. Implement `to_str()` and `from_str()` manually
|
||||
4. Add macros: `impl_enum_display_and_debug_using_to_str!(MyEnum);` and `impl_mysql_enum_coders!(MyEnum);`
|
||||
5. Add to parent `mod.rs`
|
||||
6. Write tests: serialization, round-trip, `all_variants()` count
|
||||
|
||||
## Rules
|
||||
|
||||
- **NEVER change existing variant string values** - they are stored in the database
|
||||
- New variants CAN be added freely - but be very careful about the Tauri API callers
|
||||
ingesting new values. Make sure to pair the client with variants with an "Unknown(String)"
|
||||
catch-all / future-proof bucket.
|
||||
- All string values must be snake_case
|
||||
- Use `#[serde(rename = "snake_case_value")]` on each variant
|
||||
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,22 @@
|
||||
# Tokens Crate
|
||||
|
||||
Strongly-typed primary key identifiers with Stripe-like prefixes (e.g. `user_abc123`, `prompt_xyz789`, etc.)
|
||||
|
||||
## Adding a New Token
|
||||
|
||||
1. Create file in `src/tokens/{entity_name}.rs`
|
||||
2. Define as a newtype wrapper:
|
||||
```rust
|
||||
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Debug, Serialize, Deserialize, ToSchema)]
|
||||
#[cfg_attr(feature = "database", derive(sqlx::Type))]
|
||||
#[cfg_attr(feature = "database", sqlx(transparent))]
|
||||
pub struct MyToken(pub String);
|
||||
```
|
||||
3. Apply macros:
|
||||
```rust
|
||||
impl_string_token!(MyToken);
|
||||
impl_mysql_token_from_row!(MyToken);
|
||||
impl_crockford_generator!(MyToken, 32usize, TokenPrefix::MyEntity, CrockfordLower);
|
||||
```
|
||||
4. Add the prefix to `TokenPrefix` enum in `src/token_prefix.rs`
|
||||
5. Add to parent `mod.rs`
|
||||
@@ -1,22 +0,0 @@
|
||||
# Tokens Crate
|
||||
|
||||
Strongly-typed primary key identifiers with Stripe-like prefixes (e.g. `user_abc123`, `prompt_xyz789`, etc.)
|
||||
|
||||
## Adding a New Token
|
||||
|
||||
1. Create file in `src/tokens/{entity_name}.rs`
|
||||
2. Define as a newtype wrapper:
|
||||
```rust
|
||||
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Debug, Serialize, Deserialize, ToSchema)]
|
||||
#[cfg_attr(feature = "database", derive(sqlx::Type))]
|
||||
#[cfg_attr(feature = "database", sqlx(transparent))]
|
||||
pub struct MyToken(pub String);
|
||||
```
|
||||
3. Apply macros:
|
||||
```rust
|
||||
impl_string_token!(MyToken);
|
||||
impl_mysql_token_from_row!(MyToken);
|
||||
impl_crockford_generator!(MyToken, 32usize, TokenPrefix::MyEntity, CrockfordLower);
|
||||
```
|
||||
4. Add the prefix to `TokenPrefix` enum in `src/token_prefix.rs`
|
||||
5. Add to parent `mod.rs`
|
||||
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,42 @@
|
||||
# Background Jobs
|
||||
|
||||
Background job crates that run as standalone processes alongside storyteller-web.
|
||||
|
||||
## Common Pattern
|
||||
|
||||
All jobs follow the same structure:
|
||||
|
||||
1. `main.rs` - Bootstrap, DB pool, bucket client, pager setup, thread spawning
|
||||
2. `job_dependencies.rs` - Shared state struct passed to the main loop
|
||||
3. `startup/build_pager.rs` - Pager initialization from env vars
|
||||
4. `http_server/` - Health check endpoint for k8s liveness probes
|
||||
5. `job/` or `jobs/` - Main processing logic
|
||||
|
||||
## Threading Model
|
||||
|
||||
- **Main thread** (`#[tokio::main]`): runs the job's main loop
|
||||
- **HTTP server thread** (`std::thread::spawn`): dedicated OS thread with its own `actix_web::rt::System`
|
||||
- **Pager worker thread** (`std::thread::spawn`): dedicated OS thread (uses blocking `Condvar::wait`)
|
||||
- **Signal handler** (`tokio::spawn`): listens for SIGTERM/SIGINT to set `application_shutdown` flag
|
||||
|
||||
Important: Do NOT use `#[actix_web::main]` - it registers signal handlers that steal SIGINT. Use `#[tokio::main]` instead.
|
||||
|
||||
Important: Do NOT spawn the pager worker via `tokio::spawn` - its `Condvar::wait` blocks the runtime. Use a dedicated OS thread.
|
||||
|
||||
## Pager Integration
|
||||
|
||||
Use the `alert_pager_and_return_err()` helper pattern for error reporting:
|
||||
|
||||
```rust
|
||||
if let Err(err) = some_operation().await {
|
||||
error!("Descriptive message: {:?}", err);
|
||||
return alert_pager_and_return_err(&deps.pager, "Alert title", err, optional_job_ref);
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Shared across jobs (via `shared_env_var_config`):
|
||||
- `ENABLE_PAGING` - Master switch for pager alerts
|
||||
- `ROOTLY_API_KEY` - Rootly API key for sending alerts
|
||||
- `ROOTLY_NOTIFICATION_TARGET_TYPE` / `ROOTLY_NOTIFICATION_TARGET_ID`
|
||||
@@ -1,42 +0,0 @@
|
||||
# Background Jobs
|
||||
|
||||
Background job crates that run as standalone processes alongside storyteller-web.
|
||||
|
||||
## Common Pattern
|
||||
|
||||
All jobs follow the same structure:
|
||||
|
||||
1. `main.rs` - Bootstrap, DB pool, bucket client, pager setup, thread spawning
|
||||
2. `job_dependencies.rs` - Shared state struct passed to the main loop
|
||||
3. `startup/build_pager.rs` - Pager initialization from env vars
|
||||
4. `http_server/` - Health check endpoint for k8s liveness probes
|
||||
5. `job/` or `jobs/` - Main processing logic
|
||||
|
||||
## Threading Model
|
||||
|
||||
- **Main thread** (`#[tokio::main]`): runs the job's main loop
|
||||
- **HTTP server thread** (`std::thread::spawn`): dedicated OS thread with its own `actix_web::rt::System`
|
||||
- **Pager worker thread** (`std::thread::spawn`): dedicated OS thread (uses blocking `Condvar::wait`)
|
||||
- **Signal handler** (`tokio::spawn`): listens for SIGTERM/SIGINT to set `application_shutdown` flag
|
||||
|
||||
Important: Do NOT use `#[actix_web::main]` - it registers signal handlers that steal SIGINT. Use `#[tokio::main]` instead.
|
||||
|
||||
Important: Do NOT spawn the pager worker via `tokio::spawn` - its `Condvar::wait` blocks the runtime. Use a dedicated OS thread.
|
||||
|
||||
## Pager Integration
|
||||
|
||||
Use the `alert_pager_and_return_err()` helper pattern for error reporting:
|
||||
|
||||
```rust
|
||||
if let Err(err) = some_operation().await {
|
||||
error!("Descriptive message: {:?}", err);
|
||||
return alert_pager_and_return_err(&deps.pager, "Alert title", err, optional_job_ref);
|
||||
}
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Shared across jobs (via `shared_env_var_config`):
|
||||
- `ENABLE_PAGING` - Master switch for pager alerts
|
||||
- `ROOTLY_API_KEY` - Rootly API key for sending alerts
|
||||
- `ROOTLY_NOTIFICATION_TARGET_TYPE` / `ROOTLY_NOTIFICATION_TARGET_ID`
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,116 @@
|
||||
# storyteller-web
|
||||
|
||||
The main HTTP API monolith. This is the backend for the ArtCraft application.
|
||||
|
||||
## Building
|
||||
|
||||
```
|
||||
cargo check -p storyteller-web
|
||||
```
|
||||
|
||||
There are ~370 pre-existing warnings. Check the last 8-10 lines of output for actual errors.
|
||||
|
||||
## Structure
|
||||
|
||||
- `src/http_server/endpoints/` - HTTP handlers organized by feature
|
||||
- `src/http_server/middleware/` - Actix middleware (error alerting, etc.)
|
||||
- `src/http_server/common_responses/` - Shared error types (`CommonWebError`)
|
||||
- `src/http_server/web_utils/` - Session checking, auth helpers
|
||||
- `src/state/` - Server state, feature flags
|
||||
- `src/startup/` - Initialization (pager, etc.)
|
||||
- `src/threads/` - Background threads (health checker, etc.)
|
||||
- `src/docs/` - OpenAPI/Swagger documentation (`api_doc.rs`)
|
||||
|
||||
## Handler Pattern
|
||||
|
||||
Handlers return `Result<Json<Response>, ErrorType>`. The `Result<HttpResponse, ErrorType>` pattern is
|
||||
deprecated or only used when setting other HTTP headers, like cookies.
|
||||
|
||||
Prefer `CommonWebError` for new handlers unless you need custom error variants.
|
||||
|
||||
The session helpers (`require_user_session`, `require_user_session_extended`, `require_moderator`
|
||||
in `http_server/web_utils/user_session/`) take a generic `mysql_executor` argument. Pass an
|
||||
already-open connection (`&mut *conn`) so the helper reuses it; do NOT pass
|
||||
`&server_state.mysql_pool` when the handler already holds a connection, since that self-acquires a
|
||||
second one and adds pool pressure (this crate has had pool-timeout incidents). There is no
|
||||
`require_user_session_using_connection` / `require_moderator_using_connection` — the
|
||||
reuse-vs-acquire behavior is determined by the argument.
|
||||
|
||||
## Database Access
|
||||
|
||||
Do NOT write SQLx queries in this crate. `storyteller-web` contains **zero** direct SQLx queries —
|
||||
all MySQL access goes through the `mysql_queries` crate. Add a function there (or reuse an existing
|
||||
one) and call it from the handler; never embed `sqlx::query!` / `sqlx::query()` in a handler or
|
||||
helper here. See `crates/schema/database/mysql_queries/CLAUDE.md` for the query-writing conventions
|
||||
(compile-time `query!` / `query_as!` macros, offline cache, the `Args`/`Executor` pattern, etc.).
|
||||
|
||||
## api_doc.rs
|
||||
|
||||
When adding new request/response/error types, add them to `src/docs/api_doc.rs` in the
|
||||
schemas section (alphabetically sorted). Types from `artcraft_api_defs` need explicit imports.
|
||||
|
||||
## Running locally
|
||||
|
||||
Environment variables are auto-loaded by Rust at startup from these files:
|
||||
|
||||
- `.env` (repo root)
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.common.env`
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.development.env`
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.development-secrets.env`
|
||||
|
||||
**Do NOT** use `SQLX_OFFLINE=true` — the server needs live DB access.
|
||||
|
||||
```bash
|
||||
# From the repo root, just run:
|
||||
cargo run --bin storyteller-web
|
||||
```
|
||||
|
||||
The server binds to `0.0.0.0:12345` by default. Compilation takes ~3 minutes on first build.
|
||||
|
||||
## Testing endpoints with curl
|
||||
|
||||
```bash
|
||||
# 1. Log in to get session cookies:
|
||||
curl -s -c /tmp/cookies.txt -X POST http://localhost:12345/v1/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username_or_email":"newuser4","password":"newuser4"}'
|
||||
|
||||
# 2. Generate video (Seedance 2.0 Fast):
|
||||
# Requires a UUID-format idempotency_token.
|
||||
curl -s -b /tmp/cookies.txt -X POST http://localhost:12345/v1/omni_gen/generate/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "seedance_2p0_fast",
|
||||
"prompt": "a corgi running through a field",
|
||||
"duration_seconds": 5,
|
||||
"aspect_ratio": "wide_sixteen_by_nine",
|
||||
"idempotency_token": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
|
||||
}'
|
||||
|
||||
# 3. Generate video (Seedance 2.0, 720p):
|
||||
curl -s -b /tmp/cookies.txt -X POST http://localhost:12345/v1/omni_gen/generate/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "seedance_2p0",
|
||||
"prompt": "a corgi running through a field",
|
||||
"duration_seconds": 5,
|
||||
"resolution": "seven_twenty_p",
|
||||
"aspect_ratio": "wide_sixteen_by_nine",
|
||||
"idempotency_token": "b2c3d4e5-f6a7-8901-bcde-f12345678901"
|
||||
}'
|
||||
```
|
||||
|
||||
A successful response returns 200 with `{"success": true, "inference_job_token": "jinf_..."}`.
|
||||
|
||||
**Notes:**
|
||||
- `idempotency_token` must be UUID format (32 hex chars or 36 with dashes)
|
||||
- `resolution` uses enum names: `four_eighty_p`, `seven_twenty_p`, `ten_eighty_p`
|
||||
- `aspect_ratio` uses enum names: `wide_sixteen_by_nine`, `tall_nine_by_sixteen`, `square`, etc.
|
||||
- Use a new `idempotency_token` for each request to avoid duplicate detection
|
||||
|
||||
## Pager Integration
|
||||
|
||||
- The `Pager` is available on `ServerState` as `server_state.pager`
|
||||
- Health check handler sends alerts when unhealthy
|
||||
- Error alerting middleware sends alerts on HTTP 500s (when enabled)
|
||||
- Use `NotificationDetailsBuilder` for constructing alerts
|
||||
@@ -1,116 +0,0 @@
|
||||
# storyteller-web
|
||||
|
||||
The main HTTP API monolith. This is the backend for the ArtCraft application.
|
||||
|
||||
## Building
|
||||
|
||||
```
|
||||
cargo check -p storyteller-web
|
||||
```
|
||||
|
||||
There are ~370 pre-existing warnings. Check the last 8-10 lines of output for actual errors.
|
||||
|
||||
## Structure
|
||||
|
||||
- `src/http_server/endpoints/` - HTTP handlers organized by feature
|
||||
- `src/http_server/middleware/` - Actix middleware (error alerting, etc.)
|
||||
- `src/http_server/common_responses/` - Shared error types (`CommonWebError`)
|
||||
- `src/http_server/web_utils/` - Session checking, auth helpers
|
||||
- `src/state/` - Server state, feature flags
|
||||
- `src/startup/` - Initialization (pager, etc.)
|
||||
- `src/threads/` - Background threads (health checker, etc.)
|
||||
- `src/docs/` - OpenAPI/Swagger documentation (`api_doc.rs`)
|
||||
|
||||
## Handler Pattern
|
||||
|
||||
Handlers return `Result<Json<Response>, ErrorType>`. The `Result<HttpResponse, ErrorType>` pattern is
|
||||
deprecated or only used when setting other HTTP headers, like cookies.
|
||||
|
||||
Prefer `CommonWebError` for new handlers unless you need custom error variants.
|
||||
|
||||
The session helpers (`require_user_session`, `require_user_session_extended`, `require_moderator`
|
||||
in `http_server/web_utils/user_session/`) take a generic `mysql_executor` argument. Pass an
|
||||
already-open connection (`&mut *conn`) so the helper reuses it; do NOT pass
|
||||
`&server_state.mysql_pool` when the handler already holds a connection, since that self-acquires a
|
||||
second one and adds pool pressure (this crate has had pool-timeout incidents). There is no
|
||||
`require_user_session_using_connection` / `require_moderator_using_connection` — the
|
||||
reuse-vs-acquire behavior is determined by the argument.
|
||||
|
||||
## Database Access
|
||||
|
||||
Do NOT write SQLx queries in this crate. `storyteller-web` contains **zero** direct SQLx queries —
|
||||
all MySQL access goes through the `mysql_queries` crate. Add a function there (or reuse an existing
|
||||
one) and call it from the handler; never embed `sqlx::query!` / `sqlx::query()` in a handler or
|
||||
helper here. See `crates/schema/database/mysql_queries/CLAUDE.md` for the query-writing conventions
|
||||
(compile-time `query!` / `query_as!` macros, offline cache, the `Args`/`Executor` pattern, etc.).
|
||||
|
||||
## api_doc.rs
|
||||
|
||||
When adding new request/response/error types, add them to `src/docs/api_doc.rs` in the
|
||||
schemas section (alphabetically sorted). Types from `artcraft_api_defs` need explicit imports.
|
||||
|
||||
## Running locally
|
||||
|
||||
Environment variables are auto-loaded by Rust at startup from these files:
|
||||
|
||||
- `.env` (repo root)
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.common.env`
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.development.env`
|
||||
- `crates/service/web/storyteller_web/config/storyteller-web.development-secrets.env`
|
||||
|
||||
**Do NOT** use `SQLX_OFFLINE=true` — the server needs live DB access.
|
||||
|
||||
```bash
|
||||
# From the repo root, just run:
|
||||
cargo run --bin storyteller-web
|
||||
```
|
||||
|
||||
The server binds to `0.0.0.0:12345` by default. Compilation takes ~3 minutes on first build.
|
||||
|
||||
## Testing endpoints with curl
|
||||
|
||||
```bash
|
||||
# 1. Log in to get session cookies:
|
||||
curl -s -c /tmp/cookies.txt -X POST http://localhost:12345/v1/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username_or_email":"newuser4","password":"newuser4"}'
|
||||
|
||||
# 2. Generate video (Seedance 2.0 Fast):
|
||||
# Requires a UUID-format idempotency_token.
|
||||
curl -s -b /tmp/cookies.txt -X POST http://localhost:12345/v1/omni_gen/generate/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "seedance_2p0_fast",
|
||||
"prompt": "a corgi running through a field",
|
||||
"duration_seconds": 5,
|
||||
"aspect_ratio": "wide_sixteen_by_nine",
|
||||
"idempotency_token": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
|
||||
}'
|
||||
|
||||
# 3. Generate video (Seedance 2.0, 720p):
|
||||
curl -s -b /tmp/cookies.txt -X POST http://localhost:12345/v1/omni_gen/generate/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "seedance_2p0",
|
||||
"prompt": "a corgi running through a field",
|
||||
"duration_seconds": 5,
|
||||
"resolution": "seven_twenty_p",
|
||||
"aspect_ratio": "wide_sixteen_by_nine",
|
||||
"idempotency_token": "b2c3d4e5-f6a7-8901-bcde-f12345678901"
|
||||
}'
|
||||
```
|
||||
|
||||
A successful response returns 200 with `{"success": true, "inference_job_token": "jinf_..."}`.
|
||||
|
||||
**Notes:**
|
||||
- `idempotency_token` must be UUID format (32 hex chars or 36 with dashes)
|
||||
- `resolution` uses enum names: `four_eighty_p`, `seven_twenty_p`, `ten_eighty_p`
|
||||
- `aspect_ratio` uses enum names: `wide_sixteen_by_nine`, `tall_nine_by_sixteen`, `square`, etc.
|
||||
- Use a new `idempotency_token` for each request to avoid duplicate detection
|
||||
|
||||
## Pager Integration
|
||||
|
||||
- The `Pager` is available on `ServerState` as `server_state.pager`
|
||||
- Health check handler sends alerts when unhealthy
|
||||
- Error alerting middleware sends alerts on HTTP 500s (when enabled)
|
||||
- Use `NotificationDetailsBuilder` for constructing alerts
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -0,0 +1,513 @@
|
||||
# CLAUDE.md — pagescene (3D Scene Editor)
|
||||
|
||||
Guidance for working in the `pagescene` library — the 3D editor (a.k.a. "pagescene" / "Stage3D").
|
||||
|
||||
## ⚠️ Engine-canvas mounting invariant (footgun)
|
||||
|
||||
`EngineProvider`'s editor-construction effect depends on three DOM nodes: `sceneContainer`,
|
||||
`editorCanvas`, and **`camViewCanvas`** (owned by `PreviewEngineCamera` → `CameraViewCanvas`).
|
||||
If ANY of these unmounts, the effect tears down + recreates the whole `Editor` (serializing to
|
||||
cache, then reloading) — which wipes a fresh/unsaved scene. **Never conditionally unmount the
|
||||
components that host these canvases** (`SceneContainer`, `EditorCanvas`, `PreviewEngineCamera`)
|
||||
for mode/chrome gating — hide them with CSS (`hidden`) instead. (Regression fixed: record mode
|
||||
had gated `PreviewEngineCamera` behind `!isRecord`, wiping the scene on record→build toggle.)
|
||||
|
||||
## Keybinds (unified registry)
|
||||
|
||||
Every viewport shortcut is a remappable action in `@storyteller/keybinds`
|
||||
(`frontend/libs/components/keybinds`): the registry owns *which key* (preset +
|
||||
user overrides, Settings → Keybinds), `engine/keymap.ts` owns *what it does*
|
||||
(action id → handler), and `useViewportKeyboard` dispatches with a per-event
|
||||
`KeybindContext` (sceneMode / encoding / timelineExpanded / timelineSelection /
|
||||
modalTransformActive). Rules:
|
||||
|
||||
- **Availability is declared, not implied.** An `ActionDef.when(ctx)` gate
|
||||
decides if a binding is live; an unavailable binding falls through without
|
||||
consuming the event. Default rule: available unless encoding. Two actions may
|
||||
share a key ONLY if their gates are mutually exclusive (`actionsCoAvailable`
|
||||
enforces this in tests and in the settings conflict UI). Examples: Space =
|
||||
timeline play/pause in playback contexts, camera-view toggle otherwise;
|
||||
Del/Backspace = valid timeline selection → keyframe/strip delete, else scene
|
||||
object; Escape = cancel-render mid-encode, clear-selection otherwise.
|
||||
- **Do NOT add raw keydown listeners for editor operations** — register an
|
||||
action + handler + `when` gate instead. Capture-phase component listeners
|
||||
(the old TimelineEditor delete path) are exactly the pattern this replaced.
|
||||
- **Record mode** gates all scene-mutating actions off (`inBuild`); the encode
|
||||
is abortable (overlay Cancel / Escape via `encodeCancelSignal`), and
|
||||
Capture/Record always `enterCameraView()` first so output comes from the
|
||||
render camera even after peeking at the scene view.
|
||||
- Camera movement (held WASD/QE/arrows) stays in `useFreeCam`, which resolves
|
||||
its codes from the same store; its imperative guards (ctrl-chords,
|
||||
timeline-expanded arrows, record lock) must stay in sync with the registry's
|
||||
`cameraMove`/`cameraLook` gates.
|
||||
|
||||
## Behavior notes (fixes)
|
||||
|
||||
- **Record mode is fully immutable.** The viewport lock (`CameraController.locked`, set by the
|
||||
record effect) gates every camera-move path: FreeCam drag/wheel/keyboard (`useFreeCam`),
|
||||
FreeCam integration (`tickPerFrame`), AND `MouseControls` mouse-look + `focus()` (gated via
|
||||
`MouseControlsDeps.isViewportLocked`). If you add a new camera-move path, gate it on `locked`.
|
||||
- **Default camera framing** (`DEFAULT_CAMERAS.main`): a Blender-style pulled-back/elevated 3/4
|
||||
view (`pos (-4.5,4,6)`, `lookAt (0,0.5,0.6)`) so the whole render-camera (`cam2`) frustum is
|
||||
visible on a blank scene.
|
||||
- **Auto-keyframe**: mutating an **already-keyframed** object (gizmo drag-end or inspector commit)
|
||||
calls `TimelineController.autoKeyIfTracked(uuid)` → `addKeyframe` at the playhead, which
|
||||
replace-or-creates a keyframe at the current scrub point. No-op for never-keyframed objects
|
||||
(first keyframe is always explicit).
|
||||
|
||||
## Architecture (existing, do not rebuild)
|
||||
|
||||
Strict one-way data flow. Do not violate it.
|
||||
|
||||
```
|
||||
Three.js engine → typed event bus → EngineStoreBridge → Zustand store → React → actions → engine methods
|
||||
```
|
||||
|
||||
- **`Stage3D.tsx`** — public entry component; takes an `adapter` + optional slots/callbacks.
|
||||
- **`Stage3DBody.tsx`** — layout composition (canvas, panels, toolbars, modals). Mounts the external `PromptBox3D`.
|
||||
- **`PageSceneStore.ts`** — the single Zustand store (~60 fields). All UI state. The engine never imports this store.
|
||||
- **`adapter.ts`** — platform-abstraction interface. All host-specific logic (HTTP, upload, auth, navigation, modals) flows through `PageSceneAdapter`. Hosts (Tauri app, web app) inject it at mount.
|
||||
- **`comps/Controls3D/Controls3D.tsx`** — top-center toolbar: `+` add, Move/Rotate/Scale (`transformMode`), World/Local (`transformSpace`).
|
||||
- **`comps/Outliner/Outliner.tsx`** — left "Scene" panel (code name: Outliner). Driven by `outlinerItems`; each `OutlinerRow` has lock + visibility toggles.
|
||||
- **`engine/editor.ts`** — main `Editor` class; owns subsystems (GizmoController, CameraController, HistoryManager, SelectionBridge) and the prompt text (`editor.positive_prompt`).
|
||||
- **Undo/redo** — `HistoryManager` + command-pattern `UndoableAction` classes. Every mutation that should be undoable records an action.
|
||||
|
||||
Key boundary: the **bottom prompt bar is NOT in this library**. It is `PromptBox3D` from the external `@storyteller/ui-promptbox` package, mounted by `Stage3DBody`. Anything that changes the prompt bar crosses the pagescene ↔ ui-promptbox boundary.
|
||||
|
||||
---
|
||||
|
||||
## Feature: Scene Builder (IN PROGRESS — branch `feature/scene-builder`)
|
||||
|
||||
Status: **spec / in development.** This section describes intended behavior; not all of it is built yet. Images referenced are from Figma design frames.
|
||||
|
||||
### Top-level mode pill (net-new)
|
||||
|
||||
A segmented toggle at top-center (above the `Controls3D` toolbar) with two modes:
|
||||
|
||||
- **Build** — default. Scene composition/layout. Used unless the user switches.
|
||||
- **Record** — read-only playback/output mode (see below).
|
||||
|
||||
Rule: a scene with **no animations/keyframes (static)** defaults to **Build**.
|
||||
|
||||
This mode does not exist in code today. It needs a new store field (e.g. `sceneMode: "build" | "record"`) and a new component rendered in `Stage3DBody`. Distinct from the existing `editorState` enum (`EDIT` | `CAMERA_VIEW`), which is a different axis — don't overload it.
|
||||
|
||||
### Launch re-scope — drop promptbox, lightbox handoff, editable duration
|
||||
|
||||
- **Promptbox removed**: `SceneBuilderPromptBox` deleted (MCP not ready). The **animation
|
||||
timeline** is the sole build-mode bottom UI: collapsed `TimelineBar` by default, expanded
|
||||
`TimelineEditor` via chevron. An empty timeline is auto-created on editor init
|
||||
(`editor.timelineController.create()`), so the bar is always functional (no "Add timeline"
|
||||
button). Auto-expands once on load when `timelineTracks.length > 0` or
|
||||
`editor.sceneHasAnimation()` (imported THREE clips).
|
||||
- **Editable max duration**: `TimelineController.setDuration(s)` (clamped `[1,60]`, covers 5–30s) +
|
||||
`comps/Timeline/DurationLabel.tsx` (click the total-time → number input). Persists via `getTimeline`.
|
||||
- **Capture/Record → review modal (appears first) → app Lightbox**:
|
||||
- `CompletionModal` opens **immediately** for both kinds (`producedArtifact`) with a **local
|
||||
preview** (object URL, works offline) + upload status. Images **auto-upload** on open; videos
|
||||
show a manual **Upload** button (large). On success → `adapter.openMediaLightbox(token, kind)`
|
||||
(destinations) + close; on failure → **Retry** (so the modal is always visible even when the
|
||||
upload API is unreachable in dev). The modal — not `RenderOverlay` — owns the upload.
|
||||
- `RecordControls`: Capture → produce image `producedArtifact`; Record → encode (overlay) →
|
||||
produce video `producedArtifact`.
|
||||
- Adapter: `openMediaLightbox(token, kind)` (host resolves cdnUrl via `MediaFilesApi` and renders
|
||||
the app `<Lightbox>`); `uploadMedia` kept; `openImageInEditor`/`openVideoInEditor` **removed**
|
||||
(the Lightbox owns destination routing — Edit-on-Canvas/Make-Video/Recreate/Share/Download).
|
||||
- Uploading first means the Lightbox gets a real token + cdnUrl → its actions route correctly
|
||||
(retires the earlier blob/data-URL handoff fragility).
|
||||
|
||||
### Spec revision — 3D delegates generation to 2D/Video (branch `feature/scene-builder`)
|
||||
|
||||
The 3D editor no longer generates. It composes; 2D (pagedraw) and the video experiences generate.
|
||||
|
||||
- **Build promptbox is now the scene-builder/MCP tool only** — `comps/SceneBuilderPromptBox/`
|
||||
replaced `PromptBox3D` in `Stage3DBody` (the Manual/Prompted toggle + `buildMode` store field are
|
||||
gone; `PromptBox3D` is no longer mounted by 3D). Update is still an MCP stub (records prompt +
|
||||
history on the engine). Hosts the collapsed `TimelineBar` + Add-animation-timeline affordance.
|
||||
- **Record mode = Capture (still) / Record (timeline → video)**:
|
||||
- `engine/recording/TimelineRecorder.ts` — deterministic `seekTo`→`renderScene`→`CanvasSource.add`
|
||||
encode via **mediabunny** (already a dep). Capture reuses `snapShotOfCurrentFrame(false)`.
|
||||
- `comps/RenderOverlay/` (opaque `LoadingDots` + progress) covers the viewport while encoding.
|
||||
- Output cached locally as `producedArtifact` in the store (`{kind, blob, objectUrl, ...}`) — no
|
||||
upload from 3D.
|
||||
- **`comps/CompletionModal/`** — preview hub: image full-preview / video playback + **Delete**
|
||||
(revokes URL), **Upload** (`adapter.uploadMedia` → library token), **Edit** → 2D for images,
|
||||
Create-Video / Video-editor for videos.
|
||||
- **Adapter handoff (new)** in `adapter.ts`: `openImageInEditor`, `openVideoInEditor(target)`,
|
||||
`uploadMedia` + `PageSceneArtifact`. Host-wired in `apps/artcraft-webapp/.../pagescene` :
|
||||
image → eager-upload + `applyEditOnCanvasFromImage` (`/edit-image`); video "generate" →
|
||||
`useCreateVideoStore.setPendingRecreate` reference-video (lazy upload at generate, `/create-video`);
|
||||
video "edit" → upload + `/video-editor` (TODO: auto-drop on timeline).
|
||||
- **Camera-view regression fixes** (Phase 0): Esc + viewport double-click exit; double-click the
|
||||
frustum to enter; robust enter/exit pose handling (removed the dead `hot_items` toggle);
|
||||
frustum **pick-proxy mesh** (line raycasting disabled) so selection matches the wireframe;
|
||||
`CameraStatusPill` (top-left) shows Viewport vs render-camera and toggles.
|
||||
|
||||
Deferred: real MCP backend; pagedraw file-only (lazy) base image; video-editor timeline
|
||||
auto-import; audio in recordings; `snapShotOfCurrentFrame` render-camera reconciliation.
|
||||
|
||||
### Persistence + undo (timeline & split camera)
|
||||
|
||||
- **Timeline persists in scene JSON**: `save_manager.getSceneJson` now writes `timeline:
|
||||
getTimeline() ?? null` (was a `""` stub); `loadFromJson` restores via `loadTimeline` (guards
|
||||
legacy `""`/null). Flows through both backend save and the unmount→cache→remount roundtrip.
|
||||
Loaded objects keep their saved `object_uuid` (proxy load, `obj.uuid = json_object.object_uuid`),
|
||||
so timeline tracks (keyed by uuid) resolve.
|
||||
- **Render camera persists**: at save, the render-camera entry in `cameras[]` is overridden with the
|
||||
**live `::CAM::` transform** (`getRenderCameraTransform` dep) so it reflects gizmo moves (the store
|
||||
config otherwise goes stale). The load path strips `::CAM::` and never recreated it — fixed:
|
||||
`loadFromJson` calls `recreateCameraObject()` (→ `Scene._create_camera_obj()`, now idempotent)
|
||||
AFTER `CamerasReplacedEvent`, so the frustum lands at the restored transform. This also fixes the
|
||||
standalone "load loses the render camera" bug.
|
||||
- **Undo/redo (PER-EDIT)**: every user-facing timeline edit is ONE undo step via
|
||||
`TimelineEditAction` — whole-timeline `{live, saved}` before/after snapshots, recorded through
|
||||
`TimelineController.recordEdit` (discrete edits) or `beginTimelineEdit`/`commitTimelineEdit`
|
||||
(continuous gestures: keyframe/strip drags coalesce at pointer-up, a Motion-popover session
|
||||
commits on close). Raw dispatchers (`moveKeyframe`, `resizeClipLane`, `setKeyframeEasing`,
|
||||
`setClipTransitionEasing`) are deliberately UNRECORDED live-previews — never call them for a
|
||||
discrete edit without wrapping. `Save` now only moves the Cancel baseline (its undo restores
|
||||
the old baseline without jumping live state); `Cancel` still reverts to last saved and is
|
||||
itself undoable. `autoKeyIfTracked` bypasses recording on purpose (it rides the gizmo's
|
||||
`TransformAction`). The old `SaveTimelineAction`/`RemoveClipLaneAction` classes are retired.
|
||||
Render-camera gizmo moves were already undoable (`TransformAction` on `::CAM::`, no special-casing);
|
||||
because save reads the live `cam_obj`, a save after an undo persists the undone position.
|
||||
|
||||
### Implementation status (branch `feature/scene-builder`)
|
||||
|
||||
- **DONE — Build/Record pill**: `comps/SceneModePill/` (reuses `@storyteller/ui-tab-selector`), mounted above `Controls3D` in `Stage3DBody`. Store: `sceneMode` (default `build`) + `setSceneMode`.
|
||||
- **DONE — Manual/Prompted toggle + prompted layout**: store `buildMode: "manual" | "prompted"` (default `manual`) + `setBuildMode`. The toggle + prompted layout live in `PromptBox3D` (`@storyteller/ui-promptbox`), gated by a new optional `buildMode` / `onBuildModeChange` / `onAddTimeline` prop trio passed from `Stage3DBody`. Toggle = a `ButtonIconSelect` (Manual = pointer icon, Prompted = wand-sparkles) in a floating glass pill above the prompt card. Prompted layout hides the manual toolbar and shows: textarea + "Add animation timeline" (`onAddTimeline`, stubbed) + prompt-history popover (`usePrompt3DStore.promptHistory`) + **Update**. `handlePromptedUpdate` is a **stub**: records the prompt on the engine + pushes to history + toasts; it does NOT call the MCP backend yet.
|
||||
- **DONE — Animation timeline (in-memory)**: keyframe system + playback + full UI.
|
||||
- Data model: `engine/timeline/types.ts` (`Keyframe`/`TimelineTrack`/`TimelineData`, `EasingSpec` = cubic-bezier ctrl pts, presets) + `interpolation.ts` (`cubicBezierYForX`, `sampleTrackAt`). Reuses `TransformSnap`/`snapshotTransform`/`writeTransform`.
|
||||
- `engine/editor/TimelineController.ts` owns the timeline; ticked in `editor.ts` `renderSingleFrame()` after `entranceAnimator`. **Writes transforms only while playing or on seek** (idle never touches objects, so manual editing is safe).
|
||||
- Events `TimelineChangedEvent`/`TimelinePlayheadEvent` → `EngineStoreBridge` → store fields (`timelineExists/Expanded/Playhead/IsPlaying/Duration/Tracks/SelectedKeyframeId`). Dispatchers in `actions/timeline.ts`.
|
||||
- Undo: `engine/editor/actions/SaveTimelineAction.ts`; `Save` records it, `Cancel` reverts to last saved.
|
||||
- UI in `comps/Timeline/`: `TimelineBar` (collapsed, via promptbox `aboveStackSlot`), `TimelineEditor` (expanded, replaces the prompt box; `Stage3DBody` hides `PromptBox3D` when `timelineExpanded`), `TimelineTrackRow` (diamonds, drag, add), `MotionPopover` (presets + draggable cubic-bezier). `onAddTimeline` → `createTimeline` + expand; `showAddTimelineButton={!timelineExists}` hides the promptbox button once a timeline exists.
|
||||
- **In-memory only** — not persisted to scene JSON yet (reload clears). Rotation is euler-lerped (quaternion slerp is a possible refinement). Camera track is absent until the camera-split slice creates `::CAM::`.
|
||||
- **DONE — Camera split (single camera) + camera view**: the render camera is now a selectable
|
||||
wireframe-frustum placeholder in the scene.
|
||||
- `scene.ts` `_create_camera_obj()` rewritten to build a `THREE.Group` named `"::CAM::"` (line
|
||||
frustum, layer 1 = editor-only/excluded from render), positioned at the first non-`main`
|
||||
store camera; called in `initialize()`/`clear()`; `refreshCamObj` resolves it. No
|
||||
`userData.media_id` → the save path (`saveToScene`, which only serializes `media_id` children)
|
||||
skips it, so scenes aren't corrupted.
|
||||
- Enter/exit camera view reuses the existing `CameraController.switchCameraView()` +
|
||||
`tickPerFrame` sync (canvas renders through the viewport camera, which is moved to the render
|
||||
camera — no render-path change). Trigger: new `CameraViewToggleRequestedEvent` → `editor.ts`
|
||||
subscription → `switchCameraView()`. Emitted by (a) `scene_manager_api.ts` `double_click()`
|
||||
when the selected object is `::CAM::` (covers viewport + outliner-row double-click), and
|
||||
(b) `actions/cameraView.ts` `toggleCameraView` (outliner icon + exit button).
|
||||
- Outliner: `OutlinerItem.isCamera` (set in `convert_object`) → `Outliner.tsx` renders a
|
||||
view-from-camera button (`faArrowRightToBracket`) left of the lock icon.
|
||||
- `comps/ExitCameraView/` shows the "EXIT CAMERA VIEW" pill while `editorState === CAMERA_VIEW`.
|
||||
- **DONE — Record mode**: `sceneMode === "record"` is read-only playback through the render camera.
|
||||
- `Stage3DBody` effect on `isRecord`: `cameraController.enterCameraView()` + `setLocked(true)` on
|
||||
entry; `setLocked(false)` + `exitCameraView()` on exit. `switchCameraView()` was refactored into
|
||||
idempotent `enterCameraView()`/`exitCameraView()`; new `CameraController.locked` gates the
|
||||
FreeCam integration in `tickPerFrame` (viewport can't be flown in record).
|
||||
- Build chrome hidden in record (ControlsTopButtons, Controls3D, top-right cluster, Outliner,
|
||||
PreviewEngineCamera, ControlPanelSceneObject, PromptBox3D, TimelineEditor). The **mode pill
|
||||
stays**. `ExitCameraView` is suppressed in record (the pill leaves record).
|
||||
- `comps/RecordControls/`: bottom-center read-only `TimelineBar` (`readOnly` hides the expand
|
||||
chevron — playback only, no editing) + **Capture** (`editor.snapShotOfCurrentFrame(true)` — still
|
||||
image download) and **Record** (red; **stub** — no video pipeline in pagescene yet).
|
||||
- **NOT STARTED**: timeline persistence to scene JSON; natural-language motion clips; multi-camera;
|
||||
render-camera transform round-trip to store `cameras[]`; **video render behind the Record button**;
|
||||
the record-mode "expand timeline into a grayed-out read-only editor" (currently the expand chevron
|
||||
is simply hidden in record).
|
||||
|
||||
### Feature: Mixamo animation clips on the timeline (IN PROGRESS)
|
||||
|
||||
Characters can carry skeletal (Mixamo) animation clips on the timeline, alongside the transform-
|
||||
keyframe tracks. Answers locked with the user: retarget onto the character's skeleton; clips +
|
||||
keyframes coexist; default clip source = the 37 curated demo clips. **Playback is sequential, one
|
||||
clip at a time** — a character's clips live on a **single row** and cannot overlap (see the overlap
|
||||
guard below). The earlier "stacked lanes" idea was dropped: stacking only pays off for *additive
|
||||
layering* (e.g. a base walk + an upper-body wave playing together), which needs per-lane weights +
|
||||
bone masking and is out of scope. If layering is ever wanted, that's the feature to build; until
|
||||
then one row per character is the right model.
|
||||
|
||||
- **Data model (phase A, done)**: `engine/timeline/types.ts` — `ClipStrip` (`sourceMediaId`,
|
||||
`startTime`, `duration`, `loop`) + `ClipLane` (`objectUuid`, `strip`); `TimelineData.clipLanes[]`.
|
||||
Only the reference + placement is serialized; the `AnimationClip` is resolved at runtime.
|
||||
`TimelineController` has `addClipLane` / `moveClipLane` / `removeClipLane` / `clipLanesFor`;
|
||||
`TimelineChangedEvent` carries `clipLanes` → store `timelineClipLanes`. Persists via the existing
|
||||
`getTimeline`/`loadTimeline` (rides `TimelineData`).
|
||||
- **Playback engine (phase B, done — UNVERIFIED at runtime)**: `engine/animation/CharacterAnimationManager.ts`.
|
||||
One `THREE.AnimationMixer` per character; each lane → an `AnimationAction`. Playhead-driven +
|
||||
**deterministic**: on every `TimelineController.evaluate()` it sets each action's absolute `time`
|
||||
from the playhead (paused actions) then `mixer.update(0)` — scrub, play, and frame-accurate record
|
||||
all pose identically. `syncClipLanes()` reconciles the runtime on every clip-lane change; a
|
||||
monotonic load token guards against superseded async clip loads. Clips are loaded via
|
||||
`Scene.loadRawGlb(media_id)` (raw GLTF, not added to the scene) and bound **by node name** —
|
||||
relies on the shared `mixorig:*` naming so a clip's tracks resolve against the character's bones
|
||||
directly (direct-bind first; `SkeletonUtils.retargetClip` is the documented fallback if a rig uses
|
||||
different bone names — **not yet wired**, pending the runtime spike).
|
||||
- **Trigger UI (done — moved)**: the old right-docked `comps/AnimationsDrawer/` is retired. The
|
||||
clip source is now the **"Animations" tab in `comps/AssetMenu/AssetModal.tsx`**: server-curated
|
||||
animations (`list_featured` with `engine_category=animation`, GLB/GLTF only) REPLACE the 37
|
||||
`demoAnimationItems` when the API returns any; the demos are only the empty-API fallback.
|
||||
Animation cards are **click-to-add** (earliest free slot on the selected character's row; toast
|
||||
if no character is selected or the row is full) **and pointer-draggable** via the shared
|
||||
`DndAsset` singleton (`ANIMATION` branch): drop on a character in the 3D scene (earliest free
|
||||
slot) or on a character's timeline row (pointer-x → time via the `data-timeline-ruler` rect;
|
||||
rows tag `data-clip-drop-uuid`). While dragging, the shared `DragGhost` shows a green/red
|
||||
**compatibility badge** (`PageSceneStore.animationDropState`): DndAsset preloads the clip's
|
||||
track node names (`loadRawGlb`, cached per media id) and checks them against the hovered
|
||||
character's nodes — the same direct-bind rule `CharacterAnimationManager` plays by; no
|
||||
retargeting. Incompatible or slot-less drops are rejected with a toast. A drop that beats the
|
||||
clip's name-inspection fetch is **queued** and completes (fully validated) when the names
|
||||
resolve — never punted back to the user (`completeAnimationDrop`; a re-drop of the SAME clip
|
||||
supersedes its earlier queued drop via a per-clip token, while queued drops of other clips
|
||||
complete independently). The old HTML5 DnD path (`ANIMATION_CLIP_MIME`) is removed. A full row
|
||||
(`resolveFreeStart` → null) **rejects** the add instead of overlapping.
|
||||
- **Standalone AnimationsModal (done)**: `comps/AssetMenu/AnimationsModal.tsx` — an
|
||||
animations-only sibling of `AssetModal` (same `@storyteller/ui-modal` floating-panel props,
|
||||
same `ItemElements` grid, same drag-under semantics) with All / Presets / Uploaded tabs.
|
||||
Opened by the **"Add Animation" button** next to "Enter Pose Mode" (`PoseModeSelector`,
|
||||
visible when a character is selected and not posing) and by the **"Animations" entry in the
|
||||
Controls3D "+" popover** (not auth-gated, like "presets"), both via
|
||||
`actions/openAnimationsModal.ts`.
|
||||
The two library modals are **mutually exclusive** (each open action closes the other), so
|
||||
`DndAsset.endDrag` and `ItemElement`'s click-add close whichever is visible (reopen-off
|
||||
path). Animations data (uploads-first + featured-else-demos fallback) lives in the shared
|
||||
`comps/AssetMenu/hooks/useAnimationLibrary.tsx`, consumed by both modals.
|
||||
**Upload from the modal**: an "Upload animation" button (sidebar bottom, above "Reopen
|
||||
after adding") plus an empty-state CTA on the Uploaded tab open the host uploader through
|
||||
the same `renderAssetUploader` adapter slot AssetModal uses. The slot (and `UploadModal3D`)
|
||||
gained an optional `initialCategory?: "animation" | "character"` (literal strings — the
|
||||
pagescene and upload-modal `FilterEngineCategories` enums aren't cross-assignable): it
|
||||
pre-checks the matching "Upload as ..." toggle and suppresses the My-Library popup on
|
||||
success (the panel shows the result itself). On success the modal refetches user
|
||||
animations and jumps to the Uploaded tab (only if the category stayed Animation).
|
||||
Anonymous users get `promptSignup("upload-3d")` instead, same as the Controls3D entries.
|
||||
- **Timeline reveal on add (done)**: a successful `addClipToCharacter` (click or drop, either
|
||||
modal) sets `timelineExpanded` + `timelineRevealObjectUuid`; an effect in `TimelineEditor`
|
||||
scrolls the row list to `[data-timeline-row-uuid]` (every row is tagged) then clears the
|
||||
field. A store field because `TimelineEditor` is unmounted while collapsed — expand + scroll
|
||||
can't happen in the caller's tick.
|
||||
- **Timeline focus toggle (done)**: crosshair button in the `TimelineEditor` transport bar
|
||||
(`timelineFocusSelected`, store-held so it survives the collapse/expand remount) narrows the
|
||||
row list to the selected object's track. Falls back to all rows when nothing is selected (or
|
||||
the selection has no outliner row), and the reveal effect switches focus off when a clip
|
||||
lands on a non-selected object so the new strip is never hidden.
|
||||
- **Timeline clip UI (phase C, done)**: `comps/Timeline/TimelineClipRow.tsx` renders **one row per
|
||||
character** (rendered even when empty, as a drop hint) holding all that character's strips
|
||||
end-to-end under its keyframe row in `TimelineEditor`. Strip body drags to **move**
|
||||
(`moveClipLane`), the right edge drags to **trim** length (`resizeClipLane`), a loop chip toggles
|
||||
repeat (`setClipLoop`), the × (selected strip only) removes it (`removeClipLane`, undoable — see
|
||||
the selection bullet below). Clip-eligible rows are drop targets via `data-clip-drop-uuid` (see
|
||||
the DnD bullet). Other clip edits (move/trim/loop) ride the timeline **Save/Cancel** session
|
||||
exactly like keyframes (`cancel()` restores + re-syncs).
|
||||
- **Overlap guard (single row)**: `TimelineController.resolveFreeStart` snaps add/move to the nearest
|
||||
gap so a character's strips never overlap; `nextStartAfter` bounds trim (`resizeClipLane`) and
|
||||
auto-length (`resolveClipDuration`) against the following clip. `evaluateAt` then has at most one
|
||||
active clip at any playhead, so playback is an unambiguous sequence.
|
||||
- **Runtime revalidation (loop-free)**: mixers/rest poses hold object-INSTANCE refs, but
|
||||
delete→undo and in-session scene reloads recreate objects under the same uuids. Every lane
|
||||
runtime lands in a TERMINAL state — bound, missing-object (`boundRoot: null`), or unbindable
|
||||
(rig-less/clip-less target; one warn, no fetch) — and `CharacterAnimationManager.revalidate()`
|
||||
(subscribed to `OutlinerRefreshedEvent` alongside the skeleton-helper sync) reassesses a lane
|
||||
ONLY when the live top-level root instance for its uuid differs from `boundRoot` (object
|
||||
appeared/vanished/replaced). One map lookup per lane per refresh; no traverses, timers, fetch
|
||||
retries, or per-refresh re-adds. A uuid-resolvable root is always fully loaded (all load paths
|
||||
stamp the uuid post-load), which is what makes rig-lessness a terminal verdict. Skeleton-helper `sync()` self-heals the same two cases: it purges and
|
||||
recreates helpers that are orphaned (scene reloads strip children; the map survives) or whose
|
||||
`helper.root` no longer matches the live object for that uuid.
|
||||
- **Rest-pose in gaps** (was bind-pose): a disabled three.js action leaves the skeleton frozen on
|
||||
its last frame, so `evaluateAt` resets any character with no clip under the playhead to a **rest
|
||||
pose captured at mixer creation** (`captureRestPose` — local transforms of every node BELOW the
|
||||
root; the root is excluded so gizmo/keyframe placement never snaps back). `Skeleton.pose()` was
|
||||
abandoned: it rebuilds bone locals from inverse bind matrices and mis-scales rigs whose armature
|
||||
carries a baked scale (mixamo cm-rigs), which made characters visually VANISH when a strip moved
|
||||
off the playhead. `pruneMixers` also restores the rest pose before dropping a character's mixer.
|
||||
- **Drag preview**: animation drags ride the same pointer-based `DndAsset` pipeline as every
|
||||
other asset card, so the shared `DragGhost` (tilt card) works unchanged — plus the
|
||||
compatibility badge described above.
|
||||
- **Clip-row eligibility (widened)**: rows are no longer characters-only. Any **skinned** object
|
||||
(creatures, rigged uploads — `OutlinerItem.hasSkeleton`, computed in `convert_object`) accepts
|
||||
animation drags (bind check still gates), and any object with **baked clips**
|
||||
(`OutlinerItem.bakedClips`, from `object.animations`) gets a clip row even without a skeleton.
|
||||
- **Baked-in clips**: `ClipStrip.bakedClipIndex` marks a strip sourced from the object's own
|
||||
`animations[]` (with `sourceMediaId: ""`); `CharacterAnimationManager.addLane` resolves it
|
||||
synchronously from the model instead of `loadRawGlb`, and **CLONES the clip per lane** —
|
||||
`mixer.clipAction` caches one action per (clip, root), so two strips of the same raw baked
|
||||
clip would fight over a single action and `uncacheClip` on removal would kill the sibling's. Added via the **"+" picker** on the
|
||||
object's clip row (`TimelineClipRow`, `addBakedClipToObject`, earliest free slot, real length
|
||||
known up front so no `autoDuration`). INVARIANT: baked clips are **never removed from the THREE
|
||||
model** — removing a baked strip only unschedules it (mixer uncache is mixer-state only); the
|
||||
clip stays on `object.animations` and in the picker. Library-sourced strips remain fully
|
||||
removable. ⚠️ Runtime-unverified: whether `object.animations` survives the save/load roundtrip
|
||||
(the stash in `scene.ts` is gated on `auto_add`), and that the outliner refresh timing surfaces
|
||||
`bakedClips` after async GLB loads.
|
||||
- **Strip width = play window (revised)**: fresh strips default to a compact
|
||||
`DEFAULT_CLIP_DURATION` (1 s) instead of adopting the clip's natural length — long clips were
|
||||
eating the whole row. The strip's on-timeline width is the **authoritative play window**
|
||||
(`LaneRuntime.stripDuration` in `evaluateAt`); the clip's natural length only drives the loop
|
||||
modulo / final-frame clamp inside it, so trimming a strip shorter genuinely cuts playback.
|
||||
Strips default to **loop: true** — dragging a strip wider than the clip keeps the motion
|
||||
cycling (a walk keeps walking, per tester expectation/old-studio behavior); the loop chip opts
|
||||
a strip into play-once/hold-last-frame instead. `resolveClipDuration` only SHRINKS an
|
||||
`autoDuration` strip when the clip is shorter than the default.
|
||||
- **Gap transitions (opt-in blending)**: `ClipStrip.transitionEasing` on the LEADING strip
|
||||
(presence = enabled; mirrors keyframes storing easing "into the next") cross-fades the strip's
|
||||
exit pose into the next strip's entry pose across the gap between them. `evaluateAt` enables
|
||||
BOTH actions pinned at their exit/entry frames with weights `(1−w, w)` from
|
||||
`cubicBezierYForX` — weights sum to 1 so the mixer computes a proper lerp; this is the one
|
||||
deliberate exception to the one-winner rule, and it's continuous at both gap edges (w=0 at the
|
||||
leading strip's end, w=1 at the next strip's start). Gaps without a transition keep the rest
|
||||
pose; leading/trailing row gaps never blend. UI: a `data-transition-chip` button at EVERY
|
||||
consecutive-strip boundary (ghost = off, solid = on) — auto-placement packs strips flush, so
|
||||
enabling on a flush/too-tight boundary first calls `ensureTransitionGap`
|
||||
(`DEFAULT_TRANSITION_GAP` = 0.3s: shift the next strip right, bounded by its follower/timeline
|
||||
end, then trim the leading strip if shifting isn't enough; a too-packed row rejects with a
|
||||
toast and mutates nothing). Click enables with `DEFAULT_EASING` and opens the generalized
|
||||
`MotionPopover` (`easing`/`onChange`/`footer` props — keyframes use the same component) with a
|
||||
"Remove transition" footer; popover selection rides `timelineEasingClipLaneId` — bridge-pruned
|
||||
both when the lane vanishes AND when it loses its transition with the id intact (Cancel /
|
||||
undo-of-Save revert the easing wholesale; an open popover would otherwise re-create it on the
|
||||
next curve drag) — and the keyframe popover takes render priority so the two never coexist.
|
||||
Edits ride the Save/Cancel session; serialization needs no special casing (`transitionEasing`
|
||||
rides `TimelineData` through every wholesale-clone path: scene JSON save/load, Save/Cancel
|
||||
snapshots, SaveTimelineAction, RemoveClipLaneAction restore).
|
||||
- **Strip selection + delete UX**: strips are click-to-select
|
||||
(`timelineSelectedClipLaneId` store field, `data-clip-strip` exempts the pointerdown from the
|
||||
click-away deselect). The **× renders only on the selected strip** (always-visible was too easy
|
||||
to hit); **Del/Backspace** removes the selected strip. Removal (either path) is **individually
|
||||
undoable** via `RemoveClipLaneAction` — the one exception to "clip edits ride Save/Cancel" —
|
||||
and to keep that honest it is **mirrored into the `saved` snapshot** (removeClipLane strips the
|
||||
lane from `saved` too, so Cancel can't resurrect a deleted strip and Save doesn't double-record
|
||||
the removal); the action captures both the live lane and its saved-snapshot entry so undo
|
||||
restores both worlds exactly (`restoreClipLane(lane, savedLane?)`).
|
||||
- **Rig-mismatch diagnostic**: `CharacterAnimationManager.clipBindsToCharacter` warns (console) when
|
||||
a clip's tracks resolve to **0** nodes on the character — the signal that `SkeletonUtils.retargetClip`
|
||||
is needed. Retarget itself is still **not wired** (direct-bind first).
|
||||
- **⚠️ Runtime unknown to verify**: whether the demo clips and character rigs share `mixamorig:*`
|
||||
bone names (→ direct bind works) or need retargeting. Drop/scrub a clip and watch for motion (and
|
||||
the console warn) to decide whether to wire the retarget fallback.
|
||||
|
||||
### ⚠️ Unverified — compiles but NOT runtime-tested
|
||||
|
||||
Everything in slices 1–3 has been validated **only** by `tsc -b` (both `pagescene` and
|
||||
`promptbox` exit 0). The app has **not** been launched; no visual, behavioral, lint, or test
|
||||
verification has been done. Outstanding items to verify live:
|
||||
|
||||
- **Slice 1 (pill)**: placement/spacing above `Controls3D` (possible double `pt-3`); white-active
|
||||
styling vs Figma; that clicking actually flips `sceneMode`.
|
||||
- **Slice 2 (prompted)**: Manual/Prompted toggle renders/positions correctly above the card;
|
||||
prompted layout hides the toolbar and shows textbox + add-timeline + history + Update; Update
|
||||
stub toasts + records history; history popover lists/restores prompts; toggle only appears when
|
||||
the host wires `onBuildModeChange`.
|
||||
- **Slice 3 (timeline)**: nothing behaviorally verified — playback actually interpolates objects;
|
||||
scrub/seek updates transforms; add/move/delete keyframes; Save+undo / Cancel; collapsed bar
|
||||
positioning via `aboveStackSlot`; expanded editor replacing the prompt box; **playhead-line
|
||||
alignment** (the `LANE_LEFT=8.75rem`/`LANE_RIGHT=2.5rem` offsets are hand-computed guesses);
|
||||
`MotionPopover` bezier drag coordinate math + popover anchoring; range-scrubber styling; euler
|
||||
rotation lerp quality; the assumption that the render loop stays alive during playback
|
||||
(`shouldRender`); "selected object gets an empty track row" behavior.
|
||||
|
||||
- **Slice 4 (camera split)**: nothing runtime-verified — that the frustum appears at the render
|
||||
camera and is excluded from the render; that selecting/moving it moves the render framing;
|
||||
double-click (viewport + outliner row) and the outliner icon enter camera view; the aspect
|
||||
frame + focal label show in camera view; EXIT returns to the prior free-cam position; the
|
||||
frustum is/ isn't viewport-click-selectable (**may need `raycaster.layers.enable(1)`** in
|
||||
MouseControls — outliner selection works regardless); `::CAM::` really is skipped by save.
|
||||
|
||||
- **Slice 5 (record mode)**: nothing runtime-verified — that switching the pill to Record enters a
|
||||
locked render-camera view (viewport truly can't be flown), hides all build chrome but keeps the
|
||||
pill, and shows the read-only playback bar + Capture/Record; that Capture downloads the composed
|
||||
frame; that switching back to Build restores the prior free-cam view and chrome; interaction
|
||||
between record and a manually-entered camera view; whether `switchCameraView` refactor (idempotent
|
||||
enter/exit) preserved the original toggle behavior. Record button is a **no-op stub**.
|
||||
|
||||
Treat all of the above as needing a live pass before shipping.
|
||||
|
||||
### Build mode has two usability sub-modes
|
||||
|
||||
- **Manual** — identical to current behavior. User manually moves/places/transforms objects with the gizmo and toolbar. This is what exists today.
|
||||
- **Prompted (Builder)** — the prompt bar changes to show only:
|
||||
- a text box,
|
||||
- an **"Add animation timeline"** button,
|
||||
- a **history** button (previous prompts),
|
||||
- an **"Update"** primary button.
|
||||
|
||||
Prompted mode drives a **JSON-style, MCP-esque backend integration** (in development). The flow: an LLM (e.g. Claude) reads the scene's **scene-descriptor data** + the user's prompt, makes edits, **outputs a new scene descriptor**, which is then applied to the 3D scene.
|
||||
|
||||
The **"Add animation timeline"** button only appears when the scene is **static** (no keyframes/animations on any object). Clicking it reveals the animation timeline bar (see below).
|
||||
|
||||
### Animation timeline
|
||||
|
||||
**Collapsed state** (a slim bar that sits above the prompt box, replacing the "Add animation timeline" button once a timeline exists):
|
||||
- ▶ play/pause · current-time readout (`0:00`) · scrubber (blue fill to playhead) · total duration (`0:10`) · **chevron ▾**.
|
||||
- The **chevron ▾ expands the timeline downward, collapsing the prompt box** (they trade vertical space).
|
||||
|
||||
**Expanded state** (full multi-track keyframe editor):
|
||||
- **Transport (top-left):** skip-to-start ⏮ · play ▶ · skip-to-end ⏭.
|
||||
- **Time ruler** with a draggable **playhead** (`0:00 … 0:04 … 0:10`).
|
||||
- **One track per object.** A track can hold either:
|
||||
- **Diamond keyframes** — each diamond stores the object's **full transform (position, rotation, scale)** at that moment. Tapping a diamond jumps the playhead to it. Each row has an "add keyframe" diamond button at its right edge.
|
||||
- **Natural-language motion clip** — a text clip spanning a time range (e.g. *"walks to car - opens door - gets in"*) that drives generated motion.
|
||||
- **Any object can have either type.** For simplicity, an early version may show only raw keyframes and drop the natural-language grouping.
|
||||
- **Motion popover** — appears when a keyframe is selected; edits the **easing/interpolation curve into the next keyframe**. Contains ~4 **easing presets** at top plus a large **editable bezier curve** with draggable **handles**; a **dotted line** shows the linear reference.
|
||||
- **Cancel / Save** (bottom-right):
|
||||
- **Cancel** — drops all changes back to the last saved timeline state.
|
||||
- **Save** — commits the changes. **This is an undoable action** (record it via `HistoryManager` / an `UndoableAction`).
|
||||
|
||||
### Camera model change (Blender-style)
|
||||
|
||||
Introduce a distinction between:
|
||||
|
||||
- **Viewport (scene) camera** — what the user flies around with; the editing view.
|
||||
- **Render camera(s)** — the camera(s) that define the final render output.
|
||||
|
||||
Render cameras become **objects within the 3D scene**: visible/selectable in the editor (with a camera gizmo/frustum, as in the Figma), but **excluded from the final render**. This mirrors professional 3D software (Blender).
|
||||
|
||||
Render cameras appear in the **Outliner** (left "Scene" panel). A camera row shows a **"view from camera"** icon positioned **right of the camera item's name and left of the lock icon** (visible in the reference frames). Clicking it **switches the viewport camera to the selected render camera** and lets the user control it with the usual navigation keybinds.
|
||||
|
||||
**This is the big architectural change** — but the engine is **already partially scaffolded** for it. Verify these anchors before building on them:
|
||||
|
||||
- **Two cameras already exist** in `engine/editor/CameraController.ts`: `camera` (viewport/free-fly, sees layer 1 incl. gizmos) and `render_camera` (layer 0 only, clean output, its own aspect ratio). Created in `editor.ts` `initialize()` from the store's camera configs.
|
||||
- **`Camera` type** lives in `frontend/libs/common/src/lib/interfaces/Camera.ts`: `{ id, label, focalLength, position, rotation, lookAt }`. Store seeds two: `"main"` (Main View, viewport) and `"cam2"` (Camera 2, render). So the store models multiple camera *configs*, but only **one** `render_camera` THREE object is currently instantiated (first non-main).
|
||||
- **Placeholder object `"::CAM::"`** is referenced throughout (`scene.getObjectByName("::CAM::")`, `CameraController.refreshCamObj`, layer 1 so it's editor-only/excluded from render) — **but `_create_camera_obj()` is commented out** in `scene.ts`, so no camera mesh is actually in the scene graph today. The Outliner already knows to render a `::CAM::` object as a "Camera" row (`scene_manager_api.ts` `convert_object`) — it just never appears because the object isn't created.
|
||||
- **`CAMERA_VIEW` state** (`enums/EditorState.ts`) + `CameraController.switchCameraView()` already implement enter/exit: on enter it copies `cam_obj` transform → viewport camera, hides `cam_obj`/objects/gizmo; on exit it restores. **Caveat:** today it *repositions the single viewport camera* rather than truly swapping to render through a separate scene camera. `PreviewEngineCamera.tsx` + an unused `camViewCanvasEl` exist as a secondary preview surface.
|
||||
- **Letterbox/aspect overlay** already built: `comps/SceneContainer/Letterbox.tsx` reads `cameraAspectRatio`, draws mattes, toggled via `editorLetterBox`. `CameraAspectRatio` enum + `getRenderDimensions()` map to pixel sizes.
|
||||
- **Focal length** is per-camera (`Camera.focalLength`), converted via `CameraController.focalLengthToFov()`, applied per-frame in `tickPerFrame()`; `focalLengthDragging` tracks the slider. The `35mm` overlay label maps to this.
|
||||
- **Render path**: `editor.ts` `renderScene()` renders `activeScene.scene` through `render_camera`; but `snapShotOfCurrentFrame()` currently snapshots through the **viewport** `camera`, not `render_camera` — reconcile so exports use the render camera.
|
||||
|
||||
**Gaps to close for the true split:** (1) actually create camera placeholder meshes as scene objects (uncomment/rework `_create_camera_obj`); (2) map `cameras: Camera[]` 1:1 to scene objects; (3) make `CAMERA_VIEW` swap which camera the viewport renders through (not just reposition one); (4) support N render cameras, not just the first non-main; (5) make cameras keyframeable timeline targets. Reconcile all of this with the existing `cameras[]` + `selectedCameraId` store state.
|
||||
|
||||
### Camera view
|
||||
|
||||
Entered by **double-clicking the camera placeholder in the viewport** or the **view-from-camera button in the outliner**. In this state the **viewport camera becomes the selected render camera** — the user flies it with the usual navigation keybinds, and that motion is what gets keyframed.
|
||||
|
||||
- **Framing overlay**: the render **aspect frame** with **rule-of-thirds** guides; everything **outside the frame is dimmed** (won't be captured).
|
||||
- **Focal-length label** (e.g. `35mm`) shown top-left inside the frame. Editable; may become a keyframable value in the future.
|
||||
- The camera's frustum is not drawn while looking through it.
|
||||
- **Timeline filters to just the selected camera's track** while in camera view.
|
||||
- **Selected keyframe** renders as a **blue diamond** (at the playhead); other keyframes are white.
|
||||
- **Exit** via the **EXIT CAMERA VIEW** button (bottom-right), or **Esc** / **double-click out**. (The text button is a dev placeholder; a proper icon comes later.)
|
||||
|
||||
Camera keyframes store the camera's transform just like object keyframes (and later, potentially focal length).
|
||||
|
||||
### Record mode
|
||||
|
||||
Record mode is the **read-only output/preview** mode — practically **immutable**. It plays back the render timeline **as seen from the main render camera** only.
|
||||
|
||||
- **All build chrome is hidden**: no outliner, no transform toolbar, no prompt box, no gizmos.
|
||||
- The **viewport is locked to the render camera** (no free-fly, no object/camera editing). Framing overlay (aspect frame, dimmed exterior) is shown.
|
||||
- **The only interactive controls are play/pause and scrubber position.** Nothing in the scene can be mutated.
|
||||
- The **chevron can still expand the timeline**, but it renders **grayed out / read-only** — no keyframe editing.
|
||||
- Two actions:
|
||||
- **Capture** — export the **current frame as a still image**.
|
||||
- **Record** — render the **full animation to video**.
|
||||
|
||||
### Build ↔ Record relationship (summary of axes)
|
||||
|
||||
- **Top pill**: `sceneMode` = `build | record` (net-new).
|
||||
- **Within Build**: a usability sub-mode = `manual | prompted`.
|
||||
- **Camera view** is an orthogonal viewport state (looking through a render camera) available while editing — not a top-level mode.
|
||||
- Keep these distinct from the pre-existing `editorState` (`EDIT | CAMERA_VIEW`) and `poseMode` (`select | pose`) fields; do not overload them.
|
||||
|
||||
### Notes / deferred
|
||||
|
||||
- **JSON scene descriptor / MCP backend contract** is out of scope for the UI work — the UI and other features can be built independently. The one thing that won't function until the backend lands is the **Update button in Prompted mode** (acceptable for dev).
|
||||
- The **view-from-camera icon** is already shown in the reference frames (outliner, right of the camera name, left of the lock icon) — no separate spec needed.
|
||||
- **EXIT CAMERA VIEW** text button is a dev placeholder; a proper icon comes later.
|
||||
@@ -1,513 +0,0 @@
|
||||
# CLAUDE.md — pagescene (3D Scene Editor)
|
||||
|
||||
Guidance for working in the `pagescene` library — the 3D editor (a.k.a. "pagescene" / "Stage3D").
|
||||
|
||||
## ⚠️ Engine-canvas mounting invariant (footgun)
|
||||
|
||||
`EngineProvider`'s editor-construction effect depends on three DOM nodes: `sceneContainer`,
|
||||
`editorCanvas`, and **`camViewCanvas`** (owned by `PreviewEngineCamera` → `CameraViewCanvas`).
|
||||
If ANY of these unmounts, the effect tears down + recreates the whole `Editor` (serializing to
|
||||
cache, then reloading) — which wipes a fresh/unsaved scene. **Never conditionally unmount the
|
||||
components that host these canvases** (`SceneContainer`, `EditorCanvas`, `PreviewEngineCamera`)
|
||||
for mode/chrome gating — hide them with CSS (`hidden`) instead. (Regression fixed: record mode
|
||||
had gated `PreviewEngineCamera` behind `!isRecord`, wiping the scene on record→build toggle.)
|
||||
|
||||
## Keybinds (unified registry)
|
||||
|
||||
Every viewport shortcut is a remappable action in `@storyteller/keybinds`
|
||||
(`frontend/libs/components/keybinds`): the registry owns *which key* (preset +
|
||||
user overrides, Settings → Keybinds), `engine/keymap.ts` owns *what it does*
|
||||
(action id → handler), and `useViewportKeyboard` dispatches with a per-event
|
||||
`KeybindContext` (sceneMode / encoding / timelineExpanded / timelineSelection /
|
||||
modalTransformActive). Rules:
|
||||
|
||||
- **Availability is declared, not implied.** An `ActionDef.when(ctx)` gate
|
||||
decides if a binding is live; an unavailable binding falls through without
|
||||
consuming the event. Default rule: available unless encoding. Two actions may
|
||||
share a key ONLY if their gates are mutually exclusive (`actionsCoAvailable`
|
||||
enforces this in tests and in the settings conflict UI). Examples: Space =
|
||||
timeline play/pause in playback contexts, camera-view toggle otherwise;
|
||||
Del/Backspace = valid timeline selection → keyframe/strip delete, else scene
|
||||
object; Escape = cancel-render mid-encode, clear-selection otherwise.
|
||||
- **Do NOT add raw keydown listeners for editor operations** — register an
|
||||
action + handler + `when` gate instead. Capture-phase component listeners
|
||||
(the old TimelineEditor delete path) are exactly the pattern this replaced.
|
||||
- **Record mode** gates all scene-mutating actions off (`inBuild`); the encode
|
||||
is abortable (overlay Cancel / Escape via `encodeCancelSignal`), and
|
||||
Capture/Record always `enterCameraView()` first so output comes from the
|
||||
render camera even after peeking at the scene view.
|
||||
- Camera movement (held WASD/QE/arrows) stays in `useFreeCam`, which resolves
|
||||
its codes from the same store; its imperative guards (ctrl-chords,
|
||||
timeline-expanded arrows, record lock) must stay in sync with the registry's
|
||||
`cameraMove`/`cameraLook` gates.
|
||||
|
||||
## Behavior notes (fixes)
|
||||
|
||||
- **Record mode is fully immutable.** The viewport lock (`CameraController.locked`, set by the
|
||||
record effect) gates every camera-move path: FreeCam drag/wheel/keyboard (`useFreeCam`),
|
||||
FreeCam integration (`tickPerFrame`), AND `MouseControls` mouse-look + `focus()` (gated via
|
||||
`MouseControlsDeps.isViewportLocked`). If you add a new camera-move path, gate it on `locked`.
|
||||
- **Default camera framing** (`DEFAULT_CAMERAS.main`): a Blender-style pulled-back/elevated 3/4
|
||||
view (`pos (-4.5,4,6)`, `lookAt (0,0.5,0.6)`) so the whole render-camera (`cam2`) frustum is
|
||||
visible on a blank scene.
|
||||
- **Auto-keyframe**: mutating an **already-keyframed** object (gizmo drag-end or inspector commit)
|
||||
calls `TimelineController.autoKeyIfTracked(uuid)` → `addKeyframe` at the playhead, which
|
||||
replace-or-creates a keyframe at the current scrub point. No-op for never-keyframed objects
|
||||
(first keyframe is always explicit).
|
||||
|
||||
## Architecture (existing, do not rebuild)
|
||||
|
||||
Strict one-way data flow. Do not violate it.
|
||||
|
||||
```
|
||||
Three.js engine → typed event bus → EngineStoreBridge → Zustand store → React → actions → engine methods
|
||||
```
|
||||
|
||||
- **`Stage3D.tsx`** — public entry component; takes an `adapter` + optional slots/callbacks.
|
||||
- **`Stage3DBody.tsx`** — layout composition (canvas, panels, toolbars, modals). Mounts the external `PromptBox3D`.
|
||||
- **`PageSceneStore.ts`** — the single Zustand store (~60 fields). All UI state. The engine never imports this store.
|
||||
- **`adapter.ts`** — platform-abstraction interface. All host-specific logic (HTTP, upload, auth, navigation, modals) flows through `PageSceneAdapter`. Hosts (Tauri app, web app) inject it at mount.
|
||||
- **`comps/Controls3D/Controls3D.tsx`** — top-center toolbar: `+` add, Move/Rotate/Scale (`transformMode`), World/Local (`transformSpace`).
|
||||
- **`comps/Outliner/Outliner.tsx`** — left "Scene" panel (code name: Outliner). Driven by `outlinerItems`; each `OutlinerRow` has lock + visibility toggles.
|
||||
- **`engine/editor.ts`** — main `Editor` class; owns subsystems (GizmoController, CameraController, HistoryManager, SelectionBridge) and the prompt text (`editor.positive_prompt`).
|
||||
- **Undo/redo** — `HistoryManager` + command-pattern `UndoableAction` classes. Every mutation that should be undoable records an action.
|
||||
|
||||
Key boundary: the **bottom prompt bar is NOT in this library**. It is `PromptBox3D` from the external `@storyteller/ui-promptbox` package, mounted by `Stage3DBody`. Anything that changes the prompt bar crosses the pagescene ↔ ui-promptbox boundary.
|
||||
|
||||
---
|
||||
|
||||
## Feature: Scene Builder (IN PROGRESS — branch `feature/scene-builder`)
|
||||
|
||||
Status: **spec / in development.** This section describes intended behavior; not all of it is built yet. Images referenced are from Figma design frames.
|
||||
|
||||
### Top-level mode pill (net-new)
|
||||
|
||||
A segmented toggle at top-center (above the `Controls3D` toolbar) with two modes:
|
||||
|
||||
- **Build** — default. Scene composition/layout. Used unless the user switches.
|
||||
- **Record** — read-only playback/output mode (see below).
|
||||
|
||||
Rule: a scene with **no animations/keyframes (static)** defaults to **Build**.
|
||||
|
||||
This mode does not exist in code today. It needs a new store field (e.g. `sceneMode: "build" | "record"`) and a new component rendered in `Stage3DBody`. Distinct from the existing `editorState` enum (`EDIT` | `CAMERA_VIEW`), which is a different axis — don't overload it.
|
||||
|
||||
### Launch re-scope — drop promptbox, lightbox handoff, editable duration
|
||||
|
||||
- **Promptbox removed**: `SceneBuilderPromptBox` deleted (MCP not ready). The **animation
|
||||
timeline** is the sole build-mode bottom UI: collapsed `TimelineBar` by default, expanded
|
||||
`TimelineEditor` via chevron. An empty timeline is auto-created on editor init
|
||||
(`editor.timelineController.create()`), so the bar is always functional (no "Add timeline"
|
||||
button). Auto-expands once on load when `timelineTracks.length > 0` or
|
||||
`editor.sceneHasAnimation()` (imported THREE clips).
|
||||
- **Editable max duration**: `TimelineController.setDuration(s)` (clamped `[1,60]`, covers 5–30s) +
|
||||
`comps/Timeline/DurationLabel.tsx` (click the total-time → number input). Persists via `getTimeline`.
|
||||
- **Capture/Record → review modal (appears first) → app Lightbox**:
|
||||
- `CompletionModal` opens **immediately** for both kinds (`producedArtifact`) with a **local
|
||||
preview** (object URL, works offline) + upload status. Images **auto-upload** on open; videos
|
||||
show a manual **Upload** button (large). On success → `adapter.openMediaLightbox(token, kind)`
|
||||
(destinations) + close; on failure → **Retry** (so the modal is always visible even when the
|
||||
upload API is unreachable in dev). The modal — not `RenderOverlay` — owns the upload.
|
||||
- `RecordControls`: Capture → produce image `producedArtifact`; Record → encode (overlay) →
|
||||
produce video `producedArtifact`.
|
||||
- Adapter: `openMediaLightbox(token, kind)` (host resolves cdnUrl via `MediaFilesApi` and renders
|
||||
the app `<Lightbox>`); `uploadMedia` kept; `openImageInEditor`/`openVideoInEditor` **removed**
|
||||
(the Lightbox owns destination routing — Edit-on-Canvas/Make-Video/Recreate/Share/Download).
|
||||
- Uploading first means the Lightbox gets a real token + cdnUrl → its actions route correctly
|
||||
(retires the earlier blob/data-URL handoff fragility).
|
||||
|
||||
### Spec revision — 3D delegates generation to 2D/Video (branch `feature/scene-builder`)
|
||||
|
||||
The 3D editor no longer generates. It composes; 2D (pagedraw) and the video experiences generate.
|
||||
|
||||
- **Build promptbox is now the scene-builder/MCP tool only** — `comps/SceneBuilderPromptBox/`
|
||||
replaced `PromptBox3D` in `Stage3DBody` (the Manual/Prompted toggle + `buildMode` store field are
|
||||
gone; `PromptBox3D` is no longer mounted by 3D). Update is still an MCP stub (records prompt +
|
||||
history on the engine). Hosts the collapsed `TimelineBar` + Add-animation-timeline affordance.
|
||||
- **Record mode = Capture (still) / Record (timeline → video)**:
|
||||
- `engine/recording/TimelineRecorder.ts` — deterministic `seekTo`→`renderScene`→`CanvasSource.add`
|
||||
encode via **mediabunny** (already a dep). Capture reuses `snapShotOfCurrentFrame(false)`.
|
||||
- `comps/RenderOverlay/` (opaque `LoadingDots` + progress) covers the viewport while encoding.
|
||||
- Output cached locally as `producedArtifact` in the store (`{kind, blob, objectUrl, ...}`) — no
|
||||
upload from 3D.
|
||||
- **`comps/CompletionModal/`** — preview hub: image full-preview / video playback + **Delete**
|
||||
(revokes URL), **Upload** (`adapter.uploadMedia` → library token), **Edit** → 2D for images,
|
||||
Create-Video / Video-editor for videos.
|
||||
- **Adapter handoff (new)** in `adapter.ts`: `openImageInEditor`, `openVideoInEditor(target)`,
|
||||
`uploadMedia` + `PageSceneArtifact`. Host-wired in `apps/artcraft-webapp/.../pagescene` :
|
||||
image → eager-upload + `applyEditOnCanvasFromImage` (`/edit-image`); video "generate" →
|
||||
`useCreateVideoStore.setPendingRecreate` reference-video (lazy upload at generate, `/create-video`);
|
||||
video "edit" → upload + `/video-editor` (TODO: auto-drop on timeline).
|
||||
- **Camera-view regression fixes** (Phase 0): Esc + viewport double-click exit; double-click the
|
||||
frustum to enter; robust enter/exit pose handling (removed the dead `hot_items` toggle);
|
||||
frustum **pick-proxy mesh** (line raycasting disabled) so selection matches the wireframe;
|
||||
`CameraStatusPill` (top-left) shows Viewport vs render-camera and toggles.
|
||||
|
||||
Deferred: real MCP backend; pagedraw file-only (lazy) base image; video-editor timeline
|
||||
auto-import; audio in recordings; `snapShotOfCurrentFrame` render-camera reconciliation.
|
||||
|
||||
### Persistence + undo (timeline & split camera)
|
||||
|
||||
- **Timeline persists in scene JSON**: `save_manager.getSceneJson` now writes `timeline:
|
||||
getTimeline() ?? null` (was a `""` stub); `loadFromJson` restores via `loadTimeline` (guards
|
||||
legacy `""`/null). Flows through both backend save and the unmount→cache→remount roundtrip.
|
||||
Loaded objects keep their saved `object_uuid` (proxy load, `obj.uuid = json_object.object_uuid`),
|
||||
so timeline tracks (keyed by uuid) resolve.
|
||||
- **Render camera persists**: at save, the render-camera entry in `cameras[]` is overridden with the
|
||||
**live `::CAM::` transform** (`getRenderCameraTransform` dep) so it reflects gizmo moves (the store
|
||||
config otherwise goes stale). The load path strips `::CAM::` and never recreated it — fixed:
|
||||
`loadFromJson` calls `recreateCameraObject()` (→ `Scene._create_camera_obj()`, now idempotent)
|
||||
AFTER `CamerasReplacedEvent`, so the frustum lands at the restored transform. This also fixes the
|
||||
standalone "load loses the render camera" bug.
|
||||
- **Undo/redo (PER-EDIT)**: every user-facing timeline edit is ONE undo step via
|
||||
`TimelineEditAction` — whole-timeline `{live, saved}` before/after snapshots, recorded through
|
||||
`TimelineController.recordEdit` (discrete edits) or `beginTimelineEdit`/`commitTimelineEdit`
|
||||
(continuous gestures: keyframe/strip drags coalesce at pointer-up, a Motion-popover session
|
||||
commits on close). Raw dispatchers (`moveKeyframe`, `resizeClipLane`, `setKeyframeEasing`,
|
||||
`setClipTransitionEasing`) are deliberately UNRECORDED live-previews — never call them for a
|
||||
discrete edit without wrapping. `Save` now only moves the Cancel baseline (its undo restores
|
||||
the old baseline without jumping live state); `Cancel` still reverts to last saved and is
|
||||
itself undoable. `autoKeyIfTracked` bypasses recording on purpose (it rides the gizmo's
|
||||
`TransformAction`). The old `SaveTimelineAction`/`RemoveClipLaneAction` classes are retired.
|
||||
Render-camera gizmo moves were already undoable (`TransformAction` on `::CAM::`, no special-casing);
|
||||
because save reads the live `cam_obj`, a save after an undo persists the undone position.
|
||||
|
||||
### Implementation status (branch `feature/scene-builder`)
|
||||
|
||||
- **DONE — Build/Record pill**: `comps/SceneModePill/` (reuses `@storyteller/ui-tab-selector`), mounted above `Controls3D` in `Stage3DBody`. Store: `sceneMode` (default `build`) + `setSceneMode`.
|
||||
- **DONE — Manual/Prompted toggle + prompted layout**: store `buildMode: "manual" | "prompted"` (default `manual`) + `setBuildMode`. The toggle + prompted layout live in `PromptBox3D` (`@storyteller/ui-promptbox`), gated by a new optional `buildMode` / `onBuildModeChange` / `onAddTimeline` prop trio passed from `Stage3DBody`. Toggle = a `ButtonIconSelect` (Manual = pointer icon, Prompted = wand-sparkles) in a floating glass pill above the prompt card. Prompted layout hides the manual toolbar and shows: textarea + "Add animation timeline" (`onAddTimeline`, stubbed) + prompt-history popover (`usePrompt3DStore.promptHistory`) + **Update**. `handlePromptedUpdate` is a **stub**: records the prompt on the engine + pushes to history + toasts; it does NOT call the MCP backend yet.
|
||||
- **DONE — Animation timeline (in-memory)**: keyframe system + playback + full UI.
|
||||
- Data model: `engine/timeline/types.ts` (`Keyframe`/`TimelineTrack`/`TimelineData`, `EasingSpec` = cubic-bezier ctrl pts, presets) + `interpolation.ts` (`cubicBezierYForX`, `sampleTrackAt`). Reuses `TransformSnap`/`snapshotTransform`/`writeTransform`.
|
||||
- `engine/editor/TimelineController.ts` owns the timeline; ticked in `editor.ts` `renderSingleFrame()` after `entranceAnimator`. **Writes transforms only while playing or on seek** (idle never touches objects, so manual editing is safe).
|
||||
- Events `TimelineChangedEvent`/`TimelinePlayheadEvent` → `EngineStoreBridge` → store fields (`timelineExists/Expanded/Playhead/IsPlaying/Duration/Tracks/SelectedKeyframeId`). Dispatchers in `actions/timeline.ts`.
|
||||
- Undo: `engine/editor/actions/SaveTimelineAction.ts`; `Save` records it, `Cancel` reverts to last saved.
|
||||
- UI in `comps/Timeline/`: `TimelineBar` (collapsed, via promptbox `aboveStackSlot`), `TimelineEditor` (expanded, replaces the prompt box; `Stage3DBody` hides `PromptBox3D` when `timelineExpanded`), `TimelineTrackRow` (diamonds, drag, add), `MotionPopover` (presets + draggable cubic-bezier). `onAddTimeline` → `createTimeline` + expand; `showAddTimelineButton={!timelineExists}` hides the promptbox button once a timeline exists.
|
||||
- **In-memory only** — not persisted to scene JSON yet (reload clears). Rotation is euler-lerped (quaternion slerp is a possible refinement). Camera track is absent until the camera-split slice creates `::CAM::`.
|
||||
- **DONE — Camera split (single camera) + camera view**: the render camera is now a selectable
|
||||
wireframe-frustum placeholder in the scene.
|
||||
- `scene.ts` `_create_camera_obj()` rewritten to build a `THREE.Group` named `"::CAM::"` (line
|
||||
frustum, layer 1 = editor-only/excluded from render), positioned at the first non-`main`
|
||||
store camera; called in `initialize()`/`clear()`; `refreshCamObj` resolves it. No
|
||||
`userData.media_id` → the save path (`saveToScene`, which only serializes `media_id` children)
|
||||
skips it, so scenes aren't corrupted.
|
||||
- Enter/exit camera view reuses the existing `CameraController.switchCameraView()` +
|
||||
`tickPerFrame` sync (canvas renders through the viewport camera, which is moved to the render
|
||||
camera — no render-path change). Trigger: new `CameraViewToggleRequestedEvent` → `editor.ts`
|
||||
subscription → `switchCameraView()`. Emitted by (a) `scene_manager_api.ts` `double_click()`
|
||||
when the selected object is `::CAM::` (covers viewport + outliner-row double-click), and
|
||||
(b) `actions/cameraView.ts` `toggleCameraView` (outliner icon + exit button).
|
||||
- Outliner: `OutlinerItem.isCamera` (set in `convert_object`) → `Outliner.tsx` renders a
|
||||
view-from-camera button (`faArrowRightToBracket`) left of the lock icon.
|
||||
- `comps/ExitCameraView/` shows the "EXIT CAMERA VIEW" pill while `editorState === CAMERA_VIEW`.
|
||||
- **DONE — Record mode**: `sceneMode === "record"` is read-only playback through the render camera.
|
||||
- `Stage3DBody` effect on `isRecord`: `cameraController.enterCameraView()` + `setLocked(true)` on
|
||||
entry; `setLocked(false)` + `exitCameraView()` on exit. `switchCameraView()` was refactored into
|
||||
idempotent `enterCameraView()`/`exitCameraView()`; new `CameraController.locked` gates the
|
||||
FreeCam integration in `tickPerFrame` (viewport can't be flown in record).
|
||||
- Build chrome hidden in record (ControlsTopButtons, Controls3D, top-right cluster, Outliner,
|
||||
PreviewEngineCamera, ControlPanelSceneObject, PromptBox3D, TimelineEditor). The **mode pill
|
||||
stays**. `ExitCameraView` is suppressed in record (the pill leaves record).
|
||||
- `comps/RecordControls/`: bottom-center read-only `TimelineBar` (`readOnly` hides the expand
|
||||
chevron — playback only, no editing) + **Capture** (`editor.snapShotOfCurrentFrame(true)` — still
|
||||
image download) and **Record** (red; **stub** — no video pipeline in pagescene yet).
|
||||
- **NOT STARTED**: timeline persistence to scene JSON; natural-language motion clips; multi-camera;
|
||||
render-camera transform round-trip to store `cameras[]`; **video render behind the Record button**;
|
||||
the record-mode "expand timeline into a grayed-out read-only editor" (currently the expand chevron
|
||||
is simply hidden in record).
|
||||
|
||||
### Feature: Mixamo animation clips on the timeline (IN PROGRESS)
|
||||
|
||||
Characters can carry skeletal (Mixamo) animation clips on the timeline, alongside the transform-
|
||||
keyframe tracks. Answers locked with the user: retarget onto the character's skeleton; clips +
|
||||
keyframes coexist; default clip source = the 37 curated demo clips. **Playback is sequential, one
|
||||
clip at a time** — a character's clips live on a **single row** and cannot overlap (see the overlap
|
||||
guard below). The earlier "stacked lanes" idea was dropped: stacking only pays off for *additive
|
||||
layering* (e.g. a base walk + an upper-body wave playing together), which needs per-lane weights +
|
||||
bone masking and is out of scope. If layering is ever wanted, that's the feature to build; until
|
||||
then one row per character is the right model.
|
||||
|
||||
- **Data model (phase A, done)**: `engine/timeline/types.ts` — `ClipStrip` (`sourceMediaId`,
|
||||
`startTime`, `duration`, `loop`) + `ClipLane` (`objectUuid`, `strip`); `TimelineData.clipLanes[]`.
|
||||
Only the reference + placement is serialized; the `AnimationClip` is resolved at runtime.
|
||||
`TimelineController` has `addClipLane` / `moveClipLane` / `removeClipLane` / `clipLanesFor`;
|
||||
`TimelineChangedEvent` carries `clipLanes` → store `timelineClipLanes`. Persists via the existing
|
||||
`getTimeline`/`loadTimeline` (rides `TimelineData`).
|
||||
- **Playback engine (phase B, done — UNVERIFIED at runtime)**: `engine/animation/CharacterAnimationManager.ts`.
|
||||
One `THREE.AnimationMixer` per character; each lane → an `AnimationAction`. Playhead-driven +
|
||||
**deterministic**: on every `TimelineController.evaluate()` it sets each action's absolute `time`
|
||||
from the playhead (paused actions) then `mixer.update(0)` — scrub, play, and frame-accurate record
|
||||
all pose identically. `syncClipLanes()` reconciles the runtime on every clip-lane change; a
|
||||
monotonic load token guards against superseded async clip loads. Clips are loaded via
|
||||
`Scene.loadRawGlb(media_id)` (raw GLTF, not added to the scene) and bound **by node name** —
|
||||
relies on the shared `mixorig:*` naming so a clip's tracks resolve against the character's bones
|
||||
directly (direct-bind first; `SkeletonUtils.retargetClip` is the documented fallback if a rig uses
|
||||
different bone names — **not yet wired**, pending the runtime spike).
|
||||
- **Trigger UI (done — moved)**: the old right-docked `comps/AnimationsDrawer/` is retired. The
|
||||
clip source is now the **"Animations" tab in `comps/AssetMenu/AssetModal.tsx`**: server-curated
|
||||
animations (`list_featured` with `engine_category=animation`, GLB/GLTF only) REPLACE the 37
|
||||
`demoAnimationItems` when the API returns any; the demos are only the empty-API fallback.
|
||||
Animation cards are **click-to-add** (earliest free slot on the selected character's row; toast
|
||||
if no character is selected or the row is full) **and pointer-draggable** via the shared
|
||||
`DndAsset` singleton (`ANIMATION` branch): drop on a character in the 3D scene (earliest free
|
||||
slot) or on a character's timeline row (pointer-x → time via the `data-timeline-ruler` rect;
|
||||
rows tag `data-clip-drop-uuid`). While dragging, the shared `DragGhost` shows a green/red
|
||||
**compatibility badge** (`PageSceneStore.animationDropState`): DndAsset preloads the clip's
|
||||
track node names (`loadRawGlb`, cached per media id) and checks them against the hovered
|
||||
character's nodes — the same direct-bind rule `CharacterAnimationManager` plays by; no
|
||||
retargeting. Incompatible or slot-less drops are rejected with a toast. A drop that beats the
|
||||
clip's name-inspection fetch is **queued** and completes (fully validated) when the names
|
||||
resolve — never punted back to the user (`completeAnimationDrop`; a re-drop of the SAME clip
|
||||
supersedes its earlier queued drop via a per-clip token, while queued drops of other clips
|
||||
complete independently). The old HTML5 DnD path (`ANIMATION_CLIP_MIME`) is removed. A full row
|
||||
(`resolveFreeStart` → null) **rejects** the add instead of overlapping.
|
||||
- **Standalone AnimationsModal (done)**: `comps/AssetMenu/AnimationsModal.tsx` — an
|
||||
animations-only sibling of `AssetModal` (same `@storyteller/ui-modal` floating-panel props,
|
||||
same `ItemElements` grid, same drag-under semantics) with All / Presets / Uploaded tabs.
|
||||
Opened by the **"Add Animation" button** next to "Enter Pose Mode" (`PoseModeSelector`,
|
||||
visible when a character is selected and not posing) and by the **"Animations" entry in the
|
||||
Controls3D "+" popover** (not auth-gated, like "presets"), both via
|
||||
`actions/openAnimationsModal.ts`.
|
||||
The two library modals are **mutually exclusive** (each open action closes the other), so
|
||||
`DndAsset.endDrag` and `ItemElement`'s click-add close whichever is visible (reopen-off
|
||||
path). Animations data (uploads-first + featured-else-demos fallback) lives in the shared
|
||||
`comps/AssetMenu/hooks/useAnimationLibrary.tsx`, consumed by both modals.
|
||||
**Upload from the modal**: an "Upload animation" button (sidebar bottom, above "Reopen
|
||||
after adding") plus an empty-state CTA on the Uploaded tab open the host uploader through
|
||||
the same `renderAssetUploader` adapter slot AssetModal uses. The slot (and `UploadModal3D`)
|
||||
gained an optional `initialCategory?: "animation" | "character"` (literal strings — the
|
||||
pagescene and upload-modal `FilterEngineCategories` enums aren't cross-assignable): it
|
||||
pre-checks the matching "Upload as ..." toggle and suppresses the My-Library popup on
|
||||
success (the panel shows the result itself). On success the modal refetches user
|
||||
animations and jumps to the Uploaded tab (only if the category stayed Animation).
|
||||
Anonymous users get `promptSignup("upload-3d")` instead, same as the Controls3D entries.
|
||||
- **Timeline reveal on add (done)**: a successful `addClipToCharacter` (click or drop, either
|
||||
modal) sets `timelineExpanded` + `timelineRevealObjectUuid`; an effect in `TimelineEditor`
|
||||
scrolls the row list to `[data-timeline-row-uuid]` (every row is tagged) then clears the
|
||||
field. A store field because `TimelineEditor` is unmounted while collapsed — expand + scroll
|
||||
can't happen in the caller's tick.
|
||||
- **Timeline focus toggle (done)**: crosshair button in the `TimelineEditor` transport bar
|
||||
(`timelineFocusSelected`, store-held so it survives the collapse/expand remount) narrows the
|
||||
row list to the selected object's track. Falls back to all rows when nothing is selected (or
|
||||
the selection has no outliner row), and the reveal effect switches focus off when a clip
|
||||
lands on a non-selected object so the new strip is never hidden.
|
||||
- **Timeline clip UI (phase C, done)**: `comps/Timeline/TimelineClipRow.tsx` renders **one row per
|
||||
character** (rendered even when empty, as a drop hint) holding all that character's strips
|
||||
end-to-end under its keyframe row in `TimelineEditor`. Strip body drags to **move**
|
||||
(`moveClipLane`), the right edge drags to **trim** length (`resizeClipLane`), a loop chip toggles
|
||||
repeat (`setClipLoop`), the × (selected strip only) removes it (`removeClipLane`, undoable — see
|
||||
the selection bullet below). Clip-eligible rows are drop targets via `data-clip-drop-uuid` (see
|
||||
the DnD bullet). Other clip edits (move/trim/loop) ride the timeline **Save/Cancel** session
|
||||
exactly like keyframes (`cancel()` restores + re-syncs).
|
||||
- **Overlap guard (single row)**: `TimelineController.resolveFreeStart` snaps add/move to the nearest
|
||||
gap so a character's strips never overlap; `nextStartAfter` bounds trim (`resizeClipLane`) and
|
||||
auto-length (`resolveClipDuration`) against the following clip. `evaluateAt` then has at most one
|
||||
active clip at any playhead, so playback is an unambiguous sequence.
|
||||
- **Runtime revalidation (loop-free)**: mixers/rest poses hold object-INSTANCE refs, but
|
||||
delete→undo and in-session scene reloads recreate objects under the same uuids. Every lane
|
||||
runtime lands in a TERMINAL state — bound, missing-object (`boundRoot: null`), or unbindable
|
||||
(rig-less/clip-less target; one warn, no fetch) — and `CharacterAnimationManager.revalidate()`
|
||||
(subscribed to `OutlinerRefreshedEvent` alongside the skeleton-helper sync) reassesses a lane
|
||||
ONLY when the live top-level root instance for its uuid differs from `boundRoot` (object
|
||||
appeared/vanished/replaced). One map lookup per lane per refresh; no traverses, timers, fetch
|
||||
retries, or per-refresh re-adds. A uuid-resolvable root is always fully loaded (all load paths
|
||||
stamp the uuid post-load), which is what makes rig-lessness a terminal verdict. Skeleton-helper `sync()` self-heals the same two cases: it purges and
|
||||
recreates helpers that are orphaned (scene reloads strip children; the map survives) or whose
|
||||
`helper.root` no longer matches the live object for that uuid.
|
||||
- **Rest-pose in gaps** (was bind-pose): a disabled three.js action leaves the skeleton frozen on
|
||||
its last frame, so `evaluateAt` resets any character with no clip under the playhead to a **rest
|
||||
pose captured at mixer creation** (`captureRestPose` — local transforms of every node BELOW the
|
||||
root; the root is excluded so gizmo/keyframe placement never snaps back). `Skeleton.pose()` was
|
||||
abandoned: it rebuilds bone locals from inverse bind matrices and mis-scales rigs whose armature
|
||||
carries a baked scale (mixamo cm-rigs), which made characters visually VANISH when a strip moved
|
||||
off the playhead. `pruneMixers` also restores the rest pose before dropping a character's mixer.
|
||||
- **Drag preview**: animation drags ride the same pointer-based `DndAsset` pipeline as every
|
||||
other asset card, so the shared `DragGhost` (tilt card) works unchanged — plus the
|
||||
compatibility badge described above.
|
||||
- **Clip-row eligibility (widened)**: rows are no longer characters-only. Any **skinned** object
|
||||
(creatures, rigged uploads — `OutlinerItem.hasSkeleton`, computed in `convert_object`) accepts
|
||||
animation drags (bind check still gates), and any object with **baked clips**
|
||||
(`OutlinerItem.bakedClips`, from `object.animations`) gets a clip row even without a skeleton.
|
||||
- **Baked-in clips**: `ClipStrip.bakedClipIndex` marks a strip sourced from the object's own
|
||||
`animations[]` (with `sourceMediaId: ""`); `CharacterAnimationManager.addLane` resolves it
|
||||
synchronously from the model instead of `loadRawGlb`, and **CLONES the clip per lane** —
|
||||
`mixer.clipAction` caches one action per (clip, root), so two strips of the same raw baked
|
||||
clip would fight over a single action and `uncacheClip` on removal would kill the sibling's. Added via the **"+" picker** on the
|
||||
object's clip row (`TimelineClipRow`, `addBakedClipToObject`, earliest free slot, real length
|
||||
known up front so no `autoDuration`). INVARIANT: baked clips are **never removed from the THREE
|
||||
model** — removing a baked strip only unschedules it (mixer uncache is mixer-state only); the
|
||||
clip stays on `object.animations` and in the picker. Library-sourced strips remain fully
|
||||
removable. ⚠️ Runtime-unverified: whether `object.animations` survives the save/load roundtrip
|
||||
(the stash in `scene.ts` is gated on `auto_add`), and that the outliner refresh timing surfaces
|
||||
`bakedClips` after async GLB loads.
|
||||
- **Strip width = play window (revised)**: fresh strips default to a compact
|
||||
`DEFAULT_CLIP_DURATION` (1 s) instead of adopting the clip's natural length — long clips were
|
||||
eating the whole row. The strip's on-timeline width is the **authoritative play window**
|
||||
(`LaneRuntime.stripDuration` in `evaluateAt`); the clip's natural length only drives the loop
|
||||
modulo / final-frame clamp inside it, so trimming a strip shorter genuinely cuts playback.
|
||||
Strips default to **loop: true** — dragging a strip wider than the clip keeps the motion
|
||||
cycling (a walk keeps walking, per tester expectation/old-studio behavior); the loop chip opts
|
||||
a strip into play-once/hold-last-frame instead. `resolveClipDuration` only SHRINKS an
|
||||
`autoDuration` strip when the clip is shorter than the default.
|
||||
- **Gap transitions (opt-in blending)**: `ClipStrip.transitionEasing` on the LEADING strip
|
||||
(presence = enabled; mirrors keyframes storing easing "into the next") cross-fades the strip's
|
||||
exit pose into the next strip's entry pose across the gap between them. `evaluateAt` enables
|
||||
BOTH actions pinned at their exit/entry frames with weights `(1−w, w)` from
|
||||
`cubicBezierYForX` — weights sum to 1 so the mixer computes a proper lerp; this is the one
|
||||
deliberate exception to the one-winner rule, and it's continuous at both gap edges (w=0 at the
|
||||
leading strip's end, w=1 at the next strip's start). Gaps without a transition keep the rest
|
||||
pose; leading/trailing row gaps never blend. UI: a `data-transition-chip` button at EVERY
|
||||
consecutive-strip boundary (ghost = off, solid = on) — auto-placement packs strips flush, so
|
||||
enabling on a flush/too-tight boundary first calls `ensureTransitionGap`
|
||||
(`DEFAULT_TRANSITION_GAP` = 0.3s: shift the next strip right, bounded by its follower/timeline
|
||||
end, then trim the leading strip if shifting isn't enough; a too-packed row rejects with a
|
||||
toast and mutates nothing). Click enables with `DEFAULT_EASING` and opens the generalized
|
||||
`MotionPopover` (`easing`/`onChange`/`footer` props — keyframes use the same component) with a
|
||||
"Remove transition" footer; popover selection rides `timelineEasingClipLaneId` — bridge-pruned
|
||||
both when the lane vanishes AND when it loses its transition with the id intact (Cancel /
|
||||
undo-of-Save revert the easing wholesale; an open popover would otherwise re-create it on the
|
||||
next curve drag) — and the keyframe popover takes render priority so the two never coexist.
|
||||
Edits ride the Save/Cancel session; serialization needs no special casing (`transitionEasing`
|
||||
rides `TimelineData` through every wholesale-clone path: scene JSON save/load, Save/Cancel
|
||||
snapshots, SaveTimelineAction, RemoveClipLaneAction restore).
|
||||
- **Strip selection + delete UX**: strips are click-to-select
|
||||
(`timelineSelectedClipLaneId` store field, `data-clip-strip` exempts the pointerdown from the
|
||||
click-away deselect). The **× renders only on the selected strip** (always-visible was too easy
|
||||
to hit); **Del/Backspace** removes the selected strip. Removal (either path) is **individually
|
||||
undoable** via `RemoveClipLaneAction` — the one exception to "clip edits ride Save/Cancel" —
|
||||
and to keep that honest it is **mirrored into the `saved` snapshot** (removeClipLane strips the
|
||||
lane from `saved` too, so Cancel can't resurrect a deleted strip and Save doesn't double-record
|
||||
the removal); the action captures both the live lane and its saved-snapshot entry so undo
|
||||
restores both worlds exactly (`restoreClipLane(lane, savedLane?)`).
|
||||
- **Rig-mismatch diagnostic**: `CharacterAnimationManager.clipBindsToCharacter` warns (console) when
|
||||
a clip's tracks resolve to **0** nodes on the character — the signal that `SkeletonUtils.retargetClip`
|
||||
is needed. Retarget itself is still **not wired** (direct-bind first).
|
||||
- **⚠️ Runtime unknown to verify**: whether the demo clips and character rigs share `mixamorig:*`
|
||||
bone names (→ direct bind works) or need retargeting. Drop/scrub a clip and watch for motion (and
|
||||
the console warn) to decide whether to wire the retarget fallback.
|
||||
|
||||
### ⚠️ Unverified — compiles but NOT runtime-tested
|
||||
|
||||
Everything in slices 1–3 has been validated **only** by `tsc -b` (both `pagescene` and
|
||||
`promptbox` exit 0). The app has **not** been launched; no visual, behavioral, lint, or test
|
||||
verification has been done. Outstanding items to verify live:
|
||||
|
||||
- **Slice 1 (pill)**: placement/spacing above `Controls3D` (possible double `pt-3`); white-active
|
||||
styling vs Figma; that clicking actually flips `sceneMode`.
|
||||
- **Slice 2 (prompted)**: Manual/Prompted toggle renders/positions correctly above the card;
|
||||
prompted layout hides the toolbar and shows textbox + add-timeline + history + Update; Update
|
||||
stub toasts + records history; history popover lists/restores prompts; toggle only appears when
|
||||
the host wires `onBuildModeChange`.
|
||||
- **Slice 3 (timeline)**: nothing behaviorally verified — playback actually interpolates objects;
|
||||
scrub/seek updates transforms; add/move/delete keyframes; Save+undo / Cancel; collapsed bar
|
||||
positioning via `aboveStackSlot`; expanded editor replacing the prompt box; **playhead-line
|
||||
alignment** (the `LANE_LEFT=8.75rem`/`LANE_RIGHT=2.5rem` offsets are hand-computed guesses);
|
||||
`MotionPopover` bezier drag coordinate math + popover anchoring; range-scrubber styling; euler
|
||||
rotation lerp quality; the assumption that the render loop stays alive during playback
|
||||
(`shouldRender`); "selected object gets an empty track row" behavior.
|
||||
|
||||
- **Slice 4 (camera split)**: nothing runtime-verified — that the frustum appears at the render
|
||||
camera and is excluded from the render; that selecting/moving it moves the render framing;
|
||||
double-click (viewport + outliner row) and the outliner icon enter camera view; the aspect
|
||||
frame + focal label show in camera view; EXIT returns to the prior free-cam position; the
|
||||
frustum is/ isn't viewport-click-selectable (**may need `raycaster.layers.enable(1)`** in
|
||||
MouseControls — outliner selection works regardless); `::CAM::` really is skipped by save.
|
||||
|
||||
- **Slice 5 (record mode)**: nothing runtime-verified — that switching the pill to Record enters a
|
||||
locked render-camera view (viewport truly can't be flown), hides all build chrome but keeps the
|
||||
pill, and shows the read-only playback bar + Capture/Record; that Capture downloads the composed
|
||||
frame; that switching back to Build restores the prior free-cam view and chrome; interaction
|
||||
between record and a manually-entered camera view; whether `switchCameraView` refactor (idempotent
|
||||
enter/exit) preserved the original toggle behavior. Record button is a **no-op stub**.
|
||||
|
||||
Treat all of the above as needing a live pass before shipping.
|
||||
|
||||
### Build mode has two usability sub-modes
|
||||
|
||||
- **Manual** — identical to current behavior. User manually moves/places/transforms objects with the gizmo and toolbar. This is what exists today.
|
||||
- **Prompted (Builder)** — the prompt bar changes to show only:
|
||||
- a text box,
|
||||
- an **"Add animation timeline"** button,
|
||||
- a **history** button (previous prompts),
|
||||
- an **"Update"** primary button.
|
||||
|
||||
Prompted mode drives a **JSON-style, MCP-esque backend integration** (in development). The flow: an LLM (e.g. Claude) reads the scene's **scene-descriptor data** + the user's prompt, makes edits, **outputs a new scene descriptor**, which is then applied to the 3D scene.
|
||||
|
||||
The **"Add animation timeline"** button only appears when the scene is **static** (no keyframes/animations on any object). Clicking it reveals the animation timeline bar (see below).
|
||||
|
||||
### Animation timeline
|
||||
|
||||
**Collapsed state** (a slim bar that sits above the prompt box, replacing the "Add animation timeline" button once a timeline exists):
|
||||
- ▶ play/pause · current-time readout (`0:00`) · scrubber (blue fill to playhead) · total duration (`0:10`) · **chevron ▾**.
|
||||
- The **chevron ▾ expands the timeline downward, collapsing the prompt box** (they trade vertical space).
|
||||
|
||||
**Expanded state** (full multi-track keyframe editor):
|
||||
- **Transport (top-left):** skip-to-start ⏮ · play ▶ · skip-to-end ⏭.
|
||||
- **Time ruler** with a draggable **playhead** (`0:00 … 0:04 … 0:10`).
|
||||
- **One track per object.** A track can hold either:
|
||||
- **Diamond keyframes** — each diamond stores the object's **full transform (position, rotation, scale)** at that moment. Tapping a diamond jumps the playhead to it. Each row has an "add keyframe" diamond button at its right edge.
|
||||
- **Natural-language motion clip** — a text clip spanning a time range (e.g. *"walks to car - opens door - gets in"*) that drives generated motion.
|
||||
- **Any object can have either type.** For simplicity, an early version may show only raw keyframes and drop the natural-language grouping.
|
||||
- **Motion popover** — appears when a keyframe is selected; edits the **easing/interpolation curve into the next keyframe**. Contains ~4 **easing presets** at top plus a large **editable bezier curve** with draggable **handles**; a **dotted line** shows the linear reference.
|
||||
- **Cancel / Save** (bottom-right):
|
||||
- **Cancel** — drops all changes back to the last saved timeline state.
|
||||
- **Save** — commits the changes. **This is an undoable action** (record it via `HistoryManager` / an `UndoableAction`).
|
||||
|
||||
### Camera model change (Blender-style)
|
||||
|
||||
Introduce a distinction between:
|
||||
|
||||
- **Viewport (scene) camera** — what the user flies around with; the editing view.
|
||||
- **Render camera(s)** — the camera(s) that define the final render output.
|
||||
|
||||
Render cameras become **objects within the 3D scene**: visible/selectable in the editor (with a camera gizmo/frustum, as in the Figma), but **excluded from the final render**. This mirrors professional 3D software (Blender).
|
||||
|
||||
Render cameras appear in the **Outliner** (left "Scene" panel). A camera row shows a **"view from camera"** icon positioned **right of the camera item's name and left of the lock icon** (visible in the reference frames). Clicking it **switches the viewport camera to the selected render camera** and lets the user control it with the usual navigation keybinds.
|
||||
|
||||
**This is the big architectural change** — but the engine is **already partially scaffolded** for it. Verify these anchors before building on them:
|
||||
|
||||
- **Two cameras already exist** in `engine/editor/CameraController.ts`: `camera` (viewport/free-fly, sees layer 1 incl. gizmos) and `render_camera` (layer 0 only, clean output, its own aspect ratio). Created in `editor.ts` `initialize()` from the store's camera configs.
|
||||
- **`Camera` type** lives in `frontend/libs/common/src/lib/interfaces/Camera.ts`: `{ id, label, focalLength, position, rotation, lookAt }`. Store seeds two: `"main"` (Main View, viewport) and `"cam2"` (Camera 2, render). So the store models multiple camera *configs*, but only **one** `render_camera` THREE object is currently instantiated (first non-main).
|
||||
- **Placeholder object `"::CAM::"`** is referenced throughout (`scene.getObjectByName("::CAM::")`, `CameraController.refreshCamObj`, layer 1 so it's editor-only/excluded from render) — **but `_create_camera_obj()` is commented out** in `scene.ts`, so no camera mesh is actually in the scene graph today. The Outliner already knows to render a `::CAM::` object as a "Camera" row (`scene_manager_api.ts` `convert_object`) — it just never appears because the object isn't created.
|
||||
- **`CAMERA_VIEW` state** (`enums/EditorState.ts`) + `CameraController.switchCameraView()` already implement enter/exit: on enter it copies `cam_obj` transform → viewport camera, hides `cam_obj`/objects/gizmo; on exit it restores. **Caveat:** today it *repositions the single viewport camera* rather than truly swapping to render through a separate scene camera. `PreviewEngineCamera.tsx` + an unused `camViewCanvasEl` exist as a secondary preview surface.
|
||||
- **Letterbox/aspect overlay** already built: `comps/SceneContainer/Letterbox.tsx` reads `cameraAspectRatio`, draws mattes, toggled via `editorLetterBox`. `CameraAspectRatio` enum + `getRenderDimensions()` map to pixel sizes.
|
||||
- **Focal length** is per-camera (`Camera.focalLength`), converted via `CameraController.focalLengthToFov()`, applied per-frame in `tickPerFrame()`; `focalLengthDragging` tracks the slider. The `35mm` overlay label maps to this.
|
||||
- **Render path**: `editor.ts` `renderScene()` renders `activeScene.scene` through `render_camera`; but `snapShotOfCurrentFrame()` currently snapshots through the **viewport** `camera`, not `render_camera` — reconcile so exports use the render camera.
|
||||
|
||||
**Gaps to close for the true split:** (1) actually create camera placeholder meshes as scene objects (uncomment/rework `_create_camera_obj`); (2) map `cameras: Camera[]` 1:1 to scene objects; (3) make `CAMERA_VIEW` swap which camera the viewport renders through (not just reposition one); (4) support N render cameras, not just the first non-main; (5) make cameras keyframeable timeline targets. Reconcile all of this with the existing `cameras[]` + `selectedCameraId` store state.
|
||||
|
||||
### Camera view
|
||||
|
||||
Entered by **double-clicking the camera placeholder in the viewport** or the **view-from-camera button in the outliner**. In this state the **viewport camera becomes the selected render camera** — the user flies it with the usual navigation keybinds, and that motion is what gets keyframed.
|
||||
|
||||
- **Framing overlay**: the render **aspect frame** with **rule-of-thirds** guides; everything **outside the frame is dimmed** (won't be captured).
|
||||
- **Focal-length label** (e.g. `35mm`) shown top-left inside the frame. Editable; may become a keyframable value in the future.
|
||||
- The camera's frustum is not drawn while looking through it.
|
||||
- **Timeline filters to just the selected camera's track** while in camera view.
|
||||
- **Selected keyframe** renders as a **blue diamond** (at the playhead); other keyframes are white.
|
||||
- **Exit** via the **EXIT CAMERA VIEW** button (bottom-right), or **Esc** / **double-click out**. (The text button is a dev placeholder; a proper icon comes later.)
|
||||
|
||||
Camera keyframes store the camera's transform just like object keyframes (and later, potentially focal length).
|
||||
|
||||
### Record mode
|
||||
|
||||
Record mode is the **read-only output/preview** mode — practically **immutable**. It plays back the render timeline **as seen from the main render camera** only.
|
||||
|
||||
- **All build chrome is hidden**: no outliner, no transform toolbar, no prompt box, no gizmos.
|
||||
- The **viewport is locked to the render camera** (no free-fly, no object/camera editing). Framing overlay (aspect frame, dimmed exterior) is shown.
|
||||
- **The only interactive controls are play/pause and scrubber position.** Nothing in the scene can be mutated.
|
||||
- The **chevron can still expand the timeline**, but it renders **grayed out / read-only** — no keyframe editing.
|
||||
- Two actions:
|
||||
- **Capture** — export the **current frame as a still image**.
|
||||
- **Record** — render the **full animation to video**.
|
||||
|
||||
### Build ↔ Record relationship (summary of axes)
|
||||
|
||||
- **Top pill**: `sceneMode` = `build | record` (net-new).
|
||||
- **Within Build**: a usability sub-mode = `manual | prompted`.
|
||||
- **Camera view** is an orthogonal viewport state (looking through a render camera) available while editing — not a top-level mode.
|
||||
- Keep these distinct from the pre-existing `editorState` (`EDIT | CAMERA_VIEW`) and `poseMode` (`select | pose`) fields; do not overload them.
|
||||
|
||||
### Notes / deferred
|
||||
|
||||
- **JSON scene descriptor / MCP backend contract** is out of scope for the UI work — the UI and other features can be built independently. The one thing that won't function until the backend lands is the **Update button in Prompted mode** (acceptable for dev).
|
||||
- The **view-from-camera icon** is already shown in the reference frames (outliner, right of the camera name, left of the lock icon) — no separate spec needed.
|
||||
- **EXIT CAMERA VIEW** text button is a dev placeholder; a proper icon comes later.
|
||||
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
Generated
+87
-145
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user