Compare commits

...
Author SHA1 Message Date
Tanish Manku e9ea2a938c Scene Builder: T-pose in clip gaps + drag-preview card for animations
- Gaps show the default pose: a disabled three.js action leaves the skeleton
  frozen on its last frame, so evaluateAt now resets any character with no clip
  under the playhead to its bind (T) pose via Skeleton.pose() (skeletons cached
  per character). Fixes blank space after a clip holding the clip's last pose.
- Animation drags now show the shared DragGhost tilt card with the clip
  thumbnail: the drawer's native-DnD handlers drive dragItem/dragPosition/
  assetDraggingUnder and suppress the browser's default drag image, matching
  object/character pickup motion.
2026-07-09 00:59:24 -07:00
Tanish Manku dc5608613d webapp: stop dev API host override pointing at localhost:12345
The DEV block set the API host to window.location.origin (so /v1 calls hit the
Vite proxy → api.storyteller.ai) and then immediately called setDevelopment(),
which overrode it to http://localhost:12345 — a local backend that isn't
running. Every media-token resolution failed, so getMediaUrlByToken returned ''
and GLB fetches fell back to localhost, serving nothing. The inline comment
already described this line as disabled; it just wasn't commented out. Now it
is, so local dev resolves media through the proxy as intended.
2026-07-09 00:59:24 -07:00
Tanish Manku 1a90bd54c2 Scene Builder: one clip row per character + overlap guard
Collapse the per-character clips onto a single timeline row (playback is
sequential — one clip at a time), replacing stacked lanes. Stacking only earned
its keep for additive layering, which is out of scope.

- TimelineController: add/move route through resolveFreeStart (snap to nearest
  non-overlapping slot); resizeClipLane and resolveClipDuration clamp against
  nextStartAfter so a strip never overruns the following clip. A character's
  strips stay a non-overlapping sequence, so evaluateAt has at most one active
  clip at any playhead.
- TimelineClipRow: renders all of a character's strips on one shared row
  (shown even when empty, as a drop hint); per-strip move/trim/loop and an
  inline x to remove.
- TimelineEditor: one TimelineClipRow per character instead of one per lane.
2026-07-08 00:04:13 -07:00
Tanish Manku 2920a189f0 Scene Builder: Mixamo clip-lane timeline UI + real clip length (phase C)
Stacked clip-strip rendering on the timeline with move/trim/loop/delete, drag-
and-drop from the Animations drawer onto a character's lane, and backfill of the
clip's real length into the strip once its GLB loads.

- TimelineClipRow: one stacked strip per clip lane under its character's
  keyframe row; body drags to move, right edge trims length, loop chip toggles
  repeat, trash removes. Clip edits ride the timeline Save/Cancel session (no
  per-op undo, matching keyframes).
- TimelineEditor: each character row is a drop target for clip drags; drop time
  is read from the ruler rect. Non-character rows ignore clip drags.
- AnimationsDrawer clips are now draggable (ANIMATION_CLIP_MIME) as well as
  click-to-add.
- ClipStrip.autoDuration: a fresh drop seeds a placeholder width, then
  CharacterAnimationManager.resolveClipDuration adopts the loaded clip's true
  length (re-clamping start); a user trim clears the flag so reload never
  clobbers a hand-set duration.
- clipBindsToCharacter warns when a clip resolves 0 tracks on the character —
  the diagnostic for whether retargeting is needed.
2026-07-07 23:51:37 -07:00
Tanish Manku 7f5e82cbd9 Scene Builder: Mixamo clip playback engine + drawer (phase B, spike)
Deterministic, playhead-driven skeletal-animation playback for character clip
lanes, plus a minimal click-to-add drawer so retargeting/binding can be verified
in the viewport before the full drag-drop / stacked-strip UI is built.

- CharacterAnimationManager: one AnimationMixer per character; each clip lane is
  an AnimationAction posed at an absolute time from the playhead + mixer.update(0)
  (scrub / play / record all pose identically). syncClipLanes() reconciles the
  runtime on every lane change; a load token guards superseded async clip loads.
- Scene.loadRawGlb(media_id): raw GLTF (scene + animations) with no scene-add /
  placeholder — sources clips for binding. Clips bind by node name (shared
  mixamorig:* naming); SkeletonUtils.retargetClip is the documented fallback.
- TimelineController: getPlayhead(); evaluate() poses characters; sync on
  add/move/remove clip lane + load/cancel/clear.
- actions: addClipToCharacter / removeClipLane.
- comps/AnimationsDrawer: right-docked, gated on a selected character; click a
  curated clip to drop it as a lane at the playhead.

Compiles clean (tsc exit 0); playback is NOT yet runtime-verified.
2026-07-07 15:42:24 -07:00
Tanish Manku 5c2fbbc9f3 Scene Builder: animation clip-lane data model (Mixamo drawer, phase A)
Foundation for placing Mixamo skeletal-animation clips on the timeline as
stacked lanes under character objects, alongside the existing keyframe tracks.

- ClipStrip / ClipLane types; TimelineData gains a `clipLanes` array (only the
  clip reference + placement is serialized — the AnimationClip is resolved and
  retargeted from sourceMediaId at runtime).
- TimelineController: addClipLane / moveClipLane / removeClipLane / clipLanesFor,
  create()/loadTimeline() init + backfill clipLanes, emitChanged carries them.
- TimelineChangedEvent → store mirror (timelineClipLanes) via EngineStoreBridge.
- Persistence rides the existing timeline save/load (getTimeline/loadTimeline).
2026-07-07 15:28:50 -07:00
Tanish Manku 11c7aa979d Scene Builder fixes: record-mode camera lock, Blender default framing, auto-keyframe
- Record mode is now fully immutable: gate MouseControls mouse-look + focus()
  on the viewport lock (new MouseControlsDeps.isViewportLocked → cameraController
  .locked), matching the existing FreeCam/tick gates.
- Default main (viewport) camera pulled back + elevated to a Blender-style 3/4
  view so a blank scene shows the full render-camera frustum.
- Auto-keyframe: mutating an already-keyframed object (gizmo drag-end or panel
  commit) replaces/creates a keyframe at the playhead via
  TimelineController.autoKeyIfTracked.
- loadTimeline backfills fps/duration/tracks defensively (legacy save safety).
2026-07-07 15:23:50 -07:00
bflatastic 26aa3d3839 3D scene editor: exact camera framing, timeline UX, outliner + chrome redesign
Camera framing
- Replace the full-canvas Letterbox with CameraFrame: an inset, aspect-correct
  frame (dimmed surround, rule-of-thirds, focal label) shown only in camera
  view and record mode — the free viewport no longer shows mattes
- The frame shows the EXACT render framing: CameraController.setCameraFrameRect
  offsets the viewport projection via camera.setViewOffset; snapshots and the
  timeline recorder clear/reapply the offset so captures stay full-bleed
- recordTimeline now encodes at the render camera's resolution/aspect with the
  grid hidden (was grabbing the raw viewport canvas)
- Frame avoids the bottom UI; expanded-timeline height is measured live

Aspect ratio
- Default is now 16:9 via DEFAULT_CAMERA_ASPECT_RATIO (store, controller,
  scene reset, newScene); removed the model cold-sync that forced Square
- New top-right aspect popover (AspectRatioMenu rewritten on PopoverMenu with
  proportional AspectRatioIcon, now exported from ui-promptbox)

Timeline
- Expanded editor shows every scene object as a track row (no selection
  needed), capped at ~3.5 rows then scrollable (scrollbar hidden so lanes
  stay width-aligned with the ruler)
- Restyled per design: dark lanes, keyframe diamonds + connecting line,
  playhead diamond handle, select-none everywhere
- Scrubbing: drag the ruler to scrub smoothly, following the pointer 1:1;
  times quantize to the timeline's frame grid (30fps) — transport readout
  shows frame timecode (m:ss:ff); keyframe drags quantize the same way
- Fixed playhead/keyframe misalignment (scrollbar shrank lanes vs ruler;
  label indent must be padding, not margin, to keep the w-32 column)
- Delete/Backspace removes the selected keyframe (capture-phase, so the
  engine's delete-scene-object keybind doesn't fire); clicking away
  deselects it
- Motion (easing) popover opens from the curve chip BETWEEN keyframes (new
  timelineEasingKeyframeId store field), centered above the chip in lane
  coordinates, backdrop-blurred, dismissed on outside click; selecting a
  keyframe no longer opens it
- Camera-track playback/scrubbing is now visible in camera view: the
  timeline mirrors the animated ::CAM:: pose onto the viewport camera

Record mode
- Viewport camera fully locked: mouse-look blocked via isMovable(), right-drag
  pan and wheel zoom gated in useFreeCam (build mode unaffected)

Chrome
- Outliner restyled as the "Scene" panel (+ Add via shared openAssetModal,
  two-line rows with friendly kind labels, blue selected card), open by
  default, vertically centered, fixed 45vh height independent of aspect ratio
- Build/Record pill fully rounded (new TabSelector listClassName prop)
- CameraStatusPill moved to the top-right camera cluster, height-matched
  with the aspect popover
- ExitCameraView is a regular labeled button docked right of the timeline bar
2026-07-07 23:15:54 +07:00
Tanish Manku 06315490b7 Scene Builder launch re-scope: drop promptbox, lightbox handoff, editable timeline duration
- Remove SceneBuilderPromptBox (MCP not ready). The animation timeline is the
  sole build-mode bottom UI: collapsed bar by default, expanded editor via
  chevron; an empty timeline is auto-created on init; auto-expands on load when
  the scene has keyframes or an imported THREE animation.
- Editable max timeline duration: TimelineController.setDuration (clamp 1-60s,
  covers 5-30) + click-to-edit DurationLabel in the bar/ruler.
- Capture/Record output → review modal (CompletionModal) that appears FIRST
  with a local preview (offline-safe), then uploads (images auto, videos manual)
  with status; on success opens the app Lightbox on the token (all destinations
  for the kind), on failure shows Retry.
- Adapter: add openMediaLightbox(token, kind); keep uploadMedia; remove
  openImageInEditor/openVideoInEditor (Lightbox owns destination routing). Host
  renders <Lightbox> and resolves cdnUrl via MediaFilesApi.

Compile-verified (tsc -b clean for pagescene + changed webapp files).
2026-07-06 23:54:47 -07:00
Tanish Manku c15d8bd1ec Scene Builder checkpoint: camera view fixes, record/capture + completion modal, save/load + undo persistence
Builds on the prior Scene Builder commit. Compile-verified (tsc -b clean for
pagescene + changed webapp files); see frontend/libs/components/pagescene/CLAUDE.md.

- Camera view: fixed record→build engine-teardown wipe (PreviewEngineCamera /
  cam-view canvas must stay mounted — hide via CSS, never unmount); Esc +
  viewport double-click exit; double-click frustum to enter; robust enter/exit
  pose; frustum pick-proxy hitbox; CameraStatusPill (Viewport vs render cam).
- Build promptbox: dedicated SceneBuilderPromptBox (MCP stub) replaced
  PromptBox3D; buildMode/toggle removed.
- Record mode: Capture (still) + Record (timeline→mp4 via mediabunny),
  RenderOverlay progress, producedArtifact cache, CompletionModal
  (preview/playback + Delete/Upload/Edit) + adapter handoff to 2D/create-video/
  video-editor (lazy, data-URL image handoff; best-effort uploads).
- Persistence: timeline + render-camera transform in scene JSON (save/load +
  unmount cache); ::CAM:: recreated on load after cameras restore.
- Undo: timeline transactional (SaveTimelineAction); camera moves via TransformAction.
2026-07-06 23:26:18 -07:00
Tanish Manku 40ec291395 Scene Builder: mode pill, prompted mode, timeline, camera split, record mode
Adds the Scene Builder feature set to the pagescene 3D editor (+ the shared
ui-promptbox). All UI-level; compile-verified only (tsc -b), not yet runtime
tested — see frontend/libs/components/pagescene/CLAUDE.md for the full spec and
per-slice "unverified" list.

- Build/Record mode pill (SceneModePill, TabSelector) + store `sceneMode`.
- Build sub-modes: Manual (current) vs Prompted. Prompted prompt bar shows a
  textbox + add-animation-timeline + prompt history + Update (Update is a stub
  pending the MCP scene-descriptor backend). Toggle via ButtonIconSelect.
- Animation timeline (in-memory): TimelineController + keyframe model +
  cubic-bezier interpolation ticked in the render loop; collapsed bar +
  expanded multi-track editor + Motion easing popover (draggable bezier);
  Save is an undoable action, Cancel reverts.
- Camera split (single camera): render camera is a selectable wireframe-frustum
  placeholder ("::CAM::") in the scene; look-through camera view via
  double-click / outliner view-from-camera icon; ExitCameraView button.
- Record mode: read-only playback through the locked render camera; build
  chrome hidden; Capture (still) works, Record (video) is a stub.
2026-07-03 05:24:48 -07:00
67 changed files with 4462 additions and 586 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ if (import.meta.env.DEV) {
// (disabled — it overrides the origin above and points at localhost:12345,
// which breaks dev unless a local backend is running. Re-enable only when
// testing against a local storyteller-web.)
StorytellerApiHostStore.getInstance().setDevelopment();
// StorytellerApiHostStore.getInstance().setDevelopment();
} catch (e) {
console.warn("Failed to set dev API host override", e);
}
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useRef } from "react";
import { useCallback, useEffect, useRef, useState } from "react";
import {
Link,
useNavigate,
@@ -10,6 +10,7 @@ import { faDesktop, faHouse } from "@fortawesome/pro-solid-svg-icons";
import { Button } from "@storyteller/ui-button";
import { MediaFilesApi } from "@storyteller/api";
import { Stage3D, usePageSceneStore } from "@storyteller/ui-pagescene";
import { Lightbox } from "../../components/lightbox/lightbox";
import {
GalleryModal,
GalleryDragComponent,
@@ -239,6 +240,41 @@ function PageSceneEditor() {
navigate("/create-image");
}, [navigate]);
// Record-mode handoff: open the app Lightbox on an uploaded media token. The
// Lightbox offers every destination for the media kind (Edit-on-Canvas,
// Make-Video, Recreate, Share, Download). Resolve the CDN URL from the token
// so the preview + actions have a real URL to work with.
const [lightbox, setLightbox] = useState<{
token: string;
cdnUrl: string;
mediaClass: string;
} | null>(null);
const openMediaLightbox = useCallback(
(token: string, kind: "image" | "video") => {
void (async () => {
let cdnUrl = "";
try {
const resp = await new MediaFilesApi().GetMediaFileByToken({
mediaFileToken: token,
});
const links = (
resp?.data as { media_links?: { cdn_url?: string } } | undefined
)?.media_links;
cdnUrl = links?.cdn_url ?? "";
} catch {
// ignore — open anyway; the sidebar metadata still loads by token
}
setLightbox({
token,
cdnUrl,
mediaClass: kind === "video" ? "video" : "image",
});
})();
},
[],
);
const wrapperRef = useRef<HTMLDivElement | null>(null);
useEffect(() => {
@@ -272,6 +308,7 @@ function PageSceneEditor() {
userToken: user?.user_token,
initialSceneToken: sceneToken,
navigateToImageTo3D,
openMediaLightbox,
getViewportSize,
promptSignup: openSignupCta,
onRequestNewSceneSelector: openSceneSplash,
@@ -328,6 +365,16 @@ function PageSceneEditor() {
adds. Both components portal themselves out of this wrapper. */}
<GalleryModal mode="view" />
<GalleryDragComponent />
{/* Capture/Record handoff: the app Lightbox opens on the uploaded token
with all destinations for the media kind. */}
<Lightbox
isOpen={!!lightbox}
onClose={() => setLightbox(null)}
mediaToken={lightbox?.token}
cdnUrl={lightbox?.cdnUrl}
mediaClass={lightbox?.mediaClass}
showBatchCarousel={false}
/>
{demoToken && (
<DemoOutputOverlay imageToken={imageToken} videoToken={videoToken} />
)}
@@ -19,6 +19,7 @@ import {
UsersApi,
} from "@storyteller/api";
import type { PageSceneAdapter } from "@storyteller/ui-pagescene";
import { uploadByKind } from "../video-editor/adapters/upload-by-kind";
import {
getActiveEditor,
ToastTypes,
@@ -155,6 +156,10 @@ export interface WebAppPageSceneAdapterOptions {
userToken: string | undefined;
initialSceneToken: string | undefined;
navigateToImageTo3D: () => void;
// Record-mode handoff: open the app Lightbox on an uploaded media token
// (all destinations for the kind). Built page-side (needs local state +
// navigate), passed through here like navigateToImageTo3D.
openMediaLightbox: (token: string, kind: "image" | "video") => void;
// Wrapper size — kept in a ref so the closure sees live values without
// rebuilding the adapter on every resize.
getViewportSize: () => { width: number; height: number };
@@ -173,6 +178,7 @@ export const useWebAppPageSceneAdapter = (
userToken,
initialSceneToken,
navigateToImageTo3D,
openMediaLightbox,
getViewportSize,
promptSignup,
onRequestNewSceneSelector,
@@ -386,6 +392,13 @@ export const useWebAppPageSceneAdapter = (
navigateToImageTo3D,
openMediaLightbox,
// Persist a produced still/video to the media library. Images
// auto-upload after Capture; videos upload on demand. Returns the token.
uploadMedia: ({ kind, blob, fileName, title }) =>
uploadByKind({ kind, blob, fileName, title }),
promptSignup,
onRequestNewSceneSelector,
@@ -425,6 +438,7 @@ export const useWebAppPageSceneAdapter = (
userToken,
initialSceneToken,
navigateToImageTo3D,
openMediaLightbox,
getViewportSize,
promptSignup,
onRequestNewSceneSelector,
@@ -0,0 +1,368 @@
# CLAUDE.md — pagescene (3D Scene Editor)
Guidance for working in the `pagescene` library — the 3D editor (a.k.a. "pagescene" / "Stage3D").
## ⚠️ Engine-canvas mounting invariant (footgun)
`EngineProvider`'s editor-construction effect depends on three DOM nodes: `sceneContainer`,
`editorCanvas`, and **`camViewCanvas`** (owned by `PreviewEngineCamera` → `CameraViewCanvas`).
If ANY of these unmounts, the effect tears down + recreates the whole `Editor` (serializing to
cache, then reloading) — which wipes a fresh/unsaved scene. **Never conditionally unmount the
components that host these canvases** (`SceneContainer`, `EditorCanvas`, `PreviewEngineCamera`)
for mode/chrome gating — hide them with CSS (`hidden`) instead. (Regression fixed: record mode
had gated `PreviewEngineCamera` behind `!isRecord`, wiping the scene on record→build toggle.)
## Behavior notes (fixes)
- **Record mode is fully immutable.** The viewport lock (`CameraController.locked`, set by the
record effect) gates every camera-move path: FreeCam drag/wheel/keyboard (`useFreeCam`),
FreeCam integration (`tickPerFrame`), AND `MouseControls` mouse-look + `focus()` (gated via
`MouseControlsDeps.isViewportLocked`). If you add a new camera-move path, gate it on `locked`.
- **Default camera framing** (`DEFAULT_CAMERAS.main`): a Blender-style pulled-back/elevated 3/4
view (`pos (-4.5,4,6)`, `lookAt (0,0.5,0.6)`) so the whole render-camera (`cam2`) frustum is
visible on a blank scene.
- **Auto-keyframe**: mutating an **already-keyframed** object (gizmo drag-end or inspector commit)
calls `TimelineController.autoKeyIfTracked(uuid)` → `addKeyframe` at the playhead, which
replace-or-creates a keyframe at the current scrub point. No-op for never-keyframed objects
(first keyframe is always explicit).
## Architecture (existing, do not rebuild)
Strict one-way data flow. Do not violate it.
```
Three.js engine → typed event bus → EngineStoreBridge → Zustand store → React → actions → engine methods
```
- **`Stage3D.tsx`** — public entry component; takes an `adapter` + optional slots/callbacks.
- **`Stage3DBody.tsx`** — layout composition (canvas, panels, toolbars, modals). Mounts the external `PromptBox3D`.
- **`PageSceneStore.ts`** — the single Zustand store (~60 fields). All UI state. The engine never imports this store.
- **`adapter.ts`** — platform-abstraction interface. All host-specific logic (HTTP, upload, auth, navigation, modals) flows through `PageSceneAdapter`. Hosts (Tauri app, web app) inject it at mount.
- **`comps/Controls3D/Controls3D.tsx`** — top-center toolbar: `+` add, Move/Rotate/Scale (`transformMode`), World/Local (`transformSpace`).
- **`comps/Outliner/Outliner.tsx`** — left "Scene" panel (code name: Outliner). Driven by `outlinerItems`; each `OutlinerRow` has lock + visibility toggles.
- **`engine/editor.ts`** — main `Editor` class; owns subsystems (GizmoController, CameraController, HistoryManager, SelectionBridge) and the prompt text (`editor.positive_prompt`).
- **Undo/redo** — `HistoryManager` + command-pattern `UndoableAction` classes. Every mutation that should be undoable records an action.
Key boundary: the **bottom prompt bar is NOT in this library**. It is `PromptBox3D` from the external `@storyteller/ui-promptbox` package, mounted by `Stage3DBody`. Anything that changes the prompt bar crosses the pagescene ↔ ui-promptbox boundary.
---
## Feature: Scene Builder (IN PROGRESS — branch `feature/scene-builder`)
Status: **spec / in development.** This section describes intended behavior; not all of it is built yet. Images referenced are from Figma design frames.
### Top-level mode pill (net-new)
A segmented toggle at top-center (above the `Controls3D` toolbar) with two modes:
- **Build** — default. Scene composition/layout. Used unless the user switches.
- **Record** — read-only playback/output mode (see below).
Rule: a scene with **no animations/keyframes (static)** defaults to **Build**.
This mode does not exist in code today. It needs a new store field (e.g. `sceneMode: "build" | "record"`) and a new component rendered in `Stage3DBody`. Distinct from the existing `editorState` enum (`EDIT` | `CAMERA_VIEW`), which is a different axis — don't overload it.
### Launch re-scope — drop promptbox, lightbox handoff, editable duration
- **Promptbox removed**: `SceneBuilderPromptBox` deleted (MCP not ready). The **animation
timeline** is the sole build-mode bottom UI: collapsed `TimelineBar` by default, expanded
`TimelineEditor` via chevron. An empty timeline is auto-created on editor init
(`editor.timelineController.create()`), so the bar is always functional (no "Add timeline"
button). Auto-expands once on load when `timelineTracks.length > 0` or
`editor.sceneHasAnimation()` (imported THREE clips).
- **Editable max duration**: `TimelineController.setDuration(s)` (clamped `[1,60]`, covers 5–30s) +
`comps/Timeline/DurationLabel.tsx` (click the total-time → number input). Persists via `getTimeline`.
- **Capture/Record → review modal (appears first) → app Lightbox**:
- `CompletionModal` opens **immediately** for both kinds (`producedArtifact`) with a **local
preview** (object URL, works offline) + upload status. Images **auto-upload** on open; videos
show a manual **Upload** button (large). On success → `adapter.openMediaLightbox(token, kind)`
(destinations) + close; on failure → **Retry** (so the modal is always visible even when the
upload API is unreachable in dev). The modal — not `RenderOverlay` — owns the upload.
- `RecordControls`: Capture → produce image `producedArtifact`; Record → encode (overlay) →
produce video `producedArtifact`.
- Adapter: `openMediaLightbox(token, kind)` (host resolves cdnUrl via `MediaFilesApi` and renders
the app `<Lightbox>`); `uploadMedia` kept; `openImageInEditor`/`openVideoInEditor` **removed**
(the Lightbox owns destination routing — Edit-on-Canvas/Make-Video/Recreate/Share/Download).
- Uploading first means the Lightbox gets a real token + cdnUrl → its actions route correctly
(retires the earlier blob/data-URL handoff fragility).
### Spec revision — 3D delegates generation to 2D/Video (branch `feature/scene-builder`)
The 3D editor no longer generates. It composes; 2D (pagedraw) and the video experiences generate.
- **Build promptbox is now the scene-builder/MCP tool only** — `comps/SceneBuilderPromptBox/`
replaced `PromptBox3D` in `Stage3DBody` (the Manual/Prompted toggle + `buildMode` store field are
gone; `PromptBox3D` is no longer mounted by 3D). Update is still an MCP stub (records prompt +
history on the engine). Hosts the collapsed `TimelineBar` + Add-animation-timeline affordance.
- **Record mode = Capture (still) / Record (timeline → video)**:
- `engine/recording/TimelineRecorder.ts` — deterministic `seekTo`→`renderScene`→`CanvasSource.add`
encode via **mediabunny** (already a dep). Capture reuses `snapShotOfCurrentFrame(false)`.
- `comps/RenderOverlay/` (opaque `LoadingDots` + progress) covers the viewport while encoding.
- Output cached locally as `producedArtifact` in the store (`{kind, blob, objectUrl, ...}`) — no
upload from 3D.
- **`comps/CompletionModal/`** — preview hub: image full-preview / video playback + **Delete**
(revokes URL), **Upload** (`adapter.uploadMedia` → library token), **Edit** → 2D for images,
Create-Video / Video-editor for videos.
- **Adapter handoff (new)** in `adapter.ts`: `openImageInEditor`, `openVideoInEditor(target)`,
`uploadMedia` + `PageSceneArtifact`. Host-wired in `apps/artcraft-webapp/.../pagescene` :
image → eager-upload + `applyEditOnCanvasFromImage` (`/edit-image`); video "generate" →
`useCreateVideoStore.setPendingRecreate` reference-video (lazy upload at generate, `/create-video`);
video "edit" → upload + `/video-editor` (TODO: auto-drop on timeline).
- **Camera-view regression fixes** (Phase 0): Esc + viewport double-click exit; double-click the
frustum to enter; robust enter/exit pose handling (removed the dead `hot_items` toggle);
frustum **pick-proxy mesh** (line raycasting disabled) so selection matches the wireframe;
`CameraStatusPill` (top-left) shows Viewport vs render-camera and toggles.
Deferred: real MCP backend; pagedraw file-only (lazy) base image; video-editor timeline
auto-import; audio in recordings; `snapShotOfCurrentFrame` render-camera reconciliation.
### Persistence + undo (timeline & split camera)
- **Timeline persists in scene JSON**: `save_manager.getSceneJson` now writes `timeline:
getTimeline() ?? null` (was a `""` stub); `loadFromJson` restores via `loadTimeline` (guards
legacy `""`/null). Flows through both backend save and the unmount→cache→remount roundtrip.
Loaded objects keep their saved `object_uuid` (proxy load, `obj.uuid = json_object.object_uuid`),
so timeline tracks (keyed by uuid) resolve.
- **Render camera persists**: at save, the render-camera entry in `cameras[]` is overridden with the
**live `::CAM::` transform** (`getRenderCameraTransform` dep) so it reflects gizmo moves (the store
config otherwise goes stale). The load path strips `::CAM::` and never recreated it — fixed:
`loadFromJson` calls `recreateCameraObject()` (→ `Scene._create_camera_obj()`, now idempotent)
AFTER `CamerasReplacedEvent`, so the frustum lands at the restored transform. This also fixes the
standalone "load loses the render camera" bug.
- **Undo/redo**: timeline is **transactional** — `Save` is one undo step (`SaveTimelineAction`;
`loadTimeline` re-seeks + emits on undo/redo); `Cancel` reverts to last saved (not history).
Render-camera gizmo moves were already undoable (`TransformAction` on `::CAM::`, no special-casing);
because save reads the live `cam_obj`, a save after an undo persists the undone position.
Per-edit timeline undo remains deferred.
### Implementation status (branch `feature/scene-builder`)
- **DONE — Build/Record pill**: `comps/SceneModePill/` (reuses `@storyteller/ui-tab-selector`), mounted above `Controls3D` in `Stage3DBody`. Store: `sceneMode` (default `build`) + `setSceneMode`.
- **DONE — Manual/Prompted toggle + prompted layout**: store `buildMode: "manual" | "prompted"` (default `manual`) + `setBuildMode`. The toggle + prompted layout live in `PromptBox3D` (`@storyteller/ui-promptbox`), gated by a new optional `buildMode` / `onBuildModeChange` / `onAddTimeline` prop trio passed from `Stage3DBody`. Toggle = a `ButtonIconSelect` (Manual = pointer icon, Prompted = wand-sparkles) in a floating glass pill above the prompt card. Prompted layout hides the manual toolbar and shows: textarea + "Add animation timeline" (`onAddTimeline`, stubbed) + prompt-history popover (`usePrompt3DStore.promptHistory`) + **Update**. `handlePromptedUpdate` is a **stub**: records the prompt on the engine + pushes to history + toasts; it does NOT call the MCP backend yet.
- **DONE — Animation timeline (in-memory)**: keyframe system + playback + full UI.
- Data model: `engine/timeline/types.ts` (`Keyframe`/`TimelineTrack`/`TimelineData`, `EasingSpec` = cubic-bezier ctrl pts, presets) + `interpolation.ts` (`cubicBezierYForX`, `sampleTrackAt`). Reuses `TransformSnap`/`snapshotTransform`/`writeTransform`.
- `engine/editor/TimelineController.ts` owns the timeline; ticked in `editor.ts` `renderSingleFrame()` after `entranceAnimator`. **Writes transforms only while playing or on seek** (idle never touches objects, so manual editing is safe).
- Events `TimelineChangedEvent`/`TimelinePlayheadEvent` → `EngineStoreBridge` → store fields (`timelineExists/Expanded/Playhead/IsPlaying/Duration/Tracks/SelectedKeyframeId`). Dispatchers in `actions/timeline.ts`.
- Undo: `engine/editor/actions/SaveTimelineAction.ts`; `Save` records it, `Cancel` reverts to last saved.
- UI in `comps/Timeline/`: `TimelineBar` (collapsed, via promptbox `aboveStackSlot`), `TimelineEditor` (expanded, replaces the prompt box; `Stage3DBody` hides `PromptBox3D` when `timelineExpanded`), `TimelineTrackRow` (diamonds, drag, add), `MotionPopover` (presets + draggable cubic-bezier). `onAddTimeline` → `createTimeline` + expand; `showAddTimelineButton={!timelineExists}` hides the promptbox button once a timeline exists.
- **In-memory only** — not persisted to scene JSON yet (reload clears). Rotation is euler-lerped (quaternion slerp is a possible refinement). Camera track is absent until the camera-split slice creates `::CAM::`.
- **DONE — Camera split (single camera) + camera view**: the render camera is now a selectable
wireframe-frustum placeholder in the scene.
- `scene.ts` `_create_camera_obj()` rewritten to build a `THREE.Group` named `"::CAM::"` (line
frustum, layer 1 = editor-only/excluded from render), positioned at the first non-`main`
store camera; called in `initialize()`/`clear()`; `refreshCamObj` resolves it. No
`userData.media_id` → the save path (`saveToScene`, which only serializes `media_id` children)
skips it, so scenes aren't corrupted.
- Enter/exit camera view reuses the existing `CameraController.switchCameraView()` +
`tickPerFrame` sync (canvas renders through the viewport camera, which is moved to the render
camera — no render-path change). Trigger: new `CameraViewToggleRequestedEvent` → `editor.ts`
subscription → `switchCameraView()`. Emitted by (a) `scene_manager_api.ts` `double_click()`
when the selected object is `::CAM::` (covers viewport + outliner-row double-click), and
(b) `actions/cameraView.ts` `toggleCameraView` (outliner icon + exit button).
- Outliner: `OutlinerItem.isCamera` (set in `convert_object`) → `Outliner.tsx` renders a
view-from-camera button (`faArrowRightToBracket`) left of the lock icon.
- `comps/ExitCameraView/` shows the "EXIT CAMERA VIEW" pill while `editorState === CAMERA_VIEW`.
- **DONE — Record mode**: `sceneMode === "record"` is read-only playback through the render camera.
- `Stage3DBody` effect on `isRecord`: `cameraController.enterCameraView()` + `setLocked(true)` on
entry; `setLocked(false)` + `exitCameraView()` on exit. `switchCameraView()` was refactored into
idempotent `enterCameraView()`/`exitCameraView()`; new `CameraController.locked` gates the
FreeCam integration in `tickPerFrame` (viewport can't be flown in record).
- Build chrome hidden in record (ControlsTopButtons, Controls3D, top-right cluster, Outliner,
PreviewEngineCamera, ControlPanelSceneObject, PromptBox3D, TimelineEditor). The **mode pill
stays**. `ExitCameraView` is suppressed in record (the pill leaves record).
- `comps/RecordControls/`: bottom-center read-only `TimelineBar` (`readOnly` hides the expand
chevron — playback only, no editing) + **Capture** (`editor.snapShotOfCurrentFrame(true)` — still
image download) and **Record** (red; **stub** — no video pipeline in pagescene yet).
- **NOT STARTED**: timeline persistence to scene JSON; natural-language motion clips; multi-camera;
render-camera transform round-trip to store `cameras[]`; **video render behind the Record button**;
the record-mode "expand timeline into a grayed-out read-only editor" (currently the expand chevron
is simply hidden in record).
### Feature: Mixamo animation clips on the timeline (IN PROGRESS)
Characters can carry skeletal (Mixamo) animation clips on the timeline, alongside the transform-
keyframe tracks. Answers locked with the user: retarget onto the character's skeleton; clips +
keyframes coexist; default clip source = the 37 curated demo clips. **Playback is sequential, one
clip at a time** — a character's clips live on a **single row** and cannot overlap (see the overlap
guard below). The earlier "stacked lanes" idea was dropped: stacking only pays off for *additive
layering* (e.g. a base walk + an upper-body wave playing together), which needs per-lane weights +
bone masking and is out of scope. If layering is ever wanted, that's the feature to build; until
then one row per character is the right model.
- **Data model (phase A, done)**: `engine/timeline/types.ts` — `ClipStrip` (`sourceMediaId`,
`startTime`, `duration`, `loop`) + `ClipLane` (`objectUuid`, `strip`); `TimelineData.clipLanes[]`.
Only the reference + placement is serialized; the `AnimationClip` is resolved at runtime.
`TimelineController` has `addClipLane` / `moveClipLane` / `removeClipLane` / `clipLanesFor`;
`TimelineChangedEvent` carries `clipLanes` → store `timelineClipLanes`. Persists via the existing
`getTimeline`/`loadTimeline` (rides `TimelineData`).
- **Playback engine (phase B, done — UNVERIFIED at runtime)**: `engine/animation/CharacterAnimationManager.ts`.
One `THREE.AnimationMixer` per character; each lane → an `AnimationAction`. Playhead-driven +
**deterministic**: on every `TimelineController.evaluate()` it sets each action's absolute `time`
from the playhead (paused actions) then `mixer.update(0)` — scrub, play, and frame-accurate record
all pose identically. `syncClipLanes()` reconciles the runtime on every clip-lane change; a
monotonic load token guards against superseded async clip loads. Clips are loaded via
`Scene.loadRawGlb(media_id)` (raw GLTF, not added to the scene) and bound **by node name** —
relies on the shared `mixorig:*` naming so a clip's tracks resolve against the character's bones
directly (direct-bind first; `SkeletonUtils.retargetClip` is the documented fallback if a rig uses
different bone names — **not yet wired**, pending the runtime spike).
- **Trigger UI (done)**: `comps/AnimationsDrawer/` — right-docked, only when a character is
selected (build mode; hidden via CSS in record). Clips are **click-to-add** (drops at the playhead)
**and draggable** onto the timeline (`ANIMATION_CLIP_MIME` payload `{media_id, name}`).
- **Timeline clip UI (phase C, done)**: `comps/Timeline/TimelineClipRow.tsx` renders **one row per
character** (rendered even when empty, as a drop hint) holding all that character's strips
end-to-end under its keyframe row in `TimelineEditor`. Strip body drags to **move**
(`moveClipLane`), the right edge drags to **trim** length (`resizeClipLane`), a loop chip toggles
repeat (`setClipLoop`), the × removes it (`removeClipLane`). Each character row is a **drop target**
for clip drags (`dropClipOnCharacter` → `addClipToCharacter(…, atTime)`, time from the ruler rect).
Non-character rows ignore clip drags. No per-op undo by design — clip edits ride the timeline
**Save/Cancel** session exactly like keyframes (`cancel()` restores + re-syncs).
- **Overlap guard (single row)**: `TimelineController.resolveFreeStart` snaps add/move to the nearest
gap so a character's strips never overlap; `nextStartAfter` bounds trim (`resizeClipLane`) and
auto-length (`resolveClipDuration`) against the following clip. `evaluateAt` then has at most one
active clip at any playhead, so playback is an unambiguous sequence.
- **Bind-pose in gaps**: a disabled three.js action leaves the skeleton frozen on its last frame, so
`evaluateAt` resets any character with no clip under the playhead to its **bind (T) pose** via
`resetToBindPose` → `Skeleton.pose()` (skeletons cached per character in `getMixer`). Gaps between
strips, before the first, and after the last therefore show the default T-pose, not a held frame.
- **Drag preview**: dragging a clip drives the shared `DragGhost` (tilt card) by setting
`dragItem`/`dragPosition`/`assetDraggingUnder` on the store in the drawer's native-DnD handlers,
and suppresses the browser's default drag image — matching object/character pickup motion.
- **Real clip length (fix, done)**: a fresh drop seeds `strip.duration` with a placeholder and flags
`autoDuration`; when the GLB loads, `CharacterAnimationManager` calls
`TimelineController.resolveClipDuration(laneId, clip.duration)` to adopt the clip's true length
(and re-clamp start). A user trim clears `autoDuration` so the natural length never clobbers a
hand-set duration on reload.
- **Rig-mismatch diagnostic**: `CharacterAnimationManager.clipBindsToCharacter` warns (console) when
a clip's tracks resolve to **0** nodes on the character — the signal that `SkeletonUtils.retargetClip`
is needed. Retarget itself is still **not wired** (direct-bind first).
- **⚠️ Runtime unknown to verify**: whether the demo clips and character rigs share `mixamorig:*`
bone names (→ direct bind works) or need retargeting. Drop/scrub a clip and watch for motion (and
the console warn) to decide whether to wire the retarget fallback.
### ⚠️ Unverified — compiles but NOT runtime-tested
Everything in slices 1–3 has been validated **only** by `tsc -b` (both `pagescene` and
`promptbox` exit 0). The app has **not** been launched; no visual, behavioral, lint, or test
verification has been done. Outstanding items to verify live:
- **Slice 1 (pill)**: placement/spacing above `Controls3D` (possible double `pt-3`); white-active
styling vs Figma; that clicking actually flips `sceneMode`.
- **Slice 2 (prompted)**: Manual/Prompted toggle renders/positions correctly above the card;
prompted layout hides the toolbar and shows textbox + add-timeline + history + Update; Update
stub toasts + records history; history popover lists/restores prompts; toggle only appears when
the host wires `onBuildModeChange`.
- **Slice 3 (timeline)**: nothing behaviorally verified — playback actually interpolates objects;
scrub/seek updates transforms; add/move/delete keyframes; Save+undo / Cancel; collapsed bar
positioning via `aboveStackSlot`; expanded editor replacing the prompt box; **playhead-line
alignment** (the `LANE_LEFT=8.75rem`/`LANE_RIGHT=2.5rem` offsets are hand-computed guesses);
`MotionPopover` bezier drag coordinate math + popover anchoring; range-scrubber styling; euler
rotation lerp quality; the assumption that the render loop stays alive during playback
(`shouldRender`); "selected object gets an empty track row" behavior.
- **Slice 4 (camera split)**: nothing runtime-verified — that the frustum appears at the render
camera and is excluded from the render; that selecting/moving it moves the render framing;
double-click (viewport + outliner row) and the outliner icon enter camera view; the aspect
frame + focal label show in camera view; EXIT returns to the prior free-cam position; the
frustum is/ isn't viewport-click-selectable (**may need `raycaster.layers.enable(1)`** in
MouseControls — outliner selection works regardless); `::CAM::` really is skipped by save.
- **Slice 5 (record mode)**: nothing runtime-verified — that switching the pill to Record enters a
locked render-camera view (viewport truly can't be flown), hides all build chrome but keeps the
pill, and shows the read-only playback bar + Capture/Record; that Capture downloads the composed
frame; that switching back to Build restores the prior free-cam view and chrome; interaction
between record and a manually-entered camera view; whether `switchCameraView` refactor (idempotent
enter/exit) preserved the original toggle behavior. Record button is a **no-op stub**.
Treat all of the above as needing a live pass before shipping.
### Build mode has two usability sub-modes
- **Manual** — identical to current behavior. User manually moves/places/transforms objects with the gizmo and toolbar. This is what exists today.
- **Prompted (Builder)** — the prompt bar changes to show only:
- a text box,
- an **"Add animation timeline"** button,
- a **history** button (previous prompts),
- an **"Update"** primary button.
Prompted mode drives a **JSON-style, MCP-esque backend integration** (in development). The flow: an LLM (e.g. Claude) reads the scene's **scene-descriptor data** + the user's prompt, makes edits, **outputs a new scene descriptor**, which is then applied to the 3D scene.
The **"Add animation timeline"** button only appears when the scene is **static** (no keyframes/animations on any object). Clicking it reveals the animation timeline bar (see below).
### Animation timeline
**Collapsed state** (a slim bar that sits above the prompt box, replacing the "Add animation timeline" button once a timeline exists):
- ▶ play/pause · current-time readout (`0:00`) · scrubber (blue fill to playhead) · total duration (`0:10`) · **chevron ▾**.
- The **chevron ▾ expands the timeline downward, collapsing the prompt box** (they trade vertical space).
**Expanded state** (full multi-track keyframe editor):
- **Transport (top-left):** skip-to-start ⏮ · play ▶ · skip-to-end ⏭.
- **Time ruler** with a draggable **playhead** (`0:00 … 0:04 … 0:10`).
- **One track per object.** A track can hold either:
- **Diamond keyframes** — each diamond stores the object's **full transform (position, rotation, scale)** at that moment. Tapping a diamond jumps the playhead to it. Each row has an "add keyframe" diamond button at its right edge.
- **Natural-language motion clip** — a text clip spanning a time range (e.g. *"walks to car - opens door - gets in"*) that drives generated motion.
- **Any object can have either type.** For simplicity, an early version may show only raw keyframes and drop the natural-language grouping.
- **Motion popover** — appears when a keyframe is selected; edits the **easing/interpolation curve into the next keyframe**. Contains ~4 **easing presets** at top plus a large **editable bezier curve** with draggable **handles**; a **dotted line** shows the linear reference.
- **Cancel / Save** (bottom-right):
- **Cancel** — drops all changes back to the last saved timeline state.
- **Save** — commits the changes. **This is an undoable action** (record it via `HistoryManager` / an `UndoableAction`).
### Camera model change (Blender-style)
Introduce a distinction between:
- **Viewport (scene) camera** — what the user flies around with; the editing view.
- **Render camera(s)** — the camera(s) that define the final render output.
Render cameras become **objects within the 3D scene**: visible/selectable in the editor (with a camera gizmo/frustum, as in the Figma), but **excluded from the final render**. This mirrors professional 3D software (Blender).
Render cameras appear in the **Outliner** (left "Scene" panel). A camera row shows a **"view from camera"** icon positioned **right of the camera item's name and left of the lock icon** (visible in the reference frames). Clicking it **switches the viewport camera to the selected render camera** and lets the user control it with the usual navigation keybinds.
**This is the big architectural change** — but the engine is **already partially scaffolded** for it. Verify these anchors before building on them:
- **Two cameras already exist** in `engine/editor/CameraController.ts`: `camera` (viewport/free-fly, sees layer 1 incl. gizmos) and `render_camera` (layer 0 only, clean output, its own aspect ratio). Created in `editor.ts` `initialize()` from the store's camera configs.
- **`Camera` type** lives in `frontend/libs/common/src/lib/interfaces/Camera.ts`: `{ id, label, focalLength, position, rotation, lookAt }`. Store seeds two: `"main"` (Main View, viewport) and `"cam2"` (Camera 2, render). So the store models multiple camera *configs*, but only **one** `render_camera` THREE object is currently instantiated (first non-main).
- **Placeholder object `"::CAM::"`** is referenced throughout (`scene.getObjectByName("::CAM::")`, `CameraController.refreshCamObj`, layer 1 so it's editor-only/excluded from render) — **but `_create_camera_obj()` is commented out** in `scene.ts`, so no camera mesh is actually in the scene graph today. The Outliner already knows to render a `::CAM::` object as a "Camera" row (`scene_manager_api.ts` `convert_object`) — it just never appears because the object isn't created.
- **`CAMERA_VIEW` state** (`enums/EditorState.ts`) + `CameraController.switchCameraView()` already implement enter/exit: on enter it copies `cam_obj` transform → viewport camera, hides `cam_obj`/objects/gizmo; on exit it restores. **Caveat:** today it *repositions the single viewport camera* rather than truly swapping to render through a separate scene camera. `PreviewEngineCamera.tsx` + an unused `camViewCanvasEl` exist as a secondary preview surface.
- **Letterbox/aspect overlay** already built: `comps/SceneContainer/Letterbox.tsx` reads `cameraAspectRatio`, draws mattes, toggled via `editorLetterBox`. `CameraAspectRatio` enum + `getRenderDimensions()` map to pixel sizes.
- **Focal length** is per-camera (`Camera.focalLength`), converted via `CameraController.focalLengthToFov()`, applied per-frame in `tickPerFrame()`; `focalLengthDragging` tracks the slider. The `35mm` overlay label maps to this.
- **Render path**: `editor.ts` `renderScene()` renders `activeScene.scene` through `render_camera`; but `snapShotOfCurrentFrame()` currently snapshots through the **viewport** `camera`, not `render_camera` — reconcile so exports use the render camera.
**Gaps to close for the true split:** (1) actually create camera placeholder meshes as scene objects (uncomment/rework `_create_camera_obj`); (2) map `cameras: Camera[]` 1:1 to scene objects; (3) make `CAMERA_VIEW` swap which camera the viewport renders through (not just reposition one); (4) support N render cameras, not just the first non-main; (5) make cameras keyframeable timeline targets. Reconcile all of this with the existing `cameras[]` + `selectedCameraId` store state.
### Camera view
Entered by **double-clicking the camera placeholder in the viewport** or the **view-from-camera button in the outliner**. In this state the **viewport camera becomes the selected render camera** — the user flies it with the usual navigation keybinds, and that motion is what gets keyframed.
- **Framing overlay**: the render **aspect frame** with **rule-of-thirds** guides; everything **outside the frame is dimmed** (won't be captured).
- **Focal-length label** (e.g. `35mm`) shown top-left inside the frame. Editable; may become a keyframable value in the future.
- The camera's frustum is not drawn while looking through it.
- **Timeline filters to just the selected camera's track** while in camera view.
- **Selected keyframe** renders as a **blue diamond** (at the playhead); other keyframes are white.
- **Exit** via the **EXIT CAMERA VIEW** button (bottom-right), or **Esc** / **double-click out**. (The text button is a dev placeholder; a proper icon comes later.)
Camera keyframes store the camera's transform just like object keyframes (and later, potentially focal length).
### Record mode
Record mode is the **read-only output/preview** mode — practically **immutable**. It plays back the render timeline **as seen from the main render camera** only.
- **All build chrome is hidden**: no outliner, no transform toolbar, no prompt box, no gizmos.
- The **viewport is locked to the render camera** (no free-fly, no object/camera editing). Framing overlay (aspect frame, dimmed exterior) is shown.
- **The only interactive controls are play/pause and scrubber position.** Nothing in the scene can be mutated.
- The **chevron can still expand the timeline**, but it renders **grayed out / read-only** — no keyframe editing.
- Two actions:
- **Capture** — export the **current frame as a still image**.
- **Record** — render the **full animation to video**.
### Build ↔ Record relationship (summary of axes)
- **Top pill**: `sceneMode` = `build | record` (net-new).
- **Within Build**: a usability sub-mode = `manual | prompted`.
- **Camera view** is an orthogonal viewport state (looking through a render camera) available while editing — not a top-level mode.
- Keep these distinct from the pre-existing `editorState` (`EDIT | CAMERA_VIEW`) and `poseMode` (`select | pose`) fields; do not overload them.
### Notes / deferred
- **JSON scene descriptor / MCP backend contract** is out of scope for the UI work — the UI and other features can be built independently. The one thing that won't function until the backend lands is the **Update button in Prompted mode** (acceptable for dev).
- The **view-from-camera icon** is already shown in the reference frames (outliner, right of the camera name, left of the lock icon) — no separate spec needed.
- **EXIT CAMERA VIEW** text button is a dev placeholder; a proper icon comes later.
@@ -7,6 +7,7 @@ export type { Stage3DProps } from "./lib/Stage3D";
// Adapter (platform abstraction)
export type {
PageSceneAdapter,
PageSceneArtifact,
PageSceneGenerateRequest,
PageSceneSavePayload,
ListMediaFilesQuery,
@@ -6,12 +6,41 @@ import {
AssetFilterOption,
CameraAspectRatio,
ClipGroup,
DEFAULT_CAMERA_ASPECT_RATIO,
EditorStates,
} from "./enums";
import { MediaItem } from "./models";
import { Simple3DVector } from "./datastructures/common";
export type { Camera, FocalLengthDragging };
export type {
EasingSpec,
Keyframe,
TimelineTrack,
TimelineData,
ClipLane,
ClipStrip,
} from "./engine/timeline/types";
import type { TimelineTrack, ClipLane } from "./engine/timeline/types";
// A still (Capture) or video (Record) produced by Record mode, cached
// locally (object URL) before any upload. Powers the completion modal's
// preview/playback and the 2D/video handoff.
export interface ProducedArtifact {
kind: "image" | "video";
blob: Blob;
objectUrl: string;
fileName: string;
mimeType: string;
aspectRatio: CameraAspectRatio;
}
// Progress of an in-flight Capture/Record so the RenderOverlay can cover the
// viewport (freeing GPU) and show progress while frames encode.
export interface RecordingProgress {
phase: "capturing" | "encoding" | "uploading";
pct: number; // 0..1
}
// Scene metadata — what host tracks in its `signalScene`. Mirrored
// into the store so ControlsTopButtons (lib-resident) can read it
@@ -48,6 +77,9 @@ export interface OutlinerItem {
type: string;
visible: boolean;
locked: boolean;
// True for the render-camera placeholder ("::CAM::") — the outliner shows a
// view-from-camera button for these rows.
isCamera?: boolean;
}
// The currently inspected object in the right-hand control panel.
@@ -76,6 +108,7 @@ export interface SelectedSceneObject {
id: string;
}
export type SceneMode = "build" | "record";
export type TransformMode = "move" | "rotate" | "scale";
export type TransformSpace = "world" | "local";
export type PoseMode = "select" | "pose";
@@ -100,11 +133,13 @@ export interface EditorLoader {
const DEFAULT_CAMERAS: Camera[] = [
{
id: "main",
// Blender-style default: viewport pulled back + elevated at a 3/4 angle so
// the whole render-camera frustum (cam2, below) is visible in the scene.
label: "Main View",
focalLength: 17,
position: { x: -2.5, y: 2.5, z: 2.5 },
position: { x: -4.5, y: 4, z: 6 },
rotation: { x: 0, y: 0, z: 0 },
lookAt: { x: 0, y: 0, z: 0 },
lookAt: { x: 0, y: 0.5, z: 0.6 },
},
{
id: "cam2",
@@ -131,6 +166,7 @@ interface PageSceneState {
cameraFilter: AssetFilterOption;
// editor mode
sceneMode: SceneMode;
editorState: EditorStates;
transformMode: TransformMode;
transformSpace: TransformSpace;
@@ -142,6 +178,24 @@ interface PageSceneState {
hotkeyStatus: HotkeyStatus;
isPromptBoxFocused: boolean;
// timeline (mirrors TimelineController; see engine/editor/TimelineController.ts)
timelineExists: boolean;
timelineExpanded: boolean;
timelinePlayhead: number;
timelineIsPlaying: boolean;
timelineDuration: number;
timelineTracks: TimelineTrack[];
timelineClipLanes: ClipLane[];
timelineSelectedKeyframeId: string | null;
// Left keyframe of the segment whose easing curve is being edited in the
// Motion popover (opened from the curve chip BETWEEN two keyframes).
// Distinct from timelineSelectedKeyframeId, which drives selection/delete.
timelineEasingKeyframeId: string | null;
// record output
producedArtifact: ProducedArtifact | null;
recordingProgress: RecordingProgress | null;
// layout / panels
assetModalVisible: boolean;
assetModalVisibleDuringDrag: boolean;
@@ -227,6 +281,7 @@ interface PageSceneState {
setCameraFilter: (filter: AssetFilterOption) => void;
// editor mode
setSceneMode: (mode: SceneMode) => void;
setEditorState: (state: EditorStates) => void;
setTransformMode: (mode: TransformMode) => void;
setTransformSpace: (space: TransformSpace) => void;
@@ -234,6 +289,18 @@ interface PageSceneState {
setPoseMode: (mode: PoseMode) => void;
setShowPoseControls: (visible: boolean) => void;
setGridVisible: (visible: boolean) => void;
setTimelineExists: (exists: boolean) => void;
setTimelineExpanded: (expanded: boolean) => void;
setTimelinePlayhead: (time: number) => void;
setTimelineIsPlaying: (playing: boolean) => void;
setTimelineDuration: (duration: number) => void;
setTimelineTracks: (tracks: TimelineTrack[]) => void;
setTimelineClipLanes: (clipLanes: ClipLane[]) => void;
setTimelineSelectedKeyframe: (id: string | null) => void;
setTimelineEasingKeyframe: (id: string | null) => void;
setProducedArtifact: (artifact: ProducedArtifact | null) => void;
clearProducedArtifact: () => void;
setRecordingProgress: (progress: RecordingProgress | null) => void;
toggleStats: () => void;
setIgnoreKeyDelete: (ignore: boolean) => void;
disableHotkeyInput: (level: DomLevels) => void;
@@ -306,17 +373,29 @@ export const usePageSceneStore = create<PageSceneState>((set, get) => ({
cameras: DEFAULT_CAMERAS,
selectedCameraId: "main",
cameraAspectRatio: CameraAspectRatio.HORIZONTAL_3_2,
cameraAspectRatio: DEFAULT_CAMERA_ASPECT_RATIO,
focalLengthDragging: { isDragging: false, focalLength: 35 },
cameraFilter: AssetFilterOption.ALL,
editorState: EditorStates.EDIT,
sceneMode: "build",
transformMode: "move",
transformSpace: "world",
selectedMode: "move",
poseMode: "select",
showPoseControls: false,
gridVisible: true,
timelineExists: false,
timelineExpanded: false,
timelinePlayhead: 0,
timelineIsPlaying: false,
timelineDuration: 10,
timelineTracks: [],
timelineClipLanes: [],
timelineSelectedKeyframeId: null,
timelineEasingKeyframeId: null,
producedArtifact: null,
recordingProgress: null,
ignoreKeyDelete: false,
hotkeyStatus: { disabled: false, disabledBy: DomLevels.NONE },
isPromptBoxFocused: false,
@@ -342,7 +421,7 @@ export const usePageSceneStore = create<PageSceneState>((set, get) => ({
outlinerItems: [],
outlinerSelectedItem: null,
outlinerShowing: false,
outlinerShowing: true,
precisionSelectorShowing: false,
precisionSelectorCoords: { x: 0, y: 0 },
@@ -405,6 +484,7 @@ export const usePageSceneStore = create<PageSceneState>((set, get) => ({
setCameraFilter: (filter) => set({ cameraFilter: filter }),
// editor mode actions
setSceneMode: (mode) => set({ sceneMode: mode }),
setEditorState: (state) => set({ editorState: state }),
setTransformMode: (mode) => set({ transformMode: mode }),
setTransformSpace: (space) => set({ transformSpace: space }),
@@ -412,6 +492,22 @@ export const usePageSceneStore = create<PageSceneState>((set, get) => ({
setPoseMode: (mode) => set({ poseMode: mode }),
setShowPoseControls: (visible) => set({ showPoseControls: visible }),
setGridVisible: (visible) => set({ gridVisible: visible }),
setTimelineExists: (exists) => set({ timelineExists: exists }),
setTimelineExpanded: (expanded) => set({ timelineExpanded: expanded }),
setTimelinePlayhead: (time) => set({ timelinePlayhead: time }),
setTimelineIsPlaying: (playing) => set({ timelineIsPlaying: playing }),
setTimelineDuration: (duration) => set({ timelineDuration: duration }),
setTimelineTracks: (tracks) => set({ timelineTracks: tracks }),
setTimelineClipLanes: (clipLanes) => set({ timelineClipLanes: clipLanes }),
setTimelineSelectedKeyframe: (id) => set({ timelineSelectedKeyframeId: id }),
setTimelineEasingKeyframe: (id) => set({ timelineEasingKeyframeId: id }),
setProducedArtifact: (artifact) => set({ producedArtifact: artifact }),
clearProducedArtifact: () =>
set((s) => {
if (s.producedArtifact) URL.revokeObjectURL(s.producedArtifact.objectUrl);
return { producedArtifact: null };
}),
setRecordingProgress: (progress) => set({ recordingProgress: progress }),
toggleStats: () => set((s) => ({ statsVisible: !s.statsVisible })),
setIgnoreKeyDelete: (ignore) => set({ ignoreKeyDelete: ignore }),
disableHotkeyInput: (level) => {
@@ -38,14 +38,15 @@ import {
useCostBreakdownModalStore,
} from "@storyteller/ui-pricing-modal";
import { LoadingDots } from "@storyteller/ui-loading";
import { PromptBox3D, commonToCameraAspect } from "@storyteller/ui-promptbox";
import type { PopoverItem } from "@storyteller/ui-popover";
import { v4 as uuidv4 } from "uuid";
import { EngineContext } from "./contexts/EngineContext/EngineContext";
import { AnonHintChip } from "./comps/AnonHintChip";
import { AspectRatioMenu } from "./comps/AspectRatioMenu";
import { ControlPanelSceneObject } from "./comps/ControlPanelSceneObject";
import { Controls3D } from "./comps/Controls3D";
import { SceneModePill } from "./comps/SceneModePill";
import { ControlsTopButtons } from "./comps/ControlsTopButtons";
import { EditorCanvas } from "./comps/EngineCanvases";
import { FocalLengthDisplay } from "./comps/FocalLengthDisplay/FocalLengthDisplay";
@@ -53,11 +54,18 @@ import { OnboardingHelper } from "./comps/OnboardingHelper";
import { PerfStatsOverlay } from "./comps/PerfStatsOverlay";
import { EntranceDebugPanel } from "./comps/EntranceDebugPanel";
import { Outliner } from "./comps/Outliner";
import { AnimationsDrawer } from "./comps/AnimationsDrawer";
import { PoseModeSelector } from "./comps/PoseModeSelector";
import { ExitCameraView } from "./comps/ExitCameraView";
import { RecordControls } from "./comps/RecordControls";
import { CameraStatusPill } from "./comps/CameraStatusPill";
import { RenderOverlay } from "./comps/RenderOverlay";
import { CompletionModal } from "./comps/CompletionModal";
import { PreviewBox } from "./comps/PreviewBox";
import { PreviewEngineCamera } from "./comps/PreviewEngineCamera";
import { SceneContainer } from "./comps/SceneContainer";
import { addCharacter, addObject, setCameraAspect } from "./actions";
import { addCharacter, addObject } from "./actions";
import { TimelineBar, TimelineEditor } from "./comps/Timeline";
import { useEditorCanvas } from "./hooks/useEditorCanvas";
import { useFreeCam } from "./hooks/useFreeCam";
import { useViewportPointer } from "./hooks/useViewportPointer";
@@ -126,6 +134,13 @@ export const Stage3DBody = ({
(s) => s.setIsPromptBoxFocused,
);
const gridVisible = usePageSceneStore((s) => s.gridVisible);
const timelineExists = usePageSceneStore((s) => s.timelineExists);
const timelineExpanded = usePageSceneStore((s) => s.timelineExpanded);
const setTimelineExpanded = usePageSceneStore((s) => s.setTimelineExpanded);
const timelineTracks = usePageSceneStore((s) => s.timelineTracks);
const is3DSceneLoaded = usePageSceneStore((s) => s.is3DSceneLoaded);
const sceneMode = usePageSceneStore((s) => s.sceneMode);
const isRecord = sceneMode === "record";
const previewImageUrl = usePageSceneStore((s) => s.sceneMeta.previewImageUrl);
const isVisitingOthersScene = useIsVisitingOthersScene();
const addCamera = usePageSceneStore((s) => s.addCamera);
@@ -219,6 +234,30 @@ export const Stage3DBody = ({
const editor = useContext(EngineContext);
// Record mode is read-only playback through the render camera: enter
// camera view + lock the viewport on entry, restore on exit.
useEffect(() => {
if (!editor) return;
if (isRecord) {
editor.cameraController.enterCameraView();
editor.cameraController.setLocked(true);
} else {
editor.cameraController.setLocked(false);
editor.cameraController.exitCameraView();
}
}, [isRecord, editor]);
// Auto-expand the timeline once when a loaded scene has keyframes or an
// imported Three.js animation; otherwise it stays collapsed by default.
const didAutoExpandTimelineRef = useRef(false);
useEffect(() => {
if (!editor || !is3DSceneLoaded || didAutoExpandTimelineRef.current) return;
didAutoExpandTimelineRef.current = true;
if (timelineTracks.length > 0 || editor.sceneHasAnimation()) {
setTimelineExpanded(true);
}
}, [editor, is3DSceneLoaded, timelineTracks, setTimelineExpanded]);
// Reactive viewport sizing. useViewportSize listens to window
// resize and re-renders the component. Falls back to
// window.innerWidth/innerHeight when the host adapter doesn't
@@ -343,28 +382,6 @@ export const Stage3DBody = ({
}
};
const onAspectRatioSelect = (newRatio: CameraAspectRatio) => {
if (!editor) return;
setCameraAspect(editor, newRatio);
};
// Cold-load sync: align the editor letterbox with the picker's
// initial display once when the engine + a `supportsNewAspectRatio()`
// model are both ready. Per-model-switch sync is intentionally NOT
// done because every model defaults to Square, which would override
// the user's pick.
const didColdSyncRef = useRef(false);
useEffect(() => {
if (didColdSyncRef.current) return;
if (!editor || !selectedImageModel?.supportsNewAspectRatio()) return;
const def = selectedImageModel.defaultAspectRatio;
if (!def) return;
const mapped = commonToCameraAspect(def);
if (!mapped) return;
setCameraAspect(editor, mapped);
didColdSyncRef.current = true;
}, [editor, selectedImageModel]);
// Gallery → 3D scene drop handler. Stage3D mounts only when 3D is
// active so this is implicitly 3D-only.
useEffect(() => {
@@ -443,6 +460,8 @@ export const Stage3DBody = ({
{import.meta.env.DEV && <EntranceDebugPanel />}
<FocalLengthDisplay />
<PoseModeSelector />
<RenderOverlay />
<CompletionModal />
<div
className="absolute left-0 top-0 w-full"
@@ -450,87 +469,87 @@ export const Stage3DBody = ({
>
<div className="grid grid-cols-3 gap-4">
<div className="flex flex-col items-start gap-2">
<ControlsTopButtons />
{topBarStartSlot && (
{!isRecord && <ControlsTopButtons />}
{!isRecord && topBarStartSlot && (
<div className="pl-3">{topBarStartSlot}</div>
)}
</div>
<Controls3D showImageTo3DButton={showImageTo3DButton} />
<div className="flex flex-col items-center">
<SceneModePill />
{!isRecord && (
<Controls3D showImageTo3DButton={showImageTo3DButton} />
)}
</div>
<div className="flex items-start justify-end gap-2 pr-3 pt-3">
{topBarEndSlot}
<AnonHintChip />
{/* Camera cluster: which camera drives the viewport +
the render-frame aspect ratio. */}
{!isRecord && <CameraStatusPill />}
{!isRecord && <AspectRatioMenu />}
{!isRecord && topBarEndSlot}
{!isRecord && <AnonHintChip />}
</div>
</div>
</div>
{/* Outliner: vertically centered against the viewport. Sized in
vh — deliberately independent of the camera aspect ratio. */}
<div
className={`absolute left-4 top-1/2 -translate-y-1/2${
isRecord ? " hidden" : ""
}`}
>
<Outliner />
</div>
{/* Animations drawer: right-docked, only when a character is
selected. Gated to build mode (hide via CSS, never unmount). */}
<div
className={`absolute right-4 top-1/2 -translate-y-1/2${
isRecord ? " hidden" : ""
}`}
>
<AnimationsDrawer />
</div>
<div
className="absolute bottom-0 left-0 right-0"
onClick={handleOverlayClick}
>
{/* Always mounted: PreviewEngineCamera owns the cam-view canvas
that EngineProvider depends on — unmounting it in record mode
tears down + recreates the whole engine (wiping the scene).
Hide with CSS in record instead. */}
<div
className="absolute bottom-20 mb-4 ml-4 flex origin-bottom-left flex-col gap-2"
className={`absolute bottom-20 mb-4 ml-4 flex origin-bottom-left flex-col gap-2${
isRecord ? " hidden" : ""
}`}
style={{ transform: `scale(${getScale()})` }}
>
<Outliner />
<PreviewEngineCamera />
</div>
<ControlPanelSceneObject />
{!isRecord && <ControlPanelSceneObject />}
</div>
{isVisitingOthersScene && <PreviewBox imageUrl={previewImageUrl} />}
<PromptBox3D
cameras={cameras}
cameraAspectRatio={camAspect}
disableHotkeyInput={disableHotkeyInput}
enableHotkeyInput={enableHotkeyInput}
gridVisibility={gridVisible}
setGridVisibility={(visible: boolean) =>
editor?.bus.emit(new GridVisibleChangedEvent(visible))
}
selectedCameraId={selectedCameraId}
deleteCamera={deleteCamera}
focalLengthDragging={focalLengthDragging}
setFocalLengthDragging={setFocalLengthDragging}
isPromptBoxFocused={isPromptBoxFocused}
setIsPromptBoxFocused={setIsPromptBoxFocused}
uploadImage={
editor
? (((arg: Parameters<typeof editor.adapter.uploadImage>[0]) =>
editor.adapter.uploadImage(arg)) as never)
: undefined
}
handleCameraSelect={handleCameraSelect}
handleAddCamera={handleAddCamera}
handleCameraNameChange={handleCameraNameChange}
handleCameraFocalLengthChange={handleCameraFocalLengthChange}
onAspectRatioSelect={onAspectRatioSelect}
selectedImageModel={selectedImageModel}
selectedProvider={selectedProvider}
credits={imageCredits}
setEnginePrompt={(prompt) => {
if (!editor) return;
editor.positive_prompt = prompt;
}}
snapshotCurrentFrame={editor?.snapShotOfCurrentFrame.bind(editor)}
modelSelector={inlineModelSelector}
aboveStackSlot={
<>
<OnboardingHelper />
{promptboxAboveStackSlot}
</>
}
onBeforeSubmit={() => {
const currentUserToken =
usePageSceneStore.getState().currentUserToken;
if (!currentUserToken && editor?.adapter.promptSignup) {
editor.adapter.promptSignup("generate");
return false;
}
return true;
}}
/>
{/* Build mode: the animation timeline is the sole bottom UI —
collapsed bar by default, expanded editor via the chevron.
The exit-camera-view button docks to the bar's right edge. */}
{!isRecord && !timelineExpanded && (
<div className="absolute bottom-4 left-1/2 w-[90vw] max-w-3xl -translate-x-1/2">
<TimelineBar />
<ExitCameraView className="absolute -right-2 top-1/2 -translate-y-1/2 translate-x-full" />
</div>
)}
{!isRecord && timelineExpanded && (
<>
<TimelineEditor />
<ExitCameraView className="absolute bottom-4 right-4" />
</>
)}
{isRecord && <RecordControls />}
<LoadingDots
className="absolute left-0 top-0 z-50"
@@ -0,0 +1,9 @@
import type Editor from "../engine/editor";
import { CameraViewToggleRequestedEvent } from "../engine/events/EngineEvent";
// Toggle "camera view" (look through / exit the render camera). Used by the
// outliner view-from-camera icon and the exit-camera-view button. Routes
// through the bus so the CameraController owns the actual transition.
export function toggleCameraView(editor: Editor): void {
editor.bus.emit(new CameraViewToggleRequestedEvent());
}
@@ -8,4 +8,7 @@ export * from "./setCameraAspect";
export * from "./setObjectColor";
export * from "./toggleObjectLock";
export * from "./toggleObjectVisibility";
export * from "./openAssetModal";
export * from "./transformObject";
export * from "./timeline";
export * from "./cameraView";
@@ -0,0 +1,12 @@
import { usePageSceneStore } from "../PageSceneStore";
// Open the add-asset (ArtCraft presets) modal. Shared by the Controls3D "+"
// popover and the Outliner's "+ Add" button. Clears any leftover drag-under
// state so the panel opens fully shown (a reopen-off drag leaves it
// faded-hidden until the next open).
export function openAssetModal(): void {
const store = usePageSceneStore.getState();
store.setAssetModalVisibleDuringDrag(true);
store.setAssetModalVisible(true);
store.setAssetDraggingUnder(false);
}
@@ -0,0 +1,125 @@
// UI-facing dispatchers for the animation timeline. Thin wrappers that
// forward to editor.timelineController (mirrors the other files in this
// folder). Pure-UI state (expanded, selected keyframe) is set directly on
// the store by components and has no dispatcher here.
import type Editor from "../engine/editor";
import type { EasingSpec } from "../engine/timeline/types";
// The clip source only needs an id + label — both a full MediaItem (drawer
// click) and a drag payload (drop) satisfy this.
interface ClipSource {
media_id: string;
name?: string;
}
// Placeholder strip length (seconds) until the clip GLB reports its real
// duration; playback uses the loaded clip's actual length regardless.
const DEFAULT_CLIP_DURATION = 2;
export function createTimeline(editor: Editor): void {
editor.timelineController.create();
}
export function playTimeline(editor: Editor): void {
editor.timelineController.play();
}
export function pauseTimeline(editor: Editor): void {
editor.timelineController.pause();
}
export function seekTimeline(editor: Editor, time: number): void {
editor.timelineController.seekTo(time);
}
export function setTimelineDuration(editor: Editor, seconds: number): void {
editor.timelineController.setDuration(seconds);
}
export function addKeyframe(
editor: Editor,
objectUuid: string,
atTime?: number,
): void {
editor.timelineController.addKeyframe(objectUuid, atTime);
}
export function deleteKeyframe(editor: Editor, keyframeId: string): void {
editor.timelineController.deleteKeyframe(keyframeId);
}
export function moveKeyframe(
editor: Editor,
keyframeId: string,
time: number,
): void {
editor.timelineController.moveKeyframe(keyframeId, time);
}
export function setKeyframeEasing(
editor: Editor,
keyframeId: string,
easing: EasingSpec,
): void {
editor.timelineController.setEasing(keyframeId, easing);
}
// Place a skeletal-animation clip (a Mixamo demo item) as a new stacked lane
// under `characterUuid`. Drops at the current playhead by default, or at an
// explicit `atTime` when dropped onto a specific point in the timeline. The
// real clip length replaces DEFAULT_CLIP_DURATION once the GLB loads (the
// strip starts at the placeholder width, flagged autoDuration).
export function addClipToCharacter(
editor: Editor,
characterUuid: string,
item: ClipSource,
atTime?: number,
): string | null {
if (!item.media_id) return null;
const startTime = atTime ?? editor.timelineController.getPlayhead();
return editor.timelineController.addClipLane(characterUuid, {
sourceMediaId: item.media_id,
name: item.name ?? "Animation",
startTime,
duration: DEFAULT_CLIP_DURATION,
loop: false,
autoDuration: true,
});
}
export function moveClipLane(
editor: Editor,
laneId: string,
startTime: number,
): void {
editor.timelineController.moveClipLane(laneId, startTime);
}
export function resizeClipLane(
editor: Editor,
laneId: string,
duration: number,
): void {
editor.timelineController.resizeClipLane(laneId, duration);
}
export function setClipLoop(
editor: Editor,
laneId: string,
loop: boolean,
): void {
editor.timelineController.setClipLoop(laneId, loop);
}
export function removeClipLane(editor: Editor, laneId: string): void {
editor.timelineController.removeClipLane(laneId);
}
export function saveTimeline(editor: Editor): void {
editor.timelineController.save();
}
export function cancelTimeline(editor: Editor): void {
editor.timelineController.cancel();
}
@@ -29,6 +29,9 @@ export function beginTransformSession(
commit() {
if (action.commit()) {
editor.history.record(action);
// Auto-key: an already-keyframed object edited via the panel updates
// its keyframe at the playhead (or creates one at the scrub point).
editor.timelineController.autoKeyIfTracked(uuid);
}
// Push the engine's post-edit transform back into the Zustand
// objectPanel slice so the panel's currentSceneObject vectors
@@ -75,6 +75,16 @@ export interface PageSceneSavePayload {
// ─── Adapter ───────────────────────────────────────────────────────────
// A still (Capture) or video (Record) produced by Record mode, cached
// locally (object URL) — handed to the 2D/video editor or uploaded on demand.
export interface PageSceneArtifact {
kind: "image" | "video";
blob: Blob;
objectUrl: string;
fileName: string;
mimeType: string;
}
export interface PageSceneAdapter {
// Generation enqueue. Same shape as PageDrawAdapter.enqueueEditImage.
enqueueGeneration(
@@ -215,6 +225,22 @@ export interface PageSceneAdapter {
// implements via useTabStore.setActiveTab; web host via router push.
navigateToImageTo3D(): void;
// Persist a produced still/video to the user's media library and return its
// media token. Images auto-upload after Capture; videos upload on demand
// (they're large). Host: MediaUploadApi via uploadByKind.
uploadMedia?(args: {
kind: "image" | "video";
blob: Blob;
fileName: string;
title?: string;
}): Promise<string>;
// Open the app's media Lightbox on an uploaded token — the shared modal that
// offers every destination for the media type (Edit-on-Canvas, Make-Video,
// Recreate, Share, Download). Host resolves the CDN URL from the token and
// renders <Lightbox>. Replaces bespoke per-destination handoffs.
openMediaLightbox?(token: string, kind: "image" | "video"): void;
// Auth/logout — the SettingsModal in Controls3D needs a logout
// callback. Tauri host: setLogoutStates. Web host: web auth flow.
performLogout(): void;
@@ -0,0 +1,100 @@
// Animations drawer: when a character is selected, lists the curated Mixamo
// clips. Click a clip to drop it on the character's timeline at the playhead,
// or drag it onto a character's timeline row (drop time follows the cursor).
// While dragging, the shared DragGhost (tilt card) previews the clip's
// thumbnail — the same pickup motion as object/character drags.
import { useContext } from "react";
import { EngineContext } from "../../contexts/EngineContext";
import { usePageSceneStore } from "../../PageSceneStore";
import { addClipToCharacter } from "../../actions";
import { demoAnimationItems } from "../../signals/demoAssets/demoAnimationItems";
import { ANIMATION_CLIP_MIME } from "../Timeline/timelineUtils";
// 1×1 transparent GIF used to suppress the browser's default drag image so only
// the DragGhost tilt card shows. Created once at module load so it's decoded by
// the time a drag starts.
const TRANSPARENT_DRAG_IMAGE =
typeof Image !== "undefined" ? new Image() : null;
if (TRANSPARENT_DRAG_IMAGE) {
TRANSPARENT_DRAG_IMAGE.src =
"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7";
}
export const AnimationsDrawer = () => {
const editor = useContext(EngineContext);
const selectedObject = usePageSceneStore((s) => s.selectedObject);
const characters = usePageSceneStore((s) => s.characters);
const character = characters.find(
(c) => c.kind === "character" && c.id === selectedObject?.id,
);
if (!character) return null;
return (
<div className="glass glass-no-hover pointer-events-auto flex w-52 flex-col gap-2 rounded-2xl p-3 shadow-xl">
<div className="px-1 text-xs font-medium text-base-fg/70">
Animations · {character.name}
</div>
<div className="grid max-h-[46vh] grid-cols-2 gap-2 overflow-y-auto pe-1">
{demoAnimationItems.map((item) => (
<button
key={item.media_id}
type="button"
draggable
title={`Add “${item.name}” to ${character.name} — click, or drag onto the timeline`}
className="group flex flex-col items-stretch gap-1 rounded-lg border border-white/10 bg-black/30 p-1 text-left transition-colors hover:border-white/40"
onDragStart={(e) => {
e.dataTransfer.setData(
ANIMATION_CLIP_MIME,
JSON.stringify({ media_id: item.media_id, name: item.name }),
);
e.dataTransfer.effectAllowed = "copy";
// Hide the native drag image; the DragGhost renders the preview.
if (TRANSPARENT_DRAG_IMAGE) {
e.dataTransfer.setDragImage(TRANSPARENT_DRAG_IMAGE, 0, 0);
}
const store = usePageSceneStore.getState();
store.setDragItem(item);
store.setDragPosition({ currX: e.pageX, currY: e.pageY });
store.setAssetDraggingUnder(true);
}}
onDrag={(e) => {
// The final native drag event can report (0,0) — ignore it so the
// ghost doesn't snap to the corner on release.
if (e.pageX === 0 && e.pageY === 0) return;
usePageSceneStore
.getState()
.setDragPosition({ currX: e.pageX, currY: e.pageY });
}}
onDragEnd={() => {
const store = usePageSceneStore.getState();
store.setDragItem(null);
store.setAssetDraggingUnder(false);
store.setDragPosition({ currX: 0, currY: 0 });
}}
onClick={() =>
editor && addClipToCharacter(editor, character.id, item)
}
>
<div className="aspect-square w-full overflow-hidden rounded-md bg-black/40">
{item.thumbnail && (
<img
src={item.thumbnail}
alt={item.name ?? "Animation"}
className="h-full w-full object-cover opacity-90 transition-opacity group-hover:opacity-100"
draggable={false}
/>
)}
</div>
<span className="truncate px-0.5 text-[11px] text-base-fg/90">
{item.name}
</span>
</button>
))}
</div>
</div>
);
};
export default AnimationsDrawer;
@@ -0,0 +1 @@
export { AnimationsDrawer } from "./AnimationsDrawer";
@@ -1,96 +1,60 @@
import {
faRectangle,
faRectangleVertical,
faSquare,
} from "@fortawesome/pro-solid-svg-icons";
import { useContext } from "react";
import { PopoverMenu, type PopoverItem } from "@storyteller/ui-popover";
import { Tooltip } from "@storyteller/ui-tooltip";
import { AspectRatioIcon } from "@storyteller/ui-promptbox";
import { CameraAspectRatio } from "../../enums";
import { setCameraAspect } from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { EngineContext } from "../../contexts/EngineContext";
import { ButtonDropdown } from "@storyteller/ui-button-dropdown";
const ICON_BY_RATIO: Record<CameraAspectRatio, typeof faRectangle> = {
[CameraAspectRatio.HORIZONTAL_16_9]: faRectangle,
[CameraAspectRatio.VERTICAL_9_16]: faRectangleVertical,
[CameraAspectRatio.HORIZONTAL_3_2]: faRectangle,
[CameraAspectRatio.VERTICAL_2_3]: faRectangleVertical,
[CameraAspectRatio.SQUARE_1_1]: faSquare,
};
// Render-frame aspect ratio picker. Same popover pattern as the prompt
// boxes' AspectRatioPicker, but driving the 3D editor's camera letterbox
// (CameraAspectRatio) instead of a generation model's ratio list.
const LABEL_BY_RATIO: Record<CameraAspectRatio, string> = {
[CameraAspectRatio.HORIZONTAL_16_9]: "16:9 Horizontal",
[CameraAspectRatio.VERTICAL_9_16]: "9:16 Vertical",
[CameraAspectRatio.HORIZONTAL_3_2]: "3:2 Horizontal",
[CameraAspectRatio.VERTICAL_2_3]: "2:3 Vertical",
[CameraAspectRatio.SQUARE_1_1]: "1:1 Squared",
};
interface RatioOption {
ratio: CameraAspectRatio;
label: string;
proportions: [number, number];
}
const RATIO_OPTIONS: RatioOption[] = [
{ ratio: CameraAspectRatio.HORIZONTAL_16_9, label: "16:9", proportions: [16, 9] },
{ ratio: CameraAspectRatio.HORIZONTAL_3_2, label: "3:2", proportions: [3, 2] },
{ ratio: CameraAspectRatio.SQUARE_1_1, label: "1:1", proportions: [1, 1] },
{ ratio: CameraAspectRatio.VERTICAL_2_3, label: "2:3", proportions: [2, 3] },
{ ratio: CameraAspectRatio.VERTICAL_9_16, label: "9:16", proportions: [9, 16] },
];
export const AspectRatioMenu = () => {
const aspect = usePageSceneStore((s) => s.cameraAspectRatio);
const editor = useContext(EngineContext);
const handleChangeAspectRatio = (newRatio: CameraAspectRatio) => {
if (!editor) return;
setCameraAspect(editor, newRatio);
const current =
RATIO_OPTIONS.find((o) => o.ratio === aspect) ?? RATIO_OPTIONS[0];
const items: PopoverItem[] = RATIO_OPTIONS.map((option) => ({
label: option.label,
selected: option.ratio === aspect,
icon: <AspectRatioIcon ratio={option.proportions} />,
}));
const handleSelect = (item: PopoverItem) => {
const option = RATIO_OPTIONS.find((o) => o.label === item.label);
if (option && editor) setCameraAspect(editor, option.ratio);
};
return (
<div className="absolute right-0 top-0 m-2 flex flex-col items-end">
<ButtonDropdown
label={LABEL_BY_RATIO[aspect]}
className="shadow-xl"
icon={ICON_BY_RATIO[aspect]}
align="right"
showSelected={true}
options={[
{
label: "16:9",
icon: faRectangle,
className: "pl-4",
description: "Horizontal",
selected: aspect === CameraAspectRatio.HORIZONTAL_16_9,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.HORIZONTAL_16_9),
},
{
label: "3:2",
icon: faRectangle,
className: "pl-4",
description: "Horizontal",
selected: aspect === CameraAspectRatio.HORIZONTAL_3_2,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.HORIZONTAL_3_2),
},
{
label: "2:3",
icon: faRectangleVertical,
className: "pl-4",
description: "Vertical",
selected: aspect === CameraAspectRatio.VERTICAL_2_3,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.VERTICAL_2_3),
},
{
label: "9:16",
icon: faRectangleVertical,
className: "pl-4",
description: "Vertical",
selected: aspect === CameraAspectRatio.VERTICAL_9_16,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.VERTICAL_9_16),
},
{
label: "1:1",
icon: faSquare,
className: "pl-4",
description: "Squared",
selected: aspect === CameraAspectRatio.SQUARE_1_1,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.SQUARE_1_1),
},
]}
<Tooltip content="Aspect ratio" position="bottom" delay={300} closeOnClick>
<PopoverMenu
items={items}
onSelect={handleSelect}
mode="toggle"
position="bottom"
panelTitle="Aspect Ratio"
showIconsInList
buttonClassName="glass glass-no-hover h-[34px] rounded-full px-3 text-xs font-medium shadow-xl"
triggerIcon={<AspectRatioIcon ratio={current.proportions} />}
/>
</div>
</Tooltip>
);
};
@@ -0,0 +1,50 @@
import { useContext } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faVideo, faUpDownLeftRight } from "@fortawesome/pro-solid-svg-icons";
import { Tooltip } from "@storyteller/ui-tooltip";
import { EngineContext } from "../../contexts/EngineContext";
import { toggleCameraView } from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { EditorStates } from "../../enums";
// Small status pill (top-left) showing which camera the viewport is driving:
// the free "Viewport" camera vs looking through a render camera. Clicking it
// toggles camera view. Hidden in record mode (that mode is self-evident).
export const CameraStatusPill = () => {
const editor = useContext(EngineContext);
const editorState = usePageSceneStore((s) => s.editorState);
const sceneMode = usePageSceneStore((s) => s.sceneMode);
const cameras = usePageSceneStore((s) => s.cameras);
if (sceneMode === "record") return null;
const inCameraView = editorState === EditorStates.CAMERA_VIEW;
const renderCameraLabel =
cameras.find((c) => c.id !== "main")?.label ?? "Camera";
return (
<Tooltip
content={inCameraView ? "Looking through render camera" : "Free viewport camera"}
position="bottom"
delay={300}
>
<button
type="button"
onClick={() => editor && toggleCameraView(editor)}
className={
inCameraView
? "flex h-[34px] items-center gap-2 rounded-full bg-brand-primary px-3 text-xs font-medium text-white shadow-xl transition-all duration-200"
: "glass glass-no-hover flex h-[34px] items-center gap-2 rounded-full px-3 text-xs font-medium text-base-fg/70 shadow-xl transition-all duration-200 hover:text-base-fg"
}
>
<FontAwesomeIcon
icon={inCameraView ? faVideo : faUpDownLeftRight}
className="h-3 w-3"
/>
{inCameraView ? renderCameraLabel : "Viewport"}
</button>
</Tooltip>
);
};
export default CameraStatusPill;
@@ -0,0 +1 @@
export { CameraStatusPill } from "./CameraStatusPill";
@@ -0,0 +1,148 @@
import { useContext, useEffect, useRef, useState } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faTrash,
faCloudArrowUp,
faSpinnerThird,
faTriangleExclamation,
faRotateRight,
} from "@fortawesome/pro-solid-svg-icons";
import { EngineContext } from "../../contexts/EngineContext";
import { usePageSceneStore } from "../../PageSceneStore";
// Output modal shown immediately after Capture/Record — it previews the local
// artifact (object URL, works offline), then uploads to the gallery and shows
// status. On success it opens the app Lightbox (all destinations) on the
// uploaded token; on failure it stays put with a Retry. Images auto-upload;
// videos wait for a manual Upload (they're large).
type UploadStatus = "idle" | "uploading" | "error";
export const CompletionModal = () => {
const editor = useContext(EngineContext);
const artifact = usePageSceneStore((s) => s.producedArtifact);
const clearProducedArtifact = usePageSceneStore(
(s) => s.clearProducedArtifact,
);
const [status, setStatus] = useState<UploadStatus>("idle");
const autoStartedRef = useRef(false);
const isVideo = artifact?.kind === "video";
const upload = async () => {
if (!editor?.adapter.uploadMedia || !artifact) return;
setStatus("uploading");
try {
const token = await editor.adapter.uploadMedia({
kind: artifact.kind,
blob: artifact.blob,
fileName: artifact.fileName,
});
// Hand off to the app Lightbox (destinations for this kind). It shows via
// the uploaded CDN URL, so revoking the local object URL now is safe.
editor.adapter.openMediaLightbox?.(token, artifact.kind);
clearProducedArtifact();
} catch {
setStatus("error");
}
};
// Images auto-upload as soon as the modal appears; reset when it closes.
useEffect(() => {
if (artifact?.kind === "image" && !autoStartedRef.current) {
autoStartedRef.current = true;
void upload();
}
if (!artifact) {
autoStartedRef.current = false;
setStatus("idle");
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [artifact]);
if (!artifact) return null;
const busy = status === "uploading";
return (
<div className="absolute inset-0 z-[70] flex items-center justify-center bg-black/60 p-6 backdrop-blur-sm">
<div className="glass glass-no-hover flex w-full max-w-xl flex-col gap-4 rounded-2xl p-5 text-white shadow-2xl">
<div className="text-sm font-semibold text-base-fg/90">
{isVideo ? "Recording complete" : "Capture complete"}
</div>
<div className="flex max-h-[52vh] items-center justify-center overflow-hidden rounded-xl bg-black/30">
{isVideo ? (
<video
src={artifact.objectUrl}
controls
autoPlay
loop
className="max-h-[52vh] w-full object-contain"
/>
) : (
<img
src={artifact.objectUrl}
alt="Captured frame"
className="max-h-[52vh] w-full object-contain"
/>
)}
</div>
{/* upload status */}
{busy && (
<div className="flex items-center gap-2 text-sm text-base-fg/70">
<FontAwesomeIcon
icon={faSpinnerThird}
className="h-4 w-4 animate-spin"
/>
Uploading to gallery…
</div>
)}
{status === "error" && (
<div className="flex items-center gap-2 text-sm text-red">
<FontAwesomeIcon
icon={faTriangleExclamation}
className="h-4 w-4"
/>
Upload failed — check your connection and retry.
</div>
)}
{/* actions */}
<div className="flex items-center justify-between gap-2">
<button
type="button"
onClick={() => clearProducedArtifact()}
disabled={busy}
className="flex h-9 items-center gap-2 rounded-full px-3 text-sm text-base-fg/60 transition-colors hover:bg-white/10 hover:text-base-fg disabled:opacity-50"
>
<FontAwesomeIcon icon={faTrash} className="h-3.5 w-3.5" />
Discard
</button>
{status === "error" ? (
<button
type="button"
onClick={() => void upload()}
className="flex h-9 items-center gap-2 rounded-full bg-brand-primary px-4 text-sm font-medium text-white transition-transform hover:scale-[1.03]"
>
<FontAwesomeIcon icon={faRotateRight} className="h-3.5 w-3.5" />
Retry
</button>
) : isVideo && status === "idle" ? (
<button
type="button"
onClick={() => void upload()}
className="flex h-9 items-center gap-2 rounded-full bg-brand-primary px-4 text-sm font-medium text-white transition-transform hover:scale-[1.03]"
>
<FontAwesomeIcon icon={faCloudArrowUp} className="h-3.5 w-3.5" />
Upload &amp; continue
</button>
) : null}
</div>
</div>
</div>
);
};
export default CompletionModal;
@@ -0,0 +1 @@
export { CompletionModal } from "./CompletionModal";
@@ -25,7 +25,7 @@ import {
import { twMerge } from "tailwind-merge";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import { setTransformMode } from "../../actions";
import { openAssetModal, setTransformMode } from "../../actions";
import { usePageSceneStore, type TransformMode } from "../../PageSceneStore";
import { AssetModal } from "../AssetMenu";
@@ -42,9 +42,6 @@ export const Controls3D = ({
const editor = useContext(EngineContext);
const {
assetModalVisible,
setAssetModalVisible,
setAssetModalVisibleDuringDrag,
setAssetDraggingUnder,
selectedMode,
transformSpace,
currentUserToken,
@@ -52,9 +49,6 @@ export const Controls3D = ({
} = usePageSceneStore(
useShallow((s) => ({
assetModalVisible: s.assetModalVisible,
setAssetModalVisible: s.setAssetModalVisible,
setAssetModalVisibleDuringDrag: s.setAssetModalVisibleDuringDrag,
setAssetDraggingUnder: s.setAssetDraggingUnder,
selectedMode: s.selectedMode,
transformSpace: s.transformSpace,
currentUserToken: s.currentUserToken,
@@ -100,14 +94,6 @@ export const Controls3D = ({
}
};
const handleOpenModal = () => {
setAssetModalVisibleDuringDrag(true);
setAssetModalVisible(true);
// Clear any leftover drag-under state so the panel opens fully shown (a
// reopen-off drag leaves it faded-hidden until the next open).
setAssetDraggingUnder(false);
};
const handleOpenCreate3dModal = () => {
editor?.adapter.navigateToImageTo3D();
};
@@ -132,7 +118,7 @@ export const Controls3D = ({
}
switch (action) {
case "presets":
handleOpenModal();
openAssetModal();
break;
case "library":
handleOpenGalleryModal();
@@ -16,7 +16,7 @@ import {
usePageSceneStore,
useIsVisitingOthersScene,
} from "../../PageSceneStore";
import { CameraAspectRatio, ToastTypes } from "../../enums";
import { DEFAULT_CAMERA_ASPECT_RATIO, ToastTypes } from "../../enums";
import { getSceneGenerationMetaData } from "../../sceneMetadata";
import { LoadUserScenes } from "./LoadUserScenes";
import { Help } from "./Help/Help";
@@ -54,7 +54,7 @@ export const ControlsTopButtons = () => {
const handleResetScene = () => {
editor?.cameraController.changeRenderCameraAspectRatio(
CameraAspectRatio.HORIZONTAL_3_2,
DEFAULT_CAMERA_ASPECT_RATIO,
);
};
@@ -0,0 +1,39 @@
import { useContext } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faArrowRightFromBracket } from "@fortawesome/pro-solid-svg-icons";
import { twMerge } from "tailwind-merge";
import { EngineContext } from "../../contexts/EngineContext";
import { toggleCameraView } from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { EditorStates } from "../../enums";
// Shown only while looking through a render camera (CAMERA_VIEW). Exits back
// to the free editing view. Positionless by default — the host (Stage3DBody)
// docks it next to the timeline bar via `className`.
export const ExitCameraView = ({ className }: { className?: string }) => {
const editor = useContext(EngineContext);
const editorState = usePageSceneStore((s) => s.editorState);
const sceneMode = usePageSceneStore((s) => s.sceneMode);
// In record mode the viewport is also CAMERA_VIEW, but the mode pill
// handles leaving — don't show the manual exit button there.
if (editorState !== EditorStates.CAMERA_VIEW || sceneMode === "record") {
return null;
}
return (
<button
type="button"
onClick={() => editor && toggleCameraView(editor)}
className={twMerge(
"flex h-11 items-center gap-2 whitespace-nowrap rounded-xl bg-brand-primary px-4 text-sm font-medium text-white shadow-xl transition-colors hover:bg-brand-primary/90",
className,
)}
>
<FontAwesomeIcon icon={faArrowRightFromBracket} className="h-3.5 w-3.5" />
Exit
</button>
);
};
export default ExitCameraView;
@@ -0,0 +1 @@
export { ExitCameraView } from "./ExitCameraView";
@@ -1,41 +1,32 @@
import { ChangeEvent, useContext, useEffect, useState } from "react";
import { useContext } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faCircleXmark,
faArrowRightToBracket,
faEye,
faEyeSlash,
faListTree,
faLock,
faLockOpen,
faPlus,
faXmark,
} from "@fortawesome/pro-solid-svg-icons";
import { Input } from "@storyteller/ui-input";
import { Button } from "@storyteller/ui-button";
import { Transition } from "@headlessui/react";
import { twMerge } from "tailwind-merge";
import { useSignals } from "@preact/signals-react/runtime";
import { EngineContext } from "../../contexts/EngineContext";
import { OutlinerItem, usePageSceneStore } from "../../PageSceneStore";
import { useViewportSize } from "../../hooks/useViewportSize";
import { CameraAspectRatio } from "../../enums";
import { openAssetModal } from "../../actions/openAssetModal";
import { toggleObjectLock } from "../../actions/toggleObjectLock";
import { toggleObjectVisibility } from "../../actions/toggleObjectVisibility";
import { toggleCameraView } from "../../actions/cameraView";
const OutlinerRow = ({ item }: { item: OutlinerItem }) => {
const [hovered, setHovered] = useState(false);
const isSelected = usePageSceneStore(
(s) => s.outlinerSelectedItem?.id === item.id,
);
const editorEngine = useContext(EngineContext);
// Delete object logic here
const handleDeleteKeyPress = (event: React.KeyboardEvent<HTMLDivElement>) => {
if (event.key === "Delete") {
//console.log("Delete key pressed for item:", item.id);
}
};
// Double click logic here
const handleDoubleClick = () => {
editorEngine?.sceneManager?.double_click();
@@ -58,55 +49,71 @@ const OutlinerRow = ({ item }: { item: OutlinerItem }) => {
toggleObjectVisibility(editorEngine, item.id);
};
const handleViewFromCamera = (e: React.MouseEvent) => {
e.stopPropagation();
if (!editorEngine) return;
toggleCameraView(editorEngine);
};
return (
<div
role="button"
className={twMerge(
"flex cursor-pointer justify-between px-4 py-[7px] text-[13px] font-normal text-white/80 outline-none transition-all duration-100 hover:bg-action-900/35 focus:outline-none",
"flex cursor-pointer items-center justify-between gap-2 rounded-xl border border-transparent px-3 py-2 outline-none transition-colors duration-100 hover:bg-white/5 focus:outline-none",
isSelected &&
"bg-brand-primary/80 font-medium text-white hover:bg-brand-primary/80",
"border-brand-primary bg-brand-primary/15 hover:bg-brand-primary/15",
)}
onMouseEnter={() => setHovered(true)}
onMouseLeave={() => setHovered(false)}
onDoubleClick={handleDoubleClick}
onKeyDown={handleDeleteKeyPress}
onClick={handleSelect}
tabIndex={0}
>
<span className="flex items-center gap-2.5">
<div className="flex w-4 items-center justify-center">
<span className="flex min-w-0 items-center gap-3">
<div className="flex w-5 shrink-0 items-center justify-center text-base text-white/85">
<FontAwesomeIcon icon={item.icon} />
</div>
{item.name}
<span className="min-w-0">
<span className="block truncate text-[13px] font-semibold text-white">
{item.name}
</span>
<span className="block truncate text-xs text-white/50">
{item.type}
</span>
</span>
</span>
<div className="flex gap-3">
<div className="flex shrink-0 items-center gap-3">
{item.isCamera && (
<button
onClick={handleViewFromCamera}
title="View from camera"
className="text-white/60 transition-colors duration-100 hover:text-white text-xs"
>
<div className="w-3">
<FontAwesomeIcon icon={faArrowRightToBracket} />
</div>
</button>
)}
<button
onClick={handleToggleLock}
style={{
opacity: hovered || item.locked ? 1 : 0,
}}
title={item.locked ? "Unlock" : "Lock"}
className={twMerge(
"text-white/60 transition-colors duration-100 hover:text-white text-xs",
item.locked && "text-white/90",
)}
>
<div className="w-3">
<FontAwesomeIcon
icon={item.locked ? faLock : faLockOpen}
className="opacity-80 transition-opacity duration-100 hover:opacity-100"
/>
<FontAwesomeIcon icon={item.locked ? faLock : faLockOpen} />
</div>
</button>
<button
onClick={handleToggleVisibility}
style={{
opacity: hovered || !item.visible ? 1 : 0,
}}
title={item.visible ? "Hide" : "Show"}
className={twMerge(
"text-white/60 transition-colors duration-100 hover:text-white text-xs",
!item.visible && "text-white/90",
)}
>
<div className="w-4">
<FontAwesomeIcon
icon={item.visible ? faEye : faEyeSlash}
className={twMerge(
"opacity-80 transition-opacity duration-100 hover:opacity-100",
item.locked && "text-white/90",
)}
/>
<div className="w-3">
<FontAwesomeIcon icon={item.visible ? faEye : faEyeSlash} />
</div>
</button>
</div>
@@ -116,58 +123,16 @@ const OutlinerRow = ({ item }: { item: OutlinerItem }) => {
export const Outliner = () => {
useSignals();
const camAspect = usePageSceneStore((s) => s.cameraAspectRatio);
const items = usePageSceneStore((s) => s.outlinerItems);
const showing = usePageSceneStore((s) => s.outlinerShowing);
const [searchTerm, setSearchTerm] = useState("");
const [editorHeight, setEditorHeight] = useState(0);
const handleSearchChange = (e: ChangeEvent<HTMLInputElement>) => {
setSearchTerm(e.target.value);
};
const clearSearch = () => {
setSearchTerm("");
};
// Filter items based on search term
const filteredItems = items.filter((item) =>
item.name.toLowerCase().includes(searchTerm.toLowerCase()),
);
const viewport = useViewportSize();
useEffect(() => {
setEditorHeight(viewport.height - 56);
}, [viewport.height]);
const getOutlinerHeightClass = () => {
if (viewport.width >= 2000) {
if (camAspect === CameraAspectRatio.VERTICAL_9_16) {
return `${editorHeight * 0.5 - 120}px`;
} else if (camAspect === CameraAspectRatio.SQUARE_1_1) {
return `${editorHeight * 0.42}px`;
} else {
return `${editorHeight * 0.54}px`;
}
}
if (camAspect === CameraAspectRatio.VERTICAL_9_16) {
return `${editorHeight * 0.7 - 10}px`;
} else if (camAspect === CameraAspectRatio.SQUARE_1_1) {
return `${editorHeight * 0.7}px`;
} else {
return `${editorHeight * 0.7}px`;
}
};
return (
<Transition
as="div"
show={showing}
className={twMerge(
"glass flex max-h-[34vh] w-[240px] origin-bottom-left flex-col overflow-hidden rounded-lg shadow-lg",
"flex h-[45vh] w-[260px] flex-col overflow-hidden rounded-xl border border-ui-panel-border bg-ui-panel shadow-lg",
)}
style={{ height: getOutlinerHeightClass() }}
enter="transition-opacity duration-150"
enterFrom="opacity-0"
enterTo="opacity-100"
@@ -175,11 +140,15 @@ export const Outliner = () => {
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div className="flex items-center px-4 pt-3">
<h1 className="grow text-base font-semibold">
<FontAwesomeIcon icon={faListTree} className="mb-0 mr-2" />
Outliner
</h1>
<div className="flex items-center px-4 pb-1 pt-3">
<h1 className="grow text-base font-semibold">Scene</h1>
<Button
icon={faPlus}
className="h-6 bg-transparent px-1.5 text-sm font-medium text-white/80 hover:bg-transparent hover:text-white"
onClick={openAssetModal}
>
Add
</Button>
<Button
icon={faXmark}
className="h-5 bg-transparent p-0 text-xl opacity-50 hover:bg-transparent hover:opacity-90"
@@ -189,26 +158,12 @@ export const Outliner = () => {
/>
</div>
<div className="relative mx-4 my-2.5">
<Input
inputClassName="h-8 rounded-lg text-sm pr-8"
placeholder="Search..."
value={searchTerm}
onInput={handleSearchChange}
/>
{searchTerm && (
<FontAwesomeIcon
icon={faCircleXmark}
className="absolute right-2 top-1/2 -translate-y-1/2 transform cursor-pointer opacity-50 transition-all duration-100 hover:opacity-100"
onClick={clearSearch}
/>
)}
</div>
<div className="grow overflow-auto">
{filteredItems.map((item) => (
<OutlinerRow key={item.id} item={item} />
))}
<div className="grow overflow-auto p-2">
<div className="flex flex-col gap-1">
{items.map((item) => (
<OutlinerRow key={item.id} item={item} />
))}
</div>
</div>
</Transition>
);
@@ -0,0 +1,115 @@
import { useContext } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faCamera, faCircle } from "@fortawesome/pro-solid-svg-icons";
import { Tooltip } from "@storyteller/ui-tooltip";
import { EngineContext } from "../../contexts/EngineContext";
import { usePageSceneStore } from "../../PageSceneStore";
import { recordTimeline } from "../../engine/recording/TimelineRecorder";
import { TimelineBar } from "../Timeline";
// Record-mode controls: a read-only playback bar (when a timeline exists) plus
// Capture (still) and Record (timeline→video).
// Capture → auto-upload to gallery → open the app Lightbox on the token.
// Record → produce the clip → hand to the video review modal (manual upload,
// since videos are large) via producedArtifact.
export const RecordControls = () => {
const editor = useContext(EngineContext);
const timelineExists = usePageSceneStore((s) => s.timelineExists);
const aspectRatio = usePageSceneStore((s) => s.cameraAspectRatio);
const recordingProgress = usePageSceneStore((s) => s.recordingProgress);
const setProducedArtifact = usePageSceneStore((s) => s.setProducedArtifact);
const setRecordingProgress = usePageSceneStore((s) => s.setRecordingProgress);
const busy = recordingProgress !== null;
// Snapshot the composed frame → hand to the review modal, which previews it
// immediately and auto-uploads to the gallery (with status) → app Lightbox.
const handleCapture = () => {
if (!editor || busy) return;
setRecordingProgress({ phase: "capturing", pct: 0 });
// Defer so the overlay paints before the (sync) snapshot work.
requestAnimationFrame(() => {
try {
const snap = editor.snapShotOfCurrentFrame(false);
if (snap) {
setProducedArtifact({
kind: "image",
blob: snap.file,
objectUrl: URL.createObjectURL(snap.file),
fileName: snap.file.name,
mimeType: "image/png",
aspectRatio,
});
}
} finally {
setRecordingProgress(null);
}
});
};
// Encode the timeline; the produced clip opens the review modal for a manual
// upload (videos are large, so we don't auto-upload).
const handleRecord = async () => {
if (!editor || busy || !timelineExists) return;
setRecordingProgress({ phase: "encoding", pct: 0 });
try {
const result = await recordTimeline(editor, {
onProgress: (pct) => setRecordingProgress({ phase: "encoding", pct }),
});
if (result) {
setProducedArtifact({
kind: "video",
blob: result.blob,
objectUrl: URL.createObjectURL(result.blob),
fileName: result.fileName,
mimeType: result.mimeType,
aspectRatio,
});
}
} finally {
setRecordingProgress(null);
}
};
return (
<div className="absolute bottom-6 left-1/2 z-30 flex -translate-x-1/2 flex-col items-center gap-3">
{timelineExists && (
<div className="w-[70vw] max-w-3xl">
<TimelineBar readOnly />
</div>
)}
<div className="flex items-center gap-3">
<button
type="button"
onClick={handleCapture}
disabled={busy}
className="glass glass-no-hover flex items-center gap-2 rounded-full px-5 py-2.5 text-sm font-medium text-base-fg shadow-xl hover:bg-white/10 disabled:opacity-50"
>
<FontAwesomeIcon icon={faCamera} className="h-3.5 w-3.5" />
Capture
</button>
<Tooltip
content={
timelineExists
? "Render the timeline to video"
: "Add an animation timeline first"
}
position="top"
delay={200}
>
<button
type="button"
onClick={handleRecord}
disabled={busy || !timelineExists}
className="flex items-center gap-2 rounded-full bg-red px-5 py-2.5 text-sm font-semibold text-white shadow-xl transition-transform hover:scale-[1.03] disabled:opacity-50 disabled:hover:scale-100"
>
<FontAwesomeIcon icon={faCircle} className="h-3 w-3" />
Record
</button>
</Tooltip>
</div>
</div>
);
};
export default RecordControls;
@@ -0,0 +1 @@
export { RecordControls } from "./RecordControls";
@@ -0,0 +1,27 @@
import { LoadingDots } from "@storyteller/ui-loading";
import { usePageSceneStore } from "../../PageSceneStore";
// Opaque full-viewport cover shown while a Capture/Record is in flight. It
// hides the 3D scene (freeing GPU during frame encoding) and shows progress.
export const RenderOverlay = () => {
const progress = usePageSceneStore((s) => s.recordingProgress);
const pct = Math.round((progress?.pct ?? 0) * 100);
const message =
progress?.phase === "encoding"
? `Rendering video… ${pct}%`
: progress?.phase === "uploading"
? "Uploading to gallery…"
: "Capturing frame…";
return (
<LoadingDots
className="absolute left-0 top-0 z-[60]"
isShowing={progress !== null}
type="bricks"
message={message}
/>
);
};
export default RenderOverlay;
@@ -0,0 +1 @@
export { RenderOverlay } from "./RenderOverlay";
@@ -0,0 +1,180 @@
import { useContext, useEffect, useMemo, useState } from "react";
import { usePageSceneStore } from "../../PageSceneStore";
import { EngineContext } from "../../contexts/EngineContext";
import { CameraAspectRatio } from "../../enums";
// Camera-view framing overlay: an inset, aspect-correct frame showing the
// EXACT render framing (the engine offsets the viewport projection into this
// rect via CameraController.setCameraFrameRect), with the surrounding scene
// dimmed, rule-of-thirds guides, and the focal-length label. The frame is
// fitted into the area left free by the surrounding chrome so the bottom
// timeline/record controls never cover the shot.
// Reserved chrome, in px. Vertical insets differ per mode:
// build/camera-view has the mode pill + transform toolbar up top and the
// collapsed timeline bar + exit-view button at the bottom; record mode has
// only the pill up top but the playback bar + capture/record cluster below.
const TOP_INSET_BUILD = 124;
const TOP_INSET_RECORD = 76;
const BOTTOM_INSET_BUILD = 88;
// Fallback while the expanded editor hasn't been measured yet.
const BOTTOM_INSET_BUILD_TIMELINE_EXPANDED = 240;
const BOTTOM_INSET_RECORD = 152;
// The expanded timeline editor sits at bottom-4 (16px); keep a little air
// between it and the frame.
const TIMELINE_EDITOR_CLEARANCE = 28;
const SIDE_INSET = 24;
// Below this the insets don't fit; fall back to a slim uniform margin so
// tiny windows still get a usable frame.
const MIN_FRAME_EDGE = 120;
const FALLBACK_MARGIN = 16;
const RATIO_VALUE: Record<CameraAspectRatio, number> = {
[CameraAspectRatio.HORIZONTAL_16_9]: 16 / 9,
[CameraAspectRatio.HORIZONTAL_3_2]: 3 / 2,
[CameraAspectRatio.SQUARE_1_1]: 1,
[CameraAspectRatio.VERTICAL_2_3]: 2 / 3,
[CameraAspectRatio.VERTICAL_9_16]: 9 / 16,
};
type FrameRect = { x: number; y: number; width: number; height: number };
const fitFrame = (
width: number,
height: number,
ratio: number,
top: number,
bottom: number,
): FrameRect => {
let safeWidth = width - SIDE_INSET * 2;
let safeHeight = height - top - bottom;
let safeTop = top;
if (safeWidth < MIN_FRAME_EDGE || safeHeight < MIN_FRAME_EDGE) {
safeWidth = width - FALLBACK_MARGIN * 2;
safeHeight = height - FALLBACK_MARGIN * 2;
safeTop = FALLBACK_MARGIN;
}
const frameWidth = Math.min(safeWidth, safeHeight * ratio);
const frameHeight = frameWidth / ratio;
return {
x: (width - frameWidth) / 2,
y: safeTop + (safeHeight - frameHeight) / 2,
width: frameWidth,
height: frameHeight,
};
};
export const CameraFrame = ({
width,
height,
}: {
width: number;
height: number;
}) => {
const editor = useContext(EngineContext);
const camAspect = usePageSceneStore((s) => s.cameraAspectRatio);
const sceneMode = usePageSceneStore((s) => s.sceneMode);
const timelineExpanded = usePageSceneStore((s) => s.timelineExpanded);
const cameras = usePageSceneStore((s) => s.cameras);
const selectedCameraId = usePageSceneStore((s) => s.selectedCameraId);
const isRecord = sceneMode === "record";
// The expanded timeline editor's height depends on its track count, so
// measure it live instead of guessing a fixed inset.
const [editorPanelHeight, setEditorPanelHeight] = useState(0);
useEffect(() => {
if (isRecord || !timelineExpanded) return undefined;
const el = document.getElementById("timeline-editor");
if (!el) return undefined;
setEditorPanelHeight(el.offsetHeight);
const observer = new ResizeObserver(() =>
setEditorPanelHeight(el.offsetHeight),
);
observer.observe(el);
return () => observer.disconnect();
}, [isRecord, timelineExpanded]);
const frame = useMemo(() => {
if (width <= 0 || height <= 0) return null;
const top = isRecord ? TOP_INSET_RECORD : TOP_INSET_BUILD;
const bottom = isRecord
? BOTTOM_INSET_RECORD
: timelineExpanded
? editorPanelHeight > 0
? editorPanelHeight + TIMELINE_EDITOR_CLEARANCE
: BOTTOM_INSET_BUILD_TIMELINE_EXPANDED
: BOTTOM_INSET_BUILD;
return fitFrame(width, height, RATIO_VALUE[camAspect], top, bottom);
}, [width, height, camAspect, isRecord, timelineExpanded, editorPanelHeight]);
// Hand the rect to the engine so the viewport projection maps the exact
// render framing into it; clear on unmount (leaving camera view/record).
useEffect(() => {
if (!editor || !frame) return;
editor.cameraController.setCameraFrameRect({
x: frame.x,
y: frame.y,
width: frame.width,
height: frame.height,
canvasWidth: width,
canvasHeight: height,
});
}, [editor, frame, width, height]);
useEffect(() => {
return () => editor?.cameraController.setCameraFrameRect(null);
}, [editor]);
if (!frame) return null;
const focalLength = cameras.find((c) => c.id === selectedCameraId)
?.focalLength;
const right = frame.x + frame.width;
const bottomEdge = frame.y + frame.height;
return (
<div
id="letterbox"
className="pointer-events-none absolute inset-0 select-none"
>
{/* Dim everything outside the frame (still visible, won't render). */}
<div
className="absolute left-0 right-0 top-0 bg-black/60"
style={{ height: frame.y }}
/>
<div
className="absolute bottom-0 left-0 right-0 bg-black/60"
style={{ height: height - bottomEdge }}
/>
<div
className="absolute left-0 bg-black/60"
style={{ top: frame.y, height: frame.height, width: frame.x }}
/>
<div
className="absolute right-0 bg-black/60"
style={{ top: frame.y, height: frame.height, width: width - right }}
/>
{/* The frame itself: border, rule-of-thirds guides, focal label. */}
<div
className="absolute border border-white/40"
style={{
left: frame.x,
top: frame.y,
width: frame.width,
height: frame.height,
}}
>
<div className="absolute bottom-0 left-1/3 top-0 w-px bg-white/15" />
<div className="absolute bottom-0 left-2/3 top-0 w-px bg-white/15" />
<div className="absolute left-0 right-0 top-1/3 h-px bg-white/15" />
<div className="absolute left-0 right-0 top-2/3 h-px bg-white/15" />
{focalLength !== undefined && (
<div className="absolute left-3 top-2 text-sm font-medium text-white/80 drop-shadow">
{focalLength}mm
</div>
)}
</div>
</div>
);
};
@@ -1,63 +0,0 @@
import { twMerge } from "tailwind-merge";
import { usePageSceneStore } from "../../PageSceneStore";
import {
calcMatteWidth,
calcMatteHeight,
getMatteOrientation,
MatteOrientation,
} from "./helpers";
export const Letterbox = ({
isShowing,
width,
height,
}: {
isShowing: boolean;
width: number;
height: number;
}) => {
const camAspect = usePageSceneStore((s) => s.cameraAspectRatio);
const matteOri = getMatteOrientation({ camAspect, width, height });
const matteWidth = calcMatteWidth({ matteOri, camAspect, width, height });
const matteHeight = calcMatteHeight({ matteOri, camAspect, width, height });
return (
<div
id="letterbox"
className={twMerge(
"user-select-none pointer-events-none absolute left-0 top-0 flex h-full w-full justify-between",
matteOri === MatteOrientation.TOP_BOTTOM ? "flex-col" : null,
)}
>
<Matte matteOri={matteOri} width={matteWidth} height={matteHeight} />
<Matte matteOri={matteOri} width={matteWidth} height={matteHeight} />
</div>
);
};
const Matte = ({
matteOri,
width,
height,
}: {
matteOri: MatteOrientation;
width?: number;
height?: number;
}) => {
if (matteOri === MatteOrientation.TOP_BOTTOM) {
return (
<div
className="pointer-events-none h-20 w-full bg-black/30 brightness-75"
style={{ height: `${height}px` }}
/>
);
}
return (
<div
className="pointer-events-none h-full w-80 bg-black/30 brightness-75"
style={{ width: `${width}px` }}
/>
);
};
@@ -1,9 +1,17 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { usePageSceneStore } from "../../PageSceneStore";
import { Letterbox } from "./Letterbox";
import { EditorStates } from "../../enums";
import { CameraFrame } from "./CameraFrame";
export const SceneContainer = ({ children }: { children: React.ReactNode }) => {
const editorLetterBox = usePageSceneStore((s) => s.editorLetterBox);
const editorState = usePageSceneStore((s) => s.editorState);
const sceneMode = usePageSceneStore((s) => s.sceneMode);
// The framing overlay only makes sense while rendering through a camera:
// camera view and record mode. The free viewport shows no aspect mattes.
const showCameraFrame =
editorLetterBox &&
(editorState === EditorStates.CAMERA_VIEW || sceneMode === "record");
const [size, setSize] = useState({ width: 0, height: 0 });
const nodeRef = useRef<HTMLDivElement | null>(null);
@@ -36,11 +44,9 @@ export const SceneContainer = ({ children }: { children: React.ReactNode }) => {
className="relative h-full w-full"
>
{children}
<Letterbox
isShowing={editorLetterBox}
width={size.width}
height={size.height}
/>
{showCameraFrame && (
<CameraFrame width={size.width} height={size.height} />
)}
</div>
);
};
@@ -1,117 +0,0 @@
import { CameraAspectRatio } from "../../enums";
export enum MatteOrientation {
TOP_BOTTOM,
LEFT_RIGHT,
}
export const getMatteOrientation = ({
camAspect,
width,
height,
}: {
camAspect: CameraAspectRatio;
width: number;
height: number;
}): MatteOrientation => {
switch (camAspect) {
case CameraAspectRatio.SQUARE_1_1: {
return width > height
? MatteOrientation.LEFT_RIGHT
: MatteOrientation.TOP_BOTTOM;
}
case CameraAspectRatio.VERTICAL_9_16: {
return (height / 16) * 9 - width >= 0
? MatteOrientation.TOP_BOTTOM
: MatteOrientation.LEFT_RIGHT;
}
case CameraAspectRatio.VERTICAL_2_3: {
return (height / 3) * 2 - width >= 0
? MatteOrientation.TOP_BOTTOM
: MatteOrientation.LEFT_RIGHT;
}
case CameraAspectRatio.HORIZONTAL_3_2: {
return (width / 3) * 2 - height >= 0
? MatteOrientation.LEFT_RIGHT
: MatteOrientation.TOP_BOTTOM;
}
case CameraAspectRatio.HORIZONTAL_16_9:
default: {
return (width / 16) * 9 - height >= 0
? MatteOrientation.LEFT_RIGHT
: MatteOrientation.TOP_BOTTOM;
}
}
};
export const calcMatteWidth = ({
matteOri,
camAspect,
width,
height,
}: {
matteOri: MatteOrientation;
camAspect: CameraAspectRatio;
width: number;
height: number;
}): number | undefined => {
if (matteOri === MatteOrientation.TOP_BOTTOM) {
return undefined;
}
switch (camAspect) {
case CameraAspectRatio.SQUARE_1_1: {
return (width - height) / 2;
}
case CameraAspectRatio.HORIZONTAL_16_9: {
return (width - (height / 9) * 16) / 2;
}
case CameraAspectRatio.VERTICAL_9_16: {
return (width - (height / 16) * 9) / 2;
}
case CameraAspectRatio.HORIZONTAL_3_2: {
return (width - (height / 2) * 3) / 2;
}
case CameraAspectRatio.VERTICAL_2_3: {
return (width - (height / 3) * 2) / 2;
}
default: {
return undefined;
}
}
};
export const calcMatteHeight = ({
matteOri,
camAspect,
width,
height,
}: {
matteOri: MatteOrientation;
camAspect: CameraAspectRatio;
width: number;
height: number;
}): number | undefined => {
if (matteOri === MatteOrientation.LEFT_RIGHT) {
return undefined;
}
switch (camAspect) {
case CameraAspectRatio.SQUARE_1_1: {
return (height - width) / 2;
}
case CameraAspectRatio.HORIZONTAL_16_9: {
return (height - (width / 16) * 9) / 2;
}
case CameraAspectRatio.VERTICAL_9_16: {
return (height - (width / 9) * 16) / 2;
}
case CameraAspectRatio.HORIZONTAL_3_2: {
return (height - (width / 3) * 2) / 2;
}
case CameraAspectRatio.VERTICAL_2_3: {
return (height - (width / 2) * 3) / 2;
}
default: {
return undefined;
}
}
};
@@ -0,0 +1,34 @@
import { useShallow } from "zustand/shallow";
import { TabSelector, type TabItem } from "@storyteller/ui-tab-selector";
import { usePageSceneStore, type SceneMode } from "../../PageSceneStore";
const TABS: TabItem[] = [
{ id: "build", label: "Build" },
{ id: "record", label: "Record" },
];
export const SceneModePill = () => {
const { sceneMode, setSceneMode } = usePageSceneStore(
useShallow((s) => ({
sceneMode: s.sceneMode,
setSceneMode: s.setSceneMode,
})),
);
return (
<div className="flex justify-center pt-3">
<TabSelector
tabs={TABS}
activeTab={sceneMode}
onTabChange={(id) => setSceneMode(id as SceneMode)}
className="w-auto"
listClassName="rounded-full px-1"
tabClassName="rounded-full px-5 py-1"
indicatorClassName="rounded-full bg-white/95"
selectedTabClassName="text-black"
/>
</div>
);
};
export default SceneModePill;
@@ -0,0 +1 @@
export { SceneModePill } from "./SceneModePill";
@@ -0,0 +1,65 @@
import { useContext, useEffect, useRef, useState } from "react";
import { EngineContext } from "../../contexts/EngineContext";
import { setTimelineDuration } from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { formatTimecode } from "./timelineUtils";
// The total-duration readout, click-to-edit. Click → number input (seconds);
// Enter/blur commits, Escape cancels. Backed by TimelineController.setDuration
// (clamped to [1,60], covers the required 5–30s range).
export const DurationLabel = ({ className = "" }: { className?: string }) => {
const editor = useContext(EngineContext);
const duration = usePageSceneStore((s) => s.timelineDuration);
const [editing, setEditing] = useState(false);
const [value, setValue] = useState("");
const inputRef = useRef<HTMLInputElement>(null);
useEffect(() => {
if (editing) inputRef.current?.select();
}, [editing]);
const commit = () => {
const secs = parseFloat(value);
if (editor && Number.isFinite(secs)) setTimelineDuration(editor, secs);
setEditing(false);
};
if (editing) {
return (
<input
ref={inputRef}
type="number"
min={1}
max={60}
step={1}
value={value}
onChange={(e) => setValue(e.target.value)}
onBlur={commit}
onKeyDown={(e) => {
if (e.key === "Enter") commit();
if (e.key === "Escape") setEditing(false);
}}
onClick={(e) => e.stopPropagation()}
onMouseDown={(e) => e.stopPropagation()}
className={`w-12 rounded bg-white/10 px-1 text-center tabular-nums text-base-fg outline-none ${className}`}
/>
);
}
return (
<button
type="button"
title="Set timeline duration (seconds)"
onClick={(e) => {
e.stopPropagation();
setValue(String(Math.round(duration)));
setEditing(true);
}}
className={`tabular-nums transition-colors hover:text-base-fg ${className}`}
>
{formatTimecode(duration)}
</button>
);
};
export default DurationLabel;
@@ -0,0 +1,189 @@
import { useContext, useRef, useState } from "react";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import { setKeyframeEasing } from "../../actions";
import {
EASING_PRESETS,
type EasingPresetName,
type EasingSpec,
type Keyframe,
} from "../../engine/timeline/types";
// The Motion popover edits a keyframe's easing curve (the interpolation
// INTO the next keyframe). Presets across the top, an editable cubic-bezier
// curve below with two draggable handles and a dotted linear reference.
const BOX = 150; // svg drawing box (px), domain [0,1]×[0,1]
const PAD = 14; // padding so handles/labels aren't clipped
const SPAN = BOX - PAD * 2;
// Map normalized (x,y) in [0,1] → svg coords (y is inverted so 1 is at top).
const sx = (x: number) => PAD + x * SPAN;
const sy = (y: number) => PAD + (1 - y) * SPAN;
const PRESETS: { name: EasingPresetName; label: string }[] = [
{ name: "linear", label: "Linear" },
{ name: "easeIn", label: "In" },
{ name: "easeOut", label: "Out" },
{ name: "easeInOut", label: "In-Out" },
];
interface MotionPopoverProps {
keyframe: Keyframe;
// Horizontal anchor as a percentage across the editor (playhead/keyframe pos).
leftPercent: number;
}
export const MotionPopover = ({ keyframe, leftPercent }: MotionPopoverProps) => {
const editor = useContext(EngineContext);
const svgRef = useRef<SVGSVGElement>(null);
const [dragging, setDragging] = useState<null | "p1" | "p2">(null);
const easing = keyframe.easing;
const commit = (next: EasingSpec) => {
if (editor) setKeyframeEasing(editor, keyframe.id, next);
};
const pointerToNorm = (e: React.PointerEvent) => {
const svg = svgRef.current;
if (!svg) return null;
const rect = svg.getBoundingClientRect();
// rect may be scaled vs the BOX viewBox; normalize via SPAN in view units.
const vx = ((e.clientX - rect.left) / rect.width) * BOX;
const vy = ((e.clientY - rect.top) / rect.height) * BOX;
const x = Math.max(0, Math.min(1, (vx - PAD) / SPAN));
const y = Math.max(0, Math.min(1, 1 - (vy - PAD) / SPAN));
return { x, y };
};
const onPointerMove = (e: React.PointerEvent) => {
if (!dragging) return;
const n = pointerToNorm(e);
if (!n) return;
if (dragging === "p1") commit({ ...easing, p1x: n.x, p1y: n.y });
else commit({ ...easing, p2x: n.x, p2y: n.y });
};
const curvePath = `M ${sx(0)} ${sy(0)} C ${sx(easing.p1x)} ${sy(easing.p1y)} ${sx(
easing.p2x,
)} ${sy(easing.p2y)} ${sx(1)} ${sy(1)}`;
return (
<div
id="motion-popover"
className="glass glass-no-hover pointer-events-auto absolute bottom-full z-50 mb-2 w-[180px] -translate-x-1/2 select-none rounded-2xl p-3 shadow-xl backdrop-blur-md"
style={{ left: `${Math.max(12, Math.min(88, leftPercent))}%` }}
onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()}
>
<div className="mb-2 text-xs font-semibold text-base-fg/80">Motion</div>
<div className="mb-3 grid grid-cols-4 gap-1">
{PRESETS.map((p) => {
const preset = EASING_PRESETS[p.name];
const active =
preset.p1x === easing.p1x &&
preset.p1y === easing.p1y &&
preset.p2x === easing.p2x &&
preset.p2y === easing.p2y;
return (
<button
key={p.name}
type="button"
title={p.label}
onClick={() => commit({ ...preset })}
className={`flex h-8 items-center justify-center rounded-md border transition-colors ${
active
? "border-brand-primary bg-brand-primary/20"
: "border-ui-controls-border/60 hover:bg-ui-controls/40"
}`}
>
<svg width="20" height="20" viewBox="0 0 20 20" fill="none">
<path
d={`M 2 18 C ${2 + preset.p1x * 16} ${18 - preset.p1y * 16} ${
2 + preset.p2x * 16
} ${18 - preset.p2y * 16} 18 2`}
stroke="currentColor"
strokeWidth="1.5"
className="text-base-fg/80"
/>
</svg>
</button>
);
})}
</div>
<svg
ref={svgRef}
viewBox={`0 0 ${BOX} ${BOX}`}
className="w-full touch-none rounded-lg bg-black/20"
onPointerMove={onPointerMove}
onPointerUp={() => setDragging(null)}
onPointerLeave={() => setDragging(null)}
>
{/* dotted linear reference */}
<line
x1={sx(0)}
y1={sy(0)}
x2={sx(1)}
y2={sy(1)}
stroke="currentColor"
strokeWidth="1"
strokeDasharray="3 3"
className="text-base-fg/25"
/>
{/* handle stems */}
<line
x1={sx(0)}
y1={sy(0)}
x2={sx(easing.p1x)}
y2={sy(easing.p1y)}
stroke="currentColor"
strokeWidth="1"
className="text-brand-primary/50"
/>
<line
x1={sx(1)}
y1={sy(1)}
x2={sx(easing.p2x)}
y2={sy(easing.p2y)}
stroke="currentColor"
strokeWidth="1"
className="text-brand-primary/50"
/>
{/* the curve */}
<path
d={curvePath}
stroke="currentColor"
strokeWidth="2"
fill="none"
className="text-base-fg"
/>
{/* endpoints */}
<circle cx={sx(0)} cy={sy(0)} r="2.5" className="fill-base-fg/50" />
<circle cx={sx(1)} cy={sy(1)} r="2.5" className="fill-base-fg/50" />
{/* draggable handles */}
<circle
cx={sx(easing.p1x)}
cy={sy(easing.p1y)}
r="5"
className="cursor-grab fill-brand-primary"
onPointerDown={(e) => {
e.currentTarget.setPointerCapture(e.pointerId);
setDragging("p1");
}}
/>
<circle
cx={sx(easing.p2x)}
cy={sy(easing.p2y)}
r="5"
className="cursor-grab fill-brand-primary"
onPointerDown={(e) => {
e.currentTarget.setPointerCapture(e.pointerId);
setDragging("p2");
}}
/>
</svg>
</div>
);
};
export default MotionPopover;
@@ -0,0 +1,76 @@
import { useContext } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faPlay, faPause, faChevronDown } from "@fortawesome/pro-solid-svg-icons";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import { pauseTimeline, playTimeline, seekTimeline } from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { formatTimecode } from "./timelineUtils";
import { DurationLabel } from "./DurationLabel";
// Collapsed timeline bar: play/pause · time · scrubber · duration · expand.
// Sits above the prompt card (rendered via the promptbox aboveStackSlot).
// `readOnly` (record mode) hides the expand chevron — playback only.
export const TimelineBar = ({ readOnly = false }: { readOnly?: boolean }) => {
const editor = useContext(EngineContext);
const playhead = usePageSceneStore((s) => s.timelinePlayhead);
const isPlaying = usePageSceneStore((s) => s.timelineIsPlaying);
const duration = usePageSceneStore((s) => s.timelineDuration);
const setExpanded = usePageSceneStore((s) => s.setTimelineExpanded);
const togglePlay = () => {
if (!editor) return;
if (isPlaying) pauseTimeline(editor);
else playTimeline(editor);
};
return (
<div
className="glass glass-no-hover flex w-full select-none items-center gap-3 rounded-2xl px-4 py-2 text-white shadow-xl"
onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()}
>
<button
type="button"
onClick={togglePlay}
className="flex h-7 w-7 items-center justify-center rounded-full text-base-fg/80 hover:bg-white/10"
>
<FontAwesomeIcon icon={isPlaying ? faPause : faPlay} className="h-3.5 w-3.5" />
</button>
<span className="w-9 shrink-0 tabular-nums text-xs text-base-fg/70">
{formatTimecode(playhead)}
</span>
<input
type="range"
min={0}
max={duration}
step={0.01}
value={playhead}
onChange={(e) => {
if (!editor) return;
pauseTimeline(editor);
seekTimeline(editor, parseFloat(e.target.value));
}}
className="h-1 flex-1 cursor-pointer accent-white"
/>
{readOnly ? (
<span className="w-9 shrink-0 text-right tabular-nums text-xs text-base-fg/70">
{formatTimecode(duration)}
</span>
) : (
<DurationLabel className="shrink-0 text-right text-xs text-base-fg/70" />
)}
{!readOnly && (
<button
type="button"
title="Expand timeline"
onClick={() => setExpanded(true)}
className="flex h-7 w-7 items-center justify-center rounded-full text-base-fg/60 hover:bg-white/10"
>
<FontAwesomeIcon icon={faChevronDown} className="h-3 w-3" />
</button>
)}
</div>
);
};
export default TimelineBar;
@@ -0,0 +1,167 @@
import { useContext, useRef } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faFilm,
faRepeat,
faXmark,
} from "@fortawesome/pro-solid-svg-icons";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import {
moveClipLane,
removeClipLane,
resizeClipLane,
setClipLoop,
} from "../../actions";
import {
usePageSceneStore,
type ClipLane,
} from "../../PageSceneStore";
import { DEFAULT_TIMELINE_FPS } from "../../engine/timeline/types";
import { fractionToTime, quantizeToFrame, timeToFraction } from "./timelineUtils";
interface TimelineClipRowProps {
lanes: ClipLane[];
duration: number;
}
// Drag intent for the current pointer gesture.
type Drag =
| { laneId: string; kind: "move"; grabOffset: number } // seconds pointer→start
| { laneId: string; kind: "resize" };
// A character's skeletal-animation clips on ONE shared row (single row per
// character; the overlap guard in TimelineController keeps them a
// non-overlapping sequence). Each strip drags to move, its right edge trims
// length, the loop chip toggles repeat, the × removes it. Dropping a clip from
// the Animations drawer is handled by the character block wrapper in
// TimelineEditor, so an empty row still accepts drops.
export const TimelineClipRow = ({ lanes, duration }: TimelineClipRowProps) => {
const editor = useContext(EngineContext);
const laneRef = useRef<HTMLDivElement>(null);
const drag = useRef<Drag | null>(null);
const fps =
editor?.timelineController.getTimeline()?.fps ?? DEFAULT_TIMELINE_FPS;
const laneTimeFromEvent = (clientX: number): number => {
const el = laneRef.current;
if (!el) return 0;
const rect = el.getBoundingClientRect();
return quantizeToFrame(
fractionToTime((clientX - rect.left) / rect.width, duration),
fps,
);
};
const onPointerMove = (e: React.PointerEvent) => {
if (!drag.current || !editor) return;
const t = laneTimeFromEvent(e.clientX);
if (drag.current.kind === "move") {
moveClipLane(editor, drag.current.laneId, t - drag.current.grabOffset);
} else {
const lane = lanes.find((l) => l.id === drag.current?.laneId);
if (lane) resizeClipLane(editor, lane.id, t - lane.strip.startTime);
}
};
const endDrag = () => {
drag.current = null;
};
return (
<div className="flex items-center gap-3 py-0.5">
<div className="flex w-32 shrink-0 items-center gap-2 truncate ps-6 text-xs text-base-fg/50">
<FontAwesomeIcon icon={faFilm} className="h-3 w-3 opacity-60" />
<span className="truncate">Animation</span>
</div>
<div
ref={laneRef}
className="relative h-6 flex-1 rounded-md bg-black/20"
onPointerMove={onPointerMove}
onPointerUp={endDrag}
onPointerLeave={endDrag}
onPointerCancel={endDrag}
>
{lanes.length === 0 && (
<div className="pointer-events-none absolute inset-0 flex items-center justify-center rounded-md border border-dashed border-white/15 text-[10px] text-base-fg/30">
Drag an animation here
</div>
)}
{lanes.map((lane) => {
const strip = lane.strip;
const leftPct = timeToFraction(strip.startTime, duration) * 100;
const widthPct = Math.max(
(strip.duration / Math.max(duration, 1e-6)) * 100,
1.5, // stay visible while a fresh clip's real length is loading
);
return (
<div
key={lane.id}
className="absolute inset-y-0 flex items-center gap-1 rounded-md border border-brand-primary/60 bg-brand-primary/25 pe-1 ps-1.5 text-[10px] text-white"
style={{ left: `${leftPct}%`, width: `${widthPct}%` }}
onPointerDown={(e) => {
e.currentTarget.setPointerCapture(e.pointerId);
drag.current = {
laneId: lane.id,
kind: "move",
grabOffset: laneTimeFromEvent(e.clientX) - strip.startTime,
};
}}
>
<button
type="button"
title={
strip.loop
? "Looping — click to play once"
: "Play once — click to loop"
}
className={`flex h-3.5 w-3.5 shrink-0 items-center justify-center rounded-[3px] ${
strip.loop
? "bg-white/90 text-brand-primary"
: "text-white/70 hover:text-white"
}`}
onPointerDown={(e) => e.stopPropagation()}
onClick={() =>
editor && setClipLoop(editor, lane.id, !strip.loop)
}
>
<FontAwesomeIcon icon={faRepeat} className="h-2.5 w-2.5" />
</button>
<span className="min-w-0 flex-1 truncate">{strip.name}</span>
<button
type="button"
title="Remove animation"
className="flex h-3.5 w-3.5 shrink-0 items-center justify-center rounded-[3px] text-white/70 hover:bg-white/20 hover:text-white"
onPointerDown={(e) => e.stopPropagation()}
onClick={() => editor && removeClipLane(editor, lane.id)}
>
<FontAwesomeIcon icon={faXmark} className="h-2.5 w-2.5" />
</button>
{/* right-edge trim handle */}
<div
className="absolute inset-y-0 right-0 w-1.5 cursor-ew-resize rounded-r-md bg-white/40 hover:bg-white/70"
title="Trim clip length"
onPointerDown={(e) => {
e.stopPropagation();
e.currentTarget.setPointerCapture(e.pointerId);
drag.current = { laneId: lane.id, kind: "resize" };
}}
/>
</div>
);
})}
</div>
{/* spacer to match the keyframe row's add-button column (keeps lane
widths aligned with the ruler) */}
<div className="w-7 shrink-0" />
</div>
);
};
export default TimelineClipRow;
@@ -0,0 +1,416 @@
import { useContext, useEffect, useRef } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faPlay,
faPause,
faBackwardStep,
faForwardStep,
faChevronUp,
faTrash,
} from "@fortawesome/pro-solid-svg-icons";
import { Button } from "@storyteller/ui-button";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import {
addClipToCharacter,
cancelTimeline,
deleteKeyframe,
pauseTimeline,
playTimeline,
saveTimeline,
seekTimeline,
} from "../../actions";
import { usePageSceneStore } from "../../PageSceneStore";
import { DEFAULT_TIMELINE_FPS } from "../../engine/timeline/types";
import {
ANIMATION_CLIP_MIME,
formatTimecode,
formatTimecodeFrames,
fractionToTime,
quantizeToFrame,
timeToFraction,
} from "./timelineUtils";
import { TimelineTrackRow } from "./TimelineTrackRow";
import { TimelineClipRow } from "./TimelineClipRow";
import { MotionPopover } from "./MotionPopover";
import { DurationLabel } from "./DurationLabel";
// Lane geometry: label column (w-32=8rem) + gap-3(0.75rem) … gap-3 + add-btn(w-7=1.75rem).
const LANE_LEFT = "8.75rem";
const LANE_RIGHT = "2.5rem";
// Expanded multi-track keyframe editor. Replaces the prompt box while open.
export const TimelineEditor = () => {
const editor = useContext(EngineContext);
const rulerRef = useRef<HTMLDivElement>(null);
const isScrubbing = useRef(false);
const tracks = usePageSceneStore((s) => s.timelineTracks);
const clipLanes = usePageSceneStore((s) => s.timelineClipLanes);
const characters = usePageSceneStore((s) => s.characters);
const duration = usePageSceneStore((s) => s.timelineDuration);
const playhead = usePageSceneStore((s) => s.timelinePlayhead);
const isPlaying = usePageSceneStore((s) => s.timelineIsPlaying);
const outlinerItems = usePageSceneStore((s) => s.outlinerItems);
const selectedKeyframeId = usePageSceneStore(
(s) => s.timelineSelectedKeyframeId,
);
const setExpanded = usePageSceneStore((s) => s.setTimelineExpanded);
// Every scene object gets a track row (empty lanes included), so the
// editor never looks blank while the scene has content — no selection
// required to see where keyframes can go.
const trackByUuid = new Map(tracks.map((t) => [t.objectUuid, t]));
const rows = outlinerItems;
// Only characters get skeletal-animation clip lanes. Clip drags are accepted
// on a character's block; other rows ignore them.
const characterIds = new Set(characters.map((c) => c.id));
const clipLanesByChar = new Map<string, typeof clipLanes>();
for (const lane of clipLanes) {
const list = clipLanesByChar.get(lane.objectUuid) ?? [];
list.push(lane);
clipLanesByChar.set(lane.objectUuid, list);
}
const selectedKeyframe = tracks
.flatMap((t) => t.keyframes)
.find((k) => k.id === selectedKeyframeId);
// The Motion popover edits the easing of the segment BETWEEN two
// keyframes (stored on the left one) and anchors at the segment midpoint.
const easingKeyframeId = usePageSceneStore((s) => s.timelineEasingKeyframeId);
let easingAnchor: {
keyframe: (typeof tracks)[number]["keyframes"][number];
leftPercent: number;
} | null = null;
if (easingKeyframeId) {
for (const track of tracks) {
const sorted = [...track.keyframes].sort((a, b) => a.time - b.time);
const index = sorted.findIndex((k) => k.id === easingKeyframeId);
if (index === -1) continue;
if (index < sorted.length - 1) {
const mid = (sorted[index].time + sorted[index + 1].time) / 2;
easingAnchor = {
keyframe: sorted[index],
leftPercent: timeToFraction(mid, duration) * 100,
};
}
break;
}
}
// Selected keyframe: Delete/Backspace removes it (capture phase, so the
// engine's document-level Delete — which deletes the selected scene
// object — never sees the key), and clicking anywhere that isn't a
// keyframe or the Delete button deselects it.
useEffect(() => {
if (!selectedKeyframeId) return undefined;
const onKeyDown = (e: KeyboardEvent) => {
if (e.key !== "Delete" && e.key !== "Backspace") return;
const target = e.target as HTMLElement | null;
if (
target instanceof HTMLInputElement ||
target instanceof HTMLTextAreaElement ||
target?.isContentEditable
) {
return;
}
e.preventDefault();
e.stopPropagation();
if (editor) deleteKeyframe(editor, selectedKeyframeId);
usePageSceneStore.getState().setTimelineSelectedKeyframe(null);
};
const onPointerDown = (e: PointerEvent) => {
const target = e.target as HTMLElement | null;
if (
target?.closest("[data-keyframe]") ||
target?.closest("[data-keyframe-delete]")
) {
return;
}
usePageSceneStore.getState().setTimelineSelectedKeyframe(null);
};
document.addEventListener("keydown", onKeyDown, true);
document.addEventListener("pointerdown", onPointerDown, true);
return () => {
document.removeEventListener("keydown", onKeyDown, true);
document.removeEventListener("pointerdown", onPointerDown, true);
};
}, [selectedKeyframeId, editor]);
// Clicking anywhere outside the popover (canvas included) dismisses it;
// the easing chips opt out so they can toggle it themselves.
useEffect(() => {
if (!easingKeyframeId) return undefined;
const onPointerDown = (e: PointerEvent) => {
const target = e.target as HTMLElement | null;
if (
target?.closest("#motion-popover") ||
target?.closest("[data-easing-chip]")
) {
return;
}
usePageSceneStore.getState().setTimelineEasingKeyframe(null);
};
document.addEventListener("pointerdown", onPointerDown, true);
return () =>
document.removeEventListener("pointerdown", onPointerDown, true);
}, [easingKeyframeId]);
const playheadFraction = timeToFraction(playhead, duration);
const togglePlay = () => {
if (!editor) return;
if (isPlaying) pauseTimeline(editor);
else playTimeline(editor);
};
// The timeline runs on a frame grid (what Record encodes at).
const fps = editor?.timelineController.getTimeline()?.fps ?? DEFAULT_TIMELINE_FPS;
// Scrubbing follows the pointer 1:1 (no magnetism toward keyframes),
// quantized to the frame grid so times are never arbitrary floats.
const seekFromRuler = (clientX: number) => {
const ruler = rulerRef.current;
if (!ruler || !editor) return;
const rect = ruler.getBoundingClientRect();
pauseTimeline(editor);
seekTimeline(
editor,
quantizeToFrame(
fractionToTime((clientX - rect.left) / rect.width, duration),
fps,
),
);
};
// Drop an animation clip (dragged from the Animations drawer) onto a
// character at the pointer's time. Lane geometry matches the ruler, so the
// ruler rect converts pointer-x → time exactly as scrubbing does.
const dropClipOnCharacter = (characterId: string, e: React.DragEvent) => {
const ruler = rulerRef.current;
if (!ruler || !editor) return;
const raw = e.dataTransfer.getData(ANIMATION_CLIP_MIME);
if (!raw) return;
e.preventDefault();
let payload: { media_id?: string; name?: string };
try {
payload = JSON.parse(raw);
} catch {
return;
}
if (!payload.media_id) return;
const rect = ruler.getBoundingClientRect();
const time = quantizeToFrame(
fractionToTime((e.clientX - rect.left) / rect.width, duration),
fps,
);
addClipToCharacter(
editor,
characterId,
{ media_id: payload.media_id, name: payload.name },
time,
);
};
return (
<div
id="timeline-editor"
className="glass glass-no-hover absolute bottom-4 left-1/2 w-[90vw] max-w-5xl -translate-x-1/2 select-none rounded-2xl p-3 text-white shadow-xl"
onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()}
>
{/* transport + collapse */}
<div className="mb-2 flex items-center gap-2">
<button
type="button"
title="Go to start"
onClick={() => editor && seekTimeline(editor, 0)}
className="flex h-7 w-7 items-center justify-center rounded-full text-base-fg/70 hover:bg-white/10"
>
<FontAwesomeIcon icon={faBackwardStep} className="h-3.5 w-3.5" />
</button>
<button
type="button"
onClick={togglePlay}
className="flex h-7 w-7 items-center justify-center rounded-full text-base-fg/90 hover:bg-white/10"
>
<FontAwesomeIcon
icon={isPlaying ? faPause : faPlay}
className="h-3.5 w-3.5"
/>
</button>
<button
type="button"
title="Go to end"
onClick={() => editor && seekTimeline(editor, duration)}
className="flex h-7 w-7 items-center justify-center rounded-full text-base-fg/70 hover:bg-white/10"
>
<FontAwesomeIcon icon={faForwardStep} className="h-3.5 w-3.5" />
</button>
<span className="ml-2 tabular-nums text-xs text-base-fg/60">
{formatTimecodeFrames(playhead, fps)} / {formatTimecode(duration)}
</span>
<button
type="button"
title="Collapse timeline"
onClick={() => setExpanded(false)}
className="ml-auto flex h-7 w-7 items-center justify-center rounded-full text-base-fg/60 hover:bg-white/10"
>
<FontAwesomeIcon icon={faChevronUp} className="h-3 w-3" />
</button>
</div>
{/* ruler + tracks (playhead line spans this region) */}
<div className="relative">
<div className="mb-1 flex items-center gap-3">
<div className="w-32 shrink-0" />
<div
ref={rulerRef}
className="relative h-5 flex-1 cursor-pointer touch-none text-[10px] text-base-fg/50"
onPointerDown={(e) => {
isScrubbing.current = true;
e.currentTarget.setPointerCapture(e.pointerId);
seekFromRuler(e.clientX);
}}
onPointerMove={(e) => {
if (isScrubbing.current) seekFromRuler(e.clientX);
}}
onPointerUp={() => {
isScrubbing.current = false;
}}
onPointerCancel={() => {
isScrubbing.current = false;
}}
>
<span className="absolute left-0">{formatTimecode(0)}</span>
<span className="absolute left-1/2 -translate-x-1/2">
{formatTimecode(duration / 2)}
</span>
<DurationLabel className="absolute right-0" />
</div>
<div className="w-7 shrink-0" />
</div>
{rows.length === 0 ? (
<div className="py-6 text-center text-xs text-base-fg/40">
Add objects to the scene to start animating.
</div>
) : (
/* Cap at ~3.5 rows (row = 36px lane + 8px padding) so the panel
stays compact; the half row hints that the list scrolls. The
scrollbar is hidden so the lanes keep the exact same width as
the ruler — a visible scrollbar would shift keyframes out of
alignment with the ruler-driven playhead. */
<div className="max-h-[154px] overflow-y-auto overscroll-contain [&::-webkit-scrollbar]:hidden [-ms-overflow-style:none] [scrollbar-width:none]">
{rows.map((item) => {
const isCharacter = characterIds.has(item.id);
const lanes = isCharacter ? clipLanesByChar.get(item.id) : undefined;
return (
<div
key={item.id}
onDragOver={
isCharacter
? (e) => {
// Allow the drop only when it carries an animation clip.
if (e.dataTransfer.types.includes(ANIMATION_CLIP_MIME)) {
e.preventDefault();
}
}
: undefined
}
onDrop={
isCharacter
? (e) => dropClipOnCharacter(item.id, e)
: undefined
}
>
<TimelineTrackRow
item={item}
track={trackByUuid.get(item.id)}
duration={duration}
/>
{isCharacter && (
<TimelineClipRow lanes={lanes ?? []} duration={duration} />
)}
</div>
);
})}
</div>
)}
{/* playhead (diamond handle in the ruler + line over the lanes) */}
<div
className="pointer-events-none absolute inset-y-0"
style={{ left: LANE_LEFT, right: LANE_RIGHT }}
>
<div
className="absolute inset-y-0 w-px bg-white"
style={{ left: `${playheadFraction * 100}%` }}
/>
<div
className="absolute top-0 h-2.5 w-2.5 -translate-x-1/2 rotate-45 rounded-[2px] bg-white"
style={{ left: `${playheadFraction * 100}%` }}
/>
</div>
{/* Anchor the popover in the lane area (same geometry as the
playhead overlay) so its left % lines up with the easing chip. */}
{easingAnchor && (
<div
className="pointer-events-none absolute inset-y-0"
style={{ left: LANE_LEFT, right: LANE_RIGHT }}
>
<MotionPopover
keyframe={easingAnchor.keyframe}
leftPercent={easingAnchor.leftPercent}
/>
</div>
)}
</div>
{/* footer */}
<div className="mt-3 flex items-center justify-between gap-2">
<span className="ms-2 text-[11px] text-base-fg/40">
Each diamond stores an object's full position, rotation and scale at
that moment. Tap a diamond to jump to it.
</span>
<div className="flex items-center gap-2">
{selectedKeyframe && (
/* data-keyframe-delete: exempt from the click-away deselect —
pointerdown would otherwise clear the selection before this
button's click fires. */
<div data-keyframe-delete>
<Button
variant="secondary"
icon={faTrash}
className="flex h-9 items-center border border-ui-controls-border bg-ui-controls/60 px-3 text-sm text-base-fg hover:bg-ui-controls/90"
onClick={() =>
editor && deleteKeyframe(editor, selectedKeyframe.id)
}
>
Delete
</Button>
</div>
)}
<Button
variant="secondary"
className="flex h-9 items-center border border-ui-controls-border bg-ui-controls/60 px-3 text-sm text-base-fg hover:bg-ui-controls/90"
onClick={() => editor && cancelTimeline(editor)}
>
Cancel
</Button>
<Button
variant="primary"
className="flex h-9 items-center border-none bg-brand-primary px-3 text-sm text-white"
onClick={() => editor && saveTimeline(editor)}
>
Save
</Button>
</div>
</div>
</div>
);
};
export default TimelineEditor;
@@ -0,0 +1,179 @@
import { useContext, useRef } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { EngineContext } from "../../contexts/EngineContext/EngineContext";
import { addKeyframe, moveKeyframe, seekTimeline } from "../../actions";
import {
usePageSceneStore,
type OutlinerItem,
type TimelineTrack,
} from "../../PageSceneStore";
import { DEFAULT_TIMELINE_FPS } from "../../engine/timeline/types";
import {
fractionToTime,
quantizeToFrame,
timeToFraction,
} from "./timelineUtils";
interface TimelineTrackRowProps {
item: OutlinerItem;
track: TimelineTrack | undefined;
duration: number;
}
export const TimelineTrackRow = ({
item,
track,
duration,
}: TimelineTrackRowProps) => {
const editor = useContext(EngineContext);
const laneRef = useRef<HTMLDivElement>(null);
const draggingId = useRef<string | null>(null);
const selectedKeyframeId = usePageSceneStore(
(s) => s.timelineSelectedKeyframeId,
);
const setSelectedKeyframe = usePageSceneStore(
(s) => s.setTimelineSelectedKeyframe,
);
const easingKeyframeId = usePageSceneStore((s) => s.timelineEasingKeyframeId);
const setEasingKeyframe = usePageSceneStore(
(s) => s.setTimelineEasingKeyframe,
);
const keyframes = track?.keyframes ?? [];
const sorted = [...keyframes].sort((a, b) => a.time - b.time);
const laneTimeFromEvent = (clientX: number): number => {
const lane = laneRef.current;
if (!lane) return 0;
const rect = lane.getBoundingClientRect();
// Keyframes live on the timeline's frame grid, same as scrubbing.
const fps = editor?.timelineController.getTimeline()?.fps ?? DEFAULT_TIMELINE_FPS;
return quantizeToFrame(
fractionToTime((clientX - rect.left) / rect.width, duration),
fps,
);
};
const selectKeyframe = (id: string, time: number) => {
setSelectedKeyframe(id);
if (editor) seekTimeline(editor, time);
};
return (
<div className="flex items-center gap-3 py-1">
{/* Keep this column exactly w-32: the ruler row and playhead overlay
assume it. Indent with padding, never margin (margin widens the
column and shifts the lanes out of alignment). */}
<div className="flex w-32 shrink-0 items-center gap-2 truncate ps-2 text-sm text-base-fg/90">
<FontAwesomeIcon icon={item.icon} className="h-3.5 w-3.5 opacity-70" />
<span className="truncate">{item.name}</span>
</div>
<div
ref={laneRef}
className="relative h-9 flex-1 rounded-lg bg-black/30"
onPointerMove={(e) => {
if (!draggingId.current || !editor) return;
moveKeyframe(
editor,
draggingId.current,
laneTimeFromEvent(e.clientX),
);
}}
onPointerUp={() => {
draggingId.current = null;
}}
onPointerLeave={() => {
draggingId.current = null;
}}
>
{/* connecting line between the first and last keyframe */}
{sorted.length >= 2 && (
<div
className="pointer-events-none absolute top-1/2 h-px -translate-y-1/2 bg-white/30"
style={{
left: `${timeToFraction(sorted[0].time, duration) * 100}%`,
right: `${(1 - timeToFraction(sorted[sorted.length - 1].time, duration)) * 100}%`,
}}
/>
)}
{/* easing chip between each pair of keyframes — toggles the Motion
popover for this segment (the curve INTO the next keyframe,
stored on the left keyframe) */}
{sorted.slice(0, -1).map((kf, i) => {
const next = sorted[i + 1];
const mid =
(timeToFraction(kf.time, duration) +
timeToFraction(next.time, duration)) /
2;
const active = easingKeyframeId === kf.id;
return (
<button
key={`ease-${kf.id}`}
type="button"
title="Edit easing"
data-easing-chip
className={`absolute top-1/2 flex h-5 w-6 -translate-x-1/2 -translate-y-1/2 items-center justify-center rounded-md border bg-black/60 ${
active
? "border-brand-primary text-white"
: "border-white/40 text-white/80 hover:border-white/80 hover:text-white"
}`}
style={{ left: `${mid * 100}%` }}
onClick={() => setEasingKeyframe(active ? null : kf.id)}
>
<svg width="12" height="10" viewBox="0 0 12 10" fill="none">
<path
d="M1 9 C 5 9, 7 1, 11 1"
stroke="currentColor"
strokeWidth="1.25"
/>
</svg>
</button>
);
})}
{sorted.map((kf) => {
const selected = kf.id === selectedKeyframeId;
return (
<div
key={kf.id}
role="button"
tabIndex={0}
data-keyframe
title={`${kf.time.toFixed(2)}s`}
className={`absolute top-1/2 h-3 w-3 -translate-x-1/2 -translate-y-1/2 rotate-45 cursor-grab rounded-[2px] border transition-colors ${
selected
? "border-brand-primary bg-brand-primary"
: "border-white/70 bg-white"
}`}
style={{ left: `${timeToFraction(kf.time, duration) * 100}%` }}
onPointerDown={(e) => {
e.currentTarget.setPointerCapture(e.pointerId);
draggingId.current = kf.id;
// Selecting a keyframe never opens the Motion popover —
// that's the easing chip's job.
setEasingKeyframe(null);
selectKeyframe(kf.id, kf.time);
}}
/>
);
})}
</div>
<button
type="button"
title="Add keyframe at playhead"
className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full text-base-fg/70 hover:bg-white/10 hover:text-base-fg"
onClick={() => {
if (editor) addKeyframe(editor, item.id);
}}
>
<span className="block h-2 w-2 rotate-45 rounded-[1px] border-[1.5px] border-current" />
</button>
</div>
);
};
export default TimelineTrackRow;
@@ -0,0 +1,2 @@
export { TimelineBar } from "./TimelineBar";
export { TimelineEditor } from "./TimelineEditor";
@@ -0,0 +1,37 @@
// Shared helpers for the timeline UI.
// DataTransfer MIME for dragging an animation clip from the Animations drawer
// onto a character's lane in the timeline. Payload is JSON: { media_id, name }.
export const ANIMATION_CLIP_MIME = "application/x-artcraft-animation-clip";
// Seconds → "m:ss" (e.g. 10 → "0:10", 65 → "1:05").
export const formatTimecode = (seconds: number): string => {
const clamped = Math.max(0, seconds);
const mins = Math.floor(clamped / 60);
const secs = Math.floor(clamped % 60);
return `${mins}:${secs.toString().padStart(2, "0")}`;
};
// Seconds → "m:ss:ff" frame timecode (e.g. 3.6667s @ 30fps → "0:03:20").
// Used for the transport readout so the scrub position reads in frames.
export const formatTimecodeFrames = (seconds: number, fps: number): string => {
const totalFrames = Math.round(Math.max(0, seconds) * fps);
const totalSeconds = Math.floor(totalFrames / fps);
const frames = totalFrames % fps;
return `${formatTimecode(totalSeconds)}:${frames
.toString()
.padStart(2, "0")}`;
};
// Quantize a scrubbed/dragged time to the timeline's frame grid so the
// playhead and keyframes land on frame boundaries, not arbitrary floats.
export const quantizeToFrame = (seconds: number, fps: number): number =>
Math.round(seconds * fps) / fps;
// Position (0..1) of a time within [0, duration].
export const timeToFraction = (time: number, duration: number): number =>
duration <= 0 ? 0 : Math.max(0, Math.min(1, time / duration));
// Convert a pointer x within a track lane element to a time in [0, duration].
export const fractionToTime = (fraction: number, duration: number): number =>
Math.max(0, Math.min(1, fraction)) * duration;
@@ -34,6 +34,8 @@ import {
SelectedCameraChangedEvent,
SelectedModeChangedEvent,
SelectionChangedEvent,
TimelineChangedEvent,
TimelinePlayheadEvent,
TransformModeChangedEvent,
TransformSpaceChangedEvent,
} from "./events/EngineEvent";
@@ -137,6 +139,20 @@ export class EngineStoreBridge {
usePageSceneStore.getState().updateCamera(e.cameraId, e.updates),
),
// timeline
bus.subscribe(TimelineChangedEvent, (e) => {
const store = usePageSceneStore.getState();
store.setTimelineExists(e.exists);
store.setTimelineDuration(e.duration);
store.setTimelineTracks(e.tracks);
store.setTimelineClipLanes(e.clipLanes);
}),
bus.subscribe(TimelinePlayheadEvent, (e) => {
const store = usePageSceneStore.getState();
store.setTimelinePlayhead(e.playhead);
store.setTimelineIsPlaying(e.isPlaying);
}),
// lifecycle
bus.subscribe(EngineInitializedEvent, (e) =>
usePageSceneStore.getState().setIs3DEditorInitialized(e.initialized),
@@ -0,0 +1,237 @@
// Drives skeletal (Mixamo) animation clips placed on the timeline as clip
// lanes. Each character gets one THREE.AnimationMixer; each clip lane becomes
// an AnimationAction on that mixer. Playback is DETERMINISTIC and playhead-
// driven: we never let the mixer advance on its own — on every timeline
// evaluate() we set each action's absolute time from the playhead and call
// mixer.update(0). This makes scrubbing, play, and frame-accurate recording all
// pose the character identically.
//
// Clips are authored on their own (source) armature. We bind them onto the
// character by node name: Mixamo rigs share the "mixamorig:*" bone naming, so a
// clip's tracks resolve directly against the character's bones. If a character
// uses a differently-named rig, retargeting (SkeletonUtils.retargetClip) is the
// fallback — see retargetOntoCharacter() (not wired until the direct path is
// verified, to keep the first slice simple).
import * as THREE from "three";
import type Editor from "../editor";
import type { ClipLane } from "../timeline/types";
// Per-lane runtime state. `action` is null while the clip GLB is still loading.
interface LaneRuntime {
laneId: string;
characterUuid: string;
sourceMediaId: string;
startTime: number;
loop: boolean;
action: THREE.AnimationAction | null;
clipDuration: number;
}
export class CharacterAnimationManager {
// characterUuid → mixer (root = the character Object3D).
private mixers = new Map<string, THREE.AnimationMixer>();
// laneId → runtime.
private lanes = new Map<string, LaneRuntime>();
// laneId → monotonic token, so a superseded async load can't clobber state.
private loadTokens = new Map<string, number>();
// characterUuid → its skeleton(s), cached so bind-pose resets don't re-traverse.
private skeletons = new Map<string, THREE.Skeleton[]>();
constructor(private readonly editor: Editor) {}
// Reconcile the live runtime to `clipLanes`: dispose lanes that vanished,
// load lanes that appeared. Idempotent — called on every timeline change.
sync(clipLanes: ClipLane[]): void {
const wanted = new Set(clipLanes.map((l) => l.id));
for (const laneId of [...this.lanes.keys()]) {
if (!wanted.has(laneId)) this.disposeLane(laneId);
}
for (const lane of clipLanes) {
const rt = this.lanes.get(lane.id);
if (!rt) {
void this.addLane(lane);
} else {
// Placement can change without a reload.
rt.startTime = lane.strip.startTime;
rt.loop = lane.strip.loop;
}
}
this.pruneMixers();
}
// Pose every character at `playhead` (seconds). Deterministic: sets each
// action's absolute time then applies via mixer.update(0).
evaluateAt(playhead: number): void {
// Characters that have a clip covering this playhead. Everyone else is
// reset to their bind (T) pose below.
const posed = new Set<string>();
for (const rt of this.lanes.values()) {
const action = rt.action;
if (!action) continue;
let local = playhead - rt.startTime;
const dur = rt.clipDuration;
const active = local >= 0 && (rt.loop || local <= dur);
if (!active) {
action.enabled = false;
continue;
}
if (rt.loop && dur > 0) local = local % dur;
else local = Math.min(local, dur);
action.enabled = true;
action.paused = true; // we drive time ourselves; don't auto-advance
action.time = local;
action.setEffectiveWeight(1);
posed.add(rt.characterUuid);
}
// Gaps between strips (and before the first / after the last) have no active
// clip. A disabled action leaves the skeleton frozen on its last evaluated
// frame, so explicitly restore the bind pose for those characters — the
// "default state" when nothing is under the scrubber.
for (const uuid of this.mixers.keys()) {
if (!posed.has(uuid)) this.resetToBindPose(uuid);
}
for (const mixer of this.mixers.values()) mixer.update(0);
}
clear(): void {
for (const laneId of [...this.lanes.keys()]) this.disposeLane(laneId);
for (const mixer of this.mixers.values()) mixer.stopAllAction();
this.mixers.clear();
this.skeletons.clear();
}
// ─── internals ────────────────────────────────────────────────────────
private async addLane(lane: ClipLane): Promise<void> {
const token = (this.loadTokens.get(lane.id) ?? 0) + 1;
this.loadTokens.set(lane.id, token);
const rt: LaneRuntime = {
laneId: lane.id,
characterUuid: lane.objectUuid,
sourceMediaId: lane.strip.sourceMediaId,
startTime: lane.strip.startTime,
loop: lane.strip.loop,
action: null,
clipDuration: lane.strip.duration,
};
this.lanes.set(lane.id, rt);
const character = this.findObject(lane.objectUuid);
if (!character) return;
const glb = await this.editor.activeScene.loadRawGlb(lane.strip.sourceMediaId);
// Bail if this lane was removed/replaced while the clip was loading.
if (this.loadTokens.get(lane.id) !== token || !this.lanes.has(lane.id)) {
return;
}
const clip = glb?.animations?.[0];
if (!clip) {
console.warn(
"Animation clip has no animations track:",
lane.strip.sourceMediaId,
);
return;
}
const mixer = this.getMixer(lane.objectUuid, character);
const action = mixer.clipAction(clip);
action.play();
action.paused = true;
action.enabled = false;
action.loop = rt.loop ? THREE.LoopRepeat : THREE.LoopOnce;
action.clampWhenFinished = true;
rt.action = action;
rt.clipDuration = clip.duration;
// Diagnostic for the direct-bind vs retarget question: if none of the
// clip's tracks resolve to a node on the character, the character won't
// move — that's the signal to wire SkeletonUtils.retargetClip.
if (!this.clipBindsToCharacter(clip, character)) {
console.warn(
`Animation clip "${clip.name}" bound 0 tracks on character ${lane.objectUuid} — likely a rig/bone-name mismatch (retarget needed).`,
);
}
// Adopt the clip's real length into the timeline data (fresh drops only;
// user trims are preserved), so the strip width and move/trim clamps are
// correct and survive save/reload.
this.editor.timelineController.resolveClipDuration(lane.id, clip.duration);
// Pose immediately at the current playhead so the drop is visible even
// when the timeline isn't playing.
this.evaluateAt(this.editor.timelineController.getPlayhead());
}
private disposeLane(laneId: string): void {
const rt = this.lanes.get(laneId);
if (!rt) return;
this.loadTokens.set(laneId, (this.loadTokens.get(laneId) ?? 0) + 1);
if (rt.action) {
const mixer = this.mixers.get(rt.characterUuid);
rt.action.stop();
mixer?.uncacheAction(rt.action.getClip());
mixer?.uncacheClip(rt.action.getClip());
}
this.lanes.delete(laneId);
}
// Drop mixers whose character no longer has any lanes.
private pruneMixers(): void {
const live = new Set([...this.lanes.values()].map((l) => l.characterUuid));
for (const uuid of [...this.mixers.keys()]) {
if (!live.has(uuid)) {
this.mixers.get(uuid)?.stopAllAction();
this.mixers.delete(uuid);
this.skeletons.delete(uuid);
}
}
}
private getMixer(uuid: string, root: THREE.Object3D): THREE.AnimationMixer {
let mixer = this.mixers.get(uuid);
if (!mixer) {
mixer = new THREE.AnimationMixer(root);
this.mixers.set(uuid, mixer);
this.skeletons.set(uuid, this.collectSkeletons(root));
}
return mixer;
}
// Reset a character's skeleton(s) to their bind (rest) pose. For Mixamo rigs
// the bind pose is the T-pose — the "default state" shown wherever no clip
// covers the playhead.
private resetToBindPose(characterUuid: string): void {
const skeletons = this.skeletons.get(characterUuid);
if (!skeletons) return;
for (const skeleton of skeletons) skeleton.pose();
}
private collectSkeletons(root: THREE.Object3D): THREE.Skeleton[] {
const skeletons: THREE.Skeleton[] = [];
root.traverse((o) => {
const sm = o as THREE.SkinnedMesh;
if (sm.isSkinnedMesh && sm.skeleton) skeletons.push(sm.skeleton);
});
return skeletons;
}
// True if at least one of the clip's tracks names a node that exists under
// `character`. Mixamo clips address bones as "mixamorig:Hips.position"; we
// strip the ".property" suffix and look the node up by name.
private clipBindsToCharacter(
clip: THREE.AnimationClip,
character: THREE.Object3D,
): boolean {
return clip.tracks.some((track) => {
const nodeName = track.name.split(".")[0];
if (!nodeName) return false;
return character.getObjectByName(nodeName) !== undefined;
});
}
private findObject(uuid: string): THREE.Object3D | undefined {
return this.editor.activeScene.scene.getObjectByProperty("uuid", uuid);
}
}
@@ -2,7 +2,11 @@ import * as THREE from "three";
import { OrbitControls } from "three/addons/controls/OrbitControls.js";
import Scene from "./scene";
import { PointerLockControls } from "three/addons/controls/PointerLockControls.js";
import { CameraAspectRatio, ClipGroup } from "../enums";
import {
CameraAspectRatio,
ClipGroup,
DEFAULT_CAMERA_ASPECT_RATIO,
} from "../enums";
import { SceneUtils } from "./helper";
import { MouseControls } from "./keybinds_controls";
import { SaveManager } from "./save_manager";
@@ -16,6 +20,8 @@ import { GizmoController } from "./editor/GizmoController";
import { SelectionBridge } from "./editor/SelectionBridge";
import { CameraController } from "./editor/CameraController";
import { HistoryManager } from "./editor/HistoryManager";
import { TimelineController } from "./editor/TimelineController";
import { CharacterAnimationManager } from "./animation/CharacterAnimationManager";
import { DeleteAction } from "./editor/actions/DeleteAction";
import { TransformAction } from "./editor/actions/TransformAction";
@@ -26,9 +32,11 @@ import { ensureAmmoLoaded } from "./ammoLoader";
import { EngineEventBus } from "./events/EngineEventBus";
import { EngineStoreBridge } from "./EngineStoreBridge";
import {
CameraAspectRatioChangedEvent,
EditorStateChangedEvent,
EngineInitializedEvent,
GridVisibleChangedEvent,
CameraViewToggleRequestedEvent,
InspectorPanelChangedEvent,
SceneLoadedEvent,
SceneResetEvent,
@@ -77,6 +85,11 @@ class Editor {
selection: SelectionBridge;
// Owns the camera state, FreeCam plumbing, and the per-frame camera tick.
cameraController: CameraController;
// Owns the animation timeline + keyframe playback (evaluated per frame).
timelineController: TimelineController;
// Drives skeletal (Mixamo) clip lanes onto characters, deterministically
// from the timeline playhead. See CharacterAnimationManager.
characterAnimationManager: CharacterAnimationManager;
// Owns the undo/redo stack. Mutation sites push UndoableAction
// instances via editor.history.record(...); each action class under
// engine/editor/actions/ encapsulates its own apply/revert.
@@ -96,6 +109,7 @@ class Editor {
// Store→engine reactor for grid visibility (toggling the gridHelper
// mesh in/out of the THREE.js scene). Cleared on unmountEngine.
private gridSubscription: () => void;
private cameraViewSubscription: () => void;
// Holds the in-flight transform action between gizmo dragstart and
// dragend. Null whenever no drag is in progress.
@@ -175,6 +189,13 @@ class Editor {
this.activeScene?.applyGridVisibility(e.visible),
);
// Look-through toggle. The double-click intercept and the outliner/exit
// buttons emit this; the controller owns the enter/exit transition.
this.cameraViewSubscription = this.bus.subscribe(
CameraViewToggleRequestedEvent,
() => this.cameraController.switchCameraView(),
);
// PostProcessingPipeline must exist before Scene because Scene's
// load paths invoke updateSurfaceIdAttributeToMesh as a callback.
this.postProcessing = new PostProcessingPipeline();
@@ -287,6 +308,18 @@ class Editor {
loadSceneState: (token) => this.adapter.loadScene(token),
getCameras: () => usePageSceneStore.getState().cameras,
getSelectedCameraId: () => usePageSceneStore.getState().selectedCameraId,
getTimeline: () => this.timelineController.getTimeline(),
loadTimeline: (timeline) =>
this.timelineController.loadTimeline(timeline),
getRenderCameraTransform: () => {
const cam = this.cameraController.cam_obj;
if (!cam) return null;
return {
position: { x: cam.position.x, y: cam.position.y, z: cam.position.z },
rotation: { x: cam.rotation.x, y: cam.rotation.y, z: cam.rotation.z },
};
},
recreateCameraObject: () => this.activeScene._create_camera_obj(),
bus: this.bus,
});
this.viewport = new ViewportController({
@@ -309,6 +342,14 @@ class Editor {
// apply/revert + dependencies. HistoryManager just stores them.
this.history = new HistoryManager({ capacity: 64 });
// Animation timeline. Holds keyframes + drives playback; ticked each
// frame in renderSingleFrame() after the entrance animator. An empty
// timeline always exists so the collapsed timeline bar (the sole
// build-mode bottom UI) is functional; a loaded scene overrides it.
this.characterAnimationManager = new CharacterAnimationManager(this);
this.timelineController = new TimelineController(this);
this.timelineController.create();
this.positive_prompt =
"((masterpiece, best quality, 8K, detailed)), colorful, epic, fantasy, (fox, red fox:1.2), no humans, 1other, ((koi pond)), outdoors, pond, rocks, stones, koi fish, ((watercolor))), lilypad, fish swimming around.";
@@ -490,6 +531,11 @@ class Editor {
} else if (this.activeTransform) {
if (this.activeTransform.commit()) {
this.history.record(this.activeTransform);
// Auto-key: if the dragged object is already keyframed, record
// its new transform at the playhead (replaces the keyframe there
// or creates one at the current scrub point).
const uuid = this.sceneManager?.selected_objects?.[0]?.uuid;
if (uuid) this.timelineController.autoKeyIfTracked(uuid);
}
this.activeTransform = null;
}
@@ -539,6 +585,7 @@ class Editor {
isHotkeyDisabled: () =>
usePageSceneStore.getState().hotkeyStatus.disabled,
getTransformSpace: () => usePageSceneStore.getState().transformSpace,
isViewportLocked: () => this.cameraController.locked,
});
this.sceneManager = new SceneManager(
@@ -601,7 +648,9 @@ class Editor {
}
public isMovable(): boolean {
return this.focused;
// A locked viewport (record mode) blocks mouse-look entirely —
// playback is read-only, so the camera must not turn.
return this.focused && !this.cameraController.locked;
}
// Toggle the three.js Stats panel (FPS / ms / mb). Bound to the
@@ -687,6 +736,14 @@ class Editor {
const originalRenderCameraAspect =
renderCamera?.aspect || originalCameraAspect;
// The camera-view framing offset maps the render frame into an inset
// rect of the viewport canvas — it must not leak into the snapshot,
// which renders the frame full-bleed at the target resolution.
// CameraController.applyFrameProjection reapplies it afterwards.
if (camera.view?.enabled) {
camera.clearViewOffset();
}
// Temporarily set renderer to high resolution
this.renderer.setSize(targetWidth, targetHeight, false);
this.renderer.setPixelRatio(1);
@@ -727,6 +784,10 @@ class Editor {
this.renderer.setSize(originalWidth, originalHeight, false);
this.renderer.setPixelRatio(originalPixelRatio);
// Reapply the camera-view framing offset (if active) before the
// restore render so the viewport doesn't flash an unframed frame.
this.cameraController.applyFrameProjection();
// Re-render at original resolution
if (this.postProcessing.composer) {
this.postProcessing.composer.setSize(originalWidth, originalHeight);
@@ -768,6 +829,14 @@ class Editor {
this.cameraController.cam_obj = this.activeScene.get_object_by_name(
this.cameraController.camera_name,
);
// Fresh scenes always start at the default aspect ratio — don't
// inherit whatever the previously loaded scene used.
this.cameraController.changeRenderCameraAspectRatio(
DEFAULT_CAMERA_ASPECT_RATIO,
);
this.bus.emit(
new CameraAspectRatioChangedEvent(DEFAULT_CAMERA_ASPECT_RATIO),
);
const sceneTitle =
sceneTitleInput && sceneTitleInput !== ""
? sceneTitleInput
@@ -824,6 +893,17 @@ class Editor {
});
}
// True if any object in the scene carries a THREE animation clip (e.g. an
// animated GLB import). Used to auto-expand the timeline on load.
sceneHasAnimation(): boolean {
let found = false;
this.activeScene.scene.traverse((o) => {
const clips = (o as unknown as { animations?: unknown[] }).animations;
if (clips && clips.length > 0) found = true;
});
return found;
}
deleteObject(uuid: string) {
const obj = this.activeScene.scene.getObjectByProperty("uuid", uuid);
if (obj) this.history.record(new DeleteAction(this, obj));
@@ -878,6 +958,10 @@ class Editor {
// Advance any in-flight entrance animations.
this.entranceAnimator.tick(delta_time);
// Evaluate the animation timeline (writes transforms only while
// playing or right after a seek).
this.timelineController.tick(delta_time);
this.activeScene.shader_objects.forEach((shader) => {
shader.material.uniforms["time"].value += 0.5 * delta_time;
});
@@ -969,6 +1053,7 @@ class Editor {
this.isMounted = false;
this.bus.emit(new EngineInitializedEvent(false));
this.gridSubscription();
this.cameraViewSubscription();
this.storeBridge.dispose();
console.log("3D Editor Engine unmounted");
}
@@ -5,7 +5,11 @@ import {
lookAtFromCamera,
type FreeCamControlState,
} from "../cameraMath";
import { CameraAspectRatio, EditorStates } from "../../enums";
import {
CameraAspectRatio,
DEFAULT_CAMERA_ASPECT_RATIO,
EditorStates,
} from "../../enums";
import type { Camera } from "@storyteller/common";
import type { EngineEventBus } from "../events/EngineEventBus";
import { CameraUpdatedEvent } from "../events/EngineEvent";
@@ -16,6 +20,20 @@ export type RenderDimensions = {
aspectRatio: number;
};
// Where the camera-view framing overlay wants the render framing to land,
// in canvas CSS pixels. While a rect is set (and we're in camera view) the
// viewport camera's projection is offset so the exact render framing maps
// into this rect instead of filling the whole canvas — the area outside the
// rect shows the surrounding scene (dimmed by the overlay).
export type CameraFrameRect = {
x: number;
y: number;
width: number;
height: number;
canvasWidth: number;
canvasHeight: number;
};
// Capabilities CameraController needs from outside its own state.
// Editor wires these in initialize() so CameraController never imports
// sibling subsystems directly.
@@ -46,7 +64,7 @@ export class CameraController {
camera: THREE.PerspectiveCamera | null = null;
render_camera: THREE.PerspectiveCamera | null = null;
render_camera_aspect_ratio: CameraAspectRatio = CameraAspectRatio.HORIZONTAL_3_2;
render_camera_aspect_ratio: CameraAspectRatio = DEFAULT_CAMERA_ASPECT_RATIO;
cam_obj: THREE.Object3D | undefined;
camera_person_mode: boolean = false;
@@ -54,6 +72,15 @@ export class CameraController {
lockControls: PointerLockControls | undefined;
freeCamState: FreeCamControlState | null = null;
// When true, FreeCam input is ignored (record mode: read-only viewport).
locked = false;
// Camera-view framing rect set by the CameraFrame overlay; null = none.
private frameRect: CameraFrameRect | null = null;
// Canvas aspect captured when the frame projection is applied, so the
// viewport camera's aspect can be restored the moment the frame clears
// (the resize cascade only corrects it on the next actual resize).
private frameCanvasAspect: number | null = null;
last_cam_pos: THREE.Vector3 = new THREE.Vector3(0, 0, 0);
last_cam_rot: THREE.Euler = new THREE.Euler(0, 0, 0);
@@ -83,69 +110,132 @@ export class CameraController {
}
// Toggles between free-camera (edit) mode and the active render
// camera's perspective. Lifted from SceneUtils.switchCameraView so
// the camera state stays in one place.
// camera's perspective. Split into idempotent enter/exit so callers
// (record mode) can force a specific state without double-toggling.
switchCameraView() {
this.camera_person_mode = !this.camera_person_mode;
if (this.freeCamState) {
this.freeCamState.velocity.set(0, 0, 0);
if (this.camera_person_mode) this.exitCameraView();
else this.enterCameraView();
}
// Look through the render camera. No-op if already in camera view.
enterCameraView() {
// Require a live viewport camera: we MUST capture its pose before
// flipping state, or exit would restore the default (0,0,0) pose and
// leave the viewport pointed at empty space (looks like "everything
// disappeared").
if (this.camera_person_mode || !this.cam_obj || !this.camera) return;
this.last_cam_pos.copy(this.camera.position);
this.last_cam_rot.copy(this.camera.rotation);
this.camera_person_mode = true;
if (this.freeCamState) this.freeCamState.velocity.set(0, 0, 0);
this.camera.position.copy(this.cam_obj.position);
this.camera.rotation.copy(this.cam_obj.rotation);
if (this.lockControls) {
this.deps.getThreeScene().add(this.lockControls.getObject());
}
if (!this.cam_obj) return;
this.cam_obj.scale.set(0, 0, 0);
if (this.camera_person_mode && this.camera) {
this.last_cam_pos.copy(this.camera.position);
this.last_cam_rot.copy(this.camera.rotation);
this.deps.removeTransformControls();
this.deps.setSelected(this.cam_obj);
this.deps.setEditorState(EditorStates.CAMERA_VIEW);
this.camera.position.copy(this.cam_obj.position);
this.camera.rotation.copy(this.cam_obj.rotation);
// Workaround: in camera mode, a right-click should pan rather than
// open the browser context menu. Defer until the letterbox settles.
setTimeout(
() =>
document
.getElementById("letterbox")
?.addEventListener("contextmenu", function (event) {
event.preventDefault();
}),
250,
);
}
if (this.lockControls) {
this.deps.getThreeScene().add(this.lockControls.getObject());
}
// useFreeCam reads editorState from the store and enables
// itself when CAMERA_VIEW; nothing to do here.
this.cam_obj.scale.set(0, 0, 0);
// Return to the free editing camera. No-op if not in camera view.
exitCameraView() {
if (!this.camera_person_mode || !this.cam_obj) return;
this.camera_person_mode = false;
if (this.freeCamState) this.freeCamState.velocity.set(0, 0, 0);
this.deps.removeTransformControls();
this.deps.setSelected(this.cam_obj);
this.deps.setEditorState(EditorStates.CAMERA_VIEW);
const hot = this.deps.getHotItems();
if (hot) {
hot.forEach((element) => {
element.visible = false;
});
}
// Workaround: in camera mode, a right-click should pan rather
// than open the browser context menu. Defer the listener attach
// until after the letterbox swap settles.
setTimeout(
() =>
document
.getElementById("letterbox")
?.addEventListener("contextmenu", function (event) {
event.preventDefault();
}),
250,
);
} else if (this.camera) {
if (this.camera) {
this.camera.position.copy(this.last_cam_pos);
this.camera.rotation.copy(this.last_cam_rot);
if (this.lockControls) {
this.deps.getThreeScene().remove(this.lockControls.getObject());
}
this.cam_obj.scale.set(1, 1, 1);
this.camera.updateProjectionMatrix();
}
// Drop the camera-view framing offset now that we're free-flying again.
this.applyFrameProjection();
if (this.lockControls) {
this.deps.getThreeScene().remove(this.lockControls.getObject());
}
this.cam_obj.scale.set(1, 1, 1);
const hot = this.deps.getHotItems();
if (hot) {
hot.forEach((element) => {
element.visible = true;
});
}
this.deps.hideObjectPanel();
this.deps.setEditorState(EditorStates.EDIT);
}
this.deps.hideObjectPanel();
this.deps.setEditorState(EditorStates.EDIT);
// Record mode locks the viewport: input is ignored so playback is
// strictly read-only. Gated in tickPerFrame's FreeCam integration.
setLocked(locked: boolean): void {
this.locked = locked;
}
// Set (or clear, with null) the camera-view framing rect. Applied
// immediately and kept in sync per-frame by tickPerFrame, so resizes and
// snapshot-time projection resets self-heal on the next tick.
setCameraFrameRect(rect: CameraFrameRect | null): void {
this.frameRect = rect;
this.applyFrameProjection();
}
// Reconcile the viewport camera's projection with the framing rect.
// While in camera view with a rect set: aspect = the render camera's
// aspect, and a view offset maps the full render framing into the rect
// (the canvas renders a wider surround around it). Otherwise: clear any
// offset and restore the full-canvas aspect. Cheap when nothing changed.
applyFrameProjection(): void {
const cam = this.camera;
if (!cam) return;
const rect = this.camera_person_mode ? this.frameRect : null;
if (rect) {
const renderAspect = this.getRenderDimensions().aspectRatio;
const view = cam.view;
const unchanged =
cam.aspect === renderAspect &&
view?.enabled === true &&
view.fullWidth === rect.width &&
view.fullHeight === rect.height &&
view.offsetX === -rect.x &&
view.offsetY === -rect.y &&
view.width === rect.canvasWidth &&
view.height === rect.canvasHeight;
if (unchanged) return;
this.frameCanvasAspect = rect.canvasWidth / rect.canvasHeight;
cam.aspect = renderAspect;
// Virtual "full view" = the framing rect; the canvas renders the
// larger region around it, so the rect shows the exact render frame.
cam.setViewOffset(
rect.width,
rect.height,
-rect.x,
-rect.y,
rect.canvasWidth,
rect.canvasHeight,
);
cam.updateProjectionMatrix();
} else if (cam.view?.enabled) {
cam.clearViewOffset();
if (this.frameCanvasAspect !== null) {
cam.aspect = this.frameCanvasAspect;
this.frameCanvasAspect = null;
}
cam.updateProjectionMatrix();
}
}
@@ -208,7 +298,11 @@ export class CameraController {
}
}
if (this.freeCamState && this.camera) {
// Keep the camera-view framing projection in sync (no-op when nothing
// changed; also heals resize/snapshot-time aspect resets).
this.applyFrameProjection();
if (this.freeCamState && this.camera && !this.locked) {
const moved = freeCamFrameTick(
this.camera,
this.freeCamState,
@@ -0,0 +1,498 @@
// TimelineController owns the animation timeline + playback for the 3D
// editor. It is the authoritative holder of the keyframe data; the store
// only mirrors it (via events → EngineStoreBridge) for the React UI.
//
// Playback contract: transforms are written to THREE objects ONLY while
// playing or on an explicit seek. When idle, the controller does not touch
// object transforms, so manual editing in build mode is unaffected.
import * as THREE from "three";
import type Editor from "../editor";
import { snapshotTransform, writeTransform } from "./actions/snapshots";
import { SaveTimelineAction } from "./actions/SaveTimelineAction";
import { sampleTrackAt } from "../timeline/interpolation";
import {
cloneTimeline,
DEFAULT_EASING,
DEFAULT_TIMELINE_DURATION,
DEFAULT_TIMELINE_FPS,
type ClipLane,
type ClipStrip,
type EasingSpec,
type Keyframe,
type TimelineData,
type TimelineTrack,
} from "../timeline/types";
import {
TimelineChangedEvent,
TimelinePlayheadEvent,
} from "../events/EngineEvent";
export class TimelineController {
private timeline: TimelineData | null = null;
private saved: TimelineData | null = null;
private playhead = 0;
private isPlaying = false;
constructor(private readonly editor: Editor) {}
// ─── queries ──────────────────────────────────────────────────────────
exists(): boolean {
return this.timeline !== null;
}
getTimeline(): TimelineData | null {
return this.timeline;
}
getPlayhead(): number {
return this.playhead;
}
// ─── lifecycle ────────────────────────────────────────────────────────
create(): void {
if (this.timeline) return;
this.timeline = {
duration: DEFAULT_TIMELINE_DURATION,
fps: DEFAULT_TIMELINE_FPS,
tracks: [],
clipLanes: [],
};
this.saved = cloneTimeline(this.timeline);
this.playhead = 0;
this.isPlaying = false;
this.emitChanged();
this.emitPlayhead();
}
clear(): void {
this.timeline = null;
this.saved = null;
this.playhead = 0;
this.isPlaying = false;
this.editor.characterAnimationManager.clear();
this.emitChanged();
this.emitPlayhead();
}
// Replace the live timeline wholesale (used by undo/redo of a save).
loadTimeline(data: TimelineData): void {
this.timeline = cloneTimeline(data);
// Backfill defensively so legacy/partial saved timelines don't break the
// frame-quantized UI (which assumes a valid fps) or produce a 0 duration.
if (!this.timeline.fps || this.timeline.fps <= 0) {
this.timeline.fps = DEFAULT_TIMELINE_FPS;
}
if (!this.timeline.duration || this.timeline.duration <= 0) {
this.timeline.duration = DEFAULT_TIMELINE_DURATION;
}
if (!Array.isArray(this.timeline.tracks)) this.timeline.tracks = [];
if (!Array.isArray(this.timeline.clipLanes)) this.timeline.clipLanes = [];
this.saved = cloneTimeline(this.timeline);
this.playhead = Math.min(this.playhead, this.timeline.duration);
this.syncClipLanes();
this.evaluate();
this.emitChanged();
this.emitPlayhead();
}
// ─── transport ────────────────────────────────────────────────────────
play(): void {
if (!this.timeline) return;
if (this.playhead >= this.timeline.duration) this.playhead = 0;
this.isPlaying = true;
this.emitPlayhead();
}
pause(): void {
this.isPlaying = false;
this.emitPlayhead();
}
seekTo(time: number): void {
if (!this.timeline) return;
this.playhead = Math.max(0, Math.min(time, this.timeline.duration));
this.evaluate();
this.emitPlayhead();
}
// Set the max timeline duration (seconds). Clamped to [1, 60] — covers the
// required 5–30s range with headroom. Keeps the playhead in range.
setDuration(seconds: number): void {
if (!this.timeline || !Number.isFinite(seconds)) return;
this.timeline.duration = Math.max(1, Math.min(60, seconds));
if (this.playhead > this.timeline.duration) {
this.playhead = this.timeline.duration;
}
this.emitChanged();
this.emitPlayhead();
}
// Advanced once per frame from editor.renderSingleFrame().
tick(delta: number): void {
if (!this.timeline || !this.isPlaying) return;
this.playhead += delta;
if (this.playhead >= this.timeline.duration) {
this.playhead = this.timeline.duration;
this.isPlaying = false;
}
this.evaluate();
this.emitPlayhead();
}
// ─── keyframe editing ──────────────────────────────────────────────────
// Auto-key: when an ALREADY-keyframed object is mutated, capture its current
// transform at the playhead — replacing the keyframe there or creating one at
// the current scrub point. No-op for objects that have never been keyframed
// (the first keyframe is always added explicitly).
autoKeyIfTracked(objectUuid: string): void {
if (!this.timeline) return;
const track = this.timeline.tracks.find(
(t) => t.objectUuid === objectUuid,
);
if (!track || track.keyframes.length === 0) return;
this.addKeyframe(objectUuid, this.playhead);
}
addKeyframe(objectUuid: string, atTime?: number): void {
if (!this.timeline) return;
const obj = this.editor.activeScene.scene.getObjectByProperty(
"uuid",
objectUuid,
);
if (!obj) return;
const time = Math.max(
0,
Math.min(atTime ?? this.playhead, this.timeline.duration),
);
const keyframe: Keyframe = {
id: THREE.MathUtils.generateUUID(),
time,
transform: snapshotTransform(obj),
easing: { ...DEFAULT_EASING },
};
let track = this.timeline.tracks.find((t) => t.objectUuid === objectUuid);
if (!track) {
track = { objectUuid, keyframes: [] };
this.timeline.tracks.push(track);
}
// Replace an existing keyframe at (nearly) the same time, else insert.
const existingIdx = track.keyframes.findIndex(
(k) => Math.abs(k.time - time) < 1e-3,
);
if (existingIdx >= 0) track.keyframes[existingIdx] = keyframe;
else track.keyframes.push(keyframe);
this.sortTrack(track);
this.emitChanged();
}
deleteKeyframe(keyframeId: string): void {
if (!this.timeline) return;
for (const track of this.timeline.tracks) {
const idx = track.keyframes.findIndex((k) => k.id === keyframeId);
if (idx >= 0) {
track.keyframes.splice(idx, 1);
break;
}
}
// Drop now-empty tracks so the row disappears.
this.timeline.tracks = this.timeline.tracks.filter(
(t) => t.keyframes.length > 0,
);
this.emitChanged();
}
moveKeyframe(keyframeId: string, time: number): void {
if (!this.timeline) return;
const clamped = Math.max(0, Math.min(time, this.timeline.duration));
for (const track of this.timeline.tracks) {
const kf = track.keyframes.find((k) => k.id === keyframeId);
if (kf) {
kf.time = clamped;
this.sortTrack(track);
break;
}
}
this.evaluate();
this.emitChanged();
}
setEasing(keyframeId: string, easing: EasingSpec): void {
if (!this.timeline) return;
for (const track of this.timeline.tracks) {
const kf = track.keyframes.find((k) => k.id === keyframeId);
if (kf) {
kf.easing = { ...easing };
break;
}
}
this.evaluate();
this.emitChanged();
}
// ─── clip lanes (skeletal animation) ─────────────────────────────────────
// Add a new stacked clip lane for `objectUuid` (a character). The clip is
// dropped at `startTime`, clamped so it starts within the timeline. Returns
// the new lane id, or null if there is no timeline.
addClipLane(objectUuid: string, strip: Omit<ClipStrip, "id">): string | null {
if (!this.timeline) return null;
// Characters hold clips on a single row: snap the drop to the nearest free
// slot so strips never overlap (see resolveFreeStart).
const startTime = this.resolveFreeStart(
objectUuid,
null,
strip.startTime,
strip.duration,
);
const lane: ClipLane = {
id: THREE.MathUtils.generateUUID(),
objectUuid,
strip: { ...strip, id: THREE.MathUtils.generateUUID(), startTime },
};
this.timeline.clipLanes.push(lane);
this.syncClipLanes();
this.evaluate();
this.emitChanged();
return lane.id;
}
// Move a clip strip along its lane. Snaps to the nearest overlap-free slot so
// it never runs into a neighbour, and stays within the timeline.
moveClipLane(laneId: string, startTime: number): void {
if (!this.timeline) return;
const lane = this.timeline.clipLanes.find((l) => l.id === laneId);
if (!lane) return;
lane.strip.startTime = this.resolveFreeStart(
lane.objectUuid,
laneId,
startTime,
lane.strip.duration,
);
this.syncClipLanes();
this.evaluate();
this.emitChanged();
}
// User trim (right edge): set the strip's on-timeline length. Clamped to at
// least one frame and so its end never crosses the next clip or the timeline
// end. Marks the duration as hand-set so a later clip (re)load won't overwrite it.
resizeClipLane(laneId: string, duration: number): void {
if (!this.timeline) return;
const lane = this.timeline.clipLanes.find((l) => l.id === laneId);
if (!lane) return;
const minDur = 1 / (this.timeline.fps || DEFAULT_TIMELINE_FPS);
const nextStart = this.nextStartAfter(
lane.objectUuid,
laneId,
lane.strip.startTime,
);
const maxDur = nextStart - lane.strip.startTime;
lane.strip.duration = Math.max(minDur, Math.min(duration, maxDur));
lane.strip.autoDuration = false;
this.syncClipLanes();
this.evaluate();
this.emitChanged();
}
// Toggle whether a clip loops for the remainder of its strip.
setClipLoop(laneId: string, loop: boolean): void {
if (!this.timeline) return;
const lane = this.timeline.clipLanes.find((l) => l.id === laneId);
if (!lane) return;
lane.strip.loop = loop;
this.syncClipLanes();
this.evaluate();
this.emitChanged();
}
// Called by CharacterAnimationManager once a clip GLB has loaded and its
// real length is known. Adopts the natural length only for a fresh drop
// (autoDuration still set) — a user-trimmed strip keeps its hand-set length.
resolveClipDuration(laneId: string, naturalDuration: number): void {
if (!this.timeline || !(naturalDuration > 0)) return;
const lane = this.timeline.clipLanes.find((l) => l.id === laneId);
if (!lane || lane.strip.autoDuration === false) return;
// Adopt the natural length but never let it overrun the next clip or the
// timeline end (single-row, non-overlapping).
const nextStart = this.nextStartAfter(
lane.objectUuid,
laneId,
lane.strip.startTime,
);
const maxDur = nextStart - lane.strip.startTime;
lane.strip.duration = Math.min(naturalDuration, maxDur);
lane.strip.autoDuration = false;
this.emitChanged();
}
removeClipLane(laneId: string): void {
if (!this.timeline) return;
this.timeline.clipLanes = this.timeline.clipLanes.filter(
(l) => l.id !== laneId,
);
this.syncClipLanes();
this.evaluate();
this.emitChanged();
}
// All clip lanes for one character (stacked lanes, drop order).
clipLanesFor(objectUuid: string): ClipLane[] {
if (!this.timeline) return [];
return this.timeline.clipLanes.filter((l) => l.objectUuid === objectUuid);
}
// ─── save / cancel ──────────────────────────────────────────────────────
save(): void {
if (!this.timeline) return;
const before = this.saved
? cloneTimeline(this.saved)
: cloneTimeline(this.timeline);
const action = new SaveTimelineAction(this, before);
this.saved = cloneTimeline(this.timeline);
if (action.commit()) this.editor.history.record(action);
this.emitChanged();
}
cancel(): void {
if (!this.saved) return;
this.timeline = cloneTimeline(this.saved);
this.syncClipLanes();
this.evaluate();
this.emitChanged();
this.emitPlayhead();
}
// ─── internals ────────────────────────────────────────────────────────
private evaluate(): void {
if (!this.timeline) return;
const cameraController = this.editor.cameraController;
for (const track of this.timeline.tracks) {
const snap = sampleTrackAt(track, this.playhead);
if (!snap) continue;
writeTransform(this.editor, track.objectUuid, snap);
// In camera view the canvas renders through the viewport camera and
// CameraController.tickPerFrame copies camera → cam_obj (the user's
// flying drives the proxy). Timeline-driven camera motion goes the
// other way, so mirror it onto the viewport camera here — otherwise
// playback/scrubbing shows no movement while looking through the
// camera (it's only visible as the frustum moving in viewport mode).
if (
cameraController.getCameraPersonMode() &&
cameraController.cam_obj?.uuid === track.objectUuid &&
cameraController.camera
) {
cameraController.camera.position.copy(
cameraController.cam_obj.position,
);
cameraController.camera.rotation.copy(
cameraController.cam_obj.rotation,
);
}
}
// Pose characters from their skeletal clip lanes at the same playhead.
this.editor.characterAnimationManager.evaluateAt(this.playhead);
}
// ─── clip-lane overlap guard (single row per character) ─────────────────
// Snap `desiredStart` to the nearest position where a `duration`-long strip
// fits without overlapping the character's other clips, within the timeline.
// Used by add + move so a character's clips stay a non-overlapping sequence.
private resolveFreeStart(
characterUuid: string,
exceptLaneId: string | null,
desiredStart: number,
duration: number,
): number {
if (!this.timeline) return desiredStart;
const maxStart = Math.max(0, this.timeline.duration - duration);
const clamp = (s: number) => Math.max(0, Math.min(s, maxStart));
const others = this.otherStripIntervals(characterUuid, exceptLaneId);
const desired = clamp(desiredStart);
if (!this.overlapsAny(desired, duration, others)) return desired;
// Candidate slots: flush before/after each neighbour, plus the two ends.
const candidates = [0, maxStart];
for (const iv of others) {
candidates.push(iv.end);
candidates.push(iv.start - duration);
}
const valid = candidates
.map(clamp)
.filter((s) => !this.overlapsAny(s, duration, others));
if (valid.length === 0) return desired; // no room — accept the clamp
valid.sort((a, b) => Math.abs(a - desired) - Math.abs(b - desired));
return valid[0];
}
// Start of the nearest clip that begins at/after `start` (excluding
// `exceptLaneId`), or the timeline end if there is none. Bounds trim/length.
private nextStartAfter(
characterUuid: string,
exceptLaneId: string | null,
start: number,
): number {
if (!this.timeline) return start;
let next = this.timeline.duration;
for (const iv of this.otherStripIntervals(characterUuid, exceptLaneId)) {
if (iv.start >= start) next = Math.min(next, iv.start);
}
return next;
}
// Occupied [start, end] intervals of a character's OTHER clip strips, sorted.
private otherStripIntervals(
characterUuid: string,
exceptLaneId: string | null,
): Array<{ start: number; end: number }> {
if (!this.timeline) return [];
return this.timeline.clipLanes
.filter((l) => l.objectUuid === characterUuid && l.id !== exceptLaneId)
.map((l) => ({
start: l.strip.startTime,
end: l.strip.startTime + l.strip.duration,
}))
.sort((a, b) => a.start - b.start);
}
private overlapsAny(
start: number,
duration: number,
intervals: Array<{ start: number; end: number }>,
): boolean {
const end = start + duration;
return intervals.some((iv) => start < iv.end && end > iv.start);
}
// Reconcile the skeletal-animation runtime whenever the clip-lane set changes.
private syncClipLanes(): void {
this.editor.characterAnimationManager.sync(this.timeline?.clipLanes ?? []);
}
private sortTrack(track: TimelineTrack): void {
track.keyframes.sort((a, b) => a.time - b.time);
}
private emitChanged(): void {
this.editor.bus.emit(
new TimelineChangedEvent(
this.timeline !== null,
this.timeline?.duration ?? DEFAULT_TIMELINE_DURATION,
this.timeline ? cloneTimeline(this.timeline).tracks : [],
this.timeline ? cloneTimeline(this.timeline).clipLanes : [],
),
);
}
private emitPlayhead(): void {
this.editor.bus.emit(
new TimelinePlayheadEvent(this.playhead, this.isPlaying),
);
}
}
@@ -0,0 +1,32 @@
import type { UndoableAction } from "../HistoryManager";
import type { TimelineController } from "../TimelineController";
import { cloneTimeline, type TimelineData } from "../../timeline/types";
// Records a timeline "Save": the whole timeline before vs. after the save.
// Mirrors TransformAction's before/commit/after shape. The controller
// constructs this with the pre-save (last-saved) timeline, calls commit()
// to capture the post-save timeline, and records it only if they differ.
export class SaveTimelineAction implements UndoableAction {
readonly label = "Save Timeline";
private after?: TimelineData;
constructor(
private readonly controller: TimelineController,
private readonly before: TimelineData,
) {}
commit(): boolean {
const current = this.controller.getTimeline();
if (!current) return false;
this.after = cloneTimeline(current);
return JSON.stringify(this.before) !== JSON.stringify(this.after);
}
apply(): void {
if (this.after) this.controller.loadTimeline(this.after);
}
revert(): void {
this.controller.loadTimeline(this.before);
}
}
@@ -21,6 +21,7 @@ import type {
TransformSpace,
} from "../../PageSceneStore";
import type { Camera } from "@storyteller/common";
import type { ClipLane, TimelineTrack } from "../timeline/types";
export abstract class EngineEvent {
readonly timestamp: number = performance.now();
@@ -181,3 +182,30 @@ export class CameraUpdatedEvent extends EngineEvent {
super();
}
}
// Request to toggle "camera view" (look through / exit the render camera).
// Emitted by the double-click intercept and the outliner/exit buttons;
// editor.ts subscribes and calls cameraController.switchCameraView().
export class CameraViewToggleRequestedEvent extends EngineEvent {}
// ─── timeline ───────────────────────────────────────────────────────────
export class TimelineChangedEvent extends EngineEvent {
constructor(
readonly exists: boolean,
readonly duration: number,
readonly tracks: TimelineTrack[],
readonly clipLanes: ClipLane[],
) {
super();
}
}
export class TimelinePlayheadEvent extends EngineEvent {
constructor(
readonly playhead: number,
readonly isPlaying: boolean,
) {
super();
}
}
@@ -95,6 +95,9 @@ export type MouseControlsDeps = {
getPoseMode: () => PoseMode;
isHotkeyDisabled: () => boolean;
getTransformSpace: () => TransformSpace;
// True when the viewport is locked (record mode) — camera look / focus
// must be fully inert so record playback is immutable.
isViewportLocked: () => boolean;
};
export class MouseControls {
@@ -189,6 +192,7 @@ export class MouseControls {
}
focus() {
if (this.deps.isViewportLocked()) return;
if (this.lockControls && this.selected) {
this.lockControls.camera.lookAt(this.selected[0].position);
this.lockControls.camera.position.copy(this.selected[0].position);
@@ -499,6 +503,8 @@ export class MouseControls {
}
handleMouseManualLock(event: MouseEvent) {
// Record mode locks the viewport — no mouse-look camera rotation.
if (this.deps.isViewportLocked()) return;
if (this.isMouseClicked && this.lockControls) {
// If the mouse is clicked and the lockControls is not locked, lock it
if (this.lockControls.isLocked == false) {
@@ -52,6 +52,11 @@ const deleteSelected = (editor: Editor) => {
};
const onEscape = (editor: Editor) => {
// Escape leaves camera view first (before touching pose/selection).
if (editor.cameraController.getCameraPersonMode()) {
editor.cameraController.exitCameraView();
return;
}
const poseMode: PoseMode = editor.getPoseMode();
if (poseMode === "pose") {
editor.mouse_controls?.toggleFKMode();
@@ -0,0 +1,120 @@
// Deterministic timeline → video encoder. Steps the timeline frame-by-frame
// (seekTo, render, grab) and encodes with mediabunny (WebCodecs) — the same
// pattern as video-editor's SceneExporter. Avoids realtime MediaRecorder
// (whose webm duration header is unreliable).
import {
Output,
Mp4OutputFormat,
WebMOutputFormat,
BufferTarget,
CanvasSource,
QUALITY_HIGH,
} from "mediabunny";
import * as THREE from "three";
import type Editor from "../editor";
import { GridVisibleChangedEvent } from "../events/EngineEvent";
import { usePageSceneStore } from "../../PageSceneStore";
export type RecordFormat = "mp4" | "webm";
export interface RecordResult {
blob: Blob;
mimeType: string;
fileName: string;
}
export interface RecordOptions {
format?: RecordFormat;
onProgress?: (pct: number) => void;
// Mutable flag the caller can flip to abort mid-encode.
signal?: { cancelled: boolean };
}
export async function recordTimeline(
editor: Editor,
opts: RecordOptions = {},
): Promise<RecordResult | null> {
const format = opts.format ?? "mp4";
const timeline = editor.timelineController.getTimeline();
const renderer = editor.renderer;
const canvas = renderer?.domElement;
const camera = editor.cameraController.camera;
if (!timeline || !renderer || !canvas || !camera) return null;
const fps = timeline.fps || 30;
const frameCount = Math.max(1, Math.round(timeline.duration * fps));
// Freeze the realtime loop so our seek+render stepping isn't fought by the
// per-frame tick, then always restore it.
editor.stopRenderLoop();
// Encode the exact render framing, not the raw viewport: drop the
// camera-view framing offset, hide the grid, and render at the render
// camera's target dimensions/aspect for the duration of the encode.
// Everything is restored in the finally block.
const { width: targetWidth, height: targetHeight, aspectRatio } =
editor.cameraController.getRenderDimensions();
const sizeVector = new THREE.Vector2();
renderer.getSize(sizeVector);
const originalWidth = sizeVector.x;
const originalHeight = sizeVector.y;
const originalPixelRatio = renderer.getPixelRatio();
const originalCameraAspect = camera.aspect;
const wasGridVisible = usePageSceneStore.getState().gridVisible;
editor.bus.emit(new GridVisibleChangedEvent(false));
if (camera.view?.enabled) camera.clearViewOffset();
camera.aspect = aspectRatio;
camera.updateProjectionMatrix();
renderer.setSize(targetWidth, targetHeight, false);
renderer.setPixelRatio(1);
editor.postProcessing.composer?.setSize(targetWidth, targetHeight);
try {
const output = new Output({
format:
format === "webm" ? new WebMOutputFormat() : new Mp4OutputFormat(),
target: new BufferTarget(),
});
const videoSource = new CanvasSource(canvas, {
codec: format === "webm" ? "vp9" : "avc",
bitrate: QUALITY_HIGH,
});
output.addVideoTrack(videoSource, { frameRate: fps });
await output.start();
for (let i = 0; i < frameCount; i++) {
if (opts.signal?.cancelled) {
await output.cancel();
return null;
}
editor.timelineController.seekTo(i / fps);
await editor.renderScene();
await videoSource.add(i / fps, 1 / fps);
opts.onProgress?.(i / frameCount);
}
videoSource.close();
await output.finalize();
opts.onProgress?.(1);
const buffer = output.target.buffer;
if (!buffer) return null;
const mimeType = format === "webm" ? "video/webm" : "video/mp4";
return {
blob: new Blob([buffer], { type: mimeType }),
mimeType,
fileName: `scene-recording.${format}`,
};
} finally {
camera.aspect = originalCameraAspect;
camera.updateProjectionMatrix();
renderer.setSize(originalWidth, originalHeight, false);
renderer.setPixelRatio(originalPixelRatio);
editor.postProcessing.composer?.setSize(originalWidth, originalHeight);
editor.cameraController.applyFrameProjection();
editor.bus.emit(new GridVisibleChangedEvent(wasGridVisible));
editor.startRenderLoop();
}
}
@@ -6,6 +6,7 @@ import { StoryTellerProxyScene } from "../proxy/storyteller_proxy_scene";
import { CameraAspectRatio } from "../enums";
import type Scene from "./scene";
import type { EngineEventBus } from "./events/EngineEventBus";
import type { TimelineData } from "./timeline/types";
import {
CameraAspectRatioChangedEvent,
CamerasReplacedEvent,
@@ -71,6 +72,21 @@ export type SaveManagerDeps = {
getCameras: () => Camera[];
getSelectedCameraId: () => string;
// Animation timeline persistence (in-memory controller ↔ scene JSON).
getTimeline: () => TimelineData | null;
loadTimeline: (timeline: TimelineData) => void;
// Live render-camera (::CAM:: frustum) transform, read at save time so the
// persisted render-camera position reflects the actual scene object rather
// than the (possibly stale) store config. Null if no frustum exists.
getRenderCameraTransform: () => {
position: { x: number; y: number; z: number };
rotation: { x: number; y: number; z: number };
} | null;
// Recreate the ::CAM:: frustum after a load (the load path strips it). Must
// run AFTER the store cameras are restored so it reads the loaded config.
recreateCameraObject: () => void;
// Typed event bus — every engine→store write goes through here.
bus: EngineEventBus;
};
@@ -131,20 +147,31 @@ export class SaveManager {
const proxyScene = new StoryTellerProxyScene(version, scene);
const scene_json = proxyScene.saveToScene(version);
const camerasData = this.deps.getCameras().map((cam: Camera) => ({
id: cam.id,
label: cam.label,
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}));
// The render camera's true position lives on the live ::CAM:: frustum,
// not the store config (which goes stale on edit-mode gizmo moves). At
// save time, override the render-camera entry (first non-"main") with the
// live transform so a reload restores where the frustum actually is.
const renderTransform = this.deps.getRenderCameraTransform();
const renderCameraId = this.deps
.getCameras()
.find((cam) => cam.id !== "main")?.id;
const camerasData = this.deps.getCameras().map((cam: Camera) => {
const useLive = renderTransform && cam.id === renderCameraId;
return {
id: cam.id,
label: cam.label,
focalLength: cam.focalLength,
position: useLive ? renderTransform.position : cam.position,
rotation: useLive ? renderTransform.rotation : cam.rotation,
lookAt: cam.lookAt,
};
});
return {
version,
scene: scene_json,
...sceneGenerationMetadata,
timeline: "",
timeline: this.deps.getTimeline() ?? null,
skybox: scene.skybox,
camera_data: {
position: camera?.position,
@@ -284,7 +311,24 @@ export class SaveManager {
}
this.deps.setVersion(scene_json["version"]);
// The load path stripped ::CAM::; recreate it now that the store cameras
// are restored (CamerasReplacedEvent above ran synchronously), so the
// frustum lands at the saved render-camera transform, then re-anchor it.
this.deps.recreateCameraObject();
this.deps.refreshCamObj();
// Restore the animation timeline. Guard legacy scenes where `timeline`
// was a `""` stub (or absent/null).
const timeline = scene_json["timeline"];
if (
timeline &&
typeof timeline === "object" &&
Array.isArray(timeline.tracks)
) {
this.deps.loadTimeline(timeline);
}
return true;
}
}
@@ -108,7 +108,7 @@ class Scene {
this._createGrid();
this._create_base_lighting();
this._create_skybox();
// this._create_camera_obj();
this._create_camera_obj();
this.helper = new MMDAnimationHelper({ afterglow: 0.0 });
this.scene.userData["helper"] = this.helper;
@@ -130,10 +130,13 @@ class Scene {
clear() {
this.scene.children = [];
// hot_items held detached keyframe-point refs across reloads; reset it
// so it never carries stale objects from a previous scene.
this.hot_items = [];
this._createGrid();
this._create_base_lighting();
this._create_skybox();
// this._create_camera_obj();
this._create_camera_obj();
}
async instantiate(
@@ -411,45 +414,79 @@ class Scene {
//this.scene.background = null;
}
async _create_camera_obj() {
// Creates the render-camera placeholder — a selectable wireframe frustum
// named "::CAM::" that lives in the scene like a Blender camera. Its
// transform mirrors the render camera (CameraController.tickPerFrame syncs
// cam_obj → render_camera). Placed on layer 1 so the viewport camera sees
// it but render_camera (layer 1 disabled) excludes it from output. Has no
// userData.media_id, so the save path skips it. Single camera for now.
_create_camera_obj() {
const cameras = this.deps.getCameras();
const selectedCameraId = this.deps.getSelectedCameraId();
cameras.forEach((cameraConfig) => {
const camera_position = new THREE.Vector3(
cameraConfig.position.x,
cameraConfig.position.y,
cameraConfig.position.z,
);
const renderCam = cameras.find((c) => c.id !== "main") ?? cameras[0];
if (!renderCam) return;
const camera_id = "m_cxh4asqhapdz10j880755dg4yevshb";
// Idempotent: drop any existing frustum so callers (initialize, clear,
// load-restore) never leave duplicate "::CAM::" objects.
const existing = this.scene.getObjectByName("::CAM::");
if (existing) this.scene.remove(existing);
this.loadGlbWithPlaceholder(
camera_id,
"Camera",
false,
camera_position,
).then((camera_obj) => {
camera_obj.userData["name"] = cameraConfig.label;
camera_obj.name = cameraConfig.label;
camera_obj.position.set(
camera_position.x,
camera_position.y,
camera_position.z,
);
camera_obj.layers.set(1);
camera_obj.children.forEach((child) => {
child.layers.set(1);
});
this.scene.add(camera_obj);
const group = this._buildCameraFrustum();
group.name = "::CAM::";
group.userData["name"] = "Camera";
group.position.set(
renderCam.position.x,
renderCam.position.y,
renderCam.position.z,
);
group.rotation.set(
renderCam.rotation.x,
renderCam.rotation.y,
renderCam.rotation.z,
);
group.layers.set(1);
group.children.forEach((child) => child.layers.set(1));
this.scene.add(group);
}
if (cameraConfig.id === selectedCameraId) {
camera_obj.visible = false;
camera_obj.layers.disableAll(); // Make the active camera not touchable
} else {
camera_obj.visible = true;
}
});
// Builds the frustum line geometry: apex at the camera origin opening
// toward -Z, with a near rectangle. White lines, matching the design.
private _buildCameraFrustum(): THREE.Group {
const group = new THREE.Group();
const depth = 0.7;
const halfW = 0.45;
const halfH = 0.3;
const apex = new THREE.Vector3(0, 0, 0);
const tl = new THREE.Vector3(-halfW, halfH, -depth);
const tr = new THREE.Vector3(halfW, halfH, -depth);
const br = new THREE.Vector3(halfW, -halfH, -depth);
const bl = new THREE.Vector3(-halfW, -halfH, -depth);
const points: THREE.Vector3[] = [
apex, tl, apex, tr, apex, br, apex, bl, // apex → corners
tl, tr, tr, br, br, bl, bl, tl, // near rectangle
];
const geometry = new THREE.BufferGeometry().setFromPoints(points);
const material = new THREE.LineBasicMaterial({ color: 0xffffff });
const lines = new THREE.LineSegments(geometry, material);
// Disable line raycasting: THREE picks LineSegments with a 1-world-unit
// threshold, which dwarfs this ~0.7u frustum and selects it from far
// away. Selection is instead handled by the invisible pick-proxy below.
lines.raycast = () => {};
group.add(lines);
// Invisible pick-proxy sized to the frustum volume so clicks match the
// wireframe rather than a 1-unit halo. Raycastable (opacity 0, not
// mesh.visible=false) but draws nothing.
const proxyGeometry = new THREE.BoxGeometry(halfW * 2, halfH * 2, depth);
const proxyMaterial = new THREE.MeshBasicMaterial({
transparent: true,
opacity: 0,
depthWrite: false,
});
const proxy = new THREE.Mesh(proxyGeometry, proxyMaterial);
proxy.position.set(0, 0, -depth / 2);
group.add(proxy);
return group;
}
renderMode(enabled: boolean = true) {
@@ -907,6 +944,25 @@ class Scene {
* @param onComplete
* @returns
*/
// Load a GLB by media id and return the raw GLTF (scene + animations)
// WITHOUT adding it to the scene or attaching a loading placeholder. Used to
// source skeletal animation clips (their `animations[0]`) for retargeting
// onto characters — the clip's own mesh/armature is never rendered.
async loadRawGlb(
media_id: string,
signal?: AbortSignal,
): Promise<GLTF | null> {
try {
const url = await this.getMediaURL(media_id);
return await this.load_glb_wrapped_no_cors(url, () => {}, signal);
} catch (error) {
if ((error as { name?: string })?.name !== "AbortError") {
console.error("loadRawGlb failed:", media_id, error);
}
return null;
}
}
private async load_glb_wrapped_no_cors(
media_url: string,
onComplete: () => void,
@@ -10,7 +10,10 @@ import { MouseControls } from "./keybinds_controls";
import { ClipGroup, AssetType } from "../enums";
import { XYZ } from "../datastructures/common";
import type { EngineEventBus } from "./events/EngineEventBus";
import { ObjectAddedEvent } from "./events/EngineEvent";
import {
CameraViewToggleRequestedEvent,
ObjectAddedEvent,
} from "./events/EngineEvent";
import { isInternalBbox } from "./internalBbox";
export type SceneObject = {
@@ -20,6 +23,7 @@ export type SceneObject = {
type: string;
visible: boolean;
locked: boolean;
isCamera: boolean;
};
export interface SceneManagerAPI {
@@ -147,6 +151,13 @@ export class SceneManager implements SceneManagerAPI {
}
public async double_click() {
// Double-clicking the render-camera placeholder enters/exits camera view
// instead of the usual focus-zoom. editor.ts owns the transition.
const selected = this.selected_objects?.[0];
if (selected?.name === "::CAM::") {
this.bus.emit(new CameraViewToggleRequestedEvent());
return;
}
this.mouse_controls.focus();
}
@@ -198,11 +209,15 @@ export class SceneManager implements SceneManagerAPI {
) {
let faicon = faCube;
let name = object.name;
// Human-readable kind shown as the outliner row's subtitle.
let kind = "3D Object";
if (object.name == "::CAM::") {
faicon = faCamera;
name = "Camera";
kind = "Camera";
} else if (object.uuid in timeline_characters) {
faicon = faPerson;
kind = "Character";
}
let locked = object.userData["locked"];
if (locked == undefined) {
@@ -212,9 +227,10 @@ export class SceneManager implements SceneManagerAPI {
id: object.uuid,
icon: faicon,
name: name.charAt(0).toUpperCase() + name.slice(1),
type: object.type,
type: kind,
visible: object.visible,
locked: object.userData["locked"],
isCamera: object.name == "::CAM::",
};
}
@@ -0,0 +1,78 @@
// Keyframe interpolation: cubic-bezier easing solver + per-track sampling.
// Kept dependency-free (only the timeline types + TransformSnap).
import type { TransformSnap } from "../editor/actions/snapshots";
import type { EasingSpec, TimelineTrack } from "./types";
// Evaluate one axis of a cubic bezier with endpoints (0,0)/(1,1) at
// parameter t. `a1`/`a2` are that axis's two control-point coordinates.
const bezierAxis = (t: number, a1: number, a2: number): number => {
const c = 3 * a1;
const b = 3 * (a2 - a1) - c;
const a = 1 - c - b;
return ((a * t + b) * t + c) * t;
};
const bezierAxisDerivative = (t: number, a1: number, a2: number): number => {
const c = 3 * a1;
const b = 3 * (a2 - a1) - c;
const a = 1 - c - b;
return (3 * a * t + 2 * b) * t + c;
};
// Given a normalized progress x in [0,1] along the segment, return the
// eased y. Solves bezierX(t) = x for t (Newton-Raphson), then reads y(t).
export const cubicBezierYForX = (easing: EasingSpec, x: number): number => {
if (x <= 0) return 0;
if (x >= 1) return 1;
let t = x;
for (let i = 0; i < 8; i++) {
const err = bezierAxis(t, easing.p1x, easing.p2x) - x;
if (Math.abs(err) < 1e-5) break;
const d = bezierAxisDerivative(t, easing.p1x, easing.p2x);
if (Math.abs(d) < 1e-6) break;
t -= err / d;
}
t = Math.max(0, Math.min(1, t));
return bezierAxis(t, easing.p1y, easing.p2y);
};
type Vec3 = { x: number; y: number; z: number };
const lerp = (a: number, b: number, t: number): number => a + (b - a) * t;
const lerpVec3 = (a: Vec3, b: Vec3, t: number): Vec3 => ({
x: lerp(a.x, b.x, t),
y: lerp(a.y, b.y, t),
z: lerp(a.z, b.z, t),
});
// Sample a track's transform at the given time. Assumes keyframes are
// sorted ascending by time. Clamps to the first/last keyframe outside the
// keyed range. Returns null when the track has no keyframes.
export const sampleTrackAt = (
track: TimelineTrack,
time: number,
): TransformSnap | null => {
const kfs = track.keyframes;
if (kfs.length === 0) return null;
if (kfs.length === 1 || time <= kfs[0].time) return kfs[0].transform;
const last = kfs[kfs.length - 1];
if (time >= last.time) return last.transform;
for (let i = 0; i < kfs.length - 1; i++) {
const a = kfs[i];
const b = kfs[i + 1];
if (time >= a.time && time <= b.time) {
const span = b.time - a.time;
const x = span <= 0 ? 0 : (time - a.time) / span;
const eased = cubicBezierYForX(a.easing, x);
return {
position: lerpVec3(a.transform.position, b.transform.position, eased),
rotation: lerpVec3(a.transform.rotation, b.transform.rotation, eased),
scale: lerpVec3(a.transform.scale, b.transform.scale, eased),
};
}
}
return last.transform;
};
@@ -0,0 +1,79 @@
// Animation-timeline data model. Pure data (no THREE / no store imports)
// so it can be shared by the engine controller, the event bus, and the
// React UI without creating runtime cycles.
//
// A keyframe stores a full transform snapshot at a point in time; playback
// interpolates between consecutive keyframes using the LEFT keyframe's
// easing curve (the curve describes the motion INTO the next keyframe).
import type { TransformSnap } from "../editor/actions/snapshots";
// Cubic-bezier control points, CSS `cubic-bezier(p1x,p1y,p2x,p2y)` semantics
// with implicit endpoints P0=(0,0) and P3=(1,1).
export interface EasingSpec {
p1x: number;
p1y: number;
p2x: number;
p2y: number;
}
export interface Keyframe {
id: string;
time: number; // seconds from timeline start
transform: TransformSnap;
easing: EasingSpec; // curve into the next keyframe
}
export interface TimelineTrack {
objectUuid: string;
keyframes: Keyframe[]; // kept sorted ascending by time
}
// A skeletal animation clip (Mixamo) placed on a character's lane. Only the
// reference + placement is serialized; the actual THREE.AnimationClip is
// resolved from `sourceMediaId` and retargeted onto the character at runtime.
export interface ClipStrip {
id: string;
sourceMediaId: string; // media id of the animation GLB (reload + retarget)
name: string; // display label
startTime: number; // seconds from timeline start
duration: number; // seconds (clip length on the timeline)
loop: boolean;
// True until the clip GLB has loaded and its real length is known. While
// true, `duration` is only a placeholder width and gets replaced by the
// clip's natural length on load. A user trim clears this so the natural
// length never clobbers a hand-set duration on reload.
autoDuration?: boolean;
}
// One animation lane under a character. Multiple lanes (stacked) = multiple
// clips on the same character; each lane holds a single clip strip. Only
// character objects get clip lanes.
export interface ClipLane {
id: string;
objectUuid: string; // the character this lane animates
strip: ClipStrip;
}
export interface TimelineData {
duration: number; // seconds
fps: number;
tracks: TimelineTrack[]; // per-object transform keyframes
clipLanes: ClipLane[]; // per-character skeletal animation clips
}
export type EasingPresetName = "linear" | "easeIn" | "easeOut" | "easeInOut";
export const EASING_PRESETS: Record<EasingPresetName, EasingSpec> = {
linear: { p1x: 0, p1y: 0, p2x: 1, p2y: 1 },
easeIn: { p1x: 0.42, p1y: 0, p2x: 1, p2y: 1 },
easeOut: { p1x: 0, p1y: 0, p2x: 0.58, p2y: 1 },
easeInOut: { p1x: 0.42, p1y: 0, p2x: 0.58, p2y: 1 },
};
export const DEFAULT_EASING: EasingSpec = EASING_PRESETS.easeInOut;
export const DEFAULT_TIMELINE_DURATION = 10; // seconds
export const DEFAULT_TIMELINE_FPS = 30;
export const cloneTimeline = (t: TimelineData): TimelineData =>
JSON.parse(JSON.stringify(t));
@@ -10,3 +10,7 @@ export enum CameraAspectRatio {
HORIZONTAL_3_2 = "horizontal_3_2",
VERTICAL_2_3 = "vertical_2_3",
}
// Single source of truth for the aspect ratio a fresh scene starts with.
// Store default, CameraController default, and scene reset all read this.
export const DEFAULT_CAMERA_ASPECT_RATIO = CameraAspectRatio.HORIZONTAL_16_9;
@@ -134,6 +134,8 @@ export const useFreeCam = (
};
const onPointerDown = (e: PointerEvent) => {
// Record mode locks the viewport — no right-drag panning.
if (editor.cameraController.locked) return;
if (e.button !== 2) return;
dragRef.current = { x: e.clientX, y: e.clientY, pointerId: e.pointerId };
state.velocity.set(0, 0, 0);
@@ -156,6 +158,7 @@ export const useFreeCam = (
};
const onPointerMove = (e: PointerEvent) => {
if (editor.cameraController.locked) return;
const drag = dragRef.current;
const camera = editor.cameraController.camera;
if (!drag || !camera) return;
@@ -173,6 +176,8 @@ export const useFreeCam = (
};
const onWheel = (e: WheelEvent) => {
// Record mode locks the viewport — no wheel zoom.
if (editor.cameraController.locked) return;
const camera = editor.cameraController.camera;
if (!camera) return;
const z = zoomFromWheel(e.deltaY);
@@ -51,16 +51,31 @@ export const useViewportPointer = (
editor.mouse_controls?.onMouseClick();
};
// Double-click toggles camera view: anywhere exits when active; on the
// render-camera frustum (selected by the preceding click) it enters.
const onDoubleClick = () => {
if (cameraViewActive) {
editor.cameraController.exitCameraView();
return;
}
const selected = editor.sceneManager?.selected_objects?.[0];
if (selected?.name === "::CAM::") {
editor.cameraController.enterCameraView();
}
};
canvas.addEventListener("pointermove", onPointerMove);
canvas.addEventListener("pointerdown", onPointerDown);
canvas.addEventListener("pointerup", onPointerUp);
canvas.addEventListener("click", onClick);
canvas.addEventListener("dblclick", onDoubleClick);
return () => {
canvas.removeEventListener("pointermove", onPointerMove);
canvas.removeEventListener("pointerdown", onPointerDown);
canvas.removeEventListener("pointerup", onPointerUp);
canvas.removeEventListener("click", onClick);
canvas.removeEventListener("dblclick", onDoubleClick);
};
}, [canvas, editor, cameraViewActive]);
};
@@ -14,6 +14,9 @@
{
"path": "../button-icon-select"
},
{
"path": "../tab-selector"
},
{
"path": "../badge"
},
@@ -62,6 +62,9 @@
{
"path": "../button-icon-select/tsconfig.lib.json"
},
{
"path": "../tab-selector/tsconfig.lib.json"
},
{
"path": "../badge/tsconfig.lib.json"
},
@@ -1,3 +1,4 @@
export * from "./lib/common/AspectRatioIcon";
export * from "./lib/PromptBox2D";
export * from "./lib/PromptBox3D";
export * from "./lib/PromptBoxEdit";
@@ -7,6 +7,10 @@ import {
faExpand,
faChevronDown,
faChevronUp,
faArrowPointer,
faWandMagicSparkles,
faClockRotateLeft,
faPlus,
} from "@fortawesome/pro-solid-svg-icons";
import {
faRectangleWide,
@@ -18,6 +22,7 @@ import {
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { PopoverItem, PopoverMenu } from "@storyteller/ui-popover";
import { Button, ToggleButton, GenerateButton } from "@storyteller/ui-button";
import { ButtonIconSelect } from "@storyteller/ui-button-icon-select";
import { Tooltip } from "@storyteller/ui-tooltip";
import {
CameraAspectRatio,
@@ -99,6 +104,19 @@ interface PromptBox3DProps {
// Optional pre-submit gate. Returns false to abort the generation
// (e.g. host wants to open a signup modal for anon visitors).
onBeforeSubmit?: () => boolean;
/** Build sub-mode. "manual" (default) renders the full manual toolbar;
* "prompted" renders the stripped scene-builder layout (textbox +
* add-animation-timeline + history + Update). */
buildMode?: "manual" | "prompted";
/** Called when the user flips the in-bar Manual/Prompted toggle. */
onBuildModeChange?: (mode: "manual" | "prompted") => void;
/** Called when the "Add animation timeline" button is pressed in
* prompted mode. Wired to the timeline feature. */
onAddTimeline?: () => void;
/** When false, the "Add animation timeline" button is hidden (a timeline
* already exists; the host shows the collapsed timeline bar instead).
* Defaults to true. */
showAddTimelineButton?: boolean;
}
export const PromptBox3D = ({
@@ -128,6 +146,10 @@ export const PromptBox3D = ({
modelSelector,
aboveStackSlot,
onBeforeSubmit,
buildMode = "manual",
onBuildModeChange,
onAddTimeline,
showAddTimelineButton = true,
}: PromptBox3DProps) => {
//const fileInputRef = useRef<HTMLInputElement>(null);
const textareaRef = useRef<HTMLTextAreaElement>(null);
@@ -136,6 +158,8 @@ export const PromptBox3D = ({
const prompt = usePrompt3DStore((s) => s.prompt);
const setPrompt = usePrompt3DStore((s) => s.setPrompt);
const promptHistory = usePrompt3DStore((s) => s.promptHistory);
const pushPromptHistory = usePrompt3DStore((s) => s.pushPromptHistory);
const useSystemPrompt = usePrompt3DStore((s) => s.useSystemPrompt);
const resolution = usePrompt3DStore((s) => s.resolution);
const setResolution = usePrompt3DStore((s) => s.setResolution);
@@ -304,6 +328,25 @@ export const PromptBox3D = ({
const maxLen = selectedImageModel?.maxPromptLength ?? 1000;
// Prompted (scene-builder) "Update". Stubbed until the MCP-esque scene
// descriptor backend lands: for now it just records the prompt on the
// engine + in local history so the flow is exercisable. It intentionally
// does NOT enqueue an image generation like the manual "Generate" button.
// TODO(scene-builder): send { sceneDescriptor, prompt } to the backend and
// apply the returned descriptor to the 3D scene.
const handlePromptedUpdate = () => {
if (onBeforeSubmit && !onBeforeSubmit()) return;
const trimmed = prompt.trim();
if (!trimmed) return;
if (isFinite(maxLen) && trimmed.length > maxLen) {
toast.error(`Prompt exceeds the ${maxLen} character limit for this model`);
return;
}
setEnginePrompt(trimmed);
pushPromptHistory(trimmed);
toast.success("Scene update saved — applying edits is coming soon.");
};
const handleEnqueue = async () => {
if (onBeforeSubmit && !onBeforeSubmit()) {
return;
@@ -615,6 +658,37 @@ export const PromptBox3D = ({
</Modal>
<div className="absolute bottom-4 left-1/2 flex w-[90vw] max-w-5xl -translate-x-1/2 flex-col gap-3">
{aboveStackSlot}
{onBuildModeChange && (
<div className="flex">
<div
className="glass glass-no-hover rounded-full p-1 shadow-xl"
onMouseDown={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()}
>
<ButtonIconSelect
options={[
{
value: "manual",
icon: faArrowPointer,
text: "Manual",
tooltip: "Manual — place and transform objects yourself",
},
{
value: "prompted",
icon: faWandMagicSparkles,
text: "Prompted",
tooltip:
"Prompted — describe changes and let AI edit the scene",
},
]}
selectedOption={buildMode}
onOptionChange={(v) =>
onBuildModeChange(v as "manual" | "prompted")
}
/>
</div>
</div>
)}
{selectedImageModel?.canUseImagePrompt && isImageRowVisible && (
<ImagePromptRow
visible={true}
@@ -713,6 +787,7 @@ export const PromptBox3D = ({
</span>
</div>
</div>
{buildMode === "manual" && (
<div className="mt-2 flex items-center justify-between gap-2">
<div className="flex items-center gap-2">
{modelSelector}
@@ -850,6 +925,70 @@ export const PromptBox3D = ({
</GenerateButton>
</div>
</div>
)}
{buildMode === "prompted" && (
<div className="mt-3 flex items-center justify-between gap-2">
{showAddTimelineButton ? (
<Button
variant="secondary"
icon={faPlus}
className="flex h-9 items-center border border-ui-controls-border bg-ui-controls/60 px-3 text-sm text-base-fg backdrop-blur-lg hover:bg-ui-controls/90"
onClick={() => onAddTimeline?.()}
>
Add animation timeline
</Button>
) : (
<span />
)}
<div className="flex items-center gap-2">
<Tooltip
content="Prompt history"
position="top"
className="z-50"
delay={200}
closeOnClick={true}
>
<PopoverMenu
mode="toggle"
panelTitle="Recent prompts"
triggerIcon={
<FontAwesomeIcon
icon={faClockRotateLeft}
className="h-4 w-4"
/>
}
items={
promptHistory.length
? promptHistory.map((p) => ({
label: p,
selected: false,
}))
: [
{
label: "No prompts yet",
selected: false,
disabled: true,
},
]
}
onSelect={(item) => {
if (item.disabled) return;
setPrompt(item.label);
}}
/>
</Tooltip>
<Button
variant="primary"
className="flex items-center border-none bg-brand-primary px-3 text-sm text-white disabled:cursor-not-allowed disabled:opacity-50"
onClick={handlePromptedUpdate}
disabled={!prompt.trim()}
>
Update
</Button>
</div>
</div>
)}
{buildMode === "manual" && (
<div className="absolute -bottom-1 left-1/2 -translate-x-1/2">
<Tooltip
content={isExpanded ? "Collapse" : "Expand"}
@@ -868,6 +1007,7 @@ export const PromptBox3D = ({
</button>
</Tooltip>
</div>
)}
</div>
<CameraSettingsModal
isOpen={isCameraSettingsOpen}
@@ -63,15 +63,20 @@ export const usePrompt2DStore = create<Prompt2DStore>()((set) => ({
export { usePrompt2DStore as usePromptStore };
// ----- 3D Prompt Box Store -----
// Newest first; capped so the history popover stays tidy.
const MAX_PROMPT_HISTORY = 20;
interface Prompt3DStore {
prompt: string;
resolution: Resolution;
useSystemPrompt: boolean;
referenceImages: RefImage[];
promptHistory: string[];
setPrompt: (prompt: string) => void;
setResolution: (resolution: Resolution) => void;
setUseSystemPrompt: (value: boolean) => void;
setReferenceImages: (images: RefImage[]) => void;
pushPromptHistory: (prompt: string) => void;
}
export const usePrompt3DStore = create<Prompt3DStore>()((set) => ({
@@ -79,10 +84,22 @@ export const usePrompt3DStore = create<Prompt3DStore>()((set) => ({
resolution: "1k",
useSystemPrompt: true,
referenceImages: [],
promptHistory: [],
setPrompt: (prompt) => set({ prompt }),
setResolution: (resolution) => set({ resolution }),
setUseSystemPrompt: (useSystemPrompt) => set({ useSystemPrompt }),
setReferenceImages: (referenceImages) => set({ referenceImages }),
pushPromptHistory: (prompt) =>
set((state) => {
const trimmed = prompt.trim();
if (!trimmed) return state;
// De-dupe: drop any existing copy, then unshift so it's newest-first.
const next = [
trimmed,
...state.promptHistory.filter((p) => p !== trimmed),
].slice(0, MAX_PROMPT_HISTORY);
return { promptHistory: next };
}),
}));
// ----- Image Prompt Box Store -----
@@ -18,6 +18,9 @@ interface TabSelectorProps {
tabClassName?: string;
indicatorClassName?: string;
selectedTabClassName?: string;
/** Extra classes for the pill container (the TabList), e.g. to make the
* whole selector fully rounded. */
listClassName?: string;
}
export const TabSelector: React.FC<TabSelectorProps> = ({
@@ -30,6 +33,7 @@ export const TabSelector: React.FC<TabSelectorProps> = ({
tabClassName,
indicatorClassName,
selectedTabClassName,
listClassName,
}) => {
// Find the index of the active tab
const selectedIndex = tabs.findIndex((tab) => tab.id === activeTab);
@@ -66,7 +70,12 @@ export const TabSelector: React.FC<TabSelectorProps> = ({
)}
>
<TabGroup selectedIndex={selectedIndex} onChange={handleTabChange}>
<TabList className="glass glass-no-hover relative inline-flex min-w-fit overflow-x-auto rounded-xl p-0.5 py-1 !shadow-none">
<TabList
className={twMerge(
"glass glass-no-hover relative inline-flex min-w-fit overflow-x-auto rounded-xl p-0.5 py-1 !shadow-none",
listClassName,
)}
>
{/* Animated indicator */}
<div
className={twMerge(