Compare commits

...
Author SHA1 Message Date
Tanish Manku bb7e631834 Audio: consolidate sound system into SoundManager, fix bootstrap + master toggle
- Move sound catalog registration off PageScene mount and into a single
  SoundManager.install() call in GlobalSettingsManager so sounds exist at
  app boot (previously silent until PageScene mounted).
- Honor the play_sounds master toggle in all five event playback sites
  via SoundManager.playEvent helpers. Settings preview keeps firing
  regardless (handled by SoundManager.playPreview).
- Replace per-callsite GetAppPreferences + SoundRegistry boilerplate in
  the four generation event hooks and the file-deleted handler with
  one-line SoundManager.* calls.
- Dedupe SOUND_OPTIONS: AudioSettingsPane now reads SoundManager.OPTIONS,
  derived from a single declarative SOUNDS array (label-bearing entries
  are user-selectable; others are still registered for code-driven use).
- Narrow the soundboard public barrel to SoundManager + SoundOption only;
  SoundRegistry and SoundEffect are now lib-internal.
2026-04-30 09:02:26 -07:00
Tanish Manku 2b5e509a29 PageScene: move engine lifecycle flags + canvas refs into PageSceneStore
The last batch of PageScene-owned signals — `is3DPageMounted`,
`is3DEditorInitialized`, `is3DSceneLoaded`, and the three canvas-ref
signals (sceneContainer, editorCanvas, camViewCanvas) — now live on
the Zustand store. EngineProvider, the canvas components, the
useEditorCanvas hook, and the cross-app TopBar consumer all read from
the store instead. The helper signals in editor.ts are gone; the
`set3DPageMounted` / `setIs3DEditorInitialized` / `setIs3DSceneLoaded`
exports stay as thin wrappers around the store actions for existing
callers.
2026-04-29 15:16:29 -07:00
Tanish Manku 28db87c1b2 PageScene + libs: drop camera/promptBox signals, switch shared libs to plain props
PromptBox3D and CameraSettingsModal now accept plain values + setters
instead of `Signal<T>` props. The camera/focalLengthDragging/
isPromptBoxFocused signals are gone — store is the single source of
truth. Engine code (editor/save_manager/scene) reads/writes via
`usePageSceneStore.getState()` and store actions. Demo asset signals
are now plain constants (they were never reactive).
2026-04-29 15:12:13 -07:00
Tanish Manku 292cf33f57 PageScene: route setCameraAspect + save_manager directly through PageSceneStore 2026-04-29 15:00:48 -07:00
Tanish Manku 58b7cd0cd4 PageScene: route useFreeCam through PageSceneStore for prompt-box focus check 2026-04-29 14:59:54 -07:00
Tanish Manku 101b735c35 PageScene: migrate outliner signals to PageSceneStore 2026-04-29 14:59:06 -07:00
Tanish Manku 32291ce080 PageScene: migrate objectPanel signals to PageSceneStore 2026-04-29 14:56:05 -07:00
Tanish Manku cd9c06d2ce PageScene: migrate precisionSelector signals to PageSceneStore 2026-04-29 14:54:24 -07:00
Tanish Manku a780779f7a PageScene: migrate dragAndDrop signals to PageSceneStore 2026-04-29 14:53:10 -07:00
Tanish Manku 076b7fe4a4 PageScene: extend PageSceneStore with drag/objectPanel/outliner/precisionSelector fields 2026-04-29 14:52:05 -07:00
Tanish Manku 7eef728d22 PageScene: trim api_manager + drop dead model/error files
api_manager.ts goes from 460 lines to ~135. The video-render
pipeline (uploadMediaFrameGeneration, stylizeVideo,
StylizeVideoParams, getMediaBatch, getMediaFile, uploadMedia,
gltfExport) was unreachable after videoGeneration was deleted.
Surviving public methods:

- saveSceneState (called by save_manager)
- loadSceneState (called by save_manager)
- uploadMediaSceneThumbnail (used internally by saveSceneState)

Also drops the local ArtStyle / Visibility / MediaFile types —
they only fed the deleted methods. The ArtStyle enum has a
separate copy in ~/enums/ArtStyle.ts that other code uses.

Delete two files that became unreferenced:
- engine/APIManagerResponseError.ts (duplicate of the local class
  in api_manager.ts; nothing imported it)
- models/generationOptions.ts (IGenerationOptions/GenerationOptions
  fed editor.generation_options, deleted last commit)

Net: -1 typecheck error, -326 lines.
2026-04-29 14:12:03 -07:00
Tanish Manku d09c7320fd PageScene: delete misc dead files surfaced by stylize cleanup
Files with zero importers across the artcraft app:

- constants/RandomTexts.ts — RandomTextsPositive/Negative seeded
  prompt suggestions in the deleted StylizeTab.
- composable_engines/scene_format.tsx — empty (1 line).
- constants/page.ts and the matching currentPage signal in
  signals/global.ts — Pages enum + signal, neither read.
- enums/StylizeTabPages.ts and enums/AudioTabPages.ts — re-exported
  via enums/index.ts but no consumer.
- models/track.ts, models/message.ts, models/voice.ts, models/tts.ts
  — timeline / TTS / audio-panel types from deleted features.
- styleList.ts — used to seed selectedArtStyle's default; that
  store field is gone.
- models/types.tsx had a duplicate AudioTabPages enum (separate
  from the deleted one in enums/) — drop the duplicate too.

models/index.ts and enums/index.ts trimmed to match. Error count
unchanged (151) — none of these had broken type errors anyway.
2026-04-29 14:08:08 -07:00
Tanish Manku 0ca91e29b7 PageScene: drop orphaned stylize state
Strip the stylize fields that nothing reads anymore now that the
StyleMoreOptions / StyleSelection UI is gone. From PageSceneStore:
selectedArtStyle, upscale, faceDetail, styleStrength, lipSync,
cinematic, enginePreProcessing, globalIPAMediaToken, adapterImage,
positivePrompt, negativePrompt, isUserInputPositive,
isUserInputNegative, showNegativePrompt — and the matching setters
+ resetSceneGenerationMetadata. Drop the styleList / ArtStyle
imports; both go unused here.

From editor.ts: drop negative_prompt, art_style,
engine_preprocessing, generation_options, globalIpAdapterImage.
Keep positive_prompt — PromptBox3D writes it via setEnginePrompt
(PageEditor.tsx) and save_manager round-trips it.

Trim save_manager.loadFromJson to the prompt-only path (legacy
saves with stylize keys load fine; the now-removed if-guards just
silently ignored them anyway). Trim SceneMetadata to
{ positivePrompt, cameraAspectRatio } and the matching
SceneGenereationMetaData interface.

ControlsTopButtons.handleResetScene drops its now-removed store
call.
2026-04-29 14:07:03 -07:00
Tanish Manku 8af4657393 PageScene: delete orphan StyleMoreOptions and StyleSelection UI
Both component trees survived earlier cleanup but never render
from PageEditor or PageScene. They were the last UI consumers
of the stylize store fields; deleting them sets up a follow-up
that drops those store fields entirely.

Net: -17 typecheck errors (the orphan files were the source of
many missing-import errors).
2026-04-29 14:04:02 -07:00
Tanish Manku f8ee94e538 Revert moodboard-html files committed by mistake on this branch
cardLayout.ts, MoodboardHtmlColorPicker.tsx, findDropTarget.ts, and
CardNodeHtml.tsx were swept into the PageEnigma -> PageScene rename
commit (76add858c9) on this 3D-refactor branch. They belong on
moodboard-page-html.

The file content is preserved on the local-only moodboard-rescue
branch (pointing at 01e0e27037). To carry them back over:

  git checkout moodboard-page-html
  git checkout moodboard-rescue --     frontend/apps/artcraft/app/src/pages/PageMoodboard/cardLayout.ts     frontend/apps/artcraft/app/src/pages/PageMoodboardHtml/MoodboardHtmlColorPicker.tsx     frontend/apps/artcraft/app/src/pages/PageMoodboardHtml/interactions/findDropTarget.ts     frontend/apps/artcraft/app/src/pages/PageMoodboardHtml/nodes/CardNodeHtml.tsx
  git commit
2026-04-29 12:29:58 -07:00
Tanish Manku 01e0e27037 PageScene: route PoseModeSelector through getActiveEditor()
The Enter Pose Mode button reached through a window.__mouseControls
global handle that MouseControls' constructor set up. With
getActiveEditor() already exposing the live editor instance for
non-React callers (used in the camera-view toggle, the keymap),
the global is unnecessary.

Drop the (window as any).__mouseControls = this assignment in
MouseControls and the matching Window augmentation in
PoseModeSelector. Replace the click handler's reach with
getActiveEditor()?.mouse_controls?.toggleFKMode() — same shape
the keymap entry already uses.
2026-04-29 11:33:30 -07:00
Tanish Manku 794f65b517 PageScene: delete unused models/sidePanel.ts
The Tab interface in this file was the type for the deleted
SidePanel/tabList.tsx tab list. Grep across app/src/ confirms no
consumer remains; AssetModal uses its own local AssetTab type.
Drop the file and the matching re-export in models/index.ts.
2026-04-29 11:32:48 -07:00
Tanish Manku c53b407952 PageScene: drop unused imports surfaced by deletions
EditorStates is no longer referenced from editor.ts now that
PREVIEW is gone from the enum and switchPreview/switchEdit are
deleted. EngineCanvases.tsx never used useContext/EngineContext;
the imports were leftover from earlier work.

Net: -3 typecheck errors.
2026-04-29 10:03:23 -07:00
Tanish Manku e0611e0147 PageScene: drop document.mouseover gate + dead selectedCanvas
The mouseOverEventHandler / enableFreeCamControls /
disableFreeCamControls trio existed to track when the cursor was on
the viewport canvas and toggle FreeCam.enabled. With useFreeCam
gating itself on editorState === CAMERA_VIEW, that gate is
redundant; the only remaining consumer of selectedCanvas was
MouseControls.onkeydown's space-bar branch, which is unreachable
after the previous keyboard rewrite.

Removes the last document.addEventListener call from PageScene.
2026-04-29 10:02:09 -07:00
Tanish Manku 8a03449128 PageScene: keymap + useViewportKeyboard hook
Replace MouseControls.onkeydown's giant if-else chain with a
declarative keymap:

- engine/keymap.ts: KeyBinding type + buildKeymap() returning every
  one-shot viewport shortcut as one entry. Each carries label /
  group metadata so a future Ctrl-hold cheatsheet overlay can read
  the same array. dispatchBinding() does the modifier match
  (treating Ctrl and Meta as equivalent for macOS).
- hooks/useViewportKeyboard.ts: attaches a single keydown listener
  that gates on hotkeyStatus + editable-element checks, then walks
  the keymap. Held movement keys (W/A/S/D etc.) live in useFreeCam
  since they're continuous; this hook is for one-shots only.

The listener is on document, not the canvas, deliberately: canvas
focus is finicky and would require the user to click the viewport
before T/R/G work. The editable-element guard prevents the keymap
from firing while inputs are focused.

MouseControls.onkeydown is now unreachable (SceneManager's window
listeners are no-op since the previous commit, and useViewportKeyboard
goes straight to keymap dispatch). The class itself stays for now —
it still implements selection / FK / pointer-lock state.
2026-04-29 10:00:56 -07:00
Tanish Manku e4af73c9ac PageScene: useViewportPointer hook (canvas-scoped)
The old SceneManager.attachEventListeners() attached window-level
mousemove / click / mousedown / mouseup / keydown handlers in dev
mode and dispatched them into MouseControls. Replace with two pieces:

- hooks/useViewportPointer.ts: canvas-scoped pointer listeners
  that call into editor.mouse_controls. NDC pointer coords are
  computed via the new engine/pointer.ts helpers (ndcFromClient,
  applyNdcToVector2). Selection clicks suppress while the
  free-cam owns input (CAMERA_VIEW state).
- engine/pointer.ts: pure NDC coordinate translation, no DOM
  side-effects.

SceneManager.attachEventListeners / detachEventListeners are now
no-ops; kept for back-compat with the editor lifecycle so future
follow-ups can carve them out cleanly.

MouseControls itself still owns selection / FK / drag state — its
dispatch methods (onMouseMove, onMouseClick, etc.) are now invoked
by the React hook instead of window listeners. A future PR can
dissolve MouseControls further; the goal here is "no window
listeners" + "React owns lifecycle".
2026-04-29 09:58:24 -07:00
Tanish Manku efa9a54bee PageScene: useFreeCam hook + delete free_cam.ts
The 446-line FreeCam class is gone. Its responsibilities split:

- Math (move/rotate vectors, pan/zoom from inputs, lookAt) lives in
  cameraMath.ts as pure functions, plus a freeCamFrameTick() that
  integrates the per-frame state.
- Listener attachment + drag/key state ownership moves into
  hooks/useFreeCam.ts. Listeners are scoped to the canvas (no
  window/document handlers); held-key state is reset on
  CAMERA_VIEW exit so a stale W keypress can't drift the camera.
- The shared FreeCamControlState struct flows through Editor via
  editor.setFreeCamState(state); the render loop reads it and
  calls freeCamFrameTick() each frame.

Side effects: keybinds_controls.ts now imports
isPointerLockSupported from browserChecks; helper.ts and
PageEditor's camera-select handler stop poking
.cameraViewControls and reset velocity via freeCamState instead.
PreviewEngineCamera's camera-view toggle now just flips the store
state — useFreeCam picks it up on its own (no more
enableFreeCamControls/disableFreeCamControls call site).
2026-04-29 09:55:40 -07:00
Tanish Manku 718b8261fd PageScene: extract cameraMath + browserChecks modules
Pure-function helpers carved out of FreeCam in preparation for the
hook rewrite:

- engine/cameraMath.ts: HeldMoveKeys / HeldRotateKeys types,
  moveVectorFromKeys, rotationVectorFromKeys, panFromDrag,
  zoomFromWheel, lookAtFromCamera, lerpVelocity. No DOM access, no
  signals.
- engine/browserChecks.ts: isSafariOrWebKit, hasApplePay, isSafari,
  isPointerLockSupported. Unrelated to camera math; were just
  hitchhiking in free_cam.ts.

free_cam.ts re-exports the browser checks for back-compat with
keybinds_controls.ts; both consumers will switch to the direct
imports when free_cam.ts is deleted in the next commit.
2026-04-29 09:50:16 -07:00
Tanish Manku 61c4936c29 PageScene: useEditorCanvas hook + tabIndex on viewport canvas
The Phase 6 hooks (useFreeCam, useViewportPointer, useViewportKeyboard)
need a React-shaped accessor for the canvas DOM node so their
useEffects re-run when the canvas mounts/unmounts. Promote the
existing editorCanvasSignal / camViewCanvasSignal into useEditorCanvas
/ useCamViewCanvas hooks that bridge signal -> React state via a
subscribe.

Add tabIndex={0} and outline:none to the editor canvas so it can
receive keyboard focus without showing a focus ring.
2026-04-29 09:48:59 -07:00
Tanish Manku c23e652499 PageScene: wire camera-view toggle button
handleButtonCameraView in PreviewEngineCamera was a TODO no-op.
Flip editorState (EDIT <-> CAMERA_VIEW) via the store, then call
editor.enableFreeCamControls() / disableFreeCamControls() on the
active editor. Drop the stale console.log useEffect that fired on
every state change.
2026-04-29 09:48:02 -07:00
Tanish Manku a26ebce483 PageScene: delete video generation entirely
The 3D editor no longer generates videos. Remove:

- engine/video_generation.ts (the VideoGeneration class)
- comps/GenerateModals/ (modal + MyMovies/Sharing/Activity/cards)
- comps/PreviewFrameImage.tsx (PREVIEW-state-only preview overlay)
- comps/ControlsTopButtons/TestFeaturesButtons.tsx (dev-only,
  rendered with debug={false})
- ~/components/signaled/TopBar/MyMoviesButton.tsx (was orphaned;
  no consumer)

editor.ts: drop the VideoGeneration import, field, instantiation,
and the dead helpers it powered (useCachedMediaTokens,
stopPlaybackAndUploadVideo, switchPreview/switchEdit, generateFrame,
generateVideo, checkAndUseCache).

PageSceneStore: drop previewSrc/viewMyMovies/generateMovieId/
generateProgress fields and setters.

EditorStates: drop the PREVIEW variant; PageEditor's state machine is
now EDIT <-> CAMERA_VIEW only. PREVIEW was set only by the deleted
video_generation.ts and read by PreviewFrameImage (deleted) and a few
gates in PreviewEngineCamera + ControlPanelSceneObject (now removed).

Net: -8 typecheck errors.
2026-04-29 09:47:14 -07:00
Tanish Manku 1d3da065a2 PageScene: drop dead side-panel store fields
sidePanelWidth, sidePanelVisible, dndSidePanelWidth and their setters
are no longer read by any rendered component. Drop them from
PageSceneStore and the lone setSidePanelWidth(340) call in
PageEditor's mount effect. PreviewFrameImage's width calc no longer
subtracts the (always-zero) panel width.
2026-04-29 09:42:38 -07:00
Tanish Manku b4a3eb88bf PageScene: delete dead side-panel tree
The <SidePanel /> component was never rendered from PageEditor.tsx; the
entire comps/SidePanel/, comps/SidePanelMenu/, and most of comps/SidePanelTabs/
tree was unreachable. Relocate the few files still consumed by
AssetModal (sharedComps/ItemElement{,s}.tsx, hooks/useUserObjects,
hooks/useFeaturedObjects, utilities/fetchMediaItems + misc) to
comps/AssetMenu/{shared,hooks,utilities}/, and the StyleOptions panel
used by StyleMoreOptions to its parent folder. Then delete the rest.

Net: -33 typecheck errors (the dead tree had its own type problems).
2026-04-29 09:41:21 -07:00
Tanish Manku 48ab5f73ee Phase 7: rename Editor/ -> engine/ and drop dead useSignals() calls
Editor/ -> engine/:
- Rename pages/PageScene/Editor/ -> pages/PageScene/engine/
- Update every "./Editor/...", "../Editor/...", "../../Editor/...",
  "~/pages/PageScene/Editor/..." import path. The class name `Editor`,
  `EditorState` types, and `editorState` field are unchanged -- only
  the directory is renamed
- ImageTo3DExperience also gets the new path

useSignals() drops where the component reads zero .value signals:
- comps/GenerateModals/Activity.tsx
- comps/SidePanelTabs/hooks/useFeatureFlags.tsx
- comps/SidePanelTabs/sharedComps/ItemElement.tsx
- comps/SidePanelTabs/tabComps/StylizeTab/PageStyleSelection.tsx
- comps/StyleMoreOptions/StyleMoreOptions.tsx

The remaining useSignals() callers still subscribe to genuine signals
(pageHeight/pageWidth from ~/signals, outlinerIsShowing,
sceneContainerSignal, etc.) and need to stay until those signals migrate.

Final grep audit confirms no residue:
- "PageEnigma" -- 0 hits
- "from \"...Editor/...\"" import paths -- 0 hits
- Queue.publish / Queue.subscribe / fromEngineActions /
  toEngineActions / toTimelineActions -- 0 hits
- TimeLine, AudioEngine, TransformEngine, EmotionEngine, LipSyncEngine,
  CharacterAnimationEngine, ClipUI, TimelineState -- 0 hits

Typecheck unchanged at 144 errors in PageScene; zero new errors.
2026-04-28 23:47:34 -07:00
Tanish Manku 76add858c9 PageEnigma -> PageScene: rename to match the route param and the page title
The directory was named after a stale codename. The route is
/$sceneToken, the top bar reads "Edit Scene", and the page is the main
scene-editing workspace. Renaming the directory, top-level component,
and Zustand store accordingly:

- pages/PageEnigma/ -> pages/PageScene/
- pages/PageScene/PageEnigma.tsx -> PageScene.tsx
- pages/PageScene/PageEnigmaStore.ts -> PageSceneStore.ts
- usePageEnigmaStore -> usePageSceneStore (function rename across the
  whole codebase via sed)
- All ~/pages/PageEnigma/* import paths -> ~/pages/PageScene/*
- Component name PageEnigma -> PageScene in routes/_index.tsx and
  routes/$sceneToken.tsx

234 files touched, all mechanical. Typecheck error count holds at 144;
zero new errors introduced by the rename.
2026-04-28 23:44:26 -07:00
Tanish Manku e39e67195e PageEnigma refactor Phase 5g: migrate stylizeTab signal slice to the store
stylizeTab.ts (deleted). 11 consumers migrated:

UI components:
- StyleOptions: cinematic / upscale / faceDetail / lipSync /
  enginePreProcessing toggles drive the matching store setters; the
  upscale<->cinematic mutual-exclusion logic preserved
- StyleStrength: slider + number input read styleStrength from store,
  call setStyleStrength
- Prompts: textBufferPositive / textBufferNegative now positivePrompt /
  negativePrompt on the store; isUserInputPositive /
  isUserInputNegative / showNegativePrompt as fields with setters
- IPAdapter: adapterImage + globalIPAMediaToken from store; useShallow
  subscribes to all four needed fields/setters in one selector
- StylizeTab: selectedArtStyle from store, setSelectedArtStyle replaces
  the old setArtStyleSelection helper
- StyleSelection: same pattern -- selectedArtStyle and
  setSelectedArtStyle
- GenerateMovieButton: snapshots the entire stylize bag from the store
  with usePageEnigmaStore.getState() inside the click handler

Engine + persistence:
- SceneMetadata.getSceneGenerationMetaData reads cameraAspectRatio,
  globalIPAMediaToken, upscale, faceDetail, styleStrength, lipSync,
  cinematic, enginePreProcessing all from the store in one getState()
- save_manager.loadFromJson restores all stylize state through the
  store's setters instead of the old restoreSceneGenerationMetadata
  signal helper. globalIpAdapterImageMediaToken restoration also goes
  through setGlobalIPAMediaToken
- video_generation: setGlobalIPAMediaToken on upload + reads
  globalIPAMediaToken from store for the gen request payload
- ControlsTopButtons: new-scene flow calls
  usePageEnigmaStore.getState().resetSceneGenerationMetadata()

Bonus cleanup:
- SidePanelTabs reverted to inlining pageHeight - 64 (the previous
  edit got lost in a stash round-trip)
- IPAdapter dropped useSignals and uses useShallow for its 4-field
  selector

Net effect on typecheck: 145 -> 144 errors in PageEnigma.
2026-04-28 23:28:55 -07:00
Tanish Manku ea49ddf60a PageEnigma refactor Phase 5f: migrate sizing slice to the store
sizing.ts (deleted). Was already minimal -- just sidePanelWidth, the
last survivor after the timeline-related sizing exports were stripped
in Phase 2. Three consumers migrated:

- PageEditor's mount-time setup: store.setSidePanelWidth(340)
- PreviewFrameImage: pulls sidePanelWidth off the store alongside
  sidePanelVisible
- SidePanel: same pattern (already reading sidePanelVisible /
  dndSidePanelWidth from the store; sidePanelWidth joins them)
- StyleSelection: subscribes to sidePanelWidth via the store

signals/ down to 10 files: camera, demoAssets, dragAndDrop, engine,
objectPanel, outliner, precisionSelectorMenu, promptBox, stylizeTab.
2026-04-28 23:22:57 -07:00
Tanish Manku e0c04efa8f PageEnigma refactor Phase 5e: migrate selectedMode + sidePanel slices
selectedMode.ts (deleted):
- PoseModeSelector reads poseMode + showPoseControls from store, calls
  setPoseMode to toggle
- keybinds_controls migrates the t/r/g hotkeys (mode change),
  x (transform space toggle), 'Escape' (exits pose mode), object
  selection (sets showPoseControls), and FK toggle to the store
- editor.change_mode + toggleTransformSpace read/write transformSpace
  through usePageEnigmaStore.getState()
- Controls3D's mode dropdown + world/local pill read selectedMode and
  transformSpace from the store
- actions/setTransformMode no longer dual-writes the legacy signal
- store's PoseMode union changed from "select" | "edit" to
  "select" | "pose" to match what the engine actually checks

sidePanel.ts (deleted):
- SidePanel reads sidePanelVisible + dndSidePanelWidth from store
- TabTitle's close button calls setSidePanelVisible
- SidePanelMenu pulls sidePanelVisible + setSidePanelVisible
- PreviewFrameImage reads sidePanelVisible (was a signal, now store)
- sidePanelHeight (was a computed signal of pageHeight - 64) inlined
  at its three call sites: PreviewEngineCamera, SidePanelTabs,
  StyleSelection. cameraFilter has no consumers; it lives on the store
  for the next round.

signals/ down to 11 leaves: camera, demoAssets, dragAndDrop, engine,
objectPanel, outliner, precisionSelectorMenu, promptBox, sizing,
stylizeTab.
2026-04-28 23:21:13 -07:00
Tanish Manku dc140c0bc1 PageEnigma refactor Phase 5d: migrate hotkeys + selectedItem signal slices
hotkeys.ts (deleted):
- keybinds_controls reads usePageEnigmaStore.getState().hotkeyStatus.disabled
- ControlPanelSceneObject pulls disableHotkeyInput / enableHotkeyInput
  off the store and passes them to InputVector
- PageEditor pulls the same and passes them to LoginModal +
  PromptBox3D. DomLevels imported from PageEnigmaStore (replacing the
  signals/hotkeys re-export)

selectedItem.ts (deleted):
- No external consumers (dead since the timeline removal).
  selectedObject lives on the store; the legacy signal had no readers
  outside its own file.

signals/index.ts no longer re-exports either slice. Down to 13 files
in signals/.
2026-04-28 23:15:45 -07:00
Tanish Manku 21d3617970 PageEnigma refactor Phase 5c: migrate assetModal signal slice to the store
assetModal.ts (deleted):
- AssetMenu: setAssetModalVisibleDuringDrag(true) on open
- AssetModal: reads assetModalVisible / assetModalVisibleDuringDrag /
  reopenAfterDrag from the store, writes via the matching setters.
  Modal close drives setAssetModalVisible(false); the reopen-after-add
  checkbox writes through setReopenAfterDrag (note: store field is
  reopenAfterDrag, not the old reopenAfterDragSignal name)
- Controls3D: handleOpenModal flips both setters from the store; the
  empty-scene tooltip useEffect reads assetModalVisible from the store
- keybinds_controls: 'b' shortcut opens the asset modal via the store
- DragAndDrop/DndAsset: pointerdown clears
  assetModalVisibleDuringDrag, pointerup restores it from
  reopenAfterDrag, click (no drag) restores it to true. All through
  the store now.

signals/index.ts no longer re-exports assetModal. Four signal files
gone so far in Phase 5; signals/ now down to 13 leaves (camera,
demoAssets, dragAndDrop, engine, hotkeys, objectPanel, outliner,
precisionSelectorMenu, promptBox, selectedItem, selectedMode,
sidePanel, sizing, stylizeTab).
2026-04-28 23:13:57 -07:00
Tanish Manku 11a0ac82b6 PageEnigma refactor Phase 5b: migrate error, myMovies, editorOverlays signal slices
Three more signal files retired in favour of the Zustand store.

error.ts (deleted):
- ErrorDialog (in ~/components/signaled): reads showErrorDialog /
  errorDialogTitle / errorDialogMessage from the store, calls
  setShowErrorDialog
- SceneTitleInput (in TopBar): handleShowErrorDialog uses the store's
  setErrorDialog action

myMovies.ts (deleted):
- MyMoviesButton (in TopBar): writes via setGenerateMovieId /
  setViewMyMovies
- GenerateModals: useEffect-on-store replaces useSignalEffect; reads
  generateMovieId from store
- MyMovies and Sharing: read viewMyMovies from store, close via
  setViewMyMovies
- Sharing also reads generateMovieId from store

editorOverlays.ts (deleted):
- PageEditor's LoadingDots: reads editorLoader from store
- SceneContainer: reads editorLetterBox from store and passes through
  to Letterbox
- PreviewEngineCamera: reads editorLetterBox + toggleEditorLetterBox
  from store
- save_manager: showEditorLoader/hideEditorLoader become thin wrappers
  over the store actions, keeping the same call sites

signals/index.ts no longer re-exports any of the three deleted files.
2026-04-28 23:11:52 -07:00
Tanish Manku 4b0618e50b PageEnigma refactor Phase 5a: migrate engine.ts signal slice to the store
Migrates components and engine code from reading signals/engine.ts
(cameraAspectRatio, editorState, gridVisibility, previewSrc) to reading
the Zustand store.

Components migrated to usePageEnigmaStore reads:
- AspectRatioMenu (also dropped useSignals + collapsed two
  duplicated lookup ladders into ICON_BY_RATIO / LABEL_BY_RATIO)
- PreviewFrameImage
- Letterbox (no longer reactive via signals; reads camAspect from store)
- StyleButtons (editorState, previewSrc, setEditorState)
- Outliner (cameraAspectRatio)
- SidePanelMenu (editorState + setEditorState)
- ControlPanelSceneObject (editorState)
- PreviewEngineCamera (cameraAspectRatio + editorState)
- PageEditor (cameraAspectRatio for getScale; the signal version still
  flows into PromptBox3D as a prop)

Engine code migrated to usePageEnigmaStore.getState():
- editor.ts: snapShotOfCurrentFrame reads cameraAspectRatio + grid;
  switchPreview / switchEdit set editorState
- helper.ts: switchCameraView writes editorState
- video_generation.ts: generateFrame writes previewSrc + editorState
- SceneMetadata.ts: cameraAspectRatio for snapshot metadata
- scene.ts: subscribes to store gridVisible (not the signal)

signals/engine.ts becomes a thin compatibility layer for PromptBox3D's
signal-typed props. Its setters dual-write into the store so any
PromptBox3D-driven changes (grid toggle, aspect ratio dropdown) keep
the store authoritative. editorState / previewSrc / setEditorState
removed entirely -- nothing outside the store needs them anymore.

Net effect on typecheck: 148 -> 147 errors in PageEnigma. Zero new
errors introduced.
2026-04-28 23:05:54 -07:00
Tanish Manku 35aae54d49 fix: let UploadSuccess actually render in upload modals
The auto-success useEffect in UploadModalSplat / UploadModalImage /
UploadModal3D was firing on UploaderStates.success and immediately
calling onClose(), which short-circuited the UploaderModalContent
switch before the UploaderStates.success branch (UploadSuccess) could
render. The OK button on UploadSuccess was effectively dead code.

Move the success-side actions (open the gallery, fire onSuccess, close
the modal) into the UploadSuccess onOk handler. The user now sees the
"upload successful" dialog and can dismiss it intentionally; the
gallery opens at dismissal so they can drag the freshly uploaded asset
onto the canvas.
2026-04-28 22:56:38 -07:00
Tanish Manku 088edd270f UploadModalSplat: open gallery on success to match image / 3D-model upload
Pre-refactor the splat upload modal had two paths: the (developer-only)
local-bytes path that handed the in-memory buffer to a now-deleted
editor.timeline.addLocalSplat, and a real upload path that registered
the file in Storyteller's media library but did nothing else with it.

The local-bytes branch was the only thing letting a freshly uploaded
splat reach the canvas. With it gone, splat upload was a dead end --
the file landed in the user's media library but they had no path back
to the 3D editor.

Align with the existing image / 3D-model upload convention: on
UploaderStates.success, set galleryModalVisibleViewMode and call
onSuccess(selectedCategory). The user can then drag the freshly
uploaded splat from the auto-opened gallery onto the canvas, which
flows through PageEditor's onImageDrop handler and into
actions/addObject.

Other changes:
- onSuccess signature: (ArrayBuffer, boolean) => (FilterEngineCategories)
  to match UploadModalImage / UploadModal3D
- Drop the unused onLocalBytes prop and plumbing in UploadFilesSplat;
  no callsite invoked it. The two existing consumers
  (PageEnigma/Controls3D, GlobalFileDropHandler) ignore the category
  argument, which is fine -- () => void is assignable.
2026-04-28 22:50:17 -07:00
Tanish Manku 61311028e2 PageEnigma refactor Phase 4: build actions/ and rewire add/select/transform flows
Replaces the four-hop signal-stub pattern (drop -> store-only update,
engine never told) with explicit action helpers that mutate the engine
and the store in one step.

actions/:
- addObject / addCharacter / addShape: call editor.sceneManager.create
  to load the THREE.js object, then update the Zustand store. addCharacter
  also stamps userData.isCharacter on the loaded object
- deleteObject: thin wrapper over editor.deleteObject (which already
  goes through SceneUtils to update the store)
- selectObject / deselectObject: invoke sceneManager.select_object;
  Editor.publishSelect already writes the selection to the store
- setTransformMode: drives editor.change_mode + store + the legacy
  selectedMode signal during the migration window
- setCameraAspect: drives editor.changeRenderCameraAspectRatio + store +
  the legacy cameraAspectRatio signal
- index.ts barrel

Wiring updates:
- PageEditor drop handler (gallery -> 3D) now calls actions/addObject
  and actions/addCharacter with the editor and world position
- DragAndDrop/DndAsset takes the editor at pointerdown time and calls
  actions/* on drop. ItemElement passes the editor in via EngineContext
- AspectRatioMenu calls actions/setCameraAspect; PageEditor's bottom-bar
  aspect-ratio select uses the action too
- Controls3D's mode toggle (move/rotate/scale) goes through
  actions/setTransformMode and the four little redundant handlers are
  collapsed into one
- ImageTo3DExperience's "Open in 3D Editor" button uses getActiveEditor
  to pick up the live editor across pages

Plumbing:
- New module-level activeEditor handle in EngineContext (with
  getActiveEditor / setActiveEditor) so non-React callers and cross-page
  actions can reach the engine; EngineProvider sets it on init and
  clears on unmount
- Drop the now-orphaned signals/objectGroup, signals/characterGroups,
  signals/shape entirely. Their job (store update + engine call) lives
  in actions/. signals/index.ts loses the corresponding re-exports
- Remove the placeholder addLocalSplat onSuccess in Controls3D (a
  developer-test path for direct splat loading; the real splat upload
  goes through GlobalFileDropHandler) and delete the now-confirmed-dead
  recordScene method in editor.ts
- video_generation.ts zipBlob: type as Blob | null (matches the
  UploadStudioShot signature; pre-existing TS error)

Net effect on typecheck: 144 -> 147 errors in PageEnigma + experiences;
zero errors introduced in actions/, PageEnigmaStore, or DragAndDrop. All
new errors are pre-existing issues in adjacent files surfaced by the
edits, not regressions.
2026-04-28 22:37:30 -07:00
Tanish Manku 6d22d0d3c3 PageEnigma refactor: strip Queue, TimeLine, and timeline engines from editor.ts
editor.ts surgery (1864 -> 1665 lines):

- Drop imports + field declarations + constructor instantiations for
  AudioEngine, TransformEngine, EmotionEngine, LipSyncEngine,
  CharacterAnimationEngine, and TimeLine; drop the
  Queue/QueueNames/fromEngineActions/toEngineActions imports
- Drop timeline state fields: can_playback, playback_location,
  current_frame, last_scrub. generating_preview stays (it tracks the
  stylize preview render, not the timeline)
- Replace Queue.publish in changeRenderCameraAspectRatio, newScene,
  loadScene, and publishSelect with direct store actions
  (usePageEnigmaStore.getState().setSelectedObject / resetScene)
- Replace timeline.characters lookups (used by SceneManager outliner)
  with a getCharactersByUuid helper that reads the store
- Replace timeline.isCharacter (passed into SceneManager) with an
  isCharacterUuid helper that checks the store's character list
- recordScene becomes a no-op stub: timeline-driven frame capture is
  gone, video generation needs to drive its own frame loop. TODO marker
  for the rewiring
- generateVideo and togglePlayback drop their timeline.is_playing /
  scrubber gating; renderSingleFrame drops the timeline.update branch
- Bonus: kill the meraAspectRatio typo (a pre-existing bug in the
  recordScene branch I deleted)

Cascading fixes:

- save_manager.ts loadScene/loadCache: drop calls to the deleted
  timeline.scrub and timeline.checkEditorCanPlay
- video_generation.ts videoProcessingPipeline: stub the
  engineFrameBuffers.payloadZip path with a TODO; the video upload now
  always errors via the existing handleError. The full rewiring is the
  same TODO as recordScene
- Controls3D UploadModalSplat onSuccess: drop the
  editor.timeline.addLocalSplat call
- SidePanelTabs: drop a long-broken import of useMouseEventsSidePanel
  (the Timeline/utils path it pointed to was already deleted before
  this refactor); resize handle gets a TODO marker for Phase 6

Net effect on typecheck: 162 -> 144 errors in PageEnigma. Every
remaining error in PageEnigma predates this branch; no new TypeScript
errors introduced by the refactor.
2026-04-28 22:23:11 -07:00
Tanish Manku 50e68b0721 PageEnigma refactor: strip Queue and timeline-only signals from comps and engine helpers
Removes references to the deleted Queue/timeline subsystem from:

- Engine helpers: scene_manager_api, video_generation, helper. Their
  Queue.publish calls (UPDATE_CHARACTER_ID/ADD_OBJECT/POP_A_TOAST/
  DELETE_OBJECT) now write to the new Zustand store / addToast directly.
- Editor/EditorState: drop TimelineState class and its JSON variant; trim
  EditorState/EditorStateJson down to just sceneState and metadata.
- Editor/scene.ts: drop the timeline animation-clip seeding path; restore
  placeholder_manager (incorrectly classified as dead in the audit).
- Signals: collapse signals/objectGroup, signals/characterGroups, and
  signals/shape to non-timeline scaffolds that update the store directly
  with a TODO marker for the engine-side rewiring. Drop
  loadObjectData/loadCharacterData/updateCharacters/updateObjectGroup
  files (all timeline-only). Strip dndTimelineHeight + overTimeline from
  signals/dragAndDrop. Add setEditorState helper to signals/engine.
- Comps: drop Queue.publish calls. Aspect-ratio menu and stylize-tab
  state transitions go through signal setters; TOGGLE_CAMERA_STATE,
  REFRESH_PREVIEW, GENERATE_VIDEO are stubbed with TODOs pending the
  editor.ts rewiring. DragComponent simplified (timeline-drop branch was
  unreachable). PageEditor drops timelineHeight/dndTimelineHeight layout
  math.
- PageEnigma drops the useQueueHandler call.

The branch still does not compile: editor.ts (1864 lines) is the next
target -- it instantiates TimeLine + the five timeline engines and
publishes select/aspect-ratio events through the queue. Cleaning it up
is the focused work for the next commit.
2026-04-28 22:07:31 -07:00
Tanish Manku 1351a40b3d PageEnigma refactor checkpoint: delete timeline subsystem and add Zustand store
Phase 1-3 of the 3D page refactor:
- Delete the timeline + event-queue subsystem (Queue/, useQueueHandler,
  Editor/timeline.ts, Editor/Engines/, clips/, and the timeline-only
  signal slices: timeline, timing, audioGroup, cameraGroup,
  promptTravelGroup, utils/publishClip)
- Delete dead-code files with no live importers: lipsync.js,
  retargeting.ts, preview_video_generation.ts, shaders.js,
  placeholder_manager.ts, both composable_engines/*.tsx stubs,
  contexts/TrackContext/, comps/TimerGrid, comps/ControlsVideo
- Add PageEnigmaStore.ts with the full state shape that will replace
  preact-signals (scene objects, cameras, transform mode, layout, error
  overlays, generation/stylize state)
- Strip signals/index.ts of deleted slice exports and reduce
  signals/sizing.ts to just sidePanelWidth

Branch does not yet compile -- subsequent commits will strip
Queue/timeline references from editor.ts, the remaining signal files,
and component consumers, then build out actions/ and event hooks.
2026-04-28 21:56:58 -07:00
341 changed files with 2733 additions and 14920 deletions
@@ -343,10 +343,10 @@
"markdownDescription": "Denies the fetch_send command without any pre-configured scope."
},
{
"description": "This permission set configures what kind of\nfetch operations are available from the http plugin.\n\nThis enables all fetch operations but does not\nallow explicitly any origins to be fetched. This needs to\nbe manually configured before usage.\n\n#### Granted Permissions\n\nAll fetch operations are enabled.\n\n\n#### This default permission set includes:\n\n- `allow-fetch`\n- `allow-fetch-cancel`\n- `allow-fetch-read-body`\n- `allow-fetch-send`",
"description": "This permission set configures what kind of\r\nfetch operations are available from the http plugin.\r\n\r\nThis enables all fetch operations but does not\r\nallow explicitly any origins to be fetched. This needs to\r\nbe manually configured before usage.\r\n\r\n#### Granted Permissions\r\n\r\nAll fetch operations are enabled.\r\n\r\n\n#### This default permission set includes:\n\n- `allow-fetch`\n- `allow-fetch-cancel`\n- `allow-fetch-read-body`\n- `allow-fetch-send`",
"type": "string",
"const": "default",
"markdownDescription": "This permission set configures what kind of\nfetch operations are available from the http plugin.\n\nThis enables all fetch operations but does not\nallow explicitly any origins to be fetched. This needs to\nbe manually configured before usage.\n\n#### Granted Permissions\n\nAll fetch operations are enabled.\n\n\n#### This default permission set includes:\n\n- `allow-fetch`\n- `allow-fetch-cancel`\n- `allow-fetch-read-body`\n- `allow-fetch-send`"
"markdownDescription": "This permission set configures what kind of\r\nfetch operations are available from the http plugin.\r\n\r\nThis enables all fetch operations but does not\r\nallow explicitly any origins to be fetched. This needs to\r\nbe manually configured before usage.\r\n\r\n#### Granted Permissions\r\n\r\nAll fetch operations are enabled.\r\n\r\n\n#### This default permission set includes:\n\n- `allow-fetch`\n- `allow-fetch-cancel`\n- `allow-fetch-read-body`\n- `allow-fetch-send`"
}
]
}
@@ -1,6 +1,6 @@
import { ApiManager, ApiResponse } from "./ApiManager";
import { BetaKey } from "~/models";
import { Pagination } from "~/pages/PageEnigma/models";
import { Pagination } from "~/pages/PageScene/models";
export class BetaKeysApi extends ApiManager {
public CreateBetaKey({
@@ -3,7 +3,7 @@ import {
MediaInfo,
Pagination,
PaginationInfinite,
} from "~/pages/PageEnigma/models";
} from "~/pages/PageScene/models";
import { ApiManager, ApiResponse } from "./ApiManager";
import { authentication } from "~/signals";
import type { Property } from "csstype";
@@ -1,5 +1,5 @@
import { ApiManager, ApiResponse } from "./ApiManager";
import { Prompts } from "~/pages/PageEnigma/models";
import { Prompts } from "~/pages/PageScene/models";
export class PromptsApi extends ApiManager {
public GetPromptsByToken({
@@ -1,5 +1,5 @@
import { ApiManager, ApiResponse } from "./ApiManager";
import { GenerateTtsAudioErrorType } from "~/pages/PageEnigma/enums";
import { GenerateTtsAudioErrorType } from "~/pages/PageScene/enums";
import { Visibility } from "~/enums";
export interface GenerateTtsAudioRequest {
@@ -4,7 +4,7 @@ import {
UserBookmarkBatch,
UserBookmarkByEntity,
UserBookmarkByUser,
} from "~/pages/PageEnigma/models";
} from "~/pages/PageScene/models";
import { authentication } from "~/signals";
export enum ScopedEntityTypes {
@@ -1,5 +1,5 @@
import { ApiManager, ApiResponse } from "./ApiManager";
import { Pagination } from "~/pages/PageEnigma/models";
import { Pagination } from "~/pages/PageScene/models";
import { authentication } from "~/signals";
import { ZsDataset } from "~/models/Dataset";
@@ -1,7 +1,7 @@
import { ApiManager, ApiResponse } from "./ApiManager";
import { authentication } from "~/signals";
import { Weight } from "~/models";
import { Pagination, PaginationInfinite } from "~/pages/PageEnigma/models";
import { Pagination, PaginationInfinite } from "~/pages/PageScene/models";
import {
FilterEngineCategories,
FilterMediaClasses,
@@ -25,10 +25,11 @@ import {
import { toast } from "react-hot-toast";
import { v4 as uuidv4 } from "uuid";
import { useTabStore } from "../../pages/Stores/TabState";
import { addObject } from "../../pages/PageEnigma/signals/objectGroup/addObject";
import { set3DPageMounted } from "../../pages/PageEnigma/Editor/editor";
import { addObject } from "../../pages/PageScene/actions";
import { getActiveEditor } from "../../pages/PageScene/contexts/EngineContext";
import { set3DPageMounted } from "../../pages/PageScene/engine/editor";
import { AssetType } from "~/enums";
import type { MediaItem } from "../../pages/PageEnigma/models";
import type { MediaItem } from "../../pages/PageScene/models";
import { SPLAT_MODELS } from "@storyteller/model-list";
import {
ClassyModelSelector,
@@ -989,8 +990,13 @@ export const ImageTo3DExperience = ({
} as MediaItem & {
position: { x: number; y: number; z: number };
};
addObject(mediaItem);
toast.success("Added to 3D scene");
const editor = getActiveEditor();
if (editor) {
void addObject(editor, mediaItem);
toast.success("Added to 3D scene");
} else {
toast.error("3D editor isn't ready yet");
}
}, 500);
}}
>
@@ -73,17 +73,6 @@ export function UploadModal3D(props: Props) {
}
}, [isOpen]);
useEffect(() => {
if (uploaderState.status === UploaderStates.success) {
// Automatically open the global Gallery modal after a successful upload
galleryModalVisibleViewMode.value = true;
galleryModalVisibleDuringDrag.value = true;
onSuccess(selectedCategory);
onClose();
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [uploaderState.status]);
const UploaderModalContent = () => {
switch (uploaderState.status) {
@@ -130,8 +119,12 @@ export function UploadModal3D(props: Props) {
<UploadSuccess
title="3D model"
onOk={() => {
onClose();
// Open the gallery so the user can drag the freshly
// uploaded model onto the canvas.
galleryModalVisibleViewMode.value = true;
galleryModalVisibleDuringDrag.value = true;
onSuccess(selectedCategory);
onClose();
}}
/>
);
@@ -48,17 +48,6 @@ export function UploadModalImage(props: Props) {
}
}, [isOpen]);
useEffect(() => {
if (uploaderState.status === UploaderStates.success) {
// Automatically open the global Gallery modal after a successful upload
galleryModalVisibleViewMode.value = true;
galleryModalVisibleDuringDrag.value = true;
onSuccess(selectedCategory);
onClose();
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [uploaderState.status]);
const UploaderModalContent = () => {
switch (uploaderState.status) {
@@ -94,8 +83,12 @@ export function UploadModalImage(props: Props) {
<UploadSuccess
title="Image"
onOk={() => {
onClose();
// Open the gallery so the user can drag the freshly
// uploaded image onto the canvas.
galleryModalVisibleViewMode.value = true;
galleryModalVisibleDuringDrag.value = true;
onSuccess(selectedCategory);
onClose();
}}
/>
);
@@ -40,7 +40,6 @@ interface Props {
};
onClose: () => void;
onUploadProgress: (newState: UploaderState) => void;
onLocalBytes: (bytes: ArrayBuffer, shouldFlip: boolean) => void;
}
export const UploadFilesSplat = ({
@@ -18,7 +18,7 @@ import { UploadFilesSplat } from "./UploadFilesSplat";
interface Props {
onClose: () => void;
onSuccess: (splatArrayBuffer: ArrayBuffer, shouldFlip: boolean) => void;
onSuccess: (category: FilterEngineCategories) => void;
isOpen: boolean;
title: string;
titleIcon: IconDefinition;
@@ -52,17 +52,6 @@ export function UploadModalSplat(props: Props) {
}
}, [isOpen]);
useEffect(() => {
if (uploaderState.status === UploaderStates.success) {
// Automatically open the global Gallery modal after a successful upload
// galleryModalVisibleViewMode.value = true;
// galleryModalVisibleDuringDrag.value = true;
// onSuccess(selectedCategory);
onClose();
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [uploaderState.status]);
const UploaderModalContent = () => {
switch (uploaderState.status) {
@@ -79,7 +68,6 @@ export function UploadModalSplat(props: Props) {
}}
onClose={onClose}
onUploadProgress={updateUploaderState}
onLocalBytes={(buffer, shouldFlip) => { onSuccess(buffer, shouldFlip); }}
/>
</div>
);
@@ -103,8 +91,12 @@ export function UploadModalSplat(props: Props) {
<UploadSuccess
title="Splat"
onOk={() => {
// Open the gallery so the user can drag the freshly
// uploaded splat onto the canvas.
galleryModalVisibleViewMode.value = true;
galleryModalVisibleDuringDrag.value = true;
onSuccess(selectedCategory);
onClose();
// onSuccess(selectedCategory);
}}
/>
);
@@ -1,32 +1,36 @@
import { useSignals } from "@preact/signals-react/runtime";
import { faTriangleExclamation } from "@fortawesome/pro-solid-svg-icons";
import { useShallow } from "zustand/shallow";
import { Modal } from "@storyteller/ui-modal";
import { Button } from "@storyteller/ui-button";
import {
showErrorDialog,
errorDialogMessage,
errorDialogTitle,
} from "~/pages/PageEnigma/signals";
import { usePageSceneStore } from "~/pages/PageScene/PageSceneStore";
export function ErrorDialog() {
useSignals();
const { showErrorDialog, errorDialogTitle, errorDialogMessage, setShowErrorDialog } =
usePageSceneStore(
useShallow((s) => ({
showErrorDialog: s.showErrorDialog,
errorDialogTitle: s.errorDialogTitle,
errorDialogMessage: s.errorDialogMessage,
setShowErrorDialog: s.setShowErrorDialog,
})),
);
return (
<Modal
title={errorDialogTitle.value}
title={errorDialogTitle}
titleIcon={faTriangleExclamation}
titleIconClassName="text-brand-primary"
isOpen={showErrorDialog.value}
onClose={() => (showErrorDialog.value = false)}
isOpen={showErrorDialog}
onClose={() => setShowErrorDialog(false)}
showClose={false}
>
<div>
{errorDialogMessage.value}
{errorDialogMessage}
<div className="flex justify-end">
<Button
type="button"
onClick={() => (showErrorDialog.value = false)}
onClick={() => setShowErrorDialog(false)}
variant="secondary"
>
Close
@@ -1,63 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { twMerge } from "tailwind-merge";
import { faFilm } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { Button } from "@storyteller/ui-button";
import { generateMovieId, viewMyMovies } from "~/pages/PageEnigma/signals";
import { activeWorkflowJobs } from "~/signals";
export const MyMoviesButton = () => {
useSignals();
const activeCount = activeWorkflowJobs.value
? activeWorkflowJobs.value.length
: 0;
return (
<div className="relative">
<Button
variant="action"
onClick={() => {
generateMovieId.value = "";
viewMyMovies.value = true;
}}
>
<div className="relative flex items-center gap-2">
{activeCount > 0 ? (
<svg
className="h-4 w-4 animate-spin text-white"
xmlns="http://www.w3.org/2000/svg"
fill="none"
viewBox="0 0 24 24"
>
<circle
className="opacity-25"
cx="12"
cy="12"
r="10"
stroke="currentColor"
strokeWidth="4"
></circle>
<path
className="opacity-75"
fill="currentColor"
d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"
></path>
</svg>
) : (
<FontAwesomeIcon icon={faFilm} />
)}
<div>My Movies</div>
</div>
</Button>
<div
className={twMerge(
"invisible absolute right-[-7px] top-[-7px] flex h-[20px] w-[20px] items-center justify-center rounded-full bg-brand-primary text-[13px] font-medium text-white",
activeCount > 0 && "visible",
)}
>
{activeCount}
</div>
</div>
);
};
@@ -4,11 +4,7 @@ import { useSignals } from "@preact/signals-react/runtime";
import { faPencil, faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { scene, signalScene, authentication } from "~/signals";
import {
showErrorDialog,
errorDialogMessage,
errorDialogTitle,
} from "~/pages/PageEnigma/signals";
import { usePageSceneStore } from "~/pages/PageScene/PageSceneStore";
import { Input } from "@storyteller/ui-input";
import { MediaFilesApi } from "~/Classes/ApiManager/MediaFilesApi";
import { getCreatorIcon, ModelCreator } from "@storyteller/model-list";
@@ -36,9 +32,7 @@ export const SceneTitleInput = ({ pageName }: Props) => {
};
const handleShowErrorDialog = (errorMessage: string) => {
errorDialogTitle.value = "Error";
errorDialogMessage.value = errorMessage;
showErrorDialog.value = true;
usePageSceneStore.getState().setErrorDialog("Error", errorMessage);
};
const handleChangeSceneTitle = (e: React.ChangeEvent<HTMLInputElement>) => {
@@ -61,11 +61,8 @@ import toast from "react-hot-toast";
import { APP_DESCRIPTORS, goToApp } from "~/config/appMenu";
import { useStoryboardStore } from "~/pages/PageStoryboard";
import { useSceneStore } from "@storyteller/ui-pagedraw";
import {
is3DEditorInitialized,
is3DSceneLoaded,
set3DPageMounted,
} from "~/pages/PageEnigma/Editor/editor";
import { set3DPageMounted } from "~/pages/PageScene/engine/editor";
import { usePageSceneStore } from "~/pages/PageScene/PageSceneStore";
import { useImageTo3DStore } from "~/pages/PageImageTo3DObject/ImageTo3DStore";
import { useImageTo3DWorldStore } from "~/pages/PageImageTo3DWorld/ImageTo3DWorldStore";
import { useRemoveBackgroundStore } from "~/pages/PageRemoveBackground/RemoveBackgroundStore";
@@ -152,8 +149,8 @@ export const TopBar = ({ pageName }: Props) => {
const tabStore = useTabStore();
const is3DSceneReady = is3DSceneLoaded.value;
const is3DEditorReady = is3DEditorInitialized.value;
const is3DSceneReady = usePageSceneStore((s) => s.is3DSceneLoaded);
const is3DEditorReady = usePageSceneStore((s) => s.is3DEditorInitialized);
const [disableSwitcher, setDisableSwitcher] = useState(false);
const switcherThrottle = useRef(false);
@@ -17,7 +17,7 @@ import {
useStoryboardPageEnabled,
} from "@storyteller/ui-settings-modal";
import { useTabStore, TabId } from "~/pages/Stores/TabState";
import { set3DPageMounted } from "~/pages/PageEnigma/Editor/editor";
import { set3DPageMounted } from "~/pages/PageScene/engine/editor";
export type AppId =
| "IMAGE"
+7 -2
View File
@@ -1,7 +1,7 @@
import { StrictMode, useEffect } from "react";
import { useSignals, useSignalEffect } from "@preact/signals-react/runtime";
import { BrowserRouter } from "react-router-dom";
import { PageEnigma } from "./pages/PageEnigma/PageEnigma";
import { PageScene } from "./pages/PageScene/PageScene";
import { GlobalFileDropHandler } from "./components/GlobalFileDropHandler/GlobalFileDropHandler";
import { createRoot } from "react-dom/client";
import "./styles/normalize.css";
@@ -13,6 +13,7 @@ import EnvironmentVariables from "~/Classes/EnvironmentVariables";
import { pageHeight, pageWidth, persistLogin } from "~/signals";
import { SyncStorytellerApiConfig } from "./api/SyncStorytellerApiConfig";
import { posthog } from "posthog-js";
import { SoundManager } from "@storyteller/soundboard";
config.autoAddCss = false; /* eslint-disable import/first */
@@ -84,6 +85,10 @@ const GlobalSettingsManager = ({ env }: { env: Record<string, string> }) => {
};
}, []);
useEffect(() => {
SoundManager.install();
}, []);
return null;
};
@@ -94,7 +99,7 @@ createRoot(document.getElementById("root")!).render(
<BrowserRouter>
<GlobalSettingsManager env={ENV} />
<div className="topbar-spacer" data-tauri-drag-region={true} />
<PageEnigma />
<PageScene />
<GlobalFileDropHandler />
</BrowserRouter>
</StrictMode>
@@ -1,188 +0,0 @@
import React from "react";
import { MediaItem } from "~/pages/PageEnigma/models";
import {
addCharacter,
addCharacterAnimation,
addCharacterAudio,
addCharacterExpression,
addGlobalAudio,
addObject,
canDrop,
currPosition,
dragItem,
overTimeline,
timelineHeight,
assetModalVisibleDuringDrag,
reopenAfterDragSignal,
} from "~/pages/PageEnigma/signals";
import { pageHeight, pageWidth } from "~/signals";
import { addShape } from "~/pages/PageEnigma/signals/shape";
import { AssetType } from "~/enums";
class DndAsset {
public dropId: string = "";
public overElement: DOMRect | null = null;
public dropOffset = 0;
public initX = 0;
public initY = 0;
public notDropText = "";
public isDragging: boolean = false;
public dragThreshold: number = 5;
constructor() {
this.onPointerMove = this.onPointerMove.bind(this);
this.onPointerUp = this.onPointerUp.bind(this);
}
onPointerDown(event: React.PointerEvent<HTMLDivElement>, item: MediaItem) {
if (event.button === 0) {
dragItem.value = item;
currPosition.value = {
currX: event.pageX,
currY: event.pageY,
};
this.initX = event.pageX;
this.initY = event.pageY;
this.isDragging = false;
canDrop.value = false;
this.notDropText = "";
assetModalVisibleDuringDrag.value = false;
window.addEventListener("pointerup", this.onPointerUp);
window.addEventListener("pointermove", this.onPointerMove);
}
}
endDrag() {
if (dragItem.value) {
dragItem.value = null;
canDrop.value = false;
this.overElement = null;
overTimeline.value = false;
this.notDropText = "";
assetModalVisibleDuringDrag.value = reopenAfterDragSignal.value;
}
}
overCanvas(positionX: number, positionY: number) {
if (positionY < 69) {
return false;
}
if (positionY > pageHeight.value) {
return false;
}
return positionX <= pageWidth.value;
}
onPointerUp(event: PointerEvent) {
window.removeEventListener("pointerup", this.onPointerUp);
window.removeEventListener("pointermove", this.onPointerMove);
if (!this.isDragging) {
// It's a click, not a drag
assetModalVisibleDuringDrag.value = true;
dragItem.value = null;
currPosition.value = { currX: 0, currY: 0 };
return;
}
if (dragItem.value) {
const positionX = event.pageX;
const positionY = event.pageY;
if (this.overCanvas(positionX, positionY)) {
const mediaItem = dragItem.value;
if (mediaItem.type === AssetType.CHARACTER) {
addCharacter(dragItem.value);
}
// if (dragItem.value.type === AssetType.CAMERA) {
// console.log("Dragged In Camera Type")
// }
/*
FIXME:
THIS IS A TEMPARARY SOLUTION TO A LONG PROBLEM WITH SKYBOXES
UPDATE THIS WHEN UPLOADING SKYBOXES ARE FULLY IMPLEMENTED.
THIS IS JUST TEMPARARY!!!
*/
if (mediaItem.type === AssetType.OBJECT || mediaItem.type === AssetType.SPLAT || mediaItem.type === AssetType.SKYBOX) {
addObject(mediaItem);
}
if (mediaItem.type === AssetType.SHAPE) {
addShape(mediaItem);
}
this.endDrag();
return;
}
}
if (canDrop.value && dragItem.value) {
if (dragItem.value.type === AssetType.ANIMATION) {
addCharacterAnimation({
dragItem: dragItem.value,
characterId: this.dropId,
offset: this.dropOffset,
});
}
if (dragItem.value.type === AssetType.EXPRESSION) {
addCharacterExpression({
dragItem: dragItem.value,
characterId: this.dropId,
offset: this.dropOffset,
});
}
if (dragItem.value.type === AssetType.AUDIO) {
addCharacterAudio({
dragItem: dragItem.value,
characterId: this.dropId,
offset: this.dropOffset,
});
addGlobalAudio({
dragItem: dragItem.value,
audioId: this.dropId,
offset: this.dropOffset,
});
}
}
this.endDrag();
}
onPointerMove(event: MouseEvent) {
if (dragItem.value) {
event.stopPropagation();
event.preventDefault();
const deltaX = event.pageX - this.initX;
const deltaY = event.pageY - this.initY;
if (
Math.abs(deltaX) > this.dragThreshold ||
Math.abs(deltaY) > this.dragThreshold
) {
this.isDragging = true;
}
currPosition.value = {
currX: this.initX + deltaX,
currY: this.initY + deltaY,
};
overTimeline.value =
event.pageY > pageHeight.value - timelineHeight.value;
if (this.overElement) {
const pos = this.overElement;
const eventY = event.pageY;
const inHeight = eventY >= pos.top && eventY <= pos.top + pos.height;
const eventX = event.pageX;
const inWidth = eventX >= pos.left && eventX <= pos.left + pos.width;
if (inHeight && inWidth) {
return;
}
canDrop.value = false;
this.dropId = "";
this.overElement = null;
this.notDropText = "";
}
}
}
}
const dragAndDrop = new DndAsset();
export default dragAndDrop;
@@ -1,12 +0,0 @@
/**
* This is designed to surface user customer facing messages as errors.
* Errors shouldn't be 404 or something confusing should be
*/
class APIManagerResponseError extends Error {
constructor(message?: string) {
super(message);
this.name = "APIManagerResponseError";
}
}
export { APIManagerResponseError };
@@ -1,9 +0,0 @@
export interface TimelineStateJson {
version: number;
timelineItemsJson: string;
transformJson: string;
animationJson: string;
audioJson: string;
lipsyncJson: string;
emotionJson: string;
}
@@ -1,66 +0,0 @@
import { deepCopySerializableObjects as deepCopy } from "~/utilities";
import { ClipUI } from "../../clips/clip_ui";
import AudioEngine from "../Engines/audio_engine";
import EmotionEngine from "../Engines/emotion_engine";
import LipSyncEngine from "../Engines/lip_sync_engine";
import { TimeLine } from "../timeline";
import TransformEngine from "../Engines/transform_engine";
import { TimelineStateJson } from "./EditorStateJSON/timeline_state_json";
import { CharacterAnimationEngine } from "../Engines/CharacterAnimationEngine";
export class TimelineState {
version: number;
timeline_items: ClipUI[] | undefined;
transform_engine: TransformEngine | undefined;
animation_engine: CharacterAnimationEngine | undefined;
audio_engine: AudioEngine | undefined;
lipsync_engine: LipSyncEngine | undefined;
emotion_engine: EmotionEngine | undefined;
constructor({ editorVersion }: { editorVersion: number; }) {
this.version = editorVersion;
}
public async initializeFromTimeline(timeline: TimeLine): Promise<void> {
return new Promise((resolve, reject) => {
try {
this.timeline_items = deepCopy(timeline.timeline_items);
this.transform_engine = deepCopy(timeline.transform_engine);
this.animation_engine = deepCopy(timeline.animation_engine);
this.audio_engine = deepCopy(timeline.audio_engine);
this.lipsync_engine = deepCopy(timeline.lipSync_engine);
this.emotion_engine = deepCopy(timeline.emotion_engine);
resolve();
} catch (e) {
reject(e);
}
});
}
public async initializeFromTimelineStateJson(
jsonObject: TimelineStateJson
): Promise<void> {
return new Promise((resolve, reject) => {
try {
console.log(jsonObject);
//TODO: parse json object back into each engine object
resolve();
} catch (e) {
reject(e);
}
});
}
public async toJSON() {
const result: TimelineStateJson = {
version: this.version,
timelineItemsJson: JSON.stringify(this.timeline_items),
transformJson: JSON.stringify(this.transform_engine),
animationJson: JSON.stringify(this.animation_engine),
audioJson: JSON.stringify(this.audio_engine),
lipsyncJson: JSON.stringify(this.lipsync_engine),
emotionJson: JSON.stringify(this.emotion_engine),
};
return result;
}
}
@@ -1,308 +0,0 @@
import * as THREE from "three";
import { Retarget } from "../retargeting";
import { FBXLoader } from "three/examples/jsm/loaders/FBXLoader.js";
import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader.js";
import { get_media_url } from "~/Classes/ApiHelpers";
import { MMDLoader } from "three/examples/jsm/loaders/MMDLoader.js";
import { ClipUI } from "../../clips/clip_ui";
import Ijson from "~/interfaces/Ijson";
export const START_FRAME_LABEL = "CharacterStartFrameLabel";
export class CharacterAnimationEngine implements Ijson {
version: number;
characterAnimations: Map<THREE.Object3D<THREE.Object3DEventMap>, ClipUI[]> = new Map();
characterMixers = new Map<THREE.Object3D<THREE.Object3DEventMap>, THREE.AnimationMixer>();
constructor(version: number) {
this.version = version;
}
toJSON(): unknown {
// Return just an array of character object IDs
const characterIds = Array.from(this.characterAnimations.keys()).map((character) => character.uuid);
return { characters: characterIds };
}
/**
* A single character model is capable of tracking multiple animations, in its own track
* These tracks can be blended with weights or additively
* The weight change can be done via custom interpolation logic
*
* @param objectUUID The UUID of the object.
* @param clip The 3JS animation clip
* @param clipUI The clip info
*/
addCharacterAnimation(character: THREE.Object3D, clip: THREE.AnimationClip, clipUI: ClipUI) {
// If character doesn't exist, create a clip object
// If it does exist, add a new animation to the existing character by extracting it
// TODO: FIXME: Duplicate prevention?
clip.name = clipUI.media_id;
character.animations.push(clip);
console.log("Adding animation track:")
console.log(clip);
if (!this.characterAnimations.has(character)) {
this.characterAnimations.set(character, []);
this.#ensureMixerExists(character);
}
// Add the clip info to the character map and sort the clips
const characterClips = this.characterAnimations.get(character)!;
characterClips.push(clipUI);
characterClips.sort((a, b) => a.offset - b.offset);
}
/**
* @param objectUUID The UUID of the object.
* @param mediaId The media ID of the animation.
* @param clipUI The clip info
*/
async addCharacterAnimationMedia(character: THREE.Object3D, mediaId: string, clipUI: ClipUI) {
// If character doesn't exist, create a clip object
// If it does exist, add a new animation to the existing character by extracting it
// TODO: FIXME: Duplicate prevention?
const animationTrack = await this.#load_animation(character, mediaId);
this.addCharacterAnimation(character, animationTrack, clipUI);
}
#ensureMixerExists(character: THREE.Object3D) {
if (!this.characterMixers.has(character)) {
this.characterMixers.set(character, new THREE.AnimationMixer(character));
}
}
getMixer(character: THREE.Object3D) {
this.#ensureMixerExists(character);
return this.characterMixers.get(character)!;
}
#interpolateClips(character: THREE.Object3D, timestamp: number, maxTime: number) {
const clips = this.characterAnimations.get(character);
const mixer = this.getMixer(character);
// Find the two clips we're interpolating between
let prevClip: ClipUI | null = null;
let nextClip: ClipUI | null = null;
const firstClip = clips ? clips[0] : null;
const lastClip = clips ? clips[clips.length - 1] : null;
// If we're before the first clip, interpolate it in from base pose
// TODO: Replace with starting frame
if (firstClip && firstClip.offset > timestamp) {
prevClip = null;
nextClip = firstClip;
} else if (lastClip && (lastClip.offset + lastClip.length) < timestamp) { // After last clip, interpolate it out
prevClip = lastClip;
nextClip = null;
} else if (clips) {
let prevClipIndex = 0;
// Loop until the next clip is after the timestamp
while (prevClipIndex < clips.length - 1 && clips[prevClipIndex + 1].offset < timestamp) {
prevClipIndex++;
}
prevClip = clips[prevClipIndex];
nextClip = clips[prevClipIndex + 1];
}
// If the clips exist, fetch their actions. If not, evaluation is either at the start or the end.
// If evalution is on either sides, check the start/end frame tracks
// Otherwise just mark it null
const prevAnimationTrack = prevClip ?
this.getCharacterAnimationTrack(character, prevClip.media_id) :
this.getCharacterAnimationTrack(character, START_FRAME_LABEL);
// TODO: Replace with ending frame
const nextAnimationTrack = nextClip ? this.getCharacterAnimationTrack(character, nextClip.media_id) : null;
const prevAction = prevAnimationTrack ? mixer.clipAction(prevAnimationTrack) : null;
const nextAction = nextAnimationTrack ? mixer.clipAction(nextAnimationTrack) : null;
// Calculate the progress of timestamp from end of prev action to start of next action
const left = (prevClip?.offset ?? 0) + (prevClip?.length ?? 0);
const right = nextClip?.offset ?? maxTime;
// Simple Linear interpolation for now
// TODO: Take an interpolation dependency, or better yet, write a transition engine
const progress = (timestamp - left) / (right - left);
prevAction?.setEffectiveWeight(1 - progress);
nextAction?.setEffectiveWeight(progress);
// Make sure we hold that last frame for the previous action
if (prevAction) {
prevAction.clampWhenFinished = true;
}
// The next action should stay at the first frame
if (nextAction) {
nextAction.paused = true;
}
// Necessary to ensure the actions are active - the default is inactive, mixer won't do anything
prevAction?.play();
nextAction?.play();
// The clip time would still be relative to the previous clip
const clipTime = timestamp - (prevClip?.offset ?? 0);
mixer.setTime(clipTime / 1000);
}
evaluateCharacter(character: THREE.Object3D, timestamp: number, maxTime: number) {
const mixer = this.getMixer(character);
const clips = this.characterAnimations.get(character);
// Find the clip we're in right now
const currentClip = clips?.find((clip) => {
return clip.offset <= timestamp && clip.offset + clip.length >= timestamp;
});
// If timestamp not in any clip, do nothing.
// If timestamp in clip, set mixer to the timestamp inside the clip
if (!currentClip) {
// Let the interpolation function handle this actions
this.#interpolateClips(character, timestamp, maxTime)
return;
}
const clipTime = timestamp - currentClip.offset;
const animationTrack = this.getCharacterAnimationTrack(character, currentClip.media_id);
// NOTE: This shouldn't really happen unless the UI was desynced at some point from the engine
if (!animationTrack) {
return;
}
// Play only this action
const animationAction = mixer.clipAction(animationTrack);
// Since it's the only clip in this timestamp, make it full weight and make sure it's not paused (from interpolation or otherwise)
animationAction.setEffectiveWeight(1);
animationAction.paused = false;
// Necessary to ensure the actions are active - the default is inactive, mixer won't do anything
animationAction.play();
mixer.setTime(clipTime / 1000);
}
getCharacterAnimationTrack(character: THREE.Object3D, mediaId: string) {
return character.animations.find((clip) => clip.name === mediaId);
}
/** Evaluate all character animations at a given timestamp (milliseconds) */
evaluate(timestamp: number, maxTime: number) {
this.characterMixers.forEach((_, character) => {
this.evaluateCharacter(character, timestamp, maxTime);
})
}
stopCharacter(character: THREE.Object3D) {
const mixer = this.characterMixers.get(character);
console.debug("Stopping all actions for character", character);
console.debug(mixer);
mixer?.stopAllAction();
console.debug(mixer);
}
stop() {
this.characterMixers.forEach((mixer) => {
mixer.stopAllAction();
});
}
async #load_animation(character: THREE.Object3D, mediaId: string): Promise<THREE.AnimationClip> {
// Get the file URL and extract the (first) animation track from it
// TODO: Support for multiple animations in a single file?
const url = await get_media_url(mediaId);
return await new Promise((resolve) => {
if (url.includes(".glb")) {
const glbLoader = new GLTFLoader();
glbLoader.load(url, (gltf) => {
const animationClip = gltf.animations[0];
resolve(animationClip);
});
} else if (url.includes(".fbx")) {
const fbxLoader = new FBXLoader();
fbxLoader.load(url, (fbx) => {
const animationClip_1 = fbx.animations[0];
animationClip_1.tracks.forEach((track) => {
const retarget = new Retarget();
const retarget_value = retarget.retarget(track.name);
track.name = retarget_value.bone;
console.log(track);
if (retarget_value.is_special) {
// TODO: Revisit special properties later
// this.special_properties.push(retarget_value);
}
});
resolve(animationClip_1);
});
} else if (url.includes(".vmd")) {
const mmdLoader = new MMDLoader();
mmdLoader.loadAnimation(url, character as THREE.SkinnedMesh, (mmd) => {
mmd.name = url;
resolve(mmd as THREE.AnimationClip);
});
}
});
}
removeCharacter(character: THREE.Object3D) {
console.debug("Removing character from animation engine", character);
this.characterAnimations.delete(character);
this.characterMixers.delete(character);
}
removeAnimation(character: THREE.Object3D, clip: ClipUI) {
console.debug("Removing animation from character", character, clip);
const clips = this.characterAnimations.get(character)!;
const clipIndex = clips.findIndex((c) => c.clip_uuid === clip.clip_uuid);
if (clipIndex < 0) {
return;
}
// Remove the clip from character's tracked clips
clips.splice(clipIndex, 1);
// Stop all character animations to cleanly reset
this.stopCharacter(character);
}
clearStartFrame(character: THREE.Object3D, resetTime?: number, maxTime = 1) {
// Stop all character animations
this.stopCharacter(character);
// Check the animations and remove any labelled START_FRAME_LABEL
character.animations = character.animations.filter((clip) => clip.name !== START_FRAME_LABEL);
// If resetTime is passed, evaluate the character to that time
if (resetTime) {
this.evaluateCharacter(character, resetTime, maxTime);
}
}
/** resetTime: The time to reset the character to. This should ideally be the current timeline time. */
async createStartFrameAnimation(character: THREE.Object3D, poseHelper: any, url: string, resetTime: number = 0, maxTime: number = 1) {
//// Clear any existing start frame
//// This will also reset the animation to default pose
//this.clearStartFrame(character);
//// We must create the track data now after resetting to default
//const poseData = await poseHelper.extractPoseData(url);
//const tracks = poseHelper.inflatePoseDataToTracks(character, poseData);
//// Create the animation track from the rotation values and hip position
//// then add it to the character
//const startFrameClip = new THREE.AnimationClip(START_FRAME_LABEL, 0, tracks);
//character.animations.push(startFrameClip);
//// Reset the character evaluation to what was passed in
//this.evaluateCharacter(character, resetTime, maxTime);
}
}
@@ -1,48 +0,0 @@
// A mapping between the model bones names and the Mediapipe output index
export const MixamoPoseMap: Record<string, number> = {
"mixamorigLeftArm": 11,
"mixamorigRightArm": 12,
"mixamorigLeftForeArm": 13,
"mixamorigRightForeArm": 14,
"mixamorigLeftHand": 15,
"mixamorigRightHand": 16,
"mixamorigLeftUpLeg": 23,
"mixamorigRightUpLeg": 24,
"mixamorigLeftLeg": 25,
"mixamorigRightLeg": 26,
"mixamorigLeftFoot": 27,
"mixamorigRightFoot": 28,
"mixamorigLeftHandThumb2": 21,
"mixamorigRightHandThumb2": 22,
}
export const MixamoInterpolationBoneNames = {
Spine: "mixamorigSpine",
Spine1: "mixamorigSpine1",
Spine2: "mixamorigSpine2",
Neck: "mixamorigNeck",
LeftShoulderBlade: "mixamorigLeftShoulder",
DefLeftUpperArm: "DEF-upper_armL001",
DefLeftForeArm: "DEF-forearmL001",
RightShoulderBlade: "mixamorigRightShoulder",
DefRightUpperArm: "DEF-upper_armR001",
DefRightForeArm: "DEF-forearmR001",
DefLeftThigh: "DEF-thighL001",
DefRightThigh: "DEF-thighR001",
}
export const PoseMap = {
NOSE: 0,
LEFT_SHOULDER: 11,
RIGHT_SHOULDER: 12,
LEFT_ELBOW: 13,
RIGHT_ELBOW: 14,
LEFT_WRIST: 15,
RIGHT_WRIST: 16,
LEFT_HIP: 23,
RIGHT_HIP: 24,
LEFT_KNEE: 25,
RIGHT_KNEE: 26,
LEFT_ANKLE: 27,
RIGHT_ANKLE: 28,
}
@@ -1,27 +0,0 @@
import { AnimationClip } from "../../clips/animation_clip";
export class AnimationEngine {
clips: { [key: string]: AnimationClip } = {};
version: number;
constructor(version: number) {
this.clips = {};
this.version = version;
}
load_object(object_uuid: string, media_id: string, clip_name: string) {
console.debug("load_object", object_uuid, media_id, clip_name);
this.clips[object_uuid + media_id] = new AnimationClip(
this.version,
media_id,
"glb",
object_uuid,
1.0,
1.0,
clip_name,
0.0,
0.0,
);
//this.clips[object_uuid + media_id]._load_animation();
}
}
@@ -1,81 +0,0 @@
import { AudioClip } from "../../clips/audio_clip";
class AudioEngine {
clips: { [key: string]: AudioClip } = {};
audio_sources: { [key: string]: AudioBufferSourceNode } = {};
version: number;
playing: string[];
last_frame: number;
constructor() {
this.clips = {};
this.playing = [];
this.audio_sources = {};
this.version = 1.0;
this.last_frame = 0;
}
// loads clips into the engine to cache
loadClip(audio_media_id: string) {
if (this.clips[audio_media_id] != null) {
return;
}
this.clips[audio_media_id] = new AudioClip(
this.version,
audio_media_id,
1.0,
);
}
// plays from the timeline.
playClip(audio_media_id: string) {
if (this.clips[audio_media_id] == null) {
this.loadClip(audio_media_id);
}
}
async step(audio_media_id: string, frame: number, offset: number) {
const frameBuffer = 12;
const pos = frame - offset;
let doPlay = true;
if (
Math.abs(this.last_frame - frame) < frameBuffer - 1 &&
this.last_frame !== 0
) {
doPlay = false;
}
if (pos <= 1) {
doPlay = true;
}
const clip = this.clips[audio_media_id];
if (doPlay && clip.audio_data?.audioContext) {
const startTime = pos / 60.0;
const endTime = frameBuffer / 60.0;
clip.audio_data.source =
clip.audio_data.audioContext.createBufferSource();
clip.audio_data.source.buffer = clip.audio_data.audioBuffer;
clip.audio_data.source.connect(clip.audio_data.audioContext.destination);
clip.audio_data.source.start(0, startTime, endTime);
this.last_frame = frame;
}
}
// stops the clips
stopClip(audio_media_id: string) {
const clip = this.clips[audio_media_id];
if (clip.audio_data?.source) {
clip.audio_data.source.stop();
clip.audio_data.source.disconnect();
} else {
console.warn(
`AudioManager: AudioClip with id "${audio_media_id}" not found.`,
);
}
}
}
export default AudioEngine;
@@ -1,24 +0,0 @@
import { EmotionClip } from "../../clips/emotion_clip";
class EmotionEngine {
clips: { [key: string]: EmotionClip } = {};
version: number;
constructor(version: number) {
this.clips = {};
this.version = version;
}
// loads clips into the engine to cache
loadClip(object_uuid: string, csv_media_id: string) {
if (this.clips[object_uuid + csv_media_id] != null) {
return;
}
this.clips[object_uuid + csv_media_id] = new EmotionClip(
this.version,
csv_media_id,
);
}
}
export default EmotionEngine;
@@ -1,22 +0,0 @@
import { LipSyncClip } from "../../clips/lipsync_clip";
// needs to be apart of the editor window.
export default class LipSyncEngine {
clips: { [key: string]: LipSyncClip } = {};
audio_sources: { [key: string]: AudioBufferSourceNode } = {};
version: number;
constructor() {
this.clips = {};
this.audio_sources = {};
this.version = 1.0;
}
load_object(object_uuid: string, audio_media_id: string) {
this.clips[object_uuid + audio_media_id] = new LipSyncClip(
this.version,
audio_media_id,
1.0,
);
}
}
@@ -1,58 +0,0 @@
import { TransformClip } from "../../clips/transform_clip";
import * as THREE from "three";
class TransformEngine {
clips: { [key: string]: TransformClip } = {};
version: number;
constructor(version: number) {
this.version = version;
this.clips = {};
}
loadObject(object_uuid: string, clip_length: number = 2) {
this.clips[object_uuid] = new TransformClip(
this.version,
object_uuid,
clip_length,
object_uuid,
); // replace the last object_uuid with the media ID when its ready.
return this.clips[object_uuid];
}
addFrame(
object_uuid: string,
clip_length: number = 2,
position: THREE.Vector3,
rotation: THREE.Vector3,
scale: THREE.Vector3,
offset: number,
keyframe_uuid: string,
): boolean {
if (this.clips[object_uuid] == null) {
this.clips[object_uuid] = new TransformClip(
this.version,
object_uuid,
clip_length,
);
this.clips[object_uuid].add_frame(
position,
rotation,
scale,
offset,
keyframe_uuid,
);
return false;
}
this.clips[object_uuid].add_frame(
position,
rotation,
scale,
offset,
keyframe_uuid,
);
return true;
}
}
export default TransformEngine;
@@ -1,32 +0,0 @@
import { getArtStyle } from "~/enums/ArtStyle";
import {
cameraAspectRatio,
cinematic,
enginePreProcessing,
faceDetail,
globalIPAMediaToken,
lipSync,
styleStrength,
upscale,
} from "~/pages/PageEnigma/signals";
import { SceneGenereationMetaData as SceneGenerationMetaData } from "~/pages/PageEnigma/models/sceneGenerationMetadata";
import Editor from "./editor";
export const getSceneGenerationMetaData = (
editorEngine: Editor,
): SceneGenerationMetaData => {
// when this is called, editor engine is guarunteed by it's caller
return {
positivePrompt: editorEngine.positive_prompt,
negativePrompt: editorEngine.negative_prompt,
artisticStyle: getArtStyle(editorEngine.art_style.toString()),
cameraAspectRatio: cameraAspectRatio.value,
globalIPAMediaToken: globalIPAMediaToken.value || undefined,
upscale: upscale.value,
faceDetail: faceDetail.value,
styleStrength: styleStrength.value,
lipSync: lipSync.value,
cinematic: cinematic.value,
enginePreProcessing: enginePreProcessing.value,
};
};
@@ -1,460 +0,0 @@
import { v4 as uuidv4 } from "uuid";
import * as THREE from "three";
import { GLTFExporter } from "three/addons/exporters/GLTFExporter.js";
import { signalScene, startPollingActiveJobs } from "~/signals";
import { updateExistingScene, uploadNewScene } from "./api_fetchers";
import environmentVariables from "~/Classes/EnvironmentVariables";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import { StorytellerApiHostStore } from "@storyteller/api";
interface StylizeVideoParams {
media_token: string;
style: ArtStyle;
positive_prompt: string;
negative_prompt: string;
visibility: Visibility;
use_face_detailer?: boolean;
use_upscaler?: boolean;
use_strength?: number;
use_lipsync?: boolean;
use_cinematic?: boolean;
use_global_ipa_media_token?: string | null;
input_depth_file?: string | null;
input_normal_file?: string | null;
input_outline_file?: string | null;
}
/**
* Storyteller Studio API Manager
* The source of truth of all these media items is the database in the cloud
*/
export class MediaFile {
public_bucket_path: string;
media_type: string;
media_token: string;
constructor(
public_bucket_path: string,
media_type: string,
media_token: string,
) {
this.media_token = media_token;
this.public_bucket_path = public_bucket_path;
this.media_type = media_type;
}
}
export enum ArtStyle {
Anime2_5D = "anime_2_5d",
Anime2DFlat = "anime_2d_flat",
Cartoon3D = "cartoon_3d",
ComicBook = "comic_book",
AnimeGhibli = "anime_ghibli",
InkPunk = "ink_punk",
InkSplash = "ink_splash",
InkBWStyle = "ink_bw_style",
JojoStyle = "jojo_style",
PaperOrigami = "paper_origami",
PixelArt = "pixel_art",
PopArt = "pop_art",
Realistic1 = "realistic_1",
Realistic2 = "realistic_2",
AnimeRetroNeon = "anime_retro_neon",
AnimeStandard = "anime_standard",
HRGiger = "hr_giger",
Simpsons = "simpsons",
Carnage = "carnage",
PastelCuteAnime = "pastel_cute_anime",
Bloom = "bloom_lighting",
Horror2_5D = "25d_horror",
Creepy = "creepy",
CreepyVHS = "creepy_vhs",
TrailCamFootage = "trail_cam_footage",
OldBWMovie = "old_black_white_movie",
HorrorNoirBW = "horror_noir_black_white",
TechnoNoirBW = "techno_noir_black_white",
BW20s = "black_white_20s",
AnimeCyberpunk = "cyberpunk_anime",
Dragonball = "dragonball",
RealisticMatrix = "realistic_matrix",
RealisticCyberpunk = "realistic_cyberpunk",
}
export enum Visibility {
Public = "public",
Hidden = "hidden",
Private = "private",
}
/**
* This is designed to surface user customer facing messages as errors.
*/
// type Data = { [key: string]: any };
// class APIManagerResponseSuccess {
// public user_message: string;
// public data: Data | null;
// constructor(user_message: string = "", data: Data | null = null) {
// this.data = data;
// this.user_message = user_message;
// }
// }
/**
* This is designed to surface user customer facing messages as errors.
* Errors shouldn't be 404 or something confusing should be
*/
class APIManagerResponseError extends Error {
constructor(message?: string) {
super(message);
this.name = "APIManagerResponseError";
}
}
export class APIManager {
constructor() {
// NB(bt,2025-07-06): Cleaning this up. The code was duplicated a half dozen times.
// I think all of the code in this file is legacy, but to be safe I'm continuing to update it.
}
protected getApiSchemeAndHost() {
return StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
}
/**
* @param scene The 3JS Scene we want to save.
* @param scene_name The Scene name we want to display
* @param scene_media_file_token If null we will we will create a new save or copy the scene, if provided we will overwrite the scene.
* @returns APIManagerResponseMessage
*/
public async saveSceneState({
saveJson,
sceneTitle,
sceneToken,
sceneThumbnail,
}: {
saveJson: string;
sceneTitle: string;
sceneToken?: string;
sceneThumbnail: Blob | undefined;
}): Promise<string> {
const file = new File([saveJson], `${sceneTitle}.glb`, {
type: "application/json",
});
const uploadSceneResponse = sceneToken
? await updateExistingScene(file, sceneToken)
: await uploadNewScene(file, sceneTitle);
// failed so catch this.
if (uploadSceneResponse["success"] == false) {
return "";
}
if (sceneThumbnail) {
const image_resp = await this.uploadMediaSceneThumbnail(
sceneThumbnail,
"render.png",
);
if (image_resp["success"] == false) {
return "";
}
if (image_resp["media_file_token"]) {
const image_token = image_resp["media_file_token"];
const endpoint = `${this.getApiSchemeAndHost()}/v1/media_files/cover_image/${uploadSceneResponse["media_file_token"]}`;
await fetch(endpoint, {
method: "POST",
credentials: "include",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
},
body: JSON.stringify({ cover_image_media_file_token: image_token }),
});
}
}
console.log(uploadSceneResponse);
return uploadSceneResponse["media_file_token"];
}
public async loadSceneState(
scene_media_file_token: string | null,
): Promise<any> {
const url = `${this.getApiSchemeAndHost()}/v1/media_files/file/${scene_media_file_token}`;
const response = await fetch(url);
if (response.status > 200) {
throw new APIManagerResponseError("Failed to load scene");
}
const json = await response.json();
if (json && json.media_file) {
if (json.media_file.maybe_title === null) {
console.warn(`Scene /w Token: ${scene_media_file_token} has no title`);
}
signalScene({
title: json.media_file.maybe_title || "Untitled Scene",
token: scene_media_file_token || undefined,
ownerToken: json.media_file.maybe_creator_user.user_token,
isModified: false,
});
}
const bucket_path = json["media_file"]["public_bucket_path"];
const media_base_url = GetCdnOrigin();
//const media_url = `${media_base_url}/vocodes-public${bucket_path}`; // gets you a bucket path
const media_url = `${media_base_url}${bucket_path}`; // gets you a bucket path
const file_response = await fetch(media_url);
if (!file_response.ok) {
throw new APIManagerResponseError("Failed to download file");
}
// Convert the response from a blob to json text
const blob = await file_response.blob();
const json_result: string = await new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onloadend = () => resolve(JSON.parse(reader.result as string));
reader.onerror = reject;
reader.readAsText(blob);
});
return json_result;
}
/**
* This recieves the bucket path of a media file
* @param media_file_token
* @returns
*/
public async getMediaFile(media_file_token: string): Promise<string> {
const url = `${this.getApiSchemeAndHost()}/v1/media_files/file/${media_file_token}`;
const response = await fetch(url);
const json = await JSON.parse(await response.text());
const bucketPath = json["media_file"]["public_bucket_path"];
const media_base_url = GetCdnOrigin();
const media_url = `${media_base_url}${bucketPath}`; // gets you a bucket path
return media_url;
}
/**
This will save the scene to keep ids positions.
It will also give the file a name which will be a uuidv4()
@param scene The 3JS Scene we want to make a file to be uploaded as multipart form.
*/
private async gltfExport(scene: THREE.Scene): Promise<File> {
const gltfExporter = new GLTFExporter();
const uuid = uuidv4();
const result = await gltfExporter.parseAsync(scene);
const file = new File([JSON.stringify(result)], `${uuid}.glb`, {
type: "application/json",
});
return file;
}
public async uploadMediaSceneThumbnail(blob: Blob | File, fileName: string) {
const url = `${this.getApiSchemeAndHost()}/v1/media_files/upload/image`;
const uuid = uuidv4();
const formData = new FormData();
formData.append("uuid_idempotency_token", uuid);
formData.append("is_intermediate_system_file", "true");
formData.append("maybe_title", "Screenshot");
formData.append("file", blob, fileName);
//formData.append("source", "file");
//formData.append("type", "video");
//formData.append("source", "file");
const response = await fetch(url, {
method: "POST",
credentials: "include",
headers: {
Accept: "application/json",
},
body: formData,
});
if (!response.ok) {
throw new APIManagerResponseError("Upload Media Failed to send file");
} else {
const json_data = await response.json();
//console.log(`uploadMedia: ${JSON.stringify(json_data)}`);
return json_data;
}
}
public async uploadMedia({
blob,
fileName,
title,
styleName,
maybe_scene_source_media_file_token,
}: {
blob: Blob;
fileName: string;
title: string;
styleName?: string;
maybe_scene_source_media_file_token: string | undefined;
}) {
// Promise<APIManagerResponseSuccess>
//TODO: UPDATE ENDPOINT!!!!
const url = `${this.getApiSchemeAndHost()}/v1/media_files/upload/new_video`;
const uuid = uuidv4();
const formData = new FormData();
formData.append("uuid_idempotency_token", uuid);
formData.append("file", blob, fileName);
// formData.append("source", "file");
// formData.append("type", "video");
if (styleName) formData.append("maybe_style_name", styleName);
formData.append("maybe_title", title);
// This signals to the backend to hide the video from view
formData.append("is_intermediate_system_file", "true");
if (maybe_scene_source_media_file_token !== undefined) {
formData.append(
"maybe_scene_source_media_file_token",
maybe_scene_source_media_file_token,
);
}
const response = await fetch(url, {
method: "POST",
credentials: "include",
headers: {
Accept: "application/json",
},
body: formData,
});
if (!response.ok) {
throw new APIManagerResponseError("Upload Media Failed to send file");
} else {
const json_data = await response.json();
//console.log(`uploadMedia: ${JSON.stringify(json_data)}`);
return json_data;
}
}
public async uploadMediaFrameGeneration(
blob: Blob | File,
fileName: string,
style: string = "comic_book",
positive_prompt: string,
negative_prompt: string,
): Promise<string> {
const url = `${environmentVariables.values.FUNNEL_API}/preview/`;
const payload = {
style: style,
positive_prompt: positive_prompt,
negative_prompt: negative_prompt,
};
const formData = new FormData();
formData.append("input_file", blob, fileName);
formData.append("request", JSON.stringify(payload));
const response = await fetch(url, {
method: "POST",
credentials: "include",
headers: {
Accept: "application/json",
},
body: formData,
});
if (!response.ok) {
throw new APIManagerResponseError("Upload Media Failed to send file");
} else {
return URL.createObjectURL(await response.blob());
}
}
public async getMediaBatch(media_tokens: string[]): Promise<MediaFile[]> {
const tokens = media_tokens;
const url = new URL(`${this.getApiSchemeAndHost()}/v1/media_files/batch`);
tokens.forEach((token) => url.searchParams.append("tokens", token));
const result = await fetch(url)
.then((response) => response.json())
.then((data) => {
const result = data["media_files"].map((element: any) => {
return new MediaFile(
element["public_bucket_path"],
element["media_type"],
element["token"],
);
});
return result;
})
.catch((error) => console.error("Error:", error));
return result;
}
public async stylizeVideo({
media_token,
style,
positive_prompt,
negative_prompt,
visibility,
use_face_detailer = false,
use_upscaler = false,
use_strength = 1.0,
use_lipsync = false,
use_cinematic = false,
use_global_ipa_media_token = null,
input_depth_file = null,
input_normal_file = null,
input_outline_file = null,
}: StylizeVideoParams) {
const uuid = uuidv4();
const data = {
uuid_idempotency_token: uuid,
style: style,
input_file: media_token,
prompt: positive_prompt,
negative_prompt: negative_prompt,
trim_start_millis: 0,
trim_end_millis: 7000,
enable_lipsync: use_lipsync,
creator_set_visibility: visibility,
use_face_detailer: use_face_detailer,
use_upscaler: use_upscaler,
use_strength: use_strength,
use_lipsync: use_lipsync,
global_ipa_media_token: use_global_ipa_media_token,
use_cinematic: use_cinematic,
input_depth_file: input_depth_file,
input_normal_file: input_normal_file,
input_outline_file: input_outline_file,
};
const json_data = JSON.stringify(data);
const response = await fetch(
`${this.getApiSchemeAndHost()}/v1/workflows/enqueue_studio`,
{
method: "POST",
credentials: "include",
headers: {
Accept: "application/json",
"Content-Type": "application/json",
},
body: json_data,
},
);
startPollingActiveJobs();
if (!response.ok) {
// Handle HTTP error responses
const errorData = await response.json();
throw new Error(`API Error: ${response.status} ${errorData.message}`);
}
// Assuming the response is JSON and matches the EnqueueVideoStyleTransferSuccessResponse interface
return await response.json();
}
}
@@ -1,445 +0,0 @@
import {
EventDispatcher,
Quaternion,
Vector3,
Vector2,
MathUtils,
} from "three";
import * as THREE from "three";
import { cameras, selectedCameraId } from "~/pages/PageEnigma/signals/camera";
import { isPromptBoxFocused } from "~/pages/PageEnigma/signals/promptBox";
export const isSafariOrWebKit = () => {
const ua = navigator.userAgent;
const isWebKit = /AppleWebKit/i.test(ua) && !/Chrome|Chromium|Edg|OPR|CriOS|FxiOS/i.test(ua);
const isSafari = /^((?!chrome|android|crios|fxios).)*safari/i.test(ua);
return isSafari || isWebKit;
};
export const hasApplePay = () => {
// @ts-expect-error Apple Pay is not defined in TypeScript
return !!window.ApplePaySession;
}
export const isSafari = () => {
return isSafariOrWebKit() || hasApplePay();
}
export const isPointerLockSupported = () => {
return !isSafari();
}
class FreeCam extends EventDispatcher {
object: THREE.PerspectiveCamera;
domElement: HTMLElement | Document;
enabled = true;
movementSpeed = 1.0;
rollSpeed = 0.005;
dragToLook = false;
autoForward = false;
dragging = false;
xOffset = 0;
yOffset = 0;
// Smoothing properties
smoothingFactor = 0.2;
currentVelocity = new Vector3();
lastPosition = new Vector3();
lastMousePosition = new Vector2();
mouseVelocity = new Vector2();
tmpQuaternion = new Quaternion();
status = 0;
moveState = {
up: 0,
down: 0,
left: 0,
right: 0,
forward: 0,
back: 0,
pitchUp: 0,
pitchDown: 0,
yawLeft: 0,
yawRight: 0,
rollLeft: 0,
rollRight: 0,
};
moveVector = new Vector3(0, 0, 0);
rotationVector = new Vector3(0, 0, 0);
constructor(
object: THREE.PerspectiveCamera,
domElement: HTMLElement | Document,
) {
super();
this.object = object;
this.domElement = domElement;
this.lastPosition.copy(object.position);
this.keydown = this.keydown.bind(this);
this.keyup = this.keyup.bind(this);
this.mousedown = this.mousedown.bind(this);
this.mouseup = this.mouseup.bind(this);
this.mousemove = this.mousemove.bind(this);
this.mousewheel = this.mousewheel.bind(this);
this.reset = this.reset.bind(this);
this.update = this.update.bind(this);
this.updateMovementVector = this.updateMovementVector.bind(this);
this.updateRotationVector = this.updateRotationVector.bind(this);
this.getContainerDimensions = this.getContainerDimensions.bind(this);
this.dispose = this.dispose.bind(this);
this.domElement.addEventListener("contextmenu", function (event) {
event.preventDefault();
});
this.updateMovementVector();
this.updateRotationVector();
}
attachEventListeners() {
window.addEventListener("keydown", this.keydown);
window.addEventListener("keyup", this.keyup);
window.addEventListener("mousedown", this.mousedown);
window.addEventListener("mouseup", this.mouseup);
window.addEventListener("mousemove", this.mousemove);
window.addEventListener("wheel", this.mousewheel);
}
detachEventListeners() {
window.removeEventListener("keydown", this.keydown);
window.removeEventListener("keyup", this.keyup);
window.removeEventListener("mousedown", this.mousedown);
window.removeEventListener("mouseup", this.mouseup);
window.removeEventListener("mousemove", this.mousemove);
window.removeEventListener("wheel", this.mousewheel);
}
keydown(event: KeyboardEvent) {
if (!this.enabled || isPromptBoxFocused.value) {
return;
}
switch (event.code) {
case "KeyW":
this.moveState.forward = 1;
break;
case "KeyS":
this.moveState.back = 1;
break;
case "KeyA":
this.moveState.left = 1;
break;
case "KeyD":
this.moveState.right = 1;
break;
case "KeyQ":
this.moveState.down = 1;
break;
case "KeyE":
this.moveState.up = 1;
break;
case "ArrowUp":
this.moveState.pitchUp = 1;
break;
case "ArrowDown":
this.moveState.pitchDown = 1;
break;
case "ArrowLeft":
this.moveState.yawLeft = 1;
break;
case "ArrowRight":
this.moveState.yawRight = 1;
break;
}
this.updateMovementVector();
this.updateRotationVector();
}
keyup(event: KeyboardEvent) {
if (!this.enabled || isPromptBoxFocused.value) {
return;
}
switch (event.code) {
case "KeyW":
this.moveState.forward = 0;
break;
case "KeyS":
this.moveState.back = 0;
break;
case "KeyA":
this.moveState.left = 0;
break;
case "KeyD":
this.moveState.right = 0;
break;
case "ArrowUp":
this.moveState.pitchUp = 0;
break;
case "ArrowDown":
this.moveState.pitchDown = 0;
break;
case "ArrowLeft":
this.moveState.yawLeft = 0;
break;
case "ArrowRight":
this.moveState.yawRight = 0;
break;
case "KeyQ":
this.moveState.down = 0;
break;
case "KeyE":
this.moveState.up = 0;
break;
}
this.updateMovementVector();
this.updateRotationVector();
}
mousedown(event: MouseEvent) {
if (!this.enabled) {
return;
}
if (event.button === 2) {
this.dragging = true;
this.xOffset = event.clientX;
this.yOffset = event.clientY;
this.resetVelocity();
}
}
mouseup(event: MouseEvent) {
if (!this.enabled) {
return;
}
if (event.button === 2) {
this.xOffset = event.clientX;
this.yOffset = event.clientY;
this.resetVelocity();
}
this.dragging = false;
}
mousemove(event: MouseEvent) {
if (!this.enabled) {
return;
}
if (this.dragging) {
const mouseX = event.clientX - this.xOffset;
const mouseY = event.clientY - this.yOffset;
this.xOffset = event.clientX;
this.yOffset = event.clientY;
if (Math.abs(mouseX + mouseY) > 0) {
// Store last position before movement
this.lastPosition.copy(this.object.position);
// Apply movement with smoothing
const translationX = -mouseX * (this.movementSpeed * 0.01);
const translationY = mouseY * (this.movementSpeed * 0.01);
this.currentVelocity.x = MathUtils.lerp(
this.currentVelocity.x,
translationX,
this.smoothingFactor,
);
this.currentVelocity.y = MathUtils.lerp(
this.currentVelocity.y,
translationY,
this.smoothingFactor,
);
this.object.translateX(this.currentVelocity.x);
this.object.translateY(this.currentVelocity.y);
}
}
}
mousewheel(event: WheelEvent) {
if (!this.enabled) {
return;
}
// Store last position before zoom
this.lastPosition.copy(this.object.position);
const zoomDelta = event.deltaY / 120;
this.currentVelocity.z = MathUtils.lerp(
this.currentVelocity.z,
zoomDelta,
this.smoothingFactor,
);
this.object.translateZ(this.currentVelocity.z);
}
resetVelocity() {
this.currentVelocity.set(0, 0, 0);
this.lastPosition.copy(this.object.position);
}
reset() {
this.moveState.forward = 0;
this.moveState.back = 0;
this.moveState.left = 0;
this.moveState.right = 0;
this.moveState.pitchUp = 0;
this.moveState.pitchDown = 0;
this.moveState.yawLeft = 0;
this.moveState.yawRight = 0;
this.moveState.down = 0;
this.moveState.up = 0;
this.resetVelocity();
this.updateMovementVector();
this.updateRotationVector();
}
isStationary() {
return (
this.moveVector.distanceTo(new Vector3(0, 0, 0)) === 0 &&
this.currentVelocity.distanceTo(new Vector3(0, 0, 0)) < 0.0001
);
}
isRotating() {
return (
this.moveState.pitchUp ||
this.moveState.pitchDown ||
this.moveState.yawLeft ||
this.moveState.yawRight
);
}
update(delta: number) {
if (!this.enabled) {
return;
}
// Skip update if there's no movement, velocity, or rotation
if (this.isStationary() && !this.isRotating()) {
return;
}
const moveMulti = delta * this.movementSpeed;
// Apply WASD movement with smoothing
const targetMovement = new Vector3(
this.moveVector.x * moveMulti,
this.moveVector.y * moveMulti,
this.moveVector.z * moveMulti,
);
// Smooth out the movement
this.currentVelocity.lerp(targetMovement, this.smoothingFactor);
// Apply the smoothed movement
this.object.translateX(this.currentVelocity.x);
this.object.translateY(this.currentVelocity.y);
this.object.translateZ(this.currentVelocity.z);
// Store last position
this.lastPosition.copy(this.object.position);
// Done with positions
// Now handle rotations
this.object.rotateX(this.rotationVector.x);
this.object.rotateY(this.rotationVector.y);
this.object.rotateZ(this.rotationVector.z);
// Update camera signals
if (selectedCameraId.value) {
const pos = this.object.position;
const rot = this.object.rotation;
// Calculate lookAt point based on camera's forward direction
const lookAtPoint = new Vector3(0, 0, -1);
lookAtPoint.applyQuaternion(this.object.quaternion);
lookAtPoint.add(pos);
cameras.value = cameras.value.map((cam) =>
cam.id === selectedCameraId.value
? {
...cam,
position: { x: pos.x, y: pos.y, z: pos.z },
rotation: { x: rot.x, y: rot.y, z: rot.z },
lookAt: { x: lookAtPoint.x, y: lookAtPoint.y, z: lookAtPoint.z },
}
: cam,
);
}
}
updateMovementVector() {
const forward =
this.moveState.forward || (this.autoForward && !this.moveState.back)
? 1
: 0;
this.moveVector.x = -this.moveState.left + this.moveState.right;
this.moveVector.y = -this.moveState.down + this.moveState.up;
this.moveVector.z = -forward + this.moveState.back;
}
updateRotationVector() {
this.rotationVector.x = -this.moveState.pitchDown + this.moveState.pitchUp;
this.rotationVector.y = -this.moveState.yawRight + this.moveState.yawLeft;
this.rotationVector.z = -this.moveState.rollRight + this.moveState.rollLeft;
this.rotationVector.multiplyScalar(this.rollSpeed);
}
getContainerDimensions() {
if (this.domElement !== document) {
return {
size: [
(this.domElement as HTMLElement).offsetWidth,
(this.domElement as HTMLElement).offsetHeight,
],
offset: [
(this.domElement as HTMLElement).offsetLeft,
(this.domElement as HTMLElement).offsetTop,
],
};
} else {
return {
size: [window.innerWidth, window.innerHeight],
offset: [0, 0],
};
}
}
dispose() {
// window.addEventListener('contextmenu', function(event) {
// event.preventDefault();
// });
// this.domElement.removeEventListener("contextmenu", this.contextmenu);
// this.domElement.removeEventListener("pointerdown", this.pointerdown);
// this.domElement.removeEventListener("pointermove", this.pointermove);
// this.domElement.removeEventListener("pointerup", this.pointerup);
// this.domElement.removeEventListener("pointercancel", this.pointercancel);
//
window.removeEventListener("keydown", this.keydown);
window.removeEventListener("keyup", this.keyup);
}
}
export { FreeCam };
@@ -1,239 +0,0 @@
const BoundingFrequencyMasc = [0, 400, 560, 2400, 4800];
const BoundingFrequencyFem = [0, 500, 700, 3000, 6000];
const IndicesFrequencyFemale = [];
const IndicesFrequencyMale = [];
const FFT_SIZE = 1024;
const samplingFrequency = 44100;
for (let m = 0; m < BoundingFrequencyMasc.length; m++) {
IndicesFrequencyMale[m] = Math.round(
((2 * FFT_SIZE) / samplingFrequency) * BoundingFrequencyMasc[m],
);
}
for (let m = 0; m < BoundingFrequencyFem.length; m++) {
IndicesFrequencyFemale[m] = Math.round(
((2 * FFT_SIZE) / samplingFrequency) * BoundingFrequencyFem[m],
);
}
export class LipSync {
constructor(face) {
this.face = face;
this.buffer = null;
this.last_frame = 0;
// WE NEED THIS TO be run at RUNTIME
// we have to modify this for our
// this.ee = this.face.morphTargetDictionary["E"];
// this.oh = this.face.morphTargetDictionary["O"];
// this.ah = this.face.morphTargetDictionary["aa"];
}
// for streaming sound?
start(stream) {
this.audioContext = new AudioContext();
this.mediaStreamSource = this.audioContext.createMediaStreamSource(stream);
this.meter = LipSync.createAudioMeter(this.audioContext);
this.mediaStreamSource.connect(this.meter);
}
// testing only
startFromAudioFile(file) {
if (!this.audioContext) this.audioContext = new AudioContext();
if (!this.userSpeechAnalyzer)
this.userSpeechAnalyzer = this.audioContext.createAnalyser();
this.userSpeechAnalyzer.smoothingTimeConstant = 0.5;
this.userSpeechAnalyzer.fftSize = FFT_SIZE;
if (this.mediaStreamSource) this.mediaStreamSource.stop();
this.audioContext.decodeAudioData(file).then((buffer) => {
this.mediaStreamSource = this.audioContext.createBufferSource();
this.mediaStreamSource.buffer = buffer;
this.meter = LipSync.createAudioMeter(this.audioContext);
this.mediaStreamSource.connect(this.meter);
this.mediaStreamSource.connect(this.audioContext.destination);
this.mediaStreamSource.start();
// connect the output of mediaStreamSource to the input of userSpeechAnalyzer
this.mediaStreamSource.connect(this.userSpeechAnalyzer);
});
}
startLipSyncFromAudioBuffer(buffer) {
if (!this.audioContext) this.audioContext = new AudioContext();
if (!this.userSpeechAnalyzer)
this.userSpeechAnalyzer = this.audioContext.createAnalyser();
this.userSpeechAnalyzer.smoothingTimeConstant = 0.5;
this.userSpeechAnalyzer.fftSize = FFT_SIZE;
if (this.mediaStreamSource) this.mediaStreamSource.stop();
this.buffer = buffer;
}
destroy() {
this.meter?.shutdown();
this.meter = null;
this.mediaStreamSource?.disconnect();
return this.audioContext?.close().catch(() => {}) || Promise.resolve();
}
update(frame, offset, rendering) {
const frameBuffer = 12;
let pos = frame - offset;
if (Math.floor(pos) <= 1){
this.last_frame = 0;
}
let doPlay = true;
if (
Math.abs(this.last_frame - frame) < frameBuffer - 1 &&
this.last_frame !== 0
) {
doPlay = false;
}
if (doPlay) {
const startTime = pos / 60.0;
const endTime = frameBuffer / 60.0;
this.mediaStreamSource = this.audioContext.createBufferSource();
this.mediaStreamSource.buffer = this.buffer;
this.meter = LipSync.createAudioMeter(this.audioContext);
this.mediaStreamSource.connect(this.meter);
this.mediaStreamSource.connect(this.audioContext.destination);
this.mediaStreamSource.start(0, startTime, endTime);
this.mediaStreamSource.connect(this.userSpeechAnalyzer);
this.last_frame = frame;
}
if (this.meter) {
const { volume } = this.meter;
let { ah, oh, ee } = this.update2();
if (volume < 0.01) {
ah = 0;
oh = 0;
ee = 0;
}
return { ah: ah, oh: oh, ee: ee };
}
return { ah: 0, oh: 0, ee: 0 };
}
update2() {
function getSensitivityMap(spectrum) {
const sensitivity_threshold = 0.5;
const stPSD = new Float32Array(spectrum.length);
for (let i = 0; i < spectrum.length; i++) {
stPSD[i] = sensitivity_threshold + (spectrum[i] + 20) / 140;
}
return stPSD;
}
const spectrum = new Float32Array(
this.userSpeechAnalyzer.frequencyBinCount,
);
// Populate frequency data for computing frequency intensities
this.userSpeechAnalyzer.getFloatFrequencyData(spectrum); // getByteTimeDomainData gets volumes over the sample time
// Populate time domain for calculating RMS
// userSpeechAnalyzer.getFloatTimeDomainData(spectrum);
// RMS (root mean square) is a better approximation of current input level than peak (just sampling this frame)
// spectrumRMS = getRMS(spectrum);
const sensitivityPerPole = getSensitivityMap(spectrum);
// Lower and higher voices have different frequency domains, so we'll separate and max them
const EnergyBinMasc = new Float32Array(BoundingFrequencyMasc.length);
const EnergyBinFem = new Float32Array(BoundingFrequencyFem.length);
// Masc energy bins (groups of frequency-depending energy)
for (let m = 0; m < BoundingFrequencyMasc.length - 1; m++) {
for (
let j = IndicesFrequencyMale[m];
j <= IndicesFrequencyMale[m + 1];
j++
)
if (sensitivityPerPole[j] > 0)
EnergyBinMasc[m] += sensitivityPerPole[j];
EnergyBinMasc[m] /= IndicesFrequencyMale[m + 1] - IndicesFrequencyMale[m];
}
// Fem energy bin
for (let m = 0; m < BoundingFrequencyFem.length - 1; m++) {
for (
let j = IndicesFrequencyMale[m];
j <= IndicesFrequencyMale[m + 1];
j++
)
if (sensitivityPerPole[j] > 0) EnergyBinFem[m] += sensitivityPerPole[j];
EnergyBinMasc[m] /= IndicesFrequencyMale[m + 1] - IndicesFrequencyMale[m];
EnergyBinFem[m] =
EnergyBinFem[m] /
(IndicesFrequencyFemale[m + 1] - IndicesFrequencyFemale[m]);
}
const oh =
Math.max(EnergyBinFem[1], EnergyBinMasc[1]) > 0.2
? 1 - 2 * Math.max(EnergyBinMasc[2], EnergyBinFem[2])
: (1 - 2 * Math.max(EnergyBinMasc[2], EnergyBinFem[2])) *
5 *
Math.max(EnergyBinMasc[1], EnergyBinFem[1]);
const ah = 3 * Math.max(EnergyBinMasc[3], EnergyBinFem[3]);
const ee =
0.8 *
(Math.max(EnergyBinMasc[1], EnergyBinFem[1]) -
Math.max(EnergyBinMasc[3], EnergyBinFem[3]));
return { oh, ee, ah };
}
static createAudioMeter(audioContext) {
const processor = audioContext.createScriptProcessor(512);
processor.onaudioprocess = (event) => {
const buf = event.inputBuffer.getChannelData(0);
const bufLength = buf.length;
let sum = 0;
let x;
// eslint-disable-next-line no-plusplus
for (let i = 0; i < bufLength; i++) {
x = buf[i];
if (Math.abs(x) >= processor.clipLevel) {
processor.clipping = true;
processor.lastClip = window.performance.now();
}
sum += x * x;
}
const rms = Math.sqrt(sum / bufLength);
processor.volume = Math.max(rms, processor.volume * processor.averaging);
};
processor.clipping = false;
processor.lastClip = 0;
processor.volume = 0;
processor.clipLevel = 0.98;
processor.averaging = 0.95;
processor.clipLag = 750;
processor.connect(audioContext.destination);
processor.checkClipping = () => {
if (!processor.clipping) {
return false;
}
if (processor.lastClip + processor.clipLag < window.performance.now()) {
processor.clipping = false;
}
return processor.clipping;
};
processor.shutdown = () => {
processor.disconnect();
processor.onaudioprocess = null;
};
return processor;
}
}
@@ -1,41 +0,0 @@
import { MediaUploadApi, VideoApi } from "~/Classes/ApiManager";
import {
ProcessStatus,
StreamingProgressResponse,
VideoAudioPreProcessor,
} from "./VideoProcessor/video_audio_preprocessor";
class PreviewVideoGeneration {
mediaUploadAPI: MediaUploadApi;
videoAPI: VideoApi;
videoAndAudioPreProcessor: VideoAudioPreProcessor;
onProgress: (response: StreamingProgressResponse<ProcessStatus>) => void;
constructor(
onProgress: (response: StreamingProgressResponse<ProcessStatus>) => void,
) {
this.onProgress = onProgress;
this.videoAndAudioPreProcessor = new VideoAudioPreProcessor(onProgress);
this.mediaUploadAPI = new MediaUploadApi();
this.videoAPI = new VideoApi();
}
// Cancel any part of the processing
public async cancel() {}
private async stageOneProcessing() {}
private async stageTwoProcessing() {}
private async processVideo() {
// check if video is cached or not if not jump to 50%
// collect frames and run preprocessing in the background.
// start polling for percentage change and convert and remap to remainint %
}
async clearFrameBuffers() {}
}
export { PreviewVideoGeneration };
@@ -1,97 +0,0 @@
export class MoveAIResult {
bone: string;
is_special: boolean;
x: number;
y: number;
z: number;
flip: boolean;
only_y: boolean;
constructor(bone: string, is_special: boolean = false, x: number = 0, y: number = 0, z: number = 0,
flip:boolean=false, only_y:boolean=false) {
this.bone = bone;
this.is_special = is_special;
this.x = x;
this.y = y;
this.z = z;
this.flip = flip;
this.only_y = only_y;
}
}
export class Retarget {
move_ai_options: { [Key: string]: MoveAIResult }
constructor() {
this.move_ai_options = {
"_1Head.quaternion": new MoveAIResult("mixamorigHead.quaternion"),
"_1Neck.quaternion": new MoveAIResult("mixamorigNeck.quaternion"),
"_Spine.quaternion": new MoveAIResult("mixamorigSpine.quaternion"),
"_1Spine.quaternion": new MoveAIResult("mixamorigSpine1.quaternion"),
"_3Spine.quaternion": new MoveAIResult("mixamorigSpine2.quaternion"),
"_1Hips.quaternion": new MoveAIResult("mixamorigHips.quaternion", true, -90, 0, 0, false, true),
"_1Hips.position": new MoveAIResult("mixamorigHips.position", true, 1, 0.05, 1),
"_1LeftFoot.quaternion": new MoveAIResult("mixamorigLeftFoot.quaternion", true, 0, 0, 0, true),
"_1RightFoot.quaternion": new MoveAIResult("mixamorigRightFoot.quaternion", true, 0, 0, 0, true),
"_1RightShoulder.quaternion": new MoveAIResult("mixamorigRightShoulder.quaternion", true, 0, 0, 0, false),
"_1LeftShoulder.quaternion": new MoveAIResult("mixamorigLeftShoulder.quaternion", true, 0, 0, 0, false),
"_1LeftArm.quaternion": new MoveAIResult("mixamorigLeftArm.quaternion", true, 0, 0, 0, false),
"_1RightArm.quaternion": new MoveAIResult("mixamorigRightArm.quaternion", true, 0, 0, 0, false),
"_1LeftForeArm.quaternion": new MoveAIResult("mixamorigLeftForeArm.quaternion"),
"_1RightForeArm.quaternion": new MoveAIResult("mixamorigRightForeArm.quaternion"),
"_1LeftLeg.quaternion": new MoveAIResult("mixamorigLeftLeg.quaternion", true, 0, 0, 0, true),
"_1RightLeg.quaternion": new MoveAIResult("mixamorigRightLeg.quaternion", true, 0, 0, 0, true),
"_1LeftUpLeg.quaternion": new MoveAIResult("mixamorigLeftUpLeg.quaternion", true, 0, 180, 0),
"_1RightUpLeg.quaternion": new MoveAIResult("mixamorigRightUpLeg.quaternion", true, 0, 180, 0),
"_1LeftHand.quaternion": new MoveAIResult("mixamorigLeftHand.quaternion"),
"_1RightHand.quaternion": new MoveAIResult("mixamorigRightHand.quaternion"),
"_1LeftHandThumb1.quaternion": new MoveAIResult("mixamorigLeftHandThumb1.quaternion"),
"_1LeftHandThumb2.quaternion": new MoveAIResult("mixamorigLeftHandThumb2.quaternion"),
"_1LeftHandThumb3.quaternion": new MoveAIResult("mixamorigLeftHandThumb3.quaternion"),
"_1LeftHandIndex1.quaternion": new MoveAIResult("mixamorigLeftHandIndex1.quaternion"),
"_1LeftHandIndex2.quaternion": new MoveAIResult("mixamorigLeftHandIndex2.quaternion"),
"_1LeftHandIndex3.quaternion": new MoveAIResult("mixamorigLeftHandIndex3.quaternion"),
"_1LeftHandMiddle1.quaternion": new MoveAIResult("mixamorigLeftHandMiddle1.quaternion"),
"_1LeftHandMiddle2.quaternion": new MoveAIResult("mixamorigLeftHandMiddle2.quaternion"),
"_1LeftHandMiddle3.quaternion": new MoveAIResult("mixamorigLeftHandMiddle3.quaternion"),
"_1LeftHandRing1.quaternion": new MoveAIResult("mixamorigLeftHandRing1.quaternion"),
"_1LeftHandRing2.quaternion": new MoveAIResult("mixamorigLeftHandRing2.quaternion"),
"_1LeftHandRing3.quaternion": new MoveAIResult("mixamorigLeftHandRing3.quaternion"),
"_1LeftHandPinky1.quaternion": new MoveAIResult("mixamorigLeftHandPinky1.quaternion"),
"_1LeftHandPinky2.quaternion": new MoveAIResult("mixamorigLeftHandPinky2.quaternion"),
"_1LeftHandPinky3.quaternion": new MoveAIResult("mixamorigLeftHandPinky3.quaternion"),
"_1RightHandThumb1.quaternion": new MoveAIResult("mixamorigRightHandThumb1.quaternion"),
"_1RightHandThumb2.quaternion": new MoveAIResult("mixamorigRightHandThumb2.quaternion"),
"_1RightHandThumb3.quaternion": new MoveAIResult("mixamorigRightHandThumb3.quaternion"),
"_1RightHandIndex1.quaternion": new MoveAIResult("mixamorigRightHandIndex1.quaternion"),
"_1RightHandIndex2.quaternion": new MoveAIResult("mixamorigRightHandIndex2.quaternion"),
"_1RightHandIndex3.quaternion": new MoveAIResult("mixamorigRightHandIndex3.quaternion"),
"_1RightHandMiddle1.quaternion": new MoveAIResult("mixamorigRightHandMiddle1.quaternion"),
"_1RightHandMiddle2.quaternion": new MoveAIResult("mixamorigRightHandMiddle2.quaternion"),
"_1RightHandMiddle3.quaternion": new MoveAIResult("mixamorigRightHandMiddle3.quaternion"),
"_1RightHandRing1.quaternion": new MoveAIResult("mixamorigRightHandRing1.quaternion"),
"_1RightHandRing2.quaternion": new MoveAIResult("mixamorigRightHandRing2.quaternion"),
"_1RightHandRing3.quaternion": new MoveAIResult("mixamorigRightHandRing3.quaternion"),
"_1RightHandPinky1.quaternion": new MoveAIResult("mixamorigRightHandPinky1.quaternion"),
"_1RightHandPinky2.quaternion": new MoveAIResult("mixamorigRightHandPinky2.quaternion"),
"_1RightHandPinky3.quaternion": new MoveAIResult("mixamorigRightHandPinky3.quaternion"),
}
}
retarget(boneName: string): MoveAIResult {
for (const [key, value] of Object.entries(this.move_ai_options)) {
if (boneName == key) {
return value;
}
}
return new MoveAIResult(boneName);
}
}
@@ -1,52 +0,0 @@
// @see https://discourse.threejs.org/t/production-ready-green-screen-with-three-js/23113/2
const VERTEX_SHADER = `
varying vec2 vUv;
void main() {
vUv = uv;
gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
}
`;
const FRAGMENT_SHADER = `
uniform sampler2D tex;
uniform float texWidth;
uniform float texHeight;
uniform vec3 keyColor;
uniform float similarity;
uniform float smoothness;
uniform float spill;
varying vec2 vUv;
// From https://github.com/libretro/glsl-shaders/blob/master/nnedi3/shaders/rgb-to-yuv.glsl
vec2 RGBtoUV(vec3 rgb) {
return vec2(
rgb.r * -0.169 + rgb.g * -0.331 + rgb.b * 0.5 + 0.5,
rgb.r * 0.5 + rgb.g * -0.419 + rgb.b * -0.081 + 0.5
);
}
vec4 ProcessChromaKey(vec2 texCoord) {
vec4 rgba = texture2D(tex, texCoord);
float chromaDist = distance(RGBtoUV(texture2D(tex, texCoord).rgb), RGBtoUV(keyColor));
float baseMask = chromaDist - similarity;
float fullMask = pow(clamp(baseMask / smoothness, 0., 1.), 1.5);
rgba.a = fullMask;
float spillVal = pow(clamp(baseMask / spill, 0., 1.), 1.5);
float desat = clamp(rgba.r * 0.2126 + rgba.g * 0.7152 + rgba.b * 0.0722, 0., 1.);
rgba.rgb = mix(vec3(desat, desat, desat), rgba.rgb, spillVal);
return rgba;
}
void main(void) {
vec2 texCoord = vUv;
gl_FragColor = ProcessChromaKey(texCoord);
}
`;
export { VERTEX_SHADER, FRAGMENT_SHADER };
@@ -1,950 +0,0 @@
import * as THREE from "three";
import { ClipUI } from "../clips/clip_ui";
import Scene from "./scene.js";
import AudioEngine from "./Engines/audio_engine";
import TransformEngine from "./Engines/transform_engine";
import LipSyncEngine from "./Engines/lip_sync_engine";
import Queue, {
UnionedActionTypes,
UnionedDataTypes,
} from "~/pages/PageEnigma/Queue/Queue";
import { QueueNames } from "../Queue/QueueNames";
import { toEngineActions } from "../Queue/toEngineActions";
import { fromEngineActions } from "../Queue/fromEngineActions";
import { ClipGroup, ClipType, AssetType } from "~/enums";
import { CameraAspectRatio, MediaFileType } from "~/pages/PageEnigma/enums";
import { Keyframe, MediaItem, UpdateTime } from "~/pages/PageEnigma/models";
import Editor from "~/pages/PageEnigma/Editor/editor";
import EmotionEngine from "./Engines/emotion_engine";
import { IGenerationOptions } from "~/pages/PageEnigma/models/generationOptions";
import { Vector3 } from "three";
import { filmLength, outlinerState } from "../signals";
import { CharacterAnimationEngine } from "./Engines/CharacterAnimationEngine";
import { SplatMesh } from "@sparkjsdev/spark";
export class TimeLine {
editorEngine: Editor;
timeline_items: ClipUI[];
timeline_limit: number;
absolute_end: number;
scrubber_frame_position: number;
is_playing: boolean;
is_repeating: boolean = true;
// plays audio
audio_engine: AudioEngine;
// key framing
transform_engine: TransformEngine;
// animation engine
animation_engine: CharacterAnimationEngine;
// lip sync engine
lipSync_engine: LipSyncEngine;
// emotion engine
emotion_engine: EmotionEngine;
// characters
characters: { [key: string]: ClipGroup };
scene: Scene;
camera: THREE.Camera | null;
mouse: THREE.Vector2 | undefined;
current_time: number;
camera_name: string;
// ensure that the elements are loaded first.
private debounce_generate_video: boolean;
constructor(
editor: Editor,
audio_engine: AudioEngine,
transform_engine: TransformEngine,
lipsync_engine: LipSyncEngine,
animation_engine: CharacterAnimationEngine,
emotion_engine: EmotionEngine,
scene: Scene,
camera: THREE.Camera | null,
mouse: THREE.Vector2 | undefined,
camera_name: string,
) {
this.editorEngine = editor;
this.timeline_items = [];
this.characters = {};
this.absolute_end = 60 * 12;
this.timeline_limit = this.getTimelineEnd();
this.camera = camera;
this.mouse = mouse;
this.is_playing = false;
this.scrubber_frame_position = 0; // in frames into the tl
// this will be used to play the audio clips
this.audio_engine = audio_engine;
this.transform_engine = transform_engine;
this.lipSync_engine = lipsync_engine;
this.animation_engine = animation_engine;
this.emotion_engine = emotion_engine;
this.scene = scene;
this.debounce_generate_video = false;
Queue.subscribe(
QueueNames.TO_ENGINE,
"engine",
this.handleTimelineActions.bind(this),
);
this.current_time = 0;
this.camera_name = camera_name;
}
public async updateUI() {
Queue.publish({
queueName: QueueNames.FROM_ENGINE,
action: fromEngineActions.UPDATE_TIME_LINE,
data: this.timeline_items,
});
}
public async pushEvent(action: fromEngineActions, data: UnionedDataTypes) {
//this.current_time += 0.75;
Queue.publish({
queueName: QueueNames.FROM_ENGINE,
action: fromEngineActions.UPDATE_TIME,
data: data,
});
}
public isCharacter(uuid: string): boolean {
this.timeline_items.forEach((clip) => {
if (clip.group == ClipGroup.CHARACTER)
this.characters[clip.object_uuid] = ClipGroup.CHARACTER;
});
let result: boolean = false;
for (const key in this.characters) {
if (key === uuid) {
result = true;
return true;
}
}
return result;
}
public async handleTimelineActions(data: {
action: UnionedActionTypes;
data: UnionedDataTypes;
}) {
const action = data.action;
switch (action) {
case toEngineActions.ADD_KEYFRAME:
await this.addKeyFrame(data.data as Keyframe);
break;
case toEngineActions.UPDATE_KEYFRAME:
await this.updateKeyFrame(data.data as Keyframe);
break;
case toEngineActions.DELETE_KEYFRAME:
await this.deleteKeyFrame(data.data as Keyframe);
break;
case toEngineActions.ADD_CLIP:
await this.addClip(data.data as ClipUI);
break;
case toEngineActions.DELETE_CLIP:
await this.deleteClip(data.data as ClipUI);
break;
case toEngineActions.UPDATE_CLIP:
await this.updateClip(data.data as ClipUI);
break;
case toEngineActions.UPDATE_TIME:
await this.scrub(data.data as UpdateTime);
break;
case toEngineActions.MUTE:
await this.mute(data.data as ClipUI, false);
break;
case toEngineActions.UNMUTE:
await this.mute(data.data as ClipUI, true);
break;
// Create operations
case toEngineActions.ADD_CHARACTER: {
const newObject = await this.addCharacter(data.data as MediaItem);
if (newObject)
this.queueNewObjectMessage(
newObject,
data.data as MediaItem,
AssetType.CHARACTER,
);
const result = this.editorEngine.sceneManager?.render_outliner(
this.characters,
);
if (result) outlinerState.items.value = result.items;
break;
}
case toEngineActions.ADD_OBJECT: {
const newObject = await this.addObject(data.data as MediaItem);
if (newObject)
this.queueNewObjectMessage(
newObject,
data.data as MediaItem,
newObject.type as AssetType
);
const result = this.editorEngine.sceneManager?.render_outliner(
this.characters,
);
if (result) outlinerState.items.value = result.items;
break;
}
case toEngineActions.ADD_SHAPE: {
const newShape = await this.addShape(data.data as MediaItem);
this.queueNewObjectMessage(
newShape,
data.data as MediaItem,
AssetType.SHAPE,
);
const result = this.editorEngine.sceneManager?.render_outliner(
this.characters,
);
if (result) outlinerState.items.value = result.items;
break;
}
case toEngineActions.ENTER_PREVIEW_STATE:
await this.editorEngine.switchPreview();
break;
case toEngineActions.ENTER_EDIT_STATE:
this.editorEngine.switchEdit();
break;
case toEngineActions.TOGGLE_CAMERA_STATE:
this.editorEngine.switchCameraView();
break;
case toEngineActions.TOGGLE_REPEATING:
this.is_repeating = !this.is_repeating;
break;
case toEngineActions.REFRESH_PREVIEW:
if (this.editorEngine.switchPreviewToggle) {
await this.editorEngine.generateFrame();
}
break;
case toEngineActions.GENERATE_VIDEO: {
console.log("Calling Generate Video");
// debounce generate video ...
if (this.debounce_generate_video == false) {
this.debounce_generate_video = true;
const options = data.data; // super overloaded talk to the devs about this. TODO... refactor
// pass this in ... rather than doing it implicitly ...
this.editorEngine.generation_options = options as IGenerationOptions;
await this.editorEngine.generateVideo();
this.debounce_generate_video = false;
}
console.log("Generate Video Press Event");
break;
}
case toEngineActions.CHANGE_CAMERA_ASPECT_RATIO: {
this.editorEngine.changeRenderCameraAspectRatio(
data.data as CameraAspectRatio,
);
break;
}
default:
console.log("Action Not Wired", action);
}
}
public async addCharacter(data: MediaItem) {
const media_id = data.media_id;
const name = data.name;
const pos = this.getPos();
const new_data = { ...data };
console.debug('Adding to engine scene manager: ', media_id, name, pos);
const obj = await this.editorEngine.sceneManager?.create(
media_id,
name,
pos,
);
if (obj) {
obj.userData["name"] = name;
obj.name = name;
obj.position.copy(pos);
obj.userData.isCharacter = true;
console.debug("Added character object to timeline; object: ", obj);
// TODO(bt,2025-01-28): Hack for debugging only. Remove soon.
(window as any).lastCharacter = obj;
// TODO(bt,2025-01-28): Hack for debugging only. Remove soon.
const lastRig = obj.children[0];
if (!!lastRig && lastRig.name.includes("gamerig")) {
(window as any).lastRig = lastRig;
}
const object_uuid = obj.uuid;
this.characters[object_uuid] = ClipGroup.CHARACTER; // TODO: Create a class to make the idea of a character.
new_data["object_uuid"] = object_uuid;
Queue.publish({
queueName: QueueNames.FROM_ENGINE,
action: fromEngineActions.UPDATE_CHARACTER_ID,
data: new_data,
});
this.addPlayableClip(
new ClipUI(
data.version,
ClipType.FAKE,
ClipGroup.CHARACTER,
"Default",
media_id,
obj.uuid,
obj.uuid,
name,
0,
0,
0,
obj.userData["media_file_type"],
),
);
await this.editorEngine.sceneManager?.add_creation_undostack(obj);
}
return obj;
}
queueNewObjectMessage(
item: THREE.Object3D<THREE.Object3DEventMap>,
data: MediaItem,
asset_type: AssetType.OBJECT | AssetType.CHARACTER | AssetType.SHAPE | AssetType.SPLAT,
) {
Queue.publish({
queueName: QueueNames.FROM_ENGINE,
action: fromEngineActions.ADD_OBJECT,
data: {
media_id: data.media_id,
type: asset_type,
name: item.name,
object_uuid: item.uuid,
version: 1,
} as MediaItem,
});
// this.addPlayableClip(
// new ClipUI(
// data["version"],
// ClipType.FAKE,
// ClipGroup.OBJECT,
// "Default",
// data.media_id,
// item.uuid,
// item.uuid,
// item.name,
// 0,
// 0,
// 0,
// this.scene.get_object_by_uuid(item.uuid)?.userData[
// "media_file_type"
// ],
// ),
// );
}
public getPos() {
this.editorEngine.utils.removeTransformControls(true);
const raycaster = new THREE.Raycaster();
raycaster.layers.enable(0);
raycaster.layers.enable(1);
if (this.editorEngine.mouse && this.camera) {
raycaster.setFromCamera(this.editorEngine.mouse, this.camera);
const intersects = raycaster.intersectObjects(
this.scene.scene.children,
true,
);
if (intersects.length > 0) {
return intersects[0].point;
}
}
return new THREE.Vector3(0, 0, 0);
}
public async addObject(data: MediaItem & { position?: { x: number; y: number; z: number } }) {
const dataPos = data.position;
const pos = dataPos
? new THREE.Vector3(dataPos.x, dataPos.y, dataPos.z)
: this.getPos();
const media_id = data.media_id;
const name = data.name;
const obj = await this.editorEngine.sceneManager?.create(
media_id,
name,
pos,
);
if (obj) {
obj.userData["name"] = name;
obj.name = name;
obj.position.copy(pos);
await this.editorEngine.sceneManager?.add_creation_undostack(obj);
}
return obj;
}
// This method is for local dev testing of objects
// and bypasses the queue system, that's kinda redundant anyway
public addLocalSplat(fileBytes: ArrayBuffer | Uint8Array, flipVertical: boolean = false) {
const splatMesh = new SplatMesh({ fileBytes: fileBytes });
splatMesh.name = `{SplatMesh-${Date.now()}}`;
if (flipVertical) {
splatMesh.rotateX(Math.PI);
}
this.editorEngine.sceneManager?.scene.scene.add(splatMesh);
}
public async addShape(data: MediaItem) {
const pos = this.getPos();
const parim = await this.editorEngine.create_parim(data.media_id, pos);
await this.editorEngine.sceneManager?.add_creation_undostack(parim);
return parim;
}
public async addKeyFrame(data: Keyframe) {
// KeyFrame Object
// version: number;
// clip_uuid: string;
// group: ClipGroup;
// object_uuid?: string;
// offset: number;
// position: XYZ
// rotation: XYZ;
// scale: XYZ;
// selected?: boolean;
const data_json = data;
const uuid = data_json.object_uuid;
const keyframe_uuid = data_json.keyframe_uuid;
let object_name = this.scene.get_object_by_uuid(uuid)?.name;
if (object_name === undefined) {
object_name = "undefined";
}
this.transform_engine.addFrame(
uuid,
data_json.offset,
data_json.position as Vector3,
data_json.rotation as Vector3,
data_json.scale as Vector3,
data_json.offset,
data_json.keyframe_uuid,
);
await this.addPlayableClip(
new ClipUI(
data_json["version"],
ClipType.TRANSFORM,
data_json["group"],
object_name,
"",
keyframe_uuid,
uuid,
object_name,
0,
data_json["offset"],
data_json["offset"],
MediaFileType.None,
),
);
const point = this.scene.createPoint(
data_json.position as Vector3,
data_json.rotation as Vector3,
data_json.scale as Vector3,
data_json.keyframe_uuid,
);
if (this.editorEngine.camera_person_mode) {
point.visible = false;
}
this.checkEditorCanPlay();
}
public checkEditorCanPlay() {
this.editorEngine.can_playback = this.getLastClipEnd() > 1;
this.editorEngine.updateSelectedUI();
}
// This method is NOT responsible for deletion of the object from scene or other editor parts
// It is only responsible for deleting the object from the timeline and its corresponding elements
public deleteObject(object: THREE.Object3D<THREE.Object3DEventMap>) {
if (!object) {
return
}
if (object.name === this.camera_name) {
return;
}
const object_uuid = object.uuid;
this.timeline_items.forEach((element) => {
if (
element.type == ClipType.TRANSFORM &&
element.object_uuid == object_uuid
) {
this.scene.deletePoint(element.clip_uuid);
}
});
// TODO: In the future object should have relations that can be deleted inside the engines
// and the engines handle the deletion of objects.
// Character Animation Engine already does this.
this.timeline_items = this.timeline_items.filter(
(element) => element.object_uuid !== object_uuid,
);
this.animation_engine.removeCharacter(object);
// Update react land here.
}
public async addClip(data: ClipUI) {
const object_uuid = data.object_uuid;
const media_id = data.media_id;
const name = data.name;
const group = data.group;
const version = 1;
const type = data.type;
const offset = data.offset;
const end_offset = data.length + offset;
const object = this.scene.get_object_by_uuid(object_uuid);
const object_name =
object?.name ?? "undefined";
const clip_uuid = data.clip_uuid;
switch (type) {
case "animation":
await this.animation_engine.addCharacterAnimationMedia(object!, media_id, data);
this.animation_engine.evaluate(this.current_time, this.timeline_limit);
break;
case "transform":
this.transform_engine.loadObject(object_uuid, data.length);
break;
case "expression":
this.emotion_engine.loadClip(object_uuid, media_id);
break;
case "audio":
if (group === "character") {
this.lipSync_engine.load_object(object_uuid, media_id);
// media id for this as well it can be downloaded
this.addPlayableClip(
new ClipUI(
version,
ClipType.AUDIO,
ClipGroup.CHARACTER,
name,
media_id,
clip_uuid,
object_uuid,
object_name,
offset,
end_offset,
0, // length
this.scene.get_object_by_uuid(object_uuid)?.userData[
"media_file_type"
],
),
);
return;
} else {
this.audio_engine.loadClip(media_id);
}
break;
}
// media id for this as well it can be downloaded
this.addPlayableClip(
new ClipUI(
version,
type,
group,
name,
media_id,
clip_uuid,
object_uuid,
object_name,
offset,
end_offset, // length
0,
this.scene.get_object_by_uuid(object_uuid)?.userData["media_file_type"],
),
);
this.checkEditorCanPlay();
}
public async addSelfAnimationClip(
data: ClipUI,
animation_clip: THREE.AnimationClip,
) {
const object_uuid = data.object_uuid;
const media_id = data.media_id;
const name = data.name;
const group = data.group;
const version = 1;
const type = data.type;
const offset = data.offset;
const end_offset = data.length + offset;
const object = this.scene.get_object_by_uuid(object_uuid);
const object_name =
object?.name ?? "undefined";
const clip_uuid = data.clip_uuid;
this.animation_engine.addCharacterAnimation(object!, animation_clip, data);
// media id for this as well it can be downloaded
this.addPlayableClip(
new ClipUI(
version,
type,
group,
name,
media_id,
clip_uuid,
object_uuid,
object_name,
offset,
end_offset, // length
0,
this.scene.get_object_by_uuid(object_uuid)?.userData["media_file_type"],
),
);
this.checkEditorCanPlay();
this.updateUI();
}
public async deleteKeyFrame(data: Keyframe) {
const keyframe_uuid = data.keyframe_uuid;
const object_uuid = data.object_uuid;
this.transform_engine.clips[object_uuid].removeKeyframe(keyframe_uuid);
this.scene.deletePoint(keyframe_uuid);
for (const element of this.timeline_items) {
if (
element.clip_uuid === keyframe_uuid &&
element.object_uuid === object_uuid
) {
this.timeline_items = this.timeline_items.filter(
(element) =>
!(
element.clip_uuid === keyframe_uuid &&
element.object_uuid === object_uuid
),
);
break;
}
}
this.checkEditorCanPlay();
}
public async updateKeyFrame(data: Keyframe) {
const keyframe_uuid = data.keyframe_uuid;
const keyframe_offset = data.offset;
const object_uuid = data.object_uuid;
const keyframe_pos = data.position;
const keyframe_rot = data.rotation;
const keyframe_scl = data.scale;
this.transform_engine.clips[object_uuid].setOffset(
keyframe_uuid,
keyframe_offset,
);
this.transform_engine.clips[object_uuid].setTransform(
keyframe_uuid,
keyframe_pos as Vector3,
keyframe_rot as Vector3,
keyframe_scl as Vector3,
);
this.scene.updatePoint(
keyframe_uuid,
keyframe_pos as Vector3,
keyframe_rot as Vector3,
);
this.checkEditorCanPlay();
}
public async updateClip(data: ClipUI) {
// only length and offset changes here.
const object_uuid = data.object_uuid;
const media_id = data.media_id;
const offset = data.offset;
const length = data.length + offset;
const clip_uuid = data.clip_uuid;
for (const element of this.timeline_items) {
if (
element.media_id === media_id &&
element.object_uuid === object_uuid &&
element.clip_uuid == clip_uuid
) {
element.length = length;
element.offset = offset;
}
}
this.checkEditorCanPlay();
}
public async deleteClip(data: ClipUI) {
const object_uuid = data.object_uuid;
const media_id = data.media_id;
const clip_uuid = data.clip_uuid;
// Remove the clip from the items array
for (let i = 0; i < this.timeline_items.length; i++) {
const element = this.timeline_items[i];
if (
element.media_id === media_id &&
element.object_uuid === object_uuid &&
element.clip_uuid == clip_uuid
) {
this.timeline_items.splice(i, 1);
break;
}
}
// Once the clip is removed, we need to make sure the engines aren't tracking it either
// TODO: Ideally the engines should just handle this individually and we check them based on clip type/group
// This way we won't need to iterate over all timeline items
if (data.type === ClipType.ANIMATION) {
const object = this.scene.get_object_by_uuid(object_uuid)!;
this.animation_engine.removeAnimation(object, data)
}
this.checkEditorCanPlay();
}
public async mute(data: ClipUI, isMute: boolean) {
this.timeline_items.forEach((element) => {
if (element.group === data.group) {
element.should_play = isMute;
}
});
}
public async addPlayableClip(clip: ClipUI): Promise<void> {
this.timeline_items.push(clip);
}
public async scrub(data: UpdateTime): Promise<void> {
if (this.is_playing) {
return;
}
const value = Math.floor(data.currentTime);
await this.setScrubberPosition(value);
this.current_time = value;
await this.update();
if (this.editorEngine.switchPreviewToggle) {
await this.editorEngine.generateFrame();
}
}
// public streaming events into the timeline from
public async setScrubberPosition(offset: number) {
this.scrubber_frame_position = offset; // in ms
}
// should play from the clip that is closest to the to scrubber
public async play(): Promise<void> {
console.log(`Play - Starting Timeline`);
this.is_playing = true;
}
public async resetScene() {
for (const element of this.timeline_items) {
if (element.type === ClipType.TRANSFORM) {
const object = this.scene.get_object_by_uuid(element.object_uuid);
if (object && this.transform_engine.clips[element.object_uuid]) {
this.transform_engine.clips[element.object_uuid].reset(object);
}
} else if (
element.type === ClipType.AUDIO &&
element.group !== ClipGroup.CHARACTER
) {
this.audio_engine.loadClip(element.media_id);
this.audio_engine.stopClip(element.media_id);
} else if (element.type === ClipType.ANIMATION) {
this.animation_engine.stop();
} else if (
element.type === ClipType.AUDIO &&
element.group === ClipGroup.CHARACTER
) {
this.lipSync_engine.clips[
element.object_uuid + element.media_id
].stop();
this.lipSync_engine.clips[
element.object_uuid + element.media_id
].reset();
} else if (element.type === ClipType.EXPRESSION) {
const object = this.scene.get_object_by_uuid(element.object_uuid);
if (object)
this.emotion_engine.clips[
element.object_uuid + element.media_id
].reset(object);
}
}
}
public getLastClipEnd(): number {
let longest = 0;
for (const element of this.timeline_items) {
if (longest < element.length) {
longest = element.length;
}
}
return longest;
}
public getTimelineEnd(): number {
return filmLength.peek() * 1000;
}
// called by the editor update loop on each frame
public async update(
isRendering = false,
delta_time: number = 0,
): Promise<boolean> {
//if (this.is_playing === false) return; // start and stop
this.timeline_limit = this.getTimelineEnd();
if (this.is_playing) {
this.current_time += delta_time * 1000;
this.pushEvent(fromEngineActions.UPDATE_TIME, {
currentTime: this.current_time,
});
this.scrubber_frame_position = this.current_time;
}
if (this.scrubber_frame_position <= 0) {
for (const video_plane of this.scene.video_planes) {
video_plane.currentTime = 0;
}
await this.resetScene();
}
for (const video_plane of this.scene.video_planes) {
// Caps to 10fps so that the buffering issue is solved and it plays i am not sure how to fix this.
// TODO: Fix buffering and make 30 fps.
const fixedPoint = 1;
let pb = parseFloat(
(this.scrubber_frame_position / this.editorEngine.cap_fps).toFixed(
fixedPoint,
),
);
pb = parseFloat((pb % video_plane.duration).toFixed(fixedPoint));
if (video_plane.currentTime !== pb) {
video_plane.playbackRate = 6;
video_plane.currentTime = pb;
}
}
//this.scrubber_frame_position += 1;
//2. allow stopping.
//3. smallest unit is a frame and it is set by the scene and is in fps, our videos will be 60fps but we can reprocess them using the pipeline.
// Since the animation engine is newer we can just call an evaluation on it instead of having the timeline process it
this.animation_engine.evaluate(this.current_time, this.timeline_limit);
// Iterate over the clips and play them as fit
for (const element of this.timeline_items) {
if (
element.offset <= this.scrubber_frame_position &&
this.scrubber_frame_position <= element.length &&
element.should_play
) {
// run async
// element.play()
// remove the element from the list
const object = this.scene.get_object_by_uuid(element.object_uuid);
if (element.type === ClipType.TRANSFORM) {
if (object && this.transform_engine.clips[element.object_uuid]) {
this.transform_engine.clips[element.object_uuid].step(
object,
element.offset,
this.scrubber_frame_position,
this.scene,
);
element.length =
this.transform_engine.clips[element.object_uuid].length;
}
} else if (
element.type === ClipType.AUDIO &&
element.group !== ClipGroup.CHARACTER &&
this.is_playing &&
!isRendering
) {
// if (this.scrubber_frame_position + 1 >= element.length) {
// this.audio_engine.playClip(element.media_id);
// } else {
// await this.audio_engine.step(
// element.object_uuid + element.media_id,
// this.scrubber_frame_position, element.offset);
// }
this.audio_engine.playClip(element.media_id);
await this.audio_engine.step(
element.media_id,
this.scrubber_frame_position,
element.offset,
);
} else if (
element.type === ClipType.AUDIO &&
element.group === ClipGroup.CHARACTER &&
this.is_playing
) {
if (object) {
await this.lipSync_engine.clips[
element.object_uuid + element.media_id
].play(object);
this.lipSync_engine.clips[
element.object_uuid + element.media_id
].step(this.scrubber_frame_position, element.offset);
}
} else if (element.type === ClipType.EXPRESSION) {
if (object) {
await this.emotion_engine.clips[
object.uuid + element.media_id
].step(this.scrubber_frame_position - element.offset, object);
}
}
}
}
if (
this.scrubber_frame_position >= this.timeline_limit &&
this.is_playing
) {
await this.stop();
return true;
}
return false;
}
private async stop(): Promise<void> {
await this.resetScene();
this.is_playing = false;
console.log(`Stop - Stopping Timeline`);
this.current_time = 0;
this.update();
this.pushEvent(fromEngineActions.UPDATE_TIME, {
currentTime: this.current_time,
});
}
}
@@ -1,474 +0,0 @@
import Editor from "./editor";
import { EditorStates } from "~/pages/PageEnigma/enums";
import { editorState, previewSrc } from "../signals/engine";
import Queue from "~/pages/PageEnigma/Queue/Queue";
import { QueueNames } from "~/pages/PageEnigma/Queue/QueueNames";
import { fromEngineActions } from "~/pages/PageEnigma/Queue/fromEngineActions";
import { ToastTypes, ClipType } from "~/enums";
import { Visibility } from "./api_manager.js";
import * as THREE from "three";
import { getSceneSignals, addToast, startPollingActiveJobs } from "~/signals";
import { v4 as uuidv4 } from "uuid";
import { SceneGenereationMetaData as SceneGenerationMetaData } from "~/pages/PageEnigma/models/sceneGenerationMetadata";
import { MediaUploadApi, StudioGen2Api, VideoApi } from "~/Classes/ApiManager";
import { globalIPAMediaToken } from "../signals";
// TODO THIS CLASS MAKES NO SENSE
// Refactor so we generate all the frames first.
// then pass it through this pipeline as a data structure process it.
// through this class.
interface MediaTokens {
// NB(bt,2025-01-27): This media token was likely called "color" because we were originally
// going to send a shaded "color" render, a depth map render, and an edge detect render to
// the backend for inference.
color: string;
}
export class VideoGeneration {
editor: Editor;
mediaUploadAPI: MediaUploadApi;
videoAPI: VideoApi;
studioGen2Api: StudioGen2Api;
// For cached style Re-Generation
private last_scene_check_sum: string;
// Last Media token IDs
private last_media_tokens: MediaTokens;
// Last toggle position for Render
public last_position_of_preprocessing: boolean;
constructor(editor: Editor) {
this.editor = editor;
this.mediaUploadAPI = new MediaUploadApi();
this.videoAPI = new VideoApi();
this.studioGen2Api = new StudioGen2Api();
this.last_scene_check_sum = "";
this.last_media_tokens = { color: "" };
this.last_position_of_preprocessing = false;
}
sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async videoProcessingPipeline(title: string): Promise<MediaTokens> {
const mediaTokens: MediaTokens = {
color: "",
};
try {
// TODO send the audio a later time.
// const audioClips = this.editor.timeline.timeline_items.filter((clip) => {
// return clip.type == ClipType.AUDIO;
// });
if (!this.editor.engineFrameBuffers.payloadZip) {
await this.handleError(
`Video Processing Failed Please Try Again (zip process failed)`,
3000,
);
}
const zipBlob = this.editor.engineFrameBuffers.payloadZip;
if (!zipBlob) {
await this.handleError(`"Payload blob is null."`, 3000);
}
const video_upload_response = await this.mediaUploadAPI.UploadStudioShot({
maybe_title: title,
uuid_idempotency_token: uuidv4(),
blob: zipBlob,
fileName: `${title}.zip`,
maybe_visibility: Visibility.Public,
});
if (!video_upload_response.data) {
console.log("Missing media_tokens.color Token");
const error = {
message: "Missing Data From Video Upload Response",
code: 1,
};
throw error;
}
mediaTokens.color = video_upload_response.data;
console.log(`Studio recording: https://storyteller.ai/media/${mediaTokens.color}`);
} finally {
}
return mediaTokens;
}
// this is to generate previews
async generateFrame() {
if (!this.editor.generating_preview && this.editor.rawRenderer) {
this.editor.generating_preview = true;
this.editor.utils.removeTransformControls();
this.editor.activeScene.renderMode(true);
if (this.editor.activeScene.hot_items) {
this.editor.activeScene.hot_items.forEach((element) => {
element.visible = false;
});
}
previewSrc.value = "";
this.editor.rawRenderer.setSize(
this.editor.render_width,
this.editor.render_height,
);
this.editor.render_camera.aspect =
this.editor.render_width / this.editor.render_height;
this.editor.render_camera.updateProjectionMatrix();
this.editor.rawRenderer.render(
this.editor.activeScene.scene,
this.editor.render_camera,
);
const imgData = this.editor.rawRenderer.domElement.toDataURL();
const response = await fetch(imgData); // Fetch the data URL
const blob = await response.blob(); // Convert to Blob
if (!this.editor.camera_person_mode) {
this.editor.switchCameraView();
editorState.value = EditorStates.PREVIEW;
}
this.editor.generating_preview = false;
try {
const url = await this.editor.api_manager.uploadMediaFrameGeneration(
blob,
"render.png",
this.editor.art_style,
this.editor.positive_prompt,
this.editor.negative_prompt,
);
previewSrc.value = url;
return Promise.resolve(url);
} catch (err: unknown) {
const errorMessage =
err instanceof Error
? err.message
: "Unknown Error in Generate Frame";
Queue.publish({
queueName: QueueNames.FROM_ENGINE,
action: fromEngineActions.POP_A_TOAST,
data: {
type: ToastTypes.ERROR,
message: errorMessage,
},
});
}
}
}
// to determine if we should capture frames again or not.
async shouldRenderScenesAgain(checkSumData: string): Promise<boolean> {
// Should skip to rendering the scene if already processed the frames inputs for running a new style.
if (this.last_scene_check_sum === "") {
// no token means input was not processed so lets go do the video re-render process
this.last_scene_check_sum = checkSumData;
return false;
} else {
if (this.last_scene_check_sum === checkSumData) {
return true;
} else {
this.last_scene_check_sum = checkSumData;
return false;
}
}
}
async stopPlaybackAndUploadVideo() {
this.editor.rendering = false; // not sure about this variable ... this has so many state issues.
// precondition checks, if we have no frames then we shouldn't try to do generations or snapshots
// This means no frames so error out
this.editor.generating_preview = true;
await this.editor.updateLoad({
progress: 50,
message:
"Please stay on this screen and do not switch tabs! while your video is being processed.",
});
let media_tokens: MediaTokens = {
color: "",
};
// properties used by the external scope to the for loop.
const title = getSceneSignals().title || "Untitled";
const style_name = this.editor.art_style.toString();
const media_token = this.editor.current_scene_media_token || undefined;
// TAKE A snap shot of the scene then use this token across all videos
// convert the ip adapter image and upload as a media token
const image_uuid = uuidv4();
let ipa_image_token = undefined;
if (this.editor.globalIpAdapterImage != undefined) {
const response = await this.mediaUploadAPI.UploadImage({
fileName: `${image_uuid}.ipa`,
blob: this.editor.globalIpAdapterImage,
uuid: image_uuid,
});
if (!response.success) {
// ip adapter upload failed
await this.handleError(
"Reference Image Upload Failed Try Generating Movie Again.",
3000,
);
return;
}
if (!response.data) {
console.log("IPA Image Token Missing");
}
ipa_image_token = response.data;
}
// update the signal with this information
if (ipa_image_token) {
globalIPAMediaToken.value = ipa_image_token;
}
// TODO Remove so many of these around wtf. SceneGenereationMetaData should only be one place
const metaData: SceneGenerationMetaData = {
artisticStyle: this.editor.art_style,
positivePrompt: this.editor.positive_prompt,
negativePrompt: this.editor.negative_prompt,
cameraAspectRatio: this.editor.render_camera_aspect_ratio,
upscale: this.editor.generation_options.upscale,
faceDetail: this.editor.generation_options.faceDetail,
styleStrength: this.editor.generation_options.styleStrength,
lipSync: this.editor.generation_options.lipSync,
cinematic: this.editor.generation_options.cinematic,
globalIPAMediaToken: ipa_image_token,
enginePreProcessing: this.editor.engine_preprocessing, // the only thing that will invalidate the cache
};
// for the one case where the engine preprecessing is turned we need to cache this.
this.last_position_of_preprocessing = this.editor.engine_preprocessing;
// This is to save the snapshot of the scene for remixing
const uuid_snapshot = uuidv4();
// Save the scene
const saveData = await this.editor.save_manager.saveData({
sceneTitle: title,
sceneToken: media_token,
sceneGenerationMetadata: metaData,
});
const file = new File([saveData], `${title}.glb`, {
type: "application/json",
});
const response =
await this.editor.media_upload.UploadSceneSnapshotMediaFileForm({
maybe_title: title,
maybe_scene_source_media_file_token: media_token, // can be undefined or null
uuid: uuid_snapshot,
blob: file,
});
let immutable_media_token = undefined;
if (response.success) {
if (response.data) {
immutable_media_token = response.data;
}
} else {
await this.handleError(
"Scene Snapshot Failed Try Generating Movie Again.",
3000,
);
return;
}
console.log(`Immutable Snapshot Token: ${immutable_media_token}`);
await this.editor.updateLoad({
progress: 50,
message:
"Please stay on this screen and do not switch tabs! while your video is being processed.",
});
// write the end point to upload the video
// upload the color frames with end point.
try {
media_tokens = await this.videoProcessingPipeline(title);
} catch (error) {
await this.handleError(`Video Processing Failed Please Try Again`, 3000);
throw error;
}
this.editor.onWindowResize();
this.editor.setColorMap();
await this.editor.updateLoad({
progress: 100,
message: "Done Check Your Movies Tab On Profile.",
label: "Success",
});
// this is so that its a check point just encase enqueue fails, if it does we can still restylize
this.last_media_tokens = media_tokens;
// Legacy studio:
//await this.handleEnqueue(media_tokens, ipa_image_token ?? "");
// NB(bt,2025-01-27): We're using a hack to set the first frame (in CharacterFrameButton)
const firstFrameMediaToken = (window as any).firstFrameMediaToken;
const videoMediaToken = media_tokens.color;
console.log(`Enqueue Studio Gen2 Render; Video: ${videoMediaToken} Image: ${firstFrameMediaToken}`);
await this.handleEnqueueStudioGen2(videoMediaToken, firstFrameMediaToken);
await this.EndLoadingState();
}
async handleCachedEnqueue() {
const enqueue_studio_response = await this.videoAPI.EnqueueStudio({
enqueueVideo: {
disable_lcm: false,
enable_lipsync: this.editor.generation_options.lipSync,
input_file: this.last_media_tokens.color,
negative_prompt: this.editor.negative_prompt,
prompt: this.editor.positive_prompt,
remove_watermark: false,
style: this.editor.art_style.toString(),
frame_skip: 2,
travel_prompt: "",
trim_end_millis: 7000,
trim_start_millis: 0,
use_cinematic: this.editor.generation_options.cinematic,
use_face_detailer: this.editor.generation_options.faceDetail,
use_strength: this.editor.generation_options.styleStrength,
use_upscaler: this.editor.generation_options.upscale,
uuid_idempotency_token: uuidv4(),
global_ipa_media_token: globalIPAMediaToken.value ?? "",
input_depth_file: "",
input_normal_file: "",
input_outline_file: "",
creator_set_visibility: Visibility.Public,
},
});
if (enqueue_studio_response.success) {
console.log("Start Polling Active Jobs");
startPollingActiveJobs();
addToast(
ToastTypes.SUCCESS,
"Done Check Your Movies Tab On Profile.",
3000,
);
} else {
addToast(ToastTypes.ERROR, "Failed To Process Movie Try Again", 3000);
}
}
async handleEnqueue(upload_tokens: MediaTokens, ipa_image_token: string) {
const enqueue_studio_response = await this.videoAPI.EnqueueStudio({
enqueueVideo: {
disable_lcm: false,
enable_lipsync: this.editor.generation_options.lipSync,
input_file: upload_tokens.color,
negative_prompt: this.editor.negative_prompt,
prompt: this.editor.positive_prompt,
remove_watermark: false,
style: this.editor.art_style.toString(),
frame_skip: 2,
travel_prompt: "",
trim_end_millis: 7000,
trim_start_millis: 0,
use_cinematic: this.editor.generation_options.cinematic,
use_face_detailer: this.editor.generation_options.faceDetail,
use_strength: this.editor.generation_options.styleStrength,
use_upscaler: this.editor.generation_options.upscale,
uuid_idempotency_token: uuidv4(),
global_ipa_media_token: ipa_image_token,
input_depth_file: "",
input_normal_file: "",
input_outline_file: "",
creator_set_visibility: Visibility.Public,
},
});
if (enqueue_studio_response.success) {
startPollingActiveJobs();
} else {
await this.handleError("Failed To Process Movie Try Again", 3000);
return;
}
this.handleSuccess("Done Check Your Movies Tab On Profile.", 3000);
}
async handleEnqueueStudioGen2(videoMediaToken: string, firstFrameImageToken: string) {
const response = await this.studioGen2Api.EnqueueStudioGen2({
enqueueVideo: {
uuid_idempotency_token: uuidv4(),
image_file: firstFrameImageToken,
video_file: videoMediaToken,
creator_set_visibility: Visibility.Public,
},
});
if (response.success) {
startPollingActiveJobs();
} else {
await this.handleError("Failed To Process Animation Try Again", 3000);
return;
}
this.handleSuccess("Done Check Your Movies Tab On Profile.", 3000);
}
async handleSuccess(message: string, timeout: number) {
addToast(ToastTypes.SUCCESS, message, timeout);
}
async handleError(message: string, timeout: number) {
addToast(ToastTypes.ERROR, message, timeout);
await this.EndLoadingState();
}
async EndLoadingState() {
this.editor.generating_preview = false;
this.editor.endLoading();
this.editor.onWindowResize();
this.editor.recorder = undefined;
if (this.editor.rawRenderer) {
this.editor.rawRenderer.setSize(
this.editor.startRenderWidth,
this.editor.startRenderHeight,
);
}
this.editor.camViewCanvasMayReset();
this.editor.rawRenderer = new THREE.WebGLRenderer({
antialias: true,
canvas: this.editor.canvasRenderCamReference || undefined,
preserveDrawingBuffer: true,
});
this.editor._configurePostProcessingRaw();
this.editor.activeScene.renderMode(false);
this.editor.switchEdit();
}
}
@@ -1,58 +0,0 @@
import { SoundRegistry, SoundEffect } from "@storyteller/soundboard";
// TODO: This is maintained in two places. Here and MiscSettingsPane.
export const InstallSounds = () => {
const r = SoundRegistry.getInstance();
if (r.hasSound("decline_normal")) {
return;
}
// Good for simple menu choices
r.setSoundOnce("click", new SoundEffect("resources/sound/smrpg_click.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("scifi_menu_beep_1", new SoundEffect("resources/sound/metroidprime_UI_15.wav", { defaultVolume: 0.3 }));
r.setSoundOnce("scifi_menu_beep_2", new SoundEffect("resources/sound/metroidprime_UI_14.wav", { defaultVolume: 0.3 }));
r.setSoundOnce("scifi_menu_select", new SoundEffect("resources/sound/metroidprime_UI_18.wav", { defaultVolume: 0.3 }));
// Good for simple immediate enqueue success
r.setSoundOnce("done", new SoundEffect("resources/sound/oot_dialogue_done.wav", { defaultVolume: 0.4 })); // DEFAULT
// Good for simple immediate failure
r.setSoundOnce("error_chirp", new SoundEffect("resources/sound/goldensun_135.wav", { defaultVolume: 0.4 }));
r.setSoundOnce("spike_throw", new SoundEffect("resources/sound/smrpg_enemy_spikethrow.wav", { defaultVolume: 0.1 })); // DEFAULT
r.setSoundOnce("giant_shell_kick", new SoundEffect("resources/sound/smrpg_mario_giantshellkick.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("wrong", new SoundEffect("resources/sound/smrpg_wrong.wav", { defaultVolume: 0.4 }));
// Good for simple async success
r.setSoundOnce("special_flower", new SoundEffect("resources/sound/smrpg_specialflower.wav", { defaultVolume: 0.2 })); // DEFAULT
r.setSoundOnce("extra_power", new SoundEffect("resources/sound/smrpg_character_extrapower.wav", { defaultVolume: 0.2 })); // DEFAULT
// Good for async errors
r.setSoundOnce("crumble", new SoundEffect("resources/sound/smrpg_drybones_crumble.wav", { defaultVolume: 0.1 })); // DEFAULT
r.setSoundOnce("ghost", new SoundEffect("resources/sound/smrpg_ghost.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("special_alert", new SoundEffect("resources/sound/goldensun_214.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("scifi_alert", new SoundEffect("resources/sound/metroidprime_UI_52.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("scifi_shrill_alert", new SoundEffect("resources/sound/metroidprime_UI_51.wav", { defaultVolume: 0.2 }));
// Good for menus
r.setSoundOnce("next", new SoundEffect("resources/sound/oot_dialogue_next.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("select", new SoundEffect("resources/sound/goldensun_111.wav", { defaultVolume: 0.4 }));
r.setSoundOnce("scifi_menu_open", new SoundEffect("resources/sound/metroidprime_UI_12.wav", { defaultVolume: 0.3 }));
r.setSoundOnce("scifi_menu_close", new SoundEffect("resources/sound/metroidprime_UI_13.wav", { defaultVolume: 0.3 }));
// Good for special reward / celebration
r.setSoundOnce("correct", new SoundEffect("resources/sound/smrpg_correct.wav", { defaultVolume: 0.1 }));
r.setSoundOnce("flower", new SoundEffect("resources/sound/smrpg_flower.wav", { defaultVolume: 0.1 }));
// File delete, trash, etc.
r.setSoundOnce("trash", new SoundEffect("resources/sound/oot_scrub_crumble.wav", { defaultVolume: 0.4 }));
// Misc / ungrouped
r.setSoundOnce("accept_chirp", new SoundEffect("resources/sound/goldensun_101.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("accept_normal_level_1", new SoundEffect("resources/sound/goldensun_173.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("accept_normal_level_2", new SoundEffect("resources/sound/goldensun_174.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("accept_normal_level_3", new SoundEffect("resources/sound/goldensun_175.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("decline_chirp", new SoundEffect("resources/sound/goldensun_102.wav", { defaultVolume: 0.2 }));
r.setSoundOnce("decline_normal", new SoundEffect("resources/sound/goldensun_113.wav", { defaultVolume: 0.2 }));
(window as any).sounds = r;
}
@@ -1,127 +0,0 @@
import { fromEngineActions } from "~/pages/PageEnigma/Queue/fromEngineActions";
import { toEngineActions } from "~/pages/PageEnigma/Queue/toEngineActions";
import { CameraAspectRatio } from "../enums";
import {
QueueClip,
QueueKeyframe,
UpdateTime,
MediaItem,
} from "~/pages/PageEnigma/models";
import { toTimelineActions } from "./toTimelineActions";
import { ClipUI } from "../clips/clip_ui";
import { IGenerationOptions } from "../models/generationOptions";
import { ToastDataType } from "~/components";
import { QueueNames } from "./QueueNames";
export type UnionedActionTypes =
| fromEngineActions
| toEngineActions
| toTimelineActions;
export type UnionedDataTypes =
| QueueClip
| UpdateTime
| QueueKeyframe
| ClipUI[]
| MediaItem
| ToastDataType
| IGenerationOptions
| CameraAspectRatio
| null;
export type QueueSubscribeType = {
action: UnionedActionTypes;
data: UnionedDataTypes;
};
export type SubscribeHandler = (entry: {
action: UnionedActionTypes;
data: UnionedDataTypes;
}) => void;
class Queue {
private _queues: Record<
QueueNames,
Array<{
action: UnionedActionTypes;
data: UnionedDataTypes;
}>
> = {
[QueueNames.FROM_ENGINE]: [],
[QueueNames.TO_ENGINE]: [],
[QueueNames.TO_TIMELINE]: []
};
private _subscribers: Record<
QueueNames,
Map<
string,
SubscribeHandler
>
> = {
[QueueNames.FROM_ENGINE]: new Map<string, SubscribeHandler>(),
[QueueNames.TO_ENGINE]: new Map<string, SubscribeHandler>(),
[QueueNames.TO_TIMELINE]: new Map<string, SubscribeHandler>(),
};
// Enqueues actions to a specified queue
public publish({
queueName,
action,
data,
}: {
queueName: QueueNames;
action: UnionedActionTypes;
data: UnionedDataTypes;
}) {
// Queues for all QueueNames should be initialized above but this is a safeguard for future enum additions
// Make sure the queue exists
if (!this._queues[queueName]) {
this._queues[queueName] = [];
}
// Queue the task in the right queue
this._queues[queueName].push({ action, data });
// TODO: Ideally the publisher shouldn't deal at all with the subscribers
// It should only publish the events to the right queue
// It also doesn't make sense to fire an event on publish because the operations are then dispersed
// leading to potential race conditions
// FIXME: Add some sort of queue loop using internal event and add queue locks using promises to serialize queue publishes
if (this._subscribers[queueName]) {
this._subscribers[queueName].forEach((subcribeHandler) =>
subcribeHandler(this._queues[queueName][0]),
);
this._queues[queueName].shift();
}
}
public subscribe(
queueName: QueueNames,
id: string,
onMessage: (entry: QueueSubscribeType) => void,
) {
// Queues for all QueueNames should be initialized above but this is a safeguard for future enum additions
// Make sure the queue exists
if (!this._subscribers[queueName]) {
this._subscribers[queueName] = new Map<string, SubscribeHandler>();
}
// Set the handler mapped to the ID
this._subscribers[queueName].set(id, onMessage);
while (this._queues[queueName]?.length) {
this._subscribers[queueName].forEach((subscribeHandler) =>
subscribeHandler(this._queues[queueName][0]),
);
this._queues[queueName].shift();
}
}
}
const queue = new Queue();
export default queue;
@@ -1,5 +0,0 @@
export enum QueueNames {
FROM_ENGINE = "FROM_ENGINE",
TO_ENGINE = "TO_ENGINE",
TO_TIMELINE = "TO_TIMELINE",
}
@@ -1,14 +0,0 @@
export enum fromEngineActions {
ADD_OBJECT = "add_object",
CAMERA_ASPECT_RATIO_CHANGED = "camera_aspect_ratio_changed",
// data = CameraAspectRatio
DELETE_OBJECT = "delete_object",
DESELECT_OBJECT = "deselect_object",
RESET_TIMELINE = "reset_timeline",
SELECT_OBJECT = "select_object",
UPDATE_CHARACTER_ID = "update_character_id",
UPDATE_TIME = "update_time",
UPDATE_TIME_LINE = "update_timeline",
POP_A_TOAST = "pop_a_toast",
// data={currentTime: number}
}
@@ -1,3 +0,0 @@
import Queue from "./Queue";
export default Queue;
export * from "./QueueNames";
@@ -1,47 +0,0 @@
export enum toEngineActions {
ADD_CLIP = "add_clip",
// data=QueueClip
ADD_KEYFRAME = "add_keyframe",
ADD_CHARACTER = "add_character",
ADD_OBJECT = "add_object",
ADD_SHAPE = "add_shape",
// data= QueueKeyframe
DELETE_CLIP = "delete_clip",
// data=QueueClip
DELETE_KEYFRAME = "delete_keyframe",
// data=QueueKeyframe
ENTER_EDIT_STATE = "enter_edit",
// data=null
ENTER_PREVIEW_STATE = "enter_preview",
// data=null
GENERATE_VIDEO = "generate_video",
// data=null
MUTE = "mute",
// data={version, type, group, object_uuid} if lipsync
// data={version, type, group} if global_audio
REFRESH_PREVIEW = "refresh_preview",
// data=QueueClip
TOGGLE_CAMERA_STATE = "toggle_camera",
// data=null
TOGGLE_REPEATING = "toggle_repeating",
// data=null
CHANGE_CAMERA_ASPECT_RATIO = "change_camera_aspect_ratio",
// data=CamperaAspectRatio
CAMERA_CHANGED = "camera_changed",
UNMUTE = "unmute",
// data={version, type, group, object_uuid} if lipsync
// data={version, type, group} if global_audio
UPDATE_CLIP = "update_clip",
// data=QueueClip
UPDATE_KEYFRAME = "update_keyframe",
// data=QueueKeyframe
UPDATE_TIME = "update_time",
// data={currentTime: number}
}
@@ -1,3 +0,0 @@
export enum toInferenceActions {
ADD_TTS_JOB = "add_tts_job"
}
@@ -1,4 +0,0 @@
export enum toTimelineActions {
ADD_KEYFRAME = "add_keyframe",
GENERATE_PROGRESS = "generate_progress",
}
@@ -1,7 +0,0 @@
export default interface IAnimationClip {
speed: number;
duration: number;
startTime: number;
endTime: number;
evaluate: (time: number) => void;
}
@@ -1,348 +0,0 @@
import * as THREE from "three";
import { GLTFLoader } from "three/addons/loaders/GLTFLoader.js";
import {
FBXLoader,
MMDAnimationHelper,
MMDAnimationHelperMixer,
} from "three/examples/jsm/Addons.js";
import { MMDLoader } from "three/addons/loaders/MMDLoader.js";
import { MoveAIResult, Retarget } from "../Editor/retargeting";
import IAnimationClip from "./ClipInterfaces/IAnimationClip";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import { StorytellerApiHostStore } from "@storyteller/api";
export class AnimationClip implements IAnimationClip {
version: number;
media_id: string; // comes from the server
object_uuid: string;
type: "animation";
location: "glb" | "remote";
speed: number;
duration: number;
startTime: number;
length: number;
clip_name: string;
mixer: THREE.AnimationMixer | undefined;
animation_clip: THREE.AnimationClip | undefined;
clip_action: THREE.AnimationAction | undefined;
special_properties: MoveAIResult[];
retargeted: boolean;
last_frame: number;
private isMMD: boolean = false;
private obj: MMDAnimationHelperMixer | undefined;
private helper: MMDAnimationHelper | undefined;
private mmd_anim_clip: THREE.AnimationClip | undefined;
private pose: Object | undefined;
private processingClip: boolean = false;
private mmd_url: string = "";
constructor(
version: number,
media_id: string,
location: "glb" | "remote",
object_uuid: string,
speed: number,
length: number,
clip_name: string,
startTime: number,
duration: number,
) {
this.version = version;
this.media_id = media_id;
this.type = "animation";
this.object_uuid = object_uuid;
this.location = location;
this.speed = speed;
this.length = length;
this.startTime = startTime;
this.duration = duration;
this.clip_name = clip_name;
this.animation_clip;
this.mixer;
this.clip_action;
this.special_properties = [];
this.retargeted = false;
this.last_frame = 0;
}
get endTime() {
return this.startTime + this.duration
}
evaluate: (time: number) => void;
async get_media_url() {
//This is for prod when we have the proper info on the url.
const apiSchemeAndHost = StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
const url = `${apiSchemeAndHost}/v1/media_files/file/${this.media_id}`;
const response = await fetch(url);
const json = await JSON.parse(await response.text());
const bucketPath = json["media_file"]["public_bucket_path"];
const media_api_base_url = GetCdnOrigin();
const media_url = `${media_api_base_url}${bucketPath}`;
return media_url;
}
_load_animation(): Promise<THREE.AnimationClip> {
// Return the promise chain starting from `this.get_media_url()`
return this.get_media_url().then((url) => {
// Return a new Promise that resolves with the animation clip
return new Promise((resolve) => {
if (url.includes(".glb")) {
const glbLoader = new GLTFLoader();
glbLoader.load(url, (gltf) => {
// Assuming the animation is the first one in the animations array
const animationClip = gltf.animations[0];
resolve(animationClip);
});
} else if (url.includes(".fbx")) {
const fbxLoader = new FBXLoader();
fbxLoader.load(url, (fbx) => {
const animationClip = fbx.animations[0];
this.retargeted = true;
animationClip.tracks.forEach((track) => {
const retarget = new Retarget();
const retarget_value = retarget.retarget(track.name);
track.name = retarget_value.bone;
console.log(track);
if (retarget_value.is_special) {
this.special_properties.push(retarget_value);
}
});
resolve(animationClip);
});
} else if (url.includes(".vmd")) {
console.log("VMD Loader");
const root = this.mixer?.getRoot();
this.isMMD = true;
if (root) {
const mmdLoader = new MMDLoader();
this.mmd_url = url;
mmdLoader.loadAnimation(url, root as THREE.SkinnedMesh, (mmd) => {
mmd.name = this.mmd_url;
resolve(mmd as THREE.AnimationClip);
});
console.log("Loaded");
}
console.log("Dont");
}
});
});
}
_create_mixer(object: THREE.Object3D) {
this.mixer = new THREE.AnimationMixer(object);
return this.mixer;
}
async _get_clip() {
if (this.animation_clip == null && this.mixer !== null) {
console.log(this.media_id);
if (this.media_id === "SelfClip") {
const rootObject = this.mixer?.getRoot();
if (rootObject)
this.animation_clip = (
rootObject as THREE.Object3D<THREE.Object3DEventMap>
).animations[0];
} else {
this.animation_clip = await this._load_animation();
}
}
return this.animation_clip;
}
async play(object: THREE.Object3D) {
if (this.mixer == null) {
this.mixer = this._create_mixer(object);
const anim_clip = await this._get_clip();
if (anim_clip == undefined) {
return;
}
this.clip_action = this.mixer?.clipAction(anim_clip);
if (this.clip_action && this.isMMD === false) {
if (this.clip_action?.isRunning() == false) {
this.clip_action.play();
}
}
}
}
update_bones() {
if (this.retargeted === false) {
return;
}
const rootObject = this.mixer?.getRoot();
if (rootObject)
for (
let index_ = 0;
index_ <
(rootObject as THREE.Object3D<THREE.Object3DEventMap>).children.length;
index_++
) {
const child_holder = (
rootObject as THREE.Object3D<THREE.Object3DEventMap>
).children[index_];
if (child_holder.type == "Bone") {
child_holder.traverse(
(bone: THREE.Object3D<THREE.Object3DEventMap>) => {
for (
let index__ = 0;
index__ < this.special_properties.length;
index__++
) {
const property = this.special_properties[index__];
if (property.bone == bone.name + ".quaternion") {
const quat_y = THREE.MathUtils.degToRad(property.y);
bone.rotateY(quat_y);
if (property.only_y == false) {
const quat_x = THREE.MathUtils.degToRad(property.x);
bone.rotateX(quat_x);
const quat_z = THREE.MathUtils.degToRad(property.z);
bone.rotateZ(quat_z);
} else {
if (child_holder.parent) {
child_holder.parent.rotation.x =
THREE.MathUtils.degToRad(-90);
}
}
if (property.flip) {
bone.rotateX(bone.rotation.x * 2 * -1);
}
} else if (property.bone == bone.name + ".position") {
bone.position.set(
bone.position.x * property.x,
bone.position.y * property.y,
bone.position.z * property.z,
);
}
}
},
);
}
}
}
private async load_vpd_wrapped(): Promise<object> {
return new Promise((resolve) => {
const poseLoader = new MMDLoader();
poseLoader.loadVPD("/resources/pose/7.vpd", false, (vpd) => {
resolve(vpd);
}); // End of loadVPD call
}); // End of new Promise
}
async animate(deltatime: number) {
// Sets the frame of the animation mixer.
this.mixer?.setTime(deltatime);
// Regtargeting for Move.AI
this.update_bones();
// Next
if (this.isMMD && this.mixer) {
if (this.obj === undefined) {
// Sets the ik helper and object for IK and animation.
this.helper = (this.mixer.getRoot() as THREE.Object3D).parent?.userData[
"helper"
];
}
this.obj = this.helper?.objects.get(
this.mixer?.getRoot() as THREE.SkinnedMesh,
);
const mesh = this.mixer?.getRoot() as THREE.SkinnedMesh;
if (this.helper) {
this.helper._restoreBones(mesh); // Privateish js function for resetting bones.
this.obj?.mixer?.setTime(deltatime); // Sets the time.
this.helper._saveBones(mesh); // Saves the bones location for restoration later.
mesh.updateMatrixWorld(true); // Updates mesh.
this.obj?.ikSolver.update(); // Updates IK Solver.
this.obj?.grantSolver.update(); // Updates Grant which moves the bones.
}
// Changing of animation.
if (
this.obj &&
this.helper &&
this.processingClip === false &&
this.obj.mixer &&
this.mixer._actions[0]._clip.name !==
this.obj.mixer._actions[0]._clip.name &&
this.mixer !== undefined
) {
// On animation change.
this.processingClip = true;
const helper_mmd = this.helper.objects.get(mesh);
if (helper_mmd?.mixer !== undefined) {
const clips = helper_mmd.mixer._actions;
clips.forEach((clip) => {
this.helper?.objects.delete(clip._clip);
});
}
this.helper.remove(mesh);
mesh.pose();
const loader = new MMDLoader();
loader.loadAnimation(this.mmd_url, mesh, (mmd) => {
mmd.name = this.mmd_url;
this.helper?.add(mesh, {
animation: mmd,
physics: false,
});
this.processingClip = false;
if (helper_mmd?.mixer !== undefined) {
const clips = helper_mmd.mixer._actions;
clips.forEach((clip) => {
this.helper?.objects.delete(clip._clip);
});
}
this.obj.mixer = this.mixer;
});
}
}
}
async step(deltatime: number, isPlaying: boolean, frame: number) {
if (this.mixer == null) {
return;
}
if (this.retargeted) {
if (isPlaying || Math.floor(frame) != this.last_frame) {
if (Math.floor(frame) != 0) {
await this.animate(deltatime);
}
}
} else {
await this.animate(deltatime);
}
this.last_frame = frame;
}
stop() {
this.mixer?.setTime(0);
}
toJSON() {
return {
version: this.version,
media_id: this.media_id,
object_uuid: this.object_uuid,
type: this.type,
speed: this.speed,
length: this.length,
clip_name: this.clip_name,
};
}
}
@@ -1,69 +0,0 @@
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import Ijson from "~/interfaces/Ijson";
import { StorytellerApiHostStore } from "@storyteller/api";
interface AudioDataInterface {
audioContext: AudioContext;
audioBuffer: AudioBuffer;
}
class AudioData implements AudioDataInterface {
audioContext: AudioContext;
audioBuffer: AudioBuffer;
source: AudioBufferSourceNode | undefined;
constructor(audioContext: AudioContext, audioBuffer: AudioBuffer) {
this.audioContext = audioContext;
this.audioBuffer = audioBuffer;
}
}
export class AudioClip implements Ijson {
version: number;
media_id: string;
type: "audio";
volume: number;
audio_data: AudioData | undefined;
constructor(version: number, media_id: string, volume: number) {
this.version = version;
this.media_id = media_id;
this.type = "audio";
this.volume = volume;
this.download_audio().then((data) => {
this.audio_data = data;
});
}
async get_media_url() {
//This is for prod when we have the proper info on the url.
const apiSchemeAndHost = StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
const url = `${apiSchemeAndHost}/v1/media_files/file/${this.media_id}`;
const response = await fetch(url);
const json = await JSON.parse(await response.text());
const bucketPath = json["media_file"]["public_bucket_path"];
const media_api_base_url = GetCdnOrigin();
const media_url = `${media_api_base_url}${bucketPath}`;
return media_url;
}
async download_audio() {
const url = await this.get_media_url();
const audioContext = new AudioContext();
const response = await fetch(url);
const arrayBuffer = await response.arrayBuffer();
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
return new AudioData(audioContext, audioBuffer);
}
toJSON() {
return {
version: this.version,
media_id: this.media_id,
type: this.type,
volume: this.volume,
};
}
}
@@ -1,71 +0,0 @@
// Clip offsets represent the state of the clip on the timeline as well as what type of clip it is.
// it is created from a media id.
import { MediaFileType } from "~/pages/PageEnigma/enums";
import { ClipGroup, ClipType } from "~/enums";
import Ijson from "~/interfaces/Ijson";
export class ClipUI implements Ijson {
version: number;
type: ClipType;
group: ClipGroup;
name: string;
media_id: string;
object_uuid: string;
object_name: string;
offset: number; // in frames
length: number; // in frames
should_play: boolean;
clip_uuid: string;
keyframe_offset: number;
media_file_type: MediaFileType;
constructor(
version: number,
type: ClipType,
group: ClipGroup,
name: string,
media_id: string,
clip_uuid: string,
object_uuid: string,
object_name: string,
offset: number,
length: number,
keyframe_offset: number = 0,
media_file_type: MediaFileType,
) {
this.version = version;
this.group = group; // Only needed for UI
this.clip_uuid = clip_uuid;
this.name = name; // UI
this.type = type; // UI and Animation / Audio / Lipsync / : Engine
this.object_uuid = object_uuid; // Animation / Audio / Lipsync / : Engine
this.object_name = object_name;
this.media_id = media_id; // Animation / Audio / Lipsync / : Engine
this.offset = offset;
this.length = length;
this.should_play = true;
this.keyframe_offset = keyframe_offset;
this.media_file_type = media_file_type;
}
toJSON() {
return {
version: this.version,
group: this.group,
name: this.name,
type: this.type,
clip_uuid: this.clip_uuid,
object_uuid: this.object_uuid,
object_name: this.object_name,
media_id: this.media_id,
start_offset: this.offset,
ending_offset: this.length,
keyframe_offset: this.keyframe_offset,
media_file_type: this.media_file_type,
};
}
}
@@ -1,143 +0,0 @@
import * as THREE from "three";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import Ijson from "~/interfaces/Ijson";
import { StorytellerApiHostStore } from "@storyteller/api";
interface CsvJson {
[key: string]: string[];
}
export class EmotionClip implements Ijson {
version: number;
media_id: string;
type: "expression";
emotion_json: any;
faces: THREE.Mesh[];
constructor(version: number, media_id: string) {
this.version = version;
this.media_id = media_id;
this.type = "expression";
this.faces = [];
this.download_csv().then((data) => {
this.emotion_json = data;
});
}
async get_media_url() {
//This is for prod when we have the proper info on the url.
const apiSchemeAndHost = StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
const url = `${apiSchemeAndHost}/v1/media_files/file/${this.media_id}`;
const response = await fetch(url);
const json = await JSON.parse(await response.text());
const bucketPath = json["media_file"]["public_bucket_path"];
const media_api_base_url = GetCdnOrigin();
const media_url = `${media_api_base_url}${bucketPath}`;
return media_url;
}
async download_json() {
const url = await this.get_media_url();
const response = await fetch(url);
return await response.json();
}
async download_csv(): Promise<CsvJson> {
const url = await this.get_media_url();
const response = await fetch(url);
const csvText = await response.text();
// Parsing the CSV text
const rows = csvText.trim().split("\n");
const header = rows[0].split(",");
const jsonData: CsvJson = {};
header.forEach((key) => (jsonData[key] = []));
for (let i = 1; i < rows.length; i++) {
const values = rows[i].split(",");
values.forEach((value, index) => {
jsonData[header[index]].push(value);
});
}
return jsonData;
}
async _detect_face(object: THREE.Object3D): Promise<THREE.Mesh> {
this.faces = [];
return new Promise((resolve) => {
object.traverse((c: THREE.Object3D) => {
if (c instanceof THREE.Mesh) {
if (c.morphTargetInfluences && c.morphTargetDictionary) {
const blendShapeIndexE = c.morphTargetDictionary["jawOpen"];
if (blendShapeIndexE !== undefined) {
this.faces.push(c);
resolve(c);
}
}
}
});
});
}
setBlends(shapes: { [key: string]: number }) {
this.faces.forEach((element: THREE.Mesh) => {
if (element.morphTargetInfluences && shapes) {
Object.keys(shapes).forEach((key) => {
let index = element.morphTargetDictionary?.[key];
if (index === undefined) {
index =
element.morphTargetDictionary?.[
key.charAt(0).toLowerCase() + key.slice(1)
];
}
if (
typeof index === "number" &&
element.morphTargetInfluences !== undefined
) {
element.morphTargetInfluences[index] = shapes[key];
}
});
}
});
}
async reset(object: THREE.Object3D) {
await this._detect_face(object);
const keys: { [key: string]: number } = {};
if (this.emotion_json === undefined || this.emotion_json === null) {
return;
}
Object.keys(this.emotion_json).forEach((key) => {
keys[key] = 0;
});
this.setBlends(keys);
}
async step(frame: number, object: THREE.Object3D) {
if (this.faces.length <= 0) {
await this._detect_face(object);
}
if (this.emotion_json === undefined || this.emotion_json === null) {
return;
}
const keys: { [key: string]: number } = {};
Object.keys(this.emotion_json).forEach((key) => {
if (Math.floor(frame) > this.emotion_json[key].length) {
return;
}
keys[key] = this.emotion_json[key][Math.floor(frame)];
});
this.setBlends(keys);
}
toJSON() {
return {
version: this.version,
media_id: this.media_id,
type: this.type,
};
}
}
@@ -1,171 +0,0 @@
import { LipSync } from "../Editor/lipsync";
import * as THREE from "three";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import Ijson from "~/interfaces/Ijson";
import { StorytellerApiHostStore } from "@storyteller/api";
interface AudioDataInterface {
audioContext: AudioContext;
audioBuffer: AudioBuffer;
}
class AudioData implements AudioDataInterface {
audioContext: AudioContext;
audioBuffer: AudioBuffer;
source: AudioBufferSourceNode | undefined;
constructor(audioContext: AudioContext, audioBuffer: AudioBuffer) {
this.audioContext = audioContext;
this.audioBuffer = audioBuffer;
}
}
class BlendShapeHelper {
ah: number;
ee: number;
oh: number;
constructor(ah: number, ee: number, oh: number) {
this.ah = ah;
this.ee = ee;
this.oh = oh;
}
}
export class LipSyncClip implements Ijson {
version: number;
media_id: string;
type: "lipsync";
volume: number;
audio_data: AudioData | undefined;
lipsync: LipSync;
blendshape_helper: BlendShapeHelper;
faces: THREE.Mesh[];
constructor(version: number, media_id: string, volume: number) {
this.version = version;
this.media_id = media_id;
this.type = "lipsync";
this.volume = volume;
this.download_audio().then((data) => {
this.audio_data = data;
});
this.blendshape_helper = new BlendShapeHelper(0, 0, 0);
// we might need 3 of these one for each character ...
this.lipsync = new LipSync();
this.faces = [];
}
// lip sync will be generated through TTS
async get_media_url() {
//This is for prod when we have the proper info on the url.
const apiSchemeAndHost = StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
const url = `${apiSchemeAndHost}/v1/media_files/file/${this.media_id}`;
console.log(`API BASE URL? ${apiSchemeAndHost}`);
console.log(`CALLED URL? ${url}`);
const response = await fetch(url);
const json = await JSON.parse(await response.text());
const bucketPath = json["media_file"]["public_bucket_path"];
const media_api_base_url = GetCdnOrigin();
//const media_base_url = `${media_api_base_url}/vocodes-public`;
//const media_url = `${media_base_url}${bucketPath}`;
const media_url = `${media_api_base_url}${bucketPath}`;
return media_url;
}
async download_audio() {
const url = await this.get_media_url();
const audioContext = new AudioContext();
const response = await fetch(url);
const arrayBuffer = await response.arrayBuffer();
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
return new AudioData(audioContext, audioBuffer);
}
async _detect_face(object: THREE.Object3D): Promise<THREE.Mesh> {
return new Promise((resolve) => {
object.traverse((c: THREE.Object3D) => {
if (c instanceof THREE.Mesh) {
if (c.morphTargetInfluences && c.morphTargetDictionary) {
let blendShapeIndexE = c.morphTargetDictionary["E"];
let blendShapeIndexO = c.morphTargetDictionary["O"];
let blendShapeIndexA = c.morphTargetDictionary["aa"];
if (blendShapeIndexE === undefined) {
blendShapeIndexE = c.morphTargetDictionary["お"]; // MMD OH
blendShapeIndexO = c.morphTargetDictionary["ω"]; // MMD O
blendShapeIndexA = c.morphTargetDictionary["あ"]; // MMD A
}
if (blendShapeIndexE != null) {
this.blendshape_helper = new BlendShapeHelper(
blendShapeIndexA,
blendShapeIndexE,
blendShapeIndexO,
);
this.faces.push(c);
resolve(c);
}
}
}
});
});
}
async play(object: THREE.Object3D) {
if (this.audio_data?.audioBuffer == null) {
await this.download_audio();
}
if (this.lipsync.face == null) {
this.lipsync = new LipSync(await this._detect_face(object));
this.lipsync.startLipSyncFromAudioBuffer(this.audio_data?.audioBuffer);
}
}
stop() {
if (this.lipsync == null) {
return;
}
this.lipsync.destroy();
}
setBlends(ah: number, ee: number, oh: number) {
this.faces.forEach((element: THREE.Mesh) => {
if (element.morphTargetInfluences) {
element.morphTargetInfluences[this.blendshape_helper.ee] = ee;
element.morphTargetInfluences[this.blendshape_helper.ah] = ah;
element.morphTargetInfluences[this.blendshape_helper.oh] = oh;
}
});
}
step(
frame: number,
offset: number,
// rendering: boolean
) {
if (this.lipsync == null) {
return;
}
const positions = this.lipsync.update(frame, offset, true);
if (positions)
this.setBlends(positions["ee"], positions["ah"], positions["oh"]);
}
reset() {
if (this.lipsync.face != undefined) {
this.setBlends(0, 0, 0);
}
this.lipsync = new LipSync();
}
toJSON() {
return {
version: this.version,
media_id: this.media_id,
type: this.type,
volume: this.volume,
};
}
}
@@ -1,238 +0,0 @@
import * as THREE from "three";
import Scene from "../Editor/scene";
export class TransformFrame {
position: THREE.Vector3;
rotation: THREE.Vector3;
scale: THREE.Vector3;
offset: number;
keyframe_uuid: string;
constructor(
position: THREE.Vector3,
rotation: THREE.Vector3,
scale: THREE.Vector3,
offset: number,
keyframe_uuid: string,
) {
this.position = position;
this.rotation = rotation;
this.scale = scale;
this.offset = offset;
this.keyframe_uuid = keyframe_uuid;
}
}
export class TransformClip {
version: number;
media_id: string;
object_uuid: string;
type: "transform";
keyframes: TransformFrame[];
length: number;
step_frame: number;
looping: boolean;
offset: number;
constructor(
version: number,
object_uuid: string,
length: number,
media_id: string = "",
) {
this.version = version;
this.media_id = media_id;
this.object_uuid = object_uuid;
this.type = "transform";
this.length = length;
this.keyframes = [];
this.step_frame = 0;
this.offset = 0;
this.looping = false;
}
private findNextNumber(location: number) {
// Sort the transform frames based on their offset in ascending order
this.keyframes.sort((a, b) => a.offset - b.offset);
let currentKeyframe: TransformFrame | undefined;
let nextKeyframe: TransformFrame | undefined;
this.length =
this.keyframes[this.keyframes.length - 1].offset + this.offset;
for (const frame of this.keyframes) {
if (frame.offset <= location) {
currentKeyframe = frame;
} else if (frame.offset > location) {
nextKeyframe = frame;
break;
}
}
return { currentKeyframe, nextKeyframe };
}
public setOffset(keyframe_uuid: string, keyframe_offset: number) {
this.keyframes.forEach((item) => {
if (item.keyframe_uuid === keyframe_uuid) {
item.offset = keyframe_offset;
return;
}
});
}
public setTransform(
keyframe_uuid: string,
position: THREE.Vector3,
rotation: THREE.Vector3,
scale: THREE.Vector3,
) {
this.keyframes.forEach((item) => {
if (item.keyframe_uuid === keyframe_uuid) {
item.position = position;
item.rotation = rotation;
item.scale = scale;
return;
}
});
}
public removeKeyframe(keyframe_uuid: string) {
this.keyframes = this.keyframes.filter(
(item) => item.keyframe_uuid !== keyframe_uuid,
);
}
request_point_location(keyframe_uuid: string, scene: Scene) {
const point = scene.getPoint(keyframe_uuid);
if (point == undefined) {
console.log("Could not find point.");
return;
}
this.keyframes.forEach((keyframe) => {
if (keyframe.keyframe_uuid == keyframe_uuid) {
keyframe.position = point.position;
keyframe.rotation = new THREE.Vector3(
point.rotation.x,
point.rotation.y,
point.rotation.z,
);
keyframe.scale = point.scale;
}
});
return point;
}
step(object: THREE.Object3D, offset: number, frame: number, scene: Scene) {
this.offset = offset;
this.step_frame = frame - offset;
//if (this.step_frame < 0) { this.step_frame = 0; }
//if (this.step_frame >= this.length) { return; } // Reached max frames.
if (this.keyframes.length < 1) {
return;
} // If there are enough points in the scene.
// Find the current and next keyframes based on time_frame
const { currentKeyframe, nextKeyframe } = this.findNextNumber(
this.step_frame,
);
if (nextKeyframe != undefined && currentKeyframe != undefined) {
const small_step_frame = this.step_frame - currentKeyframe.offset;
let location =
small_step_frame / (nextKeyframe.offset - currentKeyframe.offset);
if (location < 0) {
location = 0;
}
const pos_s = currentKeyframe?.position;
let pos_e = nextKeyframe?.position;
if (pos_e === undefined) {
this.request_point_location(nextKeyframe.keyframe_uuid, scene);
pos_e = nextKeyframe?.position;
}
const points_pos = [
new THREE.Vector3(pos_s.x, pos_s.y, pos_s.z),
new THREE.Vector3(pos_e.x, pos_e.y, pos_e.z),
];
const curve_pos = new THREE.CatmullRomCurve3(points_pos);
const point_pos = curve_pos.getPoint(location);
object.position.copy(point_pos);
const rot_s = currentKeyframe?.rotation;
const rot_e = nextKeyframe?.rotation;
const quat_s = new THREE.Quaternion().setFromEuler(
new THREE.Euler(
THREE.MathUtils.degToRad(rot_s.x),
THREE.MathUtils.degToRad(rot_s.y),
THREE.MathUtils.degToRad(rot_s.z),
),
);
const quat_e = new THREE.Quaternion().setFromEuler(
new THREE.Euler(
THREE.MathUtils.degToRad(rot_e.x),
THREE.MathUtils.degToRad(rot_e.y),
THREE.MathUtils.degToRad(rot_e.z),
),
);
const quaternion = new THREE.Quaternion();
quaternion.slerpQuaternions(quat_s, quat_e, location);
object.quaternion.copy(quaternion);
const scale_s = currentKeyframe?.scale;
const scale_e = nextKeyframe?.scale;
const points_scale = [
new THREE.Vector3(scale_s.x, scale_s.y, scale_s.z),
new THREE.Vector3(scale_e.x, scale_e.y, scale_e.z),
];
const curve_scale = new THREE.CatmullRomCurve3(points_scale);
const point_scale = curve_scale.getPoint(location);
object.scale.copy(point_scale);
}
}
reset(object: THREE.Object3D) {
if (this.keyframes.length > 0) {
const first_pos = this.keyframes[0].position;
const first_rot = this.keyframes[0].rotation;
const first_quat = new THREE.Euler(
THREE.MathUtils.degToRad(first_rot.x),
THREE.MathUtils.degToRad(first_rot.y),
THREE.MathUtils.degToRad(first_rot.z),
);
const first_scl = this.keyframes[0].scale;
object.position.copy(first_pos);
object.rotation.copy(first_quat);
object.scale.copy(first_scl);
this.step_frame = 0;
}
}
add_frame(
position: THREE.Vector3,
rotation: THREE.Vector3,
scale: THREE.Vector3,
offset: number,
keyframe_uuid: string,
) {
this.keyframes.push(
new TransformFrame(position, rotation, scale, offset, keyframe_uuid),
);
}
toJSON(): any {
return {
version: this.version,
media_id: this.media_id,
object_uuid: this.object_uuid,
length: this.length,
type: this.type,
keyframes: this.keyframes,
};
}
}
@@ -1,106 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import {
faRectangle,
faRectangleVertical,
faSquare,
} from "@fortawesome/pro-solid-svg-icons";
import { CameraAspectRatio } from "~/pages/PageEnigma/enums";
import { cameraAspectRatio } from "~/pages/PageEnigma/signals";
import { QueueNames } from "~/pages/PageEnigma/Queue/QueueNames";
import Queue from "~/pages/PageEnigma/Queue/Queue";
import { toEngineActions } from "~/pages/PageEnigma/Queue/toEngineActions";
import { ButtonDropdown } from "@storyteller/ui-button-dropdown";
export const AspectRatioMenu = () => {
useSignals();
const buttonIcon =
cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_16_9
? faRectangle
: cameraAspectRatio.value === CameraAspectRatio.VERTICAL_9_16
? faRectangleVertical
: cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_3_2
? faRectangle
: cameraAspectRatio.value === CameraAspectRatio.VERTICAL_2_3
? faRectangleVertical
: faSquare;
const buttonText =
cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_16_9
? "16:9 Horizontal"
: cameraAspectRatio.value === CameraAspectRatio.VERTICAL_9_16
? "9:16 Vertical"
: cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_3_2
? "3:2 Horizontal"
: cameraAspectRatio.value === CameraAspectRatio.VERTICAL_2_3
? "2:3 Vertical"
: "1:1 Squared";
const handleChangeAspectRatio = (newRatio: CameraAspectRatio) => {
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.CHANGE_CAMERA_ASPECT_RATIO,
data: newRatio,
});
};
return (
<div className="absolute right-0 top-0 m-2 flex flex-col items-end">
<ButtonDropdown
label={`${buttonText}`}
className="shadow-xl"
icon={buttonIcon}
align="right"
showSelected={true}
options={[
{
label: "16:9",
icon: faRectangle,
className: "pl-4",
description: "Horizontal",
selected:
cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_16_9,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.HORIZONTAL_16_9),
},
{
label: "3:2",
icon: faRectangle,
className: "pl-4",
description: "Horizontal",
selected:
cameraAspectRatio.value === CameraAspectRatio.HORIZONTAL_3_2,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.HORIZONTAL_3_2),
},
{
label: "2:3",
icon: faRectangleVertical,
className: "pl-4",
description: "Vertical",
selected:
cameraAspectRatio.value === CameraAspectRatio.VERTICAL_2_3,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.VERTICAL_2_3),
},
{
label: "9:16",
icon: faRectangleVertical,
className: "pl-4",
description: "Vertical",
selected:
cameraAspectRatio.value === CameraAspectRatio.VERTICAL_9_16,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.VERTICAL_9_16),
},
{
label: "1:1",
icon: faSquare,
className: "pl-4",
description: "Squared",
selected: cameraAspectRatio.value === CameraAspectRatio.SQUARE_1_1,
onClick: () =>
handleChangeAspectRatio(CameraAspectRatio.SQUARE_1_1),
},
]}
/>
</div>
);
};
@@ -1,230 +0,0 @@
import { Modal } from "@storyteller/ui-modal";
import { faPlus, faTrashAlt } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { PopoverItem } from "@storyteller/ui-popover";
import { Input } from "@storyteller/ui-input";
import { Label } from "@storyteller/ui-label";
import { Button } from "@storyteller/ui-button";
import { SliderV2 } from "@storyteller/ui-sliderv2";
import { focalLengthDragging } from "~/pages/PageEnigma/signals/camera";
import { useState, useEffect } from "react";
import { twMerge } from "tailwind-merge";
import { Tooltip } from "@storyteller/ui-tooltip";
interface ExtendedPopoverItem extends PopoverItem {
id: string;
focalLength: number;
position: { x: number; y: number; z: number };
rotation: { x: number; y: number; z: number };
lookAt: { x: number; y: number; z: number };
}
interface CameraSettingsModalProps {
isOpen: boolean;
onClose: () => void;
cameras: ExtendedPopoverItem[];
onCameraNameChange: (id: string, newName: string) => void;
onCameraFocalLengthChange: (id: string, value: number) => void;
onAddCamera: () => void;
selectedCameraId: string;
handleCameraSelect: (selectedItem: PopoverItem) => void;
onDeleteCamera: (id: string) => void;
}
export const CameraSettingsModal = ({
isOpen,
onClose,
cameras,
onCameraNameChange,
onCameraFocalLengthChange,
onAddCamera,
selectedCameraId,
handleCameraSelect,
onDeleteCamera,
}: CameraSettingsModalProps) => {
const selectedCamera = cameras.find((cam) => cam.id === selectedCameraId);
const [isDragging, setIsDragging] = useState(false);
const [tempCameraName, setTempCameraName] = useState<string>("");
const [editingCameraId, setEditingCameraId] = useState<string | null>(null);
useEffect(() => {
if (selectedCamera) {
setTempCameraName(selectedCamera.label);
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [selectedCamera?.id]);
useEffect(() => {
const handlePointerUp = () => {
setIsDragging(false);
focalLengthDragging.value = {
isDragging: false,
focalLength: selectedCamera?.focalLength || 35,
};
};
document.addEventListener("pointerup", handlePointerUp);
return () => document.removeEventListener("pointerup", handlePointerUp);
}, [selectedCamera?.focalLength]);
const handlePointerDown = () => {
setIsDragging(true);
focalLengthDragging.value = {
isDragging: true,
focalLength: selectedCamera?.focalLength || 35,
};
};
const handleFocalLengthChange = (id: string, value: number) => {
focalLengthDragging.value = { isDragging: true, focalLength: value };
onCameraFocalLengthChange(id, value);
};
const handleNameBlur = () => {
if (selectedCamera && tempCameraName.trim() !== "") {
onCameraNameChange(selectedCamera.id, tempCameraName);
} else if (selectedCamera) {
setTempCameraName(selectedCamera.label);
}
setEditingCameraId(null);
};
const handleNameFocus = (id: string) => {
setEditingCameraId(id);
};
return (
<Modal
isOpen={isOpen}
onClose={onClose}
className={twMerge(
"h-[500px] max-w-3xl duration-200",
isDragging ? "opacity-10" : "opacity-100",
)}
backdropClassName={twMerge(
"duration-200",
isDragging ? "opacity-0" : "opacity-100",
)}
childPadding={false}
>
<div className="grid h-full grid-cols-12 gap-3">
<div className="relative col-span-4 p-3 pt-2 after:absolute after:right-0 after:top-0 after:h-full after:w-px after:bg-gray-200 after:dark:bg-white/10">
<div className="flex items-center justify-between gap-2.5 py-0.5">
<h2 className="text-[18px] font-semibold opacity-80">Camera</h2>
<Tooltip content="Add camera" position="top" delay={200}>
<button
className="h-6 w-6 rounded-full text-white/70 transition-colors hover:text-white/100"
onClick={onAddCamera}
>
<FontAwesomeIcon icon={faPlus} className="text-xl" />
</button>
</Tooltip>
</div>
<hr className="my-2 w-full border-white/10" />
<div className="space-y-1">
{cameras.map((camera) => (
<button
key={camera.id}
className={`h-9 w-full rounded-lg p-2 text-left transition-colors duration-100 hover:bg-[#63636B]/40 ${
camera.id === selectedCameraId ? "bg-[#63636B]/40" : ""
}`}
onClick={() => handleCameraSelect(camera)}
>
<div className="flex items-center gap-2.5 text-sm">
{camera.icon}
{editingCameraId === camera.id
? tempCameraName
: camera.label}
</div>
</button>
))}
</div>
</div>
<div className="col-span-8 p-3 ps-0 pt-2">
<div className="flex h-full flex-col">
<div>
<div className="flex items-center justify-between gap-2.5 py-0.5 opacity-100">
<h2 className="text-[18px] font-semibold">
{selectedCamera?.label || "Camera"}
</h2>
<Tooltip
content={
selectedCamera?.id === "main"
? "Cannot delete the main camera"
: "Delete camera"
}
position="top"
delay={200}
>
<button
className={`h-6 w-6 rounded-lg transition-colors ${
selectedCamera?.id === "main"
? "cursor-not-allowed text-white/30"
: "text-white/60 hover:text-white/100"
}`}
onClick={() =>
selectedCamera &&
selectedCamera.id !== "main" &&
onDeleteCamera(selectedCamera.id)
}
disabled={selectedCamera?.id === "main"}
>
<FontAwesomeIcon icon={faTrashAlt} className="text-lg" />
</button>
</Tooltip>
</div>
<hr className="my-2 w-full border-white/10" />
<div className="space-y-4">
<div className="space-y-1">
<Label htmlFor="camera-name" className="text-sm opacity-70">
Name
</Label>
<Input
id="camera-name"
type="text"
value={tempCameraName}
onChange={(e) => setTempCameraName(e.target.value)}
onBlur={handleNameBlur}
onFocus={() =>
selectedCamera && handleNameFocus(selectedCamera.id)
}
className="text-sm"
/>
</div>
<div className="space-y-2">
<Label htmlFor="focal-length" className="text-sm opacity-70">
Focal Length
</Label>
<div
className="mt-1 flex items-center gap-4"
onPointerDown={handlePointerDown}
>
<SliderV2
min={10}
max={200}
value={selectedCamera?.focalLength || 35}
onChange={(value) =>
selectedCamera &&
handleFocalLengthChange(selectedCamera.id, value)
}
step={1}
suffix="mm"
showDecrement={true}
showIncrement={true}
className="w-full"
/>
<span className="min-w-[60px] text-sm">
{selectedCamera?.focalLength || 35}mm
</span>
</div>
</div>
</div>
</div>
<div className="mt-auto flex justify-end pt-4">
<Button onClick={onClose}>Done</Button>
</div>
</div>
</div>
</div>
</Modal>
);
};
@@ -1 +0,0 @@
export { CameraSettingsModal } from "./CameraSettingsModal";
@@ -1,74 +0,0 @@
//TODO: DELETE
import { useContext } from "react";
import { Button } from "@storyteller/ui-button"
import { EngineContext } from "~/pages/PageEnigma/contexts/EngineContext";
export const TestFeaturesButtons = ({ debug }: { debug: boolean }) => {
if (!debug) return null;
const editorEngine = useContext(EngineContext);
const handleButtonCameraView = () => {
editorEngine?.switchCameraView();
};
const testStylizeRequest = () => {
// editorEngine?.testStylizeRequest();
//console.log("editorEnging does not have testStylizeRequest");
};
const handleButtonRender = () => {
editorEngine?.generateVideo();
};
const handleButtonTakeFrame = () => {
// editorEngine?.take_timeline_cam_clip();
//console.log("editorEnging does not have take_timeline_cam_clip");
};
const handleButtonSingleFrame = () => {
editorEngine?.generateFrame();
};
const handleButtonPlayBack = () => {
editorEngine?.togglePlayback();
};
const smallButtons = "text-xs p-2 h-6 ";
const tertiaryColor =
"bg-brand-tertiary hover:bg-brand-teriary-400 focus-visible:outline-brand-tertiary ";
return (
<>
<div className="flex gap-1">
<Button
variant="secondary"
onClick={handleButtonCameraView}
className={smallButtons}
>
Toggle Camera View
</Button>
</div>
<div className="flex gap-1">
<Button onClick={handleButtonSingleFrame} className={smallButtons}>
Render Single Frame
</Button>
<Button onClick={handleButtonTakeFrame} className={smallButtons}>
Take Frame
</Button>
<Button onClick={handleButtonRender} className={smallButtons}>
Render
</Button>
<Button
onClick={testStylizeRequest}
className={
"hover:bg-brand-teriary-400 bg-brand-tertiary focus-visible:outline-brand-tertiary " +
smallButtons
}
style={{ zIndex: 9001 }}
>
Test Stylize
</Button>
</div>
</>
);
};
@@ -1,158 +0,0 @@
import { useContext, useState } from "react";
import { useSignals } from "@preact/signals-react/runtime";
import {
faBackwardStep,
faForwardStep,
faCirclePause,
faCirclePlay,
faPlay,
faRepeat,
} from "@fortawesome/pro-solid-svg-icons";
import { ButtonIcon } from "@storyteller/ui-button-icon";
import { Tooltip } from "@storyteller/ui-tooltip";
import { EngineContext } from "~/pages/PageEnigma/contexts/EngineContext";
import {
currentTime,
filmLength,
secondaryScrubber,
} from "~/pages/PageEnigma/signals";
import Queue from "~/pages/PageEnigma/Queue/Queue";
import { QueueNames } from "~/pages/PageEnigma/Queue/QueueNames";
import { toEngineActions } from "~/pages/PageEnigma/Queue/toEngineActions";
import { editorState } from "~/pages/PageEnigma/signals/engine";
import { EditorStates } from "~/pages/PageEnigma/enums";
import { twMerge } from "tailwind-merge";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
import { FeatureFlags } from "~/enums";
export const ControlsVideo = () => {
useSignals();
const editorEngine = useContext(EngineContext);
const [refresh, setRefresh] = useState(0);
const showStylePage = usePosthogFeatureFlag(FeatureFlags.SHOW_STYLE_PAGE);
function formatTime(millis: number) {
const date = new Date(millis);
const secs = String(date.getUTCSeconds()).padStart(2, "0");
const ms = String(date.getUTCMilliseconds()).padStart(3, "0");
return `00:${secs}:${ms}`;
}
const isPlaying =
editorEngine !== null ? editorEngine.timeline.is_playing : false;
const isRepeating =
editorEngine !== null ? editorEngine.timeline.is_repeating : false;
const handleToStart = () => {
currentTime.value = 0;
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.UPDATE_TIME,
data: { currentTime: currentTime.value },
});
};
const handleToEnd = () => {
currentTime.value = filmLength.value * 1000;
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.UPDATE_TIME,
data: { currentTime: currentTime.value },
});
};
const handleBackwardStep = () => {
currentTime.value = Math.max(currentTime.value - 1, 0);
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.UPDATE_TIME,
data: { currentTime: currentTime.value },
});
};
const handleRepeat = () => {
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.TOGGLE_REPEATING,
data: null,
});
setRefresh((refresh + 1) % 10);
};
const handlePlayback = () => {
editorEngine?.togglePlayback();
};
const handleForwardStep = () => {
currentTime.value = Math.min(
currentTime.value + 1,
filmLength.value * 1000,
);
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.UPDATE_TIME,
data: { currentTime: currentTime.value },
});
};
if (editorState.value === EditorStates.PREVIEW) {
return null;
}
return (
<div className="flex items-center justify-center">
<div className="flex items-center gap-4 rounded-t-lg bg-ui-controls p-1.5 text-white shadow-md">
<div className="flex items-center gap-1">
<div className="flex gap-0">
<ButtonIcon icon={faBackwardStep} onClick={handleToStart} />
<ButtonIcon
icon={faPlay}
onClick={handleBackwardStep}
className="rotate-180 text-sm"
/>
</div>
<ButtonIcon
icon={isPlaying ? faCirclePause : faCirclePlay}
onClick={handlePlayback}
className="p-0 text-2xl"
/>
<div className="flex gap-0">
<ButtonIcon
icon={faPlay}
onClick={handleForwardStep}
className="text-sm"
/>
<ButtonIcon icon={faForwardStep} onClick={handleToEnd} />
</div>
</div>
<div className="flex items-center gap-3">
<div className="mr-2 flex items-center gap-1.5 text-sm font-medium">
<span className="w-[54px]">
{formatTime(secondaryScrubber.value)}
</span>
<span className="opacity-50">/</span>
<span className="w-[54px] opacity-50">
{formatTime(filmLength.value * 1000)}
</span>
</div>
{showStylePage && (
<>
<div className="h-[18px] w-0.5 rounded-full bg-white/20" />
<Tooltip content="Loop" position={"top"}>
<ButtonIcon
icon={faRepeat}
onClick={handleRepeat}
className={twMerge(
"h-7 w-7 p-0 text-sm",
isRepeating
? "border border-brand-primary"
: "border border-transparent",
)}
/>
</Tooltip>
</>
)}
</div>
</div>
</div>
);
};
@@ -1 +0,0 @@
export { ControlsVideo } from './ControlsVideo';
@@ -1,108 +0,0 @@
import {
canDrop,
currPosition,
dragItem,
overTimeline,
scale,
timelineHeight,
} from "~/pages/PageEnigma/signals";
import { useSignals } from "@preact/signals-react/runtime";
import DndAsset from "~/pages/PageEnigma/DragAndDrop/DndAsset";
import { MIN_ANIM_DURATION } from "~/constants";
export const DragComponent = () => {
useSignals();
if (!dragItem.value) {
return null;
}
const { currX, currY } = currPosition.value;
const thumbnail = dragItem.value.thumbnail
? dragItem.value.thumbnail
: `/resources/images/default-covers/${dragItem.value.imageIndex || 0}.webp`;
if (overTimeline.value) {
if (canDrop.value && DndAsset.overElement) {
return (
<>
<div
id={`ani-dnd-${dragItem.value.media_id}`}
className={[
"absolute p-2",
"rounded-md",
!canDrop.value
? "bg-brand-primary"
: "border border-dashed border-dnd-canDropBorder bg-dnd-canDrop",
"block",
].join(" ")}
style={{
top: DndAsset.overElement.top,
left: currX + 1,
zIndex: 10000,
width: Math.max(dragItem.value.length ?? 0, MIN_ANIM_DURATION) * 4 * scale.value,
height: 60,
}}
/>
<div
id={`ani-dnd-${dragItem.value.media_id}`}
className={[
"absolute p-2",
"rounded opacity-60",
"border border-dnd-timeGridBorder bg-dnd-timeGrid",
].join(" ")}
style={{
bottom: timelineHeight.value - 60,
left: currX + 1,
zIndex: 10000,
width: Math.max(dragItem.value.length ?? 0, MIN_ANIM_DURATION) * 4 * scale.value,
height: 16,
}}
/>
</>
);
}
return (
<div
id={`ani-dnd-${dragItem.value.media_id}`}
className={[
"absolute p-1",
"rounded-lg text-xs",
"bg-dnd-cannotDrop",
"block text-nowrap",
].join(" ")}
style={{
top: currY - 16,
left: currX + 1,
zIndex: 10000,
}}
>
{DndAsset.notDropText || "Cannot drop here"}
</div>
);
}
return (
<div
className="absolute rounded-lg"
style={{
width: 91,
height: 114,
top: currY - 57,
left: currX + 1,
zIndex: 10000,
}}
>
<img
{...{
crossOrigin: "anonymous",
src: thumbnail,
}}
alt={dragItem.value.name}
className="pointer-events-none select-none rounded-t-lg bg-gradient-to-b from-[#CCCCCC] to-[#A0A0A0]"
/>
<div className="w-full truncate rounded-b-lg bg-ui-controls px-2 py-1 text-center text-[12px]">
{dragItem.value.name || dragItem.value.media_id}
</div>
</div>
);
};
@@ -1,269 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { faBell, faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { PopoverMenu } from "@storyteller/ui-popover";
import { CompletedCard } from "./CompletedCard";
import { InProgressCard } from "./InProgressCard";
import { Tooltip } from "@storyteller/ui-tooltip";
import { useEffect, useState, useRef, useCallback } from "react";
import { JobsApi } from "~/Classes/ApiManager/JobsApi";
import { toast } from "@storyteller/ui-toaster";
import { ActiveJob } from "~/pages/PageEnigma/models";
import { IsJobPollingDisabled } from "~/flags/DevConsoleFlags";
// TODO ensure we de-dupe all this extra code.
interface CompletedItem {
token: string;
maybe_title: string;
public_bucket_path: string; // TODO: STOP USING THIS.
maybe_media_links_thumbnail?: string;
updated_at: string;
maybe_result?: {
entity_token?: string;
};
}
// {
// "job_token": "jinf_421nnr6rj8wy2sj0s11v9fns32s",
// "request": {
// "inference_category": "image_generation",
// "maybe_model_type": "image_gen_api",
// "maybe_model_token": null,
// "maybe_model_title": "Image Generation",
// "maybe_raw_inference_text": null,
// "maybe_style_name": null,
// "maybe_live_portrait_details": null,
// "maybe_lipsync_details": null
// },
// "status": {
// "status": "complete_success",
// "maybe_extra_status_description": null,
// "maybe_assigned_worker": "inference-job-4o-image-848c55d5dd-555kn",
// "maybe_assigned_cluster": "storyteller-eks-1",
// "maybe_first_started_at": "2025-04-15T04:41:01Z",
// "maybe_current_execution_duration_seconds": null,
// "attempt_count": 1,
// "requires_keepalive": false,
// "maybe_failure_category": null,
// "progress_percentage": 100
// },
// "maybe_result": {
// "entity_type": "media_file",
// "entity_token": "m_ytrjp1fcd5tb43ebcs3ewhq6y05tfb",
// "maybe_public_bucket_media_path": "/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "media_links": {
// "cdn_url": "https://cdn-2.fakeyou.com/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "maybe_thumbnail_template": "https://cdn-2.fakeyou.com/cdn-cgi/image/width={WIDTH},quality=95/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "maybe_video_previews": null
// },
// "maybe_successfully_completed_at": "2025-04-15T04:42:14Z"
// },
// "created_at": "2025-04-15T04:41:01Z",
// "updated_at": "2025-04-15T04:42:14Z"
// }
export function Activity() {
useSignals();
const [completedItems, setCompletedItems] = useState<CompletedItem[]>([]);
const [jobs, setJobs] = useState<ActiveJob[]>([]);
const processedJobs = useRef(new Set<string>());
const isFirstLoad = useRef(true);
const toastTimeoutRef = useRef<NodeJS.Timeout | null>(null);
// Debounced toast function
const showToast = useCallback((items: CompletedItem[]) => {
if (toastTimeoutRef.current) {
clearTimeout(toastTimeoutRef.current);
}
toastTimeoutRef.current = setTimeout(() => {
const count = items.length;
toast.success(
count === 1
? `${items[0].maybe_title} completed successfully`
: `${count} images completed successfully`,
);
toastTimeoutRef.current = null;
}, 100); // Small delay to catch potential double calls
}, []);
useEffect(() => {
let mounted = true;
async function fetchJobs() {
if (IsJobPollingDisabled()) {
return;
}
try {
const jobsApi = new JobsApi();
const response = await jobsApi.ListRecentJobs();
if (!mounted || !response.success || !response.data) return;
// Handle active jobs
const activeJobs = response.data
.filter((job) => {
const status = job.status?.status?.toLowerCase();
return (
status && status !== "complete_success" && status !== "failed"
);
})
.map(
(job) =>
({
job_token: job.job_token,
request: {
maybe_model_title:
job.request?.maybe_model_title || "Image Generation",
},
status: {
status: job.status?.status?.toUpperCase() || "",
progress_percentage: job.status?.progress_percentage || 0,
},
created_at: job.created_at || new Date().toISOString(),
updated_at: job.updated_at || new Date().toISOString(),
maybe_result: job.maybe_result || null,
}) as ActiveJob,
);
// Handle completed jobs
const newlyCompleted = response.data.filter(
(job) =>
job.status?.status?.toLowerCase() === "complete_success" &&
!processedJobs.current.has(job.job_token),
);
if (newlyCompleted.length > 0) {
// Add to processed set
newlyCompleted.forEach((job) =>
processedJobs.current.add(job.job_token),
);
// Create completed items
const newItems = newlyCompleted.map((job) => ({
token: job.job_token,
maybe_title: job.request?.maybe_model_title || "Image Generation",
public_bucket_path:
job.maybe_result?.maybe_public_bucket_media_path || "",
maybe_media_links_thumbnail: ((job) => {
// Try the video thumbnail first, then the regular thumbnail.
const maybeVideoThumb =
job.maybe_result?.media_links?.maybe_video_previews?.animated;
const maybeGenericThumb = (
job.maybe_result?.media_links?.maybe_thumbnail_template || ""
).replace("{WIDTH}", "500");
return maybeVideoThumb || maybeGenericThumb;
})(job),
updated_at: job.updated_at || new Date().toISOString(),
maybe_result: job.maybe_result,
}));
// NB(bt,2025-07-28): Don't use eventing from here anymore.
// Tauri is sending direct events we can bind toasts to.
// // Show toast only if not first load
// if (!isFirstLoad.current) {
// showToast(newItems);
// }
// Update state
setCompletedItems((prev) => {
const combined = [...newItems, ...prev];
return combined
.filter(
(item, index, self) =>
index === self.findIndex((t) => t.token === item.token),
)
.sort(
(a, b) =>
new Date(b.updated_at).getTime() -
new Date(a.updated_at).getTime(),
)
.slice(0, 50);
});
}
setJobs(activeJobs);
isFirstLoad.current = false;
} catch (error) {
console.error("Failed to fetch jobs:", error);
}
}
// Initial fetch
fetchJobs();
// Set up polling
const interval = setInterval(fetchJobs, 5000);
// Cleanup
return () => {
mounted = false;
clearInterval(interval);
if (toastTimeoutRef.current) {
clearTimeout(toastTimeoutRef.current);
}
};
}, [showToast]);
return (
<Tooltip content="Activity" position="bottom" closeOnClick={true}>
<PopoverMenu
mode="default"
buttonClassName="h-[38px] w-[38px] !p-0"
panelClassName="w-[360px] p-2 bg-ui-panel"
position="bottom"
align="end"
triggerIcon={
<div>
{jobs.length > 0 ? (
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
) : (
<FontAwesomeIcon icon={faBell} />
)}
{jobs.length > 0 && (
<div className="absolute -right-1 -top-1 h-3 w-3 animate-pulse rounded-full bg-brand-primary-400" />
)}
</div>
}
>
{(close) => (
<div className="max-h-[480px] overflow-y-auto">
{jobs.length > 0 && (
<div>
{jobs.map((job) => (
<InProgressCard key={job.job_token} job={job} />
))}
</div>
)}
{isFirstLoad.current ? (
<div className="flex h-48 w-full flex-col justify-center gap-4 p-4 text-center align-middle">
<FontAwesomeIcon
icon={faSpinnerThird}
spin
size="2x"
className="text-base-fg/40"
/>
<h3 className="text-base-fg/60">Retrieving Activities</h3>
</div>
) : jobs.length === 0 && completedItems.length === 0 ? (
<div className="flex h-48 w-full flex-col justify-center gap-4 p-4 text-center align-middle">
<h3 className="text-base-fg/60 text-lg">No activities yet</h3>
</div>
) : (
<div>
<div className="flex flex-col">
{completedItems.map((item) => (
<div key={item.token} onClick={close}>
<CompletedCard job={item} />
</div>
))}
</div>
</div>
)}
</div>
)}
</PopoverMenu>
</Tooltip>
);
}
@@ -1,90 +0,0 @@
import { useState } from "react";
import dayjs from "dayjs";
import {
galleryModalLightboxMediaId,
galleryModalLightboxVisible,
galleryModalLightboxImage,
} from "@storyteller/ui-gallery-modal";
import type { GalleryItem } from "@storyteller/ui-gallery-modal";
interface Props {
job: {
token: string;
maybe_title: string;
updated_at: string;
public_bucket_path: string;
maybe_style_name?: string;
maybe_media_links_thumbnail?: string;
maybe_result?: {
entity_token?: string;
media_links?: {
cdn_url?: string;
};
};
};
}
export function CompletedCard({ job }: Props) {
const [loadError, setLoadError] = useState(false);
// Info: Gallery modal will handle displaying full media. Compute thumbnail only.
// NB: Try to use the thumbnail instead of the link to the full asset.
// Passing around full asset URLs and putting them into image tags is causing CORS errors if those assets are not images.
const thumbnailUrl = job?.maybe_media_links_thumbnail;
if (!job) return null;
return (
<>
<div
className="flex w-full items-center justify-between rounded-lg p-2 text-start transition-all duration-150 hover:cursor-pointer hover:bg-white/10"
onClick={() => {
galleryModalLightboxMediaId.value =
job.maybe_result?.entity_token || null;
galleryModalLightboxImage.value = {
id: job.maybe_result?.entity_token || job.token,
label: job.maybe_title || "Image",
thumbnail: thumbnailUrl,
fullImage: job.maybe_result?.media_links?.cdn_url || thumbnailUrl,
createdAt: job.updated_at,
mediaClass: undefined,
} as GalleryItem;
galleryModalLightboxVisible.value = true;
}}
>
<div className="flex gap-4">
<div className="flex aspect-square h-14 w-14 justify-center overflow-hidden rounded-lg border border-[#A9A9A9]/50 bg-black/60">
<img
src={
loadError
? "/resources/images/movie-placeholder.png"
: thumbnailUrl
}
className="h-full w-full object-cover"
alt={job.maybe_title ?? "unknown"}
//crossOrigin="anonymous"
onError={() => setLoadError(true)}
loading="lazy"
/>
</div>
<div className="flex flex-col justify-center gap-1">
<div className="font-medium">{job.maybe_title || "Untitled"}</div>
<div className="text-sm text-white/60">
{dayjs(job.updated_at).format("MMM D, YYYY HH:mm:ss")}
</div>
</div>
</div>
{/* <a
href={downloadLink}
download
onClick={(e) => e.stopPropagation()}
className="flex h-9 w-9 items-center justify-center text-lg text-white/60 transition-all hover:text-white"
>
<FontAwesomeIcon icon={faDownToLine} />
</a> */}
</div>
{/* LightboxModal removed – now using top level GalleryModal */}
</>
);
}
@@ -1,46 +0,0 @@
import { useCallback, useState } from "react";
import { useSignals, useSignalEffect } from "@preact/signals-react/runtime";
import { MediaFile } from "~/pages/PageEnigma/models";
import { generateMovieId } from "~/pages/PageEnigma/signals";
import { ToastTypes } from "~/enums";
import { addToast } from "~/signals";
import { MyMovies } from "~/pages/PageEnigma/comps/GenerateModals/MyMovies";
import { Sharing } from "~/pages/PageEnigma/comps/GenerateModals/Sharing";
import { MediaFilesApi } from "~/Classes/ApiManager/MediaFilesApi";
export function GenerateModals() {
useSignals();
const [mediaFile, setMediaFile] = useState<MediaFile | null>(null);
const GetMediaFileByToken = useCallback(async (movieId: string) => {
const mediaFilesApi = new MediaFilesApi();
const response = await mediaFilesApi.GetMediaFileByToken({
mediaFileToken: movieId,
});
if (response.success && response.data) {
setMediaFile(response.data);
return;
}
addToast(
ToastTypes.ERROR,
response.errorMessage ||
`Unknown Error in Getting Movie (token=${movieId}`,
);
}, []);
const setMovieId = useCallback((movieId: string) => {
generateMovieId.value = movieId;
}, []);
useSignalEffect(() => {
if (generateMovieId.value) {
GetMediaFileByToken(generateMovieId.value);
}
});
if (!mediaFile) {
return <MyMovies setMovieId={setMovieId} />;
}
return <Sharing mediaFile={mediaFile!} setMediaFile={setMediaFile} />;
}
@@ -1,90 +0,0 @@
import { useCallback } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faClose, faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { ToastTypes } from "~/enums";
import { addToast } from "~/signals";
import { JobState } from "~/pages/PageEnigma/enums";
import { ActiveJob } from "~/pages/PageEnigma/models";
import { Tooltip } from "@storyteller/ui-tooltip";
import { JobsApi } from "~/Classes/ApiManager";
import { PollRecentJobs } from "~/hooks/useActiveJobs/utilities";
interface Props {
job: ActiveJob;
}
function getPercent(status: JobState, percentage: number) {
if (
[
JobState.COMPLETE_FAILURE,
JobState.ATTEMPT_FAILED,
JobState.DEAD,
].includes(status)
) {
return 0;
} else {
return percentage;
}
}
export function InProgressCard({ job }: Props) {
const completePercent = getPercent(
job.status.status as JobState,
job.status.progress_percentage,
);
const deleteJob = useCallback(async (job: ActiveJob) => {
const jobsApi = new JobsApi();
const response = await jobsApi.DeleteJobByToken(job.job_token);
if (response.success) {
PollRecentJobs();
addToast(ToastTypes.SUCCESS, "File successfully deleted.");
return;
}
addToast(
ToastTypes.ERROR,
response.errorMessage || "Error deleting the file.",
);
}, []);
return (
<div className="flex w-full items-center justify-between rounded-lg p-2 text-start transition-all duration-150 hover:bg-white/10">
<div className="flex gap-4">
<div className="flex aspect-square h-14 w-14 items-center justify-center rounded-lg border border-[#A9A9A9]/50 bg-black/60 text-white">
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
{/* {(job.status.status === JobState.STARTED ||
job.status.status === JobState.PENDING) && (
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
)}
{(job.status.status === JobState.COMPLETE_FAILURE ||
job.status.status === JobState.ATTEMPT_FAILED) && (
<FontAwesomeIcon icon={faCircleExclamation} size={"lg"} />
)} */}
</div>
<div className="flex flex-col justify-center gap-1">
<div className="font-medium">
{job.request.maybe_model_title || "Image Generation"}
</div>
<div className="text-sm capitalize text-white/60">
{job.status.status.replaceAll("_", " ")}... {completePercent}%
</div>
</div>
</div>
{job.status.status !== JobState.STARTED && (
<div className="pr-5">
<Tooltip content="Cancel" position="left">
<button
onClick={() => deleteJob(job)}
className="text-[15px] font-medium text-white/50 transition-all duration-150 hover:text-white/100"
>
<FontAwesomeIcon icon={faClose} />
</button>
</Tooltip>
</div>
)}
</div>
);
}
@@ -1,83 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { faFilm, faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { CompletedCard } from "~/pages/PageEnigma/comps/GenerateModals/CompletedCard";
import { InProgressCard } from "~/pages/PageEnigma/comps/GenerateModals/InProgressCard";
import { Modal } from "@storyteller/ui-modal";
import { viewMyMovies } from "~/pages/PageEnigma/signals";
import { activeWorkflowJobs, userMovies } from "~/signals";
interface Props {
setMovieId: (page: string) => void;
}
export function MyMovies({ setMovieId }: Props) {
useSignals();
return (
<Modal
title="My Movies"
titleIcon={faFilm}
className="max-w-4xl"
childPadding={false}
isOpen={viewMyMovies.value}
onClose={() => {
setMovieId("");
viewMyMovies.value = false;
}}
>
<div className="h-[560px] overflow-y-auto overflow-x-hidden rounded-b-lg">
{activeWorkflowJobs.value && activeWorkflowJobs.value.length > 0 && (
<div className="mb-3">
<div className="mx-5 mb-1 font-medium">In Progress</div>
{activeWorkflowJobs.value.map((movieJob) => (
<InProgressCard key={movieJob.job_token} movie={movieJob} />
))}
</div>
)}
<MovieList setMovieId={setMovieId} />
</div>
</Modal>
);
}
const MovieList = ({ setMovieId }: Props) => {
useSignals();
if (!userMovies.value && !activeWorkflowJobs.value) {
return (
<div className="flex h-full w-full flex-col justify-center gap-6 text-center align-middle">
<FontAwesomeIcon icon={faSpinnerThird} spin size={"3x"} />
<h3>Retrieving Completed Movies</h3>
</div>
);
}
if (
userMovies.value &&
userMovies.value.length === 0 &&
!activeWorkflowJobs.value
) {
return (
<div className="flex h-full w-full flex-col justify-center gap-6 text-center align-middle">
<h3>You have not created any movies yet!</h3>
<p>
Try start a new scene from our featured scenes, and generate a movie
via the <b>AI Stylize</b> tab.
</p>
</div>
);
}
return (
<div>
<div className="mx-5 mb-1 font-medium">Completed</div>
<div className="flex flex-col">
{userMovies.value?.map((movie) => (
<CompletedCard
key={movie.token}
movie={movie}
setMovieId={setMovieId}
/>
))}
</div>
</div>
);
};
@@ -1,150 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { useState } from "react";
import { MediaFile } from "~/pages/PageEnigma/models";
import { Input } from "@storyteller/ui-input";
import { Label } from "@storyteller/ui-label";
import { Button } from "@storyteller/ui-button";
import { Modal } from "@storyteller/ui-modal";
import {
faArrowDownToLine,
faArrowRight,
faChevronLeft,
faFilm,
faLink,
} from "@fortawesome/pro-solid-svg-icons";
import SocialButton from "./SocialButton";
import { generateMovieId, viewMyMovies } from "~/pages/PageEnigma/signals";
import dayjs from "dayjs";
import { downloadFile } from "~/pages/PageEnigma/comps/GenerateModals/utils/downloadFile";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
interface Props {
mediaFile: MediaFile;
setMediaFile: (file: MediaFile | null) => void;
}
export function Sharing({ mediaFile, setMediaFile }: Props) {
useSignals();
const mediaTitle = mediaFile?.maybe_title ?? mediaFile?.token;
// TODO: ApiManager should provide all endpoints
const shareUrl = `https://storyteller.ai/media/${mediaFile?.token || ""}`;
const shareText = "Check out this media on StoryTeller.ai";
const [buttonLabel, setButtonLabel] = useState("Copy");
const media_api_base_url = GetCdnOrigin();
const downloadLink = `${media_api_base_url}${mediaFile?.public_bucket_path}`;
const handleCopyLink = () => {
if (navigator.clipboard) {
navigator.clipboard.writeText(shareUrl);
}
setButtonLabel("Copied!");
setTimeout(() => setButtonLabel("Copy"), 2000);
};
const generateTitle = () => {
return (
<div className="flex items-center gap-2.5">
<span className="font-xl font-bold">{mediaTitle}</span>
<span className="text-sm font-medium text-white/60">
{dayjs(mediaFile?.updated_at).format("MMM DD, YYYY HH:mm:ss")}
</span>
</div>
);
};
return (
<Modal
title={generateTitle()}
titleIcon={generateMovieId.value ? faChevronLeft : faFilm}
titleIconClassName="text-white/60 hover:text-white/80 transition-colors duration-150"
onTitleIconClick={
generateMovieId.value ? () => setMediaFile(null) : undefined
}
className="max-w-6xl"
childPadding={false}
isOpen={viewMyMovies.value}
width={1049}
onClose={() => {
viewMyMovies.value = false;
setMediaFile(null);
}}
>
<div className="flex gap-6 px-5 pb-5">
<div className="max-h-[420px] w-full overflow-hidden rounded-lg">
<video controls crossOrigin="anonymous" className="h-full w-full">
<source src={downloadLink} type="video/mp4" />
Your browser does not support the video tag.
</video>
</div>
<div className="flex w-[500px] flex-col">
<Label>Share movie to:</Label>
<div className="flex w-full flex-wrap justify-between">
<SocialButton
social="x"
shareUrl={shareUrl}
shareText={shareText}
/>
<SocialButton
social="whatsapp"
shareUrl={shareUrl}
shareText={shareText}
/>
<SocialButton
social="facebook"
shareUrl={shareUrl}
shareText={shareText}
/>
<SocialButton
social="reddit"
shareUrl={shareUrl}
shareText={shareText}
/>
<SocialButton
social="email"
shareUrl={shareUrl}
shareText={shareText}
/>
</div>
<div className="my-6 flex w-full gap-2">
<div className="w-full">
<Input type="text" value={shareUrl} readOnly />
</div>
<Button icon={faLink} onClick={handleCopyLink} variant="primary">
{buttonLabel}
</Button>
</div>
<div className="flex flex-col gap-3">
<Button
icon={faArrowDownToLine}
className="h-10 w-full"
onClick={(event) => {
event.preventDefault();
event.stopPropagation();
downloadFile({
url: downloadLink,
title: mediaTitle,
});
}}
variant="secondary"
>
Download
</Button>
<Button
className="h-10 w-full"
onClick={() => {
window.open(shareUrl, "_blank");
}}
icon={faArrowRight}
iconFlip={true}
variant="secondary"
>
View on Media Page
</Button>
</div>
</div>
</div>
</Modal>
);
}
@@ -1,82 +0,0 @@
import { IconProp } from "@fortawesome/fontawesome-svg-core";
import {
faFacebookF,
faRedditAlien,
faWhatsapp,
faXTwitter,
} from "@fortawesome/free-brands-svg-icons";
import { faXmark, faEnvelope } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
interface SocialButtonProps {
social: "x" | "whatsapp" | "facebook" | "reddit" | "email";
shareUrl: string;
shareText: string;
}
export default function SocialButton({
social,
shareUrl,
shareText,
}: SocialButtonProps) {
const socialIcons = {
x: faXTwitter,
whatsapp: faWhatsapp,
facebook: faFacebookF,
reddit: faRedditAlien,
email: faEnvelope,
};
const socialIcon = socialIcons[social] || faXmark;
const getShareLink = (social: string, url: string, text: string) => {
switch (social) {
case "x":
return `https://twitter.com/intent/tweet?url=${encodeURIComponent(
url,
)}&text=${encodeURIComponent(text)}`;
case "facebook":
return `https://www.facebook.com/sharer/sharer.php?u=${encodeURIComponent(
url,
)}`;
case "whatsapp":
return `https://wa.me/?text=${encodeURIComponent(
text,
)} ${encodeURIComponent(url)}`;
case "reddit":
return `https://www.reddit.com/submit?url=${encodeURIComponent(
url,
)}&title=${encodeURIComponent(text)}`;
case "email":
return `mailto:?subject=${encodeURIComponent(
text,
)}&body=${encodeURIComponent(url)}`;
default:
return "#";
}
};
const handleClick = () => {
const shareLink = getShareLink(social, shareUrl, shareText);
window.open(shareLink, "_blank");
};
return (
<button className="flex flex-col items-center" onClick={handleClick}>
<div
className={[
"flex h-[56px] w-[56px]",
"items-center justify-center rounded-lg",
"text-[32px] text-white",
`bg-${social}`,
"transition-all duration-200 hover:opacity-80",
].join(" ")}>
<FontAwesomeIcon icon={socialIcon as IconProp} />
</div>
<p className="mt-2 text-xs">
{social.charAt(0).toUpperCase() + social.slice(1)}
</p>
</button>
);
}
@@ -1,51 +0,0 @@
import { ToastTypes } from "~/enums";
import { addToast } from "~/signals";
export const downloadFile = ({
url,
title,
}: {
url: string;
title: string;
}) => {
// console.log(url);
fetch(url)
.then((resp) => resp.blob())
.then((blob) => {
const blobUrl = window.URL.createObjectURL(blob);
const a = document.createElement("a");
a.style.display = "none";
a.href = blobUrl;
a.download = `${title}.mp4`;
document.body.appendChild(a);
a.click();
window.URL.revokeObjectURL(url);
})
.catch(() => addToast(ToastTypes.ERROR, "Could not download file."));
};
export const downloadFileImage = ({
url,
title,
}: {
url: string;
title: string;
}) => {
// console.log(url);
fetch(url)
.then((resp) => resp.blob())
.then((blob) => {
const blobUrl = window.URL.createObjectURL(blob);
const a = document.createElement("a");
a.style.display = "none";
a.href = blobUrl;
a.download = `${title}.png`;
document.body.appendChild(a);
a.click();
window.URL.revokeObjectURL(url);
})
.catch(() => addToast(ToastTypes.ERROR, "Could not download file."));
};
//check
@@ -1,70 +0,0 @@
import React from "react";
import { useSignals } from "@preact/signals-react/runtime";
import { poseMode, showPoseControls } from "../signals/selectedMode";
import { Tooltip } from "@storyteller/ui-tooltip";
import { Button } from "@storyteller/ui-button";
import { faCheck, faPersonRunning } from "@fortawesome/pro-solid-svg-icons";
declare global {
interface Window {
__mouseControls?: {
toggleFKMode: () => void;
};
}
}
interface PoseModeSelectorProps {}
export const PoseModeSelector: React.FC<PoseModeSelectorProps> = () => {
useSignals();
const handleModeChange = () => {
if (poseMode.value === "select") {
poseMode.value = "pose";
} else {
poseMode.value = "select";
}
window.__mouseControls?.toggleFKMode();
};
if (showPoseControls.value === false) {
return null;
}
return (
<>
<div
className="fixed left-1/2 top-32 flex -translate-x-1/2 transform items-center justify-center gap-2"
onClick={(e) => e.stopPropagation()}
onMouseDown={(e) => e.stopPropagation()}
>
<Tooltip
content={"Toggle pose mode (K)"}
position={"bottom"}
delay={300}
closeOnClick={true}
>
<>
{poseMode.value === "select" ? (
<Button
icon={faPersonRunning}
onClick={handleModeChange}
className="rounded-xl shadow-xl outline-none focus-visible:outline-none"
>
Enter Pose Mode
</Button>
) : (
<Button
icon={faCheck}
onClick={handleModeChange}
className="rounded-xl outline-none focus-visible:outline-none"
>
Done
</Button>
)}
</>
</Tooltip>
</div>
</>
);
};
@@ -1,43 +0,0 @@
import { Signal } from "@preact/signals-react";
import { useSignals } from "@preact/signals-react/runtime";
export const PrecisionSelector = ({
showSignal,
coordSignal,
valuesSignal,
selectedValueSignal
}: {
showSignal: Signal<boolean>,
coordSignal: Signal<{ x: number; y: number }>,
valuesSignal: Signal<number[]>
selectedValueSignal: Signal<number>
}) => {
useSignals();
const handleMouseLeave = () => {
showSignal.value = false;
}
const handleMouseEnterItem = (scale: number) => {
selectedValueSignal.value = scale;
console.log("selectedValueSignal", selectedValueSignal.value);
}
return (
<div
className="fixed z-50 bg-red-600 -translate-x-1/2 -translate-y-1/2 flex-col gap-[1px] shadow-md bg-ui-divider border-ui-divider border-2 rounded-md overflow-clip"
style={{
top: coordSignal.value.y,
left: coordSignal.value.x,
display: showSignal.value ? "flex" : "none"
}}
onMouseLeave={handleMouseLeave}
>
{valuesSignal.value.map((scale, index) => (
<span onMouseEnter={() => handleMouseEnterItem(scale)} key={index}
className="flex bg-ui-panel text-sm justify-center align-middle justify-items-center px-2 hover:bg-ui-controls"
>{scale}</span>
))}
</div>
)
}
@@ -1,53 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { EditorStates } from "~/pages/PageEnigma/enums";
import { pageHeight, pageWidth } from "~/signals";
import { editorState, previewSrc } from "~/pages/PageEnigma/signals/engine";
import {
timelineHeight,
sidePanelWidth,
sidePanelVisible,
} from "~/pages/PageEnigma/signals";
export const PreviewFrameImage = () => {
useSignals();
if (editorState.value === EditorStates.PREVIEW) {
if (previewSrc.value === "") {
return (
<div
className="absolute inset-0"
style={{
width:
pageWidth.value -
(sidePanelVisible.value ? sidePanelWidth.value : 0) -
84,
height: pageHeight.value - timelineHeight.value - 56,
}}
>
<div className="relative flex h-full w-full flex-col items-center justify-center gap-5">
<span className="absolute h-full w-full bg-black opacity-50" />
<FontAwesomeIcon icon={faSpinnerThird} spin size="4x" />
<h3 className="z-20 text-white">Generating Preview...</h3>
</div>
</div>
);
} else {
return (
<img
alt="preview of the art style that renders over the 3d scene"
className="absolute inset-0 object-cover"
src={previewSrc.value}
style={{
width:
pageWidth.value -
(sidePanelVisible.value ? sidePanelWidth.value : 0) -
84,
height: pageHeight.value - timelineHeight.value - 56,
}}
/>
);
}
}
return null;
};
@@ -1,51 +0,0 @@
import { useState } from "react";
import { useSignals } from "@preact/signals-react/runtime";
import {
dndSidePanelWidth,
sidePanelVisible,
sidePanelWidth,
} from "~/pages/PageEnigma/signals";
import { SidePanelTabs } from "~/pages/PageEnigma/comps/SidePanelTabs";
import { SidePanelMenu } from "~/pages/PageEnigma/comps/SidePanelMenu";
import { TabItem, tabList } from "./tabList";
export const SidePanel = () => {
useSignals();
const initialTabIdx = 0;
const [selectedTab, setSelectedTab] = useState<TabItem>(
tabList[initialTabIdx],
);
const displayWidth =
dndSidePanelWidth.value > -1
? dndSidePanelWidth.value
: sidePanelWidth.value;
return (
<>
<div
className={[
"fixed flex border-l border-[#363636] bg-ui-panel transition-all duration-100",
].join(" ")}
style={{
top: 64,
right: sidePanelVisible.value ? 84 : -400,
width: displayWidth,
}}
>
<div className="relative block h-full w-full bg-ui-panel">
<SidePanelTabs tabs={tabList} selectedTab={selectedTab} />
</div>
</div>
<SidePanelMenu
tabs={tabList}
selectedTab={selectedTab}
selectTab={(newSelectedTab) => {
setSelectedTab(newSelectedTab);
}}
/>
</>
);
};
@@ -1 +0,0 @@
export { SidePanel } from './SidePanel';
@@ -1,58 +0,0 @@
import { ReactNode } from "react";
import { FontAwesomeIconProps } from "@fortawesome/react-fontawesome";
import {
faAlienMonster,
faCloud,
faCube,
faMountainCity,
faPresentationScreen,
faUserAstronaut,
} from "@fortawesome/pro-solid-svg-icons";
import { TabTitles } from "~/enums";
import { CartoonsTab } from "../SidePanelTabs/tabComps/CharactersComboTabs";
import { ObjectsTab } from "../SidePanelTabs/tabComps/ObjectsTab";
import { SkyboxesTab } from "../SidePanelTabs/tabComps/SkyboxesTab";
import { CreaturesTab } from "../SidePanelTabs/tabComps/CreaturesTab";
import { PlanesTab } from "~/pages/PageEnigma/comps/SidePanelTabs/tabComps/PlanesTab";
import { SetsTab } from "~/pages/PageEnigma/comps/SidePanelTabs/tabComps/SetsTab";
export interface TabItem {
icon: FontAwesomeIconProps["icon"];
title: string;
component: ReactNode;
}
export const tabList: TabItem[] = [
{
icon: faMountainCity,
title: TabTitles.OBJECTS_SETS,
component: <SetsTab />,
},
{
icon: faUserAstronaut,
title: TabTitles.GROUP_CARTOONS,
component: <CartoonsTab />,
},
{
icon: faCube,
title: TabTitles.OBJECTS,
component: <ObjectsTab />,
},
{
icon: faAlienMonster,
title: TabTitles.OBJECTS_CREATURES,
component: <CreaturesTab />,
},
{
icon: faPresentationScreen,
title: TabTitles.PLANES,
component: <PlanesTab />,
},
{
icon: faCloud,
title: TabTitles.SKYBOXES,
component: <SkyboxesTab />,
},
];
@@ -1,91 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { twMerge } from "tailwind-merge";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { FeatureFlags, TabTitles } from "~/enums";
import { EditorStates } from "~/pages/PageEnigma/enums";
import { editorState } from "~/pages/PageEnigma/signals/engine";
import { sidePanelVisible } from "~/pages/PageEnigma/signals/sidePanel";
import Queue from "~/pages/PageEnigma/Queue/Queue";
import { QueueNames } from "~/pages/PageEnigma/Queue/QueueNames";
import { toEngineActions } from "~/pages/PageEnigma/Queue/toEngineActions";
import { TabItem } from "../SidePanel/tabList";
import { currentPage, pageHeight } from "~/signals";
import { Pages } from "~/pages/PageEnigma/constants/page";
export const SidePanelMenu = ({
tabs,
selectTab,
selectedTab,
}: {
tabs: TabItem[];
selectTab: (newSelectedTab: TabItem) => void;
selectedTab: TabItem;
}) => {
useSignals();
const showStylePage = usePosthogFeatureFlag(FeatureFlags.SHOW_STYLE_PAGE);
return (
<div
className={twMerge(
"fixed bg-assets-background",
"right-0 top-[64px] w-[84px] overflow-auto border-l px-2 py-2",
sidePanelVisible.value ? "border-transparent" : "border-[#363636]",
)}
style={{
height: pageHeight.value - 56,
}}
>
<div className="flex w-full flex-col gap-2">
{(tabs ?? []).map((tab) => {
return (
<button
key={tab.title}
className={twMerge([
"flex flex-col items-center rounded-lg border border-transparent px-2 py-3 transition-all duration-200 hover:bg-brand-secondary-900/75",
tab.title === selectedTab.title && sidePanelVisible.value
? "border-[#363636] bg-brand-secondary-900/60 opacity-100 hover:bg-brand-secondary-900/60"
: "opacity-60",
tab.title === TabTitles.STYLIZE &&
"bg-brand-primary font-medium opacity-90 hover:border-white/25 hover:bg-brand-primary hover:opacity-100",
tab.title === selectedTab.title &&
tab.title === TabTitles.STYLIZE
? "border-white/50 opacity-100 hover:border-white/50"
: "",
])}
onClick={() => {
if (tab.title === TabTitles.RENDER) {
currentPage.value = Pages.STYLE;
return;
}
selectTab(tab);
if (!sidePanelVisible.value) {
sidePanelVisible.value = true;
}
if (editorState.value === EditorStates.PREVIEW) {
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.ENTER_EDIT_STATE,
data: null,
});
}
}}
>
<div>
<FontAwesomeIcon icon={tab.icon} size="lg" />
</div>
<div
className="-mb-1 mt-1 leading-[14px]"
style={{ fontSize: 11 }}
>
{tab.title}
</div>
</button>
);
})}
</div>
</div>
);
};
@@ -1 +0,0 @@
export * from "./SidePanelMenu";
@@ -1,40 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { sidePanelHeight } from "~/pages/PageEnigma/signals";
import { useMouseEventsSidePanel } from "~/pages/PageEnigma/comps/Timeline/utils/useMouseEventsSidePanel";
import { TabItem } from "../SidePanel/tabList";
export const SidePanelTabs = ({
selectedTab,
tabs,
}: {
selectedTab: TabItem;
tabs: TabItem[];
}) => {
useSignals();
const { onPointerDown } = useMouseEventsSidePanel();
return (
<>
<div style={{ height: sidePanelHeight.value, width: "100%" }}>
{tabs.map((tab, index) => (
<div
key={index}
className={
tab.title === selectedTab.title
? "flex h-full flex-col gap-3.5 overflow-y-auto"
: "hidden"
}
>
{tab.component}
</div>
))}
</div>
<div
className="absolute inset-0 block w-1 cursor-ew-resize"
style={{ height: sidePanelHeight.value }}
onPointerDown={onPointerDown}
/>
</>
);
};
@@ -1,5 +0,0 @@
export * from "./useUserObjects";
export * from "./useFeaturedObjects";
export * from "./useSearchFeaturedObjects";
export * from "./useSearchUserObjects";
export * from "./useFeatureFlags";
@@ -1,19 +0,0 @@
import { useSignals } from "@preact/signals-react/runtime";
import { authentication } from "~/signals";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
import { FeatureFlags } from "~/enums";
export const useFeatureFlags = () => {
useSignals();
const showSearchObjectComponent = usePosthogFeatureFlag(
FeatureFlags.SHOW_SEARCH_OBJECTS,
);
const showUploadButton =
usePosthogFeatureFlag(FeatureFlags.DEV_ONLY) || authentication.canUpload3D;
return {
showSearchObjectComponent,
showUploadButton,
};
};
@@ -1,115 +0,0 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { FetchStatus } from "~/pages/PageEnigma/enums";
import {
FetchMediaItemStates,
fetchFeaturedMediaItemsSearchResults,
} from "../utilities";
import { FilterEngineCategories, FilterMediaType } from "~/enums";
import { MediaItem } from "~/pages/PageEnigma/models";
import deepEqual from "deep-equal";
interface useSearchFeaturedObjectsProps {
defaultErrorMessage: string;
filterEngineCategories: FilterEngineCategories[];
demoFeaturedObjects?: MediaItem[];
filterMediaTypes?: FilterMediaType[];
}
export const useSearchFeaturedObjects = ({
demoFeaturedObjects,
...props
}: useSearchFeaturedObjectsProps) => {
const [searchTerm, setSearchTerm] = useState("");
const lastSearchTerm = useRef(searchTerm);
const updateSearchTermForFeaturedObjects = (newTerm: string) => {
setSearchTerm(newTerm);
};
const [
{
mediaItems: featuredObjectsSearchResults,
status: featuredObjectsSearchFetchStatus,
},
setFeaturedSearchFetch,
] = useState<FetchMediaItemStates>({
mediaItems: undefined,
status: FetchStatus.READY,
});
const demoItemsRef = useRef<MediaItem[]>([]);
if (
demoFeaturedObjects &&
!deepEqual(demoItemsRef.current, demoFeaturedObjects)
) {
demoItemsRef.current = demoFeaturedObjects;
}
const fetchFeaturedObjectSearchResults = useCallback(
async (term: string) => {
const filteredObjectItems = demoItemsRef.current.filter((item) =>
item.name.toLowerCase().includes(term.toLowerCase()),
);
if (!term || !term.trim()) {
//if after trim it's empty, do nothing
return;
}
let breakFlag = false;
setFeaturedSearchFetch((curr) => {
if (curr.status === FetchStatus.IN_PROGRESS) {
breakFlag = true;
return curr;
}
return {
...curr,
status: FetchStatus.IN_PROGRESS,
};
});
if (breakFlag) {
return;
}
const { filterEngineCategories, filterMediaTypes, defaultErrorMessage } =
props;
const result = await fetchFeaturedMediaItemsSearchResults({
filterEngineCategories: filterEngineCategories,
filterMediaType: filterMediaTypes,
defaultErrorMessage: defaultErrorMessage,
searchTerm: term,
});
setFeaturedSearchFetch({
status: result.status,
mediaItems: [...filteredObjectItems, ...(result.mediaItems ?? [])],
});
},
[props],
);
useEffect(() => {
let timer: NodeJS.Timeout;
const KEY_DELAY = 500;
const delayedFetch = (term: string) => {
if (term === lastSearchTerm.current) {
fetchFeaturedObjectSearchResults(searchTerm);
}
};
if (searchTerm !== lastSearchTerm.current) {
lastSearchTerm.current = searchTerm;
timer = setTimeout(() => delayedFetch(searchTerm), KEY_DELAY);
}
return () => {
clearTimeout(timer);
};
}, [fetchFeaturedObjectSearchResults, searchTerm]);
return {
searchTermForFeaturedObjects: searchTerm,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
fetchFeaturedObjectSearchResults,
updateSearchTermForFeaturedObjects,
};
};
@@ -1,95 +0,0 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { FetchStatus } from "~/pages/PageEnigma/enums";
import {
FetchMediaItemStates,
fetchUserMediaItemsSearchResults,
} from "../utilities";
import { FilterEngineCategories, FilterMediaType } from "~/enums";
interface useSearchUserObjectsProps {
defaultErrorMessage: string;
filterEngineCategories: FilterEngineCategories[];
filterMediaTypes?: FilterMediaType[];
}
export const useSearchUserObjects = (props: useSearchUserObjectsProps) => {
const [searchTerm, setSearchTerm] = useState("");
const lastSearchTerm = useRef(searchTerm);
const updateSearchTermForUserObjects = (newTerm: string) => {
setSearchTerm(newTerm);
};
const [
{
mediaItems: userObjectsSearchResults,
status: userObjectsSearchFetchStatus,
},
setUserSearchFetch,
] = useState<FetchMediaItemStates>({
mediaItems: undefined,
status: FetchStatus.READY,
});
const fetchUserObjectsSearchResults = useCallback(
async (term: string) => {
if (!term || !term.trim()) {
//if after trim it's empty, do nothing
return;
}
let breakFlag = false;
setUserSearchFetch((curr) => {
if (curr.status === FetchStatus.IN_PROGRESS) {
breakFlag = true;
return curr;
}
return {
...curr,
status: FetchStatus.IN_PROGRESS,
};
});
if (breakFlag) {
return;
}
const { filterEngineCategories, filterMediaTypes, defaultErrorMessage } =
props;
const result = await fetchUserMediaItemsSearchResults({
filterEngineCategories: filterEngineCategories,
filterMediaType: filterMediaTypes,
defaultErrorMessage: defaultErrorMessage,
searchTerm: term,
});
setUserSearchFetch({
status: result.status,
mediaItems: result.mediaItems,
});
},
[props],
);
useEffect(() => {
let timer: NodeJS.Timeout;
const KEY_DELAY = 500;
const delayedFetch = (term: string) => {
if (term === lastSearchTerm.current) {
fetchUserObjectsSearchResults(searchTerm);
}
};
if (searchTerm !== lastSearchTerm.current) {
lastSearchTerm.current = searchTerm;
timer = setTimeout(() => delayedFetch(searchTerm), KEY_DELAY);
}
return () => {
clearTimeout(timer);
};
}, [fetchUserObjectsSearchResults, searchTerm]);
return {
searchTermForUserObjects: searchTerm,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
fetchUserObjectsSearchResults,
updateSearchTermForUserObjects,
};
};
@@ -1 +0,0 @@
export * from "./SidePanelTabs";
@@ -1,45 +0,0 @@
import { IconDefinition } from "@fortawesome/fontawesome-svg-core";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { twMerge } from "tailwind-merge";
import { TabTitles } from "~/enums";
export const SubTabButtons = ({
currSubpage,
subPageTitles,
setSubpage,
subPageTitleIcons,
}: {
currSubpage: TabTitles;
subPageTitles: TabTitles[];
subPageTitleIcons?: IconDefinition[];
setSubpage: (newTab: TabTitles) => void;
}) => {
const leftMostButtonCss = "rounded-l-lg";
const rightMostButtonCss = "rounded-r-lg";
return (
<div className="mx-4 flex">
{subPageTitles.map((subPageTitle, idx) => {
return (
<button
key={idx}
className={twMerge(
"flex h-8 grow cursor-pointer items-center justify-center gap-2 bg-brand-secondary p-2 text-xs font-medium transition-all",
idx === 0 && leftMostButtonCss,
idx === subPageTitles.length - 1 && rightMostButtonCss,
currSubpage === subPageTitle
? "bg-brand-primary/85"
: "hover:bg-brand-secondary-900",
)}
disabled={currSubpage === subPageTitle}
onClick={() => setSubpage(subPageTitle)}
>
{subPageTitleIcons && subPageTitleIcons[idx] && (
<FontAwesomeIcon icon={subPageTitleIcons[idx]} />
)}
{subPageTitle}
</button>
);
})}
</div>
);
};
@@ -1,36 +0,0 @@
import { faChevronLeft, faClose } from "@fortawesome/pro-solid-svg-icons";
import { ButtonIcon } from "@storyteller/ui-button-icon";
import { sidePanelVisible } from "~/pages/PageEnigma/signals";
interface Props {
title: string;
onBack?: () => void;
}
export function TabTitle({ title, onBack }: Props) {
const onClose = () => {
sidePanelVisible.value = false;
};
return (
<div className="flex items-center justify-between px-4 pt-4">
{onBack ? (
<div className="flex items-center gap-3">
<ButtonIcon
onClick={onBack}
icon={faChevronLeft}
className="h-auto w-auto text-xl opacity-50 hover:opacity-90"
/>
<div className="align-middle text-base font-bold">{title}</div>
</div>
) : (
<div className="align-middle text-base font-bold">{title}</div>
)}
<ButtonIcon
onClick={onClose}
icon={faClose}
className="h-auto w-auto text-xl opacity-50 hover:opacity-90"
/>
</div>
);
}
@@ -1,34 +0,0 @@
import { useState } from "react";
import { faPerson, faRabbitRunning } from "@fortawesome/pro-solid-svg-icons";
import { MediaFileAnimationType, TabTitles } from "~/enums";
import { TabTitle } from "../../sharedComps";
import { AnimationsTab } from "./subtabAnimations";
import { CharactersTab } from "./subtabCharacters";
import { SubTabButtons } from "../../sharedComps/SubTabButtons";
export const AnimeTab = () => {
const [currSubpage, setCurrSubpage] = useState<TabTitles>(
TabTitles.CHARACTERS,
);
return (
<>
<TabTitle title={TabTitles.GROUP_ANIME} />
<SubTabButtons
currSubpage={currSubpage}
setSubpage={(newPage) => {
setCurrSubpage(newPage);
}}
subPageTitles={[TabTitles.CHARACTERS]} // , TabTitles.ANIMATION
subPageTitleIcons={[faPerson]} // faRabbitRunning
/>
{currSubpage === TabTitles.CHARACTERS && (
<CharactersTab animationType={MediaFileAnimationType.MikuMikuDance} />
)}
{/* {currSubpage === TabTitles.ANIMATION && (
<AnimationsTab animationType={MediaFileAnimationType.MikuMikuDance} />
)} */}
</>
);
};
@@ -1,52 +0,0 @@
import { useState } from "react";
import { useSignals } from "@preact/signals-react/runtime";
import {
faFaceSmileWink,
faPerson,
faRabbitRunning,
} from "@fortawesome/pro-solid-svg-icons";
import { MediaFileAnimationType, TabTitles } from "~/enums";
import { TabTitle } from "../../sharedComps";
import { AnimationsTab } from "./subtabAnimations";
import { CharactersTab } from "./subtabCharacters";
import { ExpressionTab } from "./subtabExpressions";
import { SubTabButtons } from "../../sharedComps/SubTabButtons";
import {
demoCharacterItems,
demoAnimationItems,
} from "~/pages/PageEnigma/signals";
export const CartoonsTab = () => {
useSignals();
const [currSubpage, setCurrSubpage] = useState<TabTitles>(
TabTitles.CHARACTERS,
);
return (
<>
<TabTitle title={TabTitles.GROUP_CARTOONS} />
<SubTabButtons
currSubpage={currSubpage}
setSubpage={(newPage) => {
setCurrSubpage(newPage);
}}
subPageTitles={[
TabTitles.CHARACTERS,
// TabTitles.ANIMATION,
// TabTitles.EXPRESSIONS,
]}
subPageTitleIcons={[faPerson, faRabbitRunning, faFaceSmileWink]}
/>
{currSubpage === TabTitles.CHARACTERS && (
<CharactersTab
animationType={MediaFileAnimationType.Mixamo}
demoCharacterItems={demoCharacterItems.value}
/>
)}
</>
);
};
@@ -1,32 +0,0 @@
import { FilterMediaType, MediaFileAnimationType } from "~/enums";
import { MediaItem } from "~/pages/PageEnigma/models";
export const filterMMDCharacters = (item: MediaItem) => {
return (
item.maybe_animation_type === MediaFileAnimationType.MikuMikuDance ||
item.media_type === FilterMediaType.PMD ||
item.media_type === FilterMediaType.PMX
);
};
export const filterMixamoCharacters = (item: MediaItem) => {
return (
item.maybe_animation_type === MediaFileAnimationType.Mixamo ||
(item.media_type !== FilterMediaType.PMD &&
item.media_type !== FilterMediaType.PMX)
);
};
export const filterMMDAnimations = (item: MediaItem) => {
return (
item.maybe_animation_type === MediaFileAnimationType.MikuMikuDance ||
item.media_type === FilterMediaType.VMD
);
};
export const filterMixamoAnimations = (item: MediaItem) => {
return (
// item.maybe_animation_type === MediaFileAnimationType.Mixamo &&
item.media_type !== FilterMediaType.VMD
);
};
@@ -1,2 +0,0 @@
export * from "./AnimeTab";
export * from "./CartoonsTab";
@@ -1,208 +0,0 @@
import { useEffect, useMemo, useState } from "react";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
ANIMATION_MIXAMO_FILE_TYPE,
ANIMATION_MMD_FILE_TYPE,
AssetFilterOption,
FilterEngineCategories,
MediaFileAnimationType,
} from "~/enums";
import {
UploadModal,
} from "@storyteller/ui-upload-modal";
import { SearchFilter } from "@storyteller/ui-search";
import { Pagination } from "@storyteller/ui-pagination";
import { Button, FilterButtons } from "@storyteller/ui-button";
import { ItemElements } from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import { isAnyStatusFetching } from "../../utilities";
import {
useUserObjects,
useFeaturedObjects,
useSearchFeaturedObjects,
useSearchUserObjects,
useFeatureFlags,
} from "../../hooks";
import {
filterMixamoAnimations,
filterMMDAnimations,
} from "./filterCharacterTypes";
import { MediaItem } from "~/pages/PageEnigma/models";
import { getFileExtension, getFileName } from "~/utilities";
const filterEngineCategories = [FilterEngineCategories.ANIMATION];
export const AnimationsTab = ({
animationType,
demoAnimationItems,
}: {
animationType: MediaFileAnimationType;
demoAnimationItems?: MediaItem[];
}) => {
const { showSearchObjectComponent, showUploadButton } = useFeatureFlags();
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, userFetchStatus, fetchUserObjects } = useUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching User Animations",
});
const { featuredObjects, featuredFetchStatus } = useFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching Featured Animations",
});
const {
searchTermForFeaturedObjects,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
updateSearchTermForFeaturedObjects,
} = useSearchFeaturedObjects({
demoFeaturedObjects: demoAnimationItems,
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching Featured Animation Search Results",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching User Animation Search Results",
});
const [filterOwnership, setFilterOwnership] = useState(
AssetFilterOption.FEATURED,
);
const filterAnimationType = useMemo(
() =>
animationType === MediaFileAnimationType.Mixamo
? filterMixamoAnimations
: filterMMDAnimations,
[animationType],
);
const displayedItems =
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
? featuredObjectsSearchResults ?? []
: [...(demoAnimationItems ?? []), ...(featuredObjects ?? [])]
: searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const filteredDisplayItems = displayedItems.filter(filterAnimationType);
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(filteredDisplayItems.length / pageSize);
const isFetching = isAnyStatusFetching([
userFetchStatus,
featuredFetchStatus,
featuredObjectsSearchFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
useEffect(() => {
if (searchTermForFeaturedObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForFeaturedObjects]);
return (
<>
<FilterButtons
value={filterOwnership}
onClick={(buttonIdx) => {
setFilterOwnership(Number(buttonIdx));
setCurrentPage(0);
}}
/>
<div className="flex w-full flex-col gap-3 px-4">
{showUploadButton && (
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Animation (Dev Only)
</Button>
)}
{showSearchObjectComponent && (
<SearchFilter
searchTerm={
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
: searchTermForUserObjects
}
onSearchChange={
filterOwnership === AssetFilterOption.FEATURED
? updateSearchTermForFeaturedObjects
: updateSearchTermForUserObjects
}
key={filterOwnership}
placeholder={
filterOwnership === AssetFilterOption.FEATURED
? "Search featured animations"
: "Search my animations"
}
/>
)}
</div>
<div className="w-full grow overflow-y-auto px-4 pb-4">
<ItemElements
busy={isFetching}
debug="animations tab"
currentPage={currentPage}
pageSize={pageSize}
items={filteredDisplayItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
/>
)}
<UploadModal
getFileName={getFileName}
getFileExtension={getFileExtension}
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
type={FilterEngineCategories.ANIMATION}
fileTypes={Object.values(
animationType === MediaFileAnimationType.Mixamo
? ANIMATION_MIXAMO_FILE_TYPE
: ANIMATION_MMD_FILE_TYPE,
)}
title="Upload Animation"
options={{
fileSubtypes: [
{ [animationType]: animationType },
// { Mixamo: MediaFileAnimationType.Mixamo },
// { MikuMikuDance: MediaFileAnimationType.MikuMikuDance },
// { MoveAi: MediaFileAnimationType.MoveAi },
// { Rigify: MediaFileAnimationType.Rigify },
// { Rokoko: MediaFileAnimationType.Rokoko },
],
hasLength: true,
hasThumbnailUpload: true,
}}
/>
</>
);
};
@@ -1,221 +0,0 @@
import { useEffect, useState } from "react";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
AssetFilterOption,
CHARACTER_MIXAMO_FILE_TYPE,
CHARACTER_MMD_FILE_TYPE,
FilterEngineCategories,
FilterMediaType,
MediaFileAnimationType,
} from "~/enums";
import { MediaItem } from "~/pages/PageEnigma/models";
import {
UploadModal,
} from "@storyteller/ui-upload-modal";
import { SearchFilter } from "@storyteller/ui-search";
import { Pagination } from "@storyteller/ui-pagination";
import { Button, FilterButtons } from "@storyteller/ui-button";
import { ItemElements } from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import { isAnyStatusFetching } from "../../utilities";
import {
useUserObjects,
useFeaturedObjects,
useSearchFeaturedObjects,
useSearchUserObjects,
useFeatureFlags,
} from "../../hooks";
import { getFileExtension, getFileName } from "~/utilities";
const filterEngineCategories = [FilterEngineCategories.CHARACTER];
export const CharactersTab = ({
animationType,
demoCharacterItems = [],
}: {
animationType: MediaFileAnimationType;
demoCharacterItems?: MediaItem[];
}) => {
const fileTypes = Object.values(
animationType === MediaFileAnimationType.Mixamo
? CHARACTER_MIXAMO_FILE_TYPE
: CHARACTER_MMD_FILE_TYPE,
);
const { showSearchObjectComponent, showUploadButton } = useFeatureFlags();
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, nextUserObjects, userFetchStatus, fetchUserObjects } =
useUserObjects({
filterEngineCategories: filterEngineCategories,
filterMediaTypes: fileTypes as FilterMediaType[],
defaultErrorMessage: "Unknown Error in Fetching User Characters",
});
const {
featuredObjects,
nextFeaturedObjects,
fetchFeaturedObjects,
featuredFetchStatus,
} = useFeaturedObjects({
filterEngineCategories: filterEngineCategories,
filterMediaTypes: fileTypes as FilterMediaType[],
defaultErrorMessage: "Unknown Error in Fetching Featured Characters",
});
const {
searchTermForFeaturedObjects,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
updateSearchTermForFeaturedObjects,
} = useSearchFeaturedObjects({
demoFeaturedObjects: demoCharacterItems,
filterEngineCategories: filterEngineCategories,
filterMediaTypes: fileTypes as FilterMediaType[],
defaultErrorMessage:
"Unknown Error in Fetching Featured Characters Search Results",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
filterMediaTypes: fileTypes as FilterMediaType[],
defaultErrorMessage:
"Unknown Error in Fetching User Characters Search Results",
});
const [filterOwnership, setFilterOwnership] = useState(
AssetFilterOption.FEATURED,
);
const displayedItems =
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
? featuredObjectsSearchResults ?? []
: [...(demoCharacterItems ?? []), ...(featuredObjects ?? [])]
: searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(displayedItems.length / pageSize);
const fetchMorePages =
filterOwnership === AssetFilterOption.FEATURED
? nextFeaturedObjects?.maybe_next
? fetchFeaturedObjects
: undefined
: nextUserObjects &&
nextUserObjects.current !== nextUserObjects.total_page_count
? () => fetchUserObjects(nextUserObjects.current)
: undefined;
const isFetching = isAnyStatusFetching([
userFetchStatus,
featuredFetchStatus,
featuredObjectsSearchFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
useEffect(() => {
if (searchTermForFeaturedObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForFeaturedObjects]);
return (
<>
<div>
<FilterButtons
value={filterOwnership}
onClick={(button) => {
setFilterOwnership(Number(button));
setCurrentPage(0);
}}
/>
</div>
<div className="flex w-full flex-col gap-3 px-4">
{showUploadButton && (
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Character (Dev Only)
</Button>
)}
{showSearchObjectComponent && (
<SearchFilter
searchTerm={
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
: searchTermForUserObjects
}
onSearchChange={
filterOwnership === AssetFilterOption.FEATURED
? updateSearchTermForFeaturedObjects
: updateSearchTermForUserObjects
}
key={filterOwnership}
placeholder={
filterOwnership === AssetFilterOption.FEATURED
? "Search featured characters"
: "Search my characters"
}
/>
)}
</div>
<div className="w-full grow overflow-y-auto rounded px-4 pb-4">
<ItemElements
busy={isFetching}
debug="characters tab"
currentPage={currentPage}
pageSize={pageSize}
items={displayedItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
onFetchMorePages={fetchMorePages}
/>
)}
<UploadModal
getFileName={getFileName}
getFileExtension={getFileExtension}
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
type={FilterEngineCategories.CHARACTER}
options={{
hasThumbnailUpload: true,
fileSubtypes: [
{ [animationType]: animationType },
// { Mixamo: MediaFileAnimationType.Mixamo },
// { MikuMikuDance: MediaFileAnimationType.MikuMikuDance },
// { MoveAi: MediaFileAnimationType.MoveAi },
// { Rigify: MediaFileAnimationType.Rigify },
// { Rokoko: MediaFileAnimationType.Rokoko },
],
}}
fileTypes={fileTypes}
title="Upload Characters"
/>
</>
);
};
@@ -1,174 +0,0 @@
import { useEffect, useState } from "react";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
AssetFilterOption,
FilterEngineCategories,
MediaFileAnimationType,
} from "~/enums";
import {
UploadModal,
} from "@storyteller/ui-upload-modal";
import { SearchFilter } from "@storyteller/ui-search";
import { Pagination } from "@storyteller/ui-pagination";
import { Button, FilterButtons } from "@storyteller/ui-button";
import { ItemElements } from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import { isAnyStatusFetching } from "../../utilities";
import {
useUserObjects,
useFeaturedObjects,
useSearchFeaturedObjects,
useSearchUserObjects,
useFeatureFlags,
} from "../../hooks";
import { getFileExtension, getFileName } from "~/utilities";
const filterEngineCategories = [FilterEngineCategories.EXPRESSION];
export const ExpressionTab = () => {
const { showSearchObjectComponent, showUploadButton } = useFeatureFlags();
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, userFetchStatus, fetchUserObjects } = useUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching User Expressions",
});
const { featuredObjects, featuredFetchStatus } = useFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching Featured Expressions",
});
const {
searchTermForFeaturedObjects,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
updateSearchTermForFeaturedObjects,
} = useSearchFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching Featured Expressions Search Results",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching User Expressions Search Results",
});
const [filterOwnership, setFilterOwnership] = useState(
AssetFilterOption.FEATURED,
);
const displayedItems =
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
? featuredObjectsSearchResults ?? []
: featuredObjects ?? []
: searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(displayedItems.length / pageSize);
const isFetching = isAnyStatusFetching([
userFetchStatus,
featuredFetchStatus,
featuredObjectsSearchFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
useEffect(() => {
if (searchTermForFeaturedObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForFeaturedObjects]);
return (
<>
<FilterButtons
value={filterOwnership}
onClick={(button) => {
setFilterOwnership(button);
setCurrentPage(0);
}}
/>
<div className="flex w-full flex-col gap-3 px-4">
{showUploadButton && (
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Expression (Dev Only)
</Button>
)}
{showSearchObjectComponent && (
<SearchFilter
searchTerm={
filterOwnership === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
: searchTermForUserObjects
}
onSearchChange={
filterOwnership === AssetFilterOption.FEATURED
? updateSearchTermForFeaturedObjects
: updateSearchTermForUserObjects
}
key={filterOwnership}
placeholder={
filterOwnership === AssetFilterOption.FEATURED
? "Search featured expressions"
: "Search my expressions"
}
/>
)}
</div>
<div className="h-full w-full overflow-y-auto px-4 pb-4">
<ItemElements
busy={isFetching}
currentPage={currentPage}
pageSize={pageSize}
items={displayedItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
/>
)}
<UploadModal
getFileName={getFileName}
getFileExtension={getFileExtension}
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
type={FilterEngineCategories.EXPRESSION}
fileTypes={["CSV"]}
title="Upload Expression"
options={{
fileSubtypes: [{ ARKit: MediaFileAnimationType.ArKit }],
hasLength: true,
hasThumbnailUpload: true,
}}
/>
</>
);
};
@@ -1,177 +0,0 @@
import { useEffect, useState } from "react";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
AssetFilterOption,
FeatureFlags,
FilterEngineCategories,
OBJECT_FILE_TYPE,
TabTitles,
} from "~/enums";
import {
TabTitle,
ItemElements,
} from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import {
UploadModal3D,
} from "~/components";
import { SearchFilter } from "@storyteller/ui-search";
import { Pagination } from "@storyteller/ui-pagination";
import { Button, FilterButtons } from "@storyteller/ui-button"
import { isAnyStatusFetching } from "../utilities";
import {
useUserObjects,
useFeaturedObjects,
useSearchFeaturedObjects,
useSearchUserObjects,
} from "../hooks";
const filterEngineCategories = [FilterEngineCategories.CREATURE];
export const CreaturesTab = () => {
const showSearchObjectComponent = usePosthogFeatureFlag(
FeatureFlags.SHOW_SEARCH_OBJECTS,
);
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, userFetchStatus, fetchUserObjects } = useUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching User Creature Objects",
});
const { featuredObjects, featuredFetchStatus } = useFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching Featured Creature Objects",
});
const {
searchTermForFeaturedObjects,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
updateSearchTermForFeaturedObjects,
} = useSearchFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching Featured Creature Objects Search Results",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching User Creature Objects Search Results",
});
const [selectedFilter, setSelectedFilter] = useState(
AssetFilterOption.FEATURED,
);
const displayedItems =
selectedFilter === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
? featuredObjectsSearchResults ?? []
: featuredObjects ?? []
: searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(displayedItems.length / pageSize);
const isFetching = isAnyStatusFetching([
userFetchStatus,
featuredFetchStatus,
featuredObjectsSearchFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
useEffect(() => {
if (searchTermForFeaturedObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForFeaturedObjects]);
return (
<>
<TabTitle title={TabTitles.OBJECTS_CREATURES} />
<FilterButtons
value={selectedFilter}
onClick={(button) => {
setSelectedFilter(Number(button));
setCurrentPage(0);
}}
/>
<div className="flex w-full flex-col gap-3 px-4">
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Creatures
</Button>
{showSearchObjectComponent && (
<SearchFilter
searchTerm={
selectedFilter === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
: searchTermForUserObjects
}
onSearchChange={
selectedFilter === AssetFilterOption.FEATURED
? updateSearchTermForFeaturedObjects
: updateSearchTermForUserObjects
}
key={selectedFilter}
placeholder={
selectedFilter === AssetFilterOption.FEATURED
? "Search featured creature objects"
: "Search my creature objects"
}
/>
)}
</div>
<div className="w-full grow overflow-y-auto rounded px-4 pb-4">
<ItemElements
busy={isFetching}
debug="objects tab"
currentPage={currentPage}
pageSize={pageSize}
items={displayedItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
/>
)}
<UploadModal3D
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
engineCategory={FilterEngineCategories.CREATURE}
fileTypes={Object.values(OBJECT_FILE_TYPE)}
title="Upload Creatures"
/>
</>
);
};
@@ -1,178 +0,0 @@
import { useEffect, useState } from "react";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
AssetFilterOption,
FeatureFlags,
FilterEngineCategories,
OBJECT_FILE_TYPE,
TabTitles,
} from "~/enums";
import { demoShapeItems } from "~/pages/PageEnigma/signals";
import {
TabTitle,
ItemElements,
} from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import {
UploadModal3D,
} from "~/components";
import { SearchFilter } from "@storyteller/ui-search";
import { Pagination } from "@storyteller/ui-pagination";
import { Button, FilterButtons } from "@storyteller/ui-button"
import { isAnyStatusFetching } from "../utilities";
import {
useUserObjects,
useFeaturedObjects,
useSearchFeaturedObjects,
useSearchUserObjects,
} from "../hooks";
const filterEngineCategories = [
FilterEngineCategories.SET_DRESSING,
FilterEngineCategories.OBJECT,
];
export const ObjectsTab = () => {
const showSearchObjectComponent = usePosthogFeatureFlag(
FeatureFlags.SHOW_SEARCH_OBJECTS,
);
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, userFetchStatus, fetchUserObjects } = useUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching User Objects",
});
const { featuredObjects, featuredFetchStatus } = useFeaturedObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching Featured Objects",
});
const {
searchTermForFeaturedObjects,
featuredObjectsSearchResults,
featuredObjectsSearchFetchStatus,
updateSearchTermForFeaturedObjects,
} = useSearchFeaturedObjects({
demoFeaturedObjects: demoShapeItems.value,
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching Featured Objects Search Results",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching User Objects Search Results",
});
const [selectedFilter, setSelectedFilter] = useState(
AssetFilterOption.FEATURED,
);
const displayedItems =
selectedFilter === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
? featuredObjectsSearchResults ?? []
: [...demoShapeItems.value, ...(featuredObjects ?? [])]
: searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(displayedItems.length / pageSize);
const isFetching = isAnyStatusFetching([
userFetchStatus,
featuredFetchStatus,
featuredObjectsSearchFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
useEffect(() => {
if (searchTermForFeaturedObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForFeaturedObjects]);
return (
<>
<TabTitle title={TabTitles.OBJECTS} />
<FilterButtons
value={selectedFilter}
onClick={(button) => {
setSelectedFilter(Number(button));
setCurrentPage(0);
}}
/>
<div className="flex w-full flex-col gap-3 px-4">
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Props or Other Objects
</Button>
{showSearchObjectComponent && (
<SearchFilter
searchTerm={
selectedFilter === AssetFilterOption.FEATURED
? searchTermForFeaturedObjects
: searchTermForUserObjects
}
onSearchChange={
selectedFilter === AssetFilterOption.FEATURED
? updateSearchTermForFeaturedObjects
: updateSearchTermForUserObjects
}
key={selectedFilter}
placeholder={
selectedFilter === AssetFilterOption.FEATURED
? "Search featured objects"
: "Search my objects"
}
/>
)}
</div>
<div className="w-full grow overflow-y-auto rounded px-4 pb-4">
<ItemElements
busy={isFetching}
debug="objects tab"
currentPage={currentPage}
pageSize={pageSize}
items={displayedItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
/>
)}
<UploadModal3D
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
engineCategory={FilterEngineCategories.SET_DRESSING}
fileTypes={Object.values(OBJECT_FILE_TYPE)}
title="Upload Props or Other Objects"
/>
</>
);
};
@@ -1,29 +0,0 @@
import { useState } from "react";
import { TabTitles } from "~/enums";
import { TabTitle } from "../../sharedComps";
import { ImagePlanesTab } from "./subTabImagePlanes";
import { SubTabButtons } from "../../sharedComps/SubTabButtons";
import { VideoPlanesTab } from "./subTabVideoPlanes";
export const PlanesTab = () => {
const [currSubpage, setCurrSubpage] = useState<TabTitles>(
TabTitles.PLANES_IMAGE,
);
return (
<>
<TabTitle title={TabTitles.PLANES} />
<SubTabButtons
currSubpage={currSubpage}
setSubpage={(newPage) => {
setCurrSubpage(newPage);
}}
subPageTitles={[TabTitles.PLANES_IMAGE, TabTitles.PLANES_VIDEO]}
/>
{currSubpage === TabTitles.PLANES_IMAGE && <ImagePlanesTab />}
{currSubpage === TabTitles.PLANES_VIDEO && <VideoPlanesTab />}
</>
);
};
@@ -1 +0,0 @@
export * from "./PlanesTab";
@@ -1,109 +0,0 @@
import { useEffect, useState } from "react";
import { faCirclePlus } from "@fortawesome/pro-solid-svg-icons";
import {
FeatureFlags,
FilterEngineCategories,
IMAGEPLANE_FILE_TYPE,
} from "~/enums";
import {
UploadModalMedia,
} from "~/components";
import { SearchFilter } from "@storyteller/ui-search";
import { Button } from "@storyteller/ui-button";
import { Pagination } from "@storyteller/ui-pagination";
import { ItemElements } from "~/pages/PageEnigma/comps/SidePanelTabs/sharedComps";
import { isAnyStatusFetching } from "../../utilities";
import { useUserObjects, useSearchUserObjects } from "../../hooks";
import { usePosthogFeatureFlag } from "~/hooks/usePosthogFeatureFlag";
const filterEngineCategories = [FilterEngineCategories.IMAGE_PLANE];
export const ImagePlanesTab = () => {
const showSearchObjectComponent = usePosthogFeatureFlag(
FeatureFlags.SHOW_SEARCH_OBJECTS,
);
const [openUploadModal, setOpenUploadModal] = useState(false);
const { userObjects, userFetchStatus, fetchUserObjects } = useUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage: "Unknown Error in Fetching User Image Panels",
});
const {
searchTermForUserObjects,
userObjectsSearchResults,
userObjectsSearchFetchStatus,
updateSearchTermForUserObjects,
} = useSearchUserObjects({
filterEngineCategories: filterEngineCategories,
defaultErrorMessage:
"Unknown Error in Fetching User Image Panel Search Results",
});
const displayedItems = searchTermForUserObjects
? userObjectsSearchResults ?? []
: userObjects ?? [];
const [currentPage, setCurrentPage] = useState<number>(0);
const pageSize = 21;
const totalPages = Math.ceil(displayedItems.length / pageSize);
const isFetching = isAnyStatusFetching([
userFetchStatus,
userObjectsSearchFetchStatus,
]);
useEffect(() => {
if (searchTermForUserObjects.length > 0) {
setCurrentPage(0);
}
}, [searchTermForUserObjects]);
return (
<>
<div className="flex w-full flex-col gap-3 px-4">
<Button
icon={faCirclePlus}
variant="action"
onClick={() => setOpenUploadModal(true)}
className="w-full py-3 text-sm font-medium"
>
Upload Image Panels
</Button>
{showSearchObjectComponent && (
<SearchFilter
searchTerm={searchTermForUserObjects}
onSearchChange={updateSearchTermForUserObjects}
placeholder={"Search my image panels"}
/>
)}
</div>
<div className="w-full grow overflow-y-auto px-4 pb-4">
<ItemElements
busy={isFetching}
currentPage={currentPage}
pageSize={pageSize}
items={displayedItems}
/>
</div>
{totalPages > 1 && (
<Pagination
className="-mt-4 px-4"
currentPage={currentPage}
totalPages={totalPages}
onPageChange={(newPage: number) => {
setCurrentPage(newPage);
}}
/>
)}
<UploadModalMedia
onClose={() => setOpenUploadModal(false)}
onSuccess={fetchUserObjects}
isOpen={openUploadModal}
fileTypes={Object.values(IMAGEPLANE_FILE_TYPE)}
title="Upload Image Panels"
/>
</>
);
};

Some files were not shown because too many files have changed in this diff Show More