Compare commits

..
Author SHA1 Message Date
bflatastic 3860f784a3 shared toaster component 2025-04-18 14:57:07 +07:00
BFlat e31a2c0a91 fix nx project discoverability (#527) 2025-04-18 14:10:40 +07:00
Kasisnu ea85abdc62 image-gen: make unused bools in task response optional 2025-04-18 10:34:52 +05:30
Brandon Thomas 4f3c93635f don't log debug level in prod 2025-04-18 00:07:51 -04:00
ArEnScandMichael Chung 82b9b90ff9 Added the apis to the api folder it loads correctly now (#525)
Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-17 22:49:27 -04:00
bflatastic ad5ea26a81 commented out test component it was pushing the ui down 2025-04-18 08:59:53 +07:00
bflatastic 6121d118f0 reduced topbar right side gap 2025-04-18 08:54:40 +07:00
bflatastic 28ca7be96c activity update 2025-04-18 08:50:55 +07:00
Brandon Thomas a2a7f7d274 could also be a bearer token issue 2025-04-17 21:39:45 -04:00
Brandon Thomas e1255a55a0 specific error message for cookie expiration 2025-04-17 21:30:16 -04:00
9f629ee5e2 Nx working test (#524)
It works now.

---------

Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Tanish Manku <tanish2k09.dev@gmail.com>
Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-17 16:32:30 -04:00
BFlat 6a82fcb6e0 Revert "shared component starting with toaster" (#522)
Reverts storytold/storyteller-rust#521

it was causing an application error on build.
2025-04-17 17:21:05 +07:00
BFlat 35eac53b9d shared component starting with toaster (#521)
- share the toaster component between both editor2d and editor 3d
2025-04-17 17:14:02 +07:00
bflatastic c84c36d774 escape key to deselect objects and exit FK pose mode 2025-04-17 13:33:12 +07:00
bflatastic 3269b62e14 editor2d: artcraft logo 2025-04-17 13:11:55 +07:00
BFlat cd49fe0015 pose mode toggle button on top when character is selected (#520)
- pose mode toggle button to go in and out of pose mode when a character
is selected
- tooltip text size change
- button styling fixes
2025-04-17 13:04:53 +07:00
ArEnScandMichael Chung 326fa8c4a4 Adding API imports (#519)
Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-16 20:16:12 -04:00
Tanish Manku 51bdd4eff8 Create a really sick precision selector (#506)
Steps to activate:
1 - Select an object
2 - In the transform box, click and hold "Scale"
3 - Drag vertically to select the incremental precision
4 - Drag horizontally to apply increments of the selected precision

Issues:
- Releasing the mouse on top of the canvas will also trigger a "mouseup"
event on the canvas (since it listens to the window, not the canvas) and
deselect the object.
2025-04-16 15:01:44 -07:00
BFlat a64ca7d4ac editor3d: activity card improvements and share lightbox with gallery (#518)
**activity:**
- dropdown cards to show better download icon for completed items
- show spinner for in progress items
- activity button itself to show a spinner when at least one job is
running
- use the new lightbox component
- click the completed item to view image in lightbox

**gallery (library modal):**
- use the new lightbox component
- the new lightbox has a download button
- use cdn images
2025-04-17 02:48:25 +07:00
Kasisnu ec3ac242b0 image-gen: generate sentinel token in rust 2025-04-16 16:19:32 +05:30
Brandon Thomas fbe7fa3c77 don't leak logged in session jobs when logging out 2025-04-15 23:26:58 -04:00
Brandon Thomas 418d824c35 session jobs for anonymous users 2025-04-15 22:43:53 -04:00
Tanish Manku 2e6823834f Create the shapes centered on the canvas (#515)
- Also match all the shape sizes
2025-04-15 17:24:29 -07:00
ArEnScandMichael Chung 7ef20bc885 Feature/system prompt modification (#514)
Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-15 18:32:43 -04:00
Michael Chung b031a68000 Redo and Undo for osx fixed correctly.
Issues with the app not properly loading not allowing the engine reference to be set.
2025-04-15 16:35:47 -04:00
Brandon Thomas e4b648b55d bump Rust toolchains to 1.86.0 2025-04-15 15:05:06 -04:00
Brandon Thomas d78ccbca25 system prompt enhancement 2025-04-15 14:57:16 -04:00
Kasisnu bae6bc6863 move sentinel token generation into rust 2025-04-15 21:44:58 +05:30
Brandon Thomas faa43a7361 remove node-sass 2025-04-15 05:14:34 -04:00
Brandon Thomas f8b3b708c0 unblock CI builds for now 2025-04-15 05:05:21 -04:00
ArEnScandMichael Chung 4c40379470 Jobs should add the image to the list (#512)
Jobs also show completed images in the gallery.

Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-15 02:49:45 -04:00
Brandon Thomas 0c32a89828 script to clean up node modules 2025-04-15 00:30:57 -04:00
Brandon Thomas d7365ea045 remove more entries from package-lock.json 2025-04-14 19:26:57 -04:00
Brandon Thomas 3386247858 remove pnpm-lock.yaml to unblock CI - it's causing new failures 2025-04-14 19:10:44 -04:00
Brandon Thomas 7ea9c96d77 remove dead references from editor3d package 2025-04-14 19:08:49 -04:00
Brandon Thomas 059f4b4e38 add pnpm to solve build issues 2025-04-14 18:56:05 -04:00
Brandon Thomas 3c36e041ca remove "workspace:*" URL. Not working on all machines? 2025-04-14 18:40:23 -04:00
Tanish MankuandTanish Manku f027fcb29a Write Gallery component with test data (#508)
Co-authored-by: Tanish Manku <-h>
2025-04-14 10:49:35 -07:00
BFlat 6f2ed9e61e dismiss onboarding helper and local storage 2025-04-14 14:57:33 +07:00
Brandon Thomas 2b40b2f8d6 add index on jobs anonymous visitor tokens 2025-04-14 02:58:49 -04:00
Brandon Thomas 9996b41bc2 Add nx library component and import it 2025-04-14 02:24:29 -04:00
ArEnScandMichael Chung b9a5768679 Feature/polling (#505)
add text back and toasters

Co-authored-by: Michael Chung <michaelchung@michaels-mbp-2.home>
2025-04-13 23:54:20 -04:00
Brandon Thomas 614a5edc55 cleanup netlify.toml files 2025-04-13 21:17:09 -04:00
Brandon Thomas 01468020bd cleanup build script 2025-04-13 21:06:24 -04:00
Brandon Thomas 289ef6f352 debug branch build (#507) 2025-04-13 21:01:09 -04:00
Brandon Thomas 14aa429878 debug netlify 2025-04-13 13:39:53 -04:00
Brandon Thomas f902be970f debug netlify 2025-04-13 13:34:23 -04:00
Brandon Thomas 783761736e debug netlify 2025-04-13 13:29:06 -04:00
Brandon Thomas 794fcf4efb fix netlify build 2025-04-13 12:06:38 -04:00
bflatastic 183b480e6d thicker logo update 2025-04-13 00:18:40 +07:00
bflatastic ae215a7901 artcraft logo update 2025-04-13 00:12:36 +07:00
bflatastic 2117566fbd artcraft logo 2025-04-13 00:07:25 +07:00
bflatastic d492b93842 alt key to slow camera down for precise framing 2025-04-12 15:07:27 +07:00
bflatastic ebcc317f31 hide transform controls and edge glow in snapshot 2025-04-12 14:22:39 +07:00
BFlat 8a19e914dc camera switching, focal length etc. (#504)
all camera updates

- adding cameras
- switching cameras
- adjusting focal length for each
- save cameras to scene
2025-04-12 14:14:38 +07:00
bflatastic 5df413af49 memoized library modal to prevent constant rerender 2025-04-12 13:06:27 +07:00
bflatastic b1eb621998 keybind B bug fix not working after adding item 2025-04-12 12:35:16 +07:00
bflatastic 00357e0fdd snapshot update - high quality, aspect ratio, captures scene without grid 2025-04-12 03:24:32 +07:00
bflatastic c7eed7b52d unhide camera view for testing 2025-04-11 23:57:19 +07:00
Kasisnu 9c4072016d image-gen: make refresh token call in-cluster for now 2025-04-11 18:35:33 +05:30
Brandon Thomas 3d964c83c0 test builds 2025-04-11 08:56:18 -04:00
Brandon Thomas 41975bf18e test 2025-04-11 08:43:04 -04:00
Brandon Thomas e0d984d1ac debug 2025-04-11 08:38:06 -04:00
Brandon Thomas b26abbd39f debug 2025-04-11 08:33:23 -04:00
Brandon Thomas 114fd66204 test 2025-04-11 08:25:51 -04:00
Brandon Thomas 4cc370d40d update 2d build script 2025-04-11 08:17:49 -04:00
Brandon Thomas d5d7bb46be make dist dir 2025-04-11 05:39:40 -04:00
Brandon Thomas f059e74b26 copy headers 2025-04-11 05:33:40 -04:00
Brandon Thomas 765b03c600 don't copy tests 2025-04-11 05:28:44 -04:00
Brandon Thomas a37e6d5692 change build command 2025-04-11 05:23:55 -04:00
Brandon Thomas baef6f170b specify in UI for now 2025-04-11 05:07:48 -04:00
Brandon Thomas 10137ddcb8 update readme with instructions 2025-04-11 04:52:07 -04:00
Brandon Thomas c02694a8ec move editor2d to nx monorepo (#498) 2025-04-11 04:28:09 -04:00
189 changed files with 10701 additions and 5101 deletions
Generated
+32
View File
@@ -5035,6 +5035,12 @@ version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
[[package]]
name = "hex-literal"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bcaaec4551594c969335c98c903c1397853d4198408ea609190f420500f6be71"
[[package]]
name = "hf-hub"
version = "0.3.2"
@@ -6130,6 +6136,15 @@ dependencies = [
"signature",
]
[[package]]
name = "keccak"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ecc2af9a1119c51f12a14607e783cb977bde58bc069ff0c3da1095e635d70654"
dependencies = [
"cpufeatures",
]
[[package]]
name = "kernel32-sys"
version = "0.2.2"
@@ -7849,16 +7864,23 @@ name = "openai_sora_client"
version = "0.0.1"
dependencies = [
"anyhow",
"base64 0.22.1",
"chrono",
"crockford",
"errors",
"filesys",
"hex-literal",
"idempotency",
"log",
"rand 0.9.0",
"reqwest 0.12.15",
"serde",
"serde_derive",
"serde_json",
"sha3",
"subprocess_common",
"testing",
"thiserror 2.0.11",
"tokio",
"toml 0.7.8",
"workspace-build-acceleration",
@@ -10604,6 +10626,16 @@ dependencies = [
"digest 0.10.7",
]
[[package]]
name = "sha3"
version = "0.10.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75872d278a8f37ef87fa0ddbda7802605cb18344497949862c0d4dcb291eba60"
dependencies = [
"digest 0.10.7",
"keccak",
]
[[package]]
name = "sharded-slab"
version = "0.1.7"
+7 -4
View File
@@ -117,7 +117,7 @@ members = [
"crates/vendor/candle/tensor-tools",
"crates/vendor/limitation",
# NB(bt,2023-12-12): Newrelic keeps breaking with upgrades. Kill for now.
#"crates/vendor/newrelic-telemetry-sdk-rust",
#"crates/vendor/newrelic-telemetry-sdk-rust",
# cargo-hakari workspace hack (compilation speedup hack)
# see usage: https://crates.io/crates/cargo-hakari
@@ -131,12 +131,12 @@ publish = false
[workspace.dependencies]
# Hakari workspace hack (faster compile times)
#
# If an unrelated leaf package starts causing general build problems,
# If an unrelated leaf package starts causing general build problems,
# for packages including hakari, look up what depends on the bad package:
#
# `cargo tree --target all -p {BAD_PACKAGE_NAME} -i`
#
# Packages to remove from hakari to ensure we can build in Netlify,
# Packages to remove from hakari to ensure we can build in Netlify,
# Github Actions (without package installs), etc.:
#
# - cudarc
@@ -152,7 +152,9 @@ workspace-build-acceleration = { path = "crates/workspace-build-acceleration" }
# Centrally pinned versions so that we can uniformly upgrade/downgrade versions across the workspace
# If a crate needs to use an older/newer version, the dependency should be managed in that crate's own Cargo.toml
anyhow = "1.0.97"
base64 = "0.22.1"
chrono = { version = "0.4.40", features = ["serde"] }
hex-literal = "1.0.0"
log = "0.4.27"
once_cell = "1.21.3"
rand = "0.9.0"
@@ -162,9 +164,10 @@ serde = { version = "1.0.219", features = ["derive"] }
serde_derive = "1.0.219"
serde_json = "1.0.140"
serde_yaml = "0.9.1"
sha3 = "0.10.8"
thiserror = "2"
toml = "0.7.4"
url = "2.5.4"
# Internal libs
crockford = { path = "crates/lib/data/crockford" }
errors = { path = "crates/lib/errors" }
@@ -0,0 +1,6 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
DROP INDEX `index_maybe_creator_anonymous_visitor_token`
ON `generic_inference_jobs`;
@@ -0,0 +1,7 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE generic_inference_jobs
ADD INDEX index_maybe_creator_anonymous_visitor_token (maybe_creator_anonymous_visitor_token),
ALGORITHM=INPLACE, LOCK=NONE;
@@ -297,6 +297,7 @@ CREATE TABLE generic_inference_jobs (
KEY fk_maybe_input_source_token_and_type (maybe_input_source_token, maybe_input_source_token_type),
KEY fk_on_success_result_entity_token (on_success_result_entity_token),
KEY fk_maybe_creator_user_token (maybe_creator_user_token),
KEY index_maybe_creator_anonymous_visitor_token (maybe_creator_anonymous_visitor_token),
KEY index_creator_ip_address (creator_ip_address),
KEY index_priority_level (priority_level),
KEY index_is_debug_request (is_debug_request),
@@ -10,7 +10,7 @@ pub fn encode_rgb_image_base64_png(
image.write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png)?;
let bytes = BASE64_STANDARD.encode(bytes);
let bytes = (&BASE64_STANDARD).encode(bytes);
Ok(bytes)
}
@@ -20,6 +20,7 @@ workspace-build-acceleration = { workspace = true }
crockford = { path = "../../data/crockford" }
errors = { path = "../../errors" }
subprocess_common = { path = "../../subprocess_common" }
idempotency = { path = "../../idempotency" }
# External
anyhow = { workspace = true }
@@ -28,8 +29,14 @@ reqwest = { workspace = true, features = ["multipart"] }
serde = { workspace = true }
serde_derive = { workspace = true }
serde_json = { workspace = true }
sha3 = { workspace = true }
hex-literal = { workspace = true }
base64 = { workspace = true }
tokio = { version = "1.38.0", features = ["macros"] }
toml = { workspace = true }
thiserror = { workspace = true }
chrono = { workspace = true}
rand = { workspace = true }
[dev-dependencies]
testing = { path = "../../testing" }
@@ -1,13 +1,30 @@
use anyhow::anyhow;
use serde_derive::{Deserialize, Serialize};
use errors::AnyhowResult;
use crate::credentials::SoraCredentials;
use thiserror::Error;
const SORA_IMAGE_GEN_URL: &str = "https://sora.com/backend/video_gen";
/// This user agent is tied to the sentinel generation. If we need to change it, we may need to change sentinel generation too.
const USER_AGENT : &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36";
#[derive(Error, Debug)]
pub enum SoraError {
#[error("Sora token expired: {0}")]
TokenExpired(String),
#[error("Sora sentinel block: {0}")]
SentinelBlock(String),
#[error("Sora too many concurrent tasks: {0}")]
TooManyConcurrentTasks(String),
#[error("Sora API error: {0}")]
GenericError(String),
#[error("Network error: {0}")]
NetworkError(String),
}
#[derive(Serialize)]
#[serde(rename_all = "snake_case")]
pub (crate) enum VideoGenType {
@@ -85,22 +102,51 @@ pub (crate) struct RawSoraResponse {
// type "invalid_request_error"
// param null
// code null
#[derive(Serialize)]
#[derive(Deserialize, Debug, Clone)]
#[serde(rename_all = "snake_case")]
pub (crate) struct RawSoraErrorResponse {
error: String,
pub struct RawSoraErrorResponse {
error: RawSoraErrorInner,
}
#[derive(Deserialize, Debug, Clone)]
#[serde(rename_all = "snake_case")]
pub enum RawSoraError {
Error(RawSoraErrorInner),
Unknown(String),
}
#[derive(Deserialize, Debug, Clone)]
#[serde(rename_all = "snake_case")]
pub enum SoraErrorCode {
SentinelBlock,
TokenExpired,
TooManyConcurrentTasks,
Unknown(String),
}
#[derive(Deserialize, Debug, Clone, Error)]
#[serde(rename_all = "snake_case")]
pub struct RawSoraErrorInner {
message: String,
r#type: String,
param: Option<String>,
code: Option<String>,
code: Option<SoraErrorCode>,
}
impl std::fmt::Display for RawSoraErrorInner {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "Sora API error: {}", self.message)
}
}
/// Don't expose the internal request implementation as there are only a few "correct" ways to call the API.
pub (crate) async fn image_gen_http_request(sora_request: RawSoraImageGenRequest, credentials: &SoraCredentials) -> AnyhowResult<RawSoraResponse> {
pub (crate) async fn image_gen_http_request(sora_request: RawSoraImageGenRequest, credentials: &SoraCredentials) -> Result<RawSoraResponse, SoraError> {
let client = reqwest::Client::new();
let sentinel = match credentials.sentinel.as_deref() {
None => return Err(anyhow!("Calls to image gen require a sentinel in the credentials payload.")),
None => return Err(SoraError::GenericError("Calls to image gen require a sentinel in the credentials payload.".to_string())),
Some(sentinel) => sentinel,
};
@@ -111,21 +157,45 @@ pub (crate) async fn image_gen_http_request(sora_request: RawSoraImageGenRequest
.header("Content-Type", "application/json")
.header("OpenAI-Sentinel-Token", sentinel);
let http_request = http_request.json(&sora_request).build()?;
let http_request = http_request.json(&sora_request).build()
.map_err(|e| SoraError::NetworkError(e.to_string()))?;
let response = client.execute(http_request)
.await?;
.await
.map_err(|e| SoraError::NetworkError(e.to_string()))?;
let status = response.status();
let response_body = &response.text().await?;
let response_body = &response.text().await
.map_err(|e| SoraError::NetworkError(e.to_string()))?;
if status != reqwest::StatusCode::OK {
//TODO: Kasisnu: We should parse the error response and return a custom error. status 400 + code "sentinel_block" means we should retry with new sentinel token.
return Err(anyhow::anyhow!("the request failed; status = {:?}, message = {:?}", status, response_body));
let error_response: RawSoraErrorResponse = serde_json::from_str(response_body)
.map_err(|e| SoraError::GenericError(format!("Failed to parse error response: {}", e)))?;
// Check for specific error codes
if let Some(code) = &error_response.error.code {
match code {
SoraErrorCode::TokenExpired => {
return Err(SoraError::TokenExpired(error_response.error.message));
}
SoraErrorCode::SentinelBlock => {
return Err(SoraError::SentinelBlock(error_response.error.message));
}
SoraErrorCode::TooManyConcurrentTasks => {
return Err(SoraError::TooManyConcurrentTasks(error_response.error.message));
}
SoraErrorCode::Unknown(_) => {
return Err(SoraError::GenericError(error_response.error.message));
}
}
}
return Err(SoraError::GenericError(error_response.error.message));
}
let response = serde_json::from_str(response_body)?;
let response = serde_json::from_str(response_body)
.map_err(|e| SoraError::GenericError(format!("Failed to parse success response: {}", e)))?;
Ok(response)
}
@@ -83,16 +83,16 @@ pub struct Generation {
pub created_at: String,
pub deleted_at: Option<String>,
pub url: String,
pub seed: i64,
pub can_download: bool,
pub download_status: String,
pub is_favorite: bool,
pub is_liked: bool,
pub is_public: bool,
pub is_archived: bool,
pub is_featured: bool,
pub has_feedback: bool,
pub like_count: i32,
pub seed: Option<i64>,
pub can_download: Option<bool>,
pub download_status: Option<String>,
pub is_favorite: Option<bool>,
pub is_liked: Option<bool>,
pub is_public: Option<bool>,
pub is_archived: Option<bool>,
pub is_featured: Option<bool>,
pub has_feedback: Option<bool>,
pub like_count: Option<i32>,
pub cloudflare_metadata: Option<serde_json::Value>,
pub cf_thumbnail_url: Option<String>,
pub encodings: Encodings,
@@ -1,4 +1,5 @@
pub (crate) mod image_gen_http_request;
pub use image_gen_http_request::SoraError;
pub mod common;
pub mod sora_image_gen_remix;
pub mod sora_image_gen_simple;
@@ -1,7 +1,7 @@
use errors::AnyhowResult;
use crate::credentials::SoraCredentials;
use crate::image_gen::common::{ImageSize, NumImages, SoraImageGenResponse};
use crate::image_gen::image_gen_http_request::{image_gen_http_request, InpaintItem, InpaintItemType, OperationType, RawSoraImageGenRequest, VideoGenType};
use crate::image_gen::image_gen_http_request::{image_gen_http_request, InpaintItem, InpaintItemType, OperationType, RawSoraImageGenRequest, SoraError, VideoGenType};
pub struct SoraImageGenRemixRequest<'a> {
pub prompt: String,
@@ -13,7 +13,7 @@ pub struct SoraImageGenRemixRequest<'a> {
/// The "remix" commands let you supply additional images as context.
/// Sora "media tokens" of previously uploaded images must be supplied.
pub async fn sora_image_gen_remix(request: SoraImageGenRemixRequest<'_>) -> AnyhowResult<SoraImageGenResponse> {
pub async fn sora_image_gen_remix(request: SoraImageGenRemixRequest<'_>) -> Result<SoraImageGenResponse, SoraError> {
let args = RawSoraImageGenRequest {
r#type: VideoGenType::ImageGen,
operation: OperationType::Remix,
@@ -36,7 +36,6 @@ pub async fn sora_image_gen_remix(request: SoraImageGenRemixRequest<'_>) -> Anyh
}).collect(),
};
// TODO: Error handling.
let result = image_gen_http_request(args, &request.credentials).await?;
Ok(SoraImageGenResponse {
@@ -1,4 +1,6 @@
pub mod credentials;
pub mod image_gen;
pub mod job;
pub mod upload;
pub mod upload;
pub mod sentinel_refresh;
mod sora_error;
@@ -0,0 +1,671 @@
// cores = [8, 16, 24, 32]
// cached_scripts = ["https://cdn.oaistatic.com/_next/static/cXh69klOLzS0Gy2joLDRS/_ssgManifest.js?dpl=453ebaec0d44c2decab71692e1bfe39be35a24b3"]
// cached_dpl = ["prod-f501fe933b3edf57aea882da888e1a544df99840"]
// navigator_key = [
// "registerProtocolHandler−function registerProtocolHandler() { [native code] }",
// "storage−[object StorageManager]",
// "locks−[object LockManager]",
// "appCodeName−Mozilla",
// "permissions−[object Permissions]",
// "share−function share() { [native code] }",
// "webdriver−false",
// "managed−[object NavigatorManagedData]",
// "canShare−function canShare() { [native code] }",
// "vendor−Google Inc.",
// "vendor−Google Inc.",
// "mediaDevices−[object MediaDevices]",
// "vibrate−function vibrate() { [native code] }",
// "storageBuckets−[object StorageBucketManager]",
// "mediaCapabilities−[object MediaCapabilities]",
// "getGamepads−function getGamepads() { [native code] }",
// "bluetooth−[object Bluetooth]",
// "share−function share() { [native code] }",
// "cookieEnabled−true",
// "virtualKeyboard−[object VirtualKeyboard]",
// "product−Gecko",
// "mediaDevices−[object MediaDevices]",
// "canShare−function canShare() { [native code] }",
// "getGamepads−function getGamepads() { [native code] }",
// "product−Gecko",
// "xr−[object XRSystem]",
// "clipboard−[object Clipboard]",
// "storageBuckets−[object StorageBucketManager]",
// "unregisterProtocolHandler−function unregisterProtocolHandler() { [native code] }",
// "productSub−20030107",
// "login−[object NavigatorLogin]",
// "vendorSub−",
// "login−[object NavigatorLogin]",
// "getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
// "mediaDevices−[object MediaDevices]",
// "locks−[object LockManager]",
// "webkitGetUserMedia−function webkitGetUserMedia() { [native code] }",
// "vendor−Google Inc.",
// "xr−[object XRSystem]",
// "mediaDevices−[object MediaDevices]",
// "virtualKeyboard−[object VirtualKeyboard]",
// "virtualKeyboard−[object VirtualKeyboard]",
// "appName−Netscape",
// "storageBuckets−[object StorageBucketManager]",
// "presentation−[object Presentation]",
// "onLine−true",
// "mimeTypes−[object MimeTypeArray]",
// "credentials−[object CredentialsContainer]",
// "presentation−[object Presentation]",
// "getGamepads−function getGamepads() { [native code] }",
// "vendorSub−",
// "virtualKeyboard−[object VirtualKeyboard]",
// "serviceWorker−[object ServiceWorkerContainer]",
// "xr−[object XRSystem]",
// "product−Gecko",
// "keyboard−[object Keyboard]",
// "gpu−[object GPU]",
// "getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
// "webkitPersistentStorage−[object DeprecatedStorageQuota]",
// "doNotTrack",
// "clearAppBadge−function clearAppBadge() { [native code] }",
// "presentation−[object Presentation]",
// "serial−[object Serial]",
// "locks−[object LockManager]",
// "requestMIDIAccess−function requestMIDIAccess() { [native code] }",
// "locks−[object LockManager]",
// "requestMediaKeySystemAccess−function requestMediaKeySystemAccess() { [native code] }",
// "vendor−Google Inc.",
// "pdfViewerEnabled−true",
// "language−zh-CN",
// "setAppBadge−function setAppBadge() { [native code] }",
// "geolocation−[object Geolocation]",
// "userAgentData−[object NavigatorUAData]",
// "mediaCapabilities−[object MediaCapabilities]",
// "requestMIDIAccess−function requestMIDIAccess() { [native code] }",
// "getUserMedia−function getUserMedia() { [native code] }",
// "mediaDevices−[object MediaDevices]",
// "webkitPersistentStorage−[object DeprecatedStorageQuota]",
// "sendBeacon−function sendBeacon() { [native code] }",
// "hardwareConcurrency−32",
// "credentials−[object CredentialsContainer]",
// "storage−[object StorageManager]",
// "cookieEnabled−true",
// "pdfViewerEnabled−true",
// "windowControlsOverlay−[object WindowControlsOverlay]",
// "scheduling−[object Scheduling]",
// "pdfViewerEnabled−true",
// "hardwareConcurrency−32",
// "xr−[object XRSystem]",
// "webdriver−false",
// "getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
// "getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
// "bluetooth−[object Bluetooth]"
// ]
// document_key = ['_reactListeningo743lnnpvdg', 'location']
// window_key = [
// "0",
// "window",
// "self",
// "document",
// "name",
// "location",
// "customElements",
// "history",
// "navigation",
// "locationbar",
// "menubar",
// "personalbar",
// "scrollbars",
// "statusbar",
// "toolbar",
// "status",
// "closed",
// "frames",
// "length",
// "top",
// "opener",
// "parent",
// "frameElement",
// "navigator",
// "origin",
// "external",
// "screen",
// "innerWidth",
// "innerHeight",
// "scrollX",
// "pageXOffset",
// "scrollY",
// "pageYOffset",
// "visualViewport",
// "screenX",
// "screenY",
// "outerWidth",
// "outerHeight",
// "devicePixelRatio",
// "clientInformation",
// "screenLeft",
// "screenTop",
// "styleMedia",
// "onsearch",
// "isSecureContext",
// "trustedTypes",
// "performance",
// "onappinstalled",
// "onbeforeinstallprompt",
// "crypto",
// "indexedDB",
// "sessionStorage",
// "localStorage",
// "onbeforexrselect",
// "onabort",
// "onbeforeinput",
// "onbeforematch",
// "onbeforetoggle",
// "onblur",
// "oncancel",
// "oncanplay",
// "oncanplaythrough",
// "onchange",
// "onclick",
// "onclose",
// "oncontentvisibilityautostatechange",
// "oncontextlost",
// "oncontextmenu",
// "oncontextrestored",
// "oncuechange",
// "ondblclick",
// "ondrag",
// "ondragend",
// "ondragenter",
// "ondragleave",
// "ondragover",
// "ondragstart",
// "ondrop",
// "ondurationchange",
// "onemptied",
// "onended",
// "onerror",
// "onfocus",
// "onformdata",
// "oninput",
// "oninvalid",
// "onkeydown",
// "onkeypress",
// "onkeyup",
// "onload",
// "onloadeddata",
// "onloadedmetadata",
// "onloadstart",
// "onmousedown",
// "onmouseenter",
// "onmouseleave",
// "onmousemove",
// "onmouseout",
// "onmouseover",
// "onmouseup",
// "onmousewheel",
// "onpause",
// "onplay",
// "onplaying",
// "onprogress",
// "onratechange",
// "onreset",
// "onresize",
// "onscroll",
// "onsecuritypolicyviolation",
// "onseeked",
// "onseeking",
// "onselect",
// "onslotchange",
// "onstalled",
// "onsubmit",
// "onsuspend",
// "ontimeupdate",
// "ontoggle",
// "onvolumechange",
// "onwaiting",
// "onwebkitanimationend",
// "onwebkitanimationiteration",
// "onwebkitanimationstart",
// "onwebkittransitionend",
// "onwheel",
// "onauxclick",
// "ongotpointercapture",
// "onlostpointercapture",
// "onpointerdown",
// "onpointermove",
// "onpointerrawupdate",
// "onpointerup",
// "onpointercancel",
// "onpointerover",
// "onpointerout",
// "onpointerenter",
// "onpointerleave",
// "onselectstart",
// "onselectionchange",
// "onanimationend",
// "onanimationiteration",
// "onanimationstart",
// "ontransitionrun",
// "ontransitionstart",
// "ontransitionend",
// "ontransitioncancel",
// "onafterprint",
// "onbeforeprint",
// "onbeforeunload",
// "onhashchange",
// "onlanguagechange",
// "onmessage",
// "onmessageerror",
// "onoffline",
// "ononline",
// "onpagehide",
// "onpageshow",
// "onpopstate",
// "onrejectionhandled",
// "onstorage",
// "onunhandledrejection",
// "onunload",
// "crossOriginIsolated",
// "scheduler",
// "alert",
// "atob",
// "blur",
// "btoa",
// "cancelAnimationFrame",
// "cancelIdleCallback",
// "captureEvents",
// "clearInterval",
// "clearTimeout",
// "close",
// "confirm",
// "createImageBitmap",
// "fetch",
// "find",
// "focus",
// "getComputedStyle",
// "getSelection",
// "matchMedia",
// "moveBy",
// "moveTo",
// "open",
// "postMessage",
// "print",
// "prompt",
// "queueMicrotask",
// "releaseEvents",
// "reportError",
// "requestAnimationFrame",
// "requestIdleCallback",
// "resizeBy",
// "resizeTo",
// "scroll",
// "scrollBy",
// "scrollTo",
// "setInterval",
// "setTimeout",
// "stop",
// "structuredClone",
// "webkitCancelAnimationFrame",
// "webkitRequestAnimationFrame",
// "chrome",
// "caches",
// "cookieStore",
// "ondevicemotion",
// "ondeviceorientation",
// "ondeviceorientationabsolute",
// "launchQueue",
// "documentPictureInPicture",
// "getScreenDetails",
// "queryLocalFonts",
// "showDirectoryPicker",
// "showOpenFilePicker",
// "showSaveFilePicker",
// "originAgentCluster",
// "onpageswap",
// "onpagereveal",
// "credentialless",
// "speechSynthesis",
// "onscrollend",
// "webkitRequestFileSystem",
// "webkitResolveLocalFileSystemURL",
// "sendMsgToSolverCS",
// "webpackChunk_N_E",
// "__next_set_public_path__",
// "next",
// "__NEXT_DATA__",
// "__SSG_MANIFEST_CB",
// "__NEXT_P",
// "_N_E",
// "regeneratorRuntime",
// "__REACT_INTL_CONTEXT__",
// "DD_RUM",
// "_",
// "filterCSS",
// "filterXSS",
// "__SEGMENT_INSPECTOR__",
// "__NEXT_PRELOADREADY",
// "Intercom",
// "__MIDDLEWARE_MATCHERS",
// "__STATSIG_SDK__",
// "__STATSIG_JS_SDK__",
// "__STATSIG_RERENDER_OVERRIDE__",
// "_oaiHandleSessionExpired",
// "__BUILD_MANIFEST",
// "__SSG_MANIFEST",
// "__intercomAssignLocation",
// "__intercomReloadLocation"
// ]
use rand::Rng;
const CORES: &[usize] = &[8, 16, 24, 32];
const CACHED_SCRIPTS: &[&str] = &["https://cdn.oaistatic.com/_next/static/cXh69klOLzS0Gy2joLDRS/_ssgManifest.js?dpl=453ebaec0d44c2decab71692e1bfe39be35a24b3"];
const CACHED_DPL: &[&str] = &["prod-f501fe933b3edf57aea882da888e1a544df99840"];
const NAVIGATOR_KEY: &[&str] = &[
"registerProtocolHandler−function registerProtocolHandler() { [native code] }",
"storage−[object StorageManager]",
"locks−[object LockManager]",
"appCodeName−Mozilla",
"permissions−[object Permissions]",
"share−function share() { [native code] }",
"webdriver−false",
"managed−[object NavigatorManagedData]",
"canShare−function canShare() { [native code] }",
"vendor−Google Inc.",
"vendor−Google Inc.",
"mediaDevices−[object MediaDevices]",
"vibrate−function vibrate() { [native code] }",
"storageBuckets−[object StorageBucketManager]",
"mediaCapabilities−[object MediaCapabilities]",
"getGamepads−function getGamepads() { [native code] }",
"bluetooth−[object Bluetooth]",
"share−function share() { [native code] }",
"cookieEnabled−true",
"pdfViewerEnabled−true",
"windowControlsOverlay−[object WindowControlsOverlay]",
"scheduling−[object Scheduling]",
"pdfViewerEnabled−true",
"hardwareConcurrency−32",
"xr−[object XRSystem]",
"webdriver−false",
"getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
"getInstalledRelatedApps−function getInstalledRelatedApps() { [native code] }",
"bluetooth−[object Bluetooth]"
];
const DOCUMENT_KEY: &[&str] = &["_reactListeningo743lnnpvdg", "location"];
const WINDOW_KEY: &[&str] = &[
"0",
"window",
"self",
"document",
"name",
"location",
"customElements",
"history",
"navigation",
"locationbar",
"menubar",
"personalbar",
"scrollbars",
"statusbar",
"toolbar",
"status",
"closed",
"frames",
"length",
"top",
"opener",
"parent",
"frameElement",
"navigator",
"origin",
"external",
"screen",
"innerWidth",
"innerHeight",
"scrollX",
"pageXOffset",
"scrollY",
"pageYOffset",
"visualViewport",
"screenX",
"screenY",
"outerWidth",
"outerHeight",
"devicePixelRatio",
"clientInformation",
"screenLeft",
"screenTop",
"styleMedia",
"onsearch",
"isSecureContext",
"trustedTypes",
"performance",
"onappinstalled",
"onbeforeinstallprompt",
"crypto",
"indexedDB",
"sessionStorage",
"localStorage",
"onbeforexrselect",
"onabort",
"onbeforeinput",
"onbeforematch",
"onbeforetoggle",
"onblur",
"oncancel",
"oncanplay",
"oncanplaythrough",
"onchange",
"onclick",
"onclose",
"oncontentvisibilityautostatechange",
"oncontextlost",
"oncontextmenu",
"oncontextrestored",
"oncuechange",
"ondblclick",
"ondrag",
"ondragend",
"ondragenter",
"ondragleave",
"ondragover",
"ondragstart",
"ondrop",
"ondurationchange",
"onemptied",
"onended",
"onerror",
"onfocus",
"onformdata",
"oninput",
"oninvalid",
"onkeydown",
"onkeypress",
"onkeyup",
"onload",
"onloadeddata",
"onloadedmetadata",
"onloadstart",
"onmousedown",
"onmouseenter",
"onmouseleave",
"onmousemove",
"onmouseout",
"onmouseover",
"onmouseup",
"onmousewheel",
"onpause",
"onplay",
"onplaying",
"onprogress",
"onratechange",
"onreset",
"onresize",
"onscroll",
"onsecuritypolicyviolation",
"onseeked",
"onseeking",
"onselect",
"onslotchange",
"onstalled",
"onsubmit",
"onsuspend",
"ontimeupdate",
"ontoggle",
"onvolumechange",
"onwaiting",
"onwebkitanimationend",
"onwebkitanimationiteration",
"onwebkitanimationstart",
"onwebkittransitionend",
"onwheel",
"onauxclick",
"ongotpointercapture",
"onlostpointercapture",
"onpointerdown",
"onpointermove",
"onpointerrawupdate",
"onpointerup",
"onpointercancel",
"onpointerover",
"onpointerout",
"onpointerenter",
"onpointerleave",
"onselectstart",
"onselectionchange",
"onanimationend",
"onanimationiteration",
"onanimationstart",
"ontransitionrun",
"ontransitionstart",
"ontransitionend",
"ontransitioncancel",
"onafterprint",
"onbeforeprint",
"onbeforeunload",
"onhashchange",
"onlanguagechange",
"onmessage",
"onmessageerror",
"onoffline",
"ononline",
"onpagehide",
"onpageshow",
"onpopstate",
"onrejectionhandled",
"onstorage",
"onunhandledrejection",
"onunload",
"crossOriginIsolated",
"scheduler",
"alert",
"atob",
"blur",
"btoa",
"cancelAnimationFrame",
"cancelIdleCallback",
"captureEvents",
"clearInterval",
"clearTimeout",
"close",
"confirm",
"createImageBitmap",
"fetch",
"find",
"focus",
"getComputedStyle",
"getSelection",
"matchMedia",
"moveBy",
"moveTo",
"open",
"postMessage",
"print",
"prompt",
"queueMicrotask",
"releaseEvents",
"reportError",
"requestAnimationFrame",
"requestIdleCallback",
"resizeBy",
"resizeTo",
"scroll",
"scrollBy",
"scrollTo",
"setInterval",
"setTimeout",
"stop",
"structuredClone",
"webkitCancelAnimationFrame",
"webkitRequestAnimationFrame",
"chrome",
"caches",
"cookieStore",
"ondevicemotion",
"ondeviceorientation",
"ondeviceorientationabsolute",
"launchQueue",
"documentPictureInPicture",
"getScreenDetails",
"queryLocalFonts",
"showDirectoryPicker",
"showOpenFilePicker",
"showSaveFilePicker",
"originAgentCluster",
"onpageswap",
"onpagereveal",
"credentialless",
"speechSynthesis",
"onscrollend",
"webkitRequestFileSystem",
"webkitResolveLocalFileSystemURL",
"sendMsgToSolverCS",
"webpackChunk_N_E",
"__next_set_public_path__",
"next",
"__NEXT_DATA__",
"__SSG_MANIFEST_CB",
"__NEXT_P",
"_N_E",
"regeneratorRuntime",
"__REACT_INTL_CONTEXT__",
"DD_RUM",
"_",
"filterCSS",
"filterXSS",
"__SEGMENT_INSPECTOR__",
"__NEXT_PRELOADREADY",
"Intercom",
"__MIDDLEWARE_MATCHERS",
"__STATSIG_SDK__",
"__STATSIG_JS_SDK__",
"__STATSIG_RERENDER_OVERRIDE__",
"_oaiHandleSessionExpired",
"__BUILD_MANIFEST",
"__SSG_MANIFEST",
"__intercomAssignLocation",
"__intercomReloadLocation"
];
pub fn get_random_core_count() -> usize {
return CORES[rand::rng().random_range(0..CORES.len())];
}
pub fn get_random_navigator_key() -> String {
return NAVIGATOR_KEY[rand::rng().random_range(0..NAVIGATOR_KEY.len())].to_string();
}
pub fn get_random_document_key() -> String {
return DOCUMENT_KEY[rand::rng().random_range(0..DOCUMENT_KEY.len())].to_string();
}
pub fn get_random_window_key() -> String {
return WINDOW_KEY[rand::rng().random_range(0..WINDOW_KEY.len())].to_string();
}
@@ -0,0 +1,3 @@
pub mod request;
pub mod token;
pub(crate) mod config;
@@ -0,0 +1,225 @@
use crate::sentinel_refresh::generate::config::{get_random_core_count, get_random_document_key, get_random_navigator_key, get_random_window_key};
use chrono::FixedOffset;
use chrono::Utc;
use idempotency::uuid::generate_random_uuid;
use rand::Rng;
use base64::prelude::BASE64_STANDARD;
use base64::Engine;
use sha3::{Digest, Sha3_512};
// [
// 4000, // first
// "Fri Apr 11 2025 13:28:14 GMT+0530 (India Standard Time)", // second
// None, // third
// 28, // fourth
// "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:137.0) Gecko/20100101 Firefox/137.0", // fifth
// "https://sora-cdn.oaistatic.com/_next/static/chunks/f90f2e60-61da6d33257661c4.js", // sixth
// None, // seventh
// "en-US", // eighth
// "en-US,en", // ninth
// 29, // tenth
// "productSub−20100101", // eleventh
// "location", // twelfth
// "ondrop", // thirteenth
// 10520, // fourteenth - generated from time.perf_counter() * 1000
// "48094014-a024-4d2d-8ec4-ba9c7cfef9c4", // fifteenth
// "", // sixteenth
// 16, // seventeenth - Number of cores
#[derive(Debug, Clone)]
pub struct GenerateSentinelRefreshRequest {
first: usize,
second: String, // Date
third: Option<String>,
fourth: Option<usize>,
fifth: String, // User agent
sixth: String, // URL
seventh: Option<String>, // Language
eighth: String, // Accept-Language
ninth: String, // Product Sub
tenth: Option<usize>,
eleventh: String, // Product Sub - Navigator key
twelfth: String, // Location - Document key
thirteenth: String, // On Drop - Window key
fourteenth: usize, // generated from time.perf_counter() * 1000
fifteenth: String, // UUID
sixteenth: String, // empty string
seventeenth: usize, // Number of cores
}
fn generate_date() -> String {
let now_utc = Utc::now();
let offset = FixedOffset::west_opt(5 * 60 * 60).unwrap();
let now_with_offset = Utc::now().with_timezone(&offset);
let formatted_date = now_with_offset.format("%a %b %d %Y %H:%M:%S GMT-0500 (Eastern Standard Time)");
return formatted_date.to_string();
}
impl GenerateSentinelRefreshRequest {
pub fn new() -> Self {
let first_candidates = vec![1920 + 1080, 2560 + 1440, 1920 + 1200, 2560 + 1600];
let first = first_candidates[rand::rng().random_range(0..first_candidates.len())];
let second = generate_date();
let third = None;
let fourth = None;
let fifth = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:137.0) Gecko/20100101 Firefox/137.0".to_string();
let sixth = "https://sora-cdn.oaistatic.com/_next/static/chunks/f90f2e60-61da6d33257661c4.js".to_string();
let seventh = None;
let eighth = "en-US".to_string();
let ninth = "en-US,en".to_string();
let tenth = None;
let eleventh = get_random_navigator_key();
let twelfth = get_random_document_key();
let thirteenth = get_random_window_key();
let fourteenth = 10520;
let fifteenth = generate_random_uuid();
let sixteenth = "".to_string();
let seventeenth = get_random_core_count();
Self { first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth, eleventh: eleventh.to_string(), twelfth: twelfth.to_string(), thirteenth: thirteenth.to_string(), fourteenth, fifteenth, sixteenth, seventeenth }
}
pub fn with_fourth(mut self, fourth: usize) -> Self {
self.fourth = Some(fourth);
self
}
pub fn with_tenth(mut self, tenth: usize) -> Self {
self.tenth = Some(tenth);
self
}
//TODO(kasisnu): This is a temporary function to build the request.
// It is extremely hacky and this whole struct should be rewritten to be more ergonomic.
pub fn build(self) -> String {
let mut request = String::new();
request.push_str("[");
request.push_str(&self.first.to_string());
request.push_str(",");
request.push_str(&string_with_quotes(self.second));
request.push_str(",");
request.push_str(&self.third.unwrap_or("null".to_string()));
request.push_str(",");
request.push_str(&self.fourth.unwrap_or(0).to_string());
request.push_str(",");
request.push_str(&string_with_quotes(self.fifth));
request.push_str(",");
request.push_str(&string_with_quotes(self.sixth));
request.push_str(",");
request.push_str(&self.seventh.unwrap_or("null".to_string()));
request.push_str(",");
request.push_str(&string_with_quotes(self.eighth));
request.push_str(",");
request.push_str(&string_with_quotes(self.ninth));
request.push_str(",");
request.push_str(&self.tenth.unwrap_or(0).to_string());
request.push_str(",");
request.push_str(&string_with_quotes(self.eleventh));
request.push_str(",");
request.push_str(&string_with_quotes(self.twelfth));
request.push_str(",");
request.push_str(&string_with_quotes(self.thirteenth));
request.push_str(",");
request.push_str(&self.fourteenth.to_string());
request.push_str(",");
request.push_str(&string_with_quotes(self.fifteenth));
request.push_str(",");
request.push_str(&string_with_quotes(self.sixteenth));
request.push_str(",");
request.push_str(&self.seventeenth.to_string());
request.push_str("]");
request
}
pub fn hash_request(seed: f64, request: String) -> Vec<u8> {
let mut hasher = Sha3_512::new();
hasher.update(seed.to_string());
let base64_request = BASE64_STANDARD.encode(request.as_bytes());
hasher.update(base64_request);
hasher.finalize().to_vec()
}
pub fn with_fourth_and_tenth(mut self) -> (Self, String) {
let mut fourth = 0;
let mut tenth = 0;
let max_iterations = 500000;
let diff_len = 6;
let target_bytes = vec![0x0f, 0xff, 0xff];
let random_seed: f64 = rand::rng().random();
let mut base64_request = BASE64_STANDARD.encode(self.clone().build().as_bytes());
for i in 0..max_iterations {
fourth = i;
tenth = i >> 1;
let request = self.clone().with_fourth(fourth).with_tenth(tenth).build();
let hash_bytes = Self::hash_request(random_seed, request.clone());
base64_request = BASE64_STANDARD.encode(request.as_bytes());
let mut is_less_or_equal = true;
for j in 0..std::cmp::min(diff_len, target_bytes.len()) {
if j < hash_bytes.len() {
if hash_bytes[j] > target_bytes[j] {
is_less_or_equal = false;
break;
} else if hash_bytes[j] < target_bytes[j] {
break;
}
}
}
if is_less_or_equal {
break;
}
}
(self.with_fourth(fourth).with_tenth(tenth), format!("gAAAAAC{}", base64_request))
}
}
fn string_with_quotes(value: String) -> String {
format!("\"{}\"", value)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_generate_sentinel_refresh_request() {
let request = GenerateSentinelRefreshRequest::new();
println!("{}", request.build());
}
#[test]
fn test_with_fourth_and_tenth() {
let request = GenerateSentinelRefreshRequest::new().with_fourth(100).with_tenth(200);
println!("{}", request.build());
}
#[test]
fn test_hash_request() {
let request = GenerateSentinelRefreshRequest::new().with_fourth(100).with_tenth(200);
let hash = GenerateSentinelRefreshRequest::hash_request(0.1, request.build());
println!("{}", hash.iter().map(|b| format!("{:02x}", b)).collect::<String>());
}
#[test]
fn test_with_fourth_and_tenth_2() {
let (request, base64_request) = GenerateSentinelRefreshRequest::new().with_fourth_and_tenth();
println!("{}", base64_request);
}
}
@@ -0,0 +1,69 @@
use errors::AnyhowResult;
use serde_derive::{Deserialize, Serialize};
use thiserror::Error;
use idempotency::uuid::generate_random_uuid;
use super::request::GenerateSentinelRefreshRequest;
const SORA_IMAGE_GEN_URL: &str = "https://chatgpt.com/backend-api/sentinel/req";
/// This user agent is tied to the sentinel generation. If we need to change it, we may need to change sentinel generation too.
const USER_AGENT : &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36";
const SENTINEL_FLOW: &str = "sora_create_task";
#[derive(Debug, Serialize, Deserialize)]
pub struct SentinelRequest {
#[serde(rename = "p")]
problem: String,
#[serde(rename = "id")]
id: String,
#[serde(rename = "flow")]
flow: String,
}
impl SentinelRequest {
pub fn new(problem: String) -> Self {
let id = generate_random_uuid();
Self { problem, id, flow: SENTINEL_FLOW.to_string() }
}
}
#[derive(Debug, Serialize, Deserialize)]
pub struct SentinelResponse {
persona: String,
token: String,
}
pub async fn generate_token() -> AnyhowResult<String> {
let (_request, base64_request) = GenerateSentinelRefreshRequest::new().with_fourth_and_tenth();
let request = SentinelRequest::new(base64_request);
let client = reqwest::Client::new();
let response = client.post(SORA_IMAGE_GEN_URL)
.header("User-Agent", USER_AGENT)
.header("Content-Type", "application/json")
.json(&request)
.send()
.await?;
let response_json: SentinelResponse = response.json().await?;
Ok(response_json.token)
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_generate_token() {
let token = generate_token().await.unwrap();
println!("{}", token);
}
}
@@ -0,0 +1,4 @@
pub use refresh::refresh_sentinel;
pub mod refresh;
pub mod generate;
@@ -0,0 +1,25 @@
// Send a POST request to env variable SORA_SENTINEL_REFRESH_URL
use reqwest::Client;
use std::env;
use errors::AnyhowResult;
const SORA_SENTINEL_REFRESH_URL: &str = "http://oai-token-server/refresh_token";
// Refresh the Sora sentinel
// The response body does _not_ contain the new sentinel
// We should read it again from redis or generate a new one here
// For now we refresh in the server handler
pub async fn refresh_sentinel() -> AnyhowResult<String> {
let client = Client::new();
let url = env::var("SORA_SENTINEL_REFRESH_URL").unwrap_or(SORA_SENTINEL_REFRESH_URL.to_string());
let response = client.post(url).send().await?;
let body = response.text().await?;
Ok(body)
}
@@ -0,0 +1,48 @@
use std::error::Error;
use std::fmt::{Display, Formatter};
#[derive(Debug)]
pub enum SoraError {
/// Unauthorized, cookie and/or bearer token expired. We'll need to ask for a refreshed login.
///
/// Example message, e.g. from the upload endpoint:
/// {
/// "error": {
/// "message": "Your authentication token has expired. Please try signing in again.",
/// "type": "invalid_request_error",
/// "param": null,
/// "code": "token_expired"
/// }
/// }
UnauthorizedCookieOrBearerExpired,
/// Another error occurred.
OtherBadStatus(anyhow::Error),
/// Reqwest Error
ReqwestError(reqwest::Error),
}
impl Error for SoraError {}
impl Display for SoraError {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
match self {
SoraError::UnauthorizedCookieOrBearerExpired => {
write!(f, "Unauthorized: cookie and/or bearer token expired")
}
SoraError::OtherBadStatus(err) => {
write!(f, "Other error: {}", err)
}
SoraError::ReqwestError(err) => {
write!(f, "Reqwest error: {}", err)
}
}
}
}
impl From<reqwest::Error> for SoraError {
fn from(err: reqwest::Error) -> SoraError {
SoraError::ReqwestError(err)
}
}
@@ -3,6 +3,7 @@ use errors::AnyhowResult;
use reqwest::multipart::{Form, Part};
use reqwest::Client;
use serde::Deserialize;
use crate::sora_error::SoraError;
const SORA_UPLOAD_MEDIA_URL: &str = "https://sora.com/backend/uploads";
@@ -25,6 +26,17 @@ Response:
"size_bytes": 118501,
"thumbnail_url": "https://videos.openai.com/vg-assets/assets%2Fclient_upload%2Fmedia%2Fuser-9wf6JFdgRSJLjvSJ53LNAbV8%2Fmedia_01jqt9vt20erx9zvryf3v1pecx.jpg?st=2025-04-02T02%3A39%3A01Z&se=2025-04-08T03%3A39%3A01Z&sks=b&skt=2025-04-02T02%3A39%3A01Z&ske=2025-04-08T03%3A39%3A01Z&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skoid=3d249c53-07fa-4ba4-9b65-0bf8eb4ea46a&skv=2019-02-02&sv=2018-11-09&sr=b&sp=r&spr=https%2Chttp&sig=9aU9xKI0BkhRtrTb0RD05%2BT6prBnmXk1I7nV2nTsFbY%3D&az=oaivgprodscus"
}
Error response:
Failed to upload scene media to Sora: Upload failed with status 401 Unauthorized: {
"error": {
"message": "Your authentication token has expired. Please try signing in again.",
"type": "invalid_request_error",
"param": null,
"code": "token_expired"
}
}
*/
#[derive(Debug, Deserialize)]
@@ -50,7 +62,13 @@ pub (crate) struct SoraMediaUploadRequest<'a> {
pub credentials: &'a SoraCredentials,
}
pub (crate) async fn upload_media_http_request(file_bytes: Vec<u8>, filename: String, mime_type: &str, credentials: &SoraCredentials) -> AnyhowResult<SoraMediaUploadResponse> {
pub (crate) async fn upload_media_http_request(
file_bytes: Vec<u8>,
filename: String,
mime_type: &str,
credentials: &SoraCredentials,
) -> Result<SoraMediaUploadResponse, SoraError> {
// Create multipart form
let part = Part::bytes(file_bytes) // NB: Reqwest needs to own the bytes.
.file_name(filename) // NB: Reqwest needs to own the bytes
@@ -70,12 +88,29 @@ pub (crate) async fn upload_media_http_request(file_bytes: Vec<u8>, filename: St
// Check response status
if !response.status().is_success() {
let status = response.status();
let text = response.text().await?;
return Err(anyhow::anyhow!("Upload failed with status {}: {}", status, text));
let error = classify_error(response).await;
return Err(error);
}
// Parse response
let upload_response = response.json::<SoraMediaUploadResponse>().await?;
Ok(upload_response)
}
async fn classify_error(response: reqwest::Response) -> SoraError {
let status = response.status();
let message = match response.text().await {
Ok(text) => text,
Err(err) => return SoraError::ReqwestError(err),
};
let cookie_expired =
message.contains("Your authentication token has expired. Please try signing in again.")
|| message.contains("token_expired");
if cookie_expired {
SoraError::UnauthorizedCookieOrBearerExpired
} else {
SoraError::OtherBadStatus(anyhow::anyhow!("Upload failed with status {}: {}", status, message))
}
}
@@ -4,7 +4,6 @@
/// NB: `sqlx::query` is spammy and dumps all queries as "info"-level log lines.
/// NB: `hyper::proto::h1::io` is incredibly spammy and logs every chunk of bytes in very large files being downloaded.
pub const DEFAULT_RUST_LOG: &str = concat!(
"debug,",
"actix_web=info,",
"sqlx::query=warn,",
"hyper::proto::h1::io=warn,",
@@ -21,9 +21,14 @@ use crate::helpers::boolean_converters::i8_to_bool;
use crate::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, PolymorphicInferenceArgs};
use crate::queries::generic_inference::web::job_status::{GenericInferenceJobStatus, RequestDetails, ResultDetails, UserDetails};
pub enum SessionUser<'a> {
User(&'a UserToken),
Anonymous(&'a AnonymousVisitorTrackingToken),
}
pub struct ListSessionJobsForUserArgs<'a> {
// TODO(bt,2024-04-23): Support AnonymousVisitorToken
pub user_token: &'a UserToken,
pub user: SessionUser<'a>,
pub maybe_include_job_statuses: Option<&'a HashSet<JobStatusPlus>>,
pub maybe_exclude_job_statuses: Option<&'a HashSet<JobStatusPlus>>,
}
@@ -63,14 +68,29 @@ pub async fn list_session_jobs_from_connection(
// Furthermore, even though the join query's `created_at` lacks an index, sort
// by id will accomplish this quickly.
{
query_builder.push(r#"
INNER JOIN (
SELECT id
FROM generic_inference_jobs
WHERE maybe_creator_user_token =
"#);
query_builder.push_bind(args.user_token.to_string());
match args.user {
SessionUser::User(user_token) => {
query_builder.push(r#"
INNER JOIN (
SELECT id
FROM generic_inference_jobs
WHERE maybe_creator_user_token =
"#);
query_builder.push_bind(user_token.to_string());
}
SessionUser::Anonymous(avt_token) => {
query_builder.push(r#"
INNER JOIN (
SELECT id
FROM generic_inference_jobs
WHERE maybe_creator_anonymous_visitor_token =
"#);
query_builder.push_bind(avt_token.to_string());
query_builder.push(r#"
AND maybe_creator_user_token IS NULL
"#);
}
}
query_builder.push(r#"
AND created_at > DATE_SUB(NOW(), INTERVAL 36 HOUR)
@@ -15,8 +15,13 @@ use log::{debug, error, info};
use log::warn;
use mysql_queries::queries::media_files::get::get_media_file::get_media_file;
use openai_sora_client::credentials::SoraCredentials;
use openai_sora_client::image_gen::SoraError;
use openai_sora_client::sentinel_refresh::generate::token::generate_token;
use openai_sora_client::sentinel_refresh::refresh_sentinel;
use serde::Deserialize;
use serde::Serialize;
use shared_service_components::sora_redis_credentials::set_sora_credential_field_in_redis::set_sora_credential_field_in_redis;
use shared_service_components::sora_redis_credentials::keys::RedisSoraCredentialSubkey;
use sqlx::MySqlPool;
use tempdir::TempDir;
use utoipa::ToSchema;
@@ -28,16 +33,9 @@ use enums::by_table::generic_inference_jobs::inference_model_type::InferenceMode
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{
GenericInferenceArgs,
InferenceCategoryAbbreviated,
PolymorphicInferenceArgs,
};
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::payloads::generic_inference_args::inner_payloads::sora_image_gen_args::SoraImageGenArgs;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{
insert_generic_inference_job,
InsertGenericInferenceArgs,
};
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use openai_sora_client::image_gen::common::{ImageSize, NumImages};
use openai_sora_client::image_gen::sora_image_gen_remix::{sora_image_gen_remix, SoraImageGenRemixRequest};
@@ -58,10 +56,10 @@ use crate::util::allowed_studio_access::allowed_studio_access;
/// This is the number of images (batch size) to generate for each request.
/// We should allow all users to have multiple images generated at once as this
/// is what other providers do.
const MINIMUM_IMAGE_COUNT : u32 = 1;
const MINIMUM_IMAGE_COUNT: u32 = 1;
/// The maximum number of images (batch size) to generate for each request.
const MAXIMUM_IMAGE_COUNT : u32 = 2;
const MAXIMUM_IMAGE_COUNT: u32 = 2;
/// Debug requests can get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
@@ -81,6 +79,9 @@ pub struct EnqueueStudioImageGenRequest {
/// The user's image generation prompt.
pub prompt: String,
/// Turn off the system prompt.
pub disable_system_prompt: Option<bool>,
/// Additional images to include (optional). Up to nine images.
pub maybe_additional_images: Option<Vec<MediaFileToken>>,
@@ -99,6 +100,7 @@ pub enum EnqueueImageGenRequestError {
NotAuthorized,
ServerError,
RateLimited,
TooManyConcurrentTasks,
}
impl ResponseError for EnqueueImageGenRequestError {
@@ -108,6 +110,7 @@ impl ResponseError for EnqueueImageGenRequestError {
EnqueueImageGenRequestError::NotAuthorized => StatusCode::UNAUTHORIZED,
EnqueueImageGenRequestError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
EnqueueImageGenRequestError::RateLimited => StatusCode::TOO_MANY_REQUESTS,
EnqueueImageGenRequestError::TooManyConcurrentTasks => StatusCode::TOO_MANY_REQUESTS,
}
}
@@ -117,6 +120,7 @@ impl ResponseError for EnqueueImageGenRequestError {
EnqueueImageGenRequestError::NotAuthorized => "unauthorized".to_string(),
EnqueueImageGenRequestError::ServerError => "server error".to_string(),
EnqueueImageGenRequestError::RateLimited => "rate limited".to_string(),
EnqueueImageGenRequestError::TooManyConcurrentTasks => "too many concurrent tasks".to_string(),
};
to_simple_json_error(&error_reason, self.status_code())
@@ -147,12 +151,7 @@ impl Display for EnqueueImageGenRequestError {
),
params(("request" = EnqueueStudioImageGenRequest, description = "Payload for Image Generation Request"))
)]
pub async fn enqueue_studio_image_generation_handler(
http_request: HttpRequest,
request: Json<EnqueueStudioImageGenRequest>,
server_state: Data<Arc<ServerState>>
) -> Result<HttpResponse, EnqueueImageGenRequestError> {
pub async fn enqueue_studio_image_generation_handler(http_request: HttpRequest, request: Json<EnqueueStudioImageGenRequest>, server_state: Data<Arc<ServerState>>) -> Result<HttpResponse, EnqueueImageGenRequestError> {
validate_request(&request)?;
let mut mysql_connection = server_state.mysql_pool.acquire().await.map_err(|err| {
@@ -162,12 +161,10 @@ pub async fn enqueue_studio_image_generation_handler(
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection).await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
let maybe_user_session = server_state.session_checker.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection).await.map_err(|e| {
warn!("Session checker error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
let mut maybe_user_token: Option<UserToken> = None;
@@ -178,10 +175,7 @@ pub async fn enqueue_studio_image_generation_handler(
// ==================== PLANS ==================== //
// Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment_old,
maybe_user_session.as_ref()
);
let plan = get_correct_plan_for_session(server_state.server_environment_old, maybe_user_session.as_ref());
// Separate priority for animation.
let priority_level = plan.web_vc_base_priority_level();
@@ -190,9 +184,7 @@ pub async fn enqueue_studio_image_generation_handler(
let is_debug_request = get_request_header_optional(&http_request, DEBUG_HEADER_NAME).is_some();
let maybe_routing_tag = get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME).map(
|routing_tag| routing_tag.trim().to_string()
);
let maybe_routing_tag = get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME).map(|routing_tag| routing_tag.trim().to_string());
// ==================== BANNED USERS ==================== //
@@ -224,12 +216,10 @@ pub async fn enqueue_studio_image_generation_handler(
return Err(EnqueueImageGenRequestError::BadInput(reason));
}
insert_idempotency_token(&request.uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
EnqueueImageGenRequestError::BadInput("invalid idempotency token".to_string())
})?;
insert_idempotency_token(&request.uuid_idempotency_token, &mut *mysql_connection).await.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
EnqueueImageGenRequestError::BadInput("invalid idempotency token".to_string())
})?;
// ==================== DOWNLOAD FILES AND UPLOAD TO SORA ==================== //
@@ -241,30 +231,19 @@ pub async fn enqueue_studio_image_generation_handler(
let scene_media_token = request.snapshot_media_token.clone();
let scene_media_path = query_and_download_media_file(
&scene_media_token,
&work_temp_dir,
&public_bucket_client,
&server_state.mysql_pool
).await.map_err(|err| {
let scene_media_path = query_and_download_media_file(&scene_media_token, &work_temp_dir, &public_bucket_client, &server_state.mysql_pool).await.map_err(|err| {
error!("Failed to download scene media file: {:?}", err);
EnqueueImageGenRequestError::ServerError
})?;
let mut files_to_upload = Vec::with_capacity(1 + request.maybe_additional_images
.as_ref().map(|v| v.len()).unwrap_or(0));
let mut files_to_upload = Vec::with_capacity(1 + request.maybe_additional_images.as_ref().map(|v| v.len()).unwrap_or(0));
files_to_upload.push(scene_media_path.clone());
let additional_images = request.maybe_additional_images.clone().unwrap_or_else(|| vec![]);
for media_file_token in &additional_images {
let media_file_path = query_and_download_media_file(
&media_file_token,
&work_temp_dir,
&public_bucket_client,
&server_state.mysql_pool
).await.map_err(|err| {
let media_file_path = query_and_download_media_file(&media_file_token, &work_temp_dir, &public_bucket_client, &server_state.mysql_pool).await.map_err(|err| {
error!("Failed to download additional media file: {:?}", err);
EnqueueImageGenRequestError::ServerError
})?;
@@ -274,30 +253,25 @@ pub async fn enqueue_studio_image_generation_handler(
// ==================== HANDLE SORA CREDENTIALS ==================== //
let mut redis = server_state.redis_pool
.get()
.map_err(|e| {
error!("redis error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
let mut redis = server_state.redis_pool.get().map_err(|e| {
error!("redis error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
let sora_credentials = get_sora_credentials_from_request(&http_request, &mut redis)
.map_err(|e| {
error!("sora credential error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
let sora_credentials = get_sora_credentials_from_request(&http_request, &mut redis).map_err(|e| {
error!("sora credential error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
// ==================== HANDLE SORA UPLOAD ==================== //
let mut sora_media_tokens = Vec::with_capacity(files_to_upload.len());
for file_path in files_to_upload {
let sora_upload_response = sora_media_upload_from_file(file_path, &sora_credentials)
.await
.map_err(|err| {
error!("Failed to upload scene media to Sora: {:?}", err);
EnqueueImageGenRequestError::ServerError
})?;
let sora_upload_response = sora_media_upload_from_file(file_path, &sora_credentials).await.map_err(|err| {
error!("Failed to upload scene media to Sora: {:?}", err);
EnqueueImageGenRequestError::ServerError
})?;
debug!("Uploaded media to Sora : {:?}", sora_upload_response);
sora_media_tokens.push(sora_upload_response.id);
@@ -306,29 +280,105 @@ pub async fn enqueue_studio_image_generation_handler(
// ==================== HANDLE SORA PROMPT ==================== //
let number_of_samples = match request.maybe_number_of_samples {
Some(val) => if val > MAXIMUM_IMAGE_COUNT {
MAXIMUM_IMAGE_COUNT
} else if val < MINIMUM_IMAGE_COUNT {
MINIMUM_IMAGE_COUNT
} else {
val
Some(val) => {
if val > MAXIMUM_IMAGE_COUNT {
MAXIMUM_IMAGE_COUNT
} else if val < MINIMUM_IMAGE_COUNT {
MINIMUM_IMAGE_COUNT
} else {
val
}
},
None => 1,
};
let prompt = request.prompt.clone();
let prompt = create_prompt(&request);
let response = sora_image_gen_remix(SoraImageGenRemixRequest {
let mut response = sora_image_gen_remix(SoraImageGenRemixRequest {
prompt: prompt.clone(),
num_images: NumImages::One,
image_size: ImageSize::Square,
sora_media_tokens: sora_media_tokens.clone(),
credentials: &sora_credentials,
}).await
.map_err(|err| {
error!("Failed to call Sora image generation: {:?}", err);
EnqueueImageGenRequestError::ServerError
})?;
credentials: &sora_credentials
}).await;
debug!("Sora image gen response: {:?}", response);
if let Err(SoraError::SentinelBlock(msg)) = &response {
error!("Sora sentinel block, attempting refresh: {}", msg);
match generate_token().await {
Ok(sentinel_token) => {
set_sora_credential_field_in_redis(
&mut redis,
RedisSoraCredentialSubkey::Sentinel,
&sentinel_token
).map_err(|e| {
EnqueueImageGenRequestError::ServerError
})?;
let updated_sora_credentials = get_sora_credentials_from_request(&http_request, &mut redis).map_err(|e| {
error!("sora credential error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
// Retry the request with new sentinel
response = sora_image_gen_remix(SoraImageGenRemixRequest {
prompt: prompt.clone(),
num_images: NumImages::One,
image_size: ImageSize::Square,
sora_media_tokens: sora_media_tokens.clone(),
credentials: &updated_sora_credentials
}).await;
},
Err(e) => {
// Fallback to the old refresh stragegy relying on the external service
match refresh_sentinel().await {
Ok(new_sentinel) => {
// Update the credentials with new sentinel
let updated_sora_credentials = get_sora_credentials_from_request(&http_request, &mut redis).map_err(|e| {
error!("sora credential error: {:?}", e);
EnqueueImageGenRequestError::ServerError
})?;
// Retry the request with new sentinel
response = sora_image_gen_remix(SoraImageGenRemixRequest {
prompt: prompt.clone(),
num_images: NumImages::One,
image_size: ImageSize::Square,
sora_media_tokens: sora_media_tokens.clone(),
credentials: &updated_sora_credentials
}).await;
},
Err(e) => {
error!("Failed to refresh Sora sentinel: {:?}", e);
}
}
}
}
}
let response = response.map_err(|err| {
match err {
SoraError::TokenExpired(msg) => {
error!("Sora token expired, needs refresh: {}", msg);
// TODO: Implement token refresh logic here
EnqueueImageGenRequestError::ServerError
},
SoraError::SentinelBlock(msg) => {
error!("Sora sentinel block still occurring after refresh: {}", msg);
EnqueueImageGenRequestError::ServerError
},
SoraError::TooManyConcurrentTasks(msg) => {
error!("Sora too many concurrent tasks: {}", msg);
EnqueueImageGenRequestError::TooManyConcurrentTasks
},
_ => {
error!("Failed to call Sora image generation: {:?}", err);
EnqueueImageGenRequestError::ServerError
},
}
})?;
debug!("Sora image gen response: {:?}", response);
@@ -339,7 +389,7 @@ pub async fn enqueue_studio_image_generation_handler(
maybe_additional_media_file_tokens: Some(additional_images),
maybe_number_of_samples: Some(number_of_samples),
maybe_sora_media_upload_tokens: Some(sora_media_tokens),
maybe_sora_task_id: Some(response.task_id),
maybe_sora_task_id: Some(response.task_id)
};
// create the inference args here
@@ -352,7 +402,7 @@ pub async fn enqueue_studio_image_generation_handler(
maybe_product_category: None, // This is not a product anymore
inference_category: InferenceCategory::ImageGeneration,
maybe_model_type: Some(InferenceModelType::ImageGenApi), // NB: Model is static during inference
maybe_model_token: None, // NB: Model is static during inference
maybe_model_token: None, // NB: Model is static during inference
maybe_input_source_token: None,
maybe_input_source_token_type: None,
maybe_download_url: None,
@@ -372,7 +422,8 @@ pub async fn enqueue_studio_image_generation_handler(
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
})
.await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
@@ -382,24 +433,18 @@ pub async fn enqueue_studio_image_generation_handler(
return Err(EnqueueImageGenRequestError::BadInput("Duplicate idempotency token".to_string()));
}
return Err(EnqueueImageGenRequestError::ServerError);
}
},
};
let response: EnqueueImageGenRequestSuccessResponse = EnqueueImageGenRequestSuccessResponse {
success: true,
inference_job_token: job_token,
};
let response: EnqueueImageGenRequestSuccessResponse = EnqueueImageGenRequestSuccessResponse { success: true, inference_job_token: job_token };
let body = serde_json::to_string(&response)
.map_err(|_e| EnqueueImageGenRequestError::ServerError)?;
let body = serde_json::to_string(&response).map_err(|_e| EnqueueImageGenRequestError::ServerError)?;
// Error handling 101 rust result type returned like so.
Ok(HttpResponse::Ok().content_type("application/json").body(body))
}
fn validate_request(
request: &Json<EnqueueStudioImageGenRequest>,
) -> Result<(), EnqueueImageGenRequestError> {
fn validate_request(request: &Json<EnqueueStudioImageGenRequest>) -> Result<(), EnqueueImageGenRequestError> {
Ok(())
}
@@ -416,28 +461,19 @@ impl From<anyhow::Error> for DownloadMediaFileError {
}
}
async fn query_and_download_media_file(
media_file_token: &MediaFileToken,
download_dir: &TempDir,
bucket_client: &BucketClient,
mysql_pool: &MySqlPool,
) -> Result<PathBuf, DownloadMediaFileError> {
let media_file_result = get_media_file(
&media_file_token,
false,
mysql_pool
).await;
async fn query_and_download_media_file(media_file_token: &MediaFileToken, download_dir: &TempDir, bucket_client: &BucketClient, mysql_pool: &MySqlPool) -> Result<PathBuf, DownloadMediaFileError> {
let media_file_result = get_media_file(&media_file_token, false, mysql_pool).await;
let media_file= match media_file_result {
let media_file = match media_file_result {
Ok(Some(result)) => result,
Ok(None) => {
warn!("could not find media file in database: {:?}", media_file_token);
return Err(DownloadMediaFileError::MediaFileNotFound)
}
return Err(DownloadMediaFileError::MediaFileNotFound);
},
Err(e) => {
error!("could not query media file: {:?}", e);
return Err(DownloadMediaFileError::Other(e))
}
return Err(DownloadMediaFileError::Other(e));
},
};
let media_file_bucket_path = MediaFileBucketPath::from_object_hash(
@@ -451,10 +487,34 @@ async fn query_and_download_media_file(
info!("Downloading from bucket...");
bucket_client.download_file_to_disk(
&media_file_bucket_path.to_full_object_pathbuf(),
&download_file_path,
).await?;
bucket_client.download_file_to_disk(&media_file_bucket_path.to_full_object_pathbuf(), &download_file_path).await?;
Ok(download_file_path)
}
const SYSTEM_PROMPT_SINGLE_IMAGE: &str =
"Use the image as a reference for the composition exactly. Copy the character pose, layout, scene, and the objects exactly - no deviation.";
const SYSTEM_PROMPT_MULTI_IMAGE: &str =
"Use the first image as a reference for the composition exactly. Copy the character pose, layout, scene, and the objects exactly from the first image - no deviation.";
fn create_prompt(request: &EnqueueStudioImageGenRequest) -> String {
let disable_system_prompt = request.disable_system_prompt
.clone()
.unwrap_or(false);
if disable_system_prompt {
return request.prompt.clone();
}
let extra_image_count = request.maybe_additional_images
.as_ref()
.map(|v| v.len())
.unwrap_or(0);
if extra_image_count == 0 {
format!("{} {}", SYSTEM_PROMPT_SINGLE_IMAGE, request.prompt)
} else {
format!("{} {}", SYSTEM_PROMPT_MULTI_IMAGE, request.prompt)
}
}
@@ -31,7 +31,7 @@ use enums::by_table::generic_inference_jobs::inference_category::InferenceCatego
use enums::common::job_status_plus::JobStatusPlus;
use enums::no_table::style_transfer::style_transfer_name::StyleTransferName;
use mysql_queries::queries::generic_inference::web::job_status::GenericInferenceJobStatus;
use mysql_queries::queries::generic_inference::web::list_session_jobs::{list_session_jobs_from_connection, ListSessionJobsForUserArgs};
use mysql_queries::queries::generic_inference::web::list_session_jobs::{list_session_jobs_from_connection, ListSessionJobsForUserArgs, SessionUser};
use primitives::numerics::i64_to_u64_zero_clamped::i64_to_u64_zero_clamped;
use redis_common::redis_keys::RedisKeys;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
@@ -238,11 +238,6 @@ pub async fn list_session_jobs_handler(
ListSessionJobsError::ServerError
})?;
let session = match maybe_user_session {
Some(session) => session,
None => return Err(ListSessionJobsError::NotAuthorized),
};
let include_states = query.include_states
.as_deref()
.map(|s| s.split(",")
@@ -255,8 +250,23 @@ pub async fn list_session_jobs_handler(
.filter_map(|status| JobStatusPlus::from_str(status).ok())
.collect::<HashSet<_>>());
let user;
match (maybe_user_session.as_ref(), maybe_avt_token.as_ref()) {
(Some(session), _) => {
user = SessionUser::User(&session.user_token_typed);
}
(None, Some(avt_token)) => {
user = SessionUser::Anonymous(avt_token);
}
(None, None) => {
// TODO(bt,2025-04-15): We should install an AVT cookie.
return Err(ListSessionJobsError::NotAuthorized);
}
}
let args = ListSessionJobsForUserArgs {
user_token: &session.user_token_typed,
user,
maybe_include_job_statuses: include_states.as_ref(),
maybe_exclude_job_statuses: exclude_states.as_ref(),
};
+1
View File
@@ -0,0 +1 @@
node_modules
+56
View File
@@ -0,0 +1,56 @@
{
"root": true,
"ignorePatterns": [
"**/*"
],
"plugins": [
"@nx"
],
"overrides": [
{
"files": [
"*.ts",
"*.tsx",
"*.js",
"*.jsx"
],
"rules": {
"@nx/enforce-module-boundaries": [
"error",
{
"enforceBuildableLibDependency": true,
"allow": [],
"depConstraints": [
{
"sourceTag": "*",
"onlyDependOnLibsWithTags": [
"*"
]
}
]
}
]
}
},
{
"files": [
"*.ts",
"*.tsx"
],
"extends": [
"plugin:@nx/typescript"
],
"rules": {}
},
{
"files": [
"*.js",
"*.jsx"
],
"extends": [
"plugin:@nx/javascript"
],
"rules": {}
}
]
}
+2 -2
View File
@@ -1,7 +1,7 @@
{
"recommendations": [
"nrwl.angular-console",
"esbenp.prettier-vscode"
"esbenp.prettier-vscode",
"dbaeumer.vscode-eslint"
]
}
+34 -88
View File
@@ -1,101 +1,47 @@
# Frontend
<a alt="Nx logo" href="https://nx.dev" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/nrwl/nx/master/images/nx-logo.png" width="45"></a>
This is an `nx` monorepo that can contain multiple apps and shared libraries.
✨ Your new, shiny [Nx workspace](https://nx.dev) is ready ✨.
All commands to run these projects are performed from _this_ directory.
[Learn more about this workspace setup and its capabilities](https://nx.dev/getting-started/tutorials/react-monorepo-tutorial?utm_source=nx_project&amp;utm_medium=readme&amp;utm_campaign=nx_projects) or run `npx nx graph` to visually explore what was created. Now, let's get you up to speed!
## Run tasks
## Install dependencies
To run the dev server for your app, use:
```sh
npx nx serve editor3d
```
npm install
```
To create a production bundle:
## Editor2d
```sh
Run dev server:
```bash
nx dev editor2d
```
## Editor3d
Run dev server:
```bash
nx dev editor3d
```
```
call when starting on main or a new branch:
./clean_modules
The names for @frontend and @storyteller come from the package.json file in the libs folder.
import { Login } from "@frontend/login";
import { api } from "@storyteller/api";
building:
npx nx build editor3d
npx nx build editor2d
dev:
npx nx dev editor3d
npx nx dev editor2d
```
To see all available targets to run for a project, run:
```sh
npx nx show project editor3d
```
These targets are either [inferred automatically](https://nx.dev/concepts/inferred-tasks?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) or defined in the `project.json` or `package.json` files.
[More about running tasks in the docs &raquo;](https://nx.dev/features/run-tasks?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
## Add new projects
While you could add new projects to your workspace manually, you might want to leverage [Nx plugins](https://nx.dev/concepts/nx-plugins?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) and their [code generation](https://nx.dev/features/generate-code?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) feature.
Use the plugin's generator to create new projects.
To generate a new application, use:
```sh
npx nx g @nx/react:app demo
```
To generate a new library, use:
```sh
npx nx g @nx/react:lib mylib
```
You can use `npx nx list` to get a list of installed plugins. Then, run `npx nx list <plugin-name>` to learn about more specific capabilities of a particular plugin. Alternatively, [install Nx Console](https://nx.dev/getting-started/editor-setup?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) to browse plugins and generators in your IDE.
[Learn more about Nx plugins &raquo;](https://nx.dev/concepts/nx-plugins?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) | [Browse the plugin registry &raquo;](https://nx.dev/plugin-registry?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
## Set up CI!
### Step 1
To connect to Nx Cloud, run the following command:
```sh
npx nx connect
```
Connecting to Nx Cloud ensures a [fast and scalable CI](https://nx.dev/ci/intro/why-nx-cloud?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects) pipeline. It includes features such as:
- [Remote caching](https://nx.dev/ci/features/remote-cache?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
- [Task distribution across multiple machines](https://nx.dev/ci/features/distribute-task-execution?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
- [Automated e2e test splitting](https://nx.dev/ci/features/split-e2e-tasks?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
- [Task flakiness detection and rerunning](https://nx.dev/ci/features/flaky-tasks?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
### Step 2
Use the following command to configure a CI workflow for your workspace:
```sh
npx nx g ci-workflow
```
[Learn more about Nx on CI](https://nx.dev/ci/intro/ci-with-nx#ready-get-started-with-your-provider?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
## Install Nx Console
Nx Console is an editor extension that enriches your developer experience. It lets you run tasks, generate code, and improves code autocompletion in your IDE. It is available for VSCode and IntelliJ.
[Install Nx Console &raquo;](https://nx.dev/getting-started/editor-setup?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
## Useful links
Learn more:
- [Learn more about this workspace setup](https://nx.dev/getting-started/tutorials/react-monorepo-tutorial?utm_source=nx_project&amp;utm_medium=readme&amp;utm_campaign=nx_projects)
- [Learn about Nx on CI](https://nx.dev/ci/intro/ci-with-nx?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
- [Releasing Packages with Nx release](https://nx.dev/features/manage-releases?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
- [What are Nx plugins?](https://nx.dev/concepts/nx-plugins?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
And join the Nx community:
- [Discord](https://go.nx.dev/community)
- [Follow us on X](https://twitter.com/nxdevtools) or [LinkedIn](https://www.linkedin.com/company/nrwl)
- [Our Youtube channel](https://www.youtube.com/@nxdevtools)
- [Our blog](https://nx.dev/blog?utm_source=nx_project&utm_medium=readme&utm_campaign=nx_projects)
+2 -3
View File
@@ -1,6 +1,6 @@
[build]
command = "./script/netlify_build.sh"
publish = "dist/"
command = "./apps/editor2d/script/netlify_build.sh"
publish = "apps/editor2d/dist/"
#[build.environment]
#REACT_APP_PUBLIC_POSTHOG_KEY = "phc_jBFgac0mVALAFk3negnSfYCcHgvkT00yBLQDmCDYNBb"
@@ -16,7 +16,6 @@ publish = "dist/"
for = "/build/*"
[headers.values]
"Cache-Control" = "public, max-age=31560000, immutable"
# [[redirects]]
+8 -1
View File
@@ -4,7 +4,14 @@
"version": "0.0.0",
"type": "module",
"nx": {
"name": "editor2d"
"name": "editor2d",
"targets": {
"build": {
"dependsOn": [
"^build"
]
}
}
},
"scripts": {
"dev": "echo 'editor2d' && vite",
Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

@@ -3,10 +3,10 @@
set -euxo pipefail
echo "Run build script (TODO: Make strict)"
yarn run build2
nx build editor2d
echo "Copy test files into dist"
cp -r test/* dist/
echo "Change to project dir"
pushd apps/editor2d/
echo "Copy netlify configs into dist"
cp _headers dist/
@@ -15,5 +15,6 @@ cp _redirects dist/
echo "Copy netlify 404.html page into dist"
cp "404.html" dist/
echo "List files after build"
echo "List final files in build"
find dist/
@@ -0,0 +1,122 @@
import { faFolder } from "@fortawesome/pro-solid-svg-icons"
import { useEffect, useState } from "react";
import { Button } from "~/components/ui"
const testUrl = "https://api.fakeyou.com/v1/media_files/list/user/hanashi?include_user_uploads=true&page_size=24&filter_media_classes=unknown";
const getTestData = (callback: (data: any) => void) => {
const netCall = async () => {
// GET the test url
const testResults = await fetch(testUrl);
const testJson = await testResults.json();
console.log("testResults", testJson);
return testJson;
}
netCall().then(callback);
}
export const GalleryRootComponent = ({
className
}: {
className?: string
}) => {
const [testData, setTestData] = useState<any>();
useEffect(() => {
getTestData(setTestData);
}, [])
const aggregatedData = aggregateGalleryData({ data: testData });
return (
<div className={className}>
<div className="m-32">
<div className="flex w-fit gap-x-6 items-center">
<span className="text-white font-bold text-5xl">My Gallery</span>
<Button variant="secondary" icon={faFolder} className="h-8">
Open Folder
</Button>
</div>
<div className="overflow-y-auto mt-12 flex flex-col gap-y-8">
{Array.from(aggregatedData.entries()).map(([dateEpoch, results]) => (<GallerySection dateEpoch={dateEpoch} results={results} />))}
</div>
</div>
</div>
)
}
const GallerySection = ({
dateEpoch,
results
}: {
dateEpoch: number,
results: any[]
}) => {
const dateObj = new Date(dateEpoch);
const title = dateObj.toLocaleDateString("en-US", {
year: "numeric",
month: "long",
day: "numeric"
});
return (
<div className="w-full">
<span className="text-lg">{title}</span>
<div className="flex gap-4 flex-wrap mt-4">
{results.map((result) => (<GalleryCard result={result} />))}
</div>
</div>
)
}
const GalleryCard = ({ result }: {
result: {
media_links: {
cdn_url: string,
maybe_thumbnail_template: string,
},
}
}) => {
return (
<div className="rounded-md size-48 flex items-center justify-center overflow-hidden">
<img src={result.media_links.cdn_url} alt="Thumbnail" className="h-full w-full object-cover" />
</div>
)
}
const aggregateGalleryData = ({
data
}: {
data?: {
results: any[]
}
}) => {
const aggregateMap = new Map<number, any[]>();
if (!data) {
return aggregateMap;
}
data.results.forEach((result) => {
if (result.media_type !== "image") {
return;
}
const dateString = result.updated_at as string;
const dateObj = new Date(dateString);
// Move it the start of the day for comparison
dateObj.setUTCHours(0, 0, 0, 0);
const dateEpoch = dateObj.getTime();
if (!aggregateMap.has(dateEpoch)) {
aggregateMap.set(dateEpoch, []);
}
aggregateMap.get(dateEpoch)!.push(result);
})
return aggregateMap;
}
@@ -0,0 +1 @@
export * from "./GalleryRootComponent";
+40 -19
View File
@@ -148,7 +148,6 @@ export class ApiManager {
formData.append("file", blob);
}
// console.log(`HELLO${formData}`);
return this.fetchMultipartFormData<T>(endpoint, {
method: "POST",
headers: {
@@ -281,10 +280,12 @@ export class Api extends ApiManager {
}
public async enqueueImageGeneration({
disableSystemPrompt,
prompt,
snapshotMediaToken,
additionalImages,
}: {
disableSystemPrompt: boolean;
prompt: string;
snapshotMediaToken: string;
additionalImages?: string[];
@@ -294,6 +295,7 @@ export class Api extends ApiManager {
const uuidIdempotencyToken = crypto.randomUUID(); // Generate a new UUID
const body = {
uuid_idempotency_token: uuidIdempotencyToken,
disable_system_prompt: disableSystemPrompt,
prompt,
snapshot_media_token: snapshotMediaToken, // Changed from scene_media_token to snapshot_media_token
additional_images: additionalImages,
@@ -302,6 +304,7 @@ export class Api extends ApiManager {
const postResponse = await this.post<
{
uuid_idempotency_token: string;
disable_system_prompt: boolean;
prompt: string;
snapshot_media_token: string;
additional_images?: string[];
@@ -327,7 +330,10 @@ export class Api extends ApiManager {
return result;
}
public async pollJobSession(jobToken: string): Promise<
public async pollJobSession(
jobToken: string,
thumbnailWidth: number = 256,
): Promise<
ApiResponse<{
job_token: string;
request: {
@@ -338,6 +344,9 @@ export class Api extends ApiManager {
progress_percentage: number;
};
result: {
image_url: string;
thumbnail_url: string;
public_bucket_path: string;
generated_images?: string[];
error?: string;
};
@@ -345,41 +354,53 @@ export class Api extends ApiManager {
> {
const endpoint = `${this.ApiTargets.BaseApi}/v1/jobs/job/${jobToken}`;
const response = await this.get<{
success?: boolean;
status?: string;
result?: {
generated_images?: string[];
error?: string;
};
BadInput?: string;
}>({
const response = await this.get<any>({
endpoint,
});
console.log("Job Polling Response:");
console.log(response);
let image_url = response.state.maybe_result?.media_links?.cdn_url;
let public_bucket_path =
response.state.maybe_result?.maybe_public_bucket_media_path;
let thumbnail_url =
response.state.maybe_result?.media_links?.maybe_thumbnail_template?.replace(
"{WIDTH}",
thumbnailWidth.toString(),
);
let progress_percentage = response.state.status.progress_percentage;
let status_string = response.state.status.status;
console.log("Image URL:", image_url);
console.log("Thumbnail URL:", thumbnail_url);
console.log("Progress Percentage:", progress_percentage);
console.log("Status", status_string);
console.log("response FROM JOBS");
console.log(response);
const success = response.success ?? false;
const status = response.status ?? "";
const result = response.result ?? {
generated_images: [],
error: undefined,
};
const errorMessage = response.BadInput;
return {
success,
data: {
result,
result: {
image_url: image_url || "",
thumbnail_url: thumbnail_url || "",
public_bucket_path: public_bucket_path || "",
error: undefined,
},
job_token: jobToken,
request: {
maybe_model_title: "Image Generation",
},
status: {
status: status.toLowerCase(),
progress_percentage: 0,
status: status_string.toLowerCase(),
progress_percentage: progress_percentage,
},
},
errorMessage,
@@ -55,7 +55,7 @@ export class Engine {
private uiLayer: Konva.Layer;
private previewLayer: Konva.Layer;
private offScreenCanvas: OffscreenCanvas;
private realTimeDrawEngine: RealTimeDrawEngine;
public realTimeDrawEngine: RealTimeDrawEngine;
private nodesManager: NodesManager;
private nodeIsolator: NodeIsolator;
private nodeTransformer: NodeTransformer;
@@ -66,8 +66,8 @@ export class Engine {
private backgroundNode: ShapeNode | null = null;
private sceneManager: SceneManager;
private undoStackManager: UndoStackManager;
private commandManager: CommandManager;
public undoStackManager: UndoStackManager;
public commandManager: CommandManager;
private videoExtractionHandler: VideoExtractionHandler;
public segmentationButtonCanBePressed: boolean = true;
@@ -56,11 +56,8 @@ export class ShapeNode extends NetworkedNode {
switch (shapeType) {
case ShapeType.CIRCLE:
shape = new Konva.Ellipse({
x: size.width,
y: size.height,
radiusX: size.width,
radiusY: size.height,
...transform,
radiusX: size.width / 2,
radiusY: size.height / 2,
fill: color,
strokeScaleEnabled: false,
draggable: true,
@@ -80,8 +77,6 @@ export class ShapeNode extends NetworkedNode {
case ShapeType.TRIANGLE:
shape = new Konva.Shape({
x: 0,
y: 0,
width: size.width,
height: size.height,
...transform,
@@ -113,12 +108,20 @@ export class ShapeNode extends NetworkedNode {
this.size = size;
this.color = color;
// Add shape to group
// Center the shape
const centerPosition = NodeUtilities.positionNodeOnCanvasCenter({
canvasOffset: canvasPosition,
componentSize: canvasSize,
componentSize: this.size,
maxSize: canvasSize,
});
// If the shape is an ellipse, the center is the origin so we need to offset it
if (shapeType === ShapeType.CIRCLE) {
centerPosition.x += size.width / 2;
centerPosition.y += size.height / 2;
}
// Add shape to group
// bring to the front
this.kNode.moveToTop();
this.kNode.setPosition(centerPosition);
@@ -1,14 +1,12 @@
import { useCallback, useRef } from "react";
import { useCallback, useRef, useState } from "react";
// Components
import { KonvaCanvasContainer } from "./KonvaCanvasContainer";
import { ContextualButtonRetry } from "./ContextualButtonRetry";
import { ContextualToolbarNode } from "./ContextualToolbarNode";
import { ContextualLoadingBar } from "./ContextualLoadingBar";
import { SignaledCanvasDragDropFiles } from "./SignaledCanvasDragDropFiles";
import { SignaledDialogs } from "./SignaledDialogs";
import { SignaledToolbarMain } from "./SignaledToolbarMain";
import { SignaledToolbarVideoExtraction } from "./SignaledToolbarVideoExtraction";
import { PromptBox } from "~/components/PromptBox";
// The KonvaApp is the root of the Konva stage
// and only entry point for anything in Konva JS
@@ -22,9 +20,8 @@ import { useLayoutContext } from "./contextSignals/layout";
// common hooks
import { useRenderCounter } from "~/hooks/useRenderCounter";
import { useNavigate } from "react-router-dom";
import { SignaledMagicBox } from "./SignaledMagicBox";
import { SignaledPromptSlider } from "./SignaledPromptSlider";
import { SignaledPromptText } from "./SignaledPromptText";
import { UndoRedo } from "~/components/reusable/UndoRedo/UndoRedo";
export const KonvaRootComponent = ({
className,
@@ -39,44 +36,66 @@ export const KonvaRootComponent = ({
const navigate = useNavigate();
const appUiContext = useAppUiContext();
const layoutContext = useLayoutContext();
// Add state to track engine initialization
const [isEngineReady, setIsEngineReady] = useState(false);
const engineRef = useRef<EngineType | null>(null);
const konvaContainerCallbackRef = useCallback((node: HTMLDivElement) => {
if (node !== null && engineRef.current === null) {
const options = {
navigate: navigate,
sceneToken: sceneToken,
};
engineRef.current = KonvaApp(node, options);
// Only initialize if we have a node and haven't initialized yet
if (node !== null && !isEngineReady && engineRef.current === null) {
try {
const options = {
navigate: navigate,
sceneToken: sceneToken,
};
engineRef.current = KonvaApp(node, options);
// Only set ready if initialization succeeded
if (engineRef.current) {
setIsEngineReady(true);
}
} catch (error) {
console.error("Failed to initialize KonvaApp:", error);
engineRef.current = null;
setIsEngineReady(false);
}
}
}, []);
}, [navigate, sceneToken, isEngineReady]);
return (
<>
{/* Always render the canvas container */}
<KonvaCanvasContainer
ref={konvaContainerCallbackRef}
className={className}
// retreive the classNames from the parent for sizing/styling
/>
{/* <SignaledMagicBox /> */}
<SignaledCanvasDragDropFiles
openAddImage={appUiContext.openAddImage}
openAddVideo={appUiContext.openAddVideo}
/>
<PromptBox />
<SignaledToolbarMain
layoutSignal={layoutContext.signal}
appUiContext={appUiContext}
/>
{/* <SignaledToolbarVideoExtraction /> */}
<SignaledDialogs
appUiSignal={appUiContext.signal}
resetAll={appUiContext.resetAll}
/>
{/* Conditionally render the loading overlay */}
{!isEngineReady && (
<div className="absolute inset-0 flex items-center justify-center">
<div className="text-gray-500">Initializing canvas...</div>
</div>
)}
{/* <ContextualLoadingBar /> */}
<ContextualToolbarNode />
{/* <ContextualButtonRetry /> */}
{/* Conditionally render the UI components that need the engine */}
{isEngineReady && engineRef.current && (
<UndoRedo engine={engineRef.current}>
<SignaledCanvasDragDropFiles
openAddImage={appUiContext.openAddImage}
openAddVideo={appUiContext.openAddVideo}
/>
<PromptBox />
<SignaledToolbarMain
layoutSignal={layoutContext.signal}
appUiContext={appUiContext}
/>
<SignaledDialogs
appUiSignal={appUiContext.signal}
resetAll={appUiContext.resetAll}
/>
<ContextualToolbarNode />
</UndoRedo>
)}
</>
);
};
@@ -17,6 +17,7 @@ import { EditModeRootComponent } from "~/EditModeRootComponent/EditModeRootCompo
import { useRef } from "react";
import { EngineType } from "~/KonvaApp";
import { GenerationEngine } from "~/KonvaApp/GenerationEngine";
import { GalleryRootComponent } from "~/GalleryRootComponent";
export const Main = withProtectionRoute(() => {
// This is a hook that will log the number of times the component has rerendered
@@ -59,8 +60,10 @@ export const Main = withProtectionRoute(() => {
}
return (
<div className="fixed grid h-full w-full grid-cols-12 grid-rows-12">
{childView}
<div className="fixed h-full w-full">
<div className="fixed grid h-full w-full grid-cols-12 grid-rows-12">
{childView}
</div>
<div className="absolute top-0 w-full p-3.5">
<div className="relative flex w-full">
<div className="absolute left-0">
@@ -74,6 +77,8 @@ export const Main = withProtectionRoute(() => {
</div>
</div>
</div>
{appModeValue === "gallery" &&
(<GalleryRootComponent className="fixed h-full w-full z-1 bg-black" />)}
</div>
);
});
@@ -1,8 +1,10 @@
import React, { createContext, useContext, useState } from "react";
interface JobContextType {
jobToken: string | null;
setJobToken: (token: string | null) => void;
jobTokens: string[];
addJobToken: (token: string) => void;
removeJobToken: (token: string) => void;
clearJobTokens: () => void;
}
const JobContext = createContext<JobContextType | undefined>(undefined);
@@ -10,10 +12,24 @@ const JobContext = createContext<JobContextType | undefined>(undefined);
export const JobProvider: React.FC<{ children: React.ReactNode }> = ({
children,
}) => {
const [jobToken, setJobToken] = useState<string | null>(null);
const [jobTokens, setJobTokens] = useState<string[]>([]);
const addJobToken = (token: string) => {
setJobTokens((prevTokens) => [...prevTokens, token]);
};
const removeJobToken = (token: string) => {
setJobTokens((prevTokens) => prevTokens.filter((t) => t !== token));
};
const clearJobTokens = () => {
setJobTokens([]);
};
return (
<JobContext.Provider value={{ jobToken, setJobToken }}>
<JobContext.Provider
value={{ jobTokens, addJobToken, removeJobToken, clearJobTokens }}
>
{children}
</JobContext.Provider>
);
@@ -47,7 +47,8 @@ export const PromptBox = () => {
const isDesktopApp = window.navigator.userAgent.includes("Tauri");
console.log("Is this a desktop app?", isDesktopApp);
const { jobToken, setJobToken } = useJobContext();
const { jobTokens, addJobToken, removeJobToken, clearJobTokens } =
useJobContext();
//const { lastRenderedBitmap } = useCanvasSignal();
const [prompt, setPrompt] = useState("");
@@ -119,7 +120,7 @@ export const PromptBox = () => {
setUploadingImages((prev) =>
prev.filter((img) => img.id !== uploadId),
);
console.log("Reference image added:", referenceImage);
toast.success("Reference image added.");
} else if (
newState.status === UploaderStates.assetError ||
newState.status === UploaderStates.imageCreateError
@@ -127,7 +128,8 @@ export const PromptBox = () => {
setUploadingImages((prev) =>
prev.filter((img) => img.id !== uploadId),
);
console.error("Upload failed");
toast.error("Upload failed. Please try again.");
}
},
});
@@ -206,7 +208,7 @@ export const PromptBox = () => {
const api = new Api();
let image = getCanvasRenderBitmap();
if (image === undefined) {
console.error(
toast.error(
"Error: Unable to generate image. Please check the input and try again.",
);
return;
@@ -229,30 +231,28 @@ export const PromptBox = () => {
});
if (snapshotMediaToken.data === undefined) {
console.error("Error: Unable to upload scene snapshot.");
toast.error(
"Error: Unable to upload scene snapshot Please try again.",
);
return;
}
console.log("useSystemPrompt", useSystemPrompt);
console.log("Snapshot media token:", snapshotMediaToken.data);
const response = await api.enqueueImageGeneration({
disableSystemPrompt: !useSystemPrompt,
prompt: prompt,
snapshotMediaToken: snapshotMediaToken.data,
additionalImages: referenceImages.map((image) => image.mediaToken),
});
if (response.success === true) {
console.error("Successfully enqueued image generation.");
toast.success("Please wait while we process your image.");
if (response.data) {
console.log("Response:", response.data);
setJobToken(response.data);
addJobToken(response.data);
}
return;
} else {
console.log("Image generation Not Successful.");
toast.error("Failed to generate image. Please try again.");
toast.error("Failed to enqueue image generation. Please try again.");
}
}
} catch (error) {
@@ -63,12 +63,11 @@ export const ToolbarMain = ({
buttonProps={buttonProps.SELECT}
tooltip="Select"
/>
{/* <ToolbarButton
<ToolbarButton
icon={faText}
buttonProps={buttonProps.ADD_TEXT}
tooltip="Add Text"
/> */}
/>
<Popover className="relative">
{({ close }) => (
<>
@@ -41,9 +41,9 @@ export const ToolbarTopLeft = () => {
)}
>
<img
src="/brand/mira-logo.png"
src="/brand/artcraft-logo.png"
alt="logo"
className="h-[30px] select-none pb-1"
className="h-[28px] select-none"
/>
<TabSelector
@@ -1,4 +1,5 @@
import { twMerge } from "tailwind-merge";
import { Login } from "~/app/pages/login";
import { Activity } from "~/components/ui/Activity/Activity";
// import { paperWrapperStyles } from "~/components/styles";
// import { faPlus, faQuestion } from "@fortawesome/pro-solid-svg-icons";
@@ -0,0 +1,73 @@
import { useRef, useCallback, useContext, ReactNode, useEffect } from "react";
import { EngineType } from "~/KonvaApp";
interface UndoRedoProps {
children: ReactNode;
engine: EngineType | null;
}
/**
* A transparent component that provides undo/redo functionality
* with keyboard shortcuts for macOS (Command+Z, Command+Shift+Z)
*/
export const UndoRedo = ({
engine,
children,
}: UndoRedoProps) => {
console.log("UndoRedo");
// We don't need a separate ref since we want to track the prop changes
// Create the event handler with useCallback
const handleKeyDown = useCallback((e: KeyboardEvent) => {
if (!engine) {
return;
}
// Don't capture events when user is editing text
const target = e.target as HTMLElement;
const isEditingText = target.tagName === 'INPUT' ||
target.tagName === 'TEXTAREA' ||
target.isContentEditable;
// Command+Z for Undo
// Ctrl+Shift+Z for Redo (Windows/Linux alternative)
// Windows/Linux: Ctrl+Shift+Z for Redo
if (!navigator.userAgent.includes('Mac') && e.ctrlKey && e.shiftKey && e.key === 'z') {
console.log("Ctrl+Shift+Z pressed: Redo (Windows/Linux)");
e.preventDefault();
engine.undoStackManager.redo();
}
// Delete key handling - only if not editing text
if (!isEditingText && (e.key === 'Delete' || e.key === 'Backspace')) {
console.log("Delete/Backspace pressed: Delete selected nodes");
e.preventDefault();
engine.commandManager.deleteNodes();
}
// macOS: Command+Z for Undo
if (navigator.userAgent.includes('Mac') && e.metaKey && e.key === 'z' && !e.shiftKey) {
console.log("Command+Z pressed: Undo");
e.preventDefault();
engine.undoStackManager.undo();
}
// macOS: Command+Shift+Z for Redo
if (navigator.userAgent.includes('Mac') && e.metaKey && e.key === 'z' && e.shiftKey) {
console.log("Command+Shift+Z pressed: Redo");
e.preventDefault();
engine.undoStackManager.redo();
engine.realTimeDrawEngine.render();
}
}, [engine]);
// Actually set up the event listeners using useEffect
useEffect(() => {
window.addEventListener('keydown', handleKeyDown);
// Cleanup function to remove the event listener
return () => {
window.removeEventListener('keydown', handleKeyDown);
};
}, [handleKeyDown]); // Depend on the memoized handler
return children;
};
@@ -6,6 +6,8 @@ import { CompletedCard } from "./CompletedCard";
import { InProgressCard } from "./InProgressCard";
import { useJobContext } from "~/components/JobContext";
import { Api } from "~/KonvaApp/Api";
import { toast } from "sonner";
// Dummy data types
interface CompletedItem {
token: string;
@@ -65,90 +67,72 @@ const dummyCompletedItems: CompletedItem[] = [
// ];
export function Activity() {
const { jobToken, setJobToken } = useJobContext();
const { jobTokens, addJobToken, removeJobToken, clearJobTokens } =
useJobContext();
const [completedItems, setCompletedItems] = useState<CompletedItem[]>([]);
const [jobs, setJobs] = useState<ActiveJob[]>([]);
const [loading, setLoading] = useState(true);
// Dummy polling function
useEffect(() => {
const pollData = async () => {
if (jobToken) {
console.log("Current Job Token:", jobToken);
if (jobTokens.length > 0) {
console.log("Current Job Tokens:", jobTokens);
const api = new Api();
const job = await api.pollJobSession(jobToken);
console.log("Job response:", job);
if (job.success && job.data) {
const { status, progress_percentage } = job.data.status;
const { job_token } = job.data;
// Create a map to store the latest job data
const updatedJobs = new Map<string, ActiveJob>();
console.log("Job status:", status);
console.log("Progress percentage:", progress_percentage);
console.log("Job token:", job_token);
for (const jobToken of jobTokens) {
const job = await api.pollJobSession(jobToken);
// Update jobs with real job data
if (
status.toLowerCase() === "completed" ||
status.toLowerCase() === "failed"
) {
// If job is completed or failed, remove it from active jobs
setJobs((prevJobs) =>
prevJobs.filter((j) => j.job_token !== job_token),
);
if (job.success && job.data) {
const { status, progress_percentage } = job.data.status;
const { job_token } = job.data;
// Add to completed items if it was successful
if (
status.toLowerCase() === "completed" &&
job.data.result.generated_images
status.toLowerCase() === "complete_success" ||
status.toLowerCase() === "failed"
) {
const newCompletedItem: CompletedItem = {
id: job_token,
title: job.data.request.maybe_model_title || "Image Generation",
timestamp: new Date().toISOString(),
type: "image",
imageUrl: job.data.result.generated_images[0] || "",
};
// If job is complete_success or failed, we'll remove it later
removeJobToken(job_token);
setCompletedItems((prev) => [newCompletedItem, ...prev]);
// Add to complete_success items if it was successful
if (status.toLowerCase() === "complete_success") {
const newCompletedItem: CompletedItem = {
token: job_token,
maybe_title:
job.data.request.maybe_model_title || "Image Generation",
public_bucket_path: job.data.result.image_url || "",
updated_at: new Date().toISOString(),
};
toast.success("Done Processing Image");
setCompletedItems((prev) => [newCompletedItem, ...prev]);
}
} else {
// Store the job in our map
const updatedJob: ActiveJob = {
job_token,
request: {
maybe_model_title:
job.data.request.maybe_model_title || "Image Generation",
},
status: {
status: status.toUpperCase(),
progress_percentage: progress_percentage || 0,
},
};
updatedJobs.set(job_token, updatedJob);
}
} else {
// Update or add the job to the active jobs list
const updatedJob: ActiveJob = {
job_token,
request: {
maybe_model_title:
job.data.request.maybe_model_title || "Image Generation",
},
status: {
status: status.toUpperCase(),
progress_percentage: progress_percentage || 0,
},
};
setJobs((prevJobs) => {
const existingJobIndex = prevJobs.findIndex(
(j) => j.job_token === job_token,
);
if (existingJobIndex >= 0) {
// Update existing job
const newJobs = [...prevJobs];
newJobs[existingJobIndex] = updatedJob;
return newJobs;
} else {
// Add new job
return [updatedJob, ...prevJobs];
}
});
console.error("Failed to fetch job status:", job.errorMessage);
}
} else {
console.error("Failed to fetch job status:", job.errorMessage);
}
} else {
console.log("No Job Token available.");
}
if (jobs.length === 0) {
// Update jobs state with the deduplicated jobs
setJobs(Array.from(updatedJobs.values()));
}
if (jobTokens.length === 0) {
setJobs([]);
setLoading(false);
}
};
@@ -163,7 +147,7 @@ export function Activity() {
}, 5000);
return () => clearInterval(interval);
}, [jobToken]);
}, [jobTokens, removeJobToken]);
return (
<PopoverMenu
@@ -17,8 +17,10 @@ interface Props {
export function CompletedCard({ job }: Props) {
const [loadError, setLoadError] = useState(false);
const [isModalOpen, setIsModalOpen] = useState(false);
const imageUrl = job.public_bucket_path + "-thumb.gif";
const imageUrl = job.public_bucket_path;
const fullImageUrl = job.public_bucket_path;
console.log("I am hjere");
console.log(job);
return (
<>
@@ -87,19 +87,22 @@ export class EngineApi extends ApiManager {
}
public async enqueueImageGeneration({
disableSystemPrompt,
prompt,
snapshotMediaToken,
additionalImages,
}: {
disableSystemPrompt: boolean;
prompt: string;
snapshotMediaToken: string;
additionalImages?: string[];
}): Promise<ApiResponse<string>> {
const endpoint = `${this.ApiTargets.BaseApi}/v1/image_studio/prompt`;
// for now ...
const uuidIdempotencyToken = crypto.randomUUID(); // Generate a new UUID
const body = {
uuid_idempotency_token: uuidIdempotencyToken,
disable_system_prompt: disableSystemPrompt,
prompt,
snapshot_media_token: snapshotMediaToken, // Changed from scene_media_token to snapshot_media_token
additional_images: additionalImages,
@@ -108,6 +111,7 @@ export class EngineApi extends ApiManager {
const postResponse = await this.post<
{
uuid_idempotency_token: string;
disable_system_prompt: boolean;
prompt: string;
snapshot_media_token: string;
additional_images?: string[];
@@ -40,7 +40,7 @@ export class JobsApi extends ApiManager {
public ListRecentJobs(): Promise<ApiResponse<Job[]>> {
const endpoint = `${this.ApiTargets.BaseApi}/v1/jobs/session`;
return this.get<{
success: boolean;
jobs: Job[];
@@ -45,7 +45,7 @@ export const Button = ({
);
const className = twMerge(
"text-sm font-medium rounded-lg px-3 py-2 border border-[#363636] shadow-sm focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 transition-all duration-150 flex gap-2 items-center justify-center",
"text-sm font-medium rounded-lg px-3 py-2 border border-transparent shadow-sm focus-visible:outline focus-visible:outline-0 focus-visible:outline-offset-0 transition-all duration-150 flex gap-2 items-center justify-center",
getVariantClassNames(propsVariant),
propsClassName,
disabledClass,
@@ -15,7 +15,7 @@ export const CloseButton = ({
}: CloseButtonProps) => {
const sizeClasses = {
sm: "h-5 w-5 text-sm",
md: "h-7 w-7 text-lg",
md: "h-7 w-7 text-md",
lg: "h-9 w-9 text-xl",
};
@@ -0,0 +1,122 @@
import { CloseButton, Button } from "~/components";
import { Transition } from "@headlessui/react";
import { createPortal } from "react-dom";
import dayjs from "dayjs";
import { faDownToLine } from "@fortawesome/pro-solid-svg-icons";
interface LightboxModalProps {
isOpen: boolean;
onClose: () => void;
imageUrl?: string | null;
imageAlt?: string;
onImageError?: () => void;
title?: string;
createdAt?: string;
additionalInfo?: React.ReactNode;
downloadUrl?: string;
}
export function LightboxModal({
isOpen,
onClose,
imageUrl,
imageAlt = "",
onImageError,
title,
createdAt,
additionalInfo,
downloadUrl,
}: LightboxModalProps) {
return createPortal(
<Transition appear show={isOpen}>
<div className="fixed inset-0 z-[100]">
<Transition.Child
enter="ease-out duration-300"
enterFrom="opacity-0"
enterTo="opacity-100"
leave="ease-in duration-200"
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div
className="fixed inset-0 cursor-pointer bg-black/60"
onClick={onClose}
/>
</Transition.Child>
<div
className="fixed inset-0 flex items-center justify-center p-4"
onClick={onClose}
>
<Transition.Child
enter="ease-out duration-300"
enterFrom="opacity-0 scale-95"
enterTo="opacity-100 scale-100"
leave="ease-in duration-200"
leaveFrom="opacity-100 scale-100"
leaveTo="opacity-0 scale-95"
>
<div
className="relative h-[90vh] w-[80vw] rounded-xl bg-[#2C2C2C]"
onClick={(e) => e.stopPropagation()}
>
<CloseButton
onClick={onClose}
className="absolute right-4 top-4 z-10"
/>
<div className="grid h-full grid-cols-3 gap-6">
<div className="col-span-2 flex h-full items-center justify-center overflow-hidden rounded-l-xl bg-black/40">
{!imageUrl ? (
<div className="flex h-full w-full items-center justify-center bg-gray-800">
<span className="text-white/60">Image not available</span>
</div>
) : (
<img
src={imageUrl}
alt={imageAlt}
className="h-full w-full object-contain"
onError={onImageError}
crossOrigin="anonymous"
referrerPolicy="no-referrer"
/>
)}
</div>
<div className="flex h-full flex-col py-5 pe-5">
<div className="flex-1 space-y-4">
<div className="text-xl font-medium">
{title || "Image Generation"}
</div>
{createdAt && (
<div className="text-sm text-white/60">
Created:{" "}
{dayjs(createdAt).format("MMM D, YYYY HH:mm:ss")}
</div>
)}
{additionalInfo}
</div>
{downloadUrl && (
<div className="flex justify-end">
<a
href={downloadUrl}
download
onClick={(e) => e.stopPropagation()}
className="no-underline"
>
<Button
icon={faDownToLine}
onClick={(e) => e.stopPropagation()}
>
Download
</Button>
</a>
</div>
)}
</div>
</div>
</div>
</Transition.Child>
</div>
</div>
</Transition>,
document.body,
);
}
@@ -17,6 +17,7 @@ interface PopoverMenuProps {
onAdd?: () => void;
triggerIcon?: ReactNode;
showAddButton?: boolean;
disableAddButton?: boolean;
showIconsInList?: boolean;
mode?: "default" | "toggle" | "button";
triggerLabel?: string;
@@ -36,6 +37,7 @@ export const PopoverMenu = ({
onAdd,
triggerIcon,
showAddButton = false,
disableAddButton = false,
showIconsInList = false,
mode = "default",
triggerLabel,
@@ -84,9 +86,15 @@ export const PopoverMenu = ({
{({ close }) => (
<>
<Popover.Button className={className}>
{triggerIcon}
{mode === "toggle" && selectedItem ? selectedItem.label : null}
{mode === "default" && triggerLabel ? triggerLabel : null}
<>
{triggerIcon}
{mode === "toggle" && selectedItem ? (
<span className="truncate">{selectedItem.label}</span>
) : null}
{mode === "default" && triggerLabel ? (
<span className="truncate">{triggerLabel}</span>
) : null}
</>
</Popover.Button>
<div
@@ -164,9 +172,9 @@ export const PopoverMenu = ({
variant="secondary"
disabled={item.disabled}
>
<div className="flex items-center gap-2">
<div className="flex items-center gap-2 truncate">
{showIconsInList && item.icon}
{item.label}
<span className="truncate">{item.label}</span>
</div>
{mode === "toggle" && (
<input
@@ -180,8 +188,14 @@ export const PopoverMenu = ({
{showAddButton && onAdd && (
<Button
variant="secondary"
className="mb-0.5 mt-2 border-none bg-[#7B7B84] py-1 hover:bg-[#8c8c96]"
className={twMerge(
"mb-0.5 mt-2 border-none py-1",
disableAddButton
? "cursor-not-allowed bg-[#7B7B84]/50 opacity-50"
: "bg-[#7B7B84] hover:bg-[#8c8c96]",
)}
onClick={onAdd}
disabled={disableAddButton}
>
+ Add
</Button>
@@ -137,7 +137,7 @@ export const Tooltip = ({
transitionTimingFunction: "ease-out",
}}
className={twMerge(
"pointer-events-none absolute z-10 w-max rounded-lg bg-[#5F5F68] px-2.5 py-1.5 text-sm font-medium text-white shadow-xl",
"pointer-events-none absolute z-10 w-max rounded-lg bg-[#5F5F68] px-2.5 py-1.5 text-[13px] font-medium text-white shadow-xl",
className ? className : "",
)}
>
@@ -27,18 +27,18 @@ export const TopBar = ({ pageName }: Props) => {
return (
<>
<header className="fixed left-0 top-0 z-30 w-full border-b border-ui-panel-border bg-ui-background">
<header className="fixed left-0 top-0 z-[60] w-full border-b border-white/5 bg-ui-background">
<nav
className="mx-auto grid h-[64px] w-screen grid-cols-3 items-center justify-between p-3"
aria-label="Global"
className="mx-auto grid h-[56px] w-screen grid-cols-3 items-center justify-between px-3"
aria-label="navigation"
>
<div className="flex gap-4">
<a href="/" className="">
<span className="sr-only">Storyteller.ai</span>
<a href="/">
<span className="sr-only">ArtCraft</span>
<img
className="h-[39px] w-auto pb-[3px]"
src="/resources/images/Storyteller-Logo-1.png"
alt="Logo StoryTeller.ai"
className="h-[28px] w-auto"
src="/resources/images/artcraft-logo-3.png"
alt="Logo ArtCraft"
/>
</a>
{!isEditorPath(currentLocation) && (
@@ -52,7 +52,7 @@ export const TopBar = ({ pageName }: Props) => {
<SceneTitleInput pageName={pageName} />
</div>
<div className="flex justify-end gap-5">
<div className="flex justify-end gap-3.5">
<div className="flex gap-2">
<Button
variant="secondary"
+1
View File
@@ -33,4 +33,5 @@ export interface JobStatusResponse {
maybe_first_started_at: string;
requires_keepalive: boolean;
status: JobStatus;
progress_percentage: number;
}
@@ -24,7 +24,11 @@ import Queue from "~/pages/PageEnigma/Queue/Queue";
import { QueueNames } from "~/pages/PageEnigma/Queue/QueueNames";
import { fromEngineActions } from "~/pages/PageEnigma/Queue/fromEngineActions";
import { MediaItem } from "~/pages/PageEnigma/models";
import { editorState } from "../signals/engine";
import {
editorState,
cameraAspectRatio,
gridVisibility,
} from "../signals/engine";
import { SceneUtils } from "./helper";
import { VideoGeneration } from "./video_generation";
import { MouseControls } from "./keybinds_controls";
@@ -193,8 +197,6 @@ class Editor {
processingHasFailed: boolean;
stats: Stats;
cameras: Map<string, THREE.PerspectiveCamera> = new Map();
constructor() {
this.processingHasFailed = false;
console.log(
@@ -325,34 +327,6 @@ class Editor {
if (this.engineFrameBuffers.frameWorkerManager) {
this.engineFrameBuffers.frameWorkerManager.type = imageFormat;
}
// Initialize cameras from signals
cameras.value.forEach((camData) => {
const camera = new THREE.PerspectiveCamera(
camData.fov,
this.getRenderDimensions().aspectRatio,
0.01,
200,
);
camera.position.set(
camData.position.x,
camData.position.y,
camData.position.z,
);
camera.rotation.set(
camData.rotation.x,
camData.rotation.y,
camData.rotation.z,
);
this.cameras.set(camData.id, camera);
});
// Set initial camera
this.camera = this.cameras.get("main") || null;
this.render_camera = this.camera;
// Subscribe to camera changes
this.setupCameraSignalHandlers();
}
// Add helper method to convert focal length to FOV
@@ -361,99 +335,6 @@ class Editor {
return 2 * Math.atan(sensorHeight / (2 * focalLength)) * (180 / Math.PI);
}
setupCameraSignalHandlers() {
// Handle camera selection changes
selectedCameraId.subscribe((id) => {
const newCamera = this.cameras.get(id);
if (newCamera) {
this.camera = newCamera;
this.render_camera = newCamera;
// Update camera properties in signals
const camData = cameras.value.find((c) => c.id === id);
if (camData) {
// Convert focal length to FOV
const fov = this.focalLengthToFov(camData.focalLength);
newCamera.fov = fov;
newCamera.updateProjectionMatrix();
this.renderScene(); // Force a render update
}
}
});
// Handle camera updates
cameras.subscribe((newCameras) => {
newCameras.forEach((camData) => {
let camera = this.cameras.get(camData.id);
// Create new camera if it doesn't exist
if (!camera) {
const fov = this.focalLengthToFov(camData.focalLength);
camera = new THREE.PerspectiveCamera(
fov,
this.getRenderDimensions().aspectRatio,
0.01,
200,
);
this.cameras.set(camData.id, camera);
}
// Update camera properties
camera.position.set(
camData.position.x,
camData.position.y,
camData.position.z,
);
camera.rotation.set(
camData.rotation.x,
camData.rotation.y,
camData.rotation.z,
);
// Update FOV based on focal length
const fov = this.focalLengthToFov(camData.focalLength);
camera.fov = fov;
camera.updateProjectionMatrix();
// Force a render update if this is the active camera
if (camData.id === selectedCameraId.value) {
this.renderScene();
}
});
// Clean up deleted cameras
const currentIds = new Set(newCameras.map((c) => c.id));
for (const [id] of this.cameras.entries()) {
if (!currentIds.has(id) && id !== "main") {
this.cameras.delete(id);
}
}
});
}
// Update camera position in signals when it changes
updateCameraPosition() {
if (this.camera && selectedCameraId.value) {
const pos = this.camera.position;
const rot = this.camera.rotation;
const camData = cameras.value.find(
(c) => c.id === selectedCameraId.value,
);
if (camData) {
cameras.value = cameras.value.map((c) =>
c.id === selectedCameraId.value
? {
...c,
position: { x: pos.x, y: pos.y, z: pos.z },
rotation: { x: rot.x, y: rot.y, z: rot.z },
}
: c,
);
}
}
}
getRenderDimensions() {
switch (this.render_camera_aspect_ratio) {
case CameraAspectRatio.HORIZONTAL_16_9: {
@@ -586,24 +467,52 @@ class Editor {
const width = this.container.offsetWidth;
const height = this.container.offsetHeight;
// Sets up camera and base position.
this.camera = new THREE.PerspectiveCamera(70, width / height, 0.01, 200);
this.camera.position.z = 2.5;
this.camera.position.y = 2.5;
this.camera.position.x = -2.5;
this.camera.lookAt(0, 0, 0);
// Sets up camera and base position using camera configurations from camera.ts
const mainCameraConfig = cameras.value.find((cam) => cam.id === "main");
if (mainCameraConfig) {
this.camera = new THREE.PerspectiveCamera(
this.focalLengthToFov(mainCameraConfig.focalLength),
width / height,
0.01,
200,
);
this.camera.position.set(
mainCameraConfig.position.x,
mainCameraConfig.position.y,
mainCameraConfig.position.z,
);
this.camera.lookAt(
mainCameraConfig.lookAt.x,
mainCameraConfig.lookAt.y,
mainCameraConfig.lookAt.z,
);
}
this.camera.layers.enable(0);
this.camera.layers.enable(1); // This camera does not see this layer
this.camera.layers.enable(1);
this.timeline.camera = this.camera;
this.render_camera = new THREE.PerspectiveCamera(
70,
width / height,
0.01,
200,
);
const otherCameras = cameras.value.filter((cam) => cam.id !== "main");
if (otherCameras.length > 0) {
const renderCameraConfig = otherCameras[0];
this.render_camera = new THREE.PerspectiveCamera(
this.focalLengthToFov(renderCameraConfig.focalLength),
width / height,
0.01,
200,
);
this.render_camera.position.set(
renderCameraConfig.position.x,
renderCameraConfig.position.y,
renderCameraConfig.position.z,
);
this.render_camera.lookAt(
renderCameraConfig.lookAt.x,
renderCameraConfig.lookAt.y,
renderCameraConfig.lookAt.z,
);
}
this.render_camera.layers.disable(1); // This camera does not see this layer );
@@ -784,15 +693,134 @@ class Editor {
document.body.appendChild(this.stats.dom);
}
// Captures the scene without the grid
public snapShotOfCurrentFrame(shouldDownload: boolean = true) {
const snapshot = this.renderer?.domElement.toDataURL("image/png");
if (!snapshot) {
console.error("Error: Snapshot could not be created.");
if (!this.renderer?.domElement || !this.camera) {
console.error("Error: Renderer or camera not available.");
return null;
}
const currentAspectRatio = cameraAspectRatio.value;
// Store grid visibility state and hide grid
const wasGridVisible = gridVisibility.value;
gridVisibility.value = false;
// Store and hide transform controls
const wasControlVisible = this.control?.visible ?? false;
if (this.control) {
this.control.visible = false;
}
// Store and disable outline pass
const wasOutlineEnabled = this.outlinePass?.enabled ?? false;
if (this.outlinePass) {
this.outlinePass.enabled = false;
}
// High quality dimensions for each aspect ratio
let targetWidth: number;
let targetHeight: number;
let aspectRatio: number;
switch (currentAspectRatio) {
case CameraAspectRatio.HORIZONTAL_16_9:
targetWidth = 1280;
targetHeight = 720;
aspectRatio = 16 / 9;
break;
case CameraAspectRatio.VERTICAL_9_16:
targetWidth = 720;
targetHeight = 1280;
aspectRatio = 9 / 16;
break;
case CameraAspectRatio.HORIZONTAL_3_2:
targetWidth = 1536;
targetHeight = 1024;
aspectRatio = 3 / 2;
break;
case CameraAspectRatio.VERTICAL_2_3:
targetWidth = 1024;
targetHeight = 1536;
aspectRatio = 2 / 3;
break;
case CameraAspectRatio.SQUARE_1_1:
default:
targetWidth = 1024;
targetHeight = 1024;
aspectRatio = 1;
break;
}
// Store original renderer and camera state
const originalWidth = this.renderer.domElement.width;
const originalHeight = this.renderer.domElement.height;
const originalPixelRatio = this.renderer.getPixelRatio();
const originalCameraAspect = this.camera.aspect;
const originalRenderCameraAspect =
this.render_camera?.aspect || originalCameraAspect;
// Temporarily set renderer to high resolution
this.renderer.setSize(targetWidth, targetHeight, false);
this.renderer.setPixelRatio(1);
// Update camera for the new aspect ratio
this.camera.aspect = aspectRatio;
this.camera.updateProjectionMatrix();
// If using render camera, update it too
if (this.render_camera) {
this.render_camera.aspect = aspectRatio;
this.render_camera.updateProjectionMatrix();
}
// Re-render the scene at high resolution
if (this.composer) {
this.composer.setSize(targetWidth, targetHeight);
this.composer.render();
} else {
this.renderer.render(this.activeScene.scene, this.camera);
}
// Get the high resolution snapshot
const snapshot = this.renderer.domElement.toDataURL("image/png", 1.0);
const base64Snapshot = snapshot.split(",")[1];
// Restore original camera aspect
this.camera.aspect = originalCameraAspect;
this.camera.updateProjectionMatrix();
// Restore render camera if it exists
if (this.render_camera) {
this.render_camera.aspect = originalRenderCameraAspect;
this.render_camera.updateProjectionMatrix();
}
// Restore original renderer size and pixel ratio
this.renderer.setSize(originalWidth, originalHeight, false);
this.renderer.setPixelRatio(originalPixelRatio);
// Re-render at original resolution
if (this.composer) {
this.composer.setSize(originalWidth, originalHeight);
this.composer.render();
} else {
this.renderer.render(this.activeScene.scene, this.camera);
}
// Restore grid visibility
gridVisibility.value = wasGridVisible;
// Restore transform controls visibility
if (this.control) {
this.control.visible = wasControlVisible;
}
// Restore outline pass
if (this.outlinePass) {
this.outlinePass.enabled = wasOutlineEnabled;
}
if (shouldDownload) {
const link = document.createElement("a");
link.download = "scene-snapshot.png";
@@ -807,7 +835,7 @@ class Editor {
for (let i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
const uuid = crypto.randomUUID(); // Generate a new UUID
const uuid = crypto.randomUUID();
const file = new File([ab], `${uuid}.png`, { type: mimeString });
return { base64Snapshot, file };
@@ -70,7 +70,7 @@ class FreeCam extends EventDispatcher {
}
keydown(event: KeyboardEvent) {
if (event.altKey || !this.enabled) {
if (!this.enabled) {
return;
}
@@ -235,12 +235,18 @@ class FreeCam extends EventDispatcher {
const pos = this.object.position;
const rot = this.object.rotation;
// Calculate lookAt point based on camera's forward direction
const lookAtPoint = new Vector3(0, 0, -1);
lookAtPoint.applyQuaternion(this.object.quaternion);
lookAtPoint.add(pos);
cameras.value = cameras.value.map((cam) =>
cam.id === selectedCameraId.value
? {
...cam,
position: { x: pos.x, y: pos.y, z: pos.z },
rotation: { x: rot.x, y: rot.y, z: rot.z },
lookAt: { x: lookAtPoint.x, y: lookAtPoint.y, z: lookAtPoint.z },
}
: cam,
);
@@ -5,6 +5,7 @@ import {
outlinerState,
showObjectPanel,
assetModalVisible,
assetModalVisibleDuringDrag,
} from "../signals";
import {
OrbitControls,
@@ -16,7 +17,11 @@ import { SceneManager, SceneObject } from "./scene_manager_api";
import { FreeCam } from "./free_cam";
import Stats from "three/examples/jsm/libs/stats.module.js";
import { FKHelper } from "./KinHelpers/FKHelper";
import { selectedMode } from "../signals/selectedMode";
import {
selectedMode,
poseMode,
showPoseControls,
} from "../signals/selectedMode";
export enum KinMode {
FK,
@@ -121,6 +126,10 @@ export class MouseControls {
scene: this.scene,
onDragChange: this.onFKControlsDragging.bind(this),
});
// Expose handlePoseModeChange to window
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(window as any).__mouseControls = this;
}
onFKControlsDragging(dragging: boolean) {
@@ -184,7 +193,11 @@ export class MouseControls {
this.setSelected(this.selected);
this.publishSelect();
// this.editor.update_properties()
if (currentObject.userData.isCharacter) {
showPoseControls.value = true;
}
// Normal selection behavior
if (currentObject.userData["locked"] !== true && this.control) {
this.scene.add(this.control);
this.control.attach(currentObject);
@@ -234,6 +247,9 @@ export class MouseControls {
this.fkHelper.clear();
this.kinMode = KinMode.NONE;
this.reattachTransformControls();
if (poseMode.value === "pose") {
poseMode.value = "select";
}
console.log("FK mode off");
return;
}
@@ -254,6 +270,9 @@ export class MouseControls {
this.hideTransformControls();
this.kinMode = KinMode.FK;
this.fkHelper.setTarget(this.selected[0]);
if (poseMode.value === "select") {
poseMode.value = "pose";
}
console.log("FK mode on");
return;
}
@@ -293,50 +312,64 @@ export class MouseControls {
return;
} else if (event.key === "k") {
this.toggleFKMode();
return;
} else if (event.key === "b") {
// Open asset modal
event.preventDefault();
assetModalVisible.value = true;
assetModalVisibleDuringDrag.value = true;
return;
}
if ((event.ctrlKey || event.metaKey) && !this.isProcessing) {
if (event.key === "Z" || event.key === "z") {
if (event.shiftKey) {
// redo
this.isProcessing = true;
await this.sceneManager?.redo();
this.isProcessing = false;
} else {
// undo
this.isProcessing = true;
await this.sceneManager?.undo();
this.isProcessing = false;
}
return;
// redo
this.isProcessing = true;
await this.sceneManager?.redo();
this.isProcessing = false;
} else if (event.key === "c") {
// redo
this.isProcessing = true;
await this.sceneManager?.copy();
this.isProcessing = false;
return;
} else if (event.key === "v") {
// redo
this.isProcessing = true;
await this.sceneManager?.paste();
this.isProcessing = false;
return;
} else if (event.key === "0") {
// Stats Menu
this.enable_stats();
}
}
// Prevent browser shortcuts for Alt combinations
if (
event.altKey &&
(event.key === "Alt" || event.key.toLowerCase() === "d")
) {
event.preventDefault();
event.stopPropagation();
}
if (event.shiftKey) {
this.cameraViewControls.movementSpeed = 4;
} else if (event.altKey) {
this.cameraViewControls.movementSpeed = 0.1;
} else {
this.cameraViewControls.movementSpeed = 1.15;
}
if (event.key === "Escape") {
if (poseMode.value === "pose") {
this.toggleFKMode();
return;
} else if (this.selected && this.selected.length > 0) {
this.removeTransformControls();
hideObjectPanel();
showPoseControls.value = false;
}
}
}
// Sets new mouse location usually used in raycasts.
@@ -440,7 +473,8 @@ export class MouseControls {
this.selected = [];
this.setSelected(this.selected);
this.removeTransformControls();
//hideObjectPanel();
hideObjectPanel();
showPoseControls.value = false;
}
if (this.sceneManager) {
@@ -11,6 +11,8 @@ import {
} from "../signals";
import Editor from "./editor";
import { getArtStyle } from "~/enums";
import { cameras, selectedCameraId, Camera } from "../signals/camera";
export type EditorInitializeConfig = {
sceneToken: string;
};
@@ -98,6 +100,16 @@ export class SaveManager {
);
const timeline_json = await proxyTimeline.saveToJson();
// Save all cameras data
const camerasData = cameras.value.map((cam: Camera) => ({
id: cam.id,
label: cam.label,
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}));
const save_data = {
version: this.editor.version,
scene: scene_json,
@@ -108,6 +120,9 @@ export class SaveManager {
position: this.editor.camera?.position,
rotation: this.editor.camera?.rotation,
},
// Add cameras array and selected camera
cameras: camerasData,
selectedCameraId: selectedCameraId.value,
};
// take json scene and figure out checksum
const jsonString = JSON.stringify(save_data);
@@ -146,6 +161,16 @@ export class SaveManager {
);
const timeline_json = await proxyTimeline.saveToJson();
// Save all cameras data
const camerasData = cameras.value.map((cam: Camera) => ({
id: cam.id,
label: cam.label,
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}));
const save_data = {
version: this.editor.version,
scene: scene_json,
@@ -156,6 +181,9 @@ export class SaveManager {
position: this.editor.camera?.position,
rotation: this.editor.camera?.rotation,
},
// Add cameras array and selected camera
cameras: camerasData,
selectedCameraId: selectedCameraId.value,
};
// TODO turn scene information into and object ...
@@ -218,6 +246,23 @@ export class SaveManager {
this.editor.camera.rotation.copy(camera_rotation);
}
// Restore cameras data
if (scene_json.cameras) {
cameras.value = scene_json.cameras.map((cam: Camera) => ({
id: cam.id,
label: cam.label,
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}));
}
// Restore selected camera
if (scene_json.selectedCameraId) {
selectedCameraId.value = scene_json.selectedCameraId;
}
// For Remixing Scenes.
// this calls the signal function to propagate the data to the UI
restoreSceneGenerationMetadata(scene_json);
@@ -17,6 +17,7 @@ import { GetCdnOrigin } from "~/api/GetCdnOrigin";
import { GetFrontendEnvironment } from "~/Classes/GetFrontendEnvironment";
import { gridVisibility } from "../signals/engine";
import { InfiniteGridHelper } from "./InfiniteGridHelper";
import { cameras, selectedCameraId } from "../signals/camera";
class Scene {
name: string;
@@ -86,7 +87,7 @@ class Scene {
this._createGrid();
this._create_base_lighting();
this._create_skybox();
this._create_camera_obj();
// this._create_camera_obj();
this.helper = new MMDAnimationHelper({ afterglow: 0.0 });
this.scene.userData["helper"] = this.helper;
@@ -110,7 +111,7 @@ class Scene {
this._createGrid();
this._create_base_lighting();
this._create_skybox();
this._create_camera_obj();
// this._create_camera_obj();
}
async instantiate(
@@ -370,31 +371,45 @@ class Scene {
}
async _create_camera_obj() {
const camera_position = new THREE.Vector3(0, 0.6, 1.5);
cameras.value.forEach((cameraConfig) => {
const camera_position = new THREE.Vector3(
cameraConfig.position.x,
cameraConfig.position.y,
cameraConfig.position.z,
);
// TODO(bt,2025-02-10): Make the local dev assets more configurable or seedable.
const camera_id = GetFrontendEnvironment().getIsLocalDev()
? "m_4djqzq89mbn6sdbbqm25ks6drhr55k" // Local development
: "m_cxh4asqhapdz10j880755dg4yevshb"; // Production
// TODO(bt,2025-02-10): Make the local dev assets more configurable or seedable.
const camera_id = GetFrontendEnvironment().getIsLocalDev()
? "m_cxh4asqhapdz10j880755dg4yevshb" // Local development
: "m_cxh4asqhapdz10j880755dg4yevshb"; // Production
const camera_obj = await this.loadGlbWithPlaceholder(
camera_id,
"Camera",
false,
camera_position,
);
camera_obj.userData["name"] = this.camera_name;
camera_obj.name = this.camera_name;
camera_obj.position.set(
camera_position.x,
camera_position.y,
camera_position.z,
);
camera_obj.layers.set(1);
camera_obj.children.forEach((child) => {
child.layers.set(1);
this.loadGlbWithPlaceholder(
camera_id,
"Camera",
false,
camera_position,
).then((camera_obj) => {
camera_obj.userData["name"] = cameraConfig.label;
camera_obj.name = cameraConfig.label;
camera_obj.position.set(
camera_position.x,
camera_position.y,
camera_position.z,
);
camera_obj.layers.set(1);
camera_obj.children.forEach((child) => {
child.layers.set(1);
});
this.scene.add(camera_obj);
if (cameraConfig.id === selectedCameraId.value) {
camera_obj.visible = false;
camera_obj.layers.disableAll(); // Make the active camera not touchable
} else {
camera_obj.visible = true;
}
});
});
this.scene.add(camera_obj);
}
renderMode(enabled: boolean = true) {
@@ -1,18 +1,16 @@
import React, { useEffect } from "react";
import { useSignals } from "@preact/signals-react/runtime";
import { LoadingDots, Toaster, TopBar } from "~/components";
import { LoadingDots, TopBar } from "~/components";
import { Controls3D } from "./comps/Controls3D";
import { ControlsTopButtons } from "./comps/ControlsTopButtons";
import { ControlPanelSceneObject } from "./comps/ControlPanelSceneObject";
import { PreviewEngineCamera } from "./comps/PreviewEngineCamera";
import { PreviewFrameImage } from "./comps/PreviewFrameImage";
import { pageHeight, pageWidth } from "~/signals";
import { PoseModeSelector } from "./comps/PoseModeSelector";
import {
timelineHeight,
sidePanelWidth,
sidePanelVisible,
dndSidePanelWidth,
dndTimelineHeight,
editorLoader,
cameraAspectRatio,
@@ -24,11 +22,13 @@ import { Outliner } from "./comps/Outliner";
import { CameraAspectRatio } from "./enums";
import { PromptBox } from "./comps/PromptBox";
import { OnboardingHelper } from "./comps/OnboardingHelper";
import { focalLengthDragging } from "./signals/camera";
import { FocalLengthDisplay } from "./comps/FocalLengthDisplay/FocalLengthDisplay";
// import { api } from "@storyteller/api";
// import { AnalyticsApi } from "@storyteller/api";
export const PageEditor = () => {
useSignals();
//console.log(api());
//To prevent the click event from propagating to the canvas: TODO: HANDLE THIS BETTER?
const handleOverlayClick = (event: React.MouseEvent<HTMLDivElement>) => {
event.stopPropagation();
@@ -42,20 +42,13 @@ export const PageEditor = () => {
};
}, []);
const dndWidth =
dndSidePanelWidth.value > -1
? dndSidePanelWidth.value
: sidePanelWidth.value;
const width = sidePanelVisible.value
? pageWidth.value - dndWidth - 84
: pageWidth.value - 84;
const height =
dndTimelineHeight.value > -1
? pageHeight.value - dndTimelineHeight.value - 64
: pageHeight.value - timelineHeight.value - 64;
? pageHeight.value - dndTimelineHeight.value - 56
: pageHeight.value - timelineHeight.value - 56;
const getScale = () => {
const height = pageHeight.value - timelineHeight.value - 64;
const height = pageHeight.value - timelineHeight.value - 56;
const scaleHeight = height < 610 ? height / 610 : 1;
if (
@@ -97,19 +90,15 @@ export const PageEditor = () => {
>
<div className="relative w-full overflow-hidden bg-transparent">
<SceneContainer>
<Toaster />
<EditorCanvas />
<PreviewFrameImage />
</SceneContainer>
{/* Focal Length Display */}
{focalLengthDragging.value.isDragging && (
<div className="absolute left-1/2 top-16 -translate-x-1/2 transform">
<div className="glass rounded-xl px-6 py-3 text-center text-3xl font-bold text-white">
{focalLengthDragging.value.focalLength}mm
</div>
</div>
)}
<FocalLengthDisplay />
{/* Pose Mode Selector */}
<PoseModeSelector />
{/* Top controls */}
<div
@@ -6,9 +6,8 @@ import { useBackgroundLoadingMedia } from "~/hooks/useBackgroundLoadingMedia";
import { useQueueHandler } from "./hooks/useQueueHandler";
import { ErrorDialog, LoadingDots } from "~/components";
import { GenerateModals } from "~/pages/PageEnigma/comps/GenerateModals/GenerateModals";
import { Toaster } from "@storyteller/ui-toaster";
import { EditorLoadingBar } from "./comps/EditorLoadingBar";
import { Wizard } from "~/pages/PageEnigma/Wizard/Wizard";
import { useSignals } from "@preact/signals-react/runtime";
import { useEffect, useState } from "react";
@@ -16,6 +15,13 @@ import * as gpu from "detect-gpu";
import { UsersApi } from "~/Classes/ApiManager";
import { PageEnigmaComponent } from "~/pages/PageEnigma/PageEnigmaComponent";
import { TurnOnGpu } from "~/pages/PageEnigma/TurnOnGpu";
import { PrecisionSelector } from "./comps/PrecisionSelector/PrecisionSelector";
import {
precisionSelectedValue,
precisionSelectorMenuCoords,
precisionSelectorValues,
showPrecisionSelector,
} from "./signals/precisionSelectorMenu";
export const PageEnigma = ({ sceneToken }: { sceneToken?: string }) => {
useSignals();
@@ -71,7 +77,7 @@ export const PageEnigma = ({ sceneToken }: { sceneToken?: string }) => {
case "apple gpu (Apple GPU)":
// TODO(bt,2025-04-08): We may want to disable this heuristic.
// We're getting lack of hardware acceleration using Tauri on Mac and Linux.
isValid = true;
isValid = true;
break;
default:
break;
@@ -93,11 +99,18 @@ export const PageEnigma = ({ sceneToken }: { sceneToken?: string }) => {
<EngineProvider sceneToken={sceneToken}>
<PageEnigmaComponent />
<DragComponent />
<PrecisionSelector
showSignal={showPrecisionSelector}
coordSignal={precisionSelectorMenuCoords}
valuesSignal={precisionSelectorValues}
selectedValueSignal={precisionSelectedValue}
/>
<GenerateModals />
<ErrorDialog />
<Wizard />
</EngineProvider>
<EditorLoadingBar />
<Toaster offsetTop={70} offsetRight={12} />
</TrackProvider>
);
};
@@ -33,6 +33,8 @@ export enum toEngineActions {
CHANGE_CAMERA_ASPECT_RATIO = "change_camera_aspect_ratio",
// data=CamperaAspectRatio
CAMERA_CHANGED = "camera_changed",
UNMUTE = "unmute",
// data={version, type, group, object_uuid} if lipsync
// data={version, type, group} if global_audio
@@ -4,16 +4,16 @@ import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { PopoverItem } from "~/components/reusable/Popover";
import { Button, Input, Label, Tooltip } from "~/components";
import { SliderV2 } from "~/components/reusable/SliderV2/SliderV2";
import {
updateCamera,
focalLengthDragging,
} from "~/pages/PageEnigma/signals/camera";
import { focalLengthDragging } from "~/pages/PageEnigma/signals/camera";
import { useState, useEffect } from "react";
import { twMerge } from "tailwind-merge";
interface ExtendedPopoverItem extends PopoverItem {
id: string;
focalLength: number;
position: { x: number; y: number; z: number };
rotation: { x: number; y: number; z: number };
lookAt: { x: number; y: number; z: number };
}
interface CameraSettingsModalProps {
@@ -24,7 +24,7 @@ interface CameraSettingsModalProps {
onCameraFocalLengthChange: (id: string, value: number) => void;
onAddCamera: () => void;
selectedCameraId: string;
onSelectCamera: (id: string) => void;
handleCameraSelect: (selectedItem: PopoverItem) => void;
onDeleteCamera: (id: string) => void;
}
@@ -36,11 +36,20 @@ export const CameraSettingsModal = ({
onCameraFocalLengthChange,
onAddCamera,
selectedCameraId,
onSelectCamera,
handleCameraSelect,
onDeleteCamera,
}: CameraSettingsModalProps) => {
const selectedCamera = cameras.find((cam) => cam.id === selectedCameraId);
const [isDragging, setIsDragging] = useState(false);
const [tempCameraName, setTempCameraName] = useState<string>("");
const [editingCameraId, setEditingCameraId] = useState<string | null>(null);
useEffect(() => {
if (selectedCamera) {
setTempCameraName(selectedCamera.label);
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [selectedCamera?.id]);
useEffect(() => {
const handlePointerUp = () => {
@@ -67,16 +76,29 @@ export const CameraSettingsModal = ({
onCameraFocalLengthChange(id, value);
};
const handleNameBlur = () => {
if (selectedCamera && tempCameraName.trim() !== "") {
onCameraNameChange(selectedCamera.id, tempCameraName);
} else if (selectedCamera) {
setTempCameraName(selectedCamera.label);
}
setEditingCameraId(null);
};
const handleNameFocus = (id: string) => {
setEditingCameraId(id);
};
return (
<TransitionDialogue
isOpen={isOpen}
onClose={onClose}
className={twMerge(
"h-[500px] max-w-3xl transition-opacity duration-200",
isDragging ? "opacity-30 hover:opacity-20" : "opacity-100",
"h-[500px] max-w-3xl duration-200",
isDragging ? "opacity-10" : "opacity-100",
)}
backdropClassName={twMerge(
"transition-opacity duration-200",
"duration-200",
isDragging ? "opacity-0" : "opacity-100",
)}
childPadding={false}
@@ -99,14 +121,16 @@ export const CameraSettingsModal = ({
{cameras.map((camera) => (
<button
key={camera.id}
className={`w-full rounded-lg p-2 text-left transition-colors duration-100 hover:bg-[#63636B]/40 ${
className={`h-9 w-full rounded-lg p-2 text-left transition-colors duration-100 hover:bg-[#63636B]/40 ${
camera.id === selectedCameraId ? "bg-[#63636B]/40" : ""
}`}
onClick={() => onSelectCamera(camera.id)}
onClick={() => handleCameraSelect(camera)}
>
<div className="flex items-center gap-2.5 text-sm">
{camera.icon}
{camera.label}
{editingCameraId === camera.id
? tempCameraName
: camera.label}
</div>
</button>
))}
@@ -154,10 +178,11 @@ export const CameraSettingsModal = ({
<Input
id="camera-name"
type="text"
value={selectedCamera?.label || ""}
onChange={(e) =>
selectedCamera &&
onCameraNameChange(selectedCamera.id, e.target.value)
value={tempCameraName}
onChange={(e) => setTempCameraName(e.target.value)}
onBlur={handleNameBlur}
onFocus={() =>
selectedCamera && handleNameFocus(selectedCamera.id)
}
className="text-sm"
/>
@@ -22,6 +22,7 @@ import { sanitize } from "./utils/sanitize";
import { objectMismatch } from "~/pages/PageEnigma/comps/ControlPanelSceneObject/utils/objectMismatch";
import { XYZ } from "~/pages/PageEnigma/datastructures/common";
import { pageHeight } from "~/signals";
import { DraggablePrecisionMutator } from "./DraggablePrecisionMutator";
// TODO this will be useful later to fix the bug on leading zeros
// const formatNumber = (input: string): number => {
@@ -147,6 +148,15 @@ export const ControlPanelSceneObject = () => {
setLocalRotation(xyz);
};
const handleUniformScaleChange = (scale: number) => {
const updatedScaleValues: Record<string, string> = {};
updatedScaleValues.x = (parseFloat(localScale.x) + scale).toString();
updatedScaleValues.y = (parseFloat(localScale.y) + scale).toString();
updatedScaleValues.z = (parseFloat(localScale.z) + scale).toString();
handleScaleChange(updatedScaleValues);
}
const handleScaleChange = (xyz: Record<string, string>) => {
if (isInvalid(xyz)) {
setLocalScale(xyz);
@@ -164,7 +174,7 @@ export const ControlPanelSceneObject = () => {
};
const getScale = () => {
const height = pageHeight.value - timelineHeight.value - 64;
const height = pageHeight.value - timelineHeight.value - 56;
return height < 620 ? height / 620 : 1;
};
@@ -255,7 +265,9 @@ export const ControlPanelSceneObject = () => {
</div>
<div className="mb-1 flex flex-col gap-1">
<H5>Scale</H5>
<DraggablePrecisionMutator onChange={handleUniformScaleChange}>
<h5>Scale</h5>
</DraggablePrecisionMutator>
<InputVector
x={localScale.x.toString()}
y={localScale.y.toString()}
@@ -0,0 +1,101 @@
import { useCallback, useRef } from "react";
import { twMerge } from "tailwind-merge";
import { precisionSelectedValue, precisionSelectorMenuCoords, precisionSelectorValues, showPrecisionSelector } from "../../signals/precisionSelectorMenu";
const DEFAULT_PRECISIONS = [
10,
1,
0.1,
0.01,
0.001
]
export const DraggablePrecisionMutator = ({
precisions = DEFAULT_PRECISIONS,
className = "",
initialValue = 0,
onChange = () => { },
onCommit = () => { },
children
}: {
precisions?: number[],
className?: string,
initialValue?: number,
onChange?: (value: number) => void,
onCommit?: (value: number) => void,
children?: React.ReactNode
}) => {
const classes = twMerge(["cursor-ew-resize", className]);
const divRef = useRef<HTMLDivElement>(null);
const isDraggingRef = useRef(false);
const dragStartCoordsRef = useRef<{ x: number, y: number } | null>(null);
const localAdjustedValueRef = useRef(initialValue);
const handleMouseMove = useCallback((event: MouseEvent) => {
// Ignore mouse movement if we're not even dragging
// or if the menu is still open (we're selecting the precision)
if (!isDraggingRef.current || showPrecisionSelector.value) {
return;
}
event.stopPropagation();
// Record the initial drag coordinates if they haven't been set yet
if (!dragStartCoordsRef.current) {
dragStartCoordsRef.current = {
x: event.clientX,
y: event.clientY
};
return;
}
// Check how far the mouse has moved, and calculate the updated change
const distX = event.x - dragStartCoordsRef.current.x;
const scaledDistX = distX / 10;
const scaledPrecision = Math.round(scaledDistX) * precisionSelectedValue.value;
localAdjustedValueRef.current = initialValue + scaledPrecision;
// Update the listeners
onChange(localAdjustedValueRef.current);
}, [initialValue, onChange]);
const handleMouseUp = useCallback((event: MouseEvent) => {
event.stopPropagation();
showPrecisionSelector.value = false;
isDraggingRef.current = false;
dragStartCoordsRef.current = null;
// clean up the mouse listeners
document.removeEventListener("mousemove", handleMouseMove);
document.removeEventListener("mouseup", handleMouseUp);
// Call back the commit listener
onCommit(localAdjustedValueRef.current);
}, [handleMouseMove, onCommit]);
const handleMouseDown = useCallback((event: React.MouseEvent) => {
if (event.button !== 0) {
return;
}
event.preventDefault();
event.stopPropagation();
precisionSelectorMenuCoords.value = {
x: event.clientX,
y: event.clientY
};
precisionSelectorValues.value = precisions;
showPrecisionSelector.value = true;
isDraggingRef.current = true;
document.addEventListener("mousemove", handleMouseMove);
document.addEventListener("mouseup", handleMouseUp);
}, [handleMouseMove, handleMouseUp, precisions]);
return (
<div className={classes} ref={divRef} onMouseDown={handleMouseDown}>
{children}
</div>
)
}
@@ -0,0 +1,25 @@
import { Transition } from "@headlessui/react";
import { useSignals } from "@preact/signals-react/runtime";
import { focalLengthDragging } from "~/pages/PageEnigma/signals/camera";
export const FocalLengthDisplay = () => {
useSignals();
return (
<Transition
show={focalLengthDragging.value.isDragging}
enter="transition-opacity duration-200"
enterFrom="opacity-0"
enterTo="opacity-100"
leave="transition-opacity duration-200"
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div className="absolute left-1/2 top-16 z-10 -translate-x-1/2 transform">
<div className="glass rounded-xl px-5 py-2.5 text-center text-2xl font-bold text-white">
{focalLengthDragging.value.focalLength}mm
</div>
</div>
</Transition>
);
};
@@ -4,11 +4,185 @@ import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { PopoverMenu } from "~/components/reusable/Popover/Popover";
import { CompletedCard } from "./CompletedCard";
import { InProgressCard } from "./InProgressCard";
import { H3, P, Tooltip } from "~/components";
import { activeImageGenerationJobs, userMovies } from "~/signals";
import { H3, Tooltip } from "~/components";
import { useEffect, useState, useRef, useCallback } from "react";
import { JobsApi } from "~/Classes/ApiManager/JobsApi";
import { toast } from "@storyteller/ui-toaster";
import { ActiveJob } from "~/pages/PageEnigma/models";
// TODO ensure we de-dupe all this extra code.
interface CompletedItem {
token: string;
maybe_title: string;
public_bucket_path: string;
updated_at: string;
}
// {
// "job_token": "jinf_421nnr6rj8wy2sj0s11v9fns32s",
// "request": {
// "inference_category": "image_generation",
// "maybe_model_type": "image_gen_api",
// "maybe_model_token": null,
// "maybe_model_title": "Image Generation",
// "maybe_raw_inference_text": null,
// "maybe_style_name": null,
// "maybe_live_portrait_details": null,
// "maybe_lipsync_details": null
// },
// "status": {
// "status": "complete_success",
// "maybe_extra_status_description": null,
// "maybe_assigned_worker": "inference-job-4o-image-848c55d5dd-555kn",
// "maybe_assigned_cluster": "storyteller-eks-1",
// "maybe_first_started_at": "2025-04-15T04:41:01Z",
// "maybe_current_execution_duration_seconds": null,
// "attempt_count": 1,
// "requires_keepalive": false,
// "maybe_failure_category": null,
// "progress_percentage": 100
// },
// "maybe_result": {
// "entity_type": "media_file",
// "entity_token": "m_ytrjp1fcd5tb43ebcs3ewhq6y05tfb",
// "maybe_public_bucket_media_path": "/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "media_links": {
// "cdn_url": "https://cdn-2.fakeyou.com/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "maybe_thumbnail_template": "https://cdn-2.fakeyou.com/cdn-cgi/image/width={WIDTH},quality=95/media/p/t/7/y/g/pt7yg2xfv4ynemssfv1mwb7yxe26e2nb/image_pt7yg2xfv4ynemssfv1mwb7yxe26e2nb.png",
// "maybe_video_previews": null
// },
// "maybe_successfully_completed_at": "2025-04-15T04:42:14Z"
// },
// "created_at": "2025-04-15T04:41:01Z",
// "updated_at": "2025-04-15T04:42:14Z"
// }
export function Activity() {
useSignals();
const [completedItems, setCompletedItems] = useState<CompletedItem[]>([]);
const [jobs, setJobs] = useState<ActiveJob[]>([]);
const processedJobs = useRef(new Set<string>());
const isFirstLoad = useRef(true);
const toastTimeoutRef = useRef<NodeJS.Timeout | null>(null);
// Debounced toast function
const showToast = useCallback((items: CompletedItem[]) => {
if (toastTimeoutRef.current) {
clearTimeout(toastTimeoutRef.current);
}
toastTimeoutRef.current = setTimeout(() => {
const count = items.length;
toast.success(
count === 1
? `${items[0].maybe_title} completed successfully`
: `${count} images completed successfully`,
);
toastTimeoutRef.current = null;
}, 100); // Small delay to catch potential double calls
}, []);
useEffect(() => {
let mounted = true;
async function fetchJobs() {
try {
const jobsApi = new JobsApi();
const response = await jobsApi.ListRecentJobs();
if (!mounted || !response.success || !response.data) return;
// Handle active jobs
const activeJobs = response.data
.filter((job) => {
const status = job.status?.status?.toLowerCase();
return (
status && status !== "complete_success" && status !== "failed"
);
})
.map(
(job) =>
({
job_token: job.job_token,
request: {
maybe_model_title:
job.request?.maybe_model_title || "Image Generation",
},
status: {
status: job.status?.status?.toUpperCase() || "",
progress_percentage: job.status?.progress_percentage || 0,
},
created_at: job.created_at || new Date().toISOString(),
updated_at: job.updated_at || new Date().toISOString(),
maybe_result: job.maybe_result || null,
}) as ActiveJob,
);
// Handle completed jobs
const newlyCompleted = response.data.filter(
(job) =>
job.status?.status?.toLowerCase() === "complete_success" &&
!processedJobs.current.has(job.job_token),
);
if (newlyCompleted.length > 0) {
// Add to processed set
newlyCompleted.forEach((job) =>
processedJobs.current.add(job.job_token),
);
// Create completed items
const newItems = newlyCompleted.map((job) => ({
token: job.job_token,
maybe_title: job.request?.maybe_model_title || "Image Generation",
public_bucket_path:
job.maybe_result?.maybe_public_bucket_media_path || "",
updated_at: job.updated_at || new Date().toISOString(),
}));
// Show toast only if not first load
if (!isFirstLoad.current) {
showToast(newItems);
}
// Update state
setCompletedItems((prev) => {
const combined = [...newItems, ...prev];
return combined
.filter(
(item, index, self) =>
index === self.findIndex((t) => t.token === item.token),
)
.sort(
(a, b) =>
new Date(b.updated_at).getTime() -
new Date(a.updated_at).getTime(),
)
.slice(0, 50);
});
}
setJobs(activeJobs);
isFirstLoad.current = false;
} catch (error) {
console.error("Failed to fetch jobs:", error);
}
}
// Initial fetch
fetchJobs();
// Set up polling
const interval = setInterval(fetchJobs, 5000);
// Cleanup
return () => {
mounted = false;
clearInterval(interval);
if (toastTimeoutRef.current) {
clearTimeout(toastTimeoutRef.current);
}
};
}, [showToast]);
return (
<Tooltip content="Activity" position="bottom" closeOnClick={true}>
@@ -20,26 +194,27 @@ export function Activity() {
align="end"
triggerIcon={
<div>
<FontAwesomeIcon icon={faBell} />
{activeImageGenerationJobs.value &&
activeImageGenerationJobs.value.length > 0 && (
<div className="bg-blue-500 absolute -right-1 -top-1 h-3 w-3 animate-pulse rounded-full" />
)}
{jobs.length > 0 ? (
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
) : (
<FontAwesomeIcon icon={faBell} />
)}
{jobs.length > 0 && (
<div className="absolute -right-1 -top-1 h-3 w-3 animate-pulse rounded-full bg-brand-primary-400" />
)}
</div>
}
>
<div className="max-h-[480px] overflow-y-auto">
{/* In progress */}
{activeImageGenerationJobs.value &&
activeImageGenerationJobs.value.length > 0 && (
<div>
{activeImageGenerationJobs.value.map((movieJob) => (
<InProgressCard key={movieJob.job_token} movie={movieJob} />
))}
</div>
)}
{jobs.length > 0 && (
<div>
{jobs.map((job) => (
<InProgressCard key={job.job_token} job={job} />
))}
</div>
)}
{!userMovies.value && !activeImageGenerationJobs.value ? (
{isFirstLoad.current ? (
<div className="flex h-48 w-full flex-col justify-center gap-4 p-4 text-center align-middle">
<FontAwesomeIcon
icon={faSpinnerThird}
@@ -49,21 +224,15 @@ export function Activity() {
/>
<H3 className="text-gray-300">Retrieving Activities</H3>
</div>
) : userMovies.value &&
userMovies.value.length === 0 &&
!activeImageGenerationJobs.value ? (
) : jobs.length === 0 && completedItems.length === 0 ? (
<div className="flex h-48 w-full flex-col justify-center gap-4 p-4 text-center align-middle">
<H3 className="text-gray-300">No activities yet</H3>
<P className="text-gray-400">
Try generating something new from the featured scenes
</P>
<h3 className="text-lg text-gray-300">No activities yet</h3>
</div>
) : (
<div>
{/* Completed */}
<div className="flex flex-col">
{userMovies.value?.map((movie) => (
<CompletedCard key={movie.token} movie={movie} />
{completedItems.map((item) => (
<CompletedCard key={item.token} job={item} />
))}
</div>
</div>
@@ -1,91 +1,84 @@
import { MediaInfo } from "~/pages/PageEnigma/models/movies";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faArrowDownToLine } from "@fortawesome/pro-solid-svg-icons";
import { useRef, useState } from "react";
import { BucketConfig } from "~/api/BucketConfig";
import { useState } from "react";
import dayjs from "dayjs";
import environmentVariables from "~/Classes/EnvironmentVariables";
import { useSignals } from "@preact/signals-react/runtime";
import { downloadFile } from "~/pages/PageEnigma/comps/GenerateModals/utils/downloadFile";
import { styleList } from "~/pages/PageEnigma/styleList";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { faDownToLine } from "@fortawesome/pro-solid-svg-icons";
import { LightboxModal } from "~/components/reusable/LightboxModal";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
export function getStyleName(typeInput: string) {
const foundStyle = styleList.find((style) => {
return style.type === typeInput;
});
return foundStyle ? foundStyle.label : "Unknown Style";
}
interface Props {
movie: MediaInfo;
setMovieId?: (id: string) => void;
job: {
token: string;
maybe_title: string;
updated_at: string;
public_bucket_path: string;
maybe_style_name?: string;
};
}
export function CompletedCard({ movie, setMovieId }: Props) {
useSignals();
const bucketConfig = useRef<BucketConfig>(new BucketConfig());
export function CompletedCard({ job }: Props) {
const [loadError, setLoadError] = useState(false);
const [isModalOpen, setIsModalOpen] = useState(false);
const cdnOrigin = GetCdnOrigin();
const downloadLink = `https://${cdnOrigin}${movie.public_bucket_path}`;
const mediaPath = job?.public_bucket_path || "";
const downloadLink = `${cdnOrigin}${mediaPath}`;
const imageUrl = downloadLink;
const fullImageUrl = downloadLink;
const imageUrl = bucketConfig.current.getCdnUrl(
movie.public_bucket_path + "-thumb.gif",
360,
20,
);
const styleName =
movie.maybe_style_name && getStyleName(movie.maybe_style_name);
const handleClick = () => {
setMovieId?.(movie.token);
};
if (!job) return null;
return (
<div
className="flex w-full items-center justify-between rounded-lg p-2 text-start transition-all duration-150 hover:cursor-pointer hover:bg-white/10"
onClick={handleClick}
>
<div className="flex gap-4">
<div className="flex aspect-square h-14 w-14 justify-center overflow-hidden rounded-lg border border-[#A9A9A9]/50 bg-black/60">
<img
src={
loadError ? "/resources/images/movie-placeholder.png" : imageUrl
}
className="h-full w-full object-cover"
alt={movie.maybe_title ?? "unknown"}
crossOrigin="anonymous"
onError={() => setLoadError(true)}
loading="lazy"
/>
</div>
<div className="flex flex-col justify-center gap-1">
<div className="font-medium">{movie.maybe_title || "Untitled"}</div>
<div>
{/* {styleName && (
<div className="text-sm text-white/60">{styleName}</div>
)} */}
<>
<div
className="flex w-full items-center justify-between rounded-lg p-2 text-start transition-all duration-150 hover:cursor-pointer hover:bg-white/10"
onClick={() => setIsModalOpen(true)}
>
<div className="flex gap-4">
<div className="flex aspect-square h-14 w-14 justify-center overflow-hidden rounded-lg border border-[#A9A9A9]/50 bg-black/60">
<img
src={
loadError ? "/resources/images/movie-placeholder.png" : imageUrl
}
className="h-full w-full object-cover"
alt={job.maybe_title ?? "unknown"}
crossOrigin="anonymous"
onError={() => setLoadError(true)}
loading="lazy"
/>
</div>
<div className="flex flex-col justify-center gap-1">
<div className="font-medium">{job.maybe_title || "Untitled"}</div>
<div className="text-sm text-white/60">
{dayjs(movie.updated_at).format("MMM D, YYYY HH:mm:ss")}
{dayjs(job.updated_at).format("MMM D, YYYY HH:mm:ss")}
</div>
</div>
</div>
</div>
{/* <div className="pr-5">
<button
onClick={(event) => {
event.preventDefault();
event.stopPropagation();
const title =
movie.maybe_title !== null ? movie.maybe_title : "Untitled";
downloadFile({ url: downloadLink, title });
}}
className="text-[15px] font-medium text-white/50 transition-all duration-150 hover:text-white/100"
<a
href={downloadLink}
download
onClick={(e) => e.stopPropagation()}
className="flex h-9 w-9 items-center justify-center text-lg text-white/60 transition-all hover:text-white"
>
<FontAwesomeIcon icon={faArrowDownToLine} className="mr-2" />
Download
</button>
</div> */}
</div>
<FontAwesomeIcon icon={faDownToLine} />
</a>
</div>
<LightboxModal
isOpen={isModalOpen}
onClose={() => setIsModalOpen(false)}
imageUrl={fullImageUrl}
imageAlt={job.maybe_title ?? "unknown"}
onImageError={() => setLoadError(true)}
title={job.maybe_title}
createdAt={job.updated_at}
downloadUrl={downloadLink}
additionalInfo={
job.maybe_style_name && (
<div className="text-sm text-white/60">
Style: {job.maybe_style_name}
</div>
)
}
/>
</>
);
}
@@ -1,22 +1,15 @@
import { useCallback } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faClose,
faCircleExclamation,
faSpinnerThird,
} from "@fortawesome/pro-solid-svg-icons";
import { faClose, faSpinnerThird } from "@fortawesome/pro-solid-svg-icons";
import { ToastTypes } from "~/enums";
import { addToast } from "~/signals";
import { JobState } from "~/pages/PageEnigma/enums";
import { ActiveJob } from "~/pages/PageEnigma/models";
import { Tooltip } from "~/components";
import { JobsApi } from "~/Classes/ApiManager";
import { PollRecentJobs } from "~/hooks/useActiveJobs/utilities";
interface Props {
movie: ActiveJob;
job: ActiveJob;
}
function getPercent(status: JobState, percentage: number) {
@@ -33,16 +26,16 @@ function getPercent(status: JobState, percentage: number) {
}
}
export function InProgressCard({ movie }: Props) {
export function InProgressCard({ job }: Props) {
const completePercent = getPercent(
movie.status.status as JobState,
movie.status.progress_percentage,
job.status.status as JobState,
job.status.progress_percentage,
);
const deleteJob = useCallback(async (movieJob: ActiveJob) => {
const deleteJob = useCallback(async (job: ActiveJob) => {
const jobsApi = new JobsApi();
const response = await jobsApi.DeleteJobByToken(movieJob.job_token);
const response = await jobsApi.DeleteJobByToken(job.job_token);
if (response.success) {
PollRecentJobs();
addToast(ToastTypes.SUCCESS, "File successfully deleted.");
@@ -58,32 +51,33 @@ export function InProgressCard({ movie }: Props) {
return (
<div className="flex w-full items-center justify-between rounded-lg p-2 text-start transition-all duration-150 hover:bg-white/10">
<div className="flex gap-4">
<div className="flex aspect-square h-14 w-14 items-center justify-center overflow-hidden rounded-lg border border-[#A9A9A9]/50 bg-black/60">
{(movie.status.status === JobState.STARTED ||
movie.status.status === JobState.PENDING) && (
<FontAwesomeIcon icon={faSpinnerThird} spin size={"lg"} />
<div className="flex aspect-square h-14 w-14 items-center justify-center rounded-lg border border-[#A9A9A9]/50 bg-black/60 text-white">
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
{/* {(job.status.status === JobState.STARTED ||
job.status.status === JobState.PENDING) && (
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
)}
{(movie.status.status === JobState.COMPLETE_FAILURE ||
movie.status.status === JobState.ATTEMPT_FAILED) && (
{(job.status.status === JobState.COMPLETE_FAILURE ||
job.status.status === JobState.ATTEMPT_FAILED) && (
<FontAwesomeIcon icon={faCircleExclamation} size={"lg"} />
)}
)} */}
</div>
<div className="flex flex-col justify-center gap-1">
<div className="font-medium">
{movie.request.maybe_model_title || "Untitled"}
{job.request.maybe_model_title || "Image Generation"}
</div>
<div className="text-sm capitalize text-white/60">
{movie.status.status.replaceAll("_", " ")}... {completePercent}%
{job.status.status.replaceAll("_", " ")}... {completePercent}%
</div>
</div>
</div>
{movie.status.status !== JobState.STARTED && (
{job.status.status !== JobState.STARTED && (
<div className="pr-5">
<Tooltip content="Cancel" position="left">
<button
onClick={() => deleteJob(movie)}
onClick={() => deleteJob(job)}
className="text-[15px] font-medium text-white/50 transition-all duration-150 hover:text-white/100"
>
<FontAwesomeIcon icon={faClose} />
@@ -24,4 +24,28 @@ export const downloadFile = ({
.catch(() => addToast(ToastTypes.ERROR, "Could not download file."));
};
export const downloadFileImage = ({
url,
title,
}: {
url: string;
title: string;
}) => {
// console.log(url);
fetch(url)
.then((resp) => resp.blob())
.then((blob) => {
const blobUrl = window.URL.createObjectURL(blob);
const a = document.createElement("a");
a.style.display = "none";
a.href = blobUrl;
a.download = `${title}.png`;
document.body.appendChild(a);
a.click();
window.URL.revokeObjectURL(url);
})
.catch(() => addToast(ToastTypes.ERROR, "Could not download file."));
};
//check
@@ -1,15 +1,14 @@
import { TransitionDialogue } from "~/components/reusable/TransitionDialogue";
import { faCheck, faXmark } from "@fortawesome/pro-solid-svg-icons";
import { faCheck } from "@fortawesome/pro-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { Button, LoadingSpinner, CloseButton } from "~/components";
import { TabSelector } from "~/components/reusable/TabSelector";
import { useState, useEffect, Fragment } from "react";
import React, { useState, useEffect, useCallback, useMemo } from "react";
import { LibraryModalApi } from "./LibraryModalApi";
import { FilterMediaClasses } from "../../../../enums";
import { BucketConfig } from "../../../../api/BucketConfig";
import { twMerge } from "tailwind-merge";
import { createPortal } from "react-dom";
import { Transition } from "@headlessui/react";
import { LightboxModal } from "~/components/reusable/LightboxModal";
import { GetCdnOrigin } from "~/api/GetCdnOrigin";
export interface LibraryItem {
id: string;
@@ -38,323 +37,292 @@ interface LibraryModalProps {
onTabChange: (tabId: string) => void;
}
export const LibraryModal = ({
isOpen,
onClose,
mode = "view",
selectedItemIds = [],
onSelectItem,
maxSelections = 4,
onUseSelected,
tabs,
activeTab,
onTabChange,
}: LibraryModalProps) => {
const [groupedItems, setGroupedItems] = useState<GroupedItems>({});
const [loading, setLoading] = useState(false);
const [lightboxImage, setLightboxImage] = useState<LibraryItem | null>(null);
const [isLightboxVisible, setIsLightboxVisible] = useState(false);
const [failedImageUrls] = useState<Set<string>>(new Set());
const bucketConfig = new BucketConfig();
const api = new LibraryModalApi();
const formatDate = (date: string) => {
const d = new Date(date);
return d.toLocaleDateString("en-US", {
weekday: "short",
month: "short",
day: "numeric",
});
};
const groupItemsByDate = (items: LibraryItem[]) => {
const grouped = items.reduce((acc: GroupedItems, item) => {
const dateKey = formatDate(item.createdAt);
if (!acc[dateKey]) {
acc[dateKey] = [];
}
acc[dateKey].push(item);
return acc;
}, {});
// Sort dates in descending order
return Object.fromEntries(
Object.entries(grouped).sort(
(a, b) =>
new Date(b[1][0].createdAt).getTime() -
new Date(a[1][0].createdAt).getTime(),
),
export const LibraryModal = React.memo(
({
isOpen,
onClose,
mode = "view",
selectedItemIds = [],
onSelectItem,
maxSelections = 4,
onUseSelected,
tabs,
activeTab,
onTabChange,
}: LibraryModalProps) => {
const [groupedItems, setGroupedItems] = useState<GroupedItems>({});
const [loading, setLoading] = useState(false);
const [lightboxImage, setLightboxImage] = useState<LibraryItem | null>(
null,
);
};
const [isLightboxVisible, setIsLightboxVisible] = useState(false);
const [failedImageUrls] = useState<Set<string>>(new Set());
const cdnOrigin = GetCdnOrigin();
const imageUrl = lightboxImage?.fullImage || "";
const api = useMemo(() => new LibraryModalApi(), []);
const handleImageError = (url: string) => {
failedImageUrls.add(url);
};
const formatDate = useCallback((date: string) => {
const d = new Date(date);
return d.toLocaleDateString("en-US", {
weekday: "short",
month: "short",
day: "numeric",
});
}, []);
const getImageUrl = (path: string | undefined | null) => {
if (!path) return null;
const url = bucketConfig.getGcsUrl(path);
return failedImageUrls.has(url) ? null : url;
};
const groupItemsByDate = useCallback(
(items: LibraryItem[]) => {
const grouped = items.reduce((acc: GroupedItems, item) => {
const dateKey = formatDate(item.createdAt);
if (!acc[dateKey]) {
acc[dateKey] = [];
}
acc[dateKey].push(item);
return acc;
}, {});
useEffect(() => {
const loadItems = async () => {
setLoading(true);
try {
const response = await api.listUserMediaFiles({
filter_media_classes: [FilterMediaClasses.IMAGE],
});
if (response.success && response.data) {
const newItems = response.data.map((item) => ({
id: item.token,
label: item.maybe_title || "Untitled",
thumbnail: getImageUrl(
item.cover_image.maybe_cover_image_public_bucket_path ||
item.public_bucket_path,
),
fullImage: getImageUrl(item.public_bucket_path),
createdAt: item.created_at,
}));
setGroupedItems(groupItemsByDate(newItems));
// Sort dates in descending order
return Object.fromEntries(
Object.entries(grouped).sort(
(a, b) =>
new Date(b[1][0].createdAt).getTime() -
new Date(a[1][0].createdAt).getTime(),
),
);
},
[formatDate],
);
const handleImageError = useCallback(
(url: string) => {
failedImageUrls.add(url);
},
[failedImageUrls],
);
const getImageUrl = useCallback(
(path: string | undefined | null) => {
if (!path) return null;
const normalizedPath = path.startsWith("/") ? path : `/${path}`;
const url = `${cdnOrigin}${normalizedPath}`;
return failedImageUrls.has(url) ? null : url;
},
[cdnOrigin, failedImageUrls],
);
useEffect(() => {
const loadItems = async () => {
setLoading(true);
try {
const response = await api.listUserMediaFiles({
filter_media_classes: [FilterMediaClasses.IMAGE],
});
if (response.success && response.data) {
const newItems = response.data.map((item) => ({
id: item.token,
label: item.maybe_title || "Image Generation",
thumbnail: getImageUrl(item.public_bucket_path),
fullImage: getImageUrl(item.public_bucket_path),
createdAt: item.created_at,
}));
setGroupedItems(groupItemsByDate(newItems));
}
} catch (error) {
console.error("Failed to fetch library items:", error);
}
} catch (error) {
console.error("Failed to fetch library items:", error);
setLoading(false);
};
if (isOpen) {
loadItems();
}
setLoading(false);
};
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [activeTab, isOpen]);
if (isOpen) {
loadItems();
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [activeTab, isOpen]);
// useEffect(() => {
// console.log("LibraryModal render caused by:", {
// isOpen,
// activeTab,
// selectedItemIds,
// });
// }, [isOpen, activeTab, selectedItemIds]);
const handleItemClick = (item: LibraryItem) => {
if (mode === "select" && onSelectItem) {
onSelectItem(item.id);
} else {
setLightboxImage(item);
setIsLightboxVisible(true);
}
};
const handleItemClick = useCallback(
(item: LibraryItem) => {
if (mode === "select" && onSelectItem) {
onSelectItem(item.id);
} else {
setLightboxImage(item);
setIsLightboxVisible(true);
}
},
[mode, onSelectItem],
);
const handleCloseLightbox = () => {
setIsLightboxVisible(false);
// Wait for animation to complete before removing the image
setTimeout(() => setLightboxImage(null), 300);
};
const handleCloseLightbox = useCallback(() => {
setIsLightboxVisible(false);
// Wait for animation to complete before removing the image
setTimeout(() => setLightboxImage(null), 300);
}, []);
return (
<>
<TransitionDialogue
isOpen={isOpen}
onClose={onClose}
className={twMerge(
"h-[620px] max-w-4xl",
mode === "view" && "h-[80vh] max-h-[870px] max-w-7xl",
)}
childPadding={false}
>
<div className="flex h-full flex-col">
<div className="border-b border-white/10 p-4 py-3">
<div className="grid grid-cols-3 items-center">
<h2 className="text-xl font-semibold">
{mode === "select" ? "Select Images" : "My Gallery"}
</h2>
<div className="flex items-center justify-center">
<TabSelector
tabs={tabs}
activeTab={activeTab}
onTabChange={onTabChange}
className="w-auto"
/>
</div>
<div className="flex justify-end">
<CloseButton onClick={onClose} />
</div>
</div>
</div>
const handleDeselectAll = useCallback(() => {
selectedItemIds.forEach((id) => onSelectItem?.(id));
}, [selectedItemIds, onSelectItem]);
<div className="flex-1 overflow-y-auto p-4">
{loading ? (
<div className="flex h-full items-center justify-center">
<div className="text-white/60">
<LoadingSpinner className="h-12 w-12" />
const handleUseSelected = useCallback(() => {
const selectedItems = Object.values(groupedItems)
.flat()
.filter((item) => selectedItemIds.includes(item.id));
onUseSelected?.(selectedItems);
}, [groupedItems, selectedItemIds, onUseSelected]);
return (
<>
<TransitionDialogue
isOpen={isOpen}
onClose={onClose}
className={twMerge(
"h-[620px] max-w-4xl",
mode === "view" && "h-[80vh] max-h-[870px] max-w-7xl",
)}
childPadding={false}
>
<div className="flex h-full flex-col">
<div className="border-b border-white/10 p-4 py-3">
<div className="grid grid-cols-3 items-center">
<h2 className="text-xl font-semibold">
{mode === "select" ? "Select Images" : "My Gallery"}
</h2>
<div className="flex items-center justify-center">
<TabSelector
tabs={tabs}
activeTab={activeTab}
onTabChange={onTabChange}
className="w-auto"
/>
</div>
<div className="flex justify-end">
<CloseButton onClick={onClose} />
</div>
</div>
) : (
<div className="space-y-6">
{Object.entries(groupedItems).map(([date, dateItems]) => (
<div key={date}>
<h3 className="text-md mb-2 font-medium text-white/60">
{date}
</h3>
<div
className={twMerge(
"grid grid-cols-5 gap-1",
mode === "view" && "grid-cols-5",
)}
>
{dateItems.map((item) => (
<button
key={item.id}
className={twMerge(
"group relative aspect-square overflow-hidden rounded-md border-[3px] transition-all",
mode === "select" &&
selectedItemIds.includes(item.id)
? "border-brand-primary"
: "border-transparent hover:border-white",
)}
onClick={() => handleItemClick(item)}
>
<div className="relative h-full w-full">
{!item.thumbnail ? (
<div className="flex h-full w-full items-center justify-center bg-gray-800">
<span className="text-white/60">
Image not available
</span>
</div>
) : (
<img
src={item.thumbnail}
alt={item.label}
className="h-full w-full object-cover"
onError={() =>
handleImageError(item.thumbnail!)
}
crossOrigin="anonymous"
referrerPolicy="no-referrer"
/>
</div>
<div className="flex-1 overflow-y-auto p-4">
{loading ? (
<div className="flex h-full items-center justify-center">
<div className="text-white/60">
<LoadingSpinner className="h-12 w-12" />
</div>
</div>
) : (
<div className="space-y-6">
{Object.entries(groupedItems).map(([date, dateItems]) => (
<div key={date}>
<h3 className="text-md mb-2 font-medium text-white/60">
{date}
</h3>
<div
className={twMerge(
"grid grid-cols-5 gap-1",
mode === "view" && "grid-cols-5",
)}
>
{dateItems.map((item) => (
<button
key={item.id}
className={twMerge(
"group relative aspect-square overflow-hidden rounded-md border-[3px] transition-all",
mode === "select" &&
selectedItemIds.includes(item.id)
? "border-brand-primary"
: "border-transparent hover:border-white",
)}
onClick={() => handleItemClick(item)}
>
<div className="relative h-full w-full">
{!item.thumbnail ? (
<div className="flex h-full w-full items-center justify-center bg-gray-800">
<span className="text-white/60">
Image not available
</span>
</div>
) : (
<img
src={item.thumbnail}
alt={item.label}
className="h-full w-full object-cover"
onError={() =>
handleImageError(item.thumbnail!)
}
crossOrigin="anonymous"
referrerPolicy="no-referrer"
/>
)}
{mode === "select" &&
selectedItemIds.includes(item.id) && (
<div className="absolute inset-0 bg-white/30" />
)}
</div>
{mode === "select" &&
selectedItemIds.includes(item.id) && (
<div className="absolute inset-0 bg-white/30" />
<div className="absolute right-2 top-2 flex h-6 w-6 items-center justify-center rounded-full bg-brand-primary">
<FontAwesomeIcon
icon={faCheck}
className="text-sm"
/>
</div>
)}
</div>
{mode === "select" &&
selectedItemIds.includes(item.id) && (
<div className="absolute right-2 top-2 flex h-6 w-6 items-center justify-center rounded-full bg-brand-primary">
<FontAwesomeIcon
icon={faCheck}
className="text-sm"
/>
</div>
)}
</button>
))}
</button>
))}
</div>
</div>
))}
</div>
)}
</div>
{mode === "select" && (
<div className="flex items-center justify-between border-t border-white/10 bg-black/25 p-4">
<div className="flex items-center gap-3">
<div className="text-sm font-semibold text-white/80">
{selectedItemIds.length}/{maxSelections} selected
</div>
))}
{selectedItemIds.length > 0 && (
<span className="text-white/10">|</span>
)}
{selectedItemIds.length > 0 && (
<button
onClick={handleDeselectAll}
className="text-sm text-white/60 hover:text-white"
>
Deselect All
</button>
)}
</div>
<Button
onClick={handleUseSelected}
disabled={selectedItemIds.length === 0}
>
Use selected
</Button>
</div>
)}
</div>
</TransitionDialogue>
{mode === "select" && (
<div className="flex items-center justify-between border-t border-white/10 bg-black/25 p-4">
<div className="flex items-center gap-3">
<div className="text-sm font-semibold text-white/80">
{selectedItemIds.length}/{maxSelections} selected
</div>
{selectedItemIds.length > 0 && (
<span className="text-white/10">|</span>
)}
{selectedItemIds.length > 0 && (
<button
onClick={() => {
selectedItemIds.forEach((id) => onSelectItem?.(id));
}}
className="text-sm text-white/60 hover:text-white"
>
Deselect All
</button>
)}
</div>
<Button
onClick={() => {
const selectedItems = Object.values(groupedItems)
.flat()
.filter((item) => selectedItemIds.includes(item.id));
onUseSelected?.(selectedItems);
}}
disabled={selectedItemIds.length === 0}
>
Use selected
</Button>
</div>
)}
</div>
</TransitionDialogue>
<LightboxModal
isOpen={isLightboxVisible}
onClose={handleCloseLightbox}
imageUrl={imageUrl}
imageAlt={lightboxImage?.label || ""}
onImageError={() => imageUrl && handleImageError(imageUrl)}
title={lightboxImage?.label}
createdAt={lightboxImage?.createdAt}
downloadUrl={imageUrl}
/>
</>
);
},
);
{/* Lightbox Modal */}
{lightboxImage &&
createPortal(
<Transition appear show={isLightboxVisible} as={Fragment}>
<div className="fixed inset-0 z-[100]">
<Transition.Child
as={Fragment}
enter="ease-out duration-300"
enterFrom="opacity-0"
enterTo="opacity-100"
leave="ease-in duration-200"
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div
className="fixed inset-0 cursor-pointer bg-black/60"
onClick={handleCloseLightbox}
/>
</Transition.Child>
<div
className="fixed inset-0 flex items-center justify-center p-4"
onClick={handleCloseLightbox}
>
<Transition.Child
as={Fragment}
enter="ease-out duration-300"
enterFrom="opacity-0 scale-95"
enterTo="opacity-100 scale-100"
leave="ease-in duration-200"
leaveFrom="opacity-100 scale-100"
leaveTo="opacity-0 scale-95"
>
<div
className="relative h-[90vh] w-[80vw] rounded-xl bg-[#2C2C2C]"
onClick={(e) => e.stopPropagation()}
>
<CloseButton
onClick={handleCloseLightbox}
className="absolute right-4 top-4 z-10"
/>
<div className="grid h-full grid-cols-3 gap-6">
<div className="col-span-2 flex h-full items-center justify-center overflow-hidden bg-black/40">
{!lightboxImage.fullImage ? (
<div className="flex h-full w-full items-center justify-center bg-gray-800">
<span className="text-white/60">
Image not available
</span>
</div>
) : (
<img
src={lightboxImage.fullImage}
alt={lightboxImage.label}
className="h-full w-full object-contain"
onError={() =>
handleImageError(lightboxImage.fullImage!)
}
crossOrigin="anonymous"
referrerPolicy="no-referrer"
/>
)}
</div>
<div className="py-5">Prompt</div>
</div>
</div>
</Transition.Child>
</div>
</div>
</Transition>,
document.body,
)}
</>
);
};
LibraryModal.displayName = "LibraryModal";
@@ -1,10 +1,41 @@
import { useState, useEffect } from "react";
import { Transition } from "@headlessui/react";
import { Key, KeyGroup, Mouse, Plus } from "./ControlsTopButtons/Help/Help";
import { CloseButton } from "~/components";
const STORAGE_KEY = "onboarding_helper_dismissed";
const EXPIRATION_DAYS = 3;
export const OnboardingHelper = () => {
const [actionPerformed, setActionPerformed] = useState(false);
const [keyPressed, setKeyPressed] = useState(false);
const [isVisible, setIsVisible] = useState(true);
useEffect(() => {
// Check if the helper was dismissed
const dismissedData = localStorage.getItem(STORAGE_KEY);
if (dismissedData) {
const { timestamp } = JSON.parse(dismissedData);
const expirationTime = timestamp + EXPIRATION_DAYS * 24 * 60 * 60 * 1000; // 3 days in milliseconds
if (Date.now() < expirationTime) {
setIsVisible(false);
} else {
// Clear expired data
localStorage.removeItem(STORAGE_KEY);
}
}
}, []);
const handleClose = () => {
setIsVisible(false);
// Store dismissal timestamp
localStorage.setItem(
STORAGE_KEY,
JSON.stringify({
timestamp: Date.now(),
})
);
};
useEffect(() => {
const handleKeyDown = (event: KeyboardEvent) => {
@@ -30,7 +61,7 @@ export const OnboardingHelper = () => {
return (
<Transition
show={!(actionPerformed && keyPressed)}
show={isVisible && !(actionPerformed && keyPressed)}
enter="transition-opacity duration-500"
enterFrom="opacity-0"
enterTo="opacity-100"
@@ -38,7 +69,7 @@ export const OnboardingHelper = () => {
leaveFrom="opacity-100"
leaveTo="opacity-0"
>
<div className="glass pointer-events-none absolute left-1/2 top-52 z-10 flex -translate-x-1/2 transform items-center justify-center gap-2 rounded-2xl border-2 border-brand-primary !bg-brand-primary/10 px-5 py-4">
<div className="glass pointer-events-none absolute left-1/2 top-48 z-10 flex -translate-x-1/2 transform items-center justify-center gap-2 rounded-2xl border-2 border-brand-primary !bg-brand-primary/10 px-5 py-4">
<div className="flex items-center">
<p className="pr-3 text-lg font-semibold">Hold</p>
<Mouse button="left" />
@@ -56,6 +87,9 @@ export const OnboardingHelper = () => {
</div>
</div>
<p className="text-lg font-semibold">to move around the scene</p>
<div className="pointer-events-auto ml-2">
<CloseButton onClick={handleClose} />
</div>
</div>
</Transition>
);
@@ -140,7 +140,7 @@ export const Outliner = () => {
useEffect(() => {
const updateEditorHeight = () => {
setEditorHeight(pageHeight.value - timelineHeight.value - 64);
setEditorHeight(pageHeight.value - timelineHeight.value - 56);
};
updateEditorHeight();
@@ -0,0 +1,73 @@
import React from "react";
import { useSignals } from "@preact/signals-react/runtime";
import { poseMode, showPoseControls } from "../signals/selectedMode";
import { Button, Tooltip } from "~/components";
import {
faCheck,
faCheckDouble,
faPersonRunning,
} from "@fortawesome/pro-solid-svg-icons";
declare global {
interface Window {
__mouseControls?: {
toggleFKMode: () => void;
};
}
}
interface PoseModeSelectorProps {}
export const PoseModeSelector: React.FC<PoseModeSelectorProps> = () => {
useSignals();
const handleModeChange = () => {
if (poseMode.value === "select") {
poseMode.value = "pose";
} else {
poseMode.value = "select";
}
window.__mouseControls?.toggleFKMode();
};
if (showPoseControls.value === false) {
return null;
}
return (
<>
<div
className="fixed left-1/2 top-32 flex -translate-x-1/2 transform items-center justify-center gap-2"
onClick={(e) => e.stopPropagation()}
onMouseDown={(e) => e.stopPropagation()}
>
<Tooltip
content={"Toggle pose mode (K)"}
position={"bottom"}
delay={300}
closeOnClick={true}
>
<>
{poseMode.value === "select" ? (
<Button
icon={faPersonRunning}
onClick={handleModeChange}
className="rounded-xl shadow-xl outline-none focus-visible:outline-none"
>
Enter Pose Mode
</Button>
) : (
<Button
icon={faCheck}
onClick={handleModeChange}
className="rounded-xl outline-none focus-visible:outline-none"
>
Done
</Button>
)}
</>
</Tooltip>
</div>
</>
);
};
@@ -0,0 +1,43 @@
import { Signal } from "@preact/signals-react";
import { useSignals } from "@preact/signals-react/runtime";
export const PrecisionSelector = ({
showSignal,
coordSignal,
valuesSignal,
selectedValueSignal
}: {
showSignal: Signal<boolean>,
coordSignal: Signal<{ x: number; y: number }>,
valuesSignal: Signal<number[]>
selectedValueSignal: Signal<number>
}) => {
useSignals();
const handleMouseLeave = () => {
showSignal.value = false;
}
const handleMouseEnterItem = (scale: number) => {
selectedValueSignal.value = scale;
console.log("selectedValueSignal", selectedValueSignal.value);
}
return (
<div
className="fixed z-50 bg-red-600 -translate-x-1/2 -translate-y-1/2 flex-col gap-[1px] shadow-md bg-ui-divider border-ui-divider border-2 rounded-md overflow-clip"
style={{
top: coordSignal.value.y,
left: coordSignal.value.x,
display: showSignal.value ? "flex" : "none"
}}
onMouseLeave={handleMouseLeave}
>
{valuesSignal.value.map((scale, index) => (
<span onMouseEnter={() => handleMouseEnterItem(scale)} key={index}
className="flex bg-ui-panel text-sm justify-center align-middle justify-items-center px-2 hover:bg-ui-controls"
>{scale}</span>
))}
</div>
)
}
@@ -24,16 +24,19 @@ import { CameraAspectRatio, EditorStates } from "~/pages/PageEnigma/enums";
import { CameraViewCanvas } from "~/pages/PageEnigma/comps/EngineCanvases";
import { Button, ButtonIcon, Tooltip } from "~/components";
import { useEffect } from "react";
export const PreviewEngineCamera = () => {
useSignals();
const handleButtonCameraView = () => {
console.log("Camera view button clicked");
Queue.publish({
queueName: QueueNames.TO_ENGINE,
action: toEngineActions.TOGGLE_CAMERA_STATE,
data: null,
});
console.log("Action published to queue");
};
const getLargeScreenHeightClass = () => {
@@ -67,6 +70,10 @@ export const PreviewEngineCamera = () => {
}
};
useEffect(() => {
console.log("Editor state changed:", editorState.value);
}, [editorState.value]);
return (
<div
id="preview-engine-camera"
@@ -27,7 +27,7 @@ export const PreviewFrameImage = () => {
pageWidth.value -
(sidePanelVisible.value ? sidePanelWidth.value : 0) -
84,
height: pageHeight.value - timelineHeight.value - 64,
height: pageHeight.value - timelineHeight.value - 56,
}}
>
<div className="relative flex h-full w-full flex-col items-center justify-center gap-5">
@@ -48,7 +48,7 @@ export const PreviewFrameImage = () => {
pageWidth.value -
(sidePanelVisible.value ? sidePanelWidth.value : 0) -
84,
height: pageHeight.value - timelineHeight.value - 64,
height: pageHeight.value - timelineHeight.value - 56,
}}
/>
);
@@ -46,7 +46,9 @@ import {
addCamera,
deleteCamera,
updateCamera,
focalLengthDragging,
} from "~/pages/PageEnigma/signals/camera";
import { toast } from "@storyteller/ui-toaster";
interface ReferenceImage {
id: string;
@@ -93,7 +95,6 @@ export const PromptBox = () => {
[],
);
const [activeLibraryTab, setActiveLibraryTab] = useState("my-media");
const [isGridVisible, setIsGridVisible] = useState(true);
useEffect(() => {
if (textareaRef.current) {
@@ -121,36 +122,138 @@ export const PromptBox = () => {
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [cameraAspectRatio.value]);
useEffect(() => {
setGridVisibility(isGridVisible);
}, [isGridVisible]);
const handleCameraSelect = (selectedItem: PopoverItem) => {
const selectedCamera = cameras.value.find(
(cam) => cam.label === selectedItem.label,
);
if (selectedCamera) {
if (selectedCamera && editorEngine) {
selectedCameraId.value = selectedCamera.id;
// Force update camera properties
// Show focal length display temporarily
// TODO: Rename dragging to visible - BFlat
focalLengthDragging.value = {
isDragging: true,
focalLength: selectedCamera.focalLength,
};
setTimeout(() => {
focalLengthDragging.value = {
isDragging: false,
focalLength: selectedCamera.focalLength,
};
}, 1500);
// Update the main camera to match the selected camera's properties
if (editorEngine.camera) {
// First update position and lookAt
editorEngine.camera.position.set(
selectedCamera.position.x,
selectedCamera.position.y,
selectedCamera.position.z,
);
editorEngine.camera.lookAt(
selectedCamera.lookAt.x,
selectedCamera.lookAt.y,
selectedCamera.lookAt.z,
);
// Update FOV
editorEngine.camera.fov = editorEngine.focalLengthToFov(
selectedCamera.focalLength,
);
editorEngine.camera.updateProjectionMatrix();
// Reset and update camera controls
if (editorEngine.cameraViewControls) {
editorEngine.cameraViewControls.reset();
editorEngine.cameraViewControls.update(0);
}
// Force a render to update the view
editorEngine.renderScene();
// Queue an update to ensure the engine processes the camera change
// Queue.publish({
// queueName: QueueNames.TO_ENGINE,
// action: toEngineActions.CAMERA_CHANGED,
// data: selectedCamera,
// });
}
// Force update camera properties in the state
updateCamera(selectedCamera.id, {
focalLength: selectedCamera.focalLength,
position: selectedCamera.position,
rotation: selectedCamera.rotation,
lookAt: selectedCamera.lookAt,
});
}
};
const handleAddCamera = () => {
// Check if we've reached the maximum number of cameras
if (cameras.value.length >= 6) {
console.warn("Maximum number of cameras (6) reached");
return;
}
const newIndex = cameras.value.length + 1;
const newId = `cam${newIndex}`;
// This is for generating random orbital position for the new camera using spherical coordinates
const radius = Math.random() * 5 + 7; // Distance from center: 7 to 12 units
const theta = Math.random() * Math.PI * 2; // Azimuthal angle: 0 to 2π
const phi = Math.PI / 3 + (Math.random() * Math.PI) / 6; // Polar angle: π/3 to π/2 (60° to 90° from horizontal)
// Convert spherical coordinates to Cartesian coordinates
const randomX = radius * Math.sin(phi) * Math.cos(theta);
const randomY = Math.abs(radius * Math.cos(phi)) + 2; // Ensure Y is positive and at least 2 units up
const randomZ = radius * Math.sin(phi) * Math.sin(theta);
addCamera({
id: newId,
label: `Camera ${newIndex}`,
focalLength: 35,
fov: 70,
position: { x: 0, y: 2.5, z: -5 },
focalLength: 24,
position: {
x: randomX,
y: randomY,
z: randomZ,
},
rotation: { x: 0, y: 0, z: 0 },
lookAt: { x: 0, y: 0, z: 0 },
});
// Switch to the newly created camera
selectedCameraId.value = newId;
// Update the engine camera to match the new camera's properties
if (editorEngine && editorEngine.camera) {
editorEngine.camera.position.set(randomX, randomY, randomZ);
editorEngine.camera.lookAt(0, 0, 0);
editorEngine.camera.fov = editorEngine.focalLengthToFov(24);
editorEngine.camera.updateProjectionMatrix();
// Reset and update camera controls
if (editorEngine.cameraViewControls) {
editorEngine.cameraViewControls.reset();
editorEngine.cameraViewControls.update(0);
}
// Force a render to update the view
editorEngine.renderScene();
// Queue.publish({
// queueName: QueueNames.TO_ENGINE,
// action: toEngineActions.CAMERA_CHANGED,
// data: {
// id: newId,
// label: `Camera ${newIndex}`,
// focalLength: 24,
// position: { x: randomX, y: randomY, z: randomZ },
// rotation: { x: 0, y: 0, z: 0 },
// lookAt: { x: 0, y: 0, z: 0 },
// },
// });
}
};
const handleCameraNameChange = (id: string, newName: string) => {
@@ -329,7 +432,9 @@ export const PromptBox = () => {
sceneMediaToken: "",
});
console.log("useSystemPrompt", useSystemPrompt);
await engineApi.enqueueImageGeneration({
disableSystemPrompt: !useSystemPrompt,
prompt: prompt,
snapshotMediaToken: snapshotResult.data || "",
additionalImages: referenceImages.map((image) => image.mediaToken),
@@ -347,6 +452,7 @@ export const PromptBox = () => {
await new Promise((resolve) => setTimeout(resolve, 2000));
} finally {
setisEnqueueing(false);
toast.success("Image added to queue");
}
}
setisEnqueueing(false);
@@ -534,6 +640,9 @@ export const PromptBox = () => {
<FontAwesomeIcon icon={faCamera} className="h-4 w-4" />
),
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}))}
onSelect={handleCameraSelect}
onAdd={handleAddCamera}
@@ -541,11 +650,13 @@ export const PromptBox = () => {
<FontAwesomeIcon icon={faCamera} className="h-4 w-4" />
}
showAddButton
disableAddButton={cameras.value.length >= 6}
showIconsInList
mode="toggle"
panelTitle="Camera"
panelActionLabel="Settings"
onPanelAction={() => setIsCameraSettingsOpen(true)}
buttonClassName="max-w-[130px] h-9"
/>
</Tooltip>
<Tooltip
@@ -630,12 +741,15 @@ export const PromptBox = () => {
selected: cam.id === selectedCameraId.value,
icon: <FontAwesomeIcon icon={faCamera} className="h-4 w-4" />,
focalLength: cam.focalLength,
position: cam.position,
rotation: cam.rotation,
lookAt: cam.lookAt,
}))}
onCameraNameChange={handleCameraNameChange}
onCameraFocalLengthChange={handleCameraFocalLengthChange}
onAddCamera={handleAddCamera}
selectedCameraId={selectedCameraId.value}
onSelectCamera={(id: string) => (selectedCameraId.value = id)}
handleCameraSelect={handleCameraSelect}
onDeleteCamera={deleteCamera}
/>
</div>
@@ -11,7 +11,7 @@ export const SceneContainer = ({ children }: { children: React.ReactNode }) => {
const containerWidth = pageWidth.value;
const containerHeight = pageHeight.value - timelineHeight.value - 64;
const containerHeight = pageHeight.value - timelineHeight.value - 56;
const callbackRef = useCallback(
(node: HTMLDivElement) => {
@@ -35,7 +35,7 @@ export const SidePanelMenu = ({
sidePanelVisible.value ? "border-transparent" : "border-[#363636]",
)}
style={{
height: pageHeight.value - 64,
height: pageHeight.value - 56,
}}
>
<div className="flex w-full flex-col gap-2">
@@ -4,9 +4,9 @@ export interface Camera {
id: string;
label: string;
focalLength: number;
fov: number;
position: { x: number; y: number; z: number };
rotation: { x: number; y: number; z: number };
lookAt: { x: number; y: number; z: number };
}
export const cameras = signal<Camera[]>([
@@ -14,17 +14,17 @@ export const cameras = signal<Camera[]>([
id: "main",
label: "Main View",
focalLength: 17,
fov: 70,
position: { x: 2.5, y: 2.5, z: -2.5 },
position: { x: -2.5, y: 2.5, z: 2.5 },
rotation: { x: 0, y: 0, z: 0 },
lookAt: { x: 0, y: 0, z: 0 },
},
{
id: "cam2",
label: "Camera 2",
focalLength: 35,
fov: 70,
position: { x: -2.5, y: 2.5, z: 2.5 },
focalLength: 10,
position: { x: 0, y: 0.6, z: 1.5 },
rotation: { x: 0, y: 0, z: 0 },
lookAt: { x: 0, y: 0, z: 0 },
},
]);
@@ -0,0 +1,6 @@
import { Signal, signal } from "@preact/signals-react";
export const showPrecisionSelector = signal(false);
export const precisionSelectorMenuCoords = signal({ x: 0, y: 0 });
export const precisionSelectorValues: Signal<number[]> = signal([]);
export const precisionSelectedValue = signal(0);
@@ -1,3 +1,5 @@
import { signal } from "@preact/signals-react";
export const selectedMode = signal("move");
export const showPoseControls = signal(false);
export const poseMode = signal("select");
+3
View File
@@ -16,6 +16,7 @@ import normalizeCss from "./styles/normalize.css?url";
import tailwindCss from "./styles/tailwind.css?url";
import baseCss from "./styles/base.css?url";
import { Environment, Configs } from "./configs";
//import { api } from "@storyteller/api";
// The following import prevents a Font Awesome icon server-side rendering bug,
// where the icons flash from a very large icon down to a properly sized one:
@@ -71,6 +72,8 @@ export async function loader() {
uploadApiVideo = "http://localhost:12345";
break;
}
//console.log(">>>> API LIBRARY IMPORT TEST", api());
const env = {
// @ts-expect-error ProvessEnv is correct
+1 -1
View File
@@ -52,7 +52,7 @@ img {
}
.topbar-spacer {
height: 64px;
height: 56px;
width: 100vw;
}
+5 -6
View File
@@ -1,14 +1,14 @@
[build]
command = "./script/netlify_build.sh"
publish = "build/client"
command = "./apps/editor3d/script/netlify_build.sh"
publish = "apps/editor3d/dist/"
[build.environment]
REACT_APP_PUBLIC_POSTHOG_KEY = "phc_jBFgac0mVALAFk3negnSfYCcHgvkT00yBLQDmCDYNBb"
REACT_APP_PUBLIC_POSTHOG_UI = "https://us.i.posthog.com"
[dev]
command = "npm run dev"
framework = "vite"
#[dev]
#command = "npm run dev"
#framework = "vite"
# Set immutable caching for static files, because they have fingerprinted filenames
@@ -16,7 +16,6 @@ framework = "vite"
for = "/build/*"
[headers.values]
"Cache-Control" = "public, max-age=31560000, immutable"
[[redirects]]
+8 -4
View File
@@ -4,7 +4,14 @@
"sideEffects": false,
"type": "module",
"nx": {
"name": "editor3d"
"name": "editor3d",
"targets": {
"build": {
"dependsOn": [
"^build"
]
}
}
},
"scripts": {
"build": "remix vite:build",
@@ -44,14 +51,12 @@
"@zip.js/zip.js": "^2.7.51",
"ammo.js": "^0.0.10",
"ccapture.js": "github:aceslowman/ccapture.js#npm-fix",
"common": "file:common",
"csstype": "^3.1.3",
"dayjs": "^1.11.10",
"deep-equal": "^2.2.3",
"detect-gpu": "^5.0.38",
"http-proxy-middleware": "^3.0.0",
"isbot": "^4.1.0",
"kalidokit": "file:./vendor/kalidokit",
"lodash.debounce": "^4.0.8",
"posthog-js": "^1.141.0",
"react": "^18.2.0",
@@ -101,7 +106,6 @@
"eslint-plugin-storybook": "^0.8.0",
"husky": "^9.0.11",
"lint-staged": "^15.2.2",
"node-sass": "^9.0.0",
"postcss": "^8.4.37",
"postcss-import": "^16.0.1",
"postcss-nesting": "^12.1.0",
Binary file not shown.

After

Width:  |  Height:  |  Size: 6.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.1 KiB

+22 -17
View File
@@ -2,24 +2,29 @@
set -euxo pipefail
echo "Run build script"
echo "Install packages"
npm install
npm run build
#echo "Emit built files."
#pwd
#find .
echo "Build Editor 3D"
nx build editor3d --verbose
echo "Change to project dir"
pushd apps/editor3d/
echo "List build directory files"
ls -lA build/
echo "Make output directory"
mkdir dist/
echo "Copy files to putput directory"
cp -r build/server/* dist/
cp -r build/client/* dist/
cp _headers dist/
#cp _redirects dist/
#cp "404.html" dist/
echo "List final files in build"
find dist/
# echo "Copy test files into dist"
# cp -r test/* dist/
#
# echo "Copy netlify configs into dist"
# cp _headers dist/
# cp _redirects dist/
#
# echo "Copy netlify 404.html page into dist"
# cp "404.html" dist/
#
# echo "List files after build"
# find dist/
+14 -17
View File
@@ -1,34 +1,31 @@
{
"include": [
"env.d.ts",
"**/*.ts",
"**/*.tsx",
"netlify-edge-plugin.ts"
],
"extends": "../../tsconfig.base.json",
"include": ["env.d.ts", "**/*.ts", "**/*.tsx", "netlify-edge-plugin.ts"],
"compilerOptions": {
"lib": [
"DOM",
"DOM.Iterable",
"ESNext"
],
"lib": ["DOM", "DOM.Iterable", "ESNext"],
"isolatedModules": true,
"esModuleInterop": true,
"jsx": "react-jsx",
"module": "ESNext",
"moduleResolution": "node",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"target": "ESNext",
"strict": true,
"allowJs": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"baseUrl": ".",
"paths": {
"~/*": [
"./app/*"
]
"~/*": ["apps/editor3d/app/*"]
},
// Vite takes care of building everything, not tsc.
"noEmit": true
}
},
"references": [
{
"path": "../../libs/login"
},
{
"path": "../../libs/components/toaster"
}
]
}
+9
View File
@@ -14,6 +14,15 @@ export default defineConfig({
// 'kalidokit': `${path.resolve(__dirname, 'src')}/`,
// },
//},
//resolve: {
// alias: [
// {
// find: /@frontend\/login/,
// //replacement: path.resolve(__dirname, 'node_modules', '@frontend', 'login'),
// replacement: path.resolve(__dirname, '../', '../', 'libs', 'login'),
// },
// ],
//},
//rollupOptions: {
// // make sure to externalize deps that shouldn't be bundled
// // into your library
+10
View File
@@ -0,0 +1,10 @@
#!/bin/bash
set -euxo pipefail
echo "Removing Node Modules..."
npx nx reset
rm -rf ./node_modules/
rm -rf ./apps/editor2d/node_modules/
rm -rf ./apps/editor3d/node_modules/
+50
View File
@@ -0,0 +1,50 @@
{
"extends": [
"../../.eslintrc.json"
],
"ignorePatterns": [
"!**/*"
],
"overrides": [
{
"files": [
"*.ts",
"*.tsx",
"*.js",
"*.jsx"
],
"rules": {}
},
{
"files": [
"*.ts",
"*.tsx"
],
"rules": {}
},
{
"files": [
"*.js",
"*.jsx"
],
"rules": {}
},
{
"files": [
"*.json"
],
"parser": "jsonc-eslint-parser",
"rules": {
"@nx/dependency-checks": [
"error",
{
"ignoredFiles": [
"{projectRoot}/eslint.config.{js,cjs,mjs}",
"{projectRoot}/vite.config.{js,ts,mjs,mts}"
]
}
]
}
}
]
}
+11
View File
@@ -0,0 +1,11 @@
# api
This library was generated with [Nx](https://nx.dev).
## Building
Run `nx build api` to build the library.
## Running unit tests
Run `nx test api` to execute the unit tests via [Vitest](https://vitest.dev/).
+19
View File
@@ -0,0 +1,19 @@
{
"name": "@storyteller/api",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "./dist/index.js",
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
"./package.json": "./package.json",
".": {
"development": "./src/index.ts",
"types": "./dist/index.d.ts",
"import": "./dist/index.js",
"default": "./dist/index.js"
}
},
"dependencies": {}
}
+17
View File
@@ -0,0 +1,17 @@
export * from './lib/api.js';
export * from "./lib/AnalyticsApi.js";
export * from "./lib/BillingApi.js";
export * from "./lib/CommentsApi.js";
export * from "./lib/EngineApi.js";
export * from "./lib/FeaturedItemsApi.js";
export * from "./lib/JobsApi.js";
export * from "./lib/MediaFilesApi.js";
export * from "./lib/MediaUploadApi.js";
export * from "./lib/MiscApi.js";
export * from "./lib/ModerationApi.js";
export * from "./lib/PromptsApi.js";
export * from "./lib/TtsApi.js";
export * from "./lib/UserBookmarksApi.js";
export * from "./lib/UsersApi.js";
export * from "./lib/VideoApi.js";
export * from "./lib/WeightsApi.js";

Some files were not shown because too many files have changed in this diff Show More