Compare commits

..
Author SHA1 Message Date
Brandon Thomas de780b0355 pass strong type 2026-06-15 19:55:30 -04:00
Brandon Thomas cb5abeff08 rename methods 2026-06-15 19:42:08 -04:00
Brandon Thomas 78c688e104 to_string() 2026-06-15 19:39:17 -04:00
Brandon Thomas b3d281a7f5 simplify 2026-06-15 19:36:50 -04:00
Brandon Thomas fe0610b779 parsing in the lib 2026-06-15 19:25:17 -04:00
Brandon Thomas 24e2b44207 debug more 2026-06-15 19:14:38 -04:00
Brandon Thomas db344608b6 rename crate 2026-06-15 19:10:11 -04:00
Brandon Thomas 146b78dcb6 crate for singualr type 2026-06-15 19:08:58 -04:00
Brandon Thomas c822872f36 api key parsing 2026-06-15 18:49:48 -04:00
73 changed files with 18 additions and 4967 deletions
@@ -1,47 +0,0 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n api_keys.token as `api_key_token: ApiKeyToken`,\n users.token as `user_token: UserToken`,\n users.is_banned\nFROM api_keys\nINNER JOIN users\n ON users.token = api_keys.owner_user_token\nWHERE api_keys.api_key = ?\n AND api_keys.maybe_deleted_at IS NULL\nLIMIT 1\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "api_key_token: ApiKeyToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "user_token: UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "is_banned",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false
]
},
"hash": "f52c495a7ff5cafb357f79c4cdb503bbbef045fd193a5a37a66180fcac77f363"
}
-203
View File
@@ -1,203 +0,0 @@
# PageDraw — Visual Design & Architecture Review
**Surface:** `@storyteller/ui-pagedraw` image editor + its `artcraft-webapp` integration (`/edit-image`).
**Lens:** Awwwards / Linear-tier "premium app chrome" rubric, adapted for an in-product creative tool.
**Date:** 2026-06-15 · **Branch:** `worktree-pagedraw-webapp`
---
## 0. How to read this
PageDraw is a **professional canvas tool**, not a marketing page. So the elite-design rubric is
applied in its *app-chrome* register — the north star is **Figma / Linear / Photoshop-web**, not an
Awwwards landing page. That means:
- **Applies:** token discipline, concentric radii, hairline + inset-highlight surfaces ("double-bezel"),
island/pill controls, custom-eased micro-motion, z-index hygiene, premium type ramp.
- **Does _not_ apply:** `py-40` macro-whitespace, hero typography, editorial-split layouts, scroll-reveal
entrances. A tool surface is dense and persistent by design; forcing landing-page whitespace would hurt it.
Citations are `file:line` against the current tree.
---
## 1. Verdict
| Axis | Score | One-liner |
|---|---|---|
| Color system | ⚠️ 5/10 | Good token foundation, but hardcoded greys (`#1b1b1b`, `#303030`, `#404040`) leak through a *themeable lib*. |
| Typography | ❌ 3/10 | No type ramp, no font-family ownership; rides whatever the host sets. Banned-by-rubric default stack risk. |
| Radii | ⚠️ 4/10 | Five radius scales in play (`sm`→`2xl`→`full`) with no concentric nesting. |
| Surfaces (border/shadow) | ⚠️ 5/10 | Flat `shadow-lg` everywhere; no inset highlights; no double-bezel. Reads "functional", not "machined". |
| Motion | ❌ 3/10 | Only `transition-colors`/`ease-in-out`, durations 75–200ms; **zero custom easing**, no spring, no entrance choreography. |
| Layout / z-index | ⚠️ 6/10 | Clean fixed-panel composition, but a `z-[9999999]` escape hatch and a 6-tier ad-hoc z-stack. |
| Icons | ✅ 7/10 | FontAwesome Pro (solid + regular) — acceptable, but solid weight is heavier than the rubric's "ultra-light line" ideal. |
| Architecture | ⚠️ 5/10 | Clean adapter boundary; but `PaintSurface.tsx` (2005 LOC), `SceneState.ts` (1539), `PageDraw.tsx` (1292) are oversized, and the lib hardcodes host-theme colors. |
**Bottom line:** the editor is *structurally competent and functional* but visually reads as
"capable internal tool," not "$150k product surface." The single highest-leverage fix is a **shared
design-token layer** (color + radius + motion + type), because today the lib both (a) reaches for
host tokens (`bg-ui-panel`, `text-base-fg`) *and* (b) hardcodes its own greys — so it can never be
fully reskinned by the webapp vs. the desktop app.
---
## 2. What's already good (keep)
- **Token-aware base.** Most surfaces use semantic tokens (`bg-ui-panel`, `bg-ui-controls`,
`text-base-fg`, `border-ui-panel-border`) — the right instinct (SideToolbar, PromptEditor, HistoryStack).
- **Clean platform boundary.** The lib has zero REST/Tauri imports; everything I/O flows through
`PageDrawAdapter` (`adapter.ts`). The webapp adapter (`web-adapter.tsx`) and desktop adapter stay
swappable. This is genuinely good architecture and should be protected.
- **Performance-minded canvas.** Composite pre-baking on idle (`PageDraw.tsx` ~661–710) and worker
encoding (`generatePipeline.worker.ts`) keep the generate click off the main thread. Don't regress this.
- **CORS already handled** for browser canvas export (`crossOrigin="anonymous"` in `Node.ts:134`,
`SceneState.ts:1473`, `imageHelpers.ts:16`) — the web port works because of this.
---
## 3. Findings & recommendations
### 3.1 Color — stop hardcoding greys in a themeable lib ⚠️ P0
**Current:** semantic tokens *and* raw hex coexist:
- `#1b1b1b` pegboard (`pagedraw.css:6`), `#1A1A1A`/`#333333` context menu (`ContextMenu.tsx:44`),
`#303030`/`#404040` dividers + popovers (`SideToolbar.tsx:381,550`), `bg-zinc-700/600` swatch
(`SideToolbar.tsx:173`), plus ad-hoc `bg-blue-600/500` accents (`PageDraw.tsx:157`, base selector).
- Accent color is inconsistent: the lib's primary action uses raw **blue-600**, while the webapp's
design system uses `--primary` / `brand-primary`. So the editor's blue doesn't match the product.
**Why it fails the bar:** a premium component library exposes *one* surface ramp and *one* accent, both
driven by tokens, so it reskins per host. Hardcoded greys mean the desktop and web themes can drift, and
the raw `blue-600` clashes with `brand-primary` used elsewhere in the webapp.
**Do:**
- Replace every raw grey with the existing ramp: `#1b1b1b`→`bg-ui-background`, `#1A1A1A`→`bg-ui-panel`,
`#303030`/`#404040`→`border-ui-panel-border` / `bg-ui-controls`. For the pegboard, add a token
`--st-canvas` so hosts can theme the canvas backdrop.
- Replace `blue-600/500` accents with `bg-primary` / `text-primary` (the token the rest of the webapp uses),
so the Generate/Edit-3D buttons match `brand-primary`.
- Move the react-colorful overrides off pure `rgb(255,255,255)` borders (`pagedraw.css:15,19`) onto
`white/15` hairlines for consistency with the rest of the chrome.
### 3.2 Typography — own a type ramp ❌ P0
**Current:** no `font-family` anywhere in the lib (grep: 0 hits); only `text-xs/sm/base/lg/xl` +
`font-medium/semibold/bold`. The editor inherits whatever the host's body font is.
**Why it fails the bar:** the rubric explicitly bans the default web stack (Inter/Roboto/Arial/Helvetica).
Inheriting an unspecified stack means the tool can render in a banned font with no recourse, and there's
no deliberate hierarchy (numerics in sliders, labels, modal titles all share one undifferentiated ramp).
**Do:**
- Adopt the product's display/grotesk token (e.g. a `font-sans` mapped to Geist / Plus Jakarta Sans at
the app level) and reference it in the lib's root container, not per-component.
- Define a small semantic ramp: `text-[11px] tracking-[0.16em] uppercase` for toolbar/section labels,
`text-sm` for controls, `text-base font-semibold` for modal titles. Apply `tabular-nums` to slider
read-outs (`SliderWithIndicator.tsx`) and dimension fields (`BlankCanvasModal`) so numbers don't jitter.
### 3.3 Radii — collapse to a concentric scale ⚠️ P1
**Current:** `rounded-sm / md / lg / xl / 2xl / full` all appear, with no parent→child relationship
(e.g. `rounded-2xl` card at `PageDraw.tsx:1123`, `rounded-xl` panels in SideToolbar, `rounded-lg`
buttons in InpaintToolBar).
**Do:** pick a 3-step system — **`rounded-2xl` shells, `rounded-xl` inner cards, `rounded-lg` controls,
`rounded-full` pills** — and apply concentrically (the "double-bezel" curve math: inner radius =
outer − padding). Drop `rounded-sm`/`rounded-md` from the editor entirely.
### 3.4 Surfaces — add the "machined" double-bezel ⚠️ P1
**Current:** panels are a single `bg-ui-panel` + `border` + `shadow-lg`. No nested enclosure, no inset
top-highlight. Reads flat.
**Do (for the four floating panels — SideToolbar, HistoryStack, PromptEditor, InpaintToolBar):**
- Outer shell: `bg-ui-panel/80 ring-1 ring-white/10 backdrop-blur-xl p-1.5 rounded-2xl`.
- Inner core: own bg + `shadow-[inset_0_1px_0_rgba(255,255,255,0.08)] rounded-xl`.
- This is the single change that most moves the surface from "tool" to "premium hardware." The panels are
already `fixed` (`PageDraw.tsx:1144,1175,1195`), so `backdrop-blur` here is GPU-safe per the perf rules.
### 3.5 Motion — replace linear/ease with spring-eased micro-motion ❌ P0 (cheap, high impact)
**Current:** `transition-colors`/`transition-all` with `ease-in-out` and `duration-75/150/200`
(`SideToolbar.tsx:381,550`, `SliderWithIndicator.tsx:65`, `ContextMenu.tsx:53`). No custom easing,
no entrance/exit on popovers, color pickers, or the context menu (they pop instantly).
**Why it fails the bar:** instant state changes and `ease-in-out` are explicitly banned. A pro tool still
deserves *interpolated* motion — it's what separates Figma from a jQuery plugin.
**Do:**
- Standardize on one curve: `transition-[transform,opacity,background-color] duration-300
ease-[cubic-bezier(0.32,0.72,0,1)]`.
- Tool buttons: `active:scale-[0.97]` press physics; selected tool gets a spring-eased indicator slide
rather than a hard background swap.
- Popovers / color picker / context menu: animate in with `opacity-0 translate-y-1 scale-[0.98]` →
resolved, ~180ms. Transform+opacity only (perf rule).
- History tiles: stagger their `opacity` reveal when a generation lands (they already use
`transition-opacity`, just add per-index delay).
### 3.6 Layout & z-index — tame the stack ⚠️ P1
**Current:** clean fixed-panel composition, but z-index is ad-hoc across 6 tiers and includes a
`z-[9999999]` on the context menu (`ContextMenu.tsx:44`) and a one-off `z-[60]` (`PageDraw.tsx:237`).
**Do:** define a named scale in one place — `canvas:0, chrome:10, floating-controls:20, overlay-button:40,
menu:50` — and replace the magic numbers. The portal menu only needs to clear the chrome, not `9999999`.
### 3.7 Responsive — declare intent ⚠️ P2
**Current:** essentially desktop-only; fixed sidebars assume a wide viewport; only LightboxDetails has
`sm:` variants. The webapp wrapper scopes the lib correctly (`pagedraw.tsx` `translateZ(0)` containing
block) but there's no mobile story.
**Do:** mirror the Edit-3D page's explicit **mobile gate** (`pagescene.tsx` `MobileGate`) on `/edit-image`
rather than letting fixed toolbars overlap on touch. A precision canvas tool is legitimately desktop-first;
just say so deliberately instead of degrading silently.
---
## 4. Architecture notes
- **Oversized modules.** `PaintSurface.tsx` (2005 LOC), `SceneState.ts` (1539), `PageDraw.tsx` (1292).
These are review/maintenance hotspots. Suggest extracting from `SceneState.ts`: history/undo slice,
base-image-load slice, and generation/pending slice into composable Zustand slices. `PageDraw.tsx`'s
generate-flow (cases 1–3, ~780–868) could move to a `useGenerate()` hook.
- **Theme ownership.** Per §3.1, the lib should *only* consume tokens. Today it mixes tokens + hex, which
defeats the adapter pattern's promise (one component, many themed hosts).
- **Adapter surface is healthy.** `PageDrawAdapter` is small and intentional; the web vs. Tauri split is
clean. The webapp's `enqueueInpaint` stub and 402→credits-modal handling are explicit and documented.
- **Icon weight.** FontAwesome **solid** is the heaviest option; if a line/light set is available
(Phosphor/Remix or FA light), the toolbar would read more refined. Low priority.
---
## 5. Prioritized roadmap
| P | Item | Effort | Payoff |
|---|---|---|---|
| **P0** | Token sweep: kill hardcoded greys; unify accent on `--primary` (§3.1) | M | Highest — enables true theming + product-consistent accent |
| **P0** | Own a font-family + semantic type ramp at the lib root (§3.2) | S | High — removes banned-font risk, adds hierarchy |
| **P0** | One custom easing curve + press/popover micro-motion (§3.5) | S | High — cheapest "feels expensive" win |
| **P1** | Double-bezel + inset highlights on the 4 floating panels (§3.4) | M | High — "machined" surface upgrade |
| **P1** | Concentric 3-step radius system (§3.3) | S | Medium — visual coherence |
| **P1** | Named z-index scale; remove `z-[9999999]` (§3.6) | S | Medium — maintainability |
| **P2** | Explicit mobile gate on `/edit-image` (§3.7) | S | Medium — correctness on touch |
| **P2** | Split `SceneState.ts` / `PaintSurface.tsx` into slices (§4) | L | Medium — maintainability |
---
## 6. Appendix — proposed token additions
```css
/* canvas + chrome surface ramp (lib should consume only these) */
--st-canvas: <theme>; /* replaces #1b1b1b pegboard */
--st-accent: var(--primary); /* replaces blue-600/500 */
--st-hairline: rgb(255 255 255 / 0.10);
--st-inset-hi: inset 0 1px 0 rgb(255 255 255 / 0.08);
/* motion */
--st-ease: cubic-bezier(0.32, 0.72, 0, 1);
--st-dur-fast: 180ms;
--st-dur: 300ms;
/* radius scale (concentric) */
--st-r-shell: 1.25rem; /* rounded-2xl */
--st-r-card: 0.875rem; /* rounded-xl */
--st-r-control: 0.5rem; /* rounded-lg */
```
> Scope note: every recommendation here is **styling/architecture only** — none change the generate,
> polling, upload, bg-removal, or adapter contracts validated during the functional debug pass.
@@ -8,7 +8,6 @@ pub mod generate;
pub mod jobs;
pub mod media_file;
pub mod moderation;
pub mod omni_api;
pub mod omni_gen;
pub mod prompts;
pub mod stripe_artcraft;
@@ -1,3 +0,0 @@
pub mod omni_upload_audio;
pub mod omni_upload_image;
pub mod omni_upload_video;
@@ -1,15 +0,0 @@
use serde_derive::{Deserialize, Serialize};
use utoipa::ToSchema;
use tokens::tokens::media_files::MediaFileToken;
/// Success response for the API-key authenticated audio upload endpoint
/// (`POST /v1/omni_api/upload/audio`).
///
/// NB: The request is a `multipart/form-data` body. Because the multipart form type is bound to the
/// web framework, it is declared alongside the handler in `storyteller_web` rather than here.
#[derive(Clone, Serialize, Deserialize, ToSchema, Debug)]
pub struct OmniUploadAudioMediaFileSuccessResponse {
pub success: bool,
pub media_file_token: MediaFileToken,
}
@@ -1,15 +0,0 @@
use serde_derive::{Deserialize, Serialize};
use utoipa::ToSchema;
use tokens::tokens::media_files::MediaFileToken;
/// Success response for the API-key authenticated image upload endpoint
/// (`POST /v1/omni_api/upload/image`).
///
/// NB: The request is a `multipart/form-data` body. Because the multipart form type is bound to the
/// web framework, it is declared alongside the handler in `storyteller_web` rather than here.
#[derive(Clone, Serialize, Deserialize, ToSchema, Debug)]
pub struct OmniUploadImageMediaFileSuccessResponse {
pub success: bool,
pub media_file_token: MediaFileToken,
}
@@ -1,15 +0,0 @@
use serde_derive::{Deserialize, Serialize};
use utoipa::ToSchema;
use tokens::tokens::media_files::MediaFileToken;
/// Success response for the API-key authenticated video upload endpoint
/// (`POST /v1/omni_api/upload/video`).
///
/// NB: The request is a `multipart/form-data` body. Because the multipart form type is bound to the
/// web framework, it is declared alongside the handler in `storyteller_web` rather than here.
#[derive(Clone, Serialize, Deserialize, ToSchema, Debug)]
pub struct OmniUploadVideoMediaFileSuccessResponse {
pub success: bool,
pub media_file_token: MediaFileToken,
}
@@ -1,63 +0,0 @@
use std::marker::PhantomData;
use sqlx::{Executor, MySql};
use artcraft_api_keys::ArtcraftApiKey;
use tokens::tokens::api_keys::ApiKeyToken;
use tokens::tokens::users::UserToken;
use crate::helpers::boolean_converters::i8_to_bool;
pub struct GetApiKeyUserByValueArgs<'e, 'c, E>
where
E: 'e + Executor<'c, Database = MySql>,
{
/// The actual API key secret presented by the client.
pub api_key: &'e ArtcraftApiKey,
pub mysql_executor: E,
pub phantom: PhantomData<&'c E>,
}
/// A live API key joined to its owning user. This is the minimal identity needed to authenticate
/// an API-key request — far smaller than a cookie `SessionUserRecord`.
pub struct ApiKeyUserRecord {
pub api_key_token: ApiKeyToken,
pub user_token: UserToken,
pub is_banned: bool,
}
/// Look up the user behind an API key secret, for authenticating an incoming request.
///
/// Returns `Ok(None)` when no row matches, when the matching API key is soft-deleted
/// (`maybe_deleted_at IS NOT NULL`), or when the owning user row does not exist (the `INNER JOIN`
/// requires both records to be present). The full secret is never echoed back.
pub async fn get_api_key_user_by_value<'e, 'c: 'e, E>(
args: GetApiKeyUserByValueArgs<'e, 'c, E>,
) -> Result<Option<ApiKeyUserRecord>, sqlx::Error>
where
E: 'e + Executor<'c, Database = MySql>,
{
let result = sqlx::query!(
r#"
SELECT
api_keys.token as `api_key_token: ApiKeyToken`,
users.token as `user_token: UserToken`,
users.is_banned
FROM api_keys
INNER JOIN users
ON users.token = api_keys.owner_user_token
WHERE api_keys.api_key = ?
AND api_keys.maybe_deleted_at IS NULL
LIMIT 1
"#,
args.api_key.as_str_be_careful(),
)
.fetch_optional(args.mysql_executor)
.await?;
Ok(result.map(|r| ApiKeyUserRecord {
api_key_token: r.api_key_token,
user_token: r.user_token,
is_banned: i8_to_bool(r.is_banned),
}))
}
@@ -1,7 +1,6 @@
pub mod api_key_row;
pub mod delete_api_key;
pub mod get_api_key_by_token;
pub mod get_api_key_user_by_value;
pub mod get_live_api_key_by_value;
pub mod insert_api_key;
pub mod list_api_keys_for_user;
@@ -19,7 +19,6 @@ pub enum UserFeatureFlag {
/// (For now, it's hidden until we get an NSFW filter.)
ExploreMedia,
/// Access to studio features
Studio,
@@ -45,10 +44,6 @@ pub enum UserFeatureFlag {
/// Access to the referrals program
#[serde(rename = "referrals")]
ReferralsProgram,
/// Access to API key creation
#[serde(rename = "api_key")]
ApiKey,
}
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
@@ -68,7 +63,6 @@ impl UserFeatureFlag {
Self::HappyHorse => "hh",
Self::HappyHorseRateLimit => "hh_rl",
Self::ReferralsProgram => "referrals",
Self::ApiKey => "api_key",
}
}
@@ -82,7 +76,6 @@ impl UserFeatureFlag {
"hh" => Ok(Self::HappyHorse),
"hh_rl" => Ok(Self::HappyHorseRateLimit),
"referrals" => Ok(Self::ReferralsProgram),
"api_key" => Ok(Self::ApiKey),
_ => Err(format!("invalid value: {:?}", value)),
}
}
@@ -97,7 +90,6 @@ impl UserFeatureFlag {
Self::HappyHorse => "Happy Horse (no Rate Limit)",
Self::HappyHorseRateLimit => "Happy Horse (with Rate Limit)",
Self::ReferralsProgram => "Referrals Program",
Self::ApiKey => "API Keys",
}
}
@@ -111,7 +103,6 @@ impl UserFeatureFlag {
Self::HappyHorse => "Access to Happy Horse (without a rate limit - dangerous)",
Self::HappyHorseRateLimit => "Access to Happy Horse (with a rate limit)",
Self::ReferralsProgram => "Access to the referrals program",
Self::ApiKey => "Access to API key creation",
}
}
@@ -127,7 +118,6 @@ impl UserFeatureFlag {
Self::HappyHorse,
Self::HappyHorseRateLimit,
Self::ReferralsProgram,
Self::ApiKey,
])
}
}
@@ -150,7 +140,6 @@ mod tests {
assert_serialization(UserFeatureFlag::HappyHorse, "hh");
assert_serialization(UserFeatureFlag::HappyHorseRateLimit, "hh_rl");
assert_serialization(UserFeatureFlag::ReferralsProgram, "referrals");
assert_serialization(UserFeatureFlag::ApiKey, "api_key");
}
#[test]
@@ -163,7 +152,6 @@ mod tests {
assert_eq!(UserFeatureFlag::HappyHorse.to_str(), "hh");
assert_eq!(UserFeatureFlag::HappyHorseRateLimit.to_str(), "hh_rl");
assert_eq!(UserFeatureFlag::ReferralsProgram.to_str(), "referrals");
assert_eq!(UserFeatureFlag::ApiKey.to_str(), "api_key");
}
#[test]
@@ -176,14 +164,13 @@ mod tests {
assert_eq!(UserFeatureFlag::from_str("hh").unwrap(), UserFeatureFlag::HappyHorse);
assert_eq!(UserFeatureFlag::from_str("hh_rl").unwrap(), UserFeatureFlag::HappyHorseRateLimit);
assert_eq!(UserFeatureFlag::from_str("referrals").unwrap(), UserFeatureFlag::ReferralsProgram);
assert_eq!(UserFeatureFlag::from_str("api_key").unwrap(), UserFeatureFlag::ApiKey);
assert!(UserFeatureFlag::from_str("foo").is_err());
}
#[test]
fn all_variants() {
let mut variants = UserFeatureFlag::all_variants();
assert_eq!(variants.len(), 9);
assert_eq!(variants.len(), 8);
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::ExploreMedia));
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::Studio));
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::Upload3d));
@@ -192,7 +179,6 @@ mod tests {
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::HappyHorse));
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::HappyHorseRateLimit));
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::ReferralsProgram));
assert_eq!(variants.pop_first(), Some(UserFeatureFlag::ApiKey));
assert_eq!(variants.pop_first(), None);
}
}
@@ -26,9 +26,6 @@ pub enum PlatformType {
/// Desktop app users
DesktopApp,
/// Requests authenticated via an `Authorization` header API key (the Omni API surface).
ApiKey,
// ========== Other clients ==========
Curl,
@@ -45,7 +42,6 @@ impl PlatformType {
match self {
Self::Web => "web",
Self::DesktopApp => "desktop_app",
Self::ApiKey => "api_key",
Self::Curl => "curl",
Self::PythonRequests => "python_requests",
Self::Postman => "postman",
@@ -56,7 +52,6 @@ impl PlatformType {
match value {
"web" => Ok(Self::Web),
"desktop_app" => Ok(Self::DesktopApp),
"api_key" => Ok(Self::ApiKey),
"curl" => Ok(Self::Curl),
"python_requests" => Ok(Self::PythonRequests),
"postman" => Ok(Self::Postman),
@@ -69,7 +64,6 @@ impl PlatformType {
BTreeSet::from([
Self::Web,
Self::DesktopApp,
Self::ApiKey,
Self::Curl,
Self::PythonRequests,
Self::Postman,
@@ -90,7 +84,6 @@ mod tests {
fn test_serialization() {
assert_serialization(PlatformType::Web, "web");
assert_serialization(PlatformType::DesktopApp, "desktop_app");
assert_serialization(PlatformType::ApiKey, "api_key");
assert_serialization(PlatformType::Curl, "curl");
assert_serialization(PlatformType::PythonRequests, "python_requests");
assert_serialization(PlatformType::Postman, "postman");
@@ -100,7 +93,6 @@ mod tests {
fn to_str() {
assert_eq!(PlatformType::Web.to_str(), "web");
assert_eq!(PlatformType::DesktopApp.to_str(), "desktop_app");
assert_eq!(PlatformType::ApiKey.to_str(), "api_key");
assert_eq!(PlatformType::Curl.to_str(), "curl");
assert_eq!(PlatformType::PythonRequests.to_str(), "python_requests");
assert_eq!(PlatformType::Postman.to_str(), "postman");
@@ -110,7 +102,6 @@ mod tests {
fn from_str() {
assert_eq!(PlatformType::from_str("web").unwrap(), PlatformType::Web);
assert_eq!(PlatformType::from_str("desktop_app").unwrap(), PlatformType::DesktopApp);
assert_eq!(PlatformType::from_str("api_key").unwrap(), PlatformType::ApiKey);
assert_eq!(PlatformType::from_str("curl").unwrap(), PlatformType::Curl);
assert_eq!(PlatformType::from_str("python_requests").unwrap(), PlatformType::PythonRequests);
assert_eq!(PlatformType::from_str("postman").unwrap(), PlatformType::Postman);
@@ -130,10 +121,9 @@ mod tests {
#[test]
fn all_variants() {
let mut variants = PlatformType::all_variants();
assert_eq!(variants.len(), 6);
assert_eq!(variants.len(), 5);
assert_eq!(variants.pop_first(), Some(PlatformType::Web));
assert_eq!(variants.pop_first(), Some(PlatformType::DesktopApp));
assert_eq!(variants.pop_first(), Some(PlatformType::ApiKey));
assert_eq!(variants.pop_first(), Some(PlatformType::Curl));
assert_eq!(variants.pop_first(), Some(PlatformType::PythonRequests));
assert_eq!(variants.pop_first(), Some(PlatformType::Postman));
@@ -80,9 +80,6 @@ use crate::http_server::endpoints::media_files::upload::upload_spz_media_file_ha
use crate::http_server::endpoints::media_files::upload::upload_studio_shot::upload_studio_shot_media_file_handler::*;
use crate::http_server::endpoints::media_files::upload::upload_video_new::upload_new_video_media_file_handler::*;
use crate::http_server::endpoints::media_files::upload::upload_video_old::upload_video_media_file_handler::*;
use crate::http_server::endpoints::omni_api::upload::omni_upload_audio_media_file_handler::*;
use crate::http_server::endpoints::omni_api::upload::omni_upload_image_media_file_handler::*;
use crate::http_server::endpoints::omni_api::upload::omni_upload_video_media_file_handler::*;
use crate::http_server::endpoints::media_files::upsert_upload::write_engine_asset::write_engine_asset_media_file_handler::*;
use crate::http_server::endpoints::media_files::upsert_upload::write_error::MediaFileWriteError;
use crate::http_server::endpoints::media_files::upsert_upload::write_scene_file::write_scene_file_media_file_handler::*;
@@ -346,9 +343,6 @@ use artcraft_api_defs::omni_gen::generate_response::omni_gen_image_generate_resp
use artcraft_api_defs::omni_gen::generate_response::omni_gen_video_generate_response::*;
use artcraft_api_defs::omni_gen::models::omni_gen_image_models::*;
use artcraft_api_defs::omni_gen::models::omni_gen_video_models::*;
use artcraft_api_defs::omni_api::omni_upload_audio::*;
use artcraft_api_defs::omni_api::omni_upload_image::*;
use artcraft_api_defs::omni_api::omni_upload_video::*;
use artcraft_api_defs::moderation::user::list_subscribing_users_by_signup_date::*;
use artcraft_api_defs::moderation::user::list_users_by_signup_date::*;
use artcraft_api_defs::moderation::user::user_lookup::*;
@@ -556,12 +550,6 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
crate::http_server::endpoints::omni_gen::generate::video::omni_gen_video_generate_handler::omni_gen_video_generate_handler,
crate::http_server::endpoints::omni_gen::models::image::omni_gen_image_models_handler::omni_gen_image_models_handler,
crate::http_server::endpoints::omni_gen::models::video::omni_gen_video_models_handler::omni_gen_video_models_handler,
// Omni API (API-key authenticated generate)
crate::http_server::endpoints::omni_api::generate::image::omni_api_image_generate_handler::omni_api_image_generate_handler,
crate::http_server::endpoints::omni_api::generate::video::omni_api_video_generate_handler::omni_api_video_generate_handler,
crate::http_server::endpoints::omni_api::upload::omni_upload_audio_media_file_handler::omni_upload_audio_media_file_handler,
crate::http_server::endpoints::omni_api::upload::omni_upload_image_media_file_handler::omni_upload_image_media_file_handler,
crate::http_server::endpoints::omni_api::upload::omni_upload_video_media_file_handler::omni_upload_video_media_file_handler,
// Moderation
crate::http_server::endpoints::moderation::alerts::moderation_send_alert_handler::moderation_send_alert_handler,
crate::http_server::endpoints::moderation::info::moderator_token_info_handler::moderator_get_token_info_handler,
@@ -1089,12 +1077,6 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
UploadEngineAssetMediaSuccessResponse,
UploadImageMediaFileForm,
UploadImageMediaFileSuccessResponse,
OmniUploadAudioMediaFileForm,
OmniUploadAudioMediaFileSuccessResponse,
OmniUploadImageMediaFileForm,
OmniUploadImageMediaFileSuccessResponse,
OmniUploadVideoMediaFileForm,
OmniUploadVideoMediaFileSuccessResponse,
UserImpersonationRequestResponse,
UploadMediaSuccessResponse,
UploadNewEngineAssetFileForm,
@@ -10,7 +10,6 @@ use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::queries::api_keys::insert_api_key::{insert_api_key, InsertApiKeyArgs};
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::session::lookup::user_session_feature_flags::UserSessionFeatureFlags;
use crate::http_server::web_utils::user_session::require_user_session::require_user_session;
use crate::state::server_state::ServerState;
use crate::util::generate_api_key::generate_api_key;
@@ -30,7 +29,6 @@ const MAX_DESCRIPTION_LEN: usize = 512;
(status = 200, body = CreateApiKeySuccessResponse),
(status = 400, body = CommonWebError),
(status = 401, body = CommonWebError),
(status = 403, body = CommonWebError),
(status = 500, body = CommonWebError),
),
)]
@@ -46,14 +44,6 @@ pub async fn create_api_key_handler(
let user_session = require_user_session(&http_request, &server_state.session_checker, &mut *conn).await?;
// API key creation is gated behind the `api_key` feature flag.
let feature_flags = UserSessionFeatureFlags::new(user_session.maybe_feature_flags.as_deref());
if !feature_flags.can_create_api_key() {
warn!("User {:?} attempted to create an API key without the api_key feature flag", user_session.user_token.as_str());
return Err(CommonWebError::Forbidden);
}
let name = request.name.trim().to_string();
let maybe_description = request.maybe_description.as_ref().map(|s| s.trim().to_string());
@@ -17,7 +17,6 @@ pub mod media_files;
pub mod misc;
pub mod model_download;
pub mod moderation;
pub mod omni_api;
pub mod omni_gen;
pub mod prompts;
pub mod service;
@@ -1,100 +0,0 @@
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_image_cost_and_generate_request::OmniGenImageCostAndGenerateRequest;
use artcraft_router::api::router_aspect_ratio::RouterAspectRatio;
use artcraft_router::api::router_image_model::RouterImageModel;
use artcraft_router::api::router_quality::RouterQuality;
use artcraft_router::api::router_resolution::RouterResolution;
use artcraft_router::api::image_list_ref::ImageListRef;
use artcraft_router::api::router_provider::RouterProvider;
use artcraft_router::client::request_mismatch_mitigation_strategy::RequestMismatchMitigationStrategy;
use artcraft_router::generate::generate_image::generate_image_request_builder::GenerateImageRequestBuilder;
use enums::common::generation::common_aspect_ratio::CommonAspectRatio as CommonAspectRatioEnum;
use enums::common::generation::common_image_model::CommonImageModel as CommonImageModelEnum;
use enums::common::generation::common_quality::CommonQuality as CommonQualityEnum;
use enums::common::generation::common_resolution::CommonResolution as CommonResolutionEnum;
use crate::http_server::common_responses::common_web_error::CommonWebError;
pub fn hydrate_to_router_request(
request: &OmniGenImageCostAndGenerateRequest,
) -> Result<GenerateImageRequestBuilder, CommonWebError> {
let api_model = request.model.as_ref()
.ok_or_else(|| CommonWebError::BadInputWithSimpleMessage(
"model is required".to_string(),
))?;
let model = convert_model(api_model)?;
let aspect_ratio = request.aspect_ratio.as_ref()
.map(convert_aspect_ratio)
.transpose()?;
let resolution = request.resolution.as_ref()
.map(convert_resolution)
.transpose()?;
let quality = request.quality.as_ref()
.map(convert_quality)
.transpose()?;
Ok(GenerateImageRequestBuilder {
model,
provider: RouterProvider::Artcraft,
prompt: request.prompt.clone(),
image_inputs: request.image_media_tokens.clone()
.map(ImageListRef::MediaFileTokens),
resolution,
aspect_ratio,
quality,
image_batch_count: request.image_batch_count,
horizontal_angle: request.adjust_horizontal_angle,
vertical_angle: request.adjust_vertical_angle,
zoom: request.adjust_zoom,
request_mismatch_mitigation_strategy: RequestMismatchMitigationStrategy::PayMoreUpgrade,
generation_mode_mismatch_strategy: None,
idempotency_token: request.idempotency_token.clone(),
})
}
fn convert_model(
model: &CommonImageModelEnum,
) -> Result<RouterImageModel, CommonWebError> {
let json = serde_json::to_string(model)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported image model: {}", e),
)
})
}
fn convert_aspect_ratio(
ar: &CommonAspectRatioEnum,
) -> Result<RouterAspectRatio, CommonWebError> {
let json = serde_json::to_string(ar)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported aspect ratio: {}", e),
)
})
}
fn convert_resolution(
res: &CommonResolutionEnum,
) -> Result<RouterResolution, CommonWebError> {
let json = serde_json::to_string(res)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported resolution: {}", e),
)
})
}
fn convert_quality(
quality: &CommonQualityEnum,
) -> Result<RouterQuality, CommonWebError> {
let json = serde_json::to_string(quality)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported quality: {}", e),
)
})
}
@@ -1,61 +0,0 @@
use log::warn;
use enums::common::visibility::Visibility;
use mysql_queries::queries::generic_inference::api_providers::fal::insert_generic_inference_job_for_fal_queue::FalCategory;
use mysql_queries::queries::generic_inference::api_providers::fal::insert_generic_inference_job_for_fal_queue_with_apriori_job_token::{
insert_generic_inference_job_for_fal_queue_with_apriori_job_token,
InsertGenericInferenceForFalWithAprioriJobTokenArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use super::shared_job_args::SharedJobArgs;
pub struct InsertFalJobArgs<'a, 'tx> {
pub external_job_id: &'a str,
pub shared: SharedJobArgs<'a, 'tx>,
}
/// Insert a single Fal-queue inference job row for an image generation.
/// Mirrors `omni_gen::generate::video::insert_db_job::insert_fal_job::insert_fal_job`
/// but tags the row with `FalCategory::ImageGeneration` so the worker
/// routes it to the image-generation queue.
pub async fn insert_fal_job(args: InsertFalJobArgs<'_, '_>) -> Result<InferenceJobToken, CommonWebError> {
let InsertFalJobArgs {
external_job_id,
shared,
} = args;
let db_result = insert_generic_inference_job_for_fal_queue_with_apriori_job_token(
InsertGenericInferenceForFalWithAprioriJobTokenArgs {
apriori_job_token: shared.apriori_job_token,
uuid_idempotency_token: shared.idempotency_token,
maybe_external_third_party_id: external_job_id,
fal_category: FalCategory::ImageGeneration,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
creator_set_visibility: Visibility::Public,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: None,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
maybe_inference_args: None,
starting_job_status_override: None,
maybe_frontend_failure_category: None,
maybe_failure_reason: None,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => Ok(token),
Err(err) => {
warn!("Error inserting fal inference job (image): {:?}", err);
Err(CommonWebError::from_error(err))
}
}
}
@@ -1,117 +0,0 @@
use log::{error, warn};
use enums::common::visibility::Visibility;
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::{
insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token,
InsertGenericInferenceForSeedance2ProWithAprioriJobTokenArgs, KinoviVersion,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::wallet_ledger_entries::WalletLedgerEntryToken;
use uuid_utils::uuid::generate_random_uuid;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use super::shared_job_args::SharedJobArgs;
pub struct InsertSeedance2proJobsArgs<'a, 'tx> {
/// The order_id for the primary job (always inserted first).
pub primary_order_id: &'a str,
/// Additional order_ids returned by Kinovi when batch_count > 1. Each
/// becomes its own inference job row.
pub maybe_additional_order_ids: Option<&'a [String]>,
/// Wallet ledger entry from the up-front deduction. Stamped on each
/// inserted row so the worker can issue a refund on failure.
pub maybe_wallet_ledger_entry_token: Option<&'a WalletLedgerEntryToken>,
/// Which Kinovi account fulfilled the request — Volcengine, BytePlus,
/// or BytePlusUltra. For Midjourney image generation today this is
/// always Volcengine, but the helper stays version-agnostic so a future
/// account-picking layer can use it unchanged.
pub kinovi_version: KinoviVersion,
pub shared: SharedJobArgs<'a, 'tx>,
}
pub struct InsertSeedance2proJobsResult {
pub primary_job_token: InferenceJobToken,
pub all_job_tokens: Vec<InferenceJobToken>,
}
/// Insert one Seedance2Pro/Kinovi inference job per returned order_id.
/// Mirrors the video-side helper at
/// `omni_gen::generate::video::insert_db_job::insert_seedance2pro_jobs`.
pub async fn insert_seedance2pro_jobs(args: InsertSeedance2proJobsArgs<'_, '_>) -> Result<InsertSeedance2proJobsResult, CommonWebError> {
let InsertSeedance2proJobsArgs {
primary_order_id,
maybe_additional_order_ids,
maybe_wallet_ledger_entry_token,
kinovi_version,
mut shared,
} = args;
// Build a deduplicated list with primary_order_id first.
let mut all_order_ids = vec![primary_order_id.to_string()];
if let Some(additional) = maybe_additional_order_ids {
for id in additional {
if id != primary_order_id {
all_order_ids.push(id.clone());
}
}
}
let mut all_job_tokens: Vec<InferenceJobToken> = Vec::with_capacity(all_order_ids.len());
for (i, order_id) in all_order_ids.iter().enumerate() {
let job_token = if i == 0 { shared.apriori_job_token.clone() } else { InferenceJobToken::generate() };
// The idempotency_token column has a max length too tight for the
// `{idempotency}-batch-{i}` suffix pattern used elsewhere; mint a
// fresh UUID for each additional row instead. The primary row still
// wants a deterministic value, but historic behavior on the video
// side has been to mint UUIDs for all rows, which we mirror here.
let idempotency_str = generate_random_uuid();
let db_result = insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token(
InsertGenericInferenceForSeedance2ProWithAprioriJobTokenArgs {
kinovi_version,
apriori_job_token: &job_token,
uuid_idempotency_token: &idempotency_str,
maybe_external_third_party_id: order_id,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_wallet_ledger_entry_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
creator_set_visibility: Visibility::Public,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: None,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
maybe_inference_args: None,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => all_job_tokens.push(token),
Err(err) => {
warn!("Error inserting seedance2pro inference job (image, order_id={}): {:?}", order_id, err);
if i == 0 { return Err(CommonWebError::from_error(err)); }
}
}
}
let primary_job_token = all_job_tokens.first().cloned().ok_or_else(|| {
error!("No inference job token was created");
CommonWebError::server_error_with_message("No inference job token was created")
})?;
Ok(InsertSeedance2proJobsResult {
primary_job_token,
all_job_tokens,
})
}
@@ -1,3 +0,0 @@
pub mod insert_fal_job;
pub mod insert_seedance2pro_jobs;
pub mod shared_job_args;
@@ -1,22 +0,0 @@
use enums::common::generation::common_model_type::CommonModelType;
use enums::common::platform_type::PlatformType;
use tokens::tokens::anonymous_visitor_tracking::AnonymousVisitorTrackingToken;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use tokens::tokens::prompts::PromptToken;
use tokens::tokens::users::UserToken;
/// Fields every image-side `insert_*_job` helper needs.
/// Parallels `omni_gen::generate::video::insert_db_job::shared_job_args::SharedJobArgs`.
pub struct SharedJobArgs<'a, 'tx> {
pub apriori_job_token: &'a InferenceJobToken,
pub idempotency_token: &'a str,
pub user_token: &'a UserToken,
pub maybe_avt_token: Option<&'a AnonymousVisitorTrackingToken>,
pub maybe_prompt_token: Option<&'a PromptToken>,
pub maybe_debug_log_event_token: Option<&'a DebugLogEventToken>,
pub maybe_model_type: Option<CommonModelType>,
pub maybe_platform_type: Option<PlatformType>,
pub ip_address: &'a str,
pub transaction: &'a mut sqlx::Transaction<'tx, sqlx::MySql>,
}
@@ -1,5 +0,0 @@
pub mod omni_api_image_generate_handler;
pub(crate) mod hydrate_to_router_request;
pub(crate) mod insert_db_job;
pub(crate) mod pipeline_result;
pub mod pipeline_v2;
@@ -1,310 +0,0 @@
use std::sync::Arc;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, info, warn};
use sqlx::Acquire;
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_image_cost_and_generate_request::OmniGenImageCostAndGenerateRequest;
use artcraft_api_defs::omni_gen::generate_response::omni_gen_image_generate_response::OmniGenImageGenerateResponse;
use artcraft_router::generate::generate_image::generate_image_response::GenerateImageResponse;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use enums::by_table::prompt_context_items::prompt_context_semantic_type::PromptContextSemanticType;
use enums::by_table::prompts::prompt_type::PromptType;
use enums::common::generation::common_generation_mode::CommonGenerationMode;
use enums::common::generation::common_model_type::CommonModelType;
use enums::common::generation_provider::GenerationProvider;
use enums::common::platform_type::PlatformType;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use mysql_queries::queries::prompt_context_items::insert_batch_prompt_context_items::{
insert_batch_prompt_context_items, InsertBatchArgs, PromptContextItem,
};
use mysql_queries::queries::prompts::insert_prompt::{insert_prompt, InsertPromptArgs};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::endpoints::omni_api::generate::image::hydrate_to_router_request::hydrate_to_router_request;
use crate::http_server::endpoints::omni_api::generate::image::insert_db_job::insert_fal_job::{insert_fal_job, InsertFalJobArgs};
use crate::http_server::endpoints::omni_api::generate::image::insert_db_job::insert_seedance2pro_jobs::{insert_seedance2pro_jobs, InsertSeedance2proJobsArgs};
use crate::http_server::endpoints::omni_api::generate::image::insert_db_job::shared_job_args::SharedJobArgs;
use crate::http_server::endpoints::omni_api::generate::image::pipeline_v2::run_pipeline_v2::{run_pipeline_v2, RunPipelineV2Args};
use crate::http_server::session::lookup::user_session_feature_flags::UserSessionFeatureFlags;
use crate::http_server::user_lookup::api_keys::require_api_key_user::require_api_key_user;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
use crate::util::lookup::lookup_media_files_as_cdn_url_list_and_map::lookup_media_files_as_cdn_url_list_and_map;
/// Generate an image using the omni-api unified endpoint (API-key authenticated).
#[utoipa::path(
post,
tag = "Omni API",
path = "/v1/omni_api/generate/image",
request_body = OmniGenImageCostAndGenerateRequest,
responses(
(status = 200, description = "Success", body = OmniGenImageGenerateResponse),
(status = 400, description = "Bad input"),
(status = 401, description = "Unauthorized"),
(status = 402, description = "Payment required"),
(status = 500, description = "Server error"),
),
)]
pub async fn omni_api_image_generate_handler(
http_request: HttpRequest,
request: Json<OmniGenImageCostAndGenerateRequest>,
server_state: web::Data<Arc<ServerState>>,
) -> Result<Json<OmniGenImageGenerateResponse>, CommonWebError> {
info!("request: {:?}", request);
payments_error_test(&request.prompt.as_deref().unwrap_or(""))?;
let debug_log_event_token = DebugLogEventToken::generate();
let maybe_prompt_model_type: Option<CommonModelType> = request.model
.as_ref()
.map(|m| m.to_common_model_type());
// ==================== API KEY USER ==================== //
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
// API-key authentication (Authorization header) instead of a session cookie. Never cached.
let api_session = require_api_key_user(&http_request, &mut *mysql_connection).await?;
let user_token = &api_session.user_token;
// ==================== MODEL ACCESS CHECK ==================== //
// API-key users have no per-user feature flags today.
let user_feature_flags = UserSessionFeatureFlags::new(None);
// ==================== IDEMPOTENCY ==================== //
let idempotency_token = request.idempotency_token.as_deref()
.unwrap_or("")
.to_string();
if let Err(reason) = validate_idempotency_token_format(&idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
insert_idempotency_token(&idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
CommonWebError::BadInputWithSimpleMessage("repeated idempotency token".to_string())
})?;
// ==================== RESOLVE MEDIA TOKENS ==================== //
// Look up media file tokens BEFORE distilling. Pipeline execution should not do I/O.
let resolved_media = lookup_media_files_as_cdn_url_list_and_map(
&http_request,
&mut mysql_connection,
server_state.server_environment,
request.image_media_tokens.as_deref().unwrap_or(&[]),
).await?;
// ==================== HYDRATE ROUTER REQUEST ==================== //
let router_builder = hydrate_to_router_request(&request)?;
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
message: &serde_json::to_string(&*request).unwrap_or_default(),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert HTTP request debug log: {:?}", err);
}
// ==================== PIPELINE ==================== //
// NB: Release the pooled DB connection before the (slow, external) generation call so we don't
// hold a pool slot idle while waiting on the provider — that's what starves the pool and causes
// PoolTimedOut on unrelated endpoints. We re-acquire below to write the result.
drop(mysql_connection);
let pipeline_result = run_pipeline_v2(RunPipelineV2Args {
router_builder: &router_builder,
server_state: &server_state,
user_token,
resolved_media: &resolved_media,
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
// ==================== DEBUG LOG: FAL REQUEST ==================== //
if let GenerateImageResponse::Fal(ref fal_payload) = pipeline_result.response {
if let Some(ref outbound_request) = fal_payload.maybe_outbound_request {
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert Fal request debug log: {:?}", err);
}
}
}
// ==================== WRITE RESULT ==================== //
let ip_address = get_request_ip(&http_request);
// Omni API requests are always API-key authenticated; hardcode the platform type.
let maybe_platform_type = Some(PlatformType::ApiKey);
let mut transaction = mysql_connection
.begin()
.await
.map_err(|err| {
error!("Error starting MySQL transaction: {:?}", err);
CommonWebError::from_error(err)
})?;
// -- Prompt --
let generation_mode = if request.image_media_tokens.is_some() {
CommonGenerationMode::Edit
} else {
CommonGenerationMode::Text
};
let prompt_result = insert_prompt(InsertPromptArgs {
maybe_apriori_prompt_token: None,
prompt_type: PromptType::ArtcraftApp,
maybe_creator_user_token: Some(user_token),
maybe_model_type: maybe_prompt_model_type,
maybe_generation_provider: Some(GenerationProvider::Artcraft),
maybe_positive_prompt: request.prompt.as_deref(),
maybe_negative_prompt: None,
maybe_other_args: None,
maybe_generation_mode: Some(generation_mode),
maybe_aspect_ratio: request.aspect_ratio, // TODO: should be saved from router's decision as it could have changed
maybe_resolution: request.resolution,// TODO: should be saved from router's decision as it could have changed
maybe_batch_count: request.image_batch_count.map(|c| c as u8),
maybe_generate_audio: None, // NB: Images, not video
maybe_duration_seconds: None, // NB: Images, not video
creator_ip_address: &ip_address,
mysql_executor: &mut *transaction,
phantom: Default::default(),
}).await;
let prompt_token = match prompt_result {
Ok(token) => Some(token),
Err(err) => {
warn!("Error inserting prompt: {:?}", err);
None
}
};
// -- Prompt context items --
if let Some(token) = prompt_token.as_ref() {
let mut context_items = Vec::new();
if let Some(ref_tokens) = &request.image_media_tokens {
for media_token in ref_tokens {
context_items.push(PromptContextItem {
media_token: media_token.clone(),
context_semantic_type: PromptContextSemanticType::Imgref,
});
}
}
if !context_items.is_empty() {
if let Err(err) = insert_batch_prompt_context_items(InsertBatchArgs {
prompt_token: token.clone(),
items: context_items,
transaction: &mut transaction,
}).await {
warn!("Error inserting batch prompt context items: {:?}", err);
}
}
}
// -- Inference job --
//
// Each provider has its own queue / worker. Branch by response variant
// so the row lands on the correct queue (the Seedance2Pro/Kinovi worker
// for Midjourney; the Fal worker for everything else).
let job_token: InferenceJobToken = match &pipeline_result.response {
GenerateImageResponse::Seedance2Pro(payload) => {
info!("Inserting seedance2pro image job(s) with token: {:?}", pipeline_result.apriori_job_token);
// The image-side omni pipeline always dispatches Midjourney via the
// Volcengine Kinovi account today. If we ever route to BytePlus /
// BytePlus Ultra here, mirror the video-side `kinovi_account` knob.
let kinovi_version = KinoviVersion::Volcengine;
let result = insert_seedance2pro_jobs(InsertSeedance2proJobsArgs {
primary_order_id: &payload.order_id,
maybe_additional_order_ids: payload.maybe_order_ids.as_deref(),
maybe_wallet_ledger_entry_token: pipeline_result.maybe_wallet_ledger_entry_token.as_ref(),
kinovi_version,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?;
result.primary_job_token
}
GenerateImageResponse::Fal(payload) => {
info!("Inserting fal image job with token: {:?}", pipeline_result.apriori_job_token);
let external_job_id = payload.request_id.clone().unwrap_or_default();
insert_fal_job(InsertFalJobArgs {
external_job_id: &external_job_id,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?
}
GenerateImageResponse::Artcraft(payload) => {
// The omni image pipeline never dispatches via the Artcraft provider
// itself today (everything routes to Fal or Kinovi), but the response
// variant exists so we cover it defensively — Artcraft jobs come back
// already inserted server-side, so just propagate the token.
payload.inference_job_token.clone()
}
};
transaction.commit().await.map_err(|err| {
error!("Error committing transaction: {:?}", err);
CommonWebError::from_error(err)
})?;
Ok(Json(OmniGenImageGenerateResponse {
success: true,
inference_job_token: job_token,
}))
}
@@ -1,15 +0,0 @@
use artcraft_router::generate::generate_image::generate_image_response::GenerateImageResponse;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::wallet_ledger_entries::WalletLedgerEntryToken;
/// Both image pipelines produce this, then the shared handler suffix writes DB rows.
pub struct ImagePipelineResult {
pub apriori_job_token: InferenceJobToken,
pub response: GenerateImageResponse,
/// The wallet ledger entry created when the user was charged for this
/// generation, if any (None when the generation is free). Used as a
/// foreign key on the inference job row so the worker can refund on
/// failure.
pub maybe_wallet_ledger_entry_token: Option<WalletLedgerEntryToken>,
}
@@ -1,208 +0,0 @@
use log::{info, warn};
use artcraft_router::api::image_list_ref::ImageListRef;
use artcraft_router::api::router_image_model::RouterImageModel;
use artcraft_router::api::router_provider::RouterProvider;
use artcraft_router::client::router_client::RouterClient;
use artcraft_router::client::router_fal_client::RouterFalClient;
use artcraft_router::client::router_seedance2pro_client::RouterSeedance2ProClient;
use artcraft_router::generate::generate_image::generate_image_request_builder::GenerateImageRequestBuilder;
use artcraft_router::generate::generate_image::generate_image_response::GenerateImageResponse;
use artcraft_router::generate::generate_image::image_generation_draft_context::ImageGenerationDraftContext;
use artcraft_router::generate::generate_image::image_generation_draft_or_request::ImageGenerationDraftOrRequest;
use artcraft_router::generate::generate_image::image_generation_request::ImageGenerationRequest;
use seedance2pro_client::creds::seedance2pro_session::Seedance2ProSession;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::users::UserToken;
use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_else_common_web_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::omni_api::generate::image::pipeline_result::ImagePipelineResult;
use crate::http_server::endpoints::omni_api::shared_utils::map_seedance2pro_router_error::map_router_error_to_web_error;
use crate::state::server_state::ServerState;
use crate::util::lookup::lookup_media_files_as_cdn_url_list_and_map::MediaFilesAsCdnUrlListAndMap;
pub struct RunPipelineV2Args<'a> {
pub router_builder: &'a GenerateImageRequestBuilder,
pub server_state: &'a ServerState,
pub user_token: &'a UserToken,
pub resolved_media: &'a MediaFilesAsCdnUrlListAndMap,
}
// NB: This pipeline does an external generation call (`finalize_and_generate`) that can take many
// seconds. It deliberately does NOT hold a pooled DB connection across that call — it acquires a
// short-lived connection only for the wallet deduction. Holding a pooled connection across the
// external call is what starves the pool and causes `PoolTimedOut`.
pub async fn run_pipeline_v2(
args: RunPipelineV2Args<'_>,
) -> Result<ImagePipelineResult, CommonWebError> {
let RunPipelineV2Args {
router_builder,
server_state,
user_token,
resolved_media,
} = args;
let hydrated_builder = apply_hydrated_media_inputs(
router_builder,
resolved_media,
);
let cost = estimate_cost_in_credits(&router_builder)?;
let draft_or_request = build_execution_request(&hydrated_builder)?;
info!("Charging wallet: {} credits", cost);
let apriori_job_token = InferenceJobToken::generate();
let maybe_wallet_ledger_entry_token = if cost > 0 {
// NB: Short-lived connection — released before the external call below.
let mut billing_connection = server_state.mysql_pool.acquire().await
.map_err(|err| {
warn!("Failed to acquire MySQL connection for image billing: {:?}", err);
CommonWebError::from_error(err)
})?;
let deduction = attempt_wallet_deduction_else_common_web_error(
user_token,
Some(apriori_job_token.as_str()),
cost,
&mut billing_connection,
).await?;
Some(deduction.ledger_entry_token)
} else {
None
};
// NB: No pooled DB connection is held across this external generation call.
let response = finalize_and_generate(draft_or_request, server_state, resolved_media).await?;
Ok(ImagePipelineResult {
apriori_job_token,
response,
maybe_wallet_ledger_entry_token,
})
}
fn build_execution_request(
router_builder: &GenerateImageRequestBuilder,
) -> Result<ImageGenerationDraftOrRequest, CommonWebError> {
let mut execution_builder = router_builder.clone();
execution_builder.provider = provider_for_model(execution_builder.model);
execution_builder.build2().map_err(|e| {
warn!("Failed to build2 for image v2 pipeline: {}", e);
CommonWebError::from_error(e)
})
}
/// Route each image model to the provider that fulfils it. Midjourney is
/// served via the Seedance2Pro/Kinovi (Volcengine) backend; everything else
/// flows through Fal.
fn provider_for_model(model: RouterImageModel) -> RouterProvider {
match model {
RouterImageModel::Midjourney7
| RouterImageModel::Midjourney7Niji
| RouterImageModel::Midjourney8 => RouterProvider::Seedance2Pro,
_ => RouterProvider::Fal,
}
}
fn apply_hydrated_media_inputs(
router_builder: &GenerateImageRequestBuilder,
resolved_media: &MediaFilesAsCdnUrlListAndMap,
) -> GenerateImageRequestBuilder {
let mut hydrated_builder = router_builder.clone();
match hydrated_builder.image_inputs.as_ref() {
Some(ImageListRef::MediaFileTokens(tokens)) if !tokens.is_empty() => {
hydrated_builder.image_inputs = Some(ImageListRef::Urls(
resolved_media.ordered_url_list.clone(),
));
},
_ => {},
}
hydrated_builder
}
fn estimate_cost_in_credits(
router_builder: &GenerateImageRequestBuilder,
) -> Result<u64, CommonWebError> {
let mut cost_builder = router_builder.clone();
cost_builder.provider = RouterProvider::Artcraft;
let request = cost_builder.build2().map_err(|e| {
warn!("Failed to build image cost request: {}", e);
CommonWebError::from_error(e)
})?;
let cost = request.estimate_cost().map_err(|e| {
warn!("Failed to estimate image cost: {}", e);
CommonWebError::from_error(e)
})?;
Ok(cost.cost_in_credits.unwrap_or(0))
}
async fn finalize_and_generate(
draft_or_request: ImageGenerationDraftOrRequest,
server_state: &ServerState,
resolved_media: &MediaFilesAsCdnUrlListAndMap,
) -> Result<GenerateImageResponse, CommonWebError> {
let provider = draft_or_request.get_provider();
let client = build_router_client(provider, server_state)?;
// Inline the draft finalization so the draft context can borrow `client`.
let request: ImageGenerationRequest = match draft_or_request {
ImageGenerationDraftOrRequest::Request(request) => request,
ImageGenerationDraftOrRequest::Draft(draft) => {
let draft_context = ImageGenerationDraftContext {
client: Some(&client),
media_file_to_artcraft_url_map: Some(&resolved_media.token_to_url_map),
};
draft.finalize(draft_context)
.await
.map_err(|err| {
warn!("Failed to finalize image v2 draft: {:?}", err);
map_router_error_to_web_error(err)
})?
}
};
request.send_request(&client)
.await
.map_err(|err| {
warn!("v2 image generation failed: {:?}", err);
map_router_error_to_web_error(err)
})
}
fn build_router_client(
provider: RouterProvider,
server_state: &ServerState,
) -> Result<RouterClient, CommonWebError> {
match provider {
RouterProvider::Fal => {
let fal_client = RouterFalClient::new_with_webhook(
server_state.inference_providers.fal.api_key.clone(),
server_state.inference_providers.fal.webhook_url.clone(),
);
Ok(RouterClient::Fal(fal_client))
},
RouterProvider::Seedance2Pro => {
// Midjourney image generation always uses the Volcengine account.
// Per-request account selection (like the video flow's
// `KinoviAccount` enum) isn't needed here yet — if it ever is, plumb
// a parallel knob through `RunPipelineV2Args`.
let cookies = server_state.inference_providers.seedance2pro.cookies_volcengine.clone();
let session = Seedance2ProSession::from_cookies_string(cookies);
Ok(RouterClient::Seedance2Pro(RouterSeedance2ProClient::new(session)))
},
other => {
Err(CommonWebError::server_error_with_message(
&format!("Unsupported provider for image v2 generation: {:?}", other),
))
},
}
}
@@ -1,2 +0,0 @@
pub mod image;
pub mod video;
@@ -1,44 +0,0 @@
//! Shared wallet billing logic for both video generation pipelines.
use log::info;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::users::UserToken;
use tokens::tokens::wallet_ledger_entries::WalletLedgerEntryToken;
use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_else_common_web_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
pub struct BillWalletResult {
pub apriori_job_token: InferenceJobToken,
pub maybe_wallet_ledger_entry_token: Option<WalletLedgerEntryToken>,
}
/// Generate an apriori job token and bill the user's wallet for the given cost.
///
/// If cost is 0, no wallet deduction is made but the apriori token is still generated.
pub async fn bill_wallet(
user_token: &UserToken,
cost: u64,
mysql_connection: &mut sqlx::pool::PoolConnection<sqlx::MySql>,
) -> Result<BillWalletResult, CommonWebError> {
let apriori_job_token = InferenceJobToken::generate();
info!("Charging wallet: {} credits", cost);
let maybe_wallet_ledger_entry_token = if cost > 0 {
let deduction_result = attempt_wallet_deduction_else_common_web_error(
user_token,
Some(apriori_job_token.as_str()),
cost,
mysql_connection,
).await?;
Some(deduction_result.ledger_entry_token)
} else {
None
};
Ok(BillWalletResult {
apriori_job_token,
maybe_wallet_ledger_entry_token,
})
}
@@ -1,61 +0,0 @@
//! Build a RouterClient for the given provider from server state.
use artcraft_router::api::router_provider::RouterProvider;
use artcraft_router::client::router_client::RouterClient;
use artcraft_router::client::router_fal_client::RouterFalClient;
use artcraft_router::client::router_gmicloud_client::RouterGmiCloudClient;
use artcraft_router::client::router_grok_api_client::RouterGrokApiClient;
use artcraft_router::client::router_seedance2pro_client::RouterSeedance2ProClient;
use seedance2pro_client::creds::seedance2pro_session::Seedance2ProSession;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::omni_api::generate::video::kinovi_account::KinoviAccount;
use crate::state::server_state::ServerState;
pub fn build_router_client(
provider: RouterProvider,
server_state: &ServerState,
kinovi_account: KinoviAccount,
) -> Result<RouterClient, CommonWebError> {
match provider {
RouterProvider::Seedance2Pro => {
kinovi_provider(server_state, kinovi_account)
}
RouterProvider::Fal => {
let fal_client = RouterFalClient::new_with_webhook(
server_state.inference_providers.fal.api_key.clone(),
server_state.inference_providers.fal.webhook_url.clone(),
);
Ok(RouterClient::Fal(fal_client))
}
RouterProvider::GmiCloud => {
Ok(RouterClient::GmiCloud(RouterGmiCloudClient::new(
server_state.inference_providers.gmicloud.api_key.clone(),
)))
}
RouterProvider::GrokApi => {
Ok(RouterClient::GrokApi(RouterGrokApiClient::new(
server_state.inference_providers.grok_api.api_key.clone(),
)))
}
other => {
Err(CommonWebError::server_error_with_message(
&format!("Unsupported provider for video generation: {:?}", other),
))
}
}
}
fn kinovi_provider(server_state: &ServerState, kinovi_account: KinoviAccount) -> Result<RouterClient, CommonWebError> {
let seedance2pro = &server_state.inference_providers.seedance2pro;
let cookies = match kinovi_account {
KinoviAccount::Volcengine => seedance2pro.cookies_volcengine.clone(),
KinoviAccount::BytePlus => seedance2pro.cookies_byteplus.clone(),
KinoviAccount::BytePlusUltra => seedance2pro.cookies_byteplus_ultra.clone(),
};
let session = Seedance2ProSession::from_cookies_string(cookies);
Ok(RouterClient::Seedance2Pro(RouterSeedance2ProClient::new(session)))
}
@@ -1,97 +0,0 @@
use crate::http_server::common_responses::common_web_error::CommonWebError;
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_video_cost_and_generate_request::OmniGenVideoCostAndGenerateRequest;
use artcraft_router::api::audio_list_ref::AudioListRef;
use artcraft_router::api::character_list_ref::CharacterListRef;
use artcraft_router::api::router_aspect_ratio::RouterAspectRatio;
use artcraft_router::api::router_resolution::RouterResolution;
use artcraft_router::api::router_video_model::RouterVideoModel;
use artcraft_router::api::image_list_ref::ImageListRef;
use artcraft_router::api::image_ref::ImageRef;
use artcraft_router::api::router_provider::RouterProvider;
use artcraft_router::api::video_list_ref::VideoListRef;
use artcraft_router::client::request_mismatch_mitigation_strategy::RequestMismatchMitigationStrategy;
use artcraft_router::generate::generate_video::generate_video_request_builder::GenerateVideoRequestBuilder;
use enums::common::generation::common_aspect_ratio::CommonAspectRatio as CommonAspectRatioEnum;
use enums::common::generation::common_resolution::CommonResolution as CommonResolutionEnum;
use enums::common::generation::common_video_model::CommonVideoModel as CommonVideoModelEnum;
pub fn hydrate_to_router_request(
request: &OmniGenVideoCostAndGenerateRequest,
) -> Result<GenerateVideoRequestBuilder, CommonWebError> {
let api_model = request.model
.as_ref()
.ok_or_else(|| CommonWebError::BadInputWithSimpleMessage(
"model is required".to_string(),
))?;
let model = convert_model(api_model)?;
let aspect_ratio = request.aspect_ratio
.as_ref()
.map(convert_aspect_ratio)
.transpose()?;
let resolution = request.resolution
.as_ref()
.map(convert_resolution)
.transpose()?;
Ok(GenerateVideoRequestBuilder {
model,
provider: RouterProvider::Artcraft,
prompt: request.prompt.clone(),
negative_prompt: request.negative_prompt.clone(),
start_frame: request.start_frame_image_media_token.clone()
.map(ImageRef::MediaFileToken),
end_frame: request.end_frame_image_media_token.clone()
.map(ImageRef::MediaFileToken),
reference_images: request.reference_image_media_tokens.clone()
.map(ImageListRef::MediaFileTokens),
reference_videos: request.reference_video_media_tokens.clone()
.map(VideoListRef::MediaFileTokens),
reference_audio: request.reference_audio_media_tokens.clone()
.map(AudioListRef::MediaFileTokens),
reference_character_tokens: request.reference_character_tokens.clone()
.map(CharacterListRef::CharacterTokens),
resolution,
aspect_ratio,
duration_seconds: request.duration_seconds,
video_batch_count: request.video_batch_count,
generate_audio: request.generate_audio,
request_mismatch_mitigation_strategy: RequestMismatchMitigationStrategy::PayMoreUpgrade,
idempotency_token: request.idempotency_token.clone(),
})
}
fn convert_model(
model: &CommonVideoModelEnum,
) -> Result<RouterVideoModel, CommonWebError> {
let json = serde_json::to_string(model)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported video model: {}", e),
)
})
}
fn convert_aspect_ratio(
ar: &CommonAspectRatioEnum,
) -> Result<RouterAspectRatio, CommonWebError> {
let json = serde_json::to_string(ar)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported aspect ratio: {}", e),
)
})
}
fn convert_resolution(
res: &CommonResolutionEnum,
) -> Result<RouterResolution, CommonWebError> {
let json = serde_json::to_string(res)?;
serde_json::from_str(&json).map_err(|e| {
CommonWebError::BadInputWithSimpleMessage(
format!("Unsupported resolution: {}", e),
)
})
}
@@ -1,6 +0,0 @@
pub mod bill_wallet;
pub mod build_router_client;
pub(crate) mod hydrate_router_request;
pub mod pipeline_result;
pub mod resolve_kinovi_character_ids;
pub mod resolve_media_tokens_to_urls;
@@ -1,14 +0,0 @@
use artcraft_router::generate::generate_video::generate_video_response::GenerateVideoResponse;
use mysql_queries::queries::generic_inference::common::job_cost_estimates::JobCostEstimates;
use crate::http_server::endpoints::omni_api::generate::video::helpers::bill_wallet::BillWalletResult;
/// Both pipelines produce this, then the shared suffix handles DB writes.
pub struct PipelineResult {
pub billing: BillWalletResult,
pub response: GenerateVideoResponse,
/// System (user-facing) and provider-side cost estimates, written onto the
/// generic_inference_jobs rows.
pub cost_estimates: JobCostEstimates,
}
@@ -1,51 +0,0 @@
//! Resolve character tokens to Kinovi character IDs for prompting.
use std::collections::HashMap;
use log::warn;
use sqlx::MySql;
use mysql_queries::queries::characters::batch_lookup_characters_by_token_for_prompting::batch_lookup_characters_by_token_for_prompting;
use tokens::tokens::characters::CharacterToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
/// Resolve character tokens to a map of CharacterToken → Kinovi character ID.
///
/// Looks up the characters, filters to active ones with kinovi IDs, and warns about
/// any that are missing or inactive (but doesn't fail the request).
pub async fn resolve_kinovi_character_ids(
maybe_tokens: Option<&[CharacterToken]>,
connection: &mut sqlx::pool::PoolConnection<MySql>,
) -> Result<Option<HashMap<CharacterToken, String>>, CommonWebError> {
let tokens = match maybe_tokens {
None => return Ok(None),
Some(tokens) if tokens.is_empty() => return Ok(None),
Some(tokens) => tokens,
};
let characters = batch_lookup_characters_by_token_for_prompting(tokens, connection)
.await?;
if characters.len() != tokens.len() {
warn!(
"Not all character tokens were found: requested {}, found {}",
tokens.len(), characters.len(),
);
}
for character in &characters {
if !character.is_active {
warn!("Character {} is not yet active, skipping", character.token);
}
}
let map: HashMap<CharacterToken, String> = characters.iter()
.filter(|c| c.is_active)
.filter_map(|c| {
c.kinovi_character_id.as_ref().map(|kid| (c.token.clone(), kid.clone()))
})
.collect();
if map.is_empty() { Ok(None) } else { Ok(Some(map)) }
}
@@ -1,64 +0,0 @@
use std::collections::HashMap;
use artcraft_router::api::audio_list_ref::AudioListRef;
use artcraft_router::api::image_list_ref::ImageListRef;
use artcraft_router::api::image_ref::ImageRef;
use artcraft_router::api::video_list_ref::VideoListRef;
use artcraft_router::generate::generate_video::generate_video_request_builder::GenerateVideoRequestBuilder;
use tokens::tokens::media_files::MediaFileToken;
/// For providers that take URLs directly (GmiCloud, Fal), swap
/// `ImageRef::MediaFileToken` → `ImageRef::Url` using the resolved map.
pub fn resolve_media_tokens_to_urls(
builder: &mut GenerateVideoRequestBuilder,
url_map: Option<&HashMap<MediaFileToken, String>>,
) {
let map = match url_map {
Some(m) => m,
None => return,
};
// start_frame
if let Some(ImageRef::MediaFileToken(ref token)) = builder.start_frame {
if let Some(url) = map.get(token) {
builder.start_frame = Some(ImageRef::Url(url.clone()));
}
}
// end_frame
if let Some(ImageRef::MediaFileToken(ref token)) = builder.end_frame {
if let Some(url) = map.get(token) {
builder.end_frame = Some(ImageRef::Url(url.clone()));
}
}
// reference_images
if let Some(ImageListRef::MediaFileTokens(ref tokens)) = builder.reference_images {
let urls: Vec<String> = tokens.iter()
.filter_map(|t| map.get(t).cloned())
.collect();
if !urls.is_empty() {
builder.reference_images = Some(ImageListRef::Urls(urls));
}
}
// reference_videos
if let Some(VideoListRef::MediaFileTokens(ref tokens)) = builder.reference_videos {
let urls: Vec<String> = tokens.iter()
.filter_map(|t| map.get(t).cloned())
.collect();
if !urls.is_empty() {
builder.reference_videos = Some(VideoListRef::Urls(urls));
}
}
// reference_audio
if let Some(AudioListRef::MediaFileTokens(ref tokens)) = builder.reference_audio {
let urls: Vec<String> = tokens.iter()
.filter_map(|t| map.get(t).cloned())
.collect();
if !urls.is_empty() {
builder.reference_audio = Some(AudioListRef::Urls(urls));
}
}
}
@@ -1,57 +0,0 @@
use log::warn;
use enums::common::visibility::Visibility;
use mysql_queries::queries::generic_inference::api_providers::fal::insert_generic_inference_job_for_fal_queue::FalCategory;
use mysql_queries::queries::generic_inference::api_providers::fal::insert_generic_inference_job_for_fal_queue_with_apriori_job_token::{
insert_generic_inference_job_for_fal_queue_with_apriori_job_token,
InsertGenericInferenceForFalWithAprioriJobTokenArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use super::shared_job_args::SharedJobArgs;
pub struct InsertFalJobArgs<'a, 'tx> {
pub external_job_id: &'a str,
pub shared: SharedJobArgs<'a, 'tx>,
}
pub async fn insert_fal_job(args: InsertFalJobArgs<'_, '_>) -> Result<InferenceJobToken, CommonWebError> {
let InsertFalJobArgs {
external_job_id,
shared,
} = args;
let db_result = insert_generic_inference_job_for_fal_queue_with_apriori_job_token(
InsertGenericInferenceForFalWithAprioriJobTokenArgs {
apriori_job_token: shared.apriori_job_token,
uuid_idempotency_token: shared.idempotency_token,
maybe_external_third_party_id: external_job_id,
fal_category: FalCategory::VideoGeneration,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
creator_set_visibility: Visibility::Public,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: shared.maybe_cost_estimates,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
maybe_inference_args: None,
starting_job_status_override: None,
maybe_frontend_failure_category: None,
maybe_failure_reason: None,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => Ok(token),
Err(err) => {
warn!("Error inserting fal inference job: {:?}", err);
Err(CommonWebError::from_error(err))
}
}
}
@@ -1,48 +0,0 @@
use log::warn;
use mysql_queries::queries::generic_inference::api_providers::gmicloud::insert_generic_inference_job_for_gmicloud::{
insert_generic_inference_job_for_gmicloud, InsertGmiCloudInferenceJobArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use super::shared_job_args::SharedJobArgs;
pub struct InsertGmiCloudJobArgs<'a, 'tx> {
pub external_request_id: &'a str,
pub shared: SharedJobArgs<'a, 'tx>,
}
pub async fn insert_gmicloud_job(args: InsertGmiCloudJobArgs<'_, '_>) -> Result<InferenceJobToken, CommonWebError> {
let InsertGmiCloudJobArgs {
external_request_id,
shared,
} = args;
let db_result = insert_generic_inference_job_for_gmicloud(
InsertGmiCloudInferenceJobArgs {
apriori_job_token: shared.apriori_job_token,
uuid_idempotency_token: shared.idempotency_token,
external_request_id,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: shared.maybe_cost_estimates,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => Ok(token),
Err(err) => {
warn!("Error inserting GmiCloud inference job: {:?}", err);
Err(CommonWebError::from_error(err))
}
}
}
@@ -1,48 +0,0 @@
use log::warn;
use mysql_queries::queries::generic_inference::api_providers::grok_api::insert_generic_inference_job_for_grok_api::{
insert_generic_inference_job_for_grok_api, InsertGrokApiInferenceJobArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use super::shared_job_args::SharedJobArgs;
pub struct InsertGrokApiJobArgs<'a, 'tx> {
pub external_request_id: &'a str,
pub shared: SharedJobArgs<'a, 'tx>,
}
pub async fn insert_grok_api_job(args: InsertGrokApiJobArgs<'_, '_>) -> Result<InferenceJobToken, CommonWebError> {
let InsertGrokApiJobArgs {
external_request_id,
shared,
} = args;
let db_result = insert_generic_inference_job_for_grok_api(
InsertGrokApiInferenceJobArgs {
apriori_job_token: shared.apriori_job_token,
uuid_idempotency_token: shared.idempotency_token,
external_request_id,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: shared.maybe_cost_estimates,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => Ok(token),
Err(err) => {
warn!("Error inserting Grok (xAI) API inference job: {:?}", err);
Err(CommonWebError::from_error(err))
}
}
}
@@ -1,95 +0,0 @@
use log::{error, warn};
use crate::http_server::common_responses::common_web_error::CommonWebError;
use enums::common::visibility::Visibility;
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::{insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token, InsertGenericInferenceForSeedance2ProWithAprioriJobTokenArgs, KinoviVersion};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::wallet_ledger_entries::WalletLedgerEntryToken;
use uuid_utils::uuid::generate_random_uuid;
use super::shared_job_args::SharedJobArgs;
pub struct InsertSeedance2proJobsArgs<'a, 'tx> {
pub primary_order_id: &'a str,
pub maybe_additional_order_ids: Option<&'a [String]>,
pub maybe_wallet_ledger_entry_token: Option<&'a WalletLedgerEntryToken>,
pub kinovi_version: KinoviVersion,
pub shared: SharedJobArgs<'a, 'tx>,
}
pub struct InsertSeedance2proJobsResult {
pub primary_job_token: InferenceJobToken,
pub all_job_tokens: Vec<InferenceJobToken>,
}
pub async fn insert_seedance2pro_jobs(args: InsertSeedance2proJobsArgs<'_, '_>) -> Result<InsertSeedance2proJobsResult, CommonWebError> {
let InsertSeedance2proJobsArgs {
primary_order_id,
maybe_additional_order_ids,
maybe_wallet_ledger_entry_token,
kinovi_version,
mut shared,
} = args;
// Build a deduplicated list with primary_order_id first.
let mut all_order_ids = vec![primary_order_id.to_string()];
if let Some(additional) = maybe_additional_order_ids {
for id in additional {
if id != primary_order_id {
all_order_ids.push(id.clone());
}
}
}
let mut all_job_tokens: Vec<InferenceJobToken> = Vec::with_capacity(all_order_ids.len());
for (i, order_id) in all_order_ids.iter().enumerate() {
let job_token = if i == 0 { shared.apriori_job_token.clone() } else { InferenceJobToken::generate() };
// TODO: These strings are too long for the database:
//let idempotency_str = if i == 0 { shared.idempotency_token.to_string() } else { format!("{}-batch-{}", shared.idempotency_token, i) };
let idempotency_str = generate_random_uuid();
let db_result = insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token(
InsertGenericInferenceForSeedance2ProWithAprioriJobTokenArgs {
kinovi_version,
apriori_job_token: &job_token,
uuid_idempotency_token: &idempotency_str,
maybe_external_third_party_id: order_id,
maybe_model_type: shared.maybe_model_type,
maybe_prompt_token: shared.maybe_prompt_token,
maybe_wallet_ledger_entry_token,
maybe_creator_user_token: Some(shared.user_token),
maybe_avt_token: shared.maybe_avt_token,
creator_ip_address: shared.ip_address,
creator_set_visibility: Visibility::Public,
maybe_platform_type: shared.maybe_platform_type,
maybe_cost_estimates: shared.maybe_cost_estimates,
maybe_debug_log_event_token: shared.maybe_debug_log_event_token,
mysql_executor: &mut **shared.transaction,
maybe_inference_args: None,
phantom: Default::default(),
}
).await;
match db_result {
Ok(token) => all_job_tokens.push(token),
Err(err) => {
warn!("Error inserting seedance2pro inference job (order_id={}): {:?}", order_id, err);
if i == 0 { return Err(CommonWebError::from_error(err)); }
}
}
}
let primary_job_token = all_job_tokens.first().cloned().ok_or_else(|| {
error!("No inference job token was created");
CommonWebError::server_error_with_message("No inference job token was created")
})?;
Ok(InsertSeedance2proJobsResult {
primary_job_token,
all_job_tokens,
})
}
@@ -1,5 +0,0 @@
pub mod insert_fal_job;
pub mod insert_gmicloud_job;
pub mod insert_grok_api_job;
pub mod insert_seedance2pro_jobs;
pub mod shared_job_args;
@@ -1,22 +0,0 @@
use enums::common::generation::common_model_type::CommonModelType;
use enums::common::platform_type::PlatformType;
use mysql_queries::queries::generic_inference::common::job_cost_estimates::JobCostEstimates;
use tokens::tokens::anonymous_visitor_tracking::AnonymousVisitorTrackingToken;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use tokens::tokens::prompts::PromptToken;
use tokens::tokens::users::UserToken;
pub struct SharedJobArgs<'a, 'tx> {
pub apriori_job_token: &'a InferenceJobToken,
pub idempotency_token: &'a str,
pub user_token: &'a UserToken,
pub maybe_avt_token: Option<&'a AnonymousVisitorTrackingToken>,
pub maybe_prompt_token: Option<&'a PromptToken>,
pub maybe_debug_log_event_token: Option<&'a DebugLogEventToken>,
pub maybe_model_type: Option<CommonModelType>,
pub maybe_platform_type: Option<PlatformType>,
pub maybe_cost_estimates: Option<JobCostEstimates>,
pub ip_address: &'a str,
pub transaction: &'a mut sqlx::Transaction<'tx, sqlx::MySql>,
}
@@ -1,7 +0,0 @@
#[derive(Debug, Clone, Copy)]
pub enum KinoviAccount {
Volcengine,
BytePlus,
BytePlusUltra,
}
@@ -1,5 +0,0 @@
pub mod omni_api_video_generate_handler;
pub mod helpers;
pub mod pipeline_v2;
pub mod insert_db_job;
pub mod kinovi_account;
@@ -1,474 +0,0 @@
use std::collections::HashMap;
use std::sync::Arc;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, info, warn};
use sqlx::Acquire;
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_video_cost_and_generate_request::OmniGenVideoCostAndGenerateRequest;
use artcraft_api_defs::omni_gen::generate_response::omni_gen_video_generate_response::OmniGenVideoGenerateResponse;
use artcraft_router::generate::generate_video::generate_video_response::GenerateVideoResponse;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use enums::by_table::prompt_context_items::prompt_context_semantic_type::PromptContextSemanticType;
use enums::by_table::prompts::prompt_type::PromptType;
use enums::common::generation::common_generation_mode::CommonGenerationMode;
use enums::common::generation::common_model_type::CommonModelType;
use enums::common::generation::common_video_model::CommonVideoModel;
use enums::common::generation_provider::GenerationProvider;
use enums::common::platform_type::PlatformType;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use mysql_queries::queries::prompt_context_items::insert_batch_prompt_context_items::{
insert_batch_prompt_context_items, InsertBatchArgs, PromptContextItem,
};
use mysql_queries::queries::prompts::insert_prompt::{insert_prompt, InsertPromptArgs};
use tokens::tokens::characters::CharacterToken;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::endpoints::omni_api::generate::video::helpers::hydrate_router_request::hydrate_to_router_request;
use crate::http_server::endpoints::omni_api::generate::video::helpers::resolve_kinovi_character_ids::resolve_kinovi_character_ids;
use crate::http_server::endpoints::omni_api::generate::video::insert_db_job::insert_fal_job::{insert_fal_job, InsertFalJobArgs};
use crate::http_server::endpoints::omni_api::generate::video::insert_db_job::insert_gmicloud_job::{insert_gmicloud_job, InsertGmiCloudJobArgs};
use crate::http_server::endpoints::omni_api::generate::video::insert_db_job::insert_grok_api_job::{insert_grok_api_job, InsertGrokApiJobArgs};
use crate::http_server::endpoints::omni_api::generate::video::insert_db_job::insert_seedance2pro_jobs::{insert_seedance2pro_jobs, InsertSeedance2proJobsArgs};
use crate::http_server::endpoints::omni_api::generate::video::insert_db_job::shared_job_args::SharedJobArgs;
use crate::http_server::endpoints::omni_api::generate::video::kinovi_account::KinoviAccount;
use crate::http_server::endpoints::omni_api::generate::video::pipeline_v2::run_pipeline_v2::{run_pipeline_v2, RunPipelineV2Args};
use crate::http_server::endpoints::omni_api::shared_utils::video::validate_video_request::validate_video_request;
use crate::http_server::session::lookup::user_session_feature_flags::UserSessionFeatureFlags;
use crate::http_server::user_lookup::api_keys::require_api_key_user::require_api_key_user;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
use crate::util::lookup::lookup_media_files_as_cdn_url_list_and_map::lookup_media_files_as_cdn_url_list_and_map;
/// Generate a video using the omni-api unified endpoint (API-key authenticated).
#[utoipa::path(
post,
tag = "Omni API",
path = "/v1/omni_api/generate/video",
request_body = OmniGenVideoCostAndGenerateRequest,
responses(
(status = 200, description = "Success", body = OmniGenVideoGenerateResponse),
(status = 400, description = "Bad input"),
(status = 401, description = "Unauthorized"),
(status = 402, description = "Payment required"),
(status = 500, description = "Server error"),
),
)]
pub async fn omni_api_video_generate_handler(
http_request: HttpRequest,
request: Json<OmniGenVideoCostAndGenerateRequest>,
server_state: web::Data<Arc<ServerState>>,
) -> Result<Json<OmniGenVideoGenerateResponse>, CommonWebError> {
info!("request: {:?}", request);
// Reject doomed combos (e.g. grok_imagine_video_1p5 without an image)
// before any billable or DB-mutating work — see helper for the rules.
validate_video_request(&request)?;
payments_error_test(&request.prompt.as_deref().unwrap_or(""))?;
let debug_log_event_token = DebugLogEventToken::generate();
let maybe_prompt_model_type: Option<CommonModelType> = request.model
.as_ref()
.map(|m| m.to_common_model_type());
// ==================== API KEY USER ==================== //
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
// API-key authentication (Authorization header) instead of a session cookie. Never cached.
let api_session = require_api_key_user(&http_request, &mut *mysql_connection).await?;
let user_token = &api_session.user_token;
// ==================== MODEL ACCESS CHECK ==================== //
// API-key users have no per-user feature flags today.
let user_feature_flags = UserSessionFeatureFlags::new(None);
// ==================== IDEMPOTENCY ==================== //
let idempotency_token = request.idempotency_token.as_deref()
.unwrap_or("")
.to_string();
if let Err(reason) = validate_idempotency_token_format(&idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
insert_idempotency_token(&idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
CommonWebError::BadInputWithSimpleMessage("repeated idempotency token".to_string())
})?;
// ==================== RESOLVE MEDIA TOKENS ==================== //
let mut all_tokens: Vec<MediaFileToken> = Vec::new();
if let Some(token) = &request.start_frame_image_media_token {
all_tokens.push(token.clone());
}
if let Some(token) = &request.end_frame_image_media_token {
all_tokens.push(token.clone());
}
if let Some(tokens) = &request.reference_image_media_tokens {
all_tokens.extend(tokens.iter().cloned());
}
if let Some(tokens) = &request.reference_video_media_tokens {
all_tokens.extend(tokens.iter().cloned());
}
if let Some(tokens) = &request.reference_audio_media_tokens {
all_tokens.extend(tokens.iter().cloned());
}
let media_file_to_url_map: Option<HashMap<MediaFileToken, String>> = if all_tokens.is_empty() {
None
} else {
info!("Resolving {} media file tokens to CDN URLs", all_tokens.len());
let resolved = lookup_media_files_as_cdn_url_list_and_map(
&http_request,
&mut mysql_connection,
server_state.server_environment,
&all_tokens,
).await?;
Some(resolved.token_to_url_map)
};
// ==================== RESOLVE CHARACTERS ==================== //
let kinovi_character_id_map: Option<HashMap<CharacterToken, String>> =
resolve_kinovi_character_ids(
request.reference_character_tokens.as_deref(),
&mut mysql_connection,
).await?;
// ==================== HYDRATE ROUTER REQUEST ==================== //
let router_builder = hydrate_to_router_request(&request)?;
// ==================== PIPELINE DISPATCH ==================== //
let kinovi_account = match request.model {
// BytePlus Ultra
Some(CommonVideoModel::Seedance2p0BytePlusUltra) => KinoviAccount::BytePlusUltra,
Some(CommonVideoModel::Seedance2p0BytePlusUltraFast) => KinoviAccount::BytePlusUltra,
// BytePlus
Some(CommonVideoModel::Seedance2p0BytePlus) => KinoviAccount::BytePlus,
Some(CommonVideoModel::Seedance2p0BytePlusFast) => KinoviAccount::BytePlus,
Some(CommonVideoModel::PreviewModel) => KinoviAccount::BytePlus,
Some(CommonVideoModel::PreviewModelFast) => KinoviAccount::BytePlus,
// Everything else goes through Volcengine
_ => KinoviAccount::Volcengine,
};
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
message: &serde_json::to_string(&*request).unwrap_or_default(),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert HTTP request debug log: {:?}", err);
}
// NB: Release the pooled DB connection before the (slow, external) generation call so we don't
// hold a pool slot idle while waiting on the provider — that's what starves the pool and causes
// PoolTimedOut on unrelated endpoints. We re-acquire below to write the result.
drop(mysql_connection);
let pipeline_result = run_pipeline_v2(RunPipelineV2Args {
router_builder: &router_builder,
server_state: &server_state,
user_token,
media_file_to_url_map: &media_file_to_url_map,
kinovi_character_id_map: &kinovi_character_id_map,
kinovi_account,
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
// ==================== DEBUG LOG: FAL REQUEST ==================== //
if let GenerateVideoResponse::Fal(ref fal_payload) = pipeline_result.response {
if let Some(ref outbound_request) = fal_payload.maybe_outbound_request {
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert Fal request debug log: {:?}", err);
}
}
}
// ==================== DEBUG LOG: GROK API REQUEST ==================== //
if let GenerateVideoResponse::Grok(ref grok_payload) = pipeline_result.response {
if let Some(ref outbound_request) = grok_payload.maybe_outbound_request {
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::GrokApiRequest,
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert Grok API request debug log: {:?}", err);
}
}
}
// ==================== WRITE RESULT ==================== //
let ip_address = get_request_ip(&http_request);
// Omni API requests are always API-key authenticated; hardcode the platform type.
let maybe_platform_type = Some(PlatformType::ApiKey);
let mut transaction = mysql_connection.begin().await.map_err(|err| {
error!("Error starting MySQL transaction: {:?}", err);
CommonWebError::from_error(err)
})?;
// -- Prompt --
let prompt_token = match insert_prompt(InsertPromptArgs {
maybe_apriori_prompt_token: None,
prompt_type: PromptType::ArtcraftApp,
maybe_creator_user_token: Some(user_token),
maybe_model_type: maybe_prompt_model_type,
maybe_generation_provider: Some(GenerationProvider::Artcraft),
maybe_positive_prompt: request.prompt.as_deref(),
maybe_negative_prompt: request.negative_prompt.as_deref(),
maybe_other_args: None,
maybe_generation_mode: Some(determine_generation_mode(&request)),
maybe_aspect_ratio: request.aspect_ratio,
maybe_resolution: request.resolution,
maybe_batch_count: request.video_batch_count.map(|c| c as u8),
maybe_generate_audio: request.generate_audio,
maybe_duration_seconds: request.duration_seconds.map(|d| d as u32),
creator_ip_address: &ip_address,
mysql_executor: &mut *transaction,
phantom: Default::default(),
}).await {
Ok(token) => Some(token),
Err(err) => {
warn!("Error inserting prompt: {:?}", err);
None
}
};
// -- Prompt context items --
if let Some(token) = prompt_token.as_ref() {
let mut context_items = Vec::new();
if let Some(media_token) = &request.start_frame_image_media_token {
context_items.push(PromptContextItem {
media_token: media_token.clone(),
context_semantic_type: PromptContextSemanticType::VidStartFrame,
});
}
if let Some(media_token) = &request.end_frame_image_media_token {
context_items.push(PromptContextItem {
media_token: media_token.clone(),
context_semantic_type: PromptContextSemanticType::VidEndFrame,
});
}
if let Some(ref_tokens) = &request.reference_image_media_tokens {
for media_token in ref_tokens {
context_items.push(PromptContextItem {
media_token: media_token.clone(),
context_semantic_type: PromptContextSemanticType::Imgref,
});
}
}
if let Some(ref_tokens) = &request.reference_video_media_tokens {
for media_token in ref_tokens {
context_items.push(PromptContextItem {
media_token: media_token.clone(),
context_semantic_type: PromptContextSemanticType::VidRef,
});
}
}
if !context_items.is_empty() {
if let Err(err) = insert_batch_prompt_context_items(InsertBatchArgs {
prompt_token: token.clone(),
items: context_items,
transaction: &mut transaction,
}).await {
warn!("Error inserting batch prompt context items: {:?}", err);
}
}
}
// -- Inference job --
let (primary_job_token, all_job_tokens) = match &pipeline_result.response {
GenerateVideoResponse::Seedance2Pro(payload) => {
info!("Inserting seedance2pro job(s) with token: {:?}", pipeline_result.billing.apriori_job_token);
let kinovi_version = match kinovi_account {
KinoviAccount::Volcengine => KinoviVersion::Volcengine,
KinoviAccount::BytePlus => KinoviVersion::BytePlus,
KinoviAccount::BytePlusUltra => KinoviVersion::BytePlusUltra,
};
let result = insert_seedance2pro_jobs(InsertSeedance2proJobsArgs {
primary_order_id: &payload.order_id,
maybe_additional_order_ids: payload.maybe_order_ids.as_deref(),
maybe_wallet_ledger_entry_token: pipeline_result.billing.maybe_wallet_ledger_entry_token.as_ref(),
kinovi_version,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.billing.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
maybe_cost_estimates: Some(pipeline_result.cost_estimates),
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?;
(result.primary_job_token, result.all_job_tokens)
}
GenerateVideoResponse::Fal(payload) => {
let external_id = payload.request_id.as_deref().ok_or_else(|| {
error!("Fal generation response missing request_id");
CommonWebError::server_error_with_message("Fal generation response missing request_id")
})?;
info!("Inserting fal job with token: {:?}", pipeline_result.billing.apriori_job_token);
let token = insert_fal_job(InsertFalJobArgs {
external_job_id: external_id,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.billing.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
maybe_cost_estimates: Some(pipeline_result.cost_estimates),
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?;
(
token.clone(),
vec![token],
)
}
GenerateVideoResponse::Artcraft(payload) => {
(
payload.inference_job_token.clone(),
vec![payload.inference_job_token.clone()],
)
}
GenerateVideoResponse::GmiCloud(payload) => {
info!("Inserting GmiCloud job with token: {:?}", pipeline_result.billing.apriori_job_token);
let token = insert_gmicloud_job(InsertGmiCloudJobArgs {
external_request_id: &payload.request_id,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.billing.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
maybe_cost_estimates: Some(pipeline_result.cost_estimates),
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?;
(
token.clone(),
vec![token],
)
}
GenerateVideoResponse::Grok(payload) => {
info!("Inserting Grok (xAI) API job with token: {:?}", pipeline_result.billing.apriori_job_token);
let token = insert_grok_api_job(InsertGrokApiJobArgs {
external_request_id: &payload.request_id,
shared: SharedJobArgs {
apriori_job_token: &pipeline_result.billing.apriori_job_token,
idempotency_token: &idempotency_token,
user_token,
maybe_avt_token: None, // AVT tokens are web-session only; API-key callers have none.
maybe_model_type: request.model.map(|v| v.to_common_model_type()),
maybe_prompt_token: prompt_token.as_ref(),
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_platform_type,
maybe_cost_estimates: Some(pipeline_result.cost_estimates),
ip_address: &ip_address,
transaction: &mut transaction,
},
}).await?;
(
token.clone(),
vec![token],
)
}
other => {
error!("Unexpected generation response variant: {:?}", other);
return Err(CommonWebError::server_error_with_message("Unexpected generation response"));
}
};
transaction.commit().await.map_err(|err| {
error!("Error committing transaction: {:?}", err);
CommonWebError::from_error(err)
})?;
Ok(Json(OmniGenVideoGenerateResponse {
success: true,
inference_job_token: primary_job_token,
all_job_tokens,
}))
}
fn determine_generation_mode(request: &OmniGenVideoCostAndGenerateRequest) -> CommonGenerationMode {
let has_keyframe = request.start_frame_image_media_token.is_some()
|| request.end_frame_image_media_token.is_some();
if has_keyframe {
return CommonGenerationMode::Keyframe;
}
let has_reference = request.reference_image_media_tokens.as_ref().is_some_and(|t| !t.is_empty())
|| request.reference_video_media_tokens.as_ref().is_some_and(|t| !t.is_empty())
|| request.reference_audio_media_tokens.as_ref().is_some_and(|t| !t.is_empty())
|| request.reference_character_tokens.as_ref().is_some_and(|t| !t.is_empty());
if has_reference {
return CommonGenerationMode::Reference;
}
CommonGenerationMode::Text
}
@@ -1,227 +0,0 @@
use std::convert::TryFrom;
use std::collections::HashMap;
use log::{error, info, warn};
use artcraft_router::api::router_video_model::RouterVideoModel;
use artcraft_router::api::router_provider::RouterProvider;
use artcraft_router::generate::generate_video::generate_video_request_builder::GenerateVideoRequestBuilder;
use artcraft_router::generate::generate_video::generate_video_response::GenerateVideoResponse;
use artcraft_router::generate::generate_video::video_generation_draft_context::VideoGenerationDraftContext;
use artcraft_router::generate::generate_video::video_generation_draft_or_request::VideoGenerationDraftOrRequest;
use tokens::tokens::characters::CharacterToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoint_helpers::refund_wallet_after_api_failure::refund_wallet_after_api_failure;
use crate::http_server::endpoints::omni_api::generate::video::helpers::bill_wallet::bill_wallet;
use crate::http_server::endpoints::omni_api::generate::video::helpers::build_router_client::build_router_client;
use crate::http_server::endpoints::omni_api::generate::video::helpers::pipeline_result::PipelineResult;
use crate::http_server::endpoints::omni_api::generate::video::helpers::resolve_media_tokens_to_urls::resolve_media_tokens_to_urls;
use mysql_queries::queries::generic_inference::common::job_cost_estimates::JobCostEstimates;
use crate::http_server::endpoints::omni_api::generate::video::kinovi_account::KinoviAccount;
use crate::http_server::endpoints::omni_api::shared_utils::map_seedance2pro_router_error::map_router_error_to_web_error;
use crate::state::server_state::ServerState;
pub struct RunPipelineV2Args<'a> {
pub router_builder: &'a GenerateVideoRequestBuilder,
pub server_state: &'a ServerState,
pub user_token: &'a UserToken,
pub media_file_to_url_map: &'a Option<HashMap<MediaFileToken, String>>,
pub kinovi_character_id_map: &'a Option<HashMap<CharacterToken, String>>,
pub kinovi_account: KinoviAccount,
}
// NB: This pipeline does an external generation call (`upload_and_generate`) that can take many
// seconds. It deliberately does NOT hold a pooled DB connection across that call — it acquires
// short-lived connections only for the billing and (on failure) refund writes. Holding a pooled
// connection across the external call is what starves the pool and causes `PoolTimedOut`.
pub async fn run_pipeline_v2(args: RunPipelineV2Args<'_>) -> Result<PipelineResult, CommonWebError> {
let RunPipelineV2Args {
router_builder,
server_state,
user_token,
media_file_to_url_map,
kinovi_character_id_map,
kinovi_account,
} = args;
let mut router_builder = router_builder.clone();
match router_builder.model {
RouterVideoModel::PreviewModel |
RouterVideoModel::Seedance2p0BytePlus |
RouterVideoModel::Seedance2p0BytePlusUltra => {
router_builder.model = RouterVideoModel::Seedance2p0;
},
RouterVideoModel::PreviewModelFast |
RouterVideoModel::Seedance2p0BytePlusFast |
RouterVideoModel::Seedance2p0BytePlusUltraFast => {
router_builder.model = RouterVideoModel::Seedance2p0Fast;
},
_ => {}, // Fall-through
}
let provider = match router_builder.model {
RouterVideoModel::HappyHorse1p0 => RouterProvider::Seedance2Pro,
RouterVideoModel::Seedance2p0 => RouterProvider::Seedance2Pro,
RouterVideoModel::Seedance2p0Fast => RouterProvider::Seedance2Pro,
//RouterVideoModel::Seedance2p0Ultra => RouterProvider::GmiCloud,
//RouterVideoModel::Seedance2p0UltraFast => RouterProvider::GmiCloud,
RouterVideoModel::GrokImagineVideo => RouterProvider::GrokApi,
RouterVideoModel::GrokImagineVideo1p5 => RouterProvider::GrokApi,
_ => RouterProvider::Fal,
};
// 1. Build execution request
let mut exec_builder = router_builder.clone();
exec_builder.provider = provider;
// Fal, GmiCloud, and Grok (xAI) take image URLs directly, not media file tokens.
// Resolve tokens to URLs before building.
if matches!(provider, RouterProvider::Fal | RouterProvider::GmiCloud | RouterProvider::GrokApi) {
resolve_media_tokens_to_urls(&mut exec_builder, media_file_to_url_map.as_ref());
}
let draft_or_request = exec_builder.build2()
.map_err(|e| {
warn!("Failed to build2 for v2 pipeline: {}", e);
CommonWebError::from_error(e)
})?;
// 2. Calculate cost.
// For Artcraft-billable models, swap provider to Artcraft so credits = cents.
// For GmiCloud, use the execution request's cost directly (no Artcraft equivalent).
let system_cost_estimate = {
let mut cost_builder = router_builder.clone();
cost_builder.provider = RouterProvider::Artcraft;
cost_builder.build2()
.map_err(|e| {
warn!("Failed to build2 cost estimate for v2: {}", e);
CommonWebError::from_error(e)
})?
.estimate_cost()
.map_err(|e| {
warn!("Failed to estimate cost for v2: {}", e);
CommonWebError::from_error(e)
})?
};
let cost = system_cost_estimate.cost_in_credits.unwrap_or(0);
// Provider-side estimate (what the fulfilling provider charges us). The
// router defers to the underlying provider crates (seedance2pro_client,
// gmicloud_client, grok_api_client, the fal pricing modules, etc.) per
// request variant. Bookkeeping only — failures must not block generation.
let maybe_provider_cost_estimate = match draft_or_request.estimate_cost() {
Ok(estimate) => Some(estimate),
Err(err) => {
warn!("Failed to estimate provider cost for v2 video: {}", err);
None
}
};
let cost_estimates = JobCostEstimates {
maybe_external_third_party_cost_credits: maybe_provider_cost_estimate.as_ref()
.and_then(|e| e.cost_in_credits)
.and_then(|v| u32::try_from(v).ok()),
maybe_external_third_party_cost_usd_cents: maybe_provider_cost_estimate.as_ref()
.and_then(|e| e.cost_in_usd_cents)
.and_then(|v| u32::try_from(v).ok()),
maybe_system_cost_credits: system_cost_estimate.cost_in_credits
.and_then(|v| u32::try_from(v).ok()),
maybe_system_cost_usd_cents: system_cost_estimate.cost_in_usd_cents
.and_then(|v| u32::try_from(v).ok()),
};
info!("v2 estimated cost: {} credits (estimates: {:?})", cost, cost_estimates);
// 3. Bill wallet (short-lived connection — released before the external call below).
let billing = {
let mut billing_connection = server_state.mysql_pool.acquire().await
.map_err(|err| {
error!("Failed to acquire MySQL connection for billing: {:?}", err);
CommonWebError::from_error(err)
})?;
bill_wallet(user_token, cost, &mut billing_connection).await?
};
// 4. Upload media (if draft) and generate video.
// The entire block is wrapped so Kinovi failures trigger a refund.
// NB: No pooled DB connection is held across this call.
let result = upload_and_generate(
draft_or_request,
server_state,
media_file_to_url_map.as_ref(),
kinovi_character_id_map.as_ref(),
kinovi_account,
).await;
// 5. On failure, refund wallet for Kinovi requests.
if let Err(ref err) = result {
if matches!(provider, RouterProvider::Seedance2Pro) {
if let Some(ledger_entry_token) = billing.maybe_wallet_ledger_entry_token.as_ref() {
warn!("Kinovi v2 generation failed, issuing refund for {}: {:?}", ledger_entry_token.as_str(), err);
match server_state.mysql_pool.acquire().await {
Ok(mut refund_connection) => {
if let Err(refund_err) = refund_wallet_after_api_failure(ledger_entry_token, &mut refund_connection).await {
error!("Failed to refund wallet after Kinovi v2 failure: {:?}", refund_err);
}
}
Err(acquire_err) => {
error!("Failed to acquire MySQL connection to refund wallet after Kinovi v2 failure: {:?}", acquire_err);
}
}
}
}
}
let response = result?;
info!("v2 generation response: {:?}", response);
Ok(PipelineResult { billing, response, cost_estimates })
}
/// Finalize the draft (uploading media if needed), then send the generation request.
///
/// This is the block that gets refunded on failure for Kinovi providers.
async fn upload_and_generate(
draft_or_request: VideoGenerationDraftOrRequest,
server_state: &ServerState,
media_file_urls_by_token: Option<&HashMap<MediaFileToken, String>>,
kinovi_character_ids: Option<&HashMap<CharacterToken, String>>,
kinovi_account: KinoviAccount,
) -> Result<GenerateVideoResponse, CommonWebError> {
let provider = draft_or_request.get_provider();
let client = build_router_client(provider, server_state, kinovi_account)?;
let video_request = match draft_or_request {
VideoGenerationDraftOrRequest::Request(request) => request,
VideoGenerationDraftOrRequest::Draft(draft) => {
let draft_context = VideoGenerationDraftContext {
client: Some(&client),
media_file_to_artcraft_url_map: media_file_urls_by_token,
character_token_to_kinovi_id_map: kinovi_character_ids,
};
draft.finalize(draft_context)
.await
.map_err(|err| {
warn!("Failed to finalize v2 draft: {:?}", err);
map_router_error_to_web_error(err)
})?
}
};
video_request.send_request(&client)
.await
.map_err(|err| {
warn!("v2 video generation failed: {:?}", err);
map_router_error_to_web_error(err)
})
}
@@ -1,3 +0,0 @@
pub mod generate;
pub mod shared_utils;
pub mod upload;
@@ -1,90 +0,0 @@
//! Maps artcraft router errors to web errors for the omni_gen pipelines.
//!
//! Known Kinovi (Seedance2Pro) bad requests (content violations, over-long
//! prompts, too many urls) become 400s with a user-facing message. Everything
//! else is unanticipated and stays a 500.
use artcraft_router::errors::artcraft_router_error::ArtcraftRouterError;
use artcraft_router::errors::provider_error::ProviderError;
use seedance2pro_client::error::seedance2pro_bad_request_api_error::Seedance2ProBadRequestApiError;
use seedance2pro_client::error::seedance2pro_error::Seedance2ProError;
use crate::http_server::common_responses::common_web_error::CommonWebError;
/// Map an artcraft router error to a web error, unwrapping Kinovi bad requests.
pub fn map_router_error_to_web_error(error: ArtcraftRouterError) -> CommonWebError {
match error {
ArtcraftRouterError::Provider(
ProviderError::Seedance2Pro(Seedance2ProError::ApiBadRequest(bad_request))
) => {
CommonWebError::BadInputWithSimpleMessage(user_facing_bad_request_message(&bad_request))
}
other => CommonWebError::from_error(other),
}
}
fn user_facing_bad_request_message(error: &Seedance2ProBadRequestApiError) -> String {
match error {
Seedance2ProBadRequestApiError::VideoGenerationViolation { .. } => {
"The generation request was flagged as a content violation. \
Please adjust your prompt or input media and try again.".to_string()
}
Seedance2ProBadRequestApiError::PromptIsTooLong { .. } => {
"The prompt is too long. Please shorten it below 10,000 characters and try again.".to_string()
}
Seedance2ProBadRequestApiError::TooManyUrls { .. } => {
"Too many input files were attached. \
Please remove some and try again.".to_string()
}
}
}
#[cfg(test)]
mod tests {
use actix_web::ResponseError;
use super::*;
#[test]
fn kinovi_bad_requests_become_400s_with_user_facing_messages() {
let cases = [
(
Seedance2ProBadRequestApiError::VideoGenerationViolation { raw_body: "{}".to_string() },
"content violation",
),
(
Seedance2ProBadRequestApiError::PromptIsTooLong { raw_body: "{}".to_string() },
"prompt is too long",
),
(
Seedance2ProBadRequestApiError::TooManyUrls { raw_body: "{}".to_string() },
"Too many input files",
),
];
for (bad_request, expected_message_fragment) in cases {
let error = map_router_error_to_web_error(
ArtcraftRouterError::Provider(
ProviderError::Seedance2Pro(
Seedance2ProError::ApiBadRequest(bad_request))));
assert_eq!(error.status_code().as_u16(), 400);
match error {
CommonWebError::BadInputWithSimpleMessage(message) => {
assert!(
message.contains(expected_message_fragment),
"message {:?} should contain {:?}", message, expected_message_fragment);
}
other => panic!("Expected BadInputWithSimpleMessage, got: {:?}", other),
}
}
}
#[test]
fn other_router_errors_stay_500s() {
let error = map_router_error_to_web_error(
ArtcraftRouterError::UnsupportedModel("some_model".to_string()));
assert_eq!(error.status_code().as_u16(), 500);
}
}
@@ -1,2 +0,0 @@
pub mod map_seedance2pro_router_error;
pub mod video;
@@ -1,2 +0,0 @@
pub mod validate_video_request;
mod validate_when_image_required;
@@ -1,11 +0,0 @@
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::omni_api::shared_utils::video::validate_when_image_required::validate_when_image_required;
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_video_cost_and_generate_request::OmniGenVideoCostAndGenerateRequest;
/// Validate requests before they incur user costs or send API requests
pub fn validate_video_request(
request: &OmniGenVideoCostAndGenerateRequest,
) -> Result<(), CommonWebError> {
validate_when_image_required(request)?;
Ok(())
}
@@ -1,157 +0,0 @@
use artcraft_api_defs::omni_gen::cost_and_generate_requests::omni_gen_video_cost_and_generate_request::OmniGenVideoCostAndGenerateRequest;
use enums::common::generation::common_video_model::CommonVideoModel;
use crate::http_server::common_responses::common_web_error::CommonWebError;
/// xAI's `grok-imagine-video-1.5-preview` model rejects text-to-video at
/// the upstream API. Reject the request at the handler layer so callers see
/// a clean 400 with an actionable message and we don't charge their wallet
/// for an inference job we know will fail.
pub (super) fn validate_when_image_required(
request: &OmniGenVideoCostAndGenerateRequest,
) -> Result<(), CommonWebError> {
match request.model {
Some(CommonVideoModel::GrokImagineVideo1p5) => {}, // Fall-through
_ => return Ok(()),
}
let requires_image = matches!(request.model, Some(CommonVideoModel::GrokImagineVideo1p5));
if !requires_image {
return Ok(());
}
let has_start_frame = request.start_frame_image_media_token.is_some();
let has_reference_images = request
.reference_image_media_tokens
.as_ref()
.is_some_and(|v| !v.is_empty());
if has_start_frame || has_reference_images {
return Ok(());
}
Err(CommonWebError::BadInputWithSimpleMessage(
"Image is required".to_string(),
))
}
#[cfg(test)]
mod tests {
use super::*;
use tokens::tokens::media_files::MediaFileToken;
fn base_request() -> OmniGenVideoCostAndGenerateRequest {
OmniGenVideoCostAndGenerateRequest {
idempotency_token: Some("a1b2c3d4-e5f6-7890-abcd-ef1234567890".to_string()),
model: None,
prompt: Some("test".to_string()),
negative_prompt: None,
start_frame_image_media_token: None,
end_frame_image_media_token: None,
reference_image_media_tokens: None,
reference_video_media_tokens: None,
reference_audio_media_tokens: None,
reference_character_tokens: None,
resolution: None,
aspect_ratio: None,
quality: None,
duration_seconds: Some(5),
video_batch_count: Some(1),
generate_audio: None,
}
}
fn token(id: &str) -> MediaFileToken {
MediaFileToken::new(id.to_string())
}
#[test]
fn passes_when_model_is_none() {
let req = base_request();
assert!(validate_when_image_required(&req).is_ok());
}
#[test]
fn passes_for_other_models_without_image() {
// Only v1.5 needs the gate; other models can do text-to-video.
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::Seedance2p0),
..base_request()
};
assert!(validate_when_image_required(&req).is_ok());
}
#[test]
fn rejects_v1p5_with_no_image_inputs() {
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
..base_request()
};
let err = validate_when_image_required(&req).expect_err("should reject");
match err {
CommonWebError::BadInputWithSimpleMessage(msg) => {
assert_eq!(msg, "Image is required");
}
other => panic!("expected BadInputWithSimpleMessage, got {:?}", other),
}
}
#[test]
fn rejects_v1p5_with_empty_reference_list_and_no_start_frame() {
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
reference_image_media_tokens: Some(vec![]),
..base_request()
};
assert!(matches!(
validate_when_image_required(&req),
Err(CommonWebError::BadInputWithSimpleMessage(_))
));
}
#[test]
fn accepts_v1p5_with_start_frame_only() {
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
start_frame_image_media_token: Some(token("mf_start")),
..base_request()
};
assert!(validate_when_image_required(&req).is_ok());
}
#[test]
fn accepts_v1p5_with_reference_images_only() {
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
reference_image_media_tokens: Some(vec![token("mf_a"), token("mf_b")]),
..base_request()
};
assert!(validate_when_image_required(&req).is_ok());
}
#[test]
fn accepts_v1p5_with_both_start_frame_and_reference_images() {
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
start_frame_image_media_token: Some(token("mf_start")),
reference_image_media_tokens: Some(vec![token("mf_a")]),
..base_request()
};
assert!(validate_when_image_required(&req).is_ok());
}
#[test]
fn end_frame_alone_does_not_count_as_image_input() {
// end_frame_image_media_token is technically an image, but xAI's v1.5
// wire shape only uses start_frame / reference_images. end_frame
// shouldn't satisfy the gate.
let req = OmniGenVideoCostAndGenerateRequest {
model: Some(CommonVideoModel::GrokImagineVideo1p5),
end_frame_image_media_token: Some(token("mf_end")),
..base_request()
};
assert!(validate_when_image_required(&req).is_err());
}
}
@@ -1,3 +0,0 @@
pub mod omni_upload_audio_media_file_handler;
pub mod omni_upload_image_media_file_handler;
pub mod omni_upload_video_media_file_handler;
@@ -1,327 +0,0 @@
use std::collections::HashSet;
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use actix_multipart::form::tempfile::TempFile;
use actix_multipart::form::text::Text;
use actix_multipart::form::MultipartForm;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, info, warn};
use once_cell::sync::Lazy;
use utoipa::ToSchema;
use artcraft_api_defs::omni_api::omni_upload_audio::OmniUploadAudioMediaFileSuccessResponse;
use bucket_paths::legacy::typified_paths::public::media_files::bucket_file_path::MediaFileBucketPath;
use enums::by_table::media_files::media_file_class::MediaFileClass;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use hashing::sha256::sha256_hash_bytes::sha256_hash_bytes;
use http_server_common::request::get_request_ip::get_request_ip;
use media::decode_basic_audio_info::decode_basic_audio_bytes_info;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mimetypes::mimetype_to_extension::mimetype_to_extension;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use mysql_queries::queries::media_files::create::specialized_insert::insert_media_file_from_file_upload::{insert_media_file_from_file_upload, InsertMediaFileFromUploadArgs, UploadType};
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::media_files::upload::upload_error::MediaFileUploadError;
use crate::http_server::endpoints::media_files::upload::common_utils::try_parse_generation_provider::try_parse_generation_provider;
use crate::http_server::user_lookup::api_keys::require_api_key_user::require_api_key_user;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
/// Form-multipart request fields.
///
/// IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadAudioMediaFileForm` (Under "Schema") FOR DETAILS ON FIELDS AND NULLABILITY.
#[derive(MultipartForm, ToSchema)]
#[multipart(duplicate_field = "deny")]
pub struct OmniUploadAudioMediaFileForm {
/// UUID for request idempotency
#[multipart(limit = "2 KiB")]
#[schema(value_type = String, format = Binary)]
uuid_idempotency_token: Text<String>,
/// The uploaded file
#[multipart(limit = "512 MiB")]
#[schema(value_type = Vec<u8>, format = Binary)]
file: TempFile,
/// Optional: Title (name) of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_title: Option<Text<String>>,
/// Optional: Visibility of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<Visibility>, format = Binary)]
maybe_visibility: Option<Text<Visibility>>,
/// Optional: The third-party generation provider (e.g. "fal", "replicate").
/// If set, `is_user_upload` will be false and `is_intermediate_system_file` will be forced false.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_generation_provider: Option<Text<String>>,
}
static ALLOWED_MIME_TYPES : Lazy<HashSet<&'static str>> = Lazy::new(|| {
HashSet::from([
// Audio
"audio/aac",
"audio/m4a",
"audio/mpeg",
"audio/ogg",
"audio/opus",
"audio/x-flac",
"audio/x-wav",
// Mixed
"audio/mp4", // iPhone seems to upload these as audio
// Video
"video/mp4",
"video/webm",
])
});
/// API-key authenticated audio upload (Omni API). Identity is read from the `Authorization` header
/// API key rather than a session cookie.
#[utoipa::path(
post,
tag = "Omni API",
path = "/v1/omni_api/upload/audio",
responses(
(status = 200, description = "Success Update", body = OmniUploadAudioMediaFileSuccessResponse),
(status = 400, description = "Bad input", body = MediaFileUploadError),
(status = 401, description = "Not authorized", body = MediaFileUploadError),
(status = 429, description = "Too many requests", body = MediaFileUploadError),
(status = 500, description = "Server error", body = MediaFileUploadError),
),
params(
(
"request" = OmniUploadAudioMediaFileForm,
description = "IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadAudioMediaFileForm` (Under 'Schema') FOR DETAILS ON FIELDS AND NULLABILITY."
),
)
)]
pub async fn omni_upload_audio_media_file_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>,
MultipartForm(mut form): MultipartForm<OmniUploadAudioMediaFileForm>,
) -> Result<Json<OmniUploadAudioMediaFileSuccessResponse>, MediaFileUploadError> {
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
error!("MySql pool error: {:?}", err);
MediaFileUploadError::ServerError
})?;
// ==================== API KEY USER ==================== //
// API-key authentication (Authorization header) instead of a session cookie. Never cached, and a
// banned owner is rejected inside `require_api_key_user`.
let api_session = require_api_key_user(&http_request, &mut *mysql_connection)
.await
.map_err(map_api_key_auth_error)?;
let maybe_user_token = Some(&api_session.user_token);
// ==================== RATE LIMIT ==================== //
let rate_limiter = &server_state.redis_rate_limiters.file_upload_logged_in;
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(MediaFileUploadError::RateLimited);
}
// ==================== HANDLE IDEMPOTENCY ==================== //
let uuid_idempotency_token = form.uuid_idempotency_token.as_ref();
if let Err(reason) = validate_idempotency_token_format(uuid_idempotency_token) {
return Err(MediaFileUploadError::BadInput(reason));
}
insert_idempotency_token(uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
MediaFileUploadError::BadInput("invalid idempotency token".to_string())
})?;
// ==================== UPLOAD METADATA ==================== //
let maybe_title = form.maybe_title
.map(|title| title.trim().to_string())
.filter(|title| !title.is_empty());
let creator_set_visibility = form.maybe_visibility
.map(|visibility| visibility.0)
.unwrap_or(Visibility::default());
// ==================== USER DATA ==================== //
let ip_address = get_request_ip(&http_request);
// ==================== FILE VALIDATION ==================== //
let mut file_bytes = Vec::new();
form.file.file.read_to_end(&mut file_bytes)
.map_err(|e| {
error!("Problem reading file: {:?}", e);
MediaFileUploadError::ServerError
})?;
let mimetype = get_mimetype_for_bytes(file_bytes.as_ref())
.map(|mimetype| mimetype.to_string())
.ok_or_else(|| {
warn!("Could not determine mimetype for file");
MediaFileUploadError::BadInput("Could not determine mimetype for file".to_string())
})?;
if !ALLOWED_MIME_TYPES.contains(mimetype.as_str()) {
// NB: Don't let our error message inject malicious strings
let filtered_mimetype = mimetype
.chars()
.filter(|c| c.is_ascii())
.filter(|c| c.is_alphanumeric() || *c == '/')
.collect::<String>();
return Err(MediaFileUploadError::BadInput(format!("unpermitted mime type: {}", &filtered_mimetype)));
}
// ==================== DURATION DETECTION ==================== //
let do_audio_decode = match mimetype.as_ref() {
// TODO: Revisit when Safari can send us this metadata consistently
"audio/mp4" | "video/mp4" => false,
// Symphonia is currently broken for Firefox's opus; skip decoding it.
"audio/opus" => false,
// Also, don't decode images
"image/jpeg" => false,
"image/png" => false,
"image/webp" => false,
_ => true,
};
let mut maybe_duration_millis = None;
if do_audio_decode {
let basic_info = decode_basic_audio_bytes_info(
file_bytes.as_ref(),
Some(&mimetype),
None
).map_err(|e| {
warn!("file decoding error: {:?}", e);
MediaFileUploadError::BadInput("could not decode file".to_string())
})?;
maybe_duration_millis = basic_info.duration_millis;
}
// ==================== OTHER FILE METADATA ==================== //
let maybe_filename = form.file.file_name.as_deref()
.as_deref()
.map(|filename| PathBuf::from(filename));
let extension = mimetype_to_extension(&mimetype)
.or_else(|| {
maybe_filename
.as_ref()
.and_then(|filename| filename.extension())
.and_then(|ext| ext.to_str())
})
.ok_or_else(|| {
warn!("Could not determine file extension for mimetype: {}", &mimetype);
MediaFileUploadError::ServerError
})?;
let extension = format!(".{extension}"); // NB: needs dot prefix
let file_size_bytes = file_bytes.len();
let hash = sha256_hash_bytes(&file_bytes)
.map_err(|io_error| {
error!("Problem hashing bytes: {:?}", io_error);
MediaFileUploadError::ServerError
})?;
// ==================== UPLOAD AND SAVE ==================== //
const PREFIX : Option<&str> = Some("aud_");
let public_upload_path = MediaFileBucketPath::generate_new(PREFIX, Some(&extension));
info!("Uploading media to bucket path: {}", public_upload_path.get_full_object_path_str());
server_state.public_bucket_client.upload_file_with_content_type(
public_upload_path.get_full_object_path_str(),
file_bytes.as_ref(),
&mimetype)
.await
.map_err(|e| {
warn!("Upload media bytes to bucket error: {:?}", e);
MediaFileUploadError::ServerError
})?;
let maybe_generation_provider = form.maybe_generation_provider
.as_ref()
.and_then(|text| try_parse_generation_provider(text.as_ref()));
let upload_type = if maybe_generation_provider.is_some() {
UploadType::ThirdPartyInference
} else {
UploadType::Filesystem
};
let (token, record_id) = insert_media_file_from_file_upload(InsertMediaFileFromUploadArgs {
maybe_media_class: Some(MediaFileClass::Audio),
media_file_type: MediaFileType::Audio,
maybe_creator_user_token: maybe_user_token,
// NB: AVT (anonymous visitor) tokens are a web-session concept; API-key callers have none.
maybe_creator_anonymous_visitor_token: None,
creator_ip_address: &ip_address,
creator_set_visibility,
upload_type,
maybe_engine_category: None,
maybe_animation_type: None,
maybe_prompt_token: None,
maybe_batch_token: None,
maybe_mime_type: Some(&mimetype),
file_size_bytes: file_size_bytes as u64,
maybe_duration_millis,
sha256_checksum: &hash,
maybe_title: maybe_title.as_deref(),
maybe_scene_source_media_file_token: None,
is_intermediate_system_file: false, // NB: is_user_upload = true
maybe_generation_provider,
public_bucket_directory_hash: public_upload_path.get_object_hash(),
maybe_public_bucket_prefix: PREFIX,
maybe_public_bucket_extension: Some(&extension),
pool: &server_state.mysql_pool,
})
.await
.map_err(|err| {
warn!("New file creation DB error: {:?}", err);
MediaFileUploadError::ServerError
})?;
info!("new media file id: {} token: {:?}", record_id, &token);
Ok(Json(OmniUploadAudioMediaFileSuccessResponse {
success: true,
media_file_token: token,
}))
}
/// Map an API-key authentication failure onto this endpoint's error type. A genuine 401 stays a
/// 401; anything else (e.g. a DB error during lookup) becomes a 500.
fn map_api_key_auth_error(err: CommonWebError) -> MediaFileUploadError {
match err {
CommonWebError::NotAuthorized => {
MediaFileUploadError::NotAuthorizedVerbose("invalid or missing API key".to_string())
}
_ => MediaFileUploadError::ServerError,
}
}
@@ -1,338 +0,0 @@
use std::collections::HashSet;
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use actix_multipart::form::tempfile::TempFile;
use actix_multipart::form::text::Text;
use actix_multipart::form::MultipartForm;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, info, warn};
use once_cell::sync::Lazy;
use utoipa::ToSchema;
use artcraft_api_defs::omni_api::omni_upload_image::OmniUploadImageMediaFileSuccessResponse;
use bucket_paths::legacy::typified_paths::public::media_files::bucket_file_path::MediaFileBucketPath;
use enums::by_table::media_files::media_file_class::MediaFileClass;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use hashing::sha256::sha256_hash_bytes::sha256_hash_bytes;
use http_server_common::request::get_request_ip::get_request_ip;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mimetypes::mimetype_to_extension::mimetype_to_extension;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use mysql_queries::queries::media_files::create::specialized_insert::insert_media_file_from_file_upload::{insert_media_file_from_file_upload, InsertMediaFileFromUploadArgs, UploadType};
use tokens::tokens::batch_generations::BatchGenerationToken;
use tokens::tokens::prompts::PromptToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::media_files::upload::upload_error::MediaFileUploadError;
use crate::http_server::endpoints::media_files::upload::common_utils::try_parse_generation_provider::try_parse_generation_provider;
use crate::http_server::user_lookup::api_keys::require_api_key_user::require_api_key_user;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
/// Form-multipart request fields.
///
/// IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadImageMediaFileForm` (Under "Schema") FOR DETAILS ON FIELDS AND NULLABILITY.
#[derive(MultipartForm, ToSchema)]
#[multipart(duplicate_field = "deny")]
pub struct OmniUploadImageMediaFileForm {
/// UUID for request idempotency
#[multipart(limit = "2 KiB")]
#[schema(value_type = String, format = Binary)]
uuid_idempotency_token: Text<String>,
/// The uploaded file
#[multipart(limit = "512 MiB")]
#[schema(value_type = Vec<u8>, format = Binary)]
file: TempFile,
/// Optional: Title (name) of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_title: Option<Text<String>>,
/// Optional: Visibility of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<Visibility>, format = Binary)]
maybe_visibility: Option<Text<Visibility>>,
/// Optional: Prompt associated with this image
/// NOTE: Cannot set `is_intermediate_system_file = true` if this is set.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_prompt_token: Option<Text<PromptToken>>,
/// Optional: Batch associated with this image
/// NOTE: Cannot set `is_intermediate_system_file = true` if this is set.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_batch_token: Option<Text<BatchGenerationToken>>,
/// Optional: Whether this is a system file (e.g. cover files we should hide)
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<bool>, format = Binary)]
is_intermediate_system_file: Option<Text<bool>>,
/// Optional: The third-party generation provider (e.g. "fal", "replicate").
/// If set, `is_user_upload` will be false and `is_intermediate_system_file` will be forced false.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_generation_provider: Option<Text<String>>,
}
static ALLOWED_MIME_TYPES : Lazy<HashSet<&'static str>> = Lazy::new(|| {
HashSet::from([
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
])
});
/// API-key authenticated image upload (Omni API). Identity is read from the `Authorization` header
/// API key rather than a session cookie.
#[utoipa::path(
post,
tag = "Omni API",
path = "/v1/omni_api/upload/image",
responses(
(status = 200, description = "Success Update", body = OmniUploadImageMediaFileSuccessResponse),
(status = 400, description = "Bad input", body = MediaFileUploadError),
(status = 401, description = "Not authorized", body = MediaFileUploadError),
(status = 429, description = "Too many requests", body = MediaFileUploadError),
(status = 500, description = "Server error", body = MediaFileUploadError),
),
params(
(
"request" = OmniUploadImageMediaFileForm,
description = "IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadImageMediaFileForm` (Under 'Schema') FOR DETAILS ON FIELDS AND NULLABILITY."
),
)
)]
pub async fn omni_upload_image_media_file_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>,
MultipartForm(mut form): MultipartForm<OmniUploadImageMediaFileForm>,
) -> Result<Json<OmniUploadImageMediaFileSuccessResponse>, MediaFileUploadError> {
validate_form(&form)?;
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
error!("MySql pool error: {:?}", err);
MediaFileUploadError::ServerError
})?;
// ==================== API KEY USER ==================== //
// API-key authentication (Authorization header) instead of a session cookie. Never cached, and a
// banned owner is rejected inside `require_api_key_user`.
let api_session = require_api_key_user(&http_request, &mut *mysql_connection)
.await
.map_err(map_api_key_auth_error)?;
let maybe_user_token = Some(&api_session.user_token);
// ==================== HANDLE IDEMPOTENCY ==================== //
let uuid_idempotency_token = form.uuid_idempotency_token.as_ref();
if let Err(reason) = validate_idempotency_token_format(uuid_idempotency_token) {
return Err(MediaFileUploadError::BadInput(reason));
}
insert_idempotency_token(uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
MediaFileUploadError::BadInput("invalid idempotency token".to_string())
})?;
// ==================== UPLOAD METADATA ==================== //
let maybe_title = form.maybe_title.map(|title| title.to_string());
let creator_set_visibility = form.maybe_visibility
.map(|visibility| visibility.0)
.unwrap_or(Visibility::default());
// ==================== USER DATA ==================== //
let ip_address = get_request_ip(&http_request);
// ==================== FILE VALIDATION ==================== //
let mut file_bytes = Vec::new();
form.file.file.read_to_end(&mut file_bytes)
.map_err(|e| {
error!("Problem reading file: {:?}", e);
MediaFileUploadError::ServerError
})?;
let mimetype = get_mimetype_for_bytes(file_bytes.as_ref())
.map(|mimetype| mimetype.to_string())
.ok_or_else(|| {
warn!("Could not determine mimetype for file");
MediaFileUploadError::BadInput("Could not determine mimetype for file".to_string())
})?;
if !ALLOWED_MIME_TYPES.contains(mimetype.as_str()) {
// NB: Don't let our error message inject malicious strings
let filtered_mimetype = mimetype
.chars()
.filter(|c| c.is_ascii())
.filter(|c| c.is_alphanumeric() || *c == '/')
.collect::<String>();
return Err(MediaFileUploadError::BadInput(format!("unpermitted mime type: {}", &filtered_mimetype)));
}
// ==================== OTHER FILE METADATA ==================== //
let maybe_filename = form.file.file_name.as_deref()
.as_deref()
.map(|filename| PathBuf::from(filename));
let extension = mimetype_to_extension(&mimetype)
.or_else(|| {
maybe_filename
.as_ref()
.and_then(|filename| filename.extension())
.and_then(|ext| ext.to_str())
})
.ok_or_else(|| {
warn!("Could not determine file extension for mimetype: {}", &mimetype);
MediaFileUploadError::ServerError
})?;
let extension = format!(".{extension}"); // NB: needs dot prefix
let file_size_bytes = file_bytes.len();
let hash = sha256_hash_bytes(&file_bytes)
.map_err(|io_error| {
error!("Problem hashing bytes: {:?}", io_error);
MediaFileUploadError::ServerError
})?;
// ==================== UPLOAD AND SAVE ==================== //
const PREFIX : Option<&str> = Some("image_");
let public_upload_path = MediaFileBucketPath::generate_new(PREFIX, Some(&extension));
info!("Uploading media to bucket path: {}", public_upload_path.get_full_object_path_str());
server_state.public_bucket_client.upload_file_with_content_type(
public_upload_path.get_full_object_path_str(),
file_bytes.as_ref(),
&mimetype)
.await
.map_err(|e| {
warn!("Upload media bytes to bucket error: {:?}", e);
MediaFileUploadError::ServerError
})?;
let is_intermediate_system_file = form.is_intermediate_system_file
.map(|b| b.0)
.unwrap_or(false);
let maybe_prompt_token = form.maybe_prompt_token
.as_ref()
.map(|token| token.0.clone());
let maybe_batch_token = form.maybe_batch_token
.as_ref()
.map(|token| token.0.clone());
let maybe_generation_provider = form.maybe_generation_provider
.as_ref()
.and_then(|text| try_parse_generation_provider(text.as_ref()));
let upload_type = if maybe_generation_provider.is_some() {
UploadType::ThirdPartyInference
} else {
UploadType::Filesystem
};
let media_file_type = match mimetype.as_str() {
"image/jpeg" => MediaFileType::Jpg,
"image/png" => MediaFileType::Png,
"image/gif" => MediaFileType::Gif,
"image/webp" => MediaFileType::Image, // Fallback
_ => MediaFileType::Image, // fallback
};
let (token, record_id) = insert_media_file_from_file_upload(InsertMediaFileFromUploadArgs {
maybe_media_class: Some(MediaFileClass::Image),
media_file_type,
maybe_creator_user_token: maybe_user_token,
// NB: AVT (anonymous visitor) tokens are a web-session concept; API-key callers have none.
maybe_creator_anonymous_visitor_token: None,
creator_ip_address: &ip_address,
creator_set_visibility,
upload_type,
maybe_engine_category: None,
maybe_animation_type: None,
maybe_mime_type: Some(&mimetype),
maybe_prompt_token: maybe_prompt_token.as_ref(),
maybe_batch_token: maybe_batch_token.as_ref(),
file_size_bytes: file_size_bytes as u64,
maybe_duration_millis: None,
sha256_checksum: &hash,
maybe_title: maybe_title.as_deref(),
maybe_scene_source_media_file_token: None,
is_intermediate_system_file,
maybe_generation_provider,
public_bucket_directory_hash: public_upload_path.get_object_hash(),
maybe_public_bucket_prefix: PREFIX,
maybe_public_bucket_extension: Some(&extension),
pool: &server_state.mysql_pool,
})
.await
.map_err(|err| {
warn!("New file creation DB error: {:?}", err);
MediaFileUploadError::ServerError
})?;
info!("new media file id: {} token: {:?}", record_id, &token);
Ok(Json(OmniUploadImageMediaFileSuccessResponse {
success: true,
media_file_token: token,
}))
}
fn validate_form(form: &OmniUploadImageMediaFileForm) -> Result<(), MediaFileUploadError> {
if form.uuid_idempotency_token.is_empty() {
return Err(MediaFileUploadError::BadInput("Idempotency token is required".to_string()));
}
let is_intermediate_system_file = form.is_intermediate_system_file
.as_ref()
.map(|b| b.0)
.unwrap_or(false);
if is_intermediate_system_file && form.maybe_prompt_token.is_some() {
return Err(MediaFileUploadError::BadInput(
"Cannot set `is_intermediate_system_file` to true if `maybe_prompt_token` is provided."
.to_string()));
}
Ok(())
}
/// Map an API-key authentication failure onto this endpoint's error type. A genuine 401 stays a
/// 401; anything else (e.g. a DB error during lookup) becomes a 500.
fn map_api_key_auth_error(err: CommonWebError) -> MediaFileUploadError {
match err {
CommonWebError::NotAuthorized => {
MediaFileUploadError::NotAuthorizedVerbose("invalid or missing API key".to_string())
}
_ => MediaFileUploadError::ServerError,
}
}
@@ -1,557 +0,0 @@
use std::collections::HashSet;
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use actix_multipart::form::tempfile::TempFile;
use actix_multipart::form::text::Text;
use actix_multipart::form::MultipartForm;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{debug, error, info, warn};
use once_cell::sync::Lazy;
use utoipa::ToSchema;
use artcraft_api_defs::omni_api::omni_upload_video::OmniUploadVideoMediaFileSuccessResponse;
use bucket_paths::legacy::typified_paths::public::media_files::bucket_file_path::MediaFileBucketPath;
use enums::by_table::media_files::media_file_class::MediaFileClass;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use ffmpeg_utils::ffmpeg::ffmpeg_transcode_to_mp4::{ffmpeg_transcode_to_mp4, FfmpegTranscodeToMp4Args};
use ffmpeg_utils::ffmpeg::ffmpeg_trim_and_resample::{ffmpeg_trim_and_resample, Args};
use ffmpeg_utils::ffprobe::ffprobe_get_info::ffprobe_get_info;
use filesys::file_read_bytes::file_read_bytes;
use filesys::path_to_string::path_to_string;
use hashing::sha256::sha256_hash_bytes::sha256_hash_bytes;
use http_server_common::request::get_request_ip::get_request_ip;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mimetypes::mimetype_to_extension::mimetype_to_extension;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use mysql_queries::queries::media_files::create::specialized_insert::insert_media_file_from_file_upload::{insert_media_file_from_file_upload, InsertMediaFileFromUploadArgs, UploadType};
use thumbnail_generator::task_client::thumbnail_task::{ThumbnailTaskBuilder, ThumbnailTaskInputMimeType};
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::prompts::PromptToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::media_files::upload::upload_error::MediaFileUploadError;
use crate::http_server::endpoints::media_files::upload::common_utils::try_parse_generation_provider::try_parse_generation_provider;
use crate::http_server::user_lookup::api_keys::require_api_key_user::require_api_key_user;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
/// Form-multipart request fields.
///
/// IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadVideoMediaFileForm` (Under "Schema") FOR DETAILS ON FIELDS AND NULLABILITY.
#[derive(MultipartForm, ToSchema)]
#[multipart(duplicate_field = "deny")]
pub struct OmniUploadVideoMediaFileForm {
/// UUID for request idempotency
#[multipart(limit = "2 KiB")]
#[schema(value_type = String, format = Binary)]
uuid_idempotency_token: Text<String>,
/// The uploaded file
#[multipart(limit = "512 MiB")]
#[schema(value_type = Vec<u8>, format = Binary)]
file: TempFile,
/// Optional: Title (name) of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_title: Option<Text<String>>,
/// Optional: Visibility of the scene
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<Visibility>, format = Binary)]
maybe_visibility: Option<Text<Visibility>>,
/// Optional: Prompt associated with this image
/// NOTE: Cannot set `is_intermediate_system_file = true` if this is set.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_prompt_token: Option<Text<PromptToken>>,
/// Optional: If an engine scene was used to generate this video, provide it here to create a link.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<MediaFileToken>, format = Binary)]
maybe_scene_source_media_file_token: Option<Text<MediaFileToken>>,
/// Optional: Whether this is a system file (eg. cover files we should hide)
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<bool>, format = Binary)]
is_intermediate_system_file: Option<Text<bool>>,
/// Optional: Trim start offset in milliseconds.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<u64>, format = Binary)]
maybe_trim_start_millis: Option<Text<u64>>,
/// Optional: Trim end offset in milliseconds.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<u64>, format = Binary)]
maybe_trim_end_millis: Option<Text<u64>>,
/// Optional: Resample the video to this FPS.
/// Only certain values equal or under 24fps are allowed.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<u64>, format = Binary)]
maybe_resample_fps: Option<Text<u8>>,
/// Optional: The third-party generation provider (e.g. "fal", "replicate").
/// If set, `is_user_upload` will be false and `is_intermediate_system_file` will be forced false.
#[multipart(limit = "2 KiB")]
#[schema(value_type = Option<String>, format = Binary)]
maybe_generation_provider: Option<Text<String>>,
}
static ALLOWED_MIME_TYPES : Lazy<HashSet<&'static str>> = Lazy::new(|| {
HashSet::from([
// Video
"video/mp4", // NB: Only mp4 for now.
])
});
static TRANSCODE_MIME_TYPES : Lazy<HashSet<&'static str>> = Lazy::new(|| {
HashSet::from([
// Video
"video/quicktime", // .qt quicktime files
])
});
/// API-key authenticated video upload (Omni API). Identity is read from the `Authorization` header
/// API key rather than a session cookie.
#[utoipa::path(
post,
tag = "Omni API",
path = "/v1/omni_api/upload/video",
responses(
(status = 200, description = "Success Update", body = OmniUploadVideoMediaFileSuccessResponse),
(status = 400, description = "Bad input", body = MediaFileUploadError),
(status = 401, description = "Not authorized", body = MediaFileUploadError),
(status = 429, description = "Too many requests", body = MediaFileUploadError),
(status = 500, description = "Server error", body = MediaFileUploadError),
),
params(
(
"request" = OmniUploadVideoMediaFileForm,
description = "IF VIEWING DOCS, PLEASE SEE BOTTOM OF PAGE `OmniUploadVideoMediaFileForm` (Under 'Schema') FOR DETAILS ON FIELDS AND NULLABILITY."
),
)
)]
pub async fn omni_upload_video_media_file_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>,
MultipartForm(mut form): MultipartForm<OmniUploadVideoMediaFileForm>,
) -> Result<Json<OmniUploadVideoMediaFileSuccessResponse>, MediaFileUploadError> {
fast_form_validations(&form)?;
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
error!("MySql pool error: {:?}", err);
MediaFileUploadError::ServerError
})?;
// ==================== API KEY USER ==================== //
// API-key authentication (Authorization header) instead of a session cookie. Never cached, and a
// banned owner is rejected inside `require_api_key_user`.
let api_session = require_api_key_user(&http_request, &mut *mysql_connection)
.await
.map_err(map_api_key_auth_error)?;
let maybe_user_token = Some(&api_session.user_token);
// ==================== RATE LIMIT ==================== //
let rate_limiter = &server_state.redis_rate_limiters.file_upload_logged_in;
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(MediaFileUploadError::RateLimited);
}
// ==================== HANDLE IDEMPOTENCY ==================== //
let uuid_idempotency_token = form.uuid_idempotency_token.as_ref();
if let Err(reason) = validate_idempotency_token_format(uuid_idempotency_token) {
return Err(MediaFileUploadError::BadInput(reason));
}
insert_idempotency_token(uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
MediaFileUploadError::BadInput("invalid idempotency token".to_string())
})?;
// ==================== UPLOAD METADATA ==================== //
let maybe_title = form.maybe_title
.map(|title| title.trim().to_string())
.filter(|title| !title.is_empty());
let creator_set_visibility = form.maybe_visibility
.map(|visibility| visibility.0)
.unwrap_or(Visibility::default());
// ==================== USER DATA ==================== //
let ip_address = get_request_ip(&http_request);
// ==================== FILE VALIDATION ==================== //
let mut file_bytes = Vec::new();
form.file.file.read_to_end(&mut file_bytes)
.map_err(|e| {
error!("Problem reading file: {:?}", e);
MediaFileUploadError::ServerError
})?;
let mut mimetype = get_mimetype_for_bytes(file_bytes.as_ref())
.map(|mimetype| mimetype.to_string())
.ok_or_else(|| {
warn!("Could not determine mimetype for file");
MediaFileUploadError::BadInput("Could not determine mimetype for file".to_string())
})?;
let needs_transcode = TRANSCODE_MIME_TYPES.contains(mimetype.as_str());
if !ALLOWED_MIME_TYPES.contains(mimetype.as_str()) && !needs_transcode {
// NB: Don't let our error message inject malicious strings
let filtered_mimetype = mimetype
.chars()
.filter(|c| c.is_ascii())
.filter(|c| c.is_alphanumeric() || *c == '/')
.collect::<String>();
return Err(MediaFileUploadError::BadInput(format!("unpermitted mime type: {}", &filtered_mimetype)));
}
// ==================== ORIGINAL FILE BOOKKEEPING ==================== //
// Hold on to the user's original file (path + mimetype + extension) so we
// can upload it as a `_original.{ext}` sibling later if we end up
// transcoding and/or resampling it.
let original_file_path: PathBuf = form.file.file.path().to_path_buf();
let original_mimetype: String = mimetype.clone();
let original_filename_extension: Option<String> = form.file.file_name.as_deref()
.and_then(|filename| std::path::Path::new(filename).extension())
.and_then(|ext| ext.to_str())
.map(|ext| ext.to_string());
let original_extension: String = mimetype_to_extension(&original_mimetype)
.map(|ext| ext.to_string())
.or(original_filename_extension)
.ok_or_else(|| {
warn!("Could not determine file extension for original mimetype: {}", &original_mimetype);
MediaFileUploadError::ServerError
})?;
// ==================== OPTIONAL TRANSCODE TO MP4 ==================== //
// Some accepted source mime types (e.g. QuickTime .mov) are not the canonical
// distribution format we want for the media file record. Transcode them to
// mp4 in a tempdir so we can resample / upload mp4 instead.
let mut transcode_tempdir_ref = None;
let mut resample_tempdir_ref = None;
let mut final_upload_file_path = form.file.file.path().to_path_buf();
if needs_transcode {
let transcode_tempdir = server_state.temp_dir_creator.new_tempdir("ffmpeg_transcode")
.map_err(|err| {
error!("Problem creating transcode temp dir: {:?}", err);
MediaFileUploadError::ServerError
})?;
let transcoded_path = transcode_tempdir.path().join("transcoded.mp4");
info!("Transcoding {} → mp4", &original_mimetype);
ffmpeg_transcode_to_mp4(FfmpegTranscodeToMp4Args {
video_input_path: form.file.file.path(),
video_output_path: &transcoded_path,
}).map_err(|err| {
error!("Problem transcoding video to mp4: {:?}", err);
MediaFileUploadError::ServerError
})?;
file_bytes = file_read_bytes(&transcoded_path)
.map_err(|e| {
error!("Problem reading transcoded mp4: {:?}", e);
MediaFileUploadError::ServerError
})?;
mimetype = "video/mp4".to_string();
final_upload_file_path = transcoded_path;
transcode_tempdir_ref = Some(transcode_tempdir);
}
// ==================== OPTIONAL VIDEO RESAMPLE ==================== //
let should_resample = form.maybe_resample_fps.is_some()
|| form.maybe_trim_start_millis.is_some()
|| form.maybe_trim_end_millis.is_some();
if should_resample {
let frame_temp_dir = server_state.temp_dir_creator.new_tempdir("ffmpeg")
.map_err(|err| {
error!("Problem creating temp dir: {:?}", err);
MediaFileUploadError::ServerError
})?;
let video_output_path = frame_temp_dir.path().join("output.mp4");
let maybe_new_frame_rate = form.maybe_resample_fps.map(|fps| fps.0);
let maybe_start_offset = form.maybe_trim_start_millis.map(|millis| Duration::from_millis(millis.0));
let maybe_end_offset = form.maybe_trim_end_millis.map(|millis| Duration::from_millis(millis.0));
ffmpeg_trim_and_resample(Args {
video_input_path: &final_upload_file_path,
video_output_path: &video_output_path,
maybe_new_frame_rate,
maybe_start_offset,
maybe_end_offset,
}).map_err(|err| {
error!("Problem resampling video: {:?}", err);
MediaFileUploadError::ServerError
})?;
file_bytes = file_read_bytes(&video_output_path)
.map_err(|e| {
error!("Problem reading file: {:?}", e);
MediaFileUploadError::ServerError
})?;
mimetype = get_mimetype_for_bytes(file_bytes.as_ref())
.map(|mimetype| mimetype.to_string())
.ok_or_else(|| {
warn!("Could not determine mimetype for file");
MediaFileUploadError::BadInput("Could not determine mimetype for file".to_string())
})?;
final_upload_file_path = video_output_path;
resample_tempdir_ref = Some(frame_temp_dir); // NB: Keep from going out of scope
}
// True if we touched the user's file in any way (transcoded, resampled, or both).
let should_upload_original = needs_transcode || should_resample;
// ==================== OTHER FILE METADATA ==================== //
let mut maybe_duration_millis = None;
match ffprobe_get_info(&final_upload_file_path) {
Ok(video_info) => {
maybe_duration_millis = video_info.duration
.map(|duration| duration.millis as u64);
}
Err(error) => {
warn!("Error reading video dimensions with ffprobe: {:?}", error);
}
}
let maybe_filename = form.file.file_name.as_deref()
.as_deref()
.map(|filename| PathBuf::from(filename));
let extension = mimetype_to_extension(&mimetype)
.or_else(|| {
maybe_filename
.as_ref()
.and_then(|filename| filename.extension())
.and_then(|ext| ext.to_str())
})
.ok_or_else(|| {
warn!("Could not determine file extension for mimetype: {}", &mimetype);
MediaFileUploadError::ServerError
})?;
let extension = format!(".{extension}"); // NB: needs dot prefix
let file_size_bytes = file_bytes.len();
let hash = sha256_hash_bytes(&file_bytes)
.map_err(|io_error| {
error!("Problem hashing bytes: {:?}", io_error);
MediaFileUploadError::ServerError
})?;
// ==================== UPLOAD AND SAVE ==================== //
const PREFIX : Option<&str> = Some("video_");
let public_upload_path = MediaFileBucketPath::generate_new(PREFIX, Some(&extension));
info!("Uploading media to bucket path: {}", public_upload_path.get_full_object_path_str());
server_state.public_bucket_client.upload_file_with_content_type(
public_upload_path.get_full_object_path_str(),
file_bytes.as_ref(),
&mimetype)
.await
.map_err(|e| {
warn!("Upload media bytes to bucket error: {:?}", e);
MediaFileUploadError::ServerError
})?;
// ==================== UPLOAD ORIGINAL SIBLING ==================== //
// If we transcoded and/or resampled the user's upload, also upload the
// unmodified original alongside the canonical file as `_original.{ext}`.
if should_upload_original {
let original_suffix = format!("_original.{}", original_extension);
let original_bucket_path = MediaFileBucketPath::from_object_hash(
public_upload_path.get_object_hash(),
PREFIX,
Some(&original_suffix),
);
info!(
"Uploading original sibling to bucket path: {}",
original_bucket_path.get_full_object_path_str(),
);
let original_bytes = file_read_bytes(&original_file_path)
.map_err(|e| {
error!("Problem reading original file for sibling upload: {:?}", e);
MediaFileUploadError::ServerError
})?;
server_state.public_bucket_client.upload_file_with_content_type(
original_bucket_path.get_full_object_path_str(),
original_bytes.as_ref(),
&original_mimetype,
)
.await
.map_err(|e| {
// NB: Sibling failure is non-fatal — the canonical upload already
// succeeded and the user can still use the media. Log loudly.
warn!("Upload original sibling to bucket error: {:?}", e);
})
.ok();
}
let maybe_scene_source_media_file_token = form.maybe_scene_source_media_file_token
.as_ref()
.map(|token| &token.0);
let maybe_prompt_token = form.maybe_prompt_token
.as_ref()
.map(|token| token.0.clone());
// NB: If we're uploading a video file that references an engine scene, then this is an engine
// render video, and we should mark it as a system (hidden) file.
let is_intermediate_system_file =
maybe_scene_source_media_file_token.is_some() ||
form.is_intermediate_system_file.map(|b| b.0).unwrap_or(false);
let maybe_generation_provider = form.maybe_generation_provider
.as_ref()
.and_then(|text| try_parse_generation_provider(text.as_ref()));
let upload_type = if maybe_generation_provider.is_some() {
UploadType::ThirdPartyInference
} else {
UploadType::Filesystem
};
let (token, record_id) = insert_media_file_from_file_upload(InsertMediaFileFromUploadArgs {
maybe_media_class: Some(MediaFileClass::Video),
media_file_type: MediaFileType::Video,
maybe_creator_user_token: maybe_user_token,
// NB: AVT (anonymous visitor) tokens are a web-session concept; API-key callers have none.
maybe_creator_anonymous_visitor_token: None,
creator_ip_address: &ip_address,
creator_set_visibility,
upload_type,
maybe_engine_category: None,
maybe_animation_type: None,
maybe_prompt_token: maybe_prompt_token.as_ref(),
maybe_batch_token: None,
maybe_mime_type: Some(&mimetype),
file_size_bytes: file_size_bytes as u64,
maybe_duration_millis,
sha256_checksum: &hash,
maybe_title: maybe_title.as_deref(),
maybe_scene_source_media_file_token,
is_intermediate_system_file, // NB: is_user_upload = TRUE
maybe_generation_provider,
public_bucket_directory_hash: public_upload_path.get_object_hash(),
maybe_public_bucket_prefix: PREFIX,
maybe_public_bucket_extension: Some(&extension),
pool: &server_state.mysql_pool,
})
.await
.map_err(|err| {
warn!("New file creation DB error: {:?}", err);
MediaFileUploadError::ServerError
})?;
info!("new media file id: {} token: {:?}", record_id, &token);
let thumbnail_task_result = ThumbnailTaskBuilder::new_for_source_mimetype(ThumbnailTaskInputMimeType::MP4)
.with_bucket(server_state.public_bucket_client.bucket_name().as_str())
.with_path(&*path_to_string(public_upload_path.to_full_object_pathbuf()))
.with_output_suffix("thumb")
.with_event_id(&token.to_string())
.send_all()
.await;
match thumbnail_task_result {
Ok(thumbnail_task) => {
debug!("Thumbnail tasks sent: {:?}", thumbnail_task);
},
Err(e) => {
error!("Failed to create some/all thumbnail tasks: {:?}", e);
}
}
Ok(Json(OmniUploadVideoMediaFileSuccessResponse {
success: true,
media_file_token: token,
}))
}
fn fast_form_validations(form: &OmniUploadVideoMediaFileForm) -> Result<(), MediaFileUploadError> {
if let Some(resample_fps) = form.maybe_resample_fps.as_ref() {
if **resample_fps > 24 {
return Err(MediaFileUploadError::BadInput("Resample FPS must be 24 or lower".to_string()));
} else if **resample_fps == 0 {
return Err(MediaFileUploadError::BadInput("Resample FPS must be greater than 0".to_string()));
}
}
form.maybe_trim_start_millis.as_ref().zip(form.maybe_trim_end_millis.as_ref())
.map(|(start, end)| {
if **start >= **end {
return Err(MediaFileUploadError::BadInput("Trim start must be less than trim end".to_string()));
}
Ok(())
})
.transpose()?;
let is_intermediate_system_file = form.is_intermediate_system_file
.as_ref()
.map(|b| b.0)
.unwrap_or(false);
if is_intermediate_system_file && form.maybe_prompt_token.is_some() {
return Err(MediaFileUploadError::BadInput(
"Cannot set `is_intermediate_system_file` to true if `maybe_prompt_token` is provided."
.to_string()));
}
Ok(())
}
/// Map an API-key authentication failure onto this endpoint's error type. A genuine 401 stays a
/// 401; anything else (e.g. a DB error during lookup) becomes a 500.
fn map_api_key_auth_error(err: CommonWebError) -> MediaFileUploadError {
match err {
CommonWebError::NotAuthorized => {
MediaFileUploadError::NotAuthorizedVerbose("invalid or missing API key".to_string())
}
_ => MediaFileUploadError::ServerError,
}
}
@@ -7,6 +7,5 @@ pub mod middleware;
pub mod requests;
pub mod routes;
pub mod session;
pub mod user_lookup;
pub mod validations;
pub mod web_utils;
@@ -11,7 +11,6 @@ use crate::http_server::routes::application_routes::generate_routes::add_generat
use crate::http_server::routes::application_routes::job_routes::add_job_routes;
use crate::http_server::routes::application_routes::media_files_routes::add_media_file_routes;
use crate::http_server::routes::application_routes::moderation_routes::add_moderator_routes;
use crate::http_server::routes::application_routes::omni_api_routes::add_omni_api_routes;
use crate::http_server::routes::application_routes::omni_gen_routes::add_omni_gen_routes;
use crate::http_server::routes::application_routes::prompts_routes::add_prompts_routes;
use crate::http_server::routes::application_routes::stripe_artcraft_routes::add_stripe_artcraft_routes;
@@ -49,7 +48,6 @@ where
// Artcraft surface area
app = add_generate_routes(app); // /v1/generate/...
app = add_omni_gen_routes(app); // /v1/omni_gen/...
app = add_omni_api_routes(app); // /v1/omni_api/... (API-key authenticated generate)
app = add_character_routes(app); // /v1/character/... and /v1/characters/...
app = add_webhook_routes(app); // /v1/webhooks/... (fal)
@@ -11,7 +11,6 @@ mod generate_routes;
mod job_routes;
mod media_files_routes;
mod moderation_routes;
mod omni_api_routes;
mod omni_gen_routes;
mod prompts_routes;
mod stripe_artcraft_routes;
@@ -1,53 +0,0 @@
use actix_http::body::MessageBody;
use actix_service::ServiceFactory;
use actix_web::dev::{ServiceRequest, ServiceResponse};
use actix_web::{web, App, Error, HttpResponse};
use crate::http_server::endpoints::omni_api::generate::image::omni_api_image_generate_handler::omni_api_image_generate_handler;
use crate::http_server::endpoints::omni_api::generate::video::omni_api_video_generate_handler::omni_api_video_generate_handler;
use crate::http_server::endpoints::omni_api::upload::omni_upload_audio_media_file_handler::omni_upload_audio_media_file_handler;
use crate::http_server::endpoints::omni_api::upload::omni_upload_image_media_file_handler::omni_upload_image_media_file_handler;
use crate::http_server::endpoints::omni_api::upload::omni_upload_video_media_file_handler::omni_upload_video_media_file_handler;
/// API-key authenticated generation endpoints. These mirror `/v1/omni_gen/generate/*` but read the
/// caller's identity from the `Authorization` header API key instead of a session cookie. The
/// `cost` and `models` endpoints are not duplicated here — they need no user and the existing
/// `/v1/omni_gen/{cost,models}/*` routes suffice.
pub fn add_omni_api_routes<T, B>(app: App<T>) -> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(web::scope("/v1/omni_api")
.service(web::scope("/generate")
.service(web::resource("/video")
.route(web::post().to(omni_api_video_generate_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/image")
.route(web::post().to(omni_api_image_generate_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
.service(web::scope("/upload")
.service(web::resource("/audio")
.route(web::post().to(omni_upload_audio_media_file_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/image")
.route(web::post().to(omni_upload_image_media_file_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/video")
.route(web::post().to(omni_upload_video_media_file_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
)
}
@@ -101,10 +101,6 @@ impl UserSessionFeatureFlags {
pub fn can_use_referrals_program(&self) -> bool {
self.feature_flags.contains(&UserFeatureFlag::ReferralsProgram)
}
pub fn can_create_api_key(&self) -> bool {
self.feature_flags.contains(&UserFeatureFlag::ApiKey)
}
}
#[cfg(test)]
@@ -1 +0,0 @@
pub mod require_api_key_user;
@@ -1,83 +0,0 @@
use std::marker::PhantomData;
use actix_web::HttpRequest;
use log::{info, warn};
use sqlx::MySqlConnection;
use mysql_queries::queries::api_keys::get_api_key_user_by_value::{
get_api_key_user_by_value, GetApiKeyUserByValueArgs,
};
use tokens::tokens::api_keys::ApiKeyToken;
use tokens::tokens::users::UserToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::web_utils::get_authorization_header_api_key::get_authorization_header_api_key;
/// The authenticated identity behind an API-key request — the API-key analogue of a cookie
/// `SessionUserRecord`, but deliberately minimal (just what the omni-api handlers need).
///
/// Unlike cookie sessions, this is **never** cached in Redis: every request performs a fresh MySQL
/// lookup.
pub struct UserApiSession {
pub api_key_token: ApiKeyToken,
pub user_token: UserToken,
pub is_banned: bool,
}
/// Authenticate a request using the API key in its `Authorization` header (see
/// `get_authorization_header_api_key` for the accepted formats).
///
/// Returns `CommonWebError::NotAuthorized` (401) when the header is missing/unparseable, when the
/// key does not match a live (non-soft-deleted) API key whose owning user still exists, or when
/// that owner is banned. `mysql_executor` can be any sqlx executor — pass `&mut *connection` to
/// reuse a connection the handler already holds.
pub async fn require_api_key_user(
http_request: &HttpRequest,
mysql_connection: &mut MySqlConnection,
) -> Result<UserApiSession, CommonWebError> {
let api_key = match get_authorization_header_api_key(http_request) {
Some(api_key) => api_key,
None => {
warn!("Missing or malformed Authorization header API key");
return Err(CommonWebError::NotAuthorized);
}
};
let maybe_record = get_api_key_user_by_value(GetApiKeyUserByValueArgs {
api_key: &api_key,
mysql_executor: &mut *mysql_connection,
phantom: PhantomData,
})
.await
.map_err(|err| {
warn!("API key user lookup error: {:?}", err);
CommonWebError::from_error(err)
})?;
// A missing or soft-deleted key (or a key whose owner no longer exists) is a 401, not a leak of
// which case occurred.
let record = match maybe_record {
Some(record) => record,
None => {
warn!("No live API key user for presented key: {:?}", api_key);
return Err(CommonWebError::NotAuthorized);
}
};
if record.is_banned {
warn!("API key owner is banned: {:?}", record.user_token.as_str());
return Err(CommonWebError::NotAuthorized);
}
info!(
"API key user authenticated: api_key_token={:?} user_token={:?}",
record.api_key_token.as_str(),
record.user_token.as_str(),
);
Ok(UserApiSession {
api_key_token: record.api_key_token,
user_token: record.user_token,
is_banned: record.is_banned,
})
}
@@ -1 +0,0 @@
pub mod api_keys;
@@ -16,7 +16,6 @@ import CreateImage from "../pages/create-image";
import CreateVideo from "../pages/create-video";
import CreateVFX from "../pages/create-vfx";
import PageScene from "../pages/pagescene";
import PageDraw from "../pages/pagedraw";
import VideoEditorPage from "../pages/video-editor";
import Pricing from "../pages/pricing";
import Support from "../pages/support/support";
@@ -150,7 +149,6 @@ export function App() {
<Route path="/background-change" element={<CreateVFX />} />
<Route path="/edit-3d" element={<PageScene />} />
<Route path="/edit-3d/:sceneToken" element={<PageScene />} />
<Route path="/edit-image" element={<PageDraw />} />
<Route path="/video-editor" element={<VideoEditorPage />} />
<Route
path="/video-editor/:projectId"
@@ -75,7 +75,6 @@ export interface LightboxDetailsProps {
onClose?: () => void;
onRecreate?: () => void;
onMakeVideo?: () => void;
onEditOnCanvas?: () => void;
onDelete?: () => void;
showDownloadAppCta?: boolean;
}
@@ -91,7 +90,6 @@ export function LightboxDetails({
onClose,
onRecreate,
onMakeVideo,
onEditOnCanvas,
onDelete,
showDownloadAppCta,
}: LightboxDetailsProps) {
@@ -476,16 +474,6 @@ export function LightboxDetails({
Make Video
</Button>
)}
{onEditOnCanvas && (
<Button
icon={faPencil}
className="w-full border border-ui-panel-border bg-ui-controls/40 hover:bg-ui-controls/60 text-white"
variant="secondary"
onClick={onEditOnCanvas}
>
Edit on Canvas
</Button>
)}
{onRecreate && promptData.hasToken && (
<Button
icon={faArrowRotateRight}
@@ -23,7 +23,6 @@ import {
type PromptData,
} from "./shared";
import { LightboxDetails } from "./LightboxDetails";
import { applyEditOnCanvasFromImage } from "../../pages/pagedraw/edit-on-canvas";
import {
applyMakeVideoFromImage,
applyRecreateFromMediaToken,
@@ -335,13 +334,6 @@ export function Lightbox({
applyMakeVideoFromImage(selectedMediaToken, selectedImageUrl, navigate);
}, [selectedMediaToken, selectedImageUrl, canMakeVideo, navigate, onClose]);
const canEditOnCanvas = !isVideo && !is3D;
const handleEditOnCanvas = useCallback(() => {
if (!selectedMediaToken || !selectedImageUrl || !canEditOnCanvas) return;
onClose();
applyEditOnCanvasFromImage(selectedMediaToken, selectedImageUrl, navigate);
}, [selectedMediaToken, selectedImageUrl, canEditOnCanvas, navigate, onClose]);
return (
<>
<Modal
@@ -513,11 +505,6 @@ export function Lightbox({
? handleMakeVideo
: undefined
}
onEditOnCanvas={
canEditOnCanvas && selectedMediaToken && selectedImageUrl
? handleEditOnCanvas
: undefined
}
onDelete={
selectedMediaToken
? () => setConfirmDeleteOpen(true)
@@ -63,9 +63,7 @@ export function ApiKeySection(_props: ApiKeySectionProps) {
// The list includes soft-deleted keys; only show live ones.
const live = fetched.filter((k) => k.maybe_deleted_at === null);
setKeys((prev) => (reset ? live : [...prev, ...live]));
setFetchedCount((prev) =>
reset ? fetched.length : prev + fetched.length,
);
setFetchedCount((prev) => (reset ? fetched.length : prev + fetched.length));
setHasMore(fetched.length === PAGE_SIZE);
} else {
toast.error(response.errorMessage ?? "Could not load API keys.");
@@ -100,8 +98,8 @@ export function ApiKeySection(_props: ApiKeySectionProps) {
};
return (
<div className="flex flex-col gap-3">
<div className="flex items-start justify-between gap-3 mb-2">
<div className="flex flex-col gap-6">
<div className="flex items-start justify-between gap-3">
<div className="flex flex-col gap-0.5">
<p className="text-sm font-medium">API keys</p>
<p className="text-xs opacity-70">
@@ -135,6 +133,8 @@ export function ApiKeySection(_props: ApiKeySectionProps) {
/>
)}
<hr className="border-ui-panel-border" />
{loading ? (
<div className="py-6 text-center text-xs opacity-60">
<FontAwesomeIcon icon={faSpinnerThird} className="animate-spin" />
@@ -299,20 +299,19 @@ function NewKeyReveal({
</code>
<Button
type="button"
variant="primary"
className="h-8 px-3 shrink-0"
variant="secondary"
className="h-9 px-3 shrink-0"
onClick={handleCopy}
>
<FontAwesomeIcon
icon={copied ? faCheck : faCopy}
className="mr-1.5"
/>
<FontAwesomeIcon icon={copied ? faCheck : faCopy} className="mr-1.5" />
{copied ? "Copied" : "Copy"}
</Button>
</div>
<div className="flex justify-end pt-0.5">
<Button
type="button"
variant="secondary"
className="h-8 px-3 shrink-0"
className="h-8 px-3"
onClick={onDismiss}
>
Done
@@ -340,7 +339,7 @@ function ApiKeyRow({
onRequestDelete: () => void;
}) {
return (
<div className="flex flex-col gap-2 rounded-lg border border-white/10 bg-white/[0.03] p-3 transition-colors hover:border-white/[0.14]">
<div className="flex flex-col gap-2 rounded-lg border border-ui-panel-border p-3">
<div className="flex items-start justify-between gap-3">
<div className="flex min-w-0 flex-col gap-0.5">
<p className="truncate text-sm font-medium">{item.name}</p>
@@ -350,10 +349,9 @@ function ApiKeyRow({
</p>
)}
<div className="flex items-center gap-2 pt-1">
{/* Truncated prefix, display-only — the full secret is only shown on
create. The trailing ellipsis signals it's not the whole value. */}
<code className="min-w-0 truncate rounded bg-black/40 px-2 py-1 font-mono text-[11px] text-white/70">
{item.truncated_api_key}…
{/* Truncated, display-only — the full secret is only shown on create. */}
<code className="min-w-0 truncate rounded bg-black/30 px-2 py-1 font-mono text-[11px] text-white/70">
{item.truncated_api_key}
</code>
<span className="text-[10px] uppercase tracking-wider text-white/40">
{formatDate(item.created_at)}
@@ -367,7 +365,6 @@ function ApiKeyRow({
variant="secondary"
className="h-8 px-3"
onClick={onOpenEdit}
title="Edit description"
>
Edit
</Button>
@@ -441,9 +438,6 @@ function EditDescriptionForm({
return (
<form onSubmit={handleSubmit} className="flex flex-col gap-2 pt-1">
<p className="text-xs opacity-70">
Edit the description for "{item.name}". The key's name can't be changed.
</p>
<Input
type="text"
value={value}
@@ -167,7 +167,7 @@ function ApiKeysPanel() {
}
return (
<div className="pb-4 pt-3">
<div className="pt-3">
<ApiKeySection user={user} />
</div>
);
@@ -6,7 +6,6 @@ import {
faVideo,
faImage,
faCube,
faPencil,
faFilm,
faWandMagicSparkles,
faGraduationCap,
@@ -54,7 +53,6 @@ const PRIMARY_ITEMS: NavItem[] = [{ label: "Home", href: "/", icon: faHouse }];
const CREATE_ITEMS_STATIC: NavItem[] = [
{ label: "Image", href: "/create-image", icon: faImage },
{ label: "Video", href: "/create-video", icon: faVideo },
{ label: "Edit Image", href: "/edit-image", icon: faPencil },
{ label: "Edit 3D", href: "/edit-3d", icon: faCube },
{ label: "Edit Video", href: "/video-editor", icon: faFilm, badge: "BETA" },
{
@@ -1,223 +0,0 @@
// The "no base image yet" state for the webapp's image editor.
//
// PageDraw renders this (via adapter.renderBaseImageSelector) until the user
// picks a base image. It offers three entry points, mirroring the Tauri app's
// BaseImageSelector but built from the webapp's own primitives:
// - upload a local file → webapp uploadImage helper → media token
// - pick from your library → GalleryModal in select mode
// - start from a blank canvas → BlankCanvasModal (from the lib)
import { useState } from "react";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import {
faExpand,
faImages,
faPencil,
faSpinnerThird,
} from "@fortawesome/pro-solid-svg-icons";
import { UploaderStates } from "@storyteller/common";
import { MediaFilesApi } from "@storyteller/api";
import {
GalleryModal,
type GalleryItem,
} from "@storyteller/ui-gallery-modal";
import {
BlankCanvasModal,
type BaseSelectorImage,
} from "@storyteller/ui-pagedraw";
import { uploadImage } from "../../components/prompt-box/upload-image";
import { showToast } from "../../components/toast/toast";
type BaseImageSelectorProps = {
onImageSelect: (image: BaseSelectorImage) => void;
showLoading?: boolean;
};
export function BaseImageSelector({
onImageSelect,
showLoading = false,
}: BaseImageSelectorProps) {
const [isGalleryOpen, setIsGalleryOpen] = useState(false);
const [gallerySelectedIds, setGallerySelectedIds] = useState<string[]>([]);
const [isBlankCanvasOpen, setIsBlankCanvasOpen] = useState(false);
const [isUploading, setIsUploading] = useState(false);
const [isDragging, setIsDragging] = useState(false);
const busy = isUploading || showLoading;
const handleGallerySelect = (id: string) =>
// Single-select: clicking always swaps to the latest pick.
setGallerySelectedIds((prev) => (prev.includes(id) ? [] : [id]));
const handleUseGalleryImages = (items: GalleryItem[]) => {
const item = items[0];
if (!item) {
showToast("error", "No image selected");
return;
}
setIsGalleryOpen(false);
setGallerySelectedIds([]);
onImageSelect({
mediaToken: item.id,
url: item.fullImage || item.thumbnail || "",
fullImageUrl: item.fullImage || undefined,
thumbnailUrlTemplate: item.thumbnailUrlTemplate,
});
};
const handleBlankCanvasConfirm = (width: number, height: number) => {
setIsBlankCanvasOpen(false);
onImageSelect({
url: "",
mediaToken: `blank_canvas_${width}x${height}_${Math.random()
.toString(36)
.substring(2, 8)}`,
isBlankCanvas: true,
blankCanvasWidth: width,
blankCanvasHeight: height,
});
};
const handleFile = (file: File) => {
setIsUploading(true);
const reader = new FileReader();
reader.onloadend = async () => {
const dataUrl = reader.result as string;
await uploadImage({
title: `edit-image-${Math.random().toString(36).substring(2, 15)}`,
assetFile: file,
progressCallback: (state) => {
if (state.status === UploaderStates.success && state.data) {
void resolveAndSelect(state.data, dataUrl);
} else if (
state.status === UploaderStates.assetError ||
state.status === UploaderStates.imageCreateError
) {
showToast("error", "Upload failed. Please try again.");
setIsUploading(false);
}
},
});
};
reader.readAsDataURL(file);
};
// The uploader hands back a media token immediately; fetch the media file to
// swap the local data URL for a durable CDN URL (and grab the thumbnail
// template) before handing the base image to the editor.
const resolveAndSelect = async (mediaToken: string, fallbackUrl: string) => {
let url = fallbackUrl;
let thumbnailUrlTemplate: string | undefined;
try {
const result = await new MediaFilesApi().GetMediaFileByToken({
mediaFileToken: mediaToken,
});
if (result.success && result.data) {
url = result.data.media_links?.cdn_url || url;
thumbnailUrlTemplate =
result.data.media_links?.thumbnail_template ?? thumbnailUrlTemplate;
}
} catch {
// Fall back to the data URL — the editor can still render it.
}
onImageSelect({ mediaToken, url, fullImageUrl: url, thumbnailUrlTemplate });
setIsUploading(false);
};
const onFileInput = (e: React.ChangeEvent<HTMLInputElement>) => {
const file = e.target.files?.[0];
if (file) handleFile(file);
e.target.value = "";
};
const onDrop = (e: React.DragEvent<HTMLLabelElement>) => {
e.preventDefault();
setIsDragging(false);
const file = e.dataTransfer.files?.[0];
if (file?.type.startsWith("image/")) handleFile(file);
};
return (
<>
<label
onDragOver={(e) => {
e.preventDefault();
setIsDragging(true);
}}
onDragLeave={() => setIsDragging(false)}
onDrop={onDrop}
className={`flex h-full w-full cursor-pointer flex-col items-center justify-center gap-4 rounded-2xl border-2 border-dashed p-8 text-center transition-colors ${
isDragging
? "border-blue-400 bg-blue-500/10"
: "border-ui-panel-border bg-ui-background hover:border-blue-400/50"
} ${busy ? "pointer-events-none opacity-60" : ""}`}
>
<input
type="file"
accept="image/*"
className="hidden"
onChange={onFileInput}
disabled={busy}
/>
<div className="flex h-14 w-14 items-center justify-center rounded-2xl border border-blue-400/30 bg-blue-500/30 text-blue-300">
<FontAwesomeIcon
icon={busy ? faSpinnerThird : faPencil}
className={`text-xl ${busy ? "animate-spin" : ""}`}
/>
</div>
<div>
<h2 className="text-lg font-semibold text-base-fg">Edit Image</h2>
<p className="mt-1 text-sm text-base-fg/60">
Click to upload or drag and drop an image here to edit
</p>
</div>
<div className="mt-2 flex flex-wrap items-center justify-center gap-2">
<button
type="button"
onClick={(e) => {
e.preventDefault();
setIsGalleryOpen(true);
}}
disabled={busy}
className="flex items-center gap-2 rounded-lg border border-ui-controls-border bg-ui-controls px-3 py-2 text-sm font-medium text-base-fg transition-colors hover:bg-ui-controls/80"
>
<FontAwesomeIcon icon={faImages} />
Pick from Library
</button>
<button
type="button"
onClick={(e) => {
e.preventDefault();
setIsBlankCanvasOpen(true);
}}
disabled={busy}
className="flex items-center gap-2 rounded-lg border border-ui-controls-border bg-ui-controls px-3 py-2 text-sm font-medium text-base-fg transition-colors hover:bg-ui-controls/80"
>
<FontAwesomeIcon icon={faExpand} />
Blank Canvas
</button>
</div>
</label>
<GalleryModal
mode="select"
isOpen={isGalleryOpen}
onClose={() => {
setIsGalleryOpen(false);
setGallerySelectedIds([]);
}}
selectedItemIds={gallerySelectedIds}
onSelectItem={handleGallerySelect}
maxSelections={1}
onUseSelected={handleUseGalleryImages}
forceFilter="image"
/>
<BlankCanvasModal
isOpen={isBlankCanvasOpen}
onClose={() => setIsBlankCanvasOpen(false)}
onConfirm={handleBlankCanvasConfirm}
/>
</>
);
}
@@ -1,27 +0,0 @@
// "Edit on Canvas" handoff: open an existing image in the PageDraw editor.
//
// Mirrors the Tauri app's handleEditFromGallery — RESET the editor, seed the
// chosen image as the base (and into history), then route to /edit-image.
// Seeding the lib's store before navigating means the editor mounts straight
// into the image, same store-based handoff the create-image/create-video
// "Recreate" and "Make Video" buttons use (see lib/recreate.ts).
import type { NavigateFunction } from "react-router-dom";
import { useSceneStore } from "@storyteller/ui-pagedraw";
export function applyEditOnCanvasFromImage(
mediaToken: string,
mediaUrl: string,
navigate: NavigateFunction,
): void {
const store = useSceneStore.getState();
const baseImage = {
url: mediaUrl,
mediaToken,
fullImageUrl: mediaUrl,
};
store.RESET();
store.addHistoryImageBundle({ images: [baseImage] });
store.setBaseImageInfo(baseImage);
navigate("/edit-image");
}
@@ -1 +0,0 @@
export { default } from "./pagedraw";
@@ -1,141 +0,0 @@
// Job polling for the PageDraw web adapter.
//
// Mirrors the create-image / create-vfx job loops: enqueue returns an
// inference job token, then we poll `GET /v1/jobs/job/<token>` until the
// job reaches a terminal state. On success we resolve the result media —
// preferring the batch (so multi-image generations all land) and falling
// back to the single media_links payload.
//
// Kept local to the pagedraw page (rather than imported from create-image)
// so the editor owns its own result shape and isn't coupled to that page's
// store types — same separation the artcraft-website pagedraw port used.
import { JobsApi, MediaFilesApi } from "@storyteller/api";
export interface PolledImage {
media_token: string;
cdn_url: string;
maybe_thumbnail_template?: string;
}
export type PollResult =
| { status: "pending"; images: PolledImage[] }
| { status: "complete"; images: PolledImage[] }
| { status: "failed"; images: PolledImage[]; error: string };
export async function pollJobOnce(jobToken: string): Promise<PollResult> {
const jobsApi = new JobsApi();
const response = await jobsApi.GetJobByToken({ token: jobToken });
if (!response.success || !response.data) {
return { status: "pending", images: [] };
}
const state = response.data;
const statusStr = state.status?.status?.toLowerCase() ?? "";
if (statusStr === "complete_success" || statusStr === "complete") {
const result = state.maybe_result as Record<string, unknown> | undefined;
const mediaLinks = (result as any)?.media_links;
const entityToken = (result as any)?.entity_token as string | undefined;
if (entityToken) {
try {
const mediaApi = new MediaFilesApi();
const mediaFile = await mediaApi.GetMediaFileByToken({
mediaFileToken: entityToken,
});
const batchToken = mediaFile.data?.maybe_batch_token;
if (batchToken) {
const batchResponse = await mediaApi.GetMediaFilesByBatchToken({
batchToken,
});
if (batchResponse.success && batchResponse.data?.length) {
const images: PolledImage[] = batchResponse.data
.map((file: any) => ({
media_token: file.token,
cdn_url: file.media_links?.cdn_url,
maybe_thumbnail_template:
file.media_links?.maybe_thumbnail_template,
}))
.filter((img: PolledImage) => img.cdn_url);
if (images.length > 0) return { status: "complete", images };
}
}
} catch {
// Fall through to single-image extraction.
}
}
if (mediaLinks?.cdn_url) {
return {
status: "complete",
images: [
{
media_token: entityToken ?? jobToken,
cdn_url: mediaLinks.cdn_url,
maybe_thumbnail_template: mediaLinks.maybe_thumbnail_template,
},
],
};
}
return { status: "complete", images: [] };
}
if (
statusStr.includes("fail") ||
statusStr.includes("error") ||
statusStr === "dead"
) {
return {
status: "failed",
images: [],
error:
state.status?.maybe_failure_message ??
state.status?.maybe_extra_status_description ??
"Generation failed",
};
}
return { status: "pending", images: [] };
}
export function startPolling(
jobToken: string,
onComplete: (images: PolledImage[]) => void,
onError: (reason: string) => void,
intervalMs = 3000,
maxAttempts = 120,
): () => void {
let attempts = 0;
let stopped = false;
const poll = async () => {
if (stopped) return;
attempts++;
try {
const result = await pollJobOnce(jobToken);
if (stopped) return;
if (result.status === "complete") return onComplete(result.images);
if (result.status === "failed") return onError(result.error);
if (attempts >= maxAttempts) return onError("Generation timed out");
setTimeout(poll, intervalMs);
} catch {
if (!stopped && attempts < maxAttempts) {
setTimeout(poll, intervalMs * 2);
} else if (!stopped) {
onError("Network error during polling");
}
}
};
setTimeout(poll, 2000);
return () => {
stopped = true;
};
}
@@ -1,25 +0,0 @@
import { PageDraw as PageDrawLib } from "@storyteller/ui-pagedraw";
import Seo from "../../components/seo";
import { useWebPageDrawAdapter } from "./web-adapter";
export default function PageDraw() {
const adapter = useWebPageDrawAdapter();
return (
<>
<Seo
title="Edit Image - ArtCraft"
description="Edit and recompose images on an AI canvas, then generate new variations."
/>
{/* PageDraw lays its canvas and toolbars out with position:fixed.
translateZ(0) makes this div their containing block so they scope
to the SidebarInset area instead of covering the whole window —
same technique the Edit 3D page uses for its lib overlays. */}
<div
className="relative h-full w-full overflow-hidden"
style={{ transform: "translateZ(0)" }}
>
<PageDrawLib adapter={adapter} />
</div>
</>
);
}
@@ -1,275 +0,0 @@
// PageDrawAdapter implementation for the artcraft webapp.
//
// Plugs the shared @storyteller/ui-pagedraw lib into the webapp by speaking
// REST instead of Tauri commands:
// - enqueueEditImage → OmniGenApi.generateImage, then poll the job and feed
// results back into the lib's Zustand store (addHistoryImageBundle /
// resolvePendingGeneration), keyed on frontendSubscriberId.
// - enqueueBgRemoval → upload base64 → media token → POST
// /v1/generate/image/remove_background → poll → finishRemoveBackground.
// - uploadImage → UploadImageMedia (shared REST uploader).
// - renderBaseImageSelector → the webapp's own upload/library/blank picker.
//
// Inpaint is intentionally stubbed: the omni-gen REST endpoint doesn't yet
// accept a raw mask payload, so we surface a clear toast until that path
// exists. Mirrors the artcraft-website pagedraw port.
import { useMemo } from "react";
import {
type PageDrawAdapter,
type ImageBundle,
type BaseSelectorImage,
useSceneStore,
} from "@storyteller/ui-pagedraw";
import {
OmniGenApi,
StorytellerApiHostStore,
UploadImageMedia,
type OmniGenImageRequest,
} from "@storyteller/api";
import { UploaderStates } from "@storyteller/common";
import { showToast } from "../../components/toast/toast";
import { useInsufficientCredits } from "../../components/insufficient-credits-modal";
import { BaseImageSelector } from "./base-image-selector";
import { startPolling, type PolledImage } from "./job-polling";
// ─── Local bg-removal call ──────────────────────────────────────────────────
// Inlined (no dedicated method in @storyteller/api yet). Mirrors ApiManager's
// auth: cookies via credentials:"include", plus the signed-session header
// fallback for browsers that block third-party cookies.
const SESSION_STORAGE_KEY = "artcraft_signed_session";
interface RemoveBgRequest {
uuid_idempotency_token: string;
media_file_token: string;
}
interface RemoveBgResponse {
success: boolean;
inference_job_token: string;
}
const sessionHeaders = (): Record<string, string> => {
const headers: Record<string, string> = {
Accept: "application/json",
"Content-Type": "application/json",
};
try {
const session = localStorage.getItem(SESSION_STORAGE_KEY);
if (session) headers.session = session;
} catch {
// localStorage unavailable — rely on the cookie.
}
return headers;
};
const removeImageBackground = async (
body: RemoveBgRequest,
): Promise<RemoveBgResponse> => {
const host = StorytellerApiHostStore.getInstance().getApiSchemeAndHost();
const response = await fetch(`${host}/v1/generate/image/remove_background`, {
method: "POST",
headers: sessionHeaders(),
credentials: "include",
body: JSON.stringify(body),
});
if (!response.ok) {
throw new Error(`Background removal HTTP ${response.status}`);
}
return response.json();
};
// ─── Helpers ─────────────────────────────────────────────────────────────────
const polledImagesToBundle = (images: PolledImage[]): ImageBundle => ({
images: images.map(
(img) =>
({
url: img.cdn_url,
mediaToken: img.media_token,
thumbnailUrlTemplate: img.maybe_thumbnail_template,
fullImageUrl: img.cdn_url,
}) as BaseSelectorImage,
),
});
const startGenerationPoll = (jobToken: string, subscriberId: string) => {
startPolling(
jobToken,
(images) => {
const store = useSceneStore.getState();
// The user may have dismissed the placeholder while the job ran.
if (!store.pendingGenerations.some((p) => p.id === subscriberId)) return;
store.addHistoryImageBundle(polledImagesToBundle(images));
store.resolvePendingGeneration(subscriberId);
},
(reason) => {
showToast("error", reason);
useSceneStore.getState().resolvePendingGeneration(subscriberId);
},
);
};
const base64ToBlob = (base64: string): Blob => {
const commaIdx = base64.indexOf(",");
const raw = commaIdx >= 0 ? base64.slice(commaIdx + 1) : base64;
const headerMatch =
commaIdx >= 0 ? base64.slice(0, commaIdx).match(/data:(.*?);base64/) : null;
const mime = headerMatch?.[1] ?? "image/png";
const binary = atob(raw);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i);
return new Blob([bytes], { type: mime });
};
const uploadBlobAsImage = async (
blob: Blob,
title: string,
): Promise<string | undefined> => {
let mediaToken: string | undefined;
const file = new File([blob], `${title}.png`, { type: blob.type });
await UploadImageMedia({
title,
assetFile: file,
progressCallback: (state) => {
if (state.status === UploaderStates.success && state.data) {
mediaToken = state.data;
}
},
});
return mediaToken;
};
// PageDrawEditRequest.model is an ImageModel object (or possibly a raw id);
// the omni-gen REST body wants the string id.
const modelIdOf = (model: unknown): string => {
if (!model) return "";
if (typeof model === "string") return model;
const m = model as { id?: string; tauriId?: string };
return m.id ?? m.tauriId ?? String(model);
};
// ApiManager throws Error("HTTP error! status: 402") on a non-2xx response, and
// our inlined bg-removal fetch throws "Background removal HTTP 402"; the JSON
// body isn't surfaced either way. Pull the 3-digit status out of the message so
// callers can special-case 402 Payment Required.
const httpStatusFromError = (e: unknown): number | undefined => {
const message = e instanceof Error ? e.message : String(e);
const match = /(\d{3})/.exec(message);
return match ? Number(match[1]) : undefined;
};
// ─── Adapter ───────────────────────────────────────────────────────────────────
export const useWebPageDrawAdapter = (): PageDrawAdapter => {
const openInsufficientCredits = useInsufficientCredits();
return useMemo<PageDrawAdapter>(
() => ({
enqueueEditImage: async (req) => {
const refTokens = [
...(req.canvasImageMediaToken ? [req.canvasImageMediaToken] : []),
...(req.imageMediaTokens ?? []),
].filter((t): t is string => !!t);
const body: OmniGenImageRequest = {
model: modelIdOf(req.model),
prompt: req.prompt ?? null,
idempotency_token: crypto.randomUUID(),
aspect_ratio: req.aspectRatio ?? null,
resolution: req.imageResolution ?? null,
image_batch_count: req.imageCount ?? 1,
image_media_tokens: refTokens.length ? refTokens : null,
};
try {
const response = await new OmniGenApi().generateImage(body);
if (
response.success &&
response.inference_job_token &&
req.frontendSubscriberId
) {
startGenerationPoll(
response.inference_job_token,
req.frontendSubscriberId,
);
return { status: "success" };
}
showToast("error", "Generation failed to enqueue");
return { status: "fail" };
} catch (e: any) {
// 402 Payment Required → the user is out of credits. Open the
// upgrade/credits modal instead of a raw error toast, matching the
// create-image / create-video generate handlers.
if (httpStatusFromError(e) === 402) {
openInsufficientCredits();
return { status: "fail" };
}
showToast("error", `Generation error: ${e?.message ?? e}`);
return { status: "fail" };
}
},
enqueueInpaint: async (req) => {
// The omni-gen REST endpoint doesn't yet accept a raw mask payload;
// surface clearly until a multipart/mask-upload path exists.
if (!req.maskImageRawBytes) {
showToast("error", "Inpaint mask missing");
return { status: "fail" };
}
showToast("error", "Inpainting is not yet available on the web");
return { status: "fail" };
},
enqueueBgRemoval: async (base64Image, nodeId) => {
try {
const blob = base64ToBlob(base64Image);
const mediaToken = await uploadBlobAsImage(
blob,
`bg-removal-${nodeId}`,
);
if (!mediaToken) {
showToast("error", "Background removal upload failed");
return;
}
const response = await removeImageBackground({
uuid_idempotency_token: crypto.randomUUID(),
media_file_token: mediaToken,
});
if (!response.success || !response.inference_job_token) {
showToast("error", "Background removal failed to enqueue");
return;
}
startPolling(
response.inference_job_token,
async (images) => {
const out = images[0];
if (!out) return;
await useSceneStore
.getState()
.finishRemoveBackground(nodeId, out.media_token, out.cdn_url);
},
(reason) => showToast("error", reason),
);
} catch (e: any) {
if (httpStatusFromError(e) === 402) {
openInsufficientCredits();
return;
}
showToast("error", `Background removal error: ${e?.message ?? e}`);
}
},
uploadImage: UploadImageMedia,
renderBaseImageSelector: ({ onImageSelect, showLoading }) => (
<BaseImageSelector
onImageSelect={onImageSelect}
showLoading={showLoading}
/>
),
}),
[openInsufficientCredits],
);
};