Compare commits

..
Author SHA1 Message Date
Tanish Manku e9ea2a938c Scene Builder: T-pose in clip gaps + drag-preview card for animations
- Gaps show the default pose: a disabled three.js action leaves the skeleton
  frozen on its last frame, so evaluateAt now resets any character with no clip
  under the playhead to its bind (T) pose via Skeleton.pose() (skeletons cached
  per character). Fixes blank space after a clip holding the clip's last pose.
- Animation drags now show the shared DragGhost tilt card with the clip
  thumbnail: the drawer's native-DnD handlers drive dragItem/dragPosition/
  assetDraggingUnder and suppress the browser's default drag image, matching
  object/character pickup motion.
2026-07-09 00:59:24 -07:00
Tanish Manku dc5608613d webapp: stop dev API host override pointing at localhost:12345
The DEV block set the API host to window.location.origin (so /v1 calls hit the
Vite proxy → api.storyteller.ai) and then immediately called setDevelopment(),
which overrode it to http://localhost:12345 — a local backend that isn't
running. Every media-token resolution failed, so getMediaUrlByToken returned ''
and GLB fetches fell back to localhost, serving nothing. The inline comment
already described this line as disabled; it just wasn't commented out. Now it
is, so local dev resolves media through the proxy as intended.
2026-07-09 00:59:24 -07:00
Tanish Manku 1a90bd54c2 Scene Builder: one clip row per character + overlap guard
Collapse the per-character clips onto a single timeline row (playback is
sequential — one clip at a time), replacing stacked lanes. Stacking only earned
its keep for additive layering, which is out of scope.

- TimelineController: add/move route through resolveFreeStart (snap to nearest
  non-overlapping slot); resizeClipLane and resolveClipDuration clamp against
  nextStartAfter so a strip never overruns the following clip. A character's
  strips stay a non-overlapping sequence, so evaluateAt has at most one active
  clip at any playhead.
- TimelineClipRow: renders all of a character's strips on one shared row
  (shown even when empty, as a drop hint); per-strip move/trim/loop and an
  inline x to remove.
- TimelineEditor: one TimelineClipRow per character instead of one per lane.
2026-07-08 00:04:13 -07:00
Tanish Manku 2920a189f0 Scene Builder: Mixamo clip-lane timeline UI + real clip length (phase C)
Stacked clip-strip rendering on the timeline with move/trim/loop/delete, drag-
and-drop from the Animations drawer onto a character's lane, and backfill of the
clip's real length into the strip once its GLB loads.

- TimelineClipRow: one stacked strip per clip lane under its character's
  keyframe row; body drags to move, right edge trims length, loop chip toggles
  repeat, trash removes. Clip edits ride the timeline Save/Cancel session (no
  per-op undo, matching keyframes).
- TimelineEditor: each character row is a drop target for clip drags; drop time
  is read from the ruler rect. Non-character rows ignore clip drags.
- AnimationsDrawer clips are now draggable (ANIMATION_CLIP_MIME) as well as
  click-to-add.
- ClipStrip.autoDuration: a fresh drop seeds a placeholder width, then
  CharacterAnimationManager.resolveClipDuration adopts the loaded clip's true
  length (re-clamping start); a user trim clears the flag so reload never
  clobbers a hand-set duration.
- clipBindsToCharacter warns when a clip resolves 0 tracks on the character —
  the diagnostic for whether retargeting is needed.
2026-07-07 23:51:37 -07:00
Tanish Manku 7f5e82cbd9 Scene Builder: Mixamo clip playback engine + drawer (phase B, spike)
Deterministic, playhead-driven skeletal-animation playback for character clip
lanes, plus a minimal click-to-add drawer so retargeting/binding can be verified
in the viewport before the full drag-drop / stacked-strip UI is built.

- CharacterAnimationManager: one AnimationMixer per character; each clip lane is
  an AnimationAction posed at an absolute time from the playhead + mixer.update(0)
  (scrub / play / record all pose identically). syncClipLanes() reconciles the
  runtime on every lane change; a load token guards superseded async clip loads.
- Scene.loadRawGlb(media_id): raw GLTF (scene + animations) with no scene-add /
  placeholder — sources clips for binding. Clips bind by node name (shared
  mixamorig:* naming); SkeletonUtils.retargetClip is the documented fallback.
- TimelineController: getPlayhead(); evaluate() poses characters; sync on
  add/move/remove clip lane + load/cancel/clear.
- actions: addClipToCharacter / removeClipLane.
- comps/AnimationsDrawer: right-docked, gated on a selected character; click a
  curated clip to drop it as a lane at the playhead.

Compiles clean (tsc exit 0); playback is NOT yet runtime-verified.
2026-07-07 15:42:24 -07:00
Tanish Manku 5c2fbbc9f3 Scene Builder: animation clip-lane data model (Mixamo drawer, phase A)
Foundation for placing Mixamo skeletal-animation clips on the timeline as
stacked lanes under character objects, alongside the existing keyframe tracks.

- ClipStrip / ClipLane types; TimelineData gains a `clipLanes` array (only the
  clip reference + placement is serialized — the AnimationClip is resolved and
  retargeted from sourceMediaId at runtime).
- TimelineController: addClipLane / moveClipLane / removeClipLane / clipLanesFor,
  create()/loadTimeline() init + backfill clipLanes, emitChanged carries them.
- TimelineChangedEvent → store mirror (timelineClipLanes) via EngineStoreBridge.
- Persistence rides the existing timeline save/load (getTimeline/loadTimeline).
2026-07-07 15:28:50 -07:00
Tanish Manku 11c7aa979d Scene Builder fixes: record-mode camera lock, Blender default framing, auto-keyframe
- Record mode is now fully immutable: gate MouseControls mouse-look + focus()
  on the viewport lock (new MouseControlsDeps.isViewportLocked → cameraController
  .locked), matching the existing FreeCam/tick gates.
- Default main (viewport) camera pulled back + elevated to a Blender-style 3/4
  view so a blank scene shows the full render-camera frustum.
- Auto-keyframe: mutating an already-keyframed object (gizmo drag-end or panel
  commit) replaces/creates a keyframe at the playhead via
  TimelineController.autoKeyIfTracked.
- loadTimeline backfills fps/duration/tracks defensively (legacy save safety).
2026-07-07 15:23:50 -07:00
bflatastic 26aa3d3839 3D scene editor: exact camera framing, timeline UX, outliner + chrome redesign
Camera framing
- Replace the full-canvas Letterbox with CameraFrame: an inset, aspect-correct
  frame (dimmed surround, rule-of-thirds, focal label) shown only in camera
  view and record mode — the free viewport no longer shows mattes
- The frame shows the EXACT render framing: CameraController.setCameraFrameRect
  offsets the viewport projection via camera.setViewOffset; snapshots and the
  timeline recorder clear/reapply the offset so captures stay full-bleed
- recordTimeline now encodes at the render camera's resolution/aspect with the
  grid hidden (was grabbing the raw viewport canvas)
- Frame avoids the bottom UI; expanded-timeline height is measured live

Aspect ratio
- Default is now 16:9 via DEFAULT_CAMERA_ASPECT_RATIO (store, controller,
  scene reset, newScene); removed the model cold-sync that forced Square
- New top-right aspect popover (AspectRatioMenu rewritten on PopoverMenu with
  proportional AspectRatioIcon, now exported from ui-promptbox)

Timeline
- Expanded editor shows every scene object as a track row (no selection
  needed), capped at ~3.5 rows then scrollable (scrollbar hidden so lanes
  stay width-aligned with the ruler)
- Restyled per design: dark lanes, keyframe diamonds + connecting line,
  playhead diamond handle, select-none everywhere
- Scrubbing: drag the ruler to scrub smoothly, following the pointer 1:1;
  times quantize to the timeline's frame grid (30fps) — transport readout
  shows frame timecode (m:ss:ff); keyframe drags quantize the same way
- Fixed playhead/keyframe misalignment (scrollbar shrank lanes vs ruler;
  label indent must be padding, not margin, to keep the w-32 column)
- Delete/Backspace removes the selected keyframe (capture-phase, so the
  engine's delete-scene-object keybind doesn't fire); clicking away
  deselects it
- Motion (easing) popover opens from the curve chip BETWEEN keyframes (new
  timelineEasingKeyframeId store field), centered above the chip in lane
  coordinates, backdrop-blurred, dismissed on outside click; selecting a
  keyframe no longer opens it
- Camera-track playback/scrubbing is now visible in camera view: the
  timeline mirrors the animated ::CAM:: pose onto the viewport camera

Record mode
- Viewport camera fully locked: mouse-look blocked via isMovable(), right-drag
  pan and wheel zoom gated in useFreeCam (build mode unaffected)

Chrome
- Outliner restyled as the "Scene" panel (+ Add via shared openAssetModal,
  two-line rows with friendly kind labels, blue selected card), open by
  default, vertically centered, fixed 45vh height independent of aspect ratio
- Build/Record pill fully rounded (new TabSelector listClassName prop)
- CameraStatusPill moved to the top-right camera cluster, height-matched
  with the aspect popover
- ExitCameraView is a regular labeled button docked right of the timeline bar
2026-07-07 23:15:54 +07:00
Tanish Manku 06315490b7 Scene Builder launch re-scope: drop promptbox, lightbox handoff, editable timeline duration
- Remove SceneBuilderPromptBox (MCP not ready). The animation timeline is the
  sole build-mode bottom UI: collapsed bar by default, expanded editor via
  chevron; an empty timeline is auto-created on init; auto-expands on load when
  the scene has keyframes or an imported THREE animation.
- Editable max timeline duration: TimelineController.setDuration (clamp 1-60s,
  covers 5-30) + click-to-edit DurationLabel in the bar/ruler.
- Capture/Record output → review modal (CompletionModal) that appears FIRST
  with a local preview (offline-safe), then uploads (images auto, videos manual)
  with status; on success opens the app Lightbox on the token (all destinations
  for the kind), on failure shows Retry.
- Adapter: add openMediaLightbox(token, kind); keep uploadMedia; remove
  openImageInEditor/openVideoInEditor (Lightbox owns destination routing). Host
  renders <Lightbox> and resolves cdnUrl via MediaFilesApi.

Compile-verified (tsc -b clean for pagescene + changed webapp files).
2026-07-06 23:54:47 -07:00
Tanish Manku c15d8bd1ec Scene Builder checkpoint: camera view fixes, record/capture + completion modal, save/load + undo persistence
Builds on the prior Scene Builder commit. Compile-verified (tsc -b clean for
pagescene + changed webapp files); see frontend/libs/components/pagescene/CLAUDE.md.

- Camera view: fixed record→build engine-teardown wipe (PreviewEngineCamera /
  cam-view canvas must stay mounted — hide via CSS, never unmount); Esc +
  viewport double-click exit; double-click frustum to enter; robust enter/exit
  pose; frustum pick-proxy hitbox; CameraStatusPill (Viewport vs render cam).
- Build promptbox: dedicated SceneBuilderPromptBox (MCP stub) replaced
  PromptBox3D; buildMode/toggle removed.
- Record mode: Capture (still) + Record (timeline→mp4 via mediabunny),
  RenderOverlay progress, producedArtifact cache, CompletionModal
  (preview/playback + Delete/Upload/Edit) + adapter handoff to 2D/create-video/
  video-editor (lazy, data-URL image handoff; best-effort uploads).
- Persistence: timeline + render-camera transform in scene JSON (save/load +
  unmount cache); ::CAM:: recreated on load after cameras restore.
- Undo: timeline transactional (SaveTimelineAction); camera moves via TransformAction.
2026-07-06 23:26:18 -07:00
Tanish Manku 40ec291395 Scene Builder: mode pill, prompted mode, timeline, camera split, record mode
Adds the Scene Builder feature set to the pagescene 3D editor (+ the shared
ui-promptbox). All UI-level; compile-verified only (tsc -b), not yet runtime
tested — see frontend/libs/components/pagescene/CLAUDE.md for the full spec and
per-slice "unverified" list.

- Build/Record mode pill (SceneModePill, TabSelector) + store `sceneMode`.
- Build sub-modes: Manual (current) vs Prompted. Prompted prompt bar shows a
  textbox + add-animation-timeline + prompt history + Update (Update is a stub
  pending the MCP scene-descriptor backend). Toggle via ButtonIconSelect.
- Animation timeline (in-memory): TimelineController + keyframe model +
  cubic-bezier interpolation ticked in the render loop; collapsed bar +
  expanded multi-track editor + Motion easing popover (draggable bezier);
  Save is an undoable action, Cancel reverts.
- Camera split (single camera): render camera is a selectable wireframe-frustum
  placeholder ("::CAM::") in the scene; look-through camera view via
  double-click / outliner view-from-camera icon; ExitCameraView button.
- Record mode: read-only playback through the locked render camera; build
  chrome hidden; Capture (still) works, Record (video) is a stub.
2026-07-03 05:24:48 -07:00
169 changed files with 11178 additions and 1468 deletions
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO voice_clone_requests\nSET\n token = ?,\n uuid_idempotency_token = ?,\n\n maybe_user_token = ?,\n email_address = ?,\n discord_username = ?,\n\n is_for_private_use = ?,\n is_for_public_use = ?,\n\n is_for_studio = ?,\n is_for_twitch_tts = ?,\n is_for_api_use = ?,\n is_for_music = ?,\n is_for_games = ?,\n is_for_other = ?,\n optional_notes_on_use = ?,\n\n is_own_voice = ?,\n is_third_party_voice = ?,\n optional_notes_on_subject = ?,\n\n has_clean_audio_recordings = ?,\n has_good_microphone = ?,\n\n optional_questions = ?,\n optional_extra_comments = ?,\n\n ip_address_creation = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 22
},
"nullable": []
},
"hash": "4cea10944b8869419ecdfea30c58f49a87c090b9083078043177c33737ecb949"
}
@@ -0,0 +1,36 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n COUNT(distinct token) as record_count,\n NOW() as present_time\nFROM\n(\n SELECT token\n FROM tts_inference_jobs\n WHERE status = \"started\"\n AND created_at > (CURDATE() - INTERVAL 5 MINUTE)\nUNION\n SELECT token\n FROM tts_inference_jobs\n WHERE status IN (\"pending\", \"attempt_failed\")\n) as t\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "record_count",
"type_info": {
"type": "LongLong",
"flags": "NOT_NULL | BINARY",
"char_set": 63,
"max_size": 21
}
},
{
"ordinal": 1,
"name": "present_time",
"type_info": {
"type": "Datetime",
"flags": "NOT_NULL | BINARY",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false
]
},
"hash": "6098826a8ada75442be5b7ac1d0199694e01c5959666f59986cd8b2ed6f51882"
}
@@ -0,0 +1,256 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n token,\n\n maybe_user_token,\n email_address,\n discord_username,\n\n is_for_private_use,\n is_for_public_use,\n\n is_for_studio,\n is_for_twitch_tts,\n is_for_api_use,\n is_for_music,\n is_for_games,\n is_for_other,\n optional_notes_on_use,\n\n is_own_voice,\n is_third_party_voice,\n optional_notes_on_subject,\n\n has_clean_audio_recordings,\n has_good_microphone,\n\n optional_questions,\n optional_extra_comments,\n\n created_at,\n updated_at\nFROM voice_clone_requests\nWHERE\n token = ?\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "maybe_user_token",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "email_address",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 3,
"name": "discord_username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 4,
"name": "is_for_private_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 5,
"name": "is_for_public_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 6,
"name": "is_for_studio",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 7,
"name": "is_for_twitch_tts",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 8,
"name": "is_for_api_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 9,
"name": "is_for_music",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 10,
"name": "is_for_games",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 11,
"name": "is_for_other",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 12,
"name": "optional_notes_on_use",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 13,
"name": "is_own_voice",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 14,
"name": "is_third_party_voice",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 15,
"name": "optional_notes_on_subject",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 16,
"name": "has_clean_audio_recordings",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 17,
"name": "has_good_microphone",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 18,
"name": "optional_questions",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 19,
"name": "optional_extra_comments",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 20,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 21,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
true,
false,
false,
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
false,
true,
false,
false,
true,
true,
false,
false
]
},
"hash": "922a7dbcfa24a2c5901d0453c229b066b3b4eea2c73fa533651bb27365f17be0"
}
@@ -0,0 +1,256 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n token,\n\n maybe_user_token,\n email_address,\n discord_username,\n\n is_for_private_use,\n is_for_public_use,\n\n is_for_studio,\n is_for_twitch_tts,\n is_for_api_use,\n is_for_music,\n is_for_games,\n is_for_other,\n optional_notes_on_use,\n\n is_own_voice,\n is_third_party_voice,\n optional_notes_on_subject,\n\n has_clean_audio_recordings,\n has_good_microphone,\n\n optional_questions,\n optional_extra_comments,\n\n created_at,\n updated_at\nFROM voice_clone_requests\nWHERE\n maybe_user_token = ?\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "maybe_user_token",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "email_address",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 3,
"name": "discord_username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 4,
"name": "is_for_private_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 5,
"name": "is_for_public_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 6,
"name": "is_for_studio",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 7,
"name": "is_for_twitch_tts",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 8,
"name": "is_for_api_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 9,
"name": "is_for_music",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 10,
"name": "is_for_games",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 11,
"name": "is_for_other",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 12,
"name": "optional_notes_on_use",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 13,
"name": "is_own_voice",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 14,
"name": "is_third_party_voice",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 15,
"name": "optional_notes_on_subject",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 16,
"name": "has_clean_audio_recordings",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 17,
"name": "has_good_microphone",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 18,
"name": "optional_questions",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 19,
"name": "optional_extra_comments",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 67108860
}
},
{
"ordinal": 20,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 21,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
true,
false,
false,
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
false,
true,
false,
false,
true,
true,
false,
false
]
},
"hash": "e5ef2bda28221a570da6dcfa73e76cae9910a0b4ee53005a87176ecf1be73dd7"
}
Generated
-15
View File
@@ -3621,7 +3621,6 @@ dependencies = [
"env_logger 0.11.9",
"log",
"storyteller_root",
"trace_id",
"workspace-build-acceleration",
]
@@ -6767,7 +6766,6 @@ version = "0.0.1"
dependencies = [
"env_logger 0.11.9",
"log",
"trace_id",
"workspace-build-acceleration",
]
@@ -8288,7 +8286,6 @@ dependencies = [
"rootly_client",
"sha2 0.11.0-rc.5",
"tokio",
"trace_id",
"workspace-build-acceleration",
]
@@ -11681,7 +11678,6 @@ dependencies = [
"tokens",
"tokio",
"toml 1.0.7+spec-1.1.0",
"trace_id",
"tts_common",
"url",
"url_config",
@@ -13320,17 +13316,6 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
[[package]]
name = "trace_id"
version = "0.0.1"
dependencies = [
"rand 0.8.5",
"serde",
"serde_derive",
"tokio",
"workspace-build-acceleration",
]
[[package]]
name = "tracing"
version = "0.1.41"
-2
View File
@@ -130,7 +130,6 @@ members = [
"crates/schema/public/enums",
"crates/schema/public/reusable_types", # TODO(bt, 2022-12-21): Deprecate
"crates/schema/public/tokens",
"crates/schema/trace_id",
# Services: API servers, jobs, workers, etc.
"crates/service/job/gmicloud_job",
@@ -294,7 +293,6 @@ subprocess_common = { path = "crates/lib/deprecated/subprocess_common" }
test_data = { path = "crates/testing/test_data" }
test_utils = { path = "crates/testing/test_utils" }
tokens = { path = "crates/schema/public/tokens" }
trace_id = { path = "crates/schema/trace_id" }
url_utils = { path = "crates/lib/url_utils" }
users = { path = "crates/lib/users" }
uuid_utils = { path = "crates/lib/uuid_utils" }
-1
View File
@@ -25,7 +25,6 @@ workspace-build-acceleration = { version = "0.1", path = "../../workspace-build-
# External
dotenv.workspace = true
env_logger = "0.11.3"
trace_id.workspace = true
log = "0.4.14"
[dev-dependencies]
+1 -32
View File
@@ -29,37 +29,6 @@ pub fn init_env_logger(default_if_absent: Option<&str>) {
env::set_var(ENV_RUST_LOG, default_log_level);
}
// Custom format so that log lines emitted while serving an HTTP request
// automatically carry the request's trace id (see the `trace_id` crate and
// the `TraceIdMiddleware` in storyteller-web). Outside of a request scope
// the trace segment is simply omitted.
env_logger::Builder::from_env(env_logger::Env::default())
.format(|buf, record| {
use std::io::Write;
let timestamp = buf.timestamp();
let level_style = buf.default_level_style(record.level());
match trace_id::current_trace_id() {
Some(trace) => writeln!(
buf,
"[{} {level_style}{}{level_style:#} {} {}] {}",
timestamp,
record.level(),
record.target(),
trace,
record.args(),
),
None => writeln!(
buf,
"[{} {level_style}{}{level_style:#} {}] {}",
timestamp,
record.level(),
record.target(),
record.args(),
),
}
})
.init();
env_logger::init();
}
-1
View File
@@ -25,7 +25,6 @@ workspace-build-acceleration = { workspace = true }
# External
env_logger = "0.11.3"
trace_id.workspace = true
log = { workspace = true }
[dev-dependencies]
+1 -32
View File
@@ -68,37 +68,6 @@ pub fn init_env_logger(default_if_absent: Option<&str>) {
env::set_var(ENV_RUST_LOG, default_log_level);
}
// Custom format so that log lines emitted while serving an HTTP request
// automatically carry the request's trace id (see the `trace_id` crate and
// the `TraceIdMiddleware` in storyteller-web). Outside of a request scope
// the trace segment is simply omitted.
env_logger::Builder::from_env(env_logger::Env::default())
.format(|buf, record| {
use std::io::Write;
let timestamp = buf.timestamp();
let level_style = buf.default_level_style(record.level());
match trace_id::current_trace_id() {
Some(trace) => writeln!(
buf,
"[{} {level_style}{}{level_style:#} {} {}] {}",
timestamp,
record.level(),
record.target(),
trace,
record.args(),
),
None => writeln!(
buf,
"[{} {level_style}{}{level_style:#} {}] {}",
timestamp,
record.level(),
record.target(),
record.args(),
),
}
})
.init();
env_logger::init();
}
-1
View File
@@ -24,7 +24,6 @@ chrono.workspace = true
sha2.workspace = true
log.workspace = true
tokio.workspace = true
trace_id.workspace = true
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
@@ -49,9 +49,6 @@ pub struct NotificationDetails {
/// Third-party identifier associated with the event, if any.
pub(crate) third_party_id: Option<String>,
/// The request-scoped trace id associated with the event, if any.
pub(crate) trace_id: Option<String>,
/// IP address of the request that triggered this notification, if any.
pub(crate) request_ip_address: Option<String>,
@@ -192,10 +189,6 @@ impl NotificationDetails {
parts.push(format!("Request IP: {}", ip));
}
if let Some(trace_id) = &self.trace_id {
parts.push(format!("Trace ID: {}", trace_id));
}
if parts.is_empty() { None } else { Some(parts.join("\n")) }
}
@@ -31,10 +31,6 @@ pub struct NotificationDetailsBuilder {
pub(crate) avt_cookie_token: Option<String>,
pub(crate) session_token: Option<String>,
pub(crate) session_user_token: Option<String>,
/// The request-scoped trace id, if any. Auto-filled from the tokio
/// task-local at `build()` time when not explicitly set.
pub(crate) trace_id: Option<String>,
}
impl NotificationDetailsBuilder {
@@ -61,7 +57,6 @@ impl NotificationDetailsBuilder {
avt_cookie_token: None,
session_token: None,
session_user_token: None,
trace_id: None,
}
}
@@ -92,7 +87,6 @@ impl NotificationDetailsBuilder {
avt_cookie_token: None,
session_token: None,
session_user_token: None,
trace_id: None,
}
}
@@ -173,11 +167,6 @@ impl NotificationDetailsBuilder {
self
}
pub fn set_trace_id(mut self, trace_id: Option<String>) -> Self {
self.trace_id = trace_id;
self
}
pub fn set_session_user_token(mut self, session_user_token: Option<String>) -> Self {
self.session_user_token = session_user_token;
self
@@ -210,11 +199,6 @@ impl NotificationDetailsBuilder {
avt_cookie_token: self.avt_cookie_token,
session_token: self.session_token,
session_user_token: self.session_user_token,
// Auto-fill from the request's task-local so any page fired while
// serving a request is correlated even if the caller never set it.
trace_id: self.trace_id.or_else(|| {
trace_id::current_trace_id().map(|t| t.to_string())
}),
}
}
@@ -47,6 +47,7 @@ pub mod user_spend_events;
pub mod user_spend_summaries;
pub mod users;
pub mod vocoder;
pub mod voice_clone_requests;
pub mod voice_conversion;
pub mod voice_designer;
pub mod wallet_ledger_entries;
@@ -0,0 +1,53 @@
use chrono::NaiveDateTime;
use sqlx::MySqlPool;
use errors::AnyhowResult;
use crate::helpers::numeric_converters::try_i64_to_u64_or_min;
#[derive(Clone)]
pub struct TtsQueueLengthResult {
pub record_count: u64,
pub present_time: NaiveDateTime,
}
struct QueryResultInternal {
record_count: i64,
present_time: NaiveDateTime,
}
pub async fn get_pending_tts_inference_job_count(
pool: &MySqlPool,
) -> AnyhowResult<TtsQueueLengthResult> {
// NB (1): We query as a union since "started" jobs can get picked off mid-run and go stale
// forever. We currently have no automated means of collecting those jobs.
// NB (2): We query the server timestamp so we can cache the results.
// The frontend can then monotonically adjust the count based on timestamp by ignoring
// past times.
let result : QueryResultInternal = sqlx::query_as!(
QueryResultInternal,
r#"
SELECT
COUNT(distinct token) as record_count,
NOW() as present_time
FROM
(
SELECT token
FROM tts_inference_jobs
WHERE status = "started"
AND created_at > (CURDATE() - INTERVAL 5 MINUTE)
UNION
SELECT token
FROM tts_inference_jobs
WHERE status IN ("pending", "attempt_failed")
) as t
"#,
)
.fetch_one(pool)
.await?;
Ok(TtsQueueLengthResult {
record_count: try_i64_to_u64_or_min(result.record_count),
present_time: result.present_time,
})
}
@@ -1,4 +1,5 @@
pub mod _keys;
pub mod get_pending_tts_inference_job_count;
pub mod get_pending_tts_inference_job_detailed_stats;
pub mod get_tts_inference_job_status;
pub mod insert_tts_inference_job;
@@ -0,0 +1,237 @@
use anyhow::anyhow;
use chrono::Utc;
use sqlx::MySqlPool;
use errors::AnyhowResult;
use crate::helpers::boolean_converters::i8_to_bool;
/// Lookup by voice clone request token
pub async fn get_voice_clone_request_by_token(
voice_clone_request_token: &str,
pool: &MySqlPool,
) -> AnyhowResult<Option<VoiceCloneRequest>> {
internal_voice_clone_request_by_token(voice_clone_request_token, pool).await
}
/// Lookup by user
pub async fn get_voice_clone_request_by_user_token(
user_token: &str,
pool: &MySqlPool,
) -> AnyhowResult<Option<VoiceCloneRequest>> {
internal_voice_clone_request_by_user_token(user_token, pool).await
}
/// Used for both *BY TOKEN* and for *BY USER* lookups.
/// The user field should be optional for one and not for the other, but code reuse is more
/// important and this is a quick / hasty feature.
#[derive(Debug)]
pub struct VoiceCloneRequest {
pub token: String,
// Contact
pub maybe_user_token: Option<String>,
pub email_address: String,
pub discord_username: String,
// Visibility
pub is_for_private_use: bool,
pub is_for_public_use: bool,
// Use
pub is_for_studio: bool,
pub is_for_twitch_tts: bool,
pub is_for_api_use: bool,
pub is_for_music: bool,
pub is_for_games: bool,
pub is_for_other: bool,
pub optional_notes_on_use: Option<String>,
// Subject/Ownership
pub is_own_voice: bool,
pub is_third_party_voice: bool,
pub optional_notes_on_subject: Option<String>,
// Equipment
pub has_clean_audio_recordings: bool,
pub has_good_microphone: bool,
// Comments
pub optional_questions: Option<String>,
pub optional_extra_comments: Option<String>,
pub created_at: chrono::DateTime<Utc>,
pub updated_at: chrono::DateTime<Utc>,
}
async fn internal_voice_clone_request_by_token(
voice_clone_request_token: &str,
pool: &MySqlPool,
) -> AnyhowResult<Option<VoiceCloneRequest>> {
let maybe_record = sqlx::query_as!(
VoiceCloneRequestInternal,
r#"
SELECT
token,
maybe_user_token,
email_address,
discord_username,
is_for_private_use,
is_for_public_use,
is_for_studio,
is_for_twitch_tts,
is_for_api_use,
is_for_music,
is_for_games,
is_for_other,
optional_notes_on_use,
is_own_voice,
is_third_party_voice,
optional_notes_on_subject,
has_clean_audio_recordings,
has_good_microphone,
optional_questions,
optional_extra_comments,
created_at,
updated_at
FROM voice_clone_requests
WHERE
token = ?
"#,
voice_clone_request_token,
)
.fetch_one(pool)
.await;
handle_result(maybe_record)
}
async fn internal_voice_clone_request_by_user_token(
user_token: &str,
pool: &MySqlPool,
) -> AnyhowResult<Option<VoiceCloneRequest>> {
let maybe_record = sqlx::query_as!(
VoiceCloneRequestInternal,
r#"
SELECT
token,
maybe_user_token,
email_address,
discord_username,
is_for_private_use,
is_for_public_use,
is_for_studio,
is_for_twitch_tts,
is_for_api_use,
is_for_music,
is_for_games,
is_for_other,
optional_notes_on_use,
is_own_voice,
is_third_party_voice,
optional_notes_on_subject,
has_clean_audio_recordings,
has_good_microphone,
optional_questions,
optional_extra_comments,
created_at,
updated_at
FROM voice_clone_requests
WHERE
maybe_user_token = ?
"#,
user_token,
)
.fetch_one(pool)
.await;
handle_result(maybe_record)
}
fn handle_result(
maybe_record: Result<VoiceCloneRequestInternal, sqlx::Error>,
) -> AnyhowResult<Option<VoiceCloneRequest>> {
let record : VoiceCloneRequestInternal = match maybe_record {
Ok(record) => record,
Err(sqlx::Error::RowNotFound) => return Ok(None),
Err(ref err) => return Err(anyhow!("database query error: {:?}", err)),
};
Ok(Some(VoiceCloneRequest {
token: record.token,
maybe_user_token: record.maybe_user_token,
email_address: record.email_address,
discord_username: record.discord_username,
is_for_private_use: i8_to_bool(record.is_for_private_use),
is_for_public_use: i8_to_bool(record.is_for_public_use),
is_for_studio: i8_to_bool(record.is_for_studio),
is_for_twitch_tts: i8_to_bool(record.is_for_twitch_tts),
is_for_api_use: i8_to_bool(record.is_for_api_use),
is_for_music: i8_to_bool(record.is_for_music),
is_for_games: i8_to_bool(record.is_for_games),
is_for_other: i8_to_bool(record.is_for_other),
optional_notes_on_use: record.optional_notes_on_use,
is_own_voice: i8_to_bool(record.is_own_voice),
is_third_party_voice: i8_to_bool(record.is_third_party_voice),
optional_notes_on_subject: record.optional_notes_on_subject,
has_clean_audio_recordings: i8_to_bool(record.has_clean_audio_recordings),
has_good_microphone: i8_to_bool(record.has_good_microphone),
optional_questions: record.optional_questions,
optional_extra_comments: record.optional_extra_comments,
created_at: record.created_at,
updated_at: record.updated_at,
}))
}
#[derive(Debug)]
struct VoiceCloneRequestInternal {
pub token: String,
// Contact
pub maybe_user_token: Option<String>,
pub email_address: String,
pub discord_username: String,
// Visibility
pub is_for_private_use: i8,
pub is_for_public_use: i8,
// Use
pub is_for_studio: i8,
pub is_for_twitch_tts: i8,
pub is_for_api_use: i8,
pub is_for_music: i8,
pub is_for_games: i8,
pub is_for_other: i8,
pub optional_notes_on_use: Option<String>,
// Subject/Ownership
pub is_own_voice: i8,
pub is_third_party_voice: i8,
pub optional_notes_on_subject: Option<String>,
// Equipment
pub has_clean_audio_recordings: i8,
pub has_good_microphone: i8,
// Comments
pub optional_questions: Option<String>,
pub optional_extra_comments: Option<String>,
pub created_at: chrono::DateTime<Utc>,
pub updated_at: chrono::DateTime<Utc>,
}
@@ -0,0 +1,125 @@
use anyhow::anyhow;
use sqlx::MySqlPool;
use errors::AnyhowResult;
use tokens::tokens::voice_clone_requests::VoiceCloneRequestToken;
pub struct InsertVoiceCloneRequestBuilder {
pub uuid_idempotency_token: String,
// Contact
pub maybe_user_token: Option<String>,
pub email_address: String,
pub discord_username: String,
// Visibility
pub is_for_private_use: bool,
pub is_for_public_use: bool,
// Use
pub is_for_studio: bool,
pub is_for_twitch_tts: bool,
pub is_for_api_use: bool,
pub is_for_music: bool,
pub is_for_games: bool,
pub is_for_other: bool,
pub optional_notes_on_use: Option<String>,
// Subject/Ownership
pub is_own_voice: bool,
pub is_third_party_voice: bool,
pub optional_notes_on_subject: Option<String>,
// Equipment
pub has_clean_audio_recordings: bool,
pub has_good_microphone: bool,
// Comments
pub optional_questions: Option<String>,
pub optional_extra_comments: Option<String>,
pub ip_address_creation: String,
}
impl InsertVoiceCloneRequestBuilder {
/// Returns the newly generated token.
pub async fn insert(&self, mysql_pool: &MySqlPool) -> AnyhowResult<String> {
let token = VoiceCloneRequestToken::generate().to_string();
let query = sqlx::query!(
r#"
INSERT INTO voice_clone_requests
SET
token = ?,
uuid_idempotency_token = ?,
maybe_user_token = ?,
email_address = ?,
discord_username = ?,
is_for_private_use = ?,
is_for_public_use = ?,
is_for_studio = ?,
is_for_twitch_tts = ?,
is_for_api_use = ?,
is_for_music = ?,
is_for_games = ?,
is_for_other = ?,
optional_notes_on_use = ?,
is_own_voice = ?,
is_third_party_voice = ?,
optional_notes_on_subject = ?,
has_clean_audio_recordings = ?,
has_good_microphone = ?,
optional_questions = ?,
optional_extra_comments = ?,
ip_address_creation = ?
"#,
&token,
&self.uuid_idempotency_token,
&self.maybe_user_token,
&self.email_address,
&self.discord_username,
&self.is_for_private_use,
&self.is_for_public_use,
&self.is_for_studio,
&self.is_for_twitch_tts,
&self.is_for_api_use,
&self.is_for_music,
&self.is_for_games,
&self.is_for_other,
&self.optional_notes_on_use,
&self.is_own_voice,
&self.is_third_party_voice,
&self.optional_notes_on_subject,
&self.has_clean_audio_recordings,
&self.has_good_microphone,
&self.optional_questions,
&self.optional_extra_comments,
&self.ip_address_creation,
);
let query_result = query.execute(mysql_pool).await;
let _record_id = match query_result {
Ok(res) => res.last_insert_id(),
Err(err) => return Err(anyhow!("Error creating Voice Clone Request: {:?}", err)),
};
Ok(token)
}
}
@@ -0,0 +1,2 @@
pub mod get_voice_clone_request;
pub mod insert_voice_clone_requests_builder;
@@ -18,16 +18,7 @@ pub const MAX_LENGTH: usize = 24;
#[derive(Clone, Copy, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "snake_case")]
pub enum DebugLogType {
// ========== First party ==========
/// A user request, typically the Json<T> actix_web handlers read.
HttpRequest,
/// A backend/pipeline failure while serving a request.
BackendFailure,
// ========== Third party ==========
FalRequest,
KinoviRequest,
GrokApiRequest,
@@ -43,7 +34,6 @@ impl DebugLogType {
pub fn to_str(&self) -> &'static str {
match self {
Self::HttpRequest => "http_request",
Self::BackendFailure => "backend_failure",
Self::FalRequest => "fal_request",
Self::KinoviRequest => "kinovi_request",
Self::GrokApiRequest => "grok_api_request",
@@ -55,7 +45,6 @@ impl DebugLogType {
pub fn from_str(value: &str) -> Result<Self, EnumError> {
match value {
"http_request" => Ok(Self::HttpRequest),
"backend_failure" => Ok(Self::BackendFailure),
"fal_request" => Ok(Self::FalRequest),
"kinovi_request" => Ok(Self::KinoviRequest),
"grok_api_request" => Ok(Self::GrokApiRequest),
@@ -68,7 +57,6 @@ impl DebugLogType {
pub fn all_variants() -> BTreeSet<Self> {
BTreeSet::from([
Self::HttpRequest,
Self::BackendFailure,
Self::FalRequest,
Self::KinoviRequest,
Self::GrokApiRequest,
@@ -91,7 +79,6 @@ mod tests {
#[test]
fn test_serialization() {
assert_serialization(DebugLogType::HttpRequest, "http_request");
assert_serialization(DebugLogType::BackendFailure, "backend_failure");
assert_serialization(DebugLogType::FalRequest, "fal_request");
assert_serialization(DebugLogType::KinoviRequest, "kinovi_request");
assert_serialization(DebugLogType::GrokApiRequest, "grok_api_request");
@@ -102,7 +89,6 @@ mod tests {
#[test]
fn to_str() {
assert_eq!(DebugLogType::HttpRequest.to_str(), "http_request");
assert_eq!(DebugLogType::BackendFailure.to_str(), "backend_failure");
assert_eq!(DebugLogType::FalRequest.to_str(), "fal_request");
assert_eq!(DebugLogType::KinoviRequest.to_str(), "kinovi_request");
assert_eq!(DebugLogType::GrokApiRequest.to_str(), "grok_api_request");
@@ -113,7 +99,6 @@ mod tests {
#[test]
fn from_str() {
assert_eq!(DebugLogType::from_str("http_request").unwrap(), DebugLogType::HttpRequest);
assert_eq!(DebugLogType::from_str("backend_failure").unwrap(), DebugLogType::BackendFailure);
assert_eq!(DebugLogType::from_str("fal_request").unwrap(), DebugLogType::FalRequest);
assert_eq!(DebugLogType::from_str("kinovi_request").unwrap(), DebugLogType::KinoviRequest);
assert_eq!(DebugLogType::from_str("grok_api_request").unwrap(), DebugLogType::GrokApiRequest);
@@ -135,9 +120,8 @@ mod tests {
#[test]
fn all_variants() {
let mut variants = DebugLogType::all_variants();
assert_eq!(variants.len(), 7);
assert_eq!(variants.len(), 6);
assert_eq!(variants.pop_first(), Some(DebugLogType::HttpRequest));
assert_eq!(variants.pop_first(), Some(DebugLogType::BackendFailure));
assert_eq!(variants.pop_first(), Some(DebugLogType::FalRequest));
assert_eq!(variants.pop_first(), Some(DebugLogType::KinoviRequest));
assert_eq!(variants.pop_first(), Some(DebugLogType::GrokApiRequest));
-25
View File
@@ -1,25 +0,0 @@
[package]
name = "trace_id"
edition = "2018"
version = "0.0.1"
authors = [
"Brandon Thomas <bt@brand.io>",
"Brandon Thomas <echelon@gmail.com>",
]
publish = false
[package.metadata.cargo-udeps.ignore]
normal = ["workspace-build-acceleration"]
[lib]
name = "trace_id"
path = "src/lib.rs"
[dependencies]
# Workspace hack (faster compile times)
workspace-build-acceleration.workspace = true
rand = { version = "0.8.5", features = ["std_rng"] }
serde.workspace = true
serde_derive.workspace = true
tokio.workspace = true
-156
View File
@@ -1,156 +0,0 @@
//! trace_id
//!
//! Request-scoped trace identifiers for observability.
//!
//! A `TraceId` is generated for every inbound HTTP request (see the
//! `trace_id_middleware` in `storyteller-web`), stored in a tokio task-local,
//! and emitted with every log line and pager notification produced while
//! serving that request.
use std::fmt;
use serde_derive::{Deserialize, Serialize};
/// The string prefix for trace ids.
pub const TRACE_ID_PREFIX: &str = "trace_";
/// Number of Crockford base32 characters after the prefix.
/// 26 characters × 5 bits = 130 bits of entropy (≥ the 128-bit industry
/// standard for trace ids). Total length: 6 + 26 = 32 characters.
pub const TRACE_ID_ENTROPY_CHARS: usize = 26;
/// Total serialized length of a trace id ("trace_" + entropy).
pub const TRACE_ID_TOTAL_LENGTH: usize = 32;
/// Crockford base32 alphabet, lowercase (no i, l, o, u).
const CROCKFORD_LOWERCASE_CHARSET: &[u8] = b"0123456789abcdefghjkmnpqrstvwxyz";
/// A unique-per-request trace identifier, eg. `trace_1h3x9k2m4p7q8r5t6v0w2y3z4a`.
#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
pub struct TraceId(String);
impl TraceId {
/// Generate a new random trace id. Infallible.
#[inline]
pub fn generate() -> Self {
use rand::Rng;
let mut rng = rand::thread_rng();
let entropy: String = (0..TRACE_ID_ENTROPY_CHARS)
.map(|_| {
let idx = rng.gen_range(0..CROCKFORD_LOWERCASE_CHARSET.len());
CROCKFORD_LOWERCASE_CHARSET[idx] as char
})
.collect();
TraceId(format!("{}{}", TRACE_ID_PREFIX, entropy))
}
#[inline]
pub fn as_str(&self) -> &str {
&self.0
}
}
impl fmt::Display for TraceId {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
impl fmt::Debug for TraceId {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
tokio::task_local! {
/// The trace id for the currently-executing request, if any.
///
/// Set by the HTTP middleware via [`TRACE_ID_TASK_LOCAL.scope(...)`], and
/// readable from anywhere within the request's task — including the
/// logger's format closure and the pager — via [`current_trace_id`].
pub static TRACE_ID_TASK_LOCAL: TraceId;
}
/// The trace id of the currently-executing request, if one is in scope.
///
/// Returns `None` outside of a request context (startup, background threads,
/// detached `tokio::spawn` tasks that weren't explicitly re-scoped).
#[inline]
pub fn current_trace_id() -> Option<TraceId> {
TRACE_ID_TASK_LOCAL.try_with(|trace_id| trace_id.clone()).ok()
}
#[cfg(test)]
mod tests {
use super::*;
mod generation {
use super::*;
#[test]
fn total_length_is_32() {
assert_eq!(TraceId::generate().as_str().len(), TRACE_ID_TOTAL_LENGTH);
}
#[test]
fn has_prefix() {
assert!(TraceId::generate().as_str().starts_with(TRACE_ID_PREFIX));
}
#[test]
fn entropy_uses_crockford_lowercase_only() {
let trace_id = TraceId::generate();
let entropy = &trace_id.as_str()[TRACE_ID_PREFIX.len()..];
assert_eq!(entropy.len(), TRACE_ID_ENTROPY_CHARS);
for c in entropy.chars() {
assert!(
CROCKFORD_LOWERCASE_CHARSET.contains(&(c as u8)),
"unexpected character: {}",
c
);
}
}
#[test]
fn ids_are_unique() {
let a = TraceId::generate();
let b = TraceId::generate();
assert_ne!(a, b);
}
}
mod task_local {
use super::*;
#[tokio::test]
async fn absent_outside_scope() {
assert!(current_trace_id().is_none());
}
#[tokio::test]
async fn present_inside_scope() {
let trace_id = TraceId::generate();
let expected = trace_id.clone();
TRACE_ID_TASK_LOCAL
.scope(trace_id, async move {
assert_eq!(current_trace_id(), Some(expected));
})
.await;
}
#[tokio::test]
async fn survives_await_points() {
let trace_id = TraceId::generate();
let expected = trace_id.clone();
TRACE_ID_TASK_LOCAL
.scope(trace_id, async move {
tokio::task::yield_now().await;
assert_eq!(current_trace_id(), Some(expected));
})
.await;
}
}
}
@@ -89,7 +89,6 @@ seedance2pro_client.workspace = true
server_environment = { path = "../../../lib/server_environment" }
thumbnail_generator = { path = "../../../api_clients/thumbnail_generator" }
tokens.workspace = true
trace_id.workspace = true
tts_common = { path = "../../../lib/deprecated/tts_common" }
url_config = { path = "../../../lib/url_config" }
url_utils = { path = "../../../lib/url_utils" }
@@ -217,6 +217,8 @@ use billing_component::stripe::http_endpoints::checkout::create::stripe_create_c
use crate::http_server::endpoints::billing_fakeyou::list_active_user_subscriptions_handler::*;
use crate::http_server::endpoints::service::status_alert_handler::*;
use crate::http_server::endpoints::stats::get_unified_queue_stats_handler::*;
use crate::http_server::endpoints::studio_gen2::enqueue_studio_gen2_handler::*;
use crate::http_server::endpoints::studio_gen2::enqueue_studio_gen2_handler::EnqueueStudioGen2Request;
use crate::http_server::endpoints::tags::list_tags_for_entity_handler::*;
use crate::http_server::endpoints::tags::set_tags_for_entity_handler::*;
use crate::http_server::endpoints::tts::enqueue_infer_tts_handler::enqueue_infer_tts_handler::*;
@@ -239,6 +241,9 @@ use crate::http_server::endpoints::users::login_handler::LoginErrorResponse;
use crate::http_server::endpoints::users::logout_handler::*;
use crate::http_server::endpoints::users::session_info_handler::*;
use crate::http_server::endpoints::users::session_token_info_handler::*;
use crate::http_server::endpoints::voice_conversion::enqueue_voice_conversion_inference_handler::*;
use crate::http_server::endpoints::voice_designer::inference::enqueue_tts_request::*;
use crate::http_server::endpoints::voice_designer::voice_datasets::list_datasets_by_user::*;
use crate::http_server::endpoints::weights::delete::delete_weight_handler::*;
use crate::http_server::endpoints::weights::get::get_weight_handler::*;
use crate::http_server::endpoints::weights::list::list_available_weights_handler::*;
@@ -383,6 +388,7 @@ use crate::http_server::endpoints::video_info::video_info_upload_handler::*;
use crate::http_server::endpoints::video_info::video_info_notes_handler::*;
use crate::http_server::endpoints::web_referrals::log_web_referral_handler::*;
use crate::http_server::endpoints::image_studio::update_gpt_image_job_status_handler::*;
use crate::http_server::endpoints::voice_conversion::enqueue_seed_vc_inference_handler::*;
use crate::http_server::endpoints::credits::get_session_credits_handler::*;
use crate::http_server::endpoints::subscriptions::get_session_subscription_handler::*;
use crate::http_server::endpoints::analytics::log_app_active_user_handler::*;
@@ -481,6 +487,7 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
crate::http_server::endpoints::prompts::get_prompt_handler::get_prompt_handler,
crate::http_server::endpoints::service::status_alert_handler::status_alert_handler,
crate::http_server::endpoints::stats::get_unified_queue_stats_handler::get_unified_queue_stats_handler,
crate::http_server::endpoints::studio_gen2::enqueue_studio_gen2_handler::enqueue_studio_gen2_handler,
crate::http_server::endpoints::tags::list_tags_for_entity_handler::list_tags_for_entity_handler,
crate::http_server::endpoints::tags::set_tags_for_entity_handler::set_tags_for_entity_handler,
crate::http_server::endpoints::tts::enqueue_infer_tts_handler::enqueue_infer_tts_handler::enqueue_infer_tts_handler,
@@ -502,6 +509,9 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
crate::http_server::endpoints::users::logout_handler::logout_handler,
crate::http_server::endpoints::users::session_info_handler::session_info_handler,
crate::http_server::endpoints::users::session_token_info_handler::session_token_info_handler,
crate::http_server::endpoints::voice_conversion::enqueue_voice_conversion_inference_handler::enqueue_voice_conversion_inference_handler,
crate::http_server::endpoints::voice_designer::inference::enqueue_tts_request::enqueue_tts_request,
crate::http_server::endpoints::voice_designer::voice_datasets::list_datasets_by_user::list_datasets_by_user_handler,
crate::http_server::endpoints::weights::delete::delete_weight_handler::delete_weight_handler,
crate::http_server::endpoints::weights::get::get_weight_handler::get_weight_handler,
crate::http_server::endpoints::weights::list::list_available_weights_handler::list_available_weights_handler,
@@ -557,7 +567,9 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
crate::http_server::endpoints::analytics::log_app_active_user_handler::log_app_active_user_handler,
crate::http_server::endpoints::analytics::log_app_active_user_json_handler::log_app_active_user_json_handler,
// Voice Conversion
crate::http_server::endpoints::voice_conversion::enqueue_seed_vc_inference_handler::enqueue_infer_seed_vc_handler,
// TTS
crate::http_server::endpoints::tts::enqueue_infer_f5_tts_handler::enqueue_infer_f5_tts_handler::enqueue_infer_f5_tts_handler,
// Characters
crate::http_server::endpoints::characters::list_characters_handler::list_characters_handler,
crate::http_server::endpoints::characters::create_character_handler::create_character_handler,
@@ -919,10 +931,17 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
EnqueueLivePortraitCropDimensions,
EnqueueLivePortraitWorkflowRequest,
EnqueueLivePortraitWorkflowSuccessResponse,
EnqueueStudioGen2Request,
EnqueueStudioGen2Response,
EnqueueTTSRequest,
EnqueueTTSRequestSuccessResponse,
EnqueueVideoStyleTransferRequest,
EnqueueVideoStyleTransferSuccessResponse,
EnqueueVoiceConversionInferenceRequest,
EnqueueVoiceConversionInferenceSuccessResponse,
FeaturedMediaFile,
FeaturedModelWeightForList,
FundamentalFrequencyMethod,
GenerateFlux1DevTextToImageAspectRatio,
GenerateFlux1DevTextToImageNumImages,
GenerateFlux1DevTextToImageRequest,
@@ -1002,6 +1021,9 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
ListCommentsError,
ListCommentsPathInfo,
ListCommentsSuccessResponse,
ListDatasetsByUserError,
ListDatasetsByUserPathInfo,
ListDatasetsByUserSuccessResponse,
ListFeaturedMediaFilesQueryParams,
ListFeaturedMediaFilesSuccessResponse,
ListFeaturedWeightsError,
@@ -1172,6 +1194,7 @@ use crate::http_server::endpoints::media_files::list::list_batch_generated_redux
WeightsData,
WriteEngineAssetMediaSuccessResponse,
WriteSceneFileMediaSuccessResponse,
ZsDatasetRecord,
// Cost Estimate types
EstimateImageCostRequest,
@@ -0,0 +1,18 @@
pub const DEFAULT_INVESTOR_REDIRECT : &str = "https://fakeyou.com";
pub fn redirect_is_allowed(redirect_url: &str) -> bool {
match redirect_url {
"http://jungle.horse" => true,
"http://jungle.horse/" => true,
"https://create.storyteller.io" => true,
"https://create.storyteller.io/" => true,
"https://fakeyou.com" => true,
"https://fakeyou.com/clone" => true,
"https://jungle.horse" => true,
"https://jungle.horse/" => true,
"https://storyteller.io" => true,
"https://storyteller.io/" => true,
_ => false,
}
}
@@ -0,0 +1,10 @@
//! A temporary cookie for investor demo
use actix_web::HttpRequest;
pub const STORYTELLER_DEMO_COOKIE_NAME : &str = "storyteller_demo";
pub fn request_has_demo_cookie(http_request: &HttpRequest) -> bool {
// NB: Payload of the cookie does not matter.
http_request.cookie(STORYTELLER_DEMO_COOKIE_NAME).is_some()
}
@@ -0,0 +1,70 @@
use std::sync::Arc;
use actix_http::header;
use actix_web::cookie::time::OffsetDateTime;
use actix_web::cookie::Cookie;
use actix_web::http::StatusCode;
use actix_web::web::Query;
use actix_web::{web, HttpRequest, HttpResponse, Responder};
use actix_helpers::extractors::get_request_host::get_request_host;
use crate::http_server::deprecated_endpoints::investor_demo::default_redirect::{redirect_is_allowed, DEFAULT_INVESTOR_REDIRECT};
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::STORYTELLER_DEMO_COOKIE_NAME;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct QueryFields {
redirect_to: Option<String>,
}
pub async fn disable_demo_mode_handler(
http_request: HttpRequest,
query: Query<QueryFields>,
server_state: web::Data<Arc<ServerState>>
) -> impl Responder
{
let unsafe_redirect = query.redirect_to
.as_deref()
.map(|r| r.to_string())
.unwrap_or(DEFAULT_INVESTOR_REDIRECT.to_string());
let redirect_allowed = redirect_is_allowed(&unsafe_redirect);
let safe_redirect_url = if redirect_allowed {
unsafe_redirect
} else {
DEFAULT_INVESTOR_REDIRECT.to_string()
};
let maybe_host = get_request_host(&http_request);
let maybe_cookie_domain = match maybe_host {
Some("jungle.horse") | Some("api.jungle.horse") => Some(".jungle.horse"),
Some("fakeyou.com") | Some("api.fakeyou.com") => Some(".fakeyou.com"),
Some("storyteller.io") | Some("api.storyteller.io") => Some(".storyteller.io"),
_ => None,
};
// Kill the cookie
let mut cookie_builder = Cookie::build(STORYTELLER_DEMO_COOKIE_NAME, "")
.secure(server_state.env_config.cookie_secure) // HTTPS-only
.expires(OffsetDateTime::UNIX_EPOCH)
.http_only(false) // This is meant to be exposed to Javascript!
// NB: Since we're setting this from direct HTTP access rather than XHR, the browser
// implicitly sets the path used to access the cookie, ie. '/demo_mode', which is not what
// we want!
.path("/")
.permanent();
if let Some(cookie_domain) = maybe_cookie_domain {
cookie_builder = cookie_builder.domain(cookie_domain);
}
let mut cookie = cookie_builder.finish();
cookie.make_removal(); // Remove the cookie
HttpResponse::build(StatusCode::FOUND)
.append_header((header::LOCATION, safe_redirect_url.to_string()))
.cookie(cookie)
.finish()
}
@@ -0,0 +1,67 @@
use std::sync::Arc;
use actix_http::header;
use actix_web::cookie::Cookie;
use actix_web::http::StatusCode;
use actix_web::web::Query;
use actix_web::{web, HttpRequest, HttpResponse, Responder};
use actix_helpers::extractors::get_request_host::get_request_host;
use crate::http_server::deprecated_endpoints::investor_demo::default_redirect::{redirect_is_allowed, DEFAULT_INVESTOR_REDIRECT};
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::STORYTELLER_DEMO_COOKIE_NAME;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct QueryFields {
redirect_to: Option<String>,
}
pub async fn enable_demo_mode_handler(
http_request: HttpRequest,
query: Query<QueryFields>,
server_state: web::Data<Arc<ServerState>>
) -> impl Responder
{
let unsafe_redirect = query.redirect_to
.as_deref()
.map(|r| r.to_string())
.unwrap_or(DEFAULT_INVESTOR_REDIRECT.to_string());
let redirect_allowed = redirect_is_allowed(&unsafe_redirect);
let safe_redirect_url = if redirect_allowed {
unsafe_redirect
} else {
DEFAULT_INVESTOR_REDIRECT.to_string()
};
let maybe_host = get_request_host(&http_request);
let maybe_cookie_domain = match maybe_host {
Some("jungle.horse") | Some("api.jungle.horse") => Some(".jungle.horse"),
Some("fakeyou.com") | Some("api.fakeyou.com") => Some(".fakeyou.com"),
Some("storyteller.io") | Some("api.storyteller.io") => Some(".storyteller.io"),
_ => None,
};
// Add the cookie.
let mut cookie_builder = Cookie::build(STORYTELLER_DEMO_COOKIE_NAME, "true")
.secure(server_state.env_config.cookie_secure) // HTTPS-only
.http_only(false) // This is meant to be exposed to Javascript!
// NB: Since we're setting this from direct HTTP access rather than XHR, the browser
// implicitly sets the path used to access the cookie, ie. '/demo_mode', which is not what
// we want!
.path("/")
.permanent();
if let Some(cookie_domain) = maybe_cookie_domain {
cookie_builder = cookie_builder.domain(cookie_domain);
}
let cookie = cookie_builder.finish();
HttpResponse::build(StatusCode::FOUND)
.append_header((header::LOCATION, safe_redirect_url.to_string()))
.cookie(cookie)
.finish()
}
@@ -0,0 +1,4 @@
pub mod demo_cookie;
pub mod disable_demo_mode_handler;
pub mod enable_demo_mode_handler;
pub mod default_redirect;
@@ -0,0 +1,255 @@
// NB: Incrementally getting rid of build warnings...
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::http::StatusCode;
use actix_web::{web, HttpRequest, HttpResponse};
use log::{error, info, warn};
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_extended::UserSessionExtended;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::by_table::generic_inference_jobs::inference_model_type::InferenceModelType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::payloads::generic_inference_args::inner_payloads::mocap_payload::{MocapArgs, MocapVideoSource};
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::http_server::web_utils::response_error_helpers::to_simple_json_error;
use crate::state::server_state::ServerState;
/// Debug requests can get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
const DEBUG_HEADER_NAME : &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
const ROUTING_TAG_HEADER_NAME : &str = "routing-tag";
#[derive(Deserialize)]
pub struct EnqueueMocapnetRequest {
uuid_idempotency_token: String,
video_source: Option<MediaFileToken>,
creator_set_visibility: Option<Visibility>,
}
/// Treated as an enum. Only one of these may be set.
#[derive(Serialize)]
pub struct EnqueueMocapnetSuccessResponse {
pub success: bool,
pub inference_job_token: InferenceJobToken,
}
#[derive(Debug)]
pub enum EnqueueMocapnetError {
BadInput(String),
NotAuthorized,
ServerError,
RateLimited,
}
impl ResponseError for EnqueueMocapnetError {
fn status_code(&self) -> StatusCode {
match *self {
EnqueueMocapnetError::BadInput(_) => StatusCode::BAD_REQUEST,
EnqueueMocapnetError::NotAuthorized => StatusCode::UNAUTHORIZED,
EnqueueMocapnetError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
EnqueueMocapnetError::RateLimited => StatusCode::TOO_MANY_REQUESTS,
}
}
fn error_response(&self) -> HttpResponse {
let error_reason = match self {
EnqueueMocapnetError::BadInput(reason) => reason.to_string(),
EnqueueMocapnetError::NotAuthorized => "unauthorized".to_string(),
EnqueueMocapnetError::ServerError => "server error".to_string(),
EnqueueMocapnetError::RateLimited => "rate limited".to_string(),
};
to_simple_json_error(&error_reason, self.status_code())
}
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
impl std::fmt::Display for EnqueueMocapnetError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{:?}", self)
}
}
pub async fn enqueue_mocapnet_handler(
http_request: HttpRequest,
request: web::Json<EnqueueMocapnetRequest>,
server_state: web::Data<Arc<ServerState>>) -> Result<HttpResponse, EnqueueMocapnetError>
{
let mut maybe_user_token : Option<UserToken> = None;
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
warn!("MySql pool error: {:?}", err);
EnqueueMocapnetError::ServerError
})?;
let maybe_avt_token = server_state.avt_cookie_manager.get_avt_token_from_request(&http_request);
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
EnqueueMocapnetError::ServerError
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(UserToken::new_from_str(&user_session.user_token));
}
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref());
// TODO: Separate priority for animation.
let priority_level = plan.web_vc_base_priority_level();
// ==================== DEBUG MODE + ROUTING TAG ==================== //
let is_debug_request =
get_request_header_optional(&http_request, DEBUG_HEADER_NAME)
.is_some();
let maybe_routing_tag=
get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME)
.map(|routing_tag| routing_tag.trim().to_string());
// ==================== RATE LIMIT ==================== //
let rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.logged_out,
Some(ref user) => {
if user.role.is_banned {
return Err(EnqueueMocapnetError::NotAuthorized);
}
&server_state.redis_rate_limiters.logged_in
},
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(EnqueueMocapnetError::RateLimited);
}
// ==================== HANDLE IDEMPOTENCY ==================== //
if let Err(reason) = validate_idempotency_token_format(&request.uuid_idempotency_token) {
return Err(EnqueueMocapnetError::BadInput(reason));
}
insert_idempotency_token(&request.uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
EnqueueMocapnetError::BadInput("invalid idempotency token".to_string())
})?;
// ==================== LOOK UP MODEL INFO ==================== //
// TODO(bt): CHECK DATABASE FOR TOKENS!
let video_source: MocapVideoSource = request.video_source
.as_ref()
.map(|token| MocapVideoSource::media_file_token(token.as_str()))
.ok_or_else(|| EnqueueMocapnetError::BadInput("video source not fully specified".to_string()))?;
let ip_address = get_request_ip(&http_request);
let maybe_user_preferred_visibility : Option<Visibility> = maybe_user_session
.as_ref()
.map(|user_session: &UserSessionExtended| user_session.preferences.preferred_tts_result_visibility); // TODO: New setting for web-vc
let set_visibility = request.creator_set_visibility
.or(maybe_user_preferred_visibility)
.unwrap_or(Visibility::Public);
let inference_args = MocapArgs {
maybe_video_source: Some(video_source),
maybe_ik1: Some(0.001f32),
maybe_ik2: Some(99i32),
maybe_ik3: Some(99i32),
maybe_smoothing1: Some(60f32),
maybe_smoothing2: Some(10f32),
maybe_size1: Some(1920i32),
maybe_size2: Some(1080i32),
};
info!("Creating mocapnet job record...");
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::MocapNet,
maybe_product_category: None, // This is not a product anymore
inference_category: InferenceCategory::Mocap,
maybe_model_type: Some(InferenceModelType::MocapNet), // NB: Model is static during inference
maybe_model_token: None, // NB: Model is static during inference
maybe_input_source_token: None, // TODO: Introduce a second foreign key ?
maybe_input_source_token_type: None, // TODO: Introduce a second foreign key ?
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: None, // No text
maybe_max_duration_seconds: None,
maybe_inference_args: Some(GenericInferenceArgs {
inference_category: Some(InferenceCategoryAbbreviated::Mocap),
args: Some(PolymorphicInferenceArgs::Mc(inference_args)),
}),
maybe_creator_user_token: maybe_user_token.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: set_visibility,
priority_level,
requires_keepalive: plan.mocapnet_requires_frontend_keepalive(),
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool, // TODO(bt,2024-02-04): Reuse connection
}).await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
Err(err) => {
warn!("New generic inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(EnqueueMocapnetError::BadInput("Duplicate idempotency token".to_string()));
}
return Err(EnqueueMocapnetError::ServerError);
}
};
let response: EnqueueMocapnetSuccessResponse = EnqueueMocapnetSuccessResponse {
success: true,
inference_job_token: job_token,
};
let body = serde_json::to_string(&response)
.map_err(|_e| EnqueueMocapnetError::ServerError)?;
Ok(HttpResponse::Ok()
.content_type("application/json")
.body(body))
}
@@ -0,0 +1 @@
pub(crate) mod enqueue_mocapnet;
@@ -6,8 +6,10 @@ pub mod engine;
pub mod events;
pub mod flags;
pub mod image_gen;
pub mod investor_demo;
pub mod leaderboard;
pub mod media_uploads;
pub mod mocap;
pub mod moderation;
pub mod stubs;
pub mod vocoders;
@@ -1,62 +0,0 @@
use log::warn;
use sqlx::{Executor, MySql};
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use tokens::tokens::users::UserToken;
/// Best-effort info-level debug log of an inbound generation HTTP request.
/// Never fails the request — insert errors are logged and swallowed.
pub async fn insert_generation_request_debug_log<'e, 'c: 'e, E>(
debug_log_event_token: &'e DebugLogEventToken,
user_token: &'e UserToken,
ip_address: &'e str,
request_url: &'e str,
request_json: &'e str,
mysql_executor: E,
) where
E: 'c + Executor<'c, Database = MySql>,
{
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(ip_address),
maybe_url: Some(request_url),
message: request_json,
mysql_executor,
phantom: Default::default(),
}).await {
warn!("Failed to insert HTTP request debug log: {:?}", err);
}
}
/// Best-effort error-level debug log of a generation backend failure.
/// Never fails the request — insert errors are logged and swallowed.
pub async fn insert_generation_failure_debug_log<'e, 'c: 'e, E>(
debug_log_event_token: &'e DebugLogEventToken,
user_token: &'e UserToken,
ip_address: &'e str,
request_url: &'e str,
error_message: &'e str,
mysql_executor: E,
) where
E: 'c + Executor<'c, Database = MySql>,
{
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::BackendFailure,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_ip_address: Some(ip_address),
maybe_url: Some(request_url),
message: error_message,
mysql_executor,
phantom: Default::default(),
}).await {
warn!("Failed to insert generation failure debug log: {:?}", err);
}
}
@@ -1,5 +1,4 @@
mod test_synthetic_failure_reason;
pub mod generation_debug_logs;
pub mod job_failure_test;
pub mod map_seedance2pro_web_errors;
pub mod payments_error_test;
mod test_synthetic_failure_reason;
@@ -3,9 +3,6 @@ use std::sync::Arc;
use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_else_common_web_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::common_responses::media::media_links_builder::MediaLinksBuilder;
use crate::http_server::endpoints::generate::common::generation_debug_logs::{
insert_generation_failure_debug_log, insert_generation_request_debug_log,
};
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::endpoints::media_files::helpers::get_media_domain::get_media_domain;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
@@ -41,7 +38,6 @@ use mysql_queries::queries::media_files::get::get_media_file::{get_media_file, g
use mysql_queries::queries::prompts::insert_prompt::{insert_prompt, InsertPromptArgs};
use sqlx::Acquire;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use utoipa::ToSchema;
/// Seedance 1.0 Lite Image to Video
@@ -88,21 +84,6 @@ pub async fn generate_seedance_1_0_lite_image_to_video_handler(
}
};
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
let debug_log_event_token = DebugLogEventToken::generate();
let ip_address = get_request_ip(&http_request);
let request_url = http_request.uri().to_string();
insert_generation_request_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&serde_json::to_string(&*request).unwrap_or_default(),
&mut *mysql_connection,
).await;
let start_frame_media_file_token = match &request.media_file_token {
Some(token) => token,
None => {
@@ -227,21 +208,12 @@ pub async fn generate_seedance_1_0_lite_image_to_video_handler(
api_key: &server_state.inference_providers.fal.api_key,
};
let fal_result = match enqueue_seedance_1_lite_image_to_video_webhook(args).await {
Ok(result) => result,
Err(err) => {
warn!("Error calling enqueue_seedance_1_lite_image_to_video_webhook: {:?}", err);
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 1.0 Lite generation failed: {:?}", err),
&mut *mysql_connection,
).await;
return Err(CommonWebError::from_error(err));
}
};
let fal_result = enqueue_seedance_1_lite_image_to_video_webhook(args)
.await
.map_err(|err| {
warn!("Error calling enqueue_seedance_1_lite_image_to_video_webhook: {:?}", err);
CommonWebError::from_error(err)
})?;
let external_job_id = fal_result.request_id
.ok_or_else(|| {
@@ -251,6 +223,8 @@ pub async fn generate_seedance_1_0_lite_image_to_video_handler(
info!("Fal request_id: {}", external_job_id);
let ip_address = get_request_ip(&http_request);
let mut transaction = mysql_connection
.begin()
.await
@@ -320,7 +294,7 @@ pub async fn generate_seedance_1_0_lite_image_to_video_handler(
starting_job_status_override: None,
maybe_frontend_failure_category: None,
maybe_failure_reason: None,
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_debug_log_event_token: None,
phantom: Default::default(),
}).await;
@@ -3,9 +3,6 @@ use std::sync::Arc;
use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_else_common_web_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::common_responses::media::media_links_builder::MediaLinksBuilder;
use crate::http_server::endpoints::generate::common::generation_debug_logs::{
insert_generation_failure_debug_log, insert_generation_request_debug_log,
};
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::endpoints::media_files::helpers::get_media_domain::get_media_domain;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
@@ -38,7 +35,6 @@ use mysql_queries::queries::media_files::get::get_media_file::{get_media_file, g
use mysql_queries::queries::prompts::insert_prompt::{insert_prompt, InsertPromptArgs};
use sqlx::Acquire;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use utoipa::ToSchema;
/// Seedance 1.0 Pro Image to Video
@@ -85,21 +81,6 @@ pub async fn generate_seedance_1_0_pro_image_to_video_handler(
}
};
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
let debug_log_event_token = DebugLogEventToken::generate();
let ip_address = get_request_ip(&http_request);
let request_url = http_request.uri().to_string();
insert_generation_request_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&serde_json::to_string(&*request).unwrap_or_default(),
&mut *mysql_connection,
).await;
let start_frame_media_file_token = match &request.media_file_token {
Some(token) => token,
None => {
@@ -201,21 +182,12 @@ pub async fn generate_seedance_1_0_pro_image_to_video_handler(
api_key: &server_state.inference_providers.fal.api_key,
};
let fal_result = match enqueue_seedance_1_pro_image_to_video_webhook(args).await {
Ok(result) => result,
Err(err) => {
warn!("Error calling enqueue_seedance_1_pro_image_to_video_webhook: {:?}", err);
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 1.0 Pro generation failed: {:?}", err),
&mut *mysql_connection,
).await;
return Err(CommonWebError::from_error(err));
}
};
let fal_result = enqueue_seedance_1_pro_image_to_video_webhook(args)
.await
.map_err(|err| {
warn!("Error calling enqueue_seedance_1_pro_image_to_video_webhook: {:?}", err);
CommonWebError::from_error(err)
})?;
let external_job_id = fal_result.request_id
.ok_or_else(|| {
@@ -225,6 +197,8 @@ pub async fn generate_seedance_1_0_pro_image_to_video_handler(
info!("Fal request_id: {}", external_job_id);
let ip_address = get_request_ip(&http_request);
let mut transaction = mysql_connection
.begin()
.await
@@ -294,7 +268,7 @@ pub async fn generate_seedance_1_0_pro_image_to_video_handler(
starting_job_status_override: None,
maybe_frontend_failure_category: None,
maybe_failure_reason: None,
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_debug_log_event_token: None,
phantom: Default::default(),
}).await;
@@ -3,9 +3,6 @@ use std::sync::Arc;
use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_else_common_web_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoints::generate::common::generation_debug_logs::{
insert_generation_failure_debug_log, insert_generation_request_debug_log,
};
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::http_server::web_utils::get_request_platform_type::get_request_platform_type;
@@ -33,7 +30,6 @@ use mysql_queries::queries::prompt_context_items::insert_batch_prompt_context_it
use mysql_queries::queries::prompts::insert_prompt::{insert_prompt, InsertPromptArgs};
use sqlx::Acquire;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
/// Seedance 1.5 Pro Multi-Function (text and image to video)
@@ -82,21 +78,6 @@ pub async fn seedance_1p5_pro_multi_function_video_gen_handler(
}
};
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
let debug_log_event_token = DebugLogEventToken::generate();
let ip_address = get_request_ip(&http_request);
let request_url = http_request.uri().to_string();
insert_generation_request_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&serde_json::to_string(&*request).unwrap_or_default(),
&mut *mysql_connection,
).await;
if let Err(reason) = validate_idempotency_token_format(&request.uuid_idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
@@ -196,21 +177,12 @@ pub async fn seedance_1p5_pro_multi_function_video_gen_handler(
api_key: &server_state.inference_providers.fal.api_key,
};
fal_result = match enqueue_seedance_1p5_pro_image_to_video_webhook(args).await {
Ok(result) => result,
Err(err) => {
warn!("Error calling enqueue_seedance_1p5_pro_image_to_video_webhook: {:?}", err);
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 1.5 Pro generation failed: {:?}", err),
&mut *mysql_connection,
).await;
return Err(CommonWebError::from_error(err));
}
};
fal_result = enqueue_seedance_1p5_pro_image_to_video_webhook(args)
.await
.map_err(|err| {
warn!("Error calling enqueue_seedance_1p5_pro_image_to_video_webhook: {:?}", err);
CommonWebError::from_error(err)
})?;
} else {
info!("text-to-video case");
@@ -245,21 +217,12 @@ pub async fn seedance_1p5_pro_multi_function_video_gen_handler(
&mut mysql_connection,
).await?;
fal_result = match enqueue_seedance_1p5_pro_text_to_video_webhook(args).await {
Ok(result) => result,
Err(err) => {
warn!("Error calling enqueue_seedance_1p5_pro_text_to_video_webhook: {:?}", err);
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 1.5 Pro generation failed: {:?}", err),
&mut *mysql_connection,
).await;
return Err(CommonWebError::from_error(err));
}
};
fal_result = enqueue_seedance_1p5_pro_text_to_video_webhook(args)
.await
.map_err(|err| {
warn!("Error calling enqueue_seedance_1p5_pro_text_to_video_webhook: {:?}", err);
CommonWebError::from_error(err)
})?;
}
let external_job_id = fal_result.request_id
@@ -270,6 +233,8 @@ pub async fn seedance_1p5_pro_multi_function_video_gen_handler(
info!("Fal request_id: {}", external_job_id);
let ip_address = get_request_ip(&http_request);
let mut transaction = mysql_connection
.begin()
.await
@@ -377,7 +342,7 @@ pub async fn seedance_1p5_pro_multi_function_video_gen_handler(
starting_job_status_override: None,
maybe_frontend_failure_category: None,
maybe_failure_reason: None,
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_debug_log_event_token: None,
phantom: Default::default(),
}).await;
@@ -5,12 +5,6 @@ use crate::billing::wallets::attempt_wallet_deduction::attempt_wallet_deduction_
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::endpoint_helpers::refund_wallet_after_api_failure::refund_wallet_after_api_failure;
use crate::http_server::endpoints::generate::common::map_seedance2pro_web_errors::map_seedance2pro_error_to_web_error;
use crate::http_server::endpoints::generate::common::generation_debug_logs::{
insert_generation_failure_debug_log, insert_generation_request_debug_log,
};
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use crate::http_server::endpoints::generate::common::payments_error_test::payments_error_test;
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_feature_flags::UserSessionFeatureFlags;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
@@ -55,9 +49,6 @@ use sqlx::MySql;
use tokens::tokens::characters::CharacterToken;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::non_unique::debug_logs_event_token::DebugLogEventToken;
use tokens::tokens::users::UserToken;
use url::Url;
use url_utils::extension::extract_extension_from_url::{extract_extension_from_url, ExtractExtensions};
@@ -69,14 +60,6 @@ struct SeedanceGenerationResult {
generation_mode: CommonGenerationMode,
}
/// Request-scoped context for writing `debug_logs` rows.
struct DebugLogContext<'a> {
event_token: &'a DebugLogEventToken,
user_token: &'a UserToken,
ip_address: &'a str,
request_url: &'a str,
}
// ======================== Handler ========================
/// Seedance 2.0 Multi-Function video generation (text-to-video, keyframe, and reference).
@@ -125,28 +108,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
let user_token = &user_session.user_token;
// ==================== DEBUG LOG: HTTP REQUEST ==================== //
let debug_log_event_token = DebugLogEventToken::generate();
let ip_address = get_request_ip(&http_request);
let request_url = http_request.uri().to_string();
insert_generation_request_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&serde_json::to_string(&*request).unwrap_or_default(),
&mut *mysql_connection,
).await;
let debug_log_context = DebugLogContext {
event_token: &debug_log_event_token,
user_token,
ip_address: &ip_address,
request_url: &request_url,
};
if let Err(reason) = validate_idempotency_token_format(&request.uuid_idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
@@ -242,8 +203,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
batch_count,
duration_seconds,
kinovi_character_ids.clone(),
&debug_log_context,
&mut mysql_connection,
).await;
match result {
@@ -267,8 +226,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
batch_count,
duration_seconds,
kinovi_character_ids.clone(),
&debug_log_context,
&mut mysql_connection,
).await;
match result {
@@ -291,14 +248,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
}
}
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 2.0 generation failed: {:?}", err),
&mut *mysql_connection,
).await;
refund_wallet_after_api_failure(&deduction_result.ledger_entry_token, &mut mysql_connection).await?;
return Err(err);
}
@@ -321,8 +270,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
batch_count,
duration_seconds,
kinovi_character_ids,
&debug_log_context,
&mut mysql_connection,
).await;
match result {
@@ -342,14 +289,6 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
}
}
insert_generation_failure_debug_log(
&debug_log_event_token,
user_token,
&ip_address,
&request_url,
&format!("Seedance 2.0 generation failed: {:?}", err),
&mut *mysql_connection,
).await;
refund_wallet_after_api_failure(&deduction_result.ledger_entry_token, &mut mysql_connection).await?;
return Err(err);
}
@@ -366,6 +305,8 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
// --- DB writes in a transaction ---
let ip_address = get_request_ip(&http_request);
let mut transaction = mysql_connection
.begin()
.await
@@ -497,7 +438,7 @@ pub async fn seedance_2p0_multi_function_video_gen_handler(
creator_set_visibility: Visibility::Public,
maybe_platform_type: get_request_platform_type(&http_request),
maybe_cost_estimates: None,
maybe_debug_log_event_token: Some(&debug_log_event_token),
maybe_debug_log_event_token: None,
mysql_executor: &mut *transaction,
phantom: Default::default(),
}
@@ -602,8 +543,6 @@ async fn upload_and_generate(
batch_count: KinoviBatchCount,
duration_seconds: u8,
kinovi_character_ids: Option<Vec<String>>,
debug_log_context: &DebugLogContext<'_>,
mysql_connection: &mut sqlx::pool::PoolConnection<MySql>,
) -> Result<SeedanceGenerationResult, CommonWebError> {
// --- Upload files to seedance2pro CDN ---
@@ -666,44 +605,24 @@ async fn upload_and_generate(
let prompt = request.prompt.clone().unwrap_or_else(|| "".to_string());
let kinovi_request = KinoviGenerateVideoRequest {
model_type: KinoviModelType::Seedance2Pro,
prompt,
aspect_ratio,
output_resolution,
duration_seconds,
batch_count,
start_frame_url,
end_frame_url,
reference_image_urls,
reference_video_urls,
reference_audio_urls,
character_ids: kinovi_character_ids,
use_face_blur_hack: None,
};
// ==================== DEBUG LOG: KINOVI REQUEST ==================== //
// Logged BEFORE the send so the outbound payload is captured even when
// the generation call fails.
if let Err(err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(debug_log_context.event_token),
maybe_creator_user_token: Some(debug_log_context.user_token),
debug_log_type: DebugLogType::KinoviRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(debug_log_context.ip_address),
maybe_url: Some(debug_log_context.request_url),
message: &format!("{:#?}", kinovi_request),
mysql_executor: &mut **mysql_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert Kinovi request debug log: {:?}", err);
}
let video_gen_args = GenerateVideoArgs {
session,
host_override: None,
request: kinovi_request,
request: KinoviGenerateVideoRequest {
model_type: KinoviModelType::Seedance2Pro,
prompt,
aspect_ratio,
output_resolution,
duration_seconds,
batch_count,
start_frame_url,
end_frame_url,
reference_image_urls,
reference_video_urls,
reference_audio_urls,
character_ids: kinovi_character_ids,
use_face_blur_hack: None,
},
};
let gen_response = generate_video(video_gen_args).await
@@ -22,6 +22,7 @@ pub mod omni_gen;
pub mod prompts;
pub mod service;
pub mod stats;
pub mod studio_gen2;
pub mod subscriptions;
pub mod tags;
pub mod trending;
@@ -32,6 +33,9 @@ pub mod user_referral_codes;
pub mod user_wallets;
pub mod users;
pub mod video_info;
pub mod voice_clone_requests;
pub mod voice_conversion;
pub mod voice_designer;
pub mod web_referrals;
pub mod webhooks;
pub mod weights;
@@ -16,7 +16,6 @@ use enums::common::generation::common_model_type::CommonModelType;
use enums::common::generation_provider::GenerationProvider;
use enums::common::platform_type::PlatformType;
use http_server_common::request::get_request_ip::get_request_ip;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
@@ -140,7 +139,7 @@ pub async fn omni_api_image_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &serde_json::to_string(&*request).unwrap_or_default(),
@@ -162,32 +161,7 @@ pub async fn omni_api_image_generate_handler(
server_state: &server_state,
user_token,
resolved_media: &resolved_media,
}).await;
// ==================== DEBUG LOG: PIPELINE ERROR ==================== //
let pipeline_result = match pipeline_result {
Ok(result) => result,
Err(err) => {
// Best-effort error log; never mask the original error.
if let Ok(mut error_log_connection) = server_state.mysql_pool.acquire().await {
if let Err(log_err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::BackendFailure,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("Image generation pipeline failed: {:?}", err),
mysql_executor: &mut *error_log_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert pipeline error debug log: {:?}", log_err);
}
}
return Err(err);
}
};
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
@@ -199,9 +173,9 @@ pub async fn omni_api_image_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -18,7 +18,6 @@ use enums::common::generation::common_video_model::CommonVideoModel;
use enums::common::generation_provider::GenerationProvider;
use enums::common::platform_type::PlatformType;
use http_server_common::request::get_request_ip::get_request_ip;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
@@ -207,7 +206,7 @@ pub async fn omni_api_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &serde_json::to_string(&*request).unwrap_or_default(),
@@ -229,32 +228,7 @@ pub async fn omni_api_video_generate_handler(
media_file_to_url_map: &media_file_to_url_map,
kinovi_character_id_map: &kinovi_character_id_map,
kinovi_account,
}).await;
// ==================== DEBUG LOG: PIPELINE ERROR ==================== //
let pipeline_result = match pipeline_result {
Ok(result) => result,
Err(err) => {
// Best-effort error log; never mask the original error.
if let Ok(mut error_log_connection) = server_state.mysql_pool.acquire().await {
if let Err(log_err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::BackendFailure,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("Video generation pipeline failed: {:?}", err),
mysql_executor: &mut *error_log_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert pipeline error debug log: {:?}", log_err);
}
}
return Err(err);
}
};
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
@@ -266,9 +240,9 @@ pub async fn omni_api_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -286,9 +260,9 @@ pub async fn omni_api_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::GrokApiRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -15,7 +15,6 @@ use enums::common::generation::common_generation_mode::CommonGenerationMode;
use enums::common::generation::common_model_type::CommonModelType;
use enums::common::generation_provider::GenerationProvider;
use http_server_common::request::get_request_ip::get_request_ip;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
@@ -138,7 +137,7 @@ pub async fn omni_gen_image_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &serde_json::to_string(&*request).unwrap_or_default(),
@@ -160,32 +159,7 @@ pub async fn omni_gen_image_generate_handler(
server_state: &server_state,
user_token,
resolved_media: &resolved_media,
}).await;
// ==================== DEBUG LOG: PIPELINE ERROR ==================== //
let pipeline_result = match pipeline_result {
Ok(result) => result,
Err(err) => {
// Best-effort error log; never mask the original error.
if let Ok(mut error_log_connection) = server_state.mysql_pool.acquire().await {
if let Err(log_err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::BackendFailure,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("Image generation pipeline failed: {:?}", err),
mysql_executor: &mut *error_log_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert pipeline error debug log: {:?}", log_err);
}
}
return Err(err);
}
};
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
@@ -197,9 +171,9 @@ pub async fn omni_gen_image_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -17,7 +17,6 @@ use enums::common::generation::common_model_type::CommonModelType;
use enums::common::generation::common_video_model::CommonVideoModel;
use enums::common::generation_provider::GenerationProvider;
use http_server_common::request::get_request_ip::get_request_ip;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use mysql_queries::queries::debug_logs::insert_debug_log::{insert_debug_log, InsertDebugLogArgs};
use mysql_queries::queries::generic_inference::api_providers::seedance2pro::insert_generic_inference_job_for_seedance2pro_queue_with_apriori_job_token::KinoviVersion;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
@@ -199,7 +198,7 @@ pub async fn omni_gen_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::HttpRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &serde_json::to_string(&*request).unwrap_or_default(),
@@ -221,32 +220,7 @@ pub async fn omni_gen_video_generate_handler(
media_file_to_url_map: &media_file_to_url_map,
kinovi_character_id_map: &kinovi_character_id_map,
kinovi_account,
}).await;
// ==================== DEBUG LOG: PIPELINE ERROR ==================== //
let pipeline_result = match pipeline_result {
Ok(result) => result,
Err(err) => {
// Best-effort error log; never mask the original error.
if let Ok(mut error_log_connection) = server_state.mysql_pool.acquire().await {
if let Err(log_err) = insert_debug_log(InsertDebugLogArgs {
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::BackendFailure,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("Video generation pipeline failed: {:?}", err),
mysql_executor: &mut *error_log_connection,
phantom: Default::default(),
}).await {
warn!("Failed to insert pipeline error debug log: {:?}", log_err);
}
}
return Err(err);
}
};
}).await?;
let mut mysql_connection = server_state.mysql_pool.acquire().await?;
@@ -258,9 +232,9 @@ pub async fn omni_gen_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::FalRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -278,9 +252,9 @@ pub async fn omni_gen_video_generate_handler(
apriori_debug_log_event_token: Some(&debug_log_event_token),
maybe_creator_user_token: Some(user_token),
debug_log_type: DebugLogType::GrokApiRequest,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
maybe_log_level: None,
maybe_ip_address: Some(&ip_address),
maybe_url: Some(&request_url),
message: &format!("{:#?}", outbound_request),
mysql_executor: &mut *mysql_connection,
phantom: Default::default(),
@@ -0,0 +1,298 @@
// NB: Incrementally getting rid of build warnings...
// #![forbid(unused_imports)]
// #![forbid(unused_mut)]
// #![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::http::StatusCode;
use actix_web::web::Json;
use actix_web::{web, HttpRequest, HttpResponse};
use log::{error, info, warn};
use utoipa::ToSchema;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::common::watermark_type::WatermarkType;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::payloads::generic_inference_args::inner_payloads::studio_gen2_payload::StudioGen2Payload;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use mysql_queries::queries::generic_inference::web::kill_jobs_in_development::kill_jobs_in_development;
use mysql_queries::queries::idepotency_tokens::insert_idempotency_token::insert_idempotency_token;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::media_files::MediaFileToken;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::configs::plans::plan_category::PlanCategory;
use crate::http_server::requests::request_headers::get_routing_tag_header::get_routing_tag_header;
use crate::http_server::requests::request_headers::has_debug_header::has_debug_header;
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_extended::UserSessionExtended;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::http_server::web_utils::response_error_helpers::to_simple_json_error;
use crate::state::server_state::ServerState;
use crate::util::allowed_video_style_transfer_access::allowed_video_style_transfer_access;
#[derive(Deserialize, ToSchema)]
pub struct EnqueueStudioGen2Request {
/// Entropy for request de-duplication (required)
pub uuid_idempotency_token: String,
/// The input image media file (required)
pub image_file: MediaFileToken,
/// The input video media file (required)
pub video_file: MediaFileToken,
/// Remove watermark from the output
/// Only for premium accounts
pub remove_watermark: Option<bool>,
/// Optional visibility setting override.
pub creator_set_visibility: Option<Visibility>,
/// Sleep for debugging
pub debug_sleep_millis: Option<u64>,
/// Kill old jobs (only in development)
pub debug_kill_old_jobs: Option<bool>,
// TODO
pub output_width: Option<u64>,
pub output_height: Option<u64>,
pub fps: Option<u64>,
// TODO
pub trim_duration_millis: Option<u64>,
// TODO
pub max_frames: Option<u64>,
pub rounds: Option<u64>,
pub skip_image_resize: Option<bool>,
/// Model image tensor width (we do not resize images to this!)
pub tensor_image_width: Option<u64>,
/// Model image tensor height (we do not resize images to this!)
pub tensor_image_height: Option<u64>,
}
#[derive(Serialize, ToSchema)]
pub struct EnqueueStudioGen2Response {
pub success: bool,
pub inference_job_token: InferenceJobToken,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
/// Enqueue Studio Gen2 jobs.
#[utoipa::path(
post,
tag = "Studio Gen2",
path = "/v1/studio_gen2/enqueue",
responses(
(status = 200, description = "Success", body = EnqueueStudioGen2Response),
(status = 400, description = "Bad input", body = CommonWebError),
(status = 401, description = "Not authorized", body = CommonWebError),
(status = 429, description = "Rate limited", body = CommonWebError),
(status = 500, description = "Server error", body = CommonWebError)
),
params(("request" = EnqueueStudioGen2Request, description = "Payload for request"))
)]
pub async fn enqueue_studio_gen2_handler(
http_request: HttpRequest,
request: Json<EnqueueStudioGen2Request>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<EnqueueStudioGen2Response>, CommonWebError> {
let a = 1;
// ==================== DB ==================== //
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
warn!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
let maybe_avt_token = server_state.avt_cookie_manager
.get_avt_token_from_request(&http_request);
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let maybe_user_token = maybe_user_session
.as_ref()
.map(|session| session.user_token_typed.clone());
// ==================== FEATURE FLAG CHECK ==================== //
//if !allowed_video_style_transfer_access(maybe_user_session.as_ref(), &server_state.flags) {
// warn!("Video style transfer access is not permitted for user");
// return Err(CommonWebError::NotAuthorized);
//}
// ==================== PAID PLAN + PRIORITY ==================== //
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref());
// TODO: Separate priority for animation.
let priority_level = plan.web_vc_base_priority_level();
let is_staff = maybe_user_session
.as_ref()
.map(|user| user.role.can_ban_users)
.unwrap_or(false);
// ==================== DEBUG MODE + ROUTING TAG ==================== //
let is_debug_request = has_debug_header(&http_request);
let maybe_routing_tag= get_routing_tag_header(&http_request);
// ==================== RATE LIMIT ==================== //
let rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.logged_out,
Some(ref user) => {
if user.role.is_banned {
return Err(CommonWebError::NotAuthorized);
}
&server_state.redis_rate_limiters.logged_in
},
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
// ==================== HANDLE IDEMPOTENCY ==================== //
if let Err(reason) = validate_idempotency_token_format(&request.uuid_idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
insert_idempotency_token(&request.uuid_idempotency_token, &mut *mysql_connection)
.await
.map_err(|err| {
error!("Error inserting idempotency token: {:?}", err);
CommonWebError::BadInputWithSimpleMessage("invalid idempotency token".to_string())
})?;
// NB: The connection isn't needed past this point (the enqueue below uses the pool directly),
// so release it now rather than holding a pool slot for the rest of the handler.
drop(mysql_connection);
// ==================== LOOK UP MODEL INFO ==================== //
// TODO(bt): CHECK DATABASE FOR TOKENS!
let ip_address = get_request_ip(&http_request);
let maybe_user_preferred_visibility : Option<Visibility> = maybe_user_session
.as_ref()
.map(|user_session: &UserSessionExtended| user_session.preferences.preferred_tts_result_visibility); // TODO: New setting for web-vc
let set_visibility = request.creator_set_visibility
.or(maybe_user_preferred_visibility)
.unwrap_or(Visibility::Public);
let has_paid_plan = plan.plan_slug() == "fakeyou_contributor" || plan.plan_category() == PlanCategory::Paid;
let watermark_type ;
let is_requested_to_remove_watermark = request.remove_watermark.unwrap_or(false);
let is_allowed_to_remove_watermark = is_staff; // TODO: Paid permission instead
if is_requested_to_remove_watermark && is_allowed_to_remove_watermark {
watermark_type = None;
} else {
watermark_type = Some(WatermarkType::Storyteller); // Studio is only on Storyteller.
}
if request.debug_kill_old_jobs.unwrap_or(false)
&& server_state.server_environment.is_development()
{
info!("Killing old jobs before inserting new job (DEVELOPMENT MODE ONLY!)");
let result = kill_jobs_in_development(&server_state.mysql_pool).await;
if let Err(err) = result {
warn!("Error killing old jobs: {:?}", err);
}
}
let inference_args = StudioGen2Payload {
image_file: Some(request.image_file.clone()),
video_file: Some(request.video_file.clone()),
creator_visibility: Some(set_visibility),
watermark_type,
after_job_debug_sleep_millis: request.debug_sleep_millis,
output_width: request.output_width,
output_height: request.output_height,
fps: request.fps,
max_frames: request.max_frames,
rounds: request.rounds,
trim_duration_millis: request.trim_duration_millis,
skip_image_resize: request.skip_image_resize,
tensor_image_width: request.tensor_image_width,
tensor_image_height: request.tensor_image_height,
};
info!("Creating ComfyUI job record...");
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::StudioGen2,
inference_category: InferenceCategory::DeprecatedField,
maybe_model_type: None,
maybe_product_category: None,
maybe_model_token: None,
maybe_input_source_token: None, // TODO: Introduce a second foreign key ?
maybe_input_source_token_type: None, // TODO: Introduce a second foreign key ?
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: None, // No text
maybe_max_duration_seconds: None,
maybe_inference_args: Some(GenericInferenceArgs {
inference_category: Some(InferenceCategoryAbbreviated::Workflow),
args: Some(PolymorphicInferenceArgs::S2(inference_args)),
}),
maybe_creator_user_token: maybe_user_token.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: set_visibility,
priority_level,
requires_keepalive: false,
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
Err(err) => {
warn!("New generic inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(CommonWebError::BadInputWithSimpleMessage("Duplicate idempotency token".to_string()));
}
return Err(CommonWebError::from_error(err));
}
};
Ok(Json(EnqueueStudioGen2Response {
success: true,
inference_job_token: job_token,
}))
}
@@ -0,0 +1 @@
pub mod enqueue_studio_gen2_handler;
@@ -0,0 +1,524 @@
// NB: Incrementally getting rid of build warnings...
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::http::StatusCode;
use actix_web::web::Json;
use actix_web::{web, HttpRequest, HttpResponse};
use log::{info, warn};
use rand::seq::IndexedRandom;
use rand::Rng;
use utoipa::ToSchema;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_input_source_token_type::InferenceInputSourceTokenType;
use enums::by_table::generic_inference_jobs::inference_job_product_category::InferenceJobProductCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_api_token::get_request_api_token;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use migration::text_to_speech::get_tts_model_for_enqueue_inference_migration::TtsModelForEnqueueInferenceMigrationWrapper;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::payloads::generic_inference_args::inner_payloads::f5_tts_payload::F5TTSPayload;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::media_uploads::MediaUploadToken;
use tokens::tokens::users::UserToken;
use tts_common::priority::{FAKEYOU_DEFAULT_VALID_API_TOKEN_PRIORITY_LEVEL, FAKEYOU_INVESTOR_PRIORITY_LEVEL};
use user_input_common::check_for_slurs::contains_slurs;
use crate::configs::app_startup::username_set::UsernameSet;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::configs::plans::plan::Plan;
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::request_has_demo_cookie;
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_extended::UserSessionExtended;
use crate::http_server::web_utils::response_error_helpers::to_simple_json_error;
use crate::state::server_state::ServerState;
// TODO: Temporary for investor demo
const STORYTELLER_DEMO_COOKIE_NAME : &str = "storyteller_demo";
/// Debug requests mean different things, depending on the type of worker (the legacy
/// tts-inference-job vs. the new inference-job):
///
/// * Legacy (tts): these requests get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
///
/// * New (generic): the flag gets associated with the request which may be used for
/// anything in the code. (There's a new system for routing requests called
/// "routing tags" that is independent of this flag.)
///
const DEBUG_HEADER_NAME : &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
/// NB: This is only for the new job system.
const ROUTING_TAG_HEADER_NAME : &str = "routing-tag";
const USER_FAKEYOU_USER_TOKEN : &str = "U:N5J8JXPW9BTYX";
const USER_NEWS_STORY_USER_TOKEN : &str = "U:XAWRARC1N89X6";
/// Safety guard against plans supporting too lengthy TTS inference text.
const MAX_TTS_LENGTH : usize = 10_000;
#[derive(Deserialize, ToSchema)]
pub struct InferTtsRequest {
uuid_idempotency_token: String,
source_media_file_token: String,
inference_text: String,
creator_set_visibility: Option<Visibility>,
}
#[derive(Serialize, ToSchema)]
pub struct InferTtsSuccessResponse {
pub success: bool,
pub inference_job_token: String,
}
#[derive(Debug, ToSchema)]
pub enum InferTtsError {
BadInput(String),
NotAuthorized,
ServerError,
RateLimited,
NotFound,
}
impl ResponseError for InferTtsError {
fn status_code(&self) -> StatusCode {
match *self {
InferTtsError::BadInput(_) => StatusCode::BAD_REQUEST,
InferTtsError::NotAuthorized => StatusCode::UNAUTHORIZED,
InferTtsError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
InferTtsError::RateLimited => StatusCode::TOO_MANY_REQUESTS,
InferTtsError::NotFound => StatusCode::NOT_FOUND,
}
}
fn error_response(&self) -> HttpResponse {
let error_reason = match self {
InferTtsError::BadInput(reason) => reason.to_string(),
InferTtsError::NotAuthorized => "unauthorized".to_string(),
InferTtsError::ServerError => "server error".to_string(),
InferTtsError::RateLimited => "rate limited".to_string(),
InferTtsError::NotFound => "not found".to_string(),
};
to_simple_json_error(&error_reason, self.status_code())
}
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
impl fmt::Display for InferTtsError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
#[utoipa::path(
post,
tag = "TTS",
path = "/v1/tts/f5_inference",
responses(
(status = 200, description = "Success response", body = InferTtsSuccessResponse),
(status = 400, description = "Bad input", body = InferTtsError),
(status = 401, description = "Not authorized", body = InferTtsError),
(status = 429, description = "Rate limited", body = InferTtsError),
(status = 500, description = "Server error", body = InferTtsError)
),
params(("request" = InferTtsRequest, description = "Payload for Request"))
)]
pub async fn enqueue_infer_f5_tts_handler(
http_request: HttpRequest,
request: Json<InferTtsRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<InferTtsSuccessResponse>, InferTtsError> {
if server_state.flags.disable_tts {
return Err(InferTtsError::RateLimited);
}
let mut is_from_api = false;
let mut maybe_user_token : Option<String> = None;
let mut priority_level ;
let mut use_high_priority_rate_limiter = false; // NB: Careful!
let mut disable_rate_limiter = false; // NB: Careful!
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
warn!("MySql pool error: {:?}", err);
InferTtsError::ServerError
})?;
// ==================== USER SESSION ==================== //
let maybe_user_session : Option<UserSessionExtended> = server_state
.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
InferTtsError::ServerError
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(user_session.user_token.to_string());
}
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref());
priority_level = plan.tts_base_priority_level();
// ==================== API TOKENS ==================== //
if let Some(api_token) = get_request_api_token(&http_request) {
let maybe_api_token_configs = server_state
.static_api_token_set
.get_api_token(&api_token);
if let Some(api_token_configs) = maybe_api_token_configs {
is_from_api = true;
priority_level = api_token_configs.maybe_priority_level
.unwrap_or(FAKEYOU_DEFAULT_VALID_API_TOKEN_PRIORITY_LEVEL);
use_high_priority_rate_limiter = api_token_configs.maybe_use_high_priority_rate_limiter
.unwrap_or(false);
disable_rate_limiter = api_token_configs.maybe_disable_rate_limiter.unwrap_or(false);
if let Some(user_token_override) = api_token_configs.maybe_user_token.as_deref() {
maybe_user_token = Some(user_token_override.trim().to_string());
}
}
}
// ==================== INVESTOR PRIORITY ==================== //
// TODO/TEMP: Give investors even more priority
let mut is_investor = false;
// TODO/TEMP: The storyteller.io website will redirect and establish this cookie.
// This is just for the YCombinator demo.
if request_has_demo_cookie(&http_request) {
is_investor = true;
}
if is_investor {
priority_level = FAKEYOU_INVESTOR_PRIORITY_LEVEL;
}
// ==================== ENGINEERING FLAGS / HEADERS: DEBUG MODE, ETC. ==================== //
let is_debug_request = get_request_header_optional(&http_request, DEBUG_HEADER_NAME)
.is_some();
let maybe_routing_tag=
get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME)
.map(|routing_tag| routing_tag.trim().to_string());
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.role.is_banned {
warn!("User is not authorized to use TTS model because they are banned.");
return Err(InferTtsError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
if !disable_rate_limiter {
let maybe_username = maybe_user_session
.as_ref()
.map(|session| session.user.username.as_str());
// FIXME(bt,2023-12-13): These boolean flags are a bit dangerous.
let rate_limiter_type = get_rate_limiter_type(
maybe_username,
&server_state.redis_rate_limiters.api_ai_streamer_username_set,
is_investor,
is_from_api,
use_high_priority_rate_limiter
);
let rate_limiter = match rate_limiter_type {
RateLimiterType::LoggedOut => &server_state.redis_rate_limiters.logged_out,
RateLimiterType::LoggedIn => &server_state.redis_rate_limiters.logged_in,
RateLimiterType::ApiHighPriority => {
info!("Using API high priority rate limiter");
&server_state.redis_rate_limiters.api_high_priority
},
RateLimiterType::ApiAiStreamer => {
info!("Using AI streamer rate limiter");
&server_state.redis_rate_limiters.api_ai_streamers
},
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(InferTtsError::RateLimited);
}
}
// let is_authorized = check_if_authorized_to_use_media_file(
// maybe_user_session.as_ref(),
// &tts_model,
// ).await;
//
// if !is_authorized {
// warn!("User is not authorized to use TTS model due to checks.");
// return Err(InferTtsError::NotAuthorized);
// }
//
// ==================== CHECK INFERENCE TEXT ==================== //
let ip_address = get_request_ip(&http_request);
let mut inference_text = request.inference_text.trim().to_string();
if let Err(reason) = validate_inference_text(&inference_text, &plan) {
return Err(InferTtsError::BadInput(reason));
}
if contains_slurs(&inference_text) {
return Err(InferTtsError::BadInput("text contains slurs".to_string()));
}
if let Some(user) = maybe_user_session.as_ref() {
let is_user_token_troll_banned = server_state.troll_bans.user_tokens
.contains_user_token(&user.user_token)
.unwrap_or(false); // NB: Fail open
let is_ip_address_troll_banned = server_state.troll_bans.ip_addresses
.contains_ip_address(&ip_address)
.unwrap_or(false); // NB: Fail open
let is_troll_banned = is_user_token_troll_banned || is_ip_address_troll_banned;
if is_troll_banned {
let test = server_state.flags.troll_ban_user_percent as i32; // NB: Should be 0 - 100.
let num = rand::thread_rng().gen_range(0..100);
if test > num {
warn!("Generating troll response for user token: {}", user.user_token);
inference_text = random_troll_text().to_string();
}
}
}
let maybe_user_preferred_visibility : Option<Visibility> = maybe_user_session
.as_ref()
.map(|user_session| user_session.preferences.preferred_tts_result_visibility);
let set_visibility = request.creator_set_visibility
.or(maybe_user_preferred_visibility)
.unwrap_or(Visibility::Public);
// ==================== ENQUEUE APPROPRIATE TTS TYPE ==================== //
let max_duration_seconds = plan.tts_max_duration_seconds();
// This branch uses the `generic-inference-job` service and tables.
info!("Creating tts inference job record (new generic job system)...");
let maybe_creator_user_token_typed = maybe_user_token
.as_deref()
.map(|token| UserToken::new_from_str(token));
let maybe_avt_token = server_state
.avt_cookie_manager
.get_avt_token_from_request(&http_request);
let maybe_model_type = None;
let maybe_inference_args = Some(GenericInferenceArgs{
inference_category: Some(InferenceCategoryAbbreviated::F5TTS),
args: Some(PolymorphicInferenceArgs::Ft(F5TTSPayload{})),
});
let media_token = request.source_media_file_token.clone();
let media_token_type =
if media_token.starts_with(MediaUploadToken::token_prefix()) {
InferenceInputSourceTokenType::MediaUpload
} else if media_token.starts_with(MediaFileToken::token_prefix()) {
InferenceInputSourceTokenType::MediaFile
} else {
return Err(InferTtsError::BadInput(
"input token is not a media_upload or media_file token".to_string()));
};
let maybe_product_category = Some(InferenceJobProductCategory::TtsF5);
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::F5TTS,
maybe_product_category,
inference_category: InferenceCategory::F5TTS,
maybe_model_type,
maybe_model_token: None,
maybe_input_source_token: Some(&media_token),
maybe_input_source_token_type: Some(media_token_type),
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: Some(inference_text.as_str()),
maybe_max_duration_seconds: Some(max_duration_seconds),
maybe_inference_args,
maybe_creator_user_token: maybe_creator_user_token_typed.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: set_visibility,
priority_level,
requires_keepalive: false, // TODO: We may want this to be the case in the future.
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let inference_job_token = match query_result {
Ok((inference_job_token, _id)) => inference_job_token,
Err(err) => {
warn!("New (generic) tts inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(InferTtsError::BadInput("Duplicate idempotency token".to_string()));
}
return Err(InferTtsError::ServerError);
}
};
let job_token = inference_job_token.to_string();
Ok(Json(InferTtsSuccessResponse {
success: true,
inference_job_token: job_token,
}))
}
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum RateLimiterType {
LoggedOut,
LoggedIn,
ApiHighPriority,
ApiAiStreamer,
}
// FIXME(bt,2023-12-13): The boolean flags and disjoint decision making are kind of terrible.
fn get_rate_limiter_type(
maybe_username: Option<&str>,
ai_streamer_username_set: &UsernameSet,
is_investor: bool,
is_from_api: bool,
use_high_priority_rate_limiter: bool,
) -> RateLimiterType {
// For AI live streamers
// https://www.notion.so/storytellerai/dd88609558a24196b4ddeeef6079da98
let mut is_ai_streamer = false;
let mut rate_limiter = match maybe_username {
None => RateLimiterType::LoggedOut,
Some(username) => {
if ai_streamer_username_set.username_is_in_set(username) {
is_ai_streamer = true;
}
RateLimiterType::LoggedIn
},
};
// TODO/TEMP
if is_investor || is_from_api {
rate_limiter = RateLimiterType::LoggedIn;
}
// TODO: This is for VidVoice.ai and should be replaced with per-API consumer rate limiters
if use_high_priority_rate_limiter {
rate_limiter = RateLimiterType::ApiHighPriority;
}
if is_ai_streamer {
rate_limiter = RateLimiterType::ApiAiStreamer;
}
rate_limiter
}
pub fn validate_inference_text(text: &str, plan: &Plan) -> Result<(), String> {
if text.len() < 3 {
return Err("text is too short".to_string());
}
if text.len() > plan.tts_max_character_length() {
return Err("text is too long".to_string());
}
if text.len() > MAX_TTS_LENGTH {
return Err("text is too long".to_string());
}
Ok(())
}
/// If the model is private, determine if the user can use the model.
/// This is designed to fail closed (read: actually open!) rather than hit resources.
async fn check_if_authorized_to_use_model(
maybe_user_session: Option<&UserSessionExtended>,
tts_model: &TtsModelForEnqueueInferenceMigrationWrapper,
) -> bool {
match tts_model.creator_set_visibility() {
Visibility::Public => return true,
Visibility::Hidden => return true,
Visibility::Private => {} // Fall through
}
let is_authorized = maybe_user_session
.map(|session| {
(session.user_token.as_str() == tts_model.creator_user_token()) // is_author
|| (session.role.can_ban_users) // is_mod
|| (session.user_token.as_str() == USER_FAKEYOU_USER_TOKEN) // is allowed user #1
|| (session.user_token.as_str() == USER_NEWS_STORY_USER_TOKEN) // is allowed user #2
})
.unwrap_or(false);
is_authorized
}
fn random_troll_text() -> &'static str {
let texts = [
"Fake You Dot Com. Use Fake You Dot Com. The website Fake You Dot Com has better voices. Use Fake You Dot Com.",
"Use Fake You Dot Com. Fake You Dot Com. Fake You Dot Com is way better.",
"Want better voices? Fake. You. Fake You Dot Com. Fake. Fake. Fake. You. You. You. Use Fake You Dot Com. Fake You Dot Com.",
r#"
You should use Fake You Dot Com. Fake You Dot Com has way better voices.
You can get everything you want from FakeYou dot com. One Oh One Sound Boards website is just using Fake You.
Use Fake You Dot Com. Fake You Dot Com. Fake You Dot com.
AAAAAAAAAAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
"#,
//"EEEEEEEEEEEEEEEEE",
//"One Oh One Soundboards is a shitty website. Use Fake You Dot Com instead.",
//r#"
//beep. beep. beep. beep. beep.
//May have installed a virus
//OOOOOOOOEEEEUUUUUOOAAAA
//Your device may be compromised
//EEEEEEEEEEEEEEEEIIIUUUUOOOAAAA
//Your location may be tracked by the advanced artificial intelligence.
//"#,
];
let random = texts.choose(&mut rand::thread_rng());
random.map(|r| *r).unwrap_or(texts[0])
}
@@ -0,0 +1 @@
pub mod enqueue_infer_f5_tts_handler;
@@ -36,6 +36,7 @@ use user_input_common::check_for_slurs::contains_slurs;
use crate::configs::app_startup::username_set::UsernameSet;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::configs::plans::plan::Plan;
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::request_has_demo_cookie;
use crate::http_server::endpoints::tts::enqueue_infer_tts_handler::get_tts_model_with_caching::get_tts_model_with_caching;
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_extended::UserSessionExtended;
use crate::http_server::web_utils::response_error_helpers::to_simple_json_error;
@@ -234,6 +235,12 @@ pub async fn enqueue_infer_tts_handler(
_ => {},
};
// TODO/TEMP: The storyteller.io website will redirect and establish this cookie.
// This is just for the YCombinator demo.
if request_has_demo_cookie(&http_request) {
is_investor = true;
}
if is_investor {
priority_level = FAKEYOU_INVESTOR_PRIORITY_LEVEL;
}
@@ -1,42 +1,120 @@
use actix_web::HttpResponse;
use chrono::Utc;
use log::error;
use std::fmt;
use std::sync::Arc;
use actix_web::{http::StatusCode, web, HttpRequest, HttpResponse, ResponseError};
use chrono::NaiveDateTime;
use log::{debug, error, warn};
use mysql_queries::queries::tts::tts_inference_jobs::get_pending_tts_inference_job_count::get_pending_tts_inference_job_count;
use crate::http_server::web_utils::response_error_helpers::to_simple_json_error;
use crate::http_server::common_responses::common_web_error::CommonWebError;
/// How fast the frontend should poll this endpoint. Effectively "never" —
/// the endpoint is retired and only zombie traffic hits it.
const REFRESH_INTERVAL_MILLIS: u64 = 1_000_000;
use crate::state::server_state::ServerState;
#[derive(Serialize)]
pub struct Response {
pub success: bool,
pub pending_job_count: u64,
pub cache_time: chrono::NaiveDateTime,
pub cache_time: NaiveDateTime,
/// Tell the frontend client how fast to refresh their view of this list.
/// During an attack, we may want this to go extremely slow.
pub refresh_interval_millis: u64,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
pub async fn get_pending_tts_inference_job_count_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>
) -> Result<HttpResponse, CommonWebError> {
/// RETIRED endpoint that still receives heavy zombie traffic (~300 rpm).
///
/// The TTS queue no longer exists, so this returns a hardcoded zero without
/// touching the cache or the database.
pub async fn get_pending_tts_inference_job_count_handler() -> Result<HttpResponse, CommonWebError> {
let response = Response {
success: true,
pending_job_count: 0,
cache_time: Utc::now().naive_utc(),
refresh_interval_millis: REFRESH_INTERVAL_MILLIS,
};
if server_state.flags.disable_tts_queue_length_endpoint {
// NB: Despite the cache being a powerful protector of the database (this is an expensive query),
// if the cache goes stale during an outage, there is no protection. This feature flag lets us
// shut off all traffic to the endpoint.
return render_response_busy(Response {
success: true,
pending_job_count: 10_000,
cache_time: NaiveDateTime::from_timestamp(0, 0),
refresh_interval_millis: server_state.flags.frontend_pending_tts_refresh_interval_millis,
});
}
let body = serde_json::to_string(&response)
let maybe_cached = server_state.caches.ephemeral.tts_queue_length.grab_copy_without_bump_if_unexpired()
.map_err(|e| {
error!("error returning response: {:?}", e);
CommonWebError::from_error(e)
error!("error consulting cache: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
let count_result = match maybe_cached {
Some(cached) => {
cached
},
None => {
debug!("populating tts queue length from database");
let count_query_result = get_pending_tts_inference_job_count(
&server_state.mysql_pool
).await;
match count_query_result {
// If the database misbehaves (eg. DDoS), let's stop spamming it.
// We'll attempt to read the old value from the cache and keep going.
Err(err) => {
warn!("error querying database / inserting into cache: {:?}", err);
let maybe_cached = server_state.caches.ephemeral.tts_queue_length.grab_even_expired_and_bump()
.map_err(|err| {
error!("error consulting cache (even expired): {:?}", err);
CommonWebError::from_anyhow_error(err)
})?;
maybe_cached.ok_or_else(|| {
error!("error querying database and subsequently reading cache: {:?}", err);
CommonWebError::server_error_with_message("uncaught server error")
})?
}
// Happy path...
Ok(count_result) => {
server_state.caches.ephemeral.tts_queue_length.store_copy(&count_result)
.map_err(|e| {
error!("error storing cache: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
count_result
}
}
},
};
render_response_ok(Response {
success: true,
pending_job_count: count_result.record_count,
cache_time: count_result.present_time,
refresh_interval_millis: server_state.flags.frontend_pending_tts_refresh_interval_millis,
})
}
pub fn render_response_busy(response: Response) -> Result<HttpResponse, CommonWebError> {
let body = render_response_payload(response)?;
Ok(HttpResponse::TooManyRequests()
.content_type("application/json")
.body(body))
}
pub fn render_response_ok(response: Response) -> Result<HttpResponse, CommonWebError> {
let body = render_response_payload(response)?;
Ok(HttpResponse::Ok()
.content_type("application/json")
.body(body))
}
pub fn render_response_payload(response: Response) -> Result<String, CommonWebError> {
let body = serde_json::to_string(&response)
.map_err(|e| {
error!("error returning response: {:?}", e);
CommonWebError::from_error(e)
})?;
Ok(body)
}
@@ -13,4 +13,5 @@ pub mod get_tts_upload_model_job_status;
pub mod list_tts_models;
pub mod list_user_tts_inference_results;
pub mod list_user_tts_models;
pub mod search_tts_models_handler;
pub mod search_tts_models_handler;
pub mod enqueue_infer_f5_tts_handler;
@@ -0,0 +1,88 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::{web, HttpRequest};
use log::warn;
use http_server_common::response::serialize_as_json_error::serialize_as_json_error;
use mysql_queries::queries::voice_clone_requests::get_voice_clone_request::{get_voice_clone_request_by_token, get_voice_clone_request_by_user_token};
use crate::state::server_state::ServerState;
use crate::http_server::common_responses::common_web_error::CommonWebError;
// =============== Request ===============
#[derive(Deserialize)]
pub struct CheckIfVoiceRequestSubmittedRequest {
/// Though we look up logged in users by user token, we can also look up non-logged-in users
/// with a token lookup. We can store this in frontend state or a cookie.
pub maybe_request_token: Option<String>,
}
// =============== Success Response ===============
#[derive(Serialize)]
pub struct CheckIfVoiceRequestSubmittedResponse {
pub success: bool,
pub has_submitted: bool,
}
// =============== Error Response ===============
// NB: Not using DeriveMore since Clion doesn't understand it.
// =============== Handler ===============
pub async fn check_if_voice_clone_request_submitted_handler(
http_request: HttpRequest,
request: Json<CheckIfVoiceRequestSubmittedRequest>,
server_state: web::Data<Arc<ServerState>>) -> Result<Json<CheckIfVoiceRequestSubmittedResponse>, CommonWebError>
{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let mut submitted_by_token = false;
let mut submitted_by_user = false;
// Non-logged in accounts (via cookie or frontend mechanism)
if let Some(token) = request.maybe_request_token.as_deref() {
let request = get_voice_clone_request_by_token(
token, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Database error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
submitted_by_token = request.is_some();
}
// Logged in users
if let Some(user) = maybe_user_session.as_ref() {
let request = get_voice_clone_request_by_user_token(
user.user_token.as_str(), &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Database error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
submitted_by_user = request.is_some();
}
let has_submitted = submitted_by_token || submitted_by_user;
let response = CheckIfVoiceRequestSubmittedResponse {
success: true,
has_submitted,
};
Ok(Json(response))
}
@@ -0,0 +1,118 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::{web, HttpRequest};
use log::warn;
use http_server_common::request::get_request_ip::get_request_ip;
use http_server_common::response::serialize_as_json_error::serialize_as_json_error;
use mysql_queries::queries::voice_clone_requests::insert_voice_clone_requests_builder::InsertVoiceCloneRequestBuilder;
use crate::state::server_state::ServerState;
use crate::http_server::common_responses::common_web_error::CommonWebError;
// =============== Request ===============
#[derive(Deserialize)]
pub struct CreateVoiceRequestRequest {
pub idempotency_token: String,
// Contact
pub email_address: String,
pub discord_username: String,
// TODO: Make enum.
// Visibility
pub is_for_private_use: bool,
pub is_for_public_use: bool,
// Use
pub is_for_studio: bool,
pub is_for_twitch_tts: bool,
pub is_for_api_use: bool,
pub is_for_music: bool,
pub is_for_games: bool,
pub is_for_other: bool,
pub optional_notes_on_use: Option<String>,
// Subject/Ownership
pub is_own_voice: bool,
pub is_third_party_voice: bool,
pub optional_notes_on_subject: Option<String>,
// Equipment
pub has_clean_audio_recordings: bool,
pub has_good_microphone: bool,
// Comments
pub optional_questions: Option<String>,
pub optional_extra_comments: Option<String>,
}
// =============== Success Response ===============
#[derive(Serialize)]
pub struct CreateVoiceRequestResponse {
pub success: bool,
}
// =============== Error Response ===============
// NB: Not using DeriveMore since Clion doesn't understand it.
// =============== Handler ===============
pub async fn create_voice_clone_request_handler(
http_request: HttpRequest,
request: Json<CreateVoiceRequestRequest>,
server_state: web::Data<Arc<ServerState>>) -> Result<Json<CreateVoiceRequestResponse>, CommonWebError>
{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let creator_ip_address = get_request_ip(&http_request);
let builder = InsertVoiceCloneRequestBuilder {
uuid_idempotency_token: request.idempotency_token.clone(),
maybe_user_token: maybe_user_session.map(|user| user.user_token.as_str().to_string()),
email_address: request.email_address.clone(),
discord_username: request.discord_username.clone(),
is_for_private_use: request.is_for_private_use,
is_for_public_use: request.is_for_public_use,
is_for_studio: request.is_for_studio,
is_for_twitch_tts: request.is_for_twitch_tts,
is_for_api_use: request.is_for_api_use,
is_for_music: request.is_for_music,
is_for_games: request.is_for_games,
is_for_other: request.is_for_other,
optional_notes_on_use: request.optional_notes_on_use.clone(),
is_own_voice: request.is_own_voice,
is_third_party_voice: request.is_third_party_voice,
optional_notes_on_subject: request.optional_notes_on_subject.clone(),
has_clean_audio_recordings: request.has_clean_audio_recordings,
has_good_microphone: request.has_good_microphone,
optional_questions: request.optional_questions.clone(),
optional_extra_comments: request.optional_extra_comments.clone(),
ip_address_creation: creator_ip_address,
};
builder.insert(&server_state.mysql_pool)
.await
.map_err(|e| {
warn!("creation error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
let response = CreateVoiceRequestResponse {
success: true,
};
Ok(Json(response))
}
@@ -0,0 +1,2 @@
pub mod create_voice_clone_request;
pub mod check_if_voice_clone_request_submitted;
@@ -0,0 +1,358 @@
// NB: Incrementally getting rid of build warnings...
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{info, warn};
use utoipa::ToSchema;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_input_source_token_type::InferenceInputSourceTokenType;
use enums::by_table::generic_inference_jobs::inference_job_product_category::InferenceJobProductCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_api_token::get_request_api_token;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::payloads::generic_inference_args::inner_payloads::seed_vc_payload::SeedVcPayload;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::media_uploads::MediaUploadToken;
use tokens::tokens::users::UserToken;
use tts_common::priority::{FAKEYOU_DEFAULT_VALID_API_TOKEN_PRIORITY_LEVEL, FAKEYOU_INVESTOR_PRIORITY_LEVEL};
use crate::configs::app_startup::username_set::UsernameSet;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::request_has_demo_cookie;
use crate::http_server::user_lookup::user_session::session_utils::lookup::user_session_extended::UserSessionExtended;
use crate::state::server_state::ServerState;
// TODO: Temporary for investor demo
const STORYTELLER_DEMO_COOKIE_NAME : &str = "storyteller_demo";
///
const DEBUG_HEADER_NAME : &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
/// NB: This is only for the new job system.
const ROUTING_TAG_HEADER_NAME : &str = "routing-tag";
const USER_FAKEYOU_USER_TOKEN : &str = "U:N5J8JXPW9BTYX";
const USER_NEWS_STORY_USER_TOKEN : &str = "U:XAWRARC1N89X6";
#[derive(Deserialize, ToSchema)]
pub struct InferSeedVcRequest {
uuid_idempotency_token: String,
source_media_file_token: String,
reference_media_file_token: String,
creator_set_visibility: Option<Visibility>,
}
#[derive(Serialize, ToSchema)]
pub struct InferSeedVcSuccessResponse {
pub success: bool,
pub inference_job_token: String,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
#[utoipa::path(
post,
tag = "Voice Conversion",
path = "/v1/voice_conversion/seed_vc_inference",
responses(
(status = 200, description = "Success response", body = InferSeedVcSuccessResponse),
(status = 400, description = "Bad input", body = CommonWebError),
(status = 401, description = "Not authorized", body = CommonWebError),
(status = 429, description = "Rate limited", body = CommonWebError),
(status = 500, description = "Server error", body = CommonWebError)
),
params(("request" = InferSeedVcRequest, description = "Payload for Request"))
)]
pub async fn enqueue_infer_seed_vc_handler(
http_request: HttpRequest,
request: Json<InferSeedVcRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<InferSeedVcSuccessResponse>, CommonWebError> {
if server_state.flags.disable_voice_conversion {
return Err(CommonWebError::TooManyRequests);
}
let mut is_from_api = false;
let mut maybe_user_token : Option<String> = None;
let mut priority_level ;
let mut use_high_priority_rate_limiter = false; // NB: Careful!
let mut disable_rate_limiter = false; // NB: Careful!
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
warn!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
// ==================== USER SESSION ==================== //
let maybe_user_session : Option<UserSessionExtended> = server_state
.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(user_session.user_token.to_string());
}
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref());
priority_level = plan.tts_base_priority_level();
// ==================== API TOKENS ==================== //
if let Some(api_token) = get_request_api_token(&http_request) {
let maybe_api_token_configs = server_state
.static_api_token_set
.get_api_token(&api_token);
if let Some(api_token_configs) = maybe_api_token_configs {
is_from_api = true;
priority_level = api_token_configs.maybe_priority_level
.unwrap_or(FAKEYOU_DEFAULT_VALID_API_TOKEN_PRIORITY_LEVEL);
use_high_priority_rate_limiter = api_token_configs.maybe_use_high_priority_rate_limiter
.unwrap_or(false);
disable_rate_limiter = api_token_configs.maybe_disable_rate_limiter.unwrap_or(false);
if let Some(user_token_override) = api_token_configs.maybe_user_token.as_deref() {
maybe_user_token = Some(user_token_override.trim().to_string());
}
}
}
// ==================== INVESTOR PRIORITY ==================== //
// TODO/TEMP: Give investors even more priority
let mut is_investor = false;
// TODO/TEMP: The storyteller.io website will redirect and establish this cookie.
// This is just for the YCombinator demo.
if request_has_demo_cookie(&http_request) {
is_investor = true;
}
if is_investor {
priority_level = FAKEYOU_INVESTOR_PRIORITY_LEVEL;
}
// ==================== ENGINEERING FLAGS / HEADERS: DEBUG MODE, ETC. ==================== //
let is_debug_request = get_request_header_optional(&http_request, DEBUG_HEADER_NAME)
.is_some();
let maybe_routing_tag=
get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME)
.map(|routing_tag| routing_tag.trim().to_string());
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.role.is_banned {
warn!("User is not authorized to use Seed-VC because they are banned.");
return Err(CommonWebError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
if !disable_rate_limiter {
let maybe_username = maybe_user_session
.as_ref()
.map(|session| session.user.username.as_str());
// FIXME(bt,2023-12-13): These boolean flags are a bit dangerous.
let rate_limiter_type = get_rate_limiter_type(
maybe_username,
&server_state.redis_rate_limiters.api_ai_streamer_username_set,
is_investor,
is_from_api,
use_high_priority_rate_limiter
);
let rate_limiter = match rate_limiter_type {
RateLimiterType::LoggedOut => &server_state.redis_rate_limiters.logged_out,
RateLimiterType::LoggedIn => &server_state.redis_rate_limiters.logged_in,
RateLimiterType::ApiHighPriority => {
info!("Using API high priority rate limiter");
&server_state.redis_rate_limiters.api_high_priority
},
RateLimiterType::ApiAiStreamer => {
info!("Using AI streamer rate limiter");
&server_state.redis_rate_limiters.api_ai_streamers
},
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
}
let ip_address = get_request_ip(&http_request);
let maybe_user_preferred_visibility : Option<Visibility> = maybe_user_session
.as_ref()
.map(|user_session| user_session.preferences.preferred_tts_result_visibility);
let set_visibility = request.creator_set_visibility
.or(maybe_user_preferred_visibility)
.unwrap_or(Visibility::Public);
let maybe_creator_user_token_typed = maybe_user_token
.as_deref()
.map(|token| UserToken::new_from_str(token));
let maybe_avt_token = server_state
.avt_cookie_manager
.get_avt_token_from_request(&http_request);
let maybe_model_type = None;
let source_media_token = request.source_media_file_token.clone();
let source_media_token_type =
if source_media_token.starts_with(MediaUploadToken::token_prefix()) {
InferenceInputSourceTokenType::MediaUpload
} else if source_media_token.starts_with(MediaFileToken::token_prefix()) {
InferenceInputSourceTokenType::MediaFile
} else {
return Err(CommonWebError::BadInputWithSimpleMessage(
"input token is not a media_upload or media_file token".to_string()));
};
let reference_media_token = request.reference_media_file_token.clone();
let _reference_media_token_type =
if reference_media_token.starts_with(MediaUploadToken::token_prefix()) {
InferenceInputSourceTokenType::MediaUpload
} else if reference_media_token.starts_with(MediaFileToken::token_prefix()) {
InferenceInputSourceTokenType::MediaFile
} else {
return Err(CommonWebError::BadInputWithSimpleMessage(
"reference token is not a media_upload or media_file token".to_string()));
};
let maybe_inference_args = Some(GenericInferenceArgs{
inference_category: Some(InferenceCategoryAbbreviated::SeedVc),
args: Some(PolymorphicInferenceArgs::Sv(SeedVcPayload{
reference_media_file_token: Option::from(MediaFileToken::new_from_str(&reference_media_token)),
})),
});
let maybe_product_category = Some(InferenceJobProductCategory::VcSeedVc);
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::SeedVc,
maybe_product_category,
inference_category: InferenceCategory::SeedVc,
maybe_model_type,
maybe_model_token: None,
maybe_input_source_token: Some(&source_media_token),
maybe_input_source_token_type: Some(source_media_token_type),
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: None,
maybe_max_duration_seconds: Some(30),
maybe_inference_args,
maybe_creator_user_token: maybe_creator_user_token_typed.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: set_visibility,
priority_level,
requires_keepalive: false, // TODO: We may want this to be the case in the future.
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let inference_job_token = match query_result {
Ok((inference_job_token, _id)) => inference_job_token,
Err(err) => {
warn!("New (generic) seed-vc inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(CommonWebError::BadInputWithSimpleMessage("Duplicate idempotency token".to_string()));
}
return Err(CommonWebError::from_error(err));
}
};
let job_token = inference_job_token.to_string();
Ok(Json(InferSeedVcSuccessResponse {
success: true,
inference_job_token: job_token,
}))
}
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum RateLimiterType {
LoggedOut,
LoggedIn,
ApiHighPriority,
ApiAiStreamer,
}
// FIXME(bt,2023-12-13): The boolean flags and disjoint decision making are kind of terrible.
fn get_rate_limiter_type(
maybe_username: Option<&str>,
ai_streamer_username_set: &UsernameSet,
is_investor: bool,
is_from_api: bool,
use_high_priority_rate_limiter: bool,
) -> RateLimiterType {
// For AI live streamers
// https://www.notion.so/storytellerai/dd88609558a24196b4ddeeef6079da98
let mut is_ai_streamer = false;
let mut rate_limiter = match maybe_username {
None => RateLimiterType::LoggedOut,
Some(username) => {
if ai_streamer_username_set.username_is_in_set(username) {
is_ai_streamer = true;
}
RateLimiterType::LoggedIn
},
};
// TODO/TEMP
if is_investor || is_from_api {
rate_limiter = RateLimiterType::LoggedIn;
}
// TODO: This is for VidVoice.ai and should be replaced with per-API consumer rate limiters
if use_high_priority_rate_limiter {
rate_limiter = RateLimiterType::ApiHighPriority;
}
if is_ai_streamer {
rate_limiter = RateLimiterType::ApiAiStreamer;
}
rate_limiter
}
@@ -0,0 +1,423 @@
// NB: Incrementally getting rid of build warnings...
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, info, warn};
use redis::Commands;
use sqlx::pool::PoolConnection;
use sqlx::MySql;
use utoipa::ToSchema;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_input_source_token_type::InferenceInputSourceTokenType;
use enums::by_table::generic_inference_jobs::inference_job_product_category::InferenceJobProductCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::by_table::generic_inference_jobs::inference_model_type::InferenceModelType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use migration::voice_conversion::query_vc_model_info_lite_for_migration::query_vc_model_info_lite_for_migration_with_connection;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{FundamentalFrequencyMethodForJob, GenericInferenceArgs, InferenceCategoryAbbreviated, PolymorphicInferenceArgs};
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{insert_generic_inference_job, InsertGenericInferenceArgs};
use redis_common::redis_keys::RedisKeys;
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::media_uploads::MediaUploadToken;
use tokens::tokens::users::UserToken;
use tts_common::priority::FAKEYOU_INVESTOR_PRIORITY_LEVEL;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::deprecated_endpoints::investor_demo::demo_cookie::request_has_demo_cookie;
use crate::state::memory_cache::model_token_to_info_cache::{ModelInfoForInferenceJob, ModelTokenToInfoCache};
use crate::state::server_state::ServerState;
/// Debug requests can get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
const DEBUG_HEADER_NAME : &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
const ROUTING_TAG_HEADER_NAME : &str = "routing-tag";
#[derive(Deserialize, ToSchema)]
pub struct EnqueueVoiceConversionInferenceRequest {
uuid_idempotency_token: String,
/// Model weights token for the voice conversion model.
/// This must be either a SoVitsSvc or RvcV2 model.
voice_conversion_model_token: String,
/// NB: This can be a `MediaUploadToken` or a `MediaFileToken` token.
/// We will eventually migrate everything to media files and remove
/// media uploads (legacy, deprecated).
source_media_upload_token: String,
creator_set_visibility: Option<Visibility>,
is_storyteller_demo: Option<bool>,
/// Argument for so-vits-svc
/// The python model defaults to true, but that sounds awful,
/// so we default to false unless specified.
auto_predict_f0: Option<bool>,
/// Argument for so-vits-svc
/// f0 estimation
override_f0_method: Option<FundamentalFrequencyMethod>,
/// Argument for so-vits-svc
/// Pitch controls for so-vits-svc
transpose: Option<i32>,
}
#[derive(Deserialize, Clone, Copy, ToSchema)]
pub enum FundamentalFrequencyMethod {
/// RMVPE is the best algorithm as of 2023-10-10.
#[serde(rename = "rmvpe")]
Rmvpe,
#[serde(rename = "crepe")]
Crepe,
#[serde(rename = "dio")]
Dio,
#[serde(rename = "harvest")]
Harvest,
}
#[derive(Serialize, ToSchema)]
pub struct EnqueueVoiceConversionInferenceSuccessResponse {
pub success: bool,
pub inference_job_token: InferenceJobToken,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
#[utoipa::path(
post,
tag = "Voice Conversion",
path = "/v1/voice_conversion/inference",
responses(
(status = 200, description = "Success", body = EnqueueVoiceConversionInferenceSuccessResponse),
(status = 400, description = "Bad input", body = CommonWebError),
(status = 401, description = "Not authorized", body = CommonWebError),
(status = 429, description = "Rate limited", body = CommonWebError),
(status = 500, description = "Server error", body = CommonWebError),
),
params(
("request" = EnqueueVoiceConversionInferenceRequest, description = "Payload for Request"),
)
)]
pub async fn enqueue_voice_conversion_inference_handler(
http_request: HttpRequest,
request: Json<EnqueueVoiceConversionInferenceRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<EnqueueVoiceConversionInferenceSuccessResponse>, CommonWebError>
{
if server_state.flags.disable_voice_conversion {
return Err(CommonWebError::TooManyRequests);
}
let mut maybe_user_token : Option<UserToken> = None;
let mut priority_level ;
let disable_rate_limiter = false; // NB: Careful!
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
warn!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
// ==================== AVT TOKEN ==================== //
let maybe_avt_token = server_state.avt_cookie_manager.get_avt_token_from_request(&http_request);
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(UserToken::new_from_str(&user_session.user_token));
}
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref());
priority_level = plan.web_vc_base_priority_level();
// ==================== INVESTOR PRIORITY ==================== //
// TODO/TEMP: Give investors even more priority
let mut is_investor = false;
{
// TODO/TEMP: The storyteller.io website's AJAX calls will set this.
// This is just for the YCombinator demo.
match request.is_storyteller_demo {
Some(true) => {
is_investor = true;
},
_ => {},
};
// TODO/TEMP: The storyteller.io website will redirect and establish this cookie.
// This is just for the YCombinator demo.
if request_has_demo_cookie(&http_request) {
is_investor = true;
}
if is_investor {
priority_level = FAKEYOU_INVESTOR_PRIORITY_LEVEL;
}
}
// ==================== DEBUG MODE + ROUTING TAG ==================== //
let is_debug_request =
get_request_header_optional(&http_request, DEBUG_HEADER_NAME)
.is_some();
let maybe_routing_tag=
get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME)
.map(|routing_tag| routing_tag.trim().to_string());
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.role.is_banned {
return Err(CommonWebError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
if !disable_rate_limiter {
let mut rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.logged_out,
Some(ref _session) => &server_state.redis_rate_limiters.logged_in
};
// TODO/TEMP
if is_investor {
rate_limiter = &server_state.redis_rate_limiters.logged_in;
}
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
}
// ==================== LOOK UP MODEL INFO ==================== //
// TODO(bt): CHECK DATABASE!
let model_token = request.voice_conversion_model_token.clone();
let model_inference_info = lookup_model_info(
&model_token,
&server_state.caches.durable.model_token_info,
&mut mysql_connection
).await?;
// ==================== HANDLE AUDIO INPUT ==================== //
let media_token = request.source_media_upload_token.clone();
let media_token_type =
if media_token.starts_with(MediaUploadToken::token_prefix()) {
InferenceInputSourceTokenType::MediaUpload
} else if media_token.starts_with(MediaFileToken::token_prefix()) {
InferenceInputSourceTokenType::MediaFile
} else {
return Err(CommonWebError::BadInputWithSimpleMessage(
"input token is not a media_upload or media_file token".to_string()));
};
// ==================== CHECK AND ENQUEUE VOICE CONVERSION ==================== //
let mut redis = server_state.redis_pool
.get()
.map_err(|e| {
error!("redis error: {:?}", e);
CommonWebError::from_error(e)
})?;
let redis_count_key = RedisKeys::web_vc_model_usage_count(&model_token);
redis.incr::<_, _, ()>(&redis_count_key, 1)
.map_err(|e| {
warn!("redis error: {:?}", e);
CommonWebError::from_error(e)
})?;
let ip_address = get_request_ip(&http_request);
let maybe_user_preferred_visibility : Option<Visibility> = maybe_user_session
.as_ref()
.map(|user_session| user_session.preferences.preferred_tts_result_visibility); // TODO: New setting for web-vc
let set_visibility = request.creator_set_visibility
.or(maybe_user_preferred_visibility)
.unwrap_or(Visibility::Public);
info!("Creating voice conversion inference job record...");
let mut maybe_args = None;
let set_args = request.auto_predict_f0.is_some()
|| request.transpose.is_some()
|| request.override_f0_method.is_some();
if set_args {
maybe_args = Some(PolymorphicInferenceArgs::Vc {
auto_predict_f0: request.auto_predict_f0,
override_f0_method: request.override_f0_method.map(|f| match f {
FundamentalFrequencyMethod::Crepe => FundamentalFrequencyMethodForJob::Crepe,
FundamentalFrequencyMethod::Dio => FundamentalFrequencyMethodForJob::Dio,
FundamentalFrequencyMethod::Harvest => FundamentalFrequencyMethodForJob::Harvest,
FundamentalFrequencyMethod::Rmvpe => FundamentalFrequencyMethodForJob::Rmvpe,
}),
transpose: request.transpose,
});
}
let job_type = match model_inference_info.job_model_type {
InferenceModelType::SoVitsSvc => InferenceJobType::SoVitsSvc,
InferenceModelType::RvcV2 => InferenceJobType::RvcV2,
_ => {
// In theory, this shouldn't catch anything.
error!("wrong model type for voice conversion: {:?}", model_inference_info.job_model_type);
return Err(CommonWebError::server_error_with_message("wrong model type for voice conversion"))
}
};
let maybe_product_category = match model_inference_info.job_model_type {
InferenceModelType::RvcV2 => Some(InferenceJobProductCategory::VcRvc2),
InferenceModelType::SoVitsSvc => Some(InferenceJobProductCategory::VcSvc),
_ => None,
};
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type,
maybe_product_category,
inference_category: InferenceCategory::VoiceConversion,
maybe_model_type: Some(model_inference_info.job_model_type),
maybe_model_token: Some(&model_token),
maybe_input_source_token: Some(&media_token),
maybe_input_source_token_type: Some(media_token_type),
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: None, // NB: Voice conversion isn't TTS, so there's no text.
maybe_max_duration_seconds: None,
maybe_inference_args: Some(GenericInferenceArgs {
inference_category: Some(InferenceCategoryAbbreviated::VoiceConversion),
args: maybe_args,
}),
maybe_creator_user_token: maybe_user_token.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: set_visibility,
priority_level,
requires_keepalive: plan.web_vc_requires_frontend_keepalive(),
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
Err(err) => {
warn!("New generic inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(CommonWebError::BadInputWithSimpleMessage("Duplicate idempotency token".to_string()));
}
return Err(CommonWebError::from_error(err));
}
};
server_state.firehose_publisher.enqueue_vc_inference(
maybe_user_token.as_ref(),
&job_token)
.await
.map_err(|e| {
warn!("error publishing event: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
Ok(Json(EnqueueVoiceConversionInferenceSuccessResponse {
success: true,
inference_job_token: job_token,
}))
}
async fn lookup_model_info(
model_token: &str,
cache: &ModelTokenToInfoCache,
mysql_connection: &mut PoolConnection<MySql>,
) -> Result<ModelInfoForInferenceJob, CommonWebError> {
let maybe_model_inference_info = cache
.get_info(model_token)
.map_err(|err| {
error!("in-memory cache error: {:?}", err);
CommonWebError::from_anyhow_error(err)
})?;
if let Some(inference_info) = maybe_model_inference_info {
return Ok(inference_info);
}
let result = query_vc_model_info_lite_for_migration_with_connection(
model_token, mysql_connection)
.await
.map_err(|err| {
error!("model lookup error: {:?}", err);
CommonWebError::from_anyhow_error(err)
})?;
let model_info = match result {
Some(model_info) => model_info,
None => {
warn!("model not found for {model_token}; failing open");
// NB: Fail open for now.
return Ok(ModelInfoForInferenceJob {
job_inference_category: InferenceCategory::VoiceConversion,
job_model_type: InferenceModelType::SoVitsSvc,
});
}
};
let inference_info = match (model_info.get_inference_category(), model_info.get_inference_model_type()) {
(Some(inference_category), Some(model_type)) => {
ModelInfoForInferenceJob {
job_inference_category: inference_category,
job_model_type: model_type,
}
}
_ => {
error!("wrong model type for inference: {:?}", model_info);
return Err(CommonWebError::BadInputWithSimpleMessage("wrong model type for inference".to_string()));
}
};
cache.insert_one(model_token, &inference_info)
.map_err(|err| {
error!("cache insertion error: {:?}", err);
CommonWebError::from_anyhow_error(err)
})?;
Ok(inference_info)
}
@@ -0,0 +1,218 @@
// NB: Incrementally getting rid of build warnings...
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::{web, HttpRequest, HttpResponse};
use chrono::{DateTime, Utc};
use lexical_sort::natural_lexical_cmp;
use log::{debug, error, warn};
use sqlx::pool::PoolConnection;
use sqlx::MySql;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use enums::by_table::voice_conversion_models::voice_conversion_model_type::VoiceConversionModelType;
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use migration::voice_conversion::list_vc_models_for_migration::list_vc_models_for_migration;
use mysql_queries::queries::users::user_sessions::get_user_session_by_token::SessionUserRecord;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct VoiceConversionModel {
/// NB: token can be a `model_weights` token or a `voice_conversion_models` token.
pub token: String,
pub model_type: VoiceConversionModelType,
pub title: String,
pub creator: UserDetailsLight,
pub creator_set_visibility: Visibility,
pub ietf_language_tag: String,
pub ietf_primary_language_subtag: String,
pub is_front_page_featured: bool,
// TODO: Add user ratings.
//pub user_ratings: UserRatingsStats,
// TODO: Add categories.
///// Category assignments
///// From non-deleted, mod-approved categories only
//pub category_tokens: HashSet<String>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
// TODO: Add user ratings.
//#[derive(Serialize, Clone)]
//pub struct UserRatingsStats {
// pub positive_count: u32,
// pub negative_count: u32,
// /// Total count does not take into account "neutral" ratings.
// pub total_count: u32,
//}
#[derive(Serialize)]
pub struct ListVoiceConversionModelsSuccessResponse {
pub success: bool,
pub models: Vec<VoiceConversionModel>,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
pub async fn list_voice_conversion_models_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>) -> Result<HttpResponse, CommonWebError>
{
if server_state.flags.disable_voice_conversion_model_list_endpoint {
// NB: Despite the cache being a powerful protector of the database (this is an expensive query),
// if the cache goes stale during an outage, there is no protection. This feature flag lets us
// shut off all traffic to the endpoint.
return render_response_busy(ListVoiceConversionModelsSuccessResponse {
success: true,
models: Vec::new(),
});
}
let maybe_models = server_state.caches.ephemeral.voice_conversion_model_list.grab_copy_without_bump_if_unexpired()
.map_err(|e| {
error!("Error consulting cache: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
// NB: We don't know if we need a MySQL connection, so don't grab one unless we do.
let mut maybe_mysql_connection = None;
let models = match maybe_models {
Some(models) => {
debug!("Serving voice conversion models from cache");
models
},
None => {
debug!("Populating voice conversion models from database");
let mut mysql_connection = server_state.mysql_pool.acquire()
.await
.map_err(|e| {
warn!("Could not acquire DB pool: {:?}", e);
CommonWebError::from_error(e)
})?;
// TODO: Fail open in case the DB is down. Pull from expired cache if query fails.
let models = get_all_models(&mut mysql_connection, true)
.await
.map_err(|e| {
error!("Error querying database: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
maybe_mysql_connection = Some(mysql_connection);
server_state.caches.ephemeral.voice_conversion_model_list.store_copy(&models)
.map_err(|e| {
error!("Error storing cache: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
models
},
};
let maybe_user_session : Option<SessionUserRecord> = match maybe_mysql_connection {
None => {
server_state.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
}
Some(mut mysql_connection) => {
server_state.session_checker
.maybe_get_user_session_from_connection(&http_request, &mut mysql_connection)
.await
}
}.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let maybe_session_user_token = maybe_user_session
.as_ref()
.map(|s| s.user_token.as_str());
let models = models.into_iter()
.filter(|model| {
match model.creator_set_visibility {
Visibility::Public => true,
Visibility::Hidden | Visibility::Private => maybe_session_user_token
.map(|token| token == model.creator.user_token.as_str())
.unwrap_or(false),
}
})
.collect();
render_response_ok(ListVoiceConversionModelsSuccessResponse {
success: true,
models,
})
}
pub fn render_response_busy(response: ListVoiceConversionModelsSuccessResponse) -> Result<HttpResponse, CommonWebError> {
let body = render_response_payload(response)?;
Ok(HttpResponse::TooManyRequests()
.content_type("application/json")
.body(body))
}
pub fn render_response_ok(response: ListVoiceConversionModelsSuccessResponse) -> Result<HttpResponse, CommonWebError> {
let body = render_response_payload(response)?;
Ok(HttpResponse::Ok()
.content_type("application/json")
.body(body))
}
pub fn render_response_payload(response: ListVoiceConversionModelsSuccessResponse) -> Result<String, CommonWebError> {
let body = serde_json::to_string(&response)
.map_err(|e| {
error!("error returning response: {:?}", e);
CommonWebError::from_error(e)
})?;
Ok(body)
}
async fn get_all_models(mysql_connection: &mut PoolConnection<MySql>, use_weights_table: bool) -> AnyhowResult<Vec<VoiceConversionModel>> {
let mut models = list_vc_models_for_migration(mysql_connection, use_weights_table).await?;
// NB: Make the list nice for human readers.
models.sort_by(|a, b|
natural_lexical_cmp(&a.title(), &b.title()));
let models_for_response = models.into_iter()
.filter(|model| model.is_voice_conversion_model())
.map(|model| {
// TODO(bt,2023-12-18): All of these clones are lame, but this is just to support the migration.
VoiceConversionModel {
token: model.token().to_string(),
model_type: model.legacy_voice_conversion_model_type()
.unwrap_or(VoiceConversionModelType::SoVitsSvc),
title: model.title().to_string(),
is_front_page_featured: false, // TODO(bt,2023-12-18): None of the models are "featured" yet.
creator: UserDetailsLight::from_db_fields(
model.creator_user_token(),
model.creator_username(),
model.creator_display_name(),
model.creator_gravatar_hash(),
),
creator_set_visibility: model.creator_set_visibility(),
ietf_language_tag: model.ietf_language_tag().to_string(),
ietf_primary_language_subtag: model.ietf_primary_language_subtag().to_string(),
created_at: *model.created_at(),
updated_at: *model.updated_at(),
}
})
.collect::<Vec<VoiceConversionModel>>();
Ok(models_for_response)
}
@@ -0,0 +1,3 @@
pub mod enqueue_voice_conversion_inference_handler;
pub mod list_voice_conversion_models_handler;
pub mod enqueue_seed_vc_inference_handler;
@@ -0,0 +1,199 @@
#![forbid(unused_imports)]
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use std::sync::Arc;
use actix_web::{web, HttpRequest};
use actix_web::web::Json;
use log::warn;
use serde::Deserialize;
use serde::Serialize;
use utoipa::ToSchema;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_job_product_category::InferenceJobProductCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::by_table::generic_inference_jobs::inference_model_type::InferenceModelType;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{
GenericInferenceArgs,
InferenceCategoryAbbreviated,
PolymorphicInferenceArgs,
};
use mysql_queries::payloads::generic_inference_args::inner_payloads::tts_payload::TTSArgs;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{
insert_generic_inference_job,
InsertGenericInferenceArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::users::UserToken;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
/// Debug requests can get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
const DEBUG_HEADER_NAME: &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
const ROUTING_TAG_HEADER_NAME: &str = "routing-tag";
#[derive(Deserialize,ToSchema)]
pub struct EnqueueTTSRequest {
uuid_idempotency_token: String,
text: String,
voice_token: String,
}
#[derive(Serialize,ToSchema)]
pub struct EnqueueTTSRequestSuccessResponse {
pub success: bool,
pub inference_job_token: InferenceJobToken,
}
// Implementation for enqueuing a TTS request
// Reference enqueue_infer_tts_handler.rs for checks: rate limiting / user sessions
// insert generic inference job.rs
// Need to convert it to generic inference job.
#[utoipa::path(
post,
tag = "Voice Designer",
path = "/v1/voice_designer/inference/enqueue_tts",
responses(
(status = 200, description = "Enqueue TTS generically", body = EnqueueTTSRequestSuccessResponse),
(status = 400, description = "Bad input", body = CommonWebError),
(status = 401, description = "Not authorized", body = CommonWebError),
(status = 429, description = "Rate limited", body = CommonWebError),
(status = 500, description = "Server error", body = CommonWebError)
),
params(
("request" = EnqueueTTSRequest, description = "Payload for TTS Request")
)
)]
pub async fn enqueue_tts_request(
http_request: HttpRequest,
request: Json<EnqueueTTSRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<EnqueueTTSRequestSuccessResponse>, CommonWebError> {
let mut maybe_user_token: Option<UserToken> = None;
let mut mysql_connection = server_state.mysql_pool.acquire().await.map_err(|err| {
warn!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection).await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(UserToken::new_from_str(&user_session.user_token));
}
// Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref()
);
// Separate priority for animation.
let priority_level = plan.web_vc_base_priority_level();
// ==================== DEBUG MODE + ROUTING TAG ==================== //
let is_debug_request = get_request_header_optional(&http_request, DEBUG_HEADER_NAME).is_some();
let maybe_routing_tag = get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME).map(
|routing_tag| routing_tag.trim().to_string()
);
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.role.is_banned {
return Err(CommonWebError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
let rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.logged_out,
Some(ref _user) => &server_state.redis_rate_limiters.logged_in,
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
// Get up IP address
let ip_address = get_request_ip(&http_request);
// package as larger component args should always have an embedding token ..
let inference_args = TTSArgs {
voice_token: Some(request.voice_token.clone()),
dataset_token: None,
};
// create the inference args here
// enqueue a zero shot tts request here...
let maybe_avt_token = server_state.avt_cookie_manager.get_avt_token_from_request(&http_request);
// create the job record here!
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::StyleTTS2,
maybe_product_category: Some(InferenceJobProductCategory::TtsStyleTts2),
inference_category: InferenceCategory::TextToSpeech,
maybe_model_type: Some(InferenceModelType::StyleTTS2), // NB: Model is static during inference
maybe_model_token: None, // NB: Model is static during inference
maybe_input_source_token: None,
maybe_input_source_token_type: None,
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: Some(&request.text),
maybe_max_duration_seconds: None,
maybe_inference_args: Some(GenericInferenceArgs {
inference_category: Some(InferenceCategoryAbbreviated::TextToSpeech),
args: Some(PolymorphicInferenceArgs::Tts(inference_args)),
}),
maybe_creator_user_token: maybe_user_token.as_ref(),
maybe_avt_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: Visibility::Public,
priority_level,
requires_keepalive: true, // do we need this? I think so
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
Err(err) => {
warn!("New generic inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(CommonWebError::BadInputWithSimpleMessage("Duplicate idempotency token".to_string()));
}
return Err(CommonWebError::from_error(err));
}
};
let response: EnqueueTTSRequestSuccessResponse = EnqueueTTSRequestSuccessResponse {
success: true,
inference_job_token: job_token,
};
Ok(Json(response))
}
@@ -0,0 +1,6 @@
use actix_web::HttpResponse;
pub async fn enqueue_vc_request() -> HttpResponse {
// Implementation for enqueuing a VC request
HttpResponse::Ok().json("VC request enqueued successfully")
}
@@ -0,0 +1,2 @@
pub mod enqueue_tts_request;
pub mod enqueue_vc_request;
@@ -0,0 +1,4 @@
pub mod inference;
pub mod voice_dataset_samples;
pub mod voice_datasets;
pub mod voices;
@@ -0,0 +1,153 @@
{
"info": {
"name": "Voice Designer API",
"description": "Endpoints for the Voice Designer API",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "Voice Designer - Datasets",
"item": [
{
"name": "Create Dataset",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/datasets/create"
}
},
{
"name": "Update Dataset",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/datasets/{{dataset_token}}/update"
}
},
{
"name": "Delete Dataset",
"request": {
"method": "DELETE",
"url": "{{base_url}}/v1/voice_designer/datasets/{{dataset_token}}/delete"
}
},
{
"name": "List Datasets By User",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/datasets/user/{{user_token}}/list"
}
}
]
},
{
"name": "Voice Designer - Voice",
"item": [
{
"name": "Create Voice",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/voice/create"
}
},
{
"name": "Update Voice",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/voice/{{voice_token}}/update"
}
},
{
"name": "Delete Voice",
"request": {
"method": "DELETE",
"url": "{{base_url}}/v1/voice_designer/voice/{{voice_token}}/delete"
}
},
{
"name": "List Voices By User",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/voice/user/{{user_token}}/list"
}
}
]
},
{
"name": "Voice Designer - Sample",
"item": [
{
"name": "Upload Sample",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/sample/upload"
}
},
{
"name": "Delete Sample",
"request": {
"method": "DELETE",
"url": "{{base_url}}/v1/voice_designer/sample/delete"
}
},
{
"name": "List Samples By Dataset",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/sample/data_set/{{data_set_token}}/list"
}
}
]
},
{
"name": "Voice Designer - Inference",
"item": [
{
"name": "Enqueue TTS Request",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/inference/enqueue_tts"
}
},
{
"name": "Enqueue VC Request",
"request": {
"method": "POST",
"url": "{{base_url}}/v1/voice_designer/inference/enqueue_vc"
}
}
]
},
{
"name": "Voice Designer - Inventory",
"item": [
{
"name": "List Available Voices",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/inventory/list"
}
},
{
"name": "List User Models",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/inventory/user_list"
}
},
{
"name": "List Favorite Models",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/inventory/favorites_list"
}
},
{
"name": "Search Voices",
"request": {
"method": "GET",
"url": "{{base_url}}/v1/voice_designer/inventory/search"
}
}
]
}
]
}
@@ -0,0 +1,125 @@
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::warn;
use artcraft_api_defs::common::responses::simple_generic_json_success::SimpleGenericJsonSuccess;
use mysql_queries::queries::voice_designer::voice_samples::delete_sample::{delete_sample_as_mod, delete_sample_as_user, undelete_sample_as_mod, undelete_sample_as_user};
use mysql_queries::queries::voice_designer::voice_samples::get_dataset_sample::get_dataset_sample_by_token;
use tokens::tokens::zs_voice_dataset_samples::ZsVoiceDatasetSampleToken;
use crate::state::server_state::ServerState;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::util::delete_role_disambiguation::{delete_role_disambiguation, DeleteRole};
// TODO(bt,2023-10-10): This is way too much boilerplate.
/// For the URL PathInfo
#[derive(Deserialize)]
pub struct DeleteSamplePathInfo {
sample_token: ZsVoiceDatasetSampleToken,
}
#[derive(Deserialize)]
pub struct DeleteSampleRequest {
set_delete: bool,
/// NB: this is only to disambiguate when a user is both a mod and an author.
as_mod: Option<bool>,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
pub async fn delete_sample_handler(
http_request: HttpRequest,
path: Path<DeleteSamplePathInfo>,
request: Json<DeleteSampleRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<SimpleGenericJsonSuccess>, CommonWebError> {
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
// NB: First permission check.
// Only mods should see deleted models (both user_* and mod_* deleted).
let is_mod_that_can_see_deleted = user_session.can_delete_other_users_tts_results;
let inference_result_query_result = get_dataset_sample_by_token(
&path.sample_token,
is_mod_that_can_see_deleted,
&server_state.mysql_pool,
).await;
let dataset_sample = match inference_result_query_result {
Err(e) => {
warn!("query error: {:?}", e);
return Err(CommonWebError::from_anyhow_error(e));
}
Ok(None) => return Err(CommonWebError::NotFound),
Ok(Some(sample)) => sample,
};
// NB: Second set of permission checks
let is_author = dataset_sample.maybe_creator_user_token
.as_ref()
.map(|creator_token| creator_token == &user_session.user_token)
.unwrap_or(false);
let is_mod = user_session.can_delete_other_users_tts_results;
if !is_author && !is_mod {
warn!("user is not allowed to delete samples: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let delete_role = delete_role_disambiguation(is_mod, is_author, request.as_mod);
let query_result = if request.set_delete {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
delete_sample_as_user(&path.sample_token, &server_state.mysql_pool).await
}
DeleteRole::AsMod => {
delete_sample_as_mod(&path.sample_token, &user_session.user_token, &server_state.mysql_pool).await
}
}
} else {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
undelete_sample_as_user(&path.sample_token, &server_state.mysql_pool).await
}
DeleteRole::AsMod => {
undelete_sample_as_mod(&path.sample_token, &user_session.user_token, &server_state.mysql_pool).await
}
}
};
match query_result {
Ok(_) => {},
Err(err) => {
warn!("Delete DB error: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
Ok(Json(SimpleGenericJsonSuccess { success: true }))
}
@@ -0,0 +1,111 @@
use std::sync::Arc;
use actix_web::{web, HttpRequest};
use actix_web::web::Json;
use chrono::{DateTime, Utc};
use log::warn;
use enums::by_table::media_files::media_file_type::MediaFileType;
use mysql_queries::queries::voice_designer::voice_samples::list_dataset_samples_for_dataset_token::list_dataset_samples_for_dataset_token;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
use tokens::tokens::zs_voice_dataset_samples::ZsVoiceDatasetSampleToken;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::state::server_state::ServerState;
use crate::http_server::common_responses::common_web_error::CommonWebError;
#[derive(Serialize, Clone)]
pub struct ZsSampleRecordForResponse {
sample_token: ZsVoiceDatasetSampleToken,
media_file_token: MediaFileToken,
media_type: MediaFileType,
public_bucket_directory_hash: String,
maybe_public_bucket_prefix: Option<String>,
maybe_public_bucket_extension: Option<String>,
maybe_creator_user_token: Option<UserToken>,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Serialize)]
pub struct ListSamplesByDatasetSuccessResponse {
pub success: bool,
pub samples: Vec<ZsSampleRecordForResponse>,
}
#[derive(Deserialize)]
pub struct ListSamplesByDatasetPathInfo {
dataset_token: String,
}
pub async fn list_samples_by_dataset_handler(
http_request: HttpRequest,
path: web::Path<ListSamplesByDatasetPathInfo>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListSamplesByDatasetSuccessResponse>, CommonWebError> {
let maybe_user_session = server_state.session_checker.maybe_get_user_session(
&http_request,
&server_state.mysql_pool
).await.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
},
};
let dataset_token = ZsVoiceDatasetToken(path.dataset_token.clone());
let _is_mod = user_session.can_ban_users;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
// We should focus on visibility-related controls instead.
const CAN_SEE_DELETED : bool = false;
let query_results = list_dataset_samples_for_dataset_token(
&dataset_token,
CAN_SEE_DELETED,
&server_state.mysql_pool
).await.map_err(|e| {
warn!("list_samples error: {:?}", e);
CommonWebError::from_anyhow_error(e)
});
let samples = match query_results {
Ok(samples) => samples,
Err(e) => {
warn!("list_samples error: {:?}", e);
return Err(CommonWebError::from_error(e));
},
};
let samples = samples
.into_iter()
.map(|sample| ZsSampleRecordForResponse {
sample_token: sample.sample_token,
media_file_token: sample.media_file_token,
media_type: sample.media_type,
public_bucket_directory_hash: sample.public_bucket_directory_hash,
maybe_public_bucket_prefix: sample.maybe_public_bucket_prefix,
maybe_public_bucket_extension: sample.maybe_public_bucket_extension,
maybe_creator_user_token: sample.maybe_creator_user_token,
created_at: sample.created_at,
updated_at: sample.updated_at,
})
.collect();
let response = ListSamplesByDatasetSuccessResponse {
success: true,
samples
};
Ok(Json(response))
}
@@ -0,0 +1,3 @@
pub mod delete_sample;
pub mod upload_zs_sample;
pub mod list_samples_by_dataset;
@@ -0,0 +1,363 @@
use std::collections::HashSet;
use std::sync::Arc;
use actix_multipart::Multipart;
use actix_web::{http::StatusCode, web, HttpRequest, HttpResponse, ResponseError};
use log::{error, info, warn};
use once_cell::sync::Lazy;
use bucket_paths::legacy::typified_paths::public::media_files::bucket_file_path::MediaFileBucketPath;
use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory;
use enums::by_table::media_uploads::media_upload_type::MediaUploadType;
use enums::common::visibility::Visibility;
use hashing::sha256::sha256_hash_bytes::sha256_hash_bytes;
use http_server_common::request::get_request_ip::get_request_ip;
use http_server_common::response::serialize_as_json_error::serialize_as_json_error;
use media::decode_basic_audio_info::decode_basic_audio_bytes_info;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mimetypes::mimetype_to_extension::mimetype_to_extension;
use mysql_queries::queries::voice_designer::voice_samples::get_dataset_sample_by_uuid::get_dataset_sample_by_uuid_with_connection;
use mysql_queries::queries::voice_designer::voice_samples::insert_dataset_sample_and_media_file::{insert_dataset_sample_and_media_file, InsertDatasetSampleAndMediaFileArgs};
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::zs_voice_dataset_samples::ZsVoiceDatasetSampleToken;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::http_server::deprecated_endpoints::media_uploads::common::drain_multipart_request::{drain_multipart_request, MediaSource};
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::http_server::validations::validate_idempotency_token_format::validate_idempotency_token_format;
use crate::state::server_state::ServerState;
#[derive(Serialize)]
pub struct UploadSampleResponse {
pub success: bool,
pub sample_token: ZsVoiceDatasetSampleToken,
pub media_file_token: MediaFileToken,
}
// NB: Not using derive_more::Display since Clion doesn't understand it.
static ALLOWED_AUDIO_MIME_TYPES : Lazy<HashSet<&'static str>> = Lazy::new(|| {
HashSet::from([
"audio/aac",
"audio/m4a",
"audio/mpeg",
"audio/ogg",
"audio/opus",
"audio/x-flac",
"audio/x-wav",
// NB(bt,2023-10-13): This is the only way to allow browser recording.
// https://air.ghost.io/recording-to-an-audio-file-using-html5-and-js/
// Chrome: "audio/webm;codecs=opus"
// Firefox: "audio/ogg;codecs=opus"
"audio/webm",
"video/webm",
])
});
pub async fn upload_zs_sample_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>,
mut multipart_payload: Multipart,
) -> Result<HttpResponse, CommonWebError> {
let mut mysql_connection = server_state.mysql_pool
.acquire()
.await
.map_err(|err| {
error!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
// ==================== READ SESSION ==================== //
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session_from_connection(&http_request, &mut mysql_connection)
.await
.map_err(|e| {
error!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.is_banned {
return Err(CommonWebError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
let rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.file_upload_logged_out,
Some(ref _user) => &server_state.redis_rate_limiters.file_upload_logged_in,
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
if let Err(_err) = server_state.redis_rate_limiters.model_upload.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
// ==================== READ MULTIPART REQUEST ==================== //
let upload_sample_request = drain_multipart_request(multipart_payload)
.await
.map_err(|e| {
// TODO: Error handling could be nicer.
CommonWebError::BadInputWithSimpleMessage("bad request".to_string())
})?;
let uuid_idempotency_token = upload_sample_request.uuid_idempotency_token
.ok_or(CommonWebError::BadInputWithSimpleMessage("no uuid".to_string()))?;
let maybe_existing_upload =
get_dataset_sample_by_uuid_with_connection(&uuid_idempotency_token, &mut mysql_connection)
.await;
match maybe_existing_upload {
Err(err) => {
error!("Error checking for previous upload: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
Ok(Some(previous_record)) => {
return success_response(
previous_record.token,
previous_record.media_file_token,
);
}
Ok(None) => {
// Proceed.
}
}
if let Err(reason) = validate_idempotency_token_format(&uuid_idempotency_token) {
return Err(CommonWebError::BadInputWithSimpleMessage(reason));
}
let creator_set_visibility = maybe_user_session
.as_ref()
.map(|user_session| user_session.preferred_tts_result_visibility)
.unwrap_or(Visibility::default());
let ip_address = get_request_ip(&http_request);
let maybe_avt_token = server_state
.avt_cookie_manager
.get_avt_token_from_request(&http_request);
let maybe_user_token = maybe_user_session
.as_ref()
.map(|session| session.get_user_token());
let maybe_file_size_bytes = upload_sample_request.file_bytes
.as_ref()
.map(|bytes| bytes.len());
let maybe_mimetype = upload_sample_request.file_bytes
.as_ref()
.map(|bytes| get_mimetype_for_bytes(bytes.as_ref()))
.flatten();
let dataset_token = match upload_sample_request.dataset_token {
None => return Err(CommonWebError::BadInputWithSimpleMessage("missing dataset token".to_string())),
Some(token) => ZsVoiceDatasetToken::new_from_str(&token),
};
let bytes = match upload_sample_request.file_bytes {
None => return Err(CommonWebError::BadInputWithSimpleMessage("missing file contents".to_string())),
Some(bytes) => bytes,
};
let hash = sha256_hash_bytes(&bytes)
.map_err(|io_error| {
error!("Problem hashing bytes: {:?}", io_error);
CommonWebError::from_anyhow_error(io_error)
})?;
let file_size_bytes = bytes.len();
let mut maybe_duration_millis = None;
let mut maybe_codec_name = None;
let mut media_upload_type = None;
if let Some(mimetype) = maybe_mimetype.as_deref() {
if !ALLOWED_AUDIO_MIME_TYPES.contains(mimetype) {
// NB: Don't let our error message inject malicious strings
let filtered_mimetype = mimetype
.chars()
.filter(|c| c.is_ascii())
.filter(|c| c.is_alphanumeric() || *c == '/')
.collect::<String>();
return Err(CommonWebError::BadInputWithSimpleMessage(format!("unpermitted mime type: {}", &filtered_mimetype)));
}
// NB: .aiff (audio/aiff) isn't supported by Symphonia:
// It contains uncompressed PCM-encoded audio similar to wav.
// See: https://github.com/pdeljanov/Symphonia/issues/75
// NB: The following formats are not supported by Symphonia and
// do not have any open issues filed. They may simply be too old:
// - .wma (audio/x-ms-wma)
// - .avi (video/x-msvideo)
media_upload_type = match mimetype {
// Audio
"audio/aac" /* .aac */ => Some(MediaUploadType::Audio),
"audio/m4a" /* .m4a */ => Some(MediaUploadType::Audio),
"audio/mpeg" /* .mp3 */ => Some(MediaUploadType::Audio),
"audio/ogg" /* .ogg */ => Some(MediaUploadType::Audio),
"audio/opus" /* .opus */ => Some(MediaUploadType::Audio),
"audio/x-flac" /* .flac */ => Some(MediaUploadType::Audio),
"audio/x-wav" /* .wav */ => Some(MediaUploadType::Audio),
// Video
"video/webm" /* .webm */ => Some(MediaUploadType::Video), // TODO: Do browsers send this for audio (?)
_ => None,
};
let do_audio_decode = match mimetype {
"audio/opus" => {
// TODO/FIXME(bt, 2023-05-19): Symphonia is currently broken for Firefox's opus.
// We're on an off-master branch that may resolve the problem in the future, but for now
// it panics as follows:
//
// [2023-05-19T10:25:34Z INFO symphonia_core::probe] found a possible format marker within [4f, 67, 67, 53, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, d4, c5] @ 0+2 bytes.
// [2023-05-19T10:25:34Z INFO symphonia_core::probe] found the format marker [4f, 67, 67, 53] @ 0+2 bytes.
// [2023-05-19T10:25:34Z DEBUG symphonia_format_ogg::page] grow page buffer to 8192 bytes
// [2023-05-19T10:25:34Z INFO symphonia_format_ogg::demuxer] starting new physical stream
// [2023-05-19T10:25:34Z INFO symphonia_format_ogg::demuxer] selected opus mapper for stream with serial=0x19aac5d4
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Probed!
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Find audio track...
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Found audio track (maybe)
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Maybe track duration: None
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Maybe codec short name: Some("opus")
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Opus handler
// [2023-05-19T10:25:34Z INFO media::decode_basic_audio_info] Media source stream
// [2023-05-19T10:25:34Z INFO media::decode_webm_opus_info] decode_mkv : options...
// [2023-05-19T10:25:34Z INFO media::decode_webm_opus_info] decode_mkv : try_read...
// [2023-05-19T10:25:34Z DEBUG symphonia_format_mkv::ebml] element with tag: 4F67
// thread 'actix-rt|system:0|arbiter:1' panicked at 'assertion failed: `(left == right)`
// left: `Unknown`,
// right: `Ebml`: EBML element type must be checked before calling this function', /Users/bt/.cargo/git/checkouts/symphonia-8fbe6c90fc095688/e1a7009/symphonia-format-mkv/src/ebml.rs:335:9
false
}
// Also, don't decode images
"image/jpeg" => false,
"image/png" => false,
"image/webp" => false,
_ => true,
};
if do_audio_decode && media_upload_type.is_some() {
let basic_info = decode_basic_audio_bytes_info(
bytes.as_ref(),
Some(mimetype),
None
).map_err(|e| {
warn!("file decoding error: {:?}", e);
CommonWebError::BadInputWithSimpleMessage("could not decode file".to_string())
})?;
maybe_duration_millis = basic_info.duration_millis;
maybe_codec_name = basic_info.codec_name;
}
}
let media_upload_type = match media_upload_type {
Some(m) => m,
None => {
warn!("Invalid mimetype: {:?}", maybe_mimetype);
return Err(CommonWebError::BadInputWithSimpleMessage(format!("unknown mimetype: {:?}", maybe_mimetype)))
},
};
let media_file_origin = match upload_sample_request.media_source {
MediaSource::Unknown => MediaFileOriginCategory::Upload, // NB: Just treat as a regular upload.
MediaSource::UserFile => MediaFileOriginCategory::Upload,
MediaSource::UserDeviceApi => MediaFileOriginCategory::DeviceApi,
};
// TODO: Clean up code
let mime_type = match maybe_mimetype {
Some(m) => m,
None => {
warn!("Missing mimetype!");
return Err(CommonWebError::BadInputWithSimpleMessage("Missing mimetype".to_string()));
},
};
let file_prefix = "sample_";
let file_extension = mimetype_to_extension(mime_type).unwrap_or("bin");
let file_extension = format!(".{file_extension}");
let public_upload_path = MediaFileBucketPath::generate_new(
Some(&file_prefix),
Some(&file_extension)
);
info!("Uploading media to bucket path: {}", public_upload_path.get_full_object_path_str());
server_state.public_bucket_client.upload_file_with_content_type(
public_upload_path.get_full_object_path_str(),
bytes.as_ref(),
mime_type)
.await
.map_err(|e| {
warn!("Upload media bytes to bucket error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
let (dataset_sample_token, media_file_token, _record_id) = insert_dataset_sample_and_media_file(InsertDatasetSampleAndMediaFileArgs {
uuid_idempotency_token: &uuid_idempotency_token,
media_type: media_upload_type,
origin_category: media_file_origin,
dataset_token: &dataset_token,
maybe_mime_type: maybe_mimetype,
file_size_bytes: file_size_bytes as u64,
maybe_original_filename: upload_sample_request.file_name.as_deref(),
maybe_original_duration_millis: maybe_duration_millis,
maybe_original_audio_encoding: maybe_codec_name.as_deref(),
checksum_sha2: &hash,
media_file_path: &public_upload_path,
maybe_public_bucket_prefix: Some(&file_prefix),
maybe_public_bucket_extension: Some(&file_extension),
//maybe_extra_file_modification_info: None,
maybe_creator_user_token: maybe_user_token,
maybe_creator_anonymous_visitor_token: maybe_avt_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility,
mysql_pool: &server_state.mysql_pool,
})
.await
.map_err(|err| {
warn!("New generic download creation DB error: {:?}", err);
CommonWebError::from_anyhow_error(err)
})?;
info!("new media file token: {:?} dataset token: {:?}", media_file_token, dataset_sample_token);
success_response(
dataset_sample_token,
media_file_token,
)
}
fn success_response(
sample_token: ZsVoiceDatasetSampleToken,
media_file_token: MediaFileToken
) -> Result<HttpResponse, CommonWebError> {
let response = UploadSampleResponse {
success: true,
sample_token,
media_file_token,
};
let body = serde_json::to_string(&response)
.map_err(CommonWebError::from_error)?;
return Ok(HttpResponse::Ok()
.content_type("application/json")
.body(body));
}
@@ -0,0 +1,80 @@
use std::fmt;
use std::sync::Arc;
use actix_web::{web, HttpRequest};
use actix_web::web::Json;
use log::{error, warn};
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::queries::voice_designer::datasets::create_dataset::{create_dataset, CreateDatasetArgs};
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::state::server_state::ServerState;
use crate::http_server::common_responses::common_web_error::CommonWebError;
#[derive(Deserialize)]
pub struct CreateDatasetRequest {
pub title: String,
pub creator_set_visibility: Option<Visibility>,
pub idempotency_token: Option<String>,
}
#[derive(Serialize)]
pub struct CreateDatasetResponse {
pub success: bool,
pub token: Option<ZsVoiceDatasetToken>,
}
// =============== Error Response ===============
// NB: Not using DeriveMore since Clion doesn't understand it.
// =============== Handler ===============
pub async fn create_dataset_handler(http_request: HttpRequest, request: Json<CreateDatasetRequest>, server_state: web::Data<Arc<ServerState>>) -> Result<Json<CreateDatasetResponse>, CommonWebError> {
let maybe_user_session = server_state.session_checker.maybe_get_user_session(&http_request, &server_state.mysql_pool).await.map_err(|e| {
error!("Error getting user session: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
},
};
let idempotency_token = request.idempotency_token.clone().ok_or(CommonWebError::BadInputWithSimpleMessage("no idempotency token provided".to_string()))?;
let title = request.title.clone();
let creator_ip_address = get_request_ip(&http_request);
let creator_set_visibility = request.creator_set_visibility.unwrap_or(Visibility::Public);
let query_result = create_dataset(CreateDatasetArgs {
dataset_title: &title,
maybe_creator_user_token: Some(user_session.user_token.as_str()),
creator_ip_address: &creator_ip_address,
creator_set_visibility,
mysql_pool: &server_state.mysql_pool
}).await;
let dataset_token = match query_result {
Ok(token) => token,
Err(e) => {
error!("Error creating dataset: {:?}", e);
return Err(CommonWebError::from_anyhow_error(e));
}
};
let response = CreateDatasetResponse {
success: true,
token: Some(dataset_token),
};
Ok(Json(response))
}
@@ -0,0 +1,143 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::warn;
use artcraft_api_defs::common::responses::simple_generic_json_success::SimpleGenericJsonSuccess;
use mysql_queries::queries::voice_designer::datasets::delete_dataset::{delete_dataset_as_mod, delete_dataset_as_user, undelete_dataset_as_mod, undelete_dataset_as_user};
use mysql_queries::queries::voice_designer::datasets::get_dataset::get_dataset_by_token;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
use crate::util::delete_role_disambiguation::{delete_role_disambiguation, DeleteRole};
#[derive(Deserialize)]
pub struct DeleteDatasetRequest {
set_delete: bool,
/// NB: this is only to disambiguate when a user is both a mod and an author.
as_mod: Option<bool>,
}
/// For the URL PathInfo
#[derive(Deserialize)]
pub struct DeleteDatasetPathInfo {
dataset_token: String,
}
// =============== Error Response ===============
// NB: Not using derive_more::Display since Clion doesn't understand it.
// =============== Handler ===============
pub async fn delete_dataset_handler(
http_request: HttpRequest,
path: Path<DeleteDatasetPathInfo>,
request: Json<DeleteDatasetRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<SimpleGenericJsonSuccess>, CommonWebError>{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let dataset_token = path.dataset_token.clone();
let is_mod = user_session.can_ban_users;
let dataset_lookup_result = get_dataset_by_token(
&ZsVoiceDatasetToken::new(dataset_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let dataset = match dataset_lookup_result {
Ok(Some(dataset)) => dataset,
Ok(None) => {
warn!("Dataset not found: {:?}", dataset_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up dataset: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
let is_creator = dataset.maybe_creator_user_token.as_deref()
.map(|creator_user_token| creator_user_token == user_session.user_token.as_str())
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to delete this dataset: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let delete_role = delete_role_disambiguation(is_mod, is_creator, request.as_mod);
let query_result = if request.set_delete {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
warn!("user is not allowed to delete datasets: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
delete_dataset_as_user(
&path.dataset_token,
&server_state.mysql_pool
).await
}
DeleteRole::AsMod => {
delete_dataset_as_mod(
&path.dataset_token,
user_session.user_token.as_str(),
&server_state.mysql_pool
).await
}
}
} else {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
warn!("user is not allowed to undelete voices: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
// NB: Technically only mods can see their own datasets
undelete_dataset_as_user(
&path.dataset_token,
&server_state.mysql_pool
).await
}
DeleteRole::AsMod => {
undelete_dataset_as_mod(
&path.dataset_token,
user_session.user_token.as_str(),
&server_state.mysql_pool
).await
}
}
};
match query_result {
Ok(_) => {},
Err(err) => {
warn!("Update dataset mod approval status DB error: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
Ok(Json(SimpleGenericJsonSuccess { success: true }))
}
@@ -0,0 +1,112 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::datasets::get_dataset::get_dataset_by_token;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct GetDatasetResponse {
success: bool,
dataset_token: ZsVoiceDatasetToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Deserialize)]
pub struct GetDatasetPathInfo {
dataset_token: String,
}
pub async fn get_dataset_handler(
http_request: HttpRequest,
path: Path<GetDatasetPathInfo>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<GetDatasetResponse>, CommonWebError> {
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let dataset_token = path.dataset_token.clone();
let creator_user_token = user_session.user_token.as_str().to_string();
let is_mod = user_session.can_ban_users;
let dataset_lookup_result = get_dataset_by_token(
&ZsVoiceDatasetToken::new(dataset_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let dataset = match dataset_lookup_result {
Ok(Some(dataset)) => dataset,
Ok(None) => {
warn!("Dataset not found: {:?}", dataset_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up dataset: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
let is_creator = dataset.maybe_creator_user_token.as_deref()
.map(|creator_user_token| creator_user_token == user_session.user_token.as_str())
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to view this dataset: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let response = GetDatasetResponse {
success: true,
dataset_token: dataset.token,
title: dataset.title,
ietf_language_tag: dataset.ietf_language_tag,
ietf_primary_language_subtag: dataset.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&user_session.user_token,
user_session.username.as_ref(),
user_session.display_name.as_ref(),
user_session.email_gravatar_hash.as_ref(),
),
creator_set_visibility: dataset.creator_set_visibility,
created_at: dataset.created_at,
updated_at: dataset.updated_at,
};
Ok(Json(response))
}
@@ -0,0 +1,127 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::datasets::list_datasets_by_username::list_datasets_by_username;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct ZsDatasetRecord {
dataset_token: ZsVoiceDatasetToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Serialize)]
pub struct ListDatasetsByUserSuccessResponse {
pub success: bool,
pub datasets: Vec<ZsDatasetRecord>,
}
#[derive(Debug)]
pub enum ListDatasetsByUserError {
NotAuthorized,
ServerError,
}
impl fmt::Display for ListDatasetsByUserError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
impl ResponseError for ListDatasetsByUserError {
fn status_code(&self) -> StatusCode {
match *self {
ListDatasetsByUserError::NotAuthorized => StatusCode::UNAUTHORIZED,
ListDatasetsByUserError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
pub async fn list_datasets_by_session_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListDatasetsByUserSuccessResponse>, ListDatasetsByUserError> {
let user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
ListDatasetsByUserError::ServerError
})?
.ok_or_else(|| {
warn!("not logged in");
ListDatasetsByUserError::NotAuthorized
})?;
let _is_mod = user_session.can_ban_users;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
// We should focus on visibility-related controls instead.
const CAN_SEE_DELETED : bool = false;
let query_results = list_datasets_by_username(
&server_state.mysql_pool,
&user_session.username,
CAN_SEE_DELETED,
).await.map_err(|e| {
warn!("Error querying for datasets: {:?}", e);
ListDatasetsByUserError::ServerError
});
let datasets = match query_results {
Ok(datasets) => datasets,
Err(e) => {
warn!("Error querying for datasets: {:?}", e);
return Err(ListDatasetsByUserError::ServerError);
}
};
let datasets = datasets.into_iter().map(|dataset| {
ZsDatasetRecord {
dataset_token: dataset.dataset_token,
title: dataset.title,
creator_set_visibility: dataset.creator_set_visibility,
ietf_language_tag: dataset.ietf_language_tag,
ietf_primary_language_subtag: dataset.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&dataset.creator_user_token,
&dataset.creator_username,
&dataset.creator_display_name,
&dataset.creator_email_gravatar_hash,
),
created_at: dataset.created_at,
updated_at: dataset.updated_at,
}
}).collect();
let response = ListDatasetsByUserSuccessResponse {
success: true,
datasets,
};
Ok(Json(response))
}
@@ -0,0 +1,153 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::web::Path;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use utoipa::ToSchema;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::datasets::list_datasets_by_username::list_datasets_by_username;
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone, ToSchema)]
pub struct ZsDatasetRecord {
dataset_token: ZsVoiceDatasetToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Serialize, ToSchema)]
pub struct ListDatasetsByUserSuccessResponse {
pub success: bool,
pub datasets: Vec<ZsDatasetRecord>,
}
#[derive(Deserialize, ToSchema)]
pub struct ListDatasetsByUserPathInfo {
username: String,
}
#[derive(Debug, ToSchema)]
pub enum ListDatasetsByUserError {
NotAuthorized,
ServerError,
}
impl fmt::Display for ListDatasetsByUserError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
impl ResponseError for ListDatasetsByUserError {
fn status_code(&self) -> StatusCode {
match *self {
ListDatasetsByUserError::NotAuthorized => StatusCode::UNAUTHORIZED,
ListDatasetsByUserError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
#[utoipa::path(
get,
tag = "Voice Designer",
path = "/v1/voice_designer/user/{username}/list",
responses(
(status = 200, description = "Found", body = ListDatasetsByUserSuccessResponse),
(status = 401, description = "Not authorized", body = ListDatasetsByUserError),
(status = 500, description = "Server error", body = ListDatasetsByUserError),
),
params(
("path" = ListDatasetsByUserPathInfo, description = "Path for Request")
)
)]
pub async fn list_datasets_by_user_handler(
http_request: HttpRequest,
path: Path<ListDatasetsByUserPathInfo>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListDatasetsByUserSuccessResponse>, ListDatasetsByUserError> {
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
ListDatasetsByUserError::ServerError
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(ListDatasetsByUserError::NotAuthorized);
}
};
let username = path.username.as_ref();
let creator_user_token = user_session.user_token.as_str().to_string();
let _is_mod = user_session.can_ban_users;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
// We should focus on visibility-related controls instead.
const CAN_SEE_DELETED : bool = false;
let query_results = list_datasets_by_username(
&server_state.mysql_pool,
&username,
CAN_SEE_DELETED,
).await.map_err(|e| {
warn!("Error querying for datasets: {:?}", e);
ListDatasetsByUserError::ServerError
});
let datasets = match query_results {
Ok(datasets) => datasets,
Err(e) => {
warn!("Error querying for datasets: {:?}", e);
return Err(ListDatasetsByUserError::ServerError);
}
};
let datasets = datasets.into_iter().map(|dataset| {
ZsDatasetRecord {
dataset_token: dataset.dataset_token,
title: dataset.title,
creator_set_visibility: dataset.creator_set_visibility,
ietf_language_tag: dataset.ietf_language_tag,
ietf_primary_language_subtag: dataset.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&dataset.creator_user_token,
&dataset.creator_username,
&dataset.creator_display_name,
&dataset.creator_email_gravatar_hash,
),
created_at: dataset.created_at,
updated_at: dataset.updated_at,
}
}).collect();
let response = ListDatasetsByUserSuccessResponse {
success: true,
datasets,
};
Ok(Json(response))
}
@@ -0,0 +1,6 @@
pub mod create_dataset;
pub mod delete_dataset;
pub mod list_datasets_by_session;
pub mod list_datasets_by_user;
pub mod update_dataset;
pub mod get_dataset;
@@ -0,0 +1,171 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, warn};
use artcraft_api_defs::common::responses::simple_generic_json_success::SimpleGenericJsonSuccess;
use enums::common::visibility::Visibility;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::queries::voice_designer::datasets::get_dataset::get_dataset_by_token;
use mysql_queries::queries::voice_designer::datasets::update_dataset::{update_dataset, UpdateDatasetArgs};
use tokens::tokens::zs_voice_datasets::ZsVoiceDatasetToken;
use crate::configs::supported_languages_for_models::get_canonicalized_language_tag_for_model;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct UpdateDatasetRequest {
// ========== Author + Moderator options ==========
pub title: Option<String>,
pub creator_set_visibility: Option<String>,
// NB: We calculate 'ietf_primary_language_subtag' from this value.
pub ietf_language_tag: Option<String>,
}
#[derive(Serialize)]
pub struct UpdateDatasetResponse {
pub success: bool,
}
/// For the URL PathInfo
#[derive(Deserialize)]
pub struct UpdateDatasetPathInfo {
dataset_token: String,
}
// =============== Error Response ===============
// NB: Not using derive_more::Display since Clion doesn't understand it.
// =============== Handler ===============
pub async fn update_dataset_handler(
http_request: HttpRequest,
path: Path<UpdateDatasetPathInfo>,
request: Json<UpdateDatasetRequest>,
server_state: web::Data<Arc<ServerState>>) -> Result<Json<SimpleGenericJsonSuccess>, CommonWebError>
{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let dataset_token = path.dataset_token.clone();
let is_mod = user_session.can_ban_users;
let dataset_lookup_result = get_dataset_by_token(
&ZsVoiceDatasetToken::new(dataset_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let dataset = match dataset_lookup_result {
Ok(Some(dataset)) => dataset,
Ok(None) => {
warn!("Dataset not found: {:?}", dataset_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up dataset: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
// let is_creator = dataset.maybe_creator_user_token == Some(user_session.user_token);
let is_creator = dataset.maybe_creator_user_token.as_deref()
.map(|creator_user_token| creator_user_token == user_session.user_token.as_str())
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to edit this dataset: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let mut title = None;
let mut ietf_language_tag = None;
let mut ietf_primary_language_subtag = None;
let mut creator_set_visibility = Visibility::Public;
if let Some(payload) = request.title.as_deref() {
if user_input_common::check_for_slurs::contains_slurs(payload) {
return Err(CommonWebError::BadInputWithSimpleMessage("title contains slurs".to_string()));
}
title = Some(payload.trim().to_string());
}
if let Some(tag) = request.ietf_language_tag.as_deref() {
// eg. en, en-US, es-419, ja-JP, etc.
let maybe_full_canonical_tag = get_canonicalized_language_tag_for_model(tag);
// eg. en, es, ja, etc.
let maybe_primary_language_subtag = maybe_full_canonical_tag
.map(|t| language_tags::LanguageTag::parse(t)
.map(|language_tag| language_tag.primary_language().to_string())
)
.transpose()
.map_err(|e| {
error!("Error parsing language tag '{}': {:?}", tag, e);
CommonWebError::BadInputWithSimpleMessage("bad locale string".to_string())
})?;
if let Some(full_tag) = maybe_full_canonical_tag {
if let Some(primary_subtag) = maybe_primary_language_subtag.as_deref() {
ietf_language_tag = Some(full_tag.to_string());
ietf_primary_language_subtag = Some(primary_subtag.to_string());
}
}
}
if let Some(visibility) = request.creator_set_visibility.as_deref() {
creator_set_visibility = Visibility::from_str(visibility)
.map_err(|_| CommonWebError::BadInputWithSimpleMessage("bad record visibility".to_string()))?;
}
let ip_address = get_request_ip(&http_request);
let mut maybe_mod_user_token = None;
if is_mod {
maybe_mod_user_token = Some(user_session.user_token.as_str().to_string());
}
let query_result = update_dataset(
UpdateDatasetArgs {
dataset_token: &ZsVoiceDatasetToken::new(dataset_token.clone()),
dataset_title: title.as_deref(),
maybe_creator_user_token: Some(user_session.user_token.as_str()),
creator_ip_address: ip_address.as_ref(),
creator_set_visibility: &creator_set_visibility,
maybe_mod_user_token: maybe_mod_user_token.as_deref(),
ietf_language_tag: ietf_language_tag.as_deref(),
ietf_primary_language_subtag: ietf_primary_language_subtag.as_deref(),
mysql_pool: &server_state.mysql_pool
}
).await;
match query_result {
Ok(_) => {},
Err(err) => {
warn!("Update Dataset DB error: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
Ok(Json(SimpleGenericJsonSuccess { success: true }))
}
@@ -0,0 +1,186 @@
// #![forbid(unused_imports)]
// #![forbid(unused_mut)]
// #![forbid(unused_variables)]
use std::fmt::Debug;
use std::sync::Arc;
use actix_web::{web, HttpRequest};
use actix_web::web::Json;
use log::{info, warn};
use serde::Deserialize;
use serde::Serialize;
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use enums::by_table::generic_inference_jobs::inference_job_product_category::InferenceJobProductCategory;
use enums::by_table::generic_inference_jobs::inference_job_type::InferenceJobType;
use enums::by_table::generic_inference_jobs::inference_model_type::InferenceModelType;
use http_server_common::request::get_request_header_optional::get_request_header_optional;
use http_server_common::request::get_request_ip::get_request_ip;
use mysql_queries::payloads::generic_inference_args::generic_inference_args::{
GenericInferenceArgs,
InferenceCategoryAbbreviated,
PolymorphicInferenceArgs,
};
use mysql_queries::payloads::generic_inference_args::inner_payloads::tts_payload::TTSArgs;
use mysql_queries::queries::generic_inference::web::insert_generic_inference_job::{
insert_generic_inference_job,
InsertGenericInferenceArgs,
};
use tokens::tokens::generic_inference_jobs::InferenceJobToken;
use tokens::tokens::users::UserToken;
use crate::configs::plans::get_correct_plan_for_session::get_correct_plan_for_session;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct EnqueueCreateVoiceRequest {
uuid_idempotency_token: String,
voice_dataset_token: String,
}
#[derive(Serialize)]
pub struct EnqueueCreateVoiceRequestSuccessResponse {
pub success: bool,
pub inference_job_token: InferenceJobToken,
}
/// Debug requests can get routed to special "debug-only" workers, which can
/// be used to trial new code, run debugging, etc.
const DEBUG_HEADER_NAME: &str = "enable-debug-mode";
/// The routing tag header can send workloads to particular k8s hosts.
/// This is useful for catching the live logs or intercepting the job.
const ROUTING_TAG_HEADER_NAME: &str = "routing-tag";
// pub async fn get_dataset_by_token
pub async fn create_voice_handler(
http_request: HttpRequest,
request: Json<EnqueueCreateVoiceRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<EnqueueCreateVoiceRequestSuccessResponse>, CommonWebError> {
// voice data set token ... zsd_wb08asm0m4a4cz42wwqked6tfg6e e.g
// need this for the job
// -> create_voice_records
// Implementation for creating a voice
let mut maybe_user_token: Option<UserToken> = None;
let mut mysql_connection = server_state.mysql_pool.acquire().await.map_err(|err| {
warn!("MySql pool error: {:?}", err);
CommonWebError::from_error(err)
})?;
// ==================== USER SESSION ==================== //
let maybe_user_session = server_state.session_checker
.maybe_get_user_session_extended_from_connection(&http_request, &mut mysql_connection).await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
if let Some(user_session) = maybe_user_session.as_ref() {
maybe_user_token = Some(UserToken::new_from_str(&user_session.user_token));
}
// TODO: Plan should handle "first anonymous use" and "investor" cases.
let plan = get_correct_plan_for_session(
server_state.server_environment,
maybe_user_session.as_ref()
);
// TODO: Separate priority for animation.
let priority_level = plan.web_vc_base_priority_level();
// ==================== DEBUG MODE + ROUTING TAG ==================== //
let is_debug_request = get_request_header_optional(&http_request, DEBUG_HEADER_NAME).is_some();
let maybe_routing_tag = get_request_header_optional(&http_request, ROUTING_TAG_HEADER_NAME).map(
|routing_tag| routing_tag.trim().to_string()
);
// ==================== BANNED USERS ==================== //
if let Some(ref user) = maybe_user_session {
if user.role.is_banned {
return Err(CommonWebError::NotAuthorized);
}
}
// ==================== RATE LIMIT ==================== //
let rate_limiter = match maybe_user_session {
None => &server_state.redis_rate_limiters.logged_out,
Some(ref _user) => &server_state.redis_rate_limiters.logged_in,
};
if let Err(_err) = rate_limiter.rate_limit_request(&http_request).await {
return Err(CommonWebError::TooManyRequests);
}
info!("Received payload for voice creation.");
// Get up IP address
let ip_address = get_request_ip(&http_request);
// package as larger component args should always have an embedding token ..
let inference_args = TTSArgs {
voice_token: None,
dataset_token: Some(request.voice_dataset_token.clone()),
};
let maybe_avt_token = server_state.avt_cookie_manager.get_avt_token_from_request(&http_request);
// create the inference args here
// enqueue a zero shot tts request here...
// create the job record here! explore the table insert the inference args in here as json! keep it short
let query_result = insert_generic_inference_job(InsertGenericInferenceArgs {
uuid_idempotency_token: &request.uuid_idempotency_token,
job_type: InferenceJobType::StyleTTS2,
maybe_product_category: Some(InferenceJobProductCategory::TtsStyleTts2), // Perhaps we should count this another way?
inference_category: InferenceCategory::TextToSpeech,
maybe_model_type: Some(InferenceModelType::StyleTTS2), // NB: Model is static during inference
maybe_model_token: None, // NB: Model is static during inference
maybe_input_source_token: None,
maybe_input_source_token_type: None,
maybe_download_url: None,
maybe_cover_image_media_file_token: None,
maybe_raw_inference_text: None,
maybe_max_duration_seconds: None,
maybe_inference_args: Some(GenericInferenceArgs {
inference_category: Some(InferenceCategoryAbbreviated::TextToSpeech),
args: Some(PolymorphicInferenceArgs::Tts(inference_args)),
}),
maybe_creator_user_token: maybe_user_token.as_ref(),
creator_ip_address: &ip_address,
creator_set_visibility: enums::common::visibility::Visibility::Public,
priority_level,
requires_keepalive: true,
maybe_avt_token: maybe_avt_token.as_ref(),
is_debug_request,
maybe_routing_tag: maybe_routing_tag.as_deref(),
mysql_pool: &server_state.mysql_pool,
}).await;
let job_token = match query_result {
Ok((job_token, _id)) => job_token,
Err(err) => {
warn!("New generic inference job creation DB error: {:?}", err);
if err.had_duplicate_idempotency_token() {
return Err(CommonWebError::BadInputWithSimpleMessage("Duplicate idempotency token".to_string()));
}
return Err(CommonWebError::from_error(err));
}
};
let response: EnqueueCreateVoiceRequestSuccessResponse =
EnqueueCreateVoiceRequestSuccessResponse {
success: true,
inference_job_token: job_token,
};
Ok(Json(response))
}
@@ -0,0 +1,141 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::warn;
use artcraft_api_defs::common::responses::simple_generic_json_success::SimpleGenericJsonSuccess;
use mysql_queries::queries::voice_designer::voices::delete_voice::{delete_voice_as_mod, delete_voice_as_user, undelete_voice_as_mod, undelete_voice_as_user};
use mysql_queries::queries::voice_designer::voices::get_voice::get_voice_by_token;
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
use crate::util::delete_role_disambiguation::{delete_role_disambiguation, DeleteRole};
#[derive(Deserialize)]
pub struct DeleteVoiceRequest {
set_delete: bool,
/// NB: this is only to disambiguate when a user is both a mod and an author.
as_mod: Option<bool>,
}
/// For the URL PathInfo
#[derive(Deserialize)]
pub struct DeleteVoicePathInfo {
voice_token: String,
}
// =============== Error Response ===============
// NB: Not using derive_more::Display since Clion doesn't understand it.
// =============== Handler ===============
pub async fn delete_voice_handler(
http_request: HttpRequest,
path: Path<DeleteVoicePathInfo>,
request: Json<DeleteVoiceRequest>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<SimpleGenericJsonSuccess>, CommonWebError>{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let voice_token = path.voice_token.clone();
let is_mod = user_session.can_ban_users;
let voice_lookup_result = get_voice_by_token(
&ZsVoiceToken::new(voice_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let voice = match voice_lookup_result {
Ok(Some(voice)) => voice,
Ok(None) => {
warn!("Voice not found: {:?}", voice_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up voice: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
let is_creator = voice.maybe_creator_user_token.as_ref()
.map(|creator_user_token| creator_user_token == &user_session.user_token)
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to delete this voice: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let delete_role = delete_role_disambiguation(is_mod, is_creator, request.as_mod);
let query_result = if request.set_delete {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
warn!("user is not allowed to delete voices: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
delete_voice_as_user(
&path.voice_token,
&server_state.mysql_pool
).await
}
DeleteRole::AsMod => {
delete_voice_as_mod(
&path.voice_token,
user_session.user_token.as_str(),
&server_state.mysql_pool
).await
}
}
} else {
match delete_role {
DeleteRole::ErrorDoNotDelete => {
warn!("user is not allowed to undelete voices: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
DeleteRole::AsUser => {
undelete_voice_as_user(
&path.voice_token,
&server_state.mysql_pool
).await
}
DeleteRole::AsMod => {
undelete_voice_as_mod(
&path.voice_token,
user_session.user_token.as_str(),
&server_state.mysql_pool
).await
}
}
};
match query_result {
Ok(_) => {},
Err(err) => {
warn!("Update voice mod approval status DB error: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
Ok(Json(SimpleGenericJsonSuccess { success: true }))
}
@@ -0,0 +1,112 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::voices::get_voice::get_voice_by_token;
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct GetVoiceResponse {
success: bool,
voice_token: ZsVoiceToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Deserialize)]
pub struct GetVoicePathInfo {
voice_token: String,
}
pub async fn get_voice_handler(
http_request: HttpRequest,
path: Path<GetVoicePathInfo>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<GetVoiceResponse>, CommonWebError> {
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let voice_token = path.voice_token.clone();
let creator_user_token = user_session.user_token.as_str().to_string();
let is_mod = user_session.can_ban_users;
let voice_lookup_result = get_voice_by_token(
&ZsVoiceToken::new(voice_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let voice = match voice_lookup_result {
Ok(Some(voice)) => voice,
Ok(None) => {
warn!("Voice not found: {:?}", voice_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up voice: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
let is_creator = voice.maybe_creator_user_token
.map(|creator_user_token| creator_user_token.as_str() == user_session.user_token.as_str())
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to view this voice: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let response = GetVoiceResponse {
success: true,
voice_token: voice.token,
title: voice.title,
ietf_language_tag: voice.ietf_language_tag,
ietf_primary_language_subtag: voice.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&user_session.user_token,
user_session.username.as_ref(),
user_session.display_name.as_ref(),
user_session.email_gravatar_hash.as_ref(),
),
creator_set_visibility: voice.creator_set_visibility,
created_at: voice.created_at,
updated_at: voice.updated_at,
};
Ok(Json(response))
}
@@ -0,0 +1,164 @@
use std::sync::Arc;
use actix_web::{web, HttpRequest};
use actix_web::web::Json;
use chrono::{DateTime, Utc};
use log::{info, warn};
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::voices::list_voices_query_builder::ListVoicesQueryBuilder;
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct ListZsVoicesQuery {
pub sort_ascending: Option<bool>,
pub limit: Option<u16>,
pub cursor: Option<String>,
pub cursor_is_reversed: Option<bool>,
}
#[derive(Serialize)]
pub struct ListZsVoicesSuccessResponse {
pub success: bool,
pub voices: Vec<ZsVoiceForList>,
pub cursor_next: Option<String>,
pub cursor_previous: Option<String>
}
#[derive(Serialize)]
pub struct ZsVoiceForList {
pub voice_token: ZsVoiceToken,
pub title: String,
pub ietf_language_tag: String,
pub ietf_primary_language_subtag: String,
pub creator: UserDetailsLight,
pub creator_set_visibility: Visibility,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
pub async fn list_available_voices_handler(
http_request: HttpRequest,
query: web::Query<ListZsVoicesQuery>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListZsVoicesSuccessResponse>, CommonWebError> {
let maybe_user_session = server_state.session_checker.maybe_get_user_session(
&http_request,
&server_state.mysql_pool
).await.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let mut is_mod = false;
let user_session = match maybe_user_session {
Some(session) => {
is_mod = session.can_ban_users;
session
}
None => {
info!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let limit = query.limit.unwrap_or(25);
let sort_ascending = query.sort_ascending.unwrap_or(false);
let cursor_is_reversed = query.cursor_is_reversed.unwrap_or(false);
let cursor = if let Some(cursor) = query.cursor.as_deref() {
let cursor = server_state.sort_key_crypto.decrypt_id(cursor)
.map_err(|e| {
warn!("crypto error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
Some(cursor)
} else {
None
};
let include_user_hidden = is_mod;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
const CAN_SEE_DELETED : bool = false;
let mut query_builder = ListVoicesQueryBuilder::new()
.sort_ascending(sort_ascending)
.scope_creator_username(None)
.include_user_hidden(include_user_hidden)
.include_user_deleted_results(CAN_SEE_DELETED)
.include_mod_deleted_results(CAN_SEE_DELETED)
.limit(limit)
.cursor_is_reversed(cursor_is_reversed)
.offset(cursor);
let query_results = query_builder.perform_query_for_page(&server_state.mysql_pool).await;
let voices_page = match query_results {
Ok(results) => results,
Err(e) => {
warn!("Query error: {:?}", e);
return Err(CommonWebError::from_anyhow_error(e));
}
};
let cursor_next = if let Some(id) = voices_page.last_id {
let cursor = server_state.sort_key_crypto.encrypt_id(id as u64)
.map_err(|e| {
warn!("crypto error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
Some(cursor)
} else {
None
};
let cursor_previous = if let Some(id) = voices_page.first_id {
let cursor = server_state.sort_key_crypto.encrypt_id(id as u64)
.map_err(|e| {
warn!("crypto error: {:?}", e);
CommonWebError::from_anyhow_error(e)
})?;
Some(cursor)
} else {
None
};
let response = ListZsVoicesSuccessResponse {
success: true,
voices: voices_page.voices.into_iter()
.map(|voice| ZsVoiceForList {
voice_token: voice.voice_token,
title: voice.title,
ietf_language_tag: voice.ietf_language_tag,
ietf_primary_language_subtag: voice.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&voice.creator_user_token,
&voice.creator_username,
&voice.creator_display_name,
&voice.creator_email_gravatar_hash,
),
creator_set_visibility: voice.creator_set_visibility,
created_at: voice.created_at,
updated_at: voice.updated_at,
})
.collect::<Vec<_>>(),
cursor_next,
cursor_previous,
};
Ok(Json(response))
}
@@ -0,0 +1,126 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use mysql_queries::queries::voice_designer::voices::list_voices_by_username::list_zs_voices_by_username;
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct Voice {
voice_token: ZsVoiceToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: String,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Serialize)]
pub struct ListVoicesByUserSuccessResponse {
pub success: bool,
pub voices: Vec<Voice>,
}
#[derive(Debug)]
pub enum ListVoicesByUserError {
NotAuthorized,
ServerError,
}
impl fmt::Display for ListVoicesByUserError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
impl ResponseError for ListVoicesByUserError {
fn status_code(&self) -> StatusCode {
match *self {
ListVoicesByUserError::NotAuthorized => StatusCode::UNAUTHORIZED,
ListVoicesByUserError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
pub async fn list_voices_by_session_handler(
http_request: HttpRequest,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListVoicesByUserSuccessResponse>, ListVoicesByUserError> {
let user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
ListVoicesByUserError::ServerError
})?
.ok_or_else(|| {
warn!("not logged in");
ListVoicesByUserError::NotAuthorized
})?;
let creator_user_token = user_session.user_token.as_str().to_string();
let _is_mod = user_session.can_ban_users;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
// We should focus on visibility-related controls instead.
const CAN_SEE_DELETED : bool = false;
let query_results = list_zs_voices_by_username(
&server_state.mysql_pool,
&user_session.username,
CAN_SEE_DELETED,
).await.map_err(|e| {
warn!("Error querying for voices: {:?}", e);
ListVoicesByUserError::ServerError
});
let voices = match query_results {
Ok(voices) => voices,
Err(e) => {
warn!("Error querying for voices: {:?}", e);
return Err(ListVoicesByUserError::ServerError);
}
};
let voices = voices.into_iter().map(|voice| {
Voice {
voice_token: voice.voice_token,
title: voice.title,
ietf_language_tag: voice.ietf_language_tag,
ietf_primary_language_subtag: voice.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&voice.creator_user_token,
&voice.creator_username,
&voice.creator_display_name,
&voice.creator_email_gravatar_hash,
),
creator_set_visibility: voice.creator_set_visibility.to_string() ,
created_at: voice.created_at,
updated_at: voice.updated_at,
}
}).collect();
let response = ListVoicesByUserSuccessResponse {
success: true,
voices,
};
Ok(Json(response))
}
@@ -0,0 +1,141 @@
use std::fmt;
use std::sync::Arc;
use actix_web::error::ResponseError;
use actix_web::web::Json;
use actix_web::http::StatusCode;
use actix_web::web::Path;
use actix_web::{web, HttpRequest};
use chrono::{DateTime, Utc};
use log::warn;
use crate::http_server::common_responses::user_details_lite::UserDetailsLight;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::voices::list_voices_by_username::list_zs_voices_by_username;
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::state::server_state::ServerState;
#[derive(Serialize, Clone)]
pub struct Voice {
voice_token: ZsVoiceToken,
title: String,
ietf_language_tag: String,
ietf_primary_language_subtag: String,
creator: UserDetailsLight,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
#[derive(Serialize)]
pub struct ListVoicesByUserSuccessResponse {
pub success: bool,
pub voices: Vec<Voice>,
}
#[derive(Deserialize)]
pub struct ListVoicesByUserPathInfo {
username: String,
}
#[derive(Debug)]
pub enum ListVoicesByUserError {
NotAuthorized,
ServerError,
}
impl fmt::Display for ListVoicesByUserError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{:?}", self)
}
}
impl ResponseError for ListVoicesByUserError {
fn status_code(&self) -> StatusCode {
match *self {
ListVoicesByUserError::NotAuthorized => StatusCode::UNAUTHORIZED,
ListVoicesByUserError::ServerError => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
pub async fn list_voices_by_user_handler(
http_request: HttpRequest,
path: Path<ListVoicesByUserPathInfo>,
server_state: web::Data<Arc<ServerState>>
) -> Result<Json<ListVoicesByUserSuccessResponse>, ListVoicesByUserError> {
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
ListVoicesByUserError::ServerError
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(ListVoicesByUserError::NotAuthorized);
}
};
let username = path.username.as_ref();
let creator_user_token = user_session.user_token.as_str().to_string();
let _is_mod = user_session.can_ban_users;
// NB(bt,2024-01-18): Showing mods deleted files is actually kind of annoying!
// We should focus on visibility-related controls instead.
const CAN_SEE_DELETED : bool = false;
let query_results = list_zs_voices_by_username(
&server_state.mysql_pool,
&username,
CAN_SEE_DELETED,
).await.map_err(|e| {
warn!("Error querying for voices: {:?}", e);
ListVoicesByUserError::ServerError
});
let voices = match query_results {
Ok(voices) => voices,
Err(e) => {
warn!("Error querying for voices: {:?}", e);
return Err(ListVoicesByUserError::ServerError);
}
};
let voices = voices.into_iter().map(|voice| {
Voice {
voice_token: voice.voice_token,
title: voice.title,
ietf_language_tag: voice.ietf_language_tag,
ietf_primary_language_subtag: voice.ietf_primary_language_subtag,
creator: UserDetailsLight::from_db_fields(
&voice.creator_user_token,
&voice.creator_username,
&voice.creator_display_name,
&voice.creator_email_gravatar_hash,
),
creator_set_visibility: voice.creator_set_visibility,
created_at: voice.created_at,
updated_at: voice.updated_at,
}
}).collect();
let response = ListVoicesByUserSuccessResponse {
success: true,
voices,
};
Ok(Json(response))
}
@@ -0,0 +1,8 @@
pub mod create_voice;
pub mod delete_voice;
pub mod list_available_voices;
pub mod list_voices_by_session;
pub mod list_voices_by_user;
pub mod search_voices;
pub mod update_voice;
pub mod get_voice;
@@ -0,0 +1,6 @@
use actix_web::HttpResponse;
pub async fn search_voices() -> HttpResponse {
// Implementation for searching voices with keywords
HttpResponse::Ok().json("Search results for voices")
}
@@ -0,0 +1,168 @@
use std::fmt;
use std::sync::Arc;
use actix_web::web::Path;
use actix_web::web::Json;
use actix_web::{web, HttpRequest};
use log::{error, warn};
use artcraft_api_defs::common::responses::simple_generic_json_success::SimpleGenericJsonSuccess;
use enums::common::visibility::Visibility;
use mysql_queries::queries::voice_designer::voices::get_voice::get_voice_by_token;
use mysql_queries::queries::voice_designer::voices::update_voice::{update_voice, UpdateVoiceArgs};
use tokens::tokens::zs_voices::ZsVoiceToken;
use crate::configs::supported_languages_for_models::get_canonicalized_language_tag_for_model;
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
#[derive(Deserialize)]
pub struct UpdateVoiceRequest {
// ========== Author + Moderator options ==========
pub title: Option<String>,
pub creator_set_visibility: Option<String>,
// NB: We calculate 'ietf_primary_language_subtag' from this value.
pub ietf_language_tag: Option<String>,
}
#[derive(Serialize)]
pub struct UpdateVoiceResponse {
pub success: bool,
}
/// For the URL PathInfo
#[derive(Deserialize)]
pub struct UpdateVoicePathInfo {
voice_token: String,
}
// =============== Error Response ===============
// NB: Not using derive_more::Display since Clion doesn't understand it.
// =============== Handler ===============
pub async fn update_voice_handler(
http_request: HttpRequest,
path: Path<UpdateVoicePathInfo>,
request: Json<UpdateVoiceRequest>,
server_state: web::Data<Arc<ServerState>>) -> Result<Json<SimpleGenericJsonSuccess>, CommonWebError>
{
let maybe_user_session = server_state
.session_checker
.maybe_get_user_session(&http_request, &server_state.mysql_pool)
.await
.map_err(|e| {
warn!("Session checker error: {:?}", e);
CommonWebError::from_error(e)
})?;
let user_session = match maybe_user_session {
Some(session) => session,
None => {
warn!("not logged in");
return Err(CommonWebError::NotAuthorized);
}
};
let voice_token = path.voice_token.clone();
let is_mod = user_session.can_ban_users;
let voice_lookup_result = get_voice_by_token(
&ZsVoiceToken::new(voice_token.clone()),
is_mod,
&server_state.mysql_pool,
).await;
let voice = match voice_lookup_result {
Ok(Some(voice)) => voice,
Ok(None) => {
warn!("Voice not found: {:?}", voice_token);
return Err(CommonWebError::NotFound);
},
Err(err) => {
warn!("Error looking up voice: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
// let is_creator = voice.maybe_creator_user_token == Some(user_session.user_token);
let is_creator = voice.maybe_creator_user_token.as_ref()
.map(|creator_user_token| creator_user_token == &user_session.user_token)
.unwrap_or(false);
if !is_creator && !is_mod {
warn!("user is not allowed to edit this voice: {:?}", user_session.user_token);
return Err(CommonWebError::NotAuthorized);
}
let mut title = None;
let mut ietf_language_tag = None;
let mut ietf_primary_language_subtag = None;
let mut creator_set_visibility = Visibility::Public;
if let Some(payload) = request.title.as_deref() {
if user_input_common::check_for_slurs::contains_slurs(payload) {
return Err(CommonWebError::BadInputWithSimpleMessage("title contains slurs".to_string()));
}
title = Some(payload.trim().to_string());
}
if let Some(tag) = request.ietf_language_tag.as_deref() {
// eg. en, en-US, es-419, ja-JP, etc.
let maybe_full_canonical_tag = get_canonicalized_language_tag_for_model(tag);
// eg. en, es, ja, etc.
let maybe_primary_language_subtag = maybe_full_canonical_tag
.map(|t| language_tags::LanguageTag::parse(t)
.map(|language_tag| language_tag.primary_language().to_string())
)
.transpose()
.map_err(|e| {
error!("Error parsing language tag '{}': {:?}", tag, e);
CommonWebError::BadInputWithSimpleMessage("bad locale string".to_string())
})?;
if let Some(full_tag) = maybe_full_canonical_tag {
if let Some(primary_subtag) = maybe_primary_language_subtag.as_deref() {
ietf_language_tag = Some(full_tag.to_string());
ietf_primary_language_subtag = Some(primary_subtag.to_string());
}
}
}
if let Some(visibility) = request.creator_set_visibility.as_deref() {
creator_set_visibility = Visibility::from_str(visibility)
.map_err(|_| CommonWebError::BadInputWithSimpleMessage("bad record visibility".to_string()))?;
}
let mut maybe_mod_user_token = None;
if is_mod {
maybe_mod_user_token = Some(user_session.user_token.as_str().to_string());
}
let query_result = update_voice(
UpdateVoiceArgs {
voice_token: &ZsVoiceToken::new(voice_token.clone()),
voice_title: title.as_deref(),
creator_set_visibility: &creator_set_visibility,
maybe_mod_user_token: maybe_mod_user_token.as_deref(),
ietf_language_tag: ietf_language_tag.as_deref(),
ietf_primary_language_subtag: ietf_primary_language_subtag.as_deref(),
mysql_pool: &server_state.mysql_pool
}
).await;
match query_result {
Ok(_) => {},
Err(err) => {
warn!("Update Voice DB error: {:?}", err);
return Err(CommonWebError::from_anyhow_error(err));
}
};
Ok(Json(SimpleGenericJsonSuccess { success: true }))
}
@@ -13,7 +13,6 @@ use beeble_client::webhook_api::beeble_webhook_payload::{
parse_beeble_webhook, BeebleWebhookPayload, BeebleWebhookStatus,
};
use bucket_paths::legacy::typified_paths::public::media_files::bucket_file_path::MediaFileBucketPath;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use enums::by_table::generic_inference_jobs::frontend_failure_category::FrontendFailureCategory;
use enums::by_table::media_files::media_file_class::MediaFileClass;
@@ -102,7 +101,7 @@ pub async fn beeble_webhook_handler(
apriori_debug_log_event_token: Some(debug_log_event_token),
maybe_creator_user_token: job.maybe_creator_user_token.as_ref(),
debug_log_type: DebugLogType::BeebleWebhook,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: None,
maybe_url: None,
message: &raw_body,
@@ -1,7 +1,6 @@
use crate::http_server::common_responses::common_web_error::CommonWebError;
use crate::state::server_state::ServerState;
use actix_web::web::Json;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use enums::by_table::generic_inference_jobs::frontend_failure_category::FrontendFailureCategory;
use fal_client::webhook_payload::raw::webhook_error_type::WebhookErrorType;
@@ -53,7 +52,7 @@ pub async fn handle_failed_fal_webhook(
apriori_debug_log_event_token: Some(debug_log_event_token),
maybe_creator_user_token: job.maybe_creator_user_token.as_ref(),
debug_log_type: DebugLogType::FalWebhook,
maybe_log_level: Some(DebugLogLevel::Error),
maybe_log_level: None,
maybe_ip_address: None,
maybe_url: None,
message: raw_body,
@@ -1,5 +1,4 @@
use actix_web::web::Json;
use enums::by_table::debug_logs::debug_log_level::DebugLogLevel;
use enums::by_table::debug_logs::debug_log_type::DebugLogType;
use fal_client::webhook_payload::hydrated::hydrated_webhook_contents::WebhookSuccessData;
use http_server_common::response::response_success_helpers::SimpleGenericJsonSuccess;
@@ -53,7 +52,7 @@ pub async fn handle_successful_fal_webhook(
apriori_debug_log_event_token: Some(debug_log_event_token),
maybe_creator_user_token: job.maybe_creator_user_token.as_ref(),
debug_log_type: DebugLogType::FalWebhook,
maybe_log_level: Some(DebugLogLevel::Info),
maybe_log_level: None,
maybe_ip_address: None,
maybe_url: None,
message: raw_body,
@@ -44,7 +44,6 @@ pub(crate) fn check_common_web_error(
.set_urgency(Some(NotificationUrgency::Medium))
.set_http_method(Some(method.to_string()))
.set_http_path(Some(path.to_string()))
.set_trace_id(metadata.trace_id.clone())
.set_http_status_code(Some(500))
.set_request_ip_address(metadata.request_ip_address.clone())
.set_avt_cookie_token(metadata.avt_cookie_token.clone())
@@ -30,7 +30,6 @@ pub(crate) fn check_status_code_fallback(
.set_urgency(Some(NotificationUrgency::Medium))
.set_http_method(Some(method.to_string()))
.set_http_path(Some(path.to_string()))
.set_trace_id(metadata.trace_id.clone())
.set_http_status_code(Some(status_code))
.set_request_ip_address(metadata.request_ip_address.clone())
.set_avt_cookie_token(metadata.avt_cookie_token.clone())
@@ -2,12 +2,9 @@ use std::sync::Arc;
use actix_web::dev::ServiceRequest;
use actix_web::web;
use actix_web::HttpMessage;
use http_server_common::request::get_request_ip::get_request_ip;
use log::debug;
use trace_id::TraceId;
use crate::state::server_state::ServerState;
/// Optional per-request context extracted by the error alerting middleware
@@ -19,8 +16,6 @@ pub(crate) struct RequestDebuggingMetadata {
pub avt_cookie_token: Option<String>,
pub session_token: Option<String>,
pub session_user_token: Option<String>,
/// Set by `TraceIdMiddleware` (outermost middleware) on every request.
pub trace_id: Option<String>,
}
impl RequestDebuggingMetadata {
@@ -31,10 +26,6 @@ impl RequestDebuggingMetadata {
let request_ip_address = Some(get_request_ip(http_request));
let trace_id = http_request.extensions()
.get::<TraceId>()
.map(|t| t.to_string());
let server_state = req.app_data::<web::Data<Arc<ServerState>>>();
let avt_cookie_token = server_state
@@ -67,7 +58,6 @@ impl RequestDebuggingMetadata {
avt_cookie_token,
session_token,
session_user_token,
trace_id,
}
}
}
@@ -1,4 +1,3 @@
pub mod error_alerting_middleware;
pub mod metrics_middleware;
pub mod pushback_filter_middleware;
pub mod trace_id_middleware;
@@ -1 +0,0 @@
pub mod trace_id_middleware;
@@ -1,95 +0,0 @@
//! Per-request trace id middleware.
//!
//! Generates a unique [`TraceId`] for every inbound HTTP request and makes it
//! available three ways:
//!
//! 1. **Request extensions** — downstream middleware and handlers can read
//! it via `req.extensions().get::<TraceId>()`.
//! 2. **Tokio task-local** — the whole request runs inside
//! [`trace_id::TRACE_ID_TASK_LOCAL`], so `trace_id::current_trace_id()`
//! works from anywhere in the request's task. The env-logger format
//! hook uses this to stamp every log line, and the pager uses it to tag
//! notifications.
//! 3. **`x-trace-id` response header** — so clients and support tickets can
//! quote the id.
//!
//! Register this middleware LAST in the actix `.wrap(...)` chain so it runs
//! FIRST (actix middleware executes in reverse registration order) and every
//! other middleware sees the trace id.
//!
//! NB: Detached `tokio::spawn` tasks do NOT inherit the task-local. Wrap
//! spawned futures in `TRACE_ID_TASK_LOCAL.scope(trace_id, fut)` if they
//! should stay correlated.
use std::future::{Future, Ready, ready};
use std::pin::Pin;
use actix_web::Error;
use actix_web::HttpMessage;
use actix_web::dev::{Service, ServiceRequest, ServiceResponse, Transform};
use actix_web::http::header::{HeaderName, HeaderValue};
use trace_id::{TraceId, TRACE_ID_TASK_LOCAL};
/// Response header carrying the request's trace id.
pub const TRACE_ID_HEADER: &str = "x-trace-id";
#[derive(Clone, Default)]
pub struct TraceIdMiddleware;
impl<S, B> Transform<S, ServiceRequest> for TraceIdMiddleware
where
S: Service<ServiceRequest, Response = ServiceResponse<B>, Error = Error> + 'static,
S::Future: 'static,
B: 'static,
{
type Response = ServiceResponse<B>;
type Error = Error;
type InitError = ();
type Transform = TraceIdService<S>;
type Future = Ready<Result<Self::Transform, Self::InitError>>;
fn new_transform(&self, service: S) -> Self::Future {
ready(Ok(TraceIdService { service }))
}
}
pub struct TraceIdService<S> {
service: S,
}
impl<S, B> Service<ServiceRequest> for TraceIdService<S>
where
S: Service<ServiceRequest, Response = ServiceResponse<B>, Error = Error> + 'static,
S::Future: 'static,
B: 'static,
{
type Response = ServiceResponse<B>;
type Error = Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>>>>;
actix_service::forward_ready!(service);
fn call(&self, req: ServiceRequest) -> Self::Future {
let trace_id = TraceId::generate();
// Make the trace id readable from the request itself.
req.extensions_mut().insert(trace_id.clone());
let response_trace_id = trace_id.clone();
let fut = self.service.call(req);
Box::pin(TRACE_ID_TASK_LOCAL.scope(trace_id, async move {
let mut response = fut.await?;
if let Ok(header_value) = HeaderValue::from_str(response_trace_id.as_str()) {
response.headers_mut().insert(
HeaderName::from_static(TRACE_ID_HEADER),
header_value,
);
}
Ok(response)
}))
}
}
@@ -18,11 +18,14 @@ use crate::http_server::deprecated_endpoints::engine::update_scene_handler::upda
use crate::http_server::deprecated_endpoints::events::list_events::list_events_handler;
use crate::http_server::deprecated_endpoints::flags::design_refresh_flag::disable_design_refresh_flag_handler::disable_design_refresh_flag_handler;
use crate::http_server::deprecated_endpoints::flags::design_refresh_flag::enable_design_refresh_flag_handler::enable_design_refresh_flag_handler;
use crate::http_server::deprecated_endpoints::investor_demo::disable_demo_mode_handler::disable_demo_mode_handler;
use crate::http_server::deprecated_endpoints::investor_demo::enable_demo_mode_handler::enable_demo_mode_handler;
use crate::http_server::deprecated_endpoints::leaderboard::get_leaderboard::leaderboard_handler;
use crate::http_server::deprecated_endpoints::media_uploads::list_user_media_uploads_of_type::list_user_media_uploads_of_type_handler;
use crate::http_server::deprecated_endpoints::media_uploads::upload_audio::upload_audio_handler;
use crate::http_server::deprecated_endpoints::media_uploads::upload_image::upload_image_handler;
use crate::http_server::deprecated_endpoints::media_uploads::upload_media::upload_media_handler;
use crate::http_server::deprecated_endpoints::mocap::enqueue_mocapnet::enqueue_mocapnet_handler;
use crate::http_server::deprecated_endpoints::vocoders::get_vocoder::get_vocoder_handler;
use crate::http_server::deprecated_endpoints::vocoders::list_vocoders::list_vocoders_handler;
use crate::http_server::endpoints::app_state::get_app_state_handler::get_app_state_handler;
@@ -32,11 +35,34 @@ use crate::http_server::endpoints::misc::enable_alpha_easy_handler::enable_alpha
use crate::http_server::endpoints::misc::enable_alpha_handler::enable_alpha_handler;
use crate::http_server::endpoints::stats::get_unified_queue_stats_handler::get_unified_queue_stats_handler;
use crate::http_server::endpoints::trending::list_trending_tts_models::list_trending_tts_models_handler;
use crate::http_server::endpoints::voice_clone_requests::check_if_voice_clone_request_submitted::check_if_voice_clone_request_submitted_handler;
use crate::http_server::endpoints::voice_clone_requests::create_voice_clone_request::create_voice_clone_request_handler;
use crate::http_server::endpoints::voice_designer::inference::enqueue_tts_request::enqueue_tts_request;
use crate::http_server::endpoints::voice_designer::inference::enqueue_vc_request::enqueue_vc_request;
use crate::http_server::endpoints::voice_designer::voice_dataset_samples::delete_sample::delete_sample_handler;
use crate::http_server::endpoints::voice_designer::voice_dataset_samples::list_samples_by_dataset::list_samples_by_dataset_handler;
use crate::http_server::endpoints::voice_designer::voice_dataset_samples::upload_zs_sample::upload_zs_sample_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::create_dataset::create_dataset_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::delete_dataset::delete_dataset_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::get_dataset::get_dataset_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::list_datasets_by_session::list_datasets_by_session_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::list_datasets_by_user::list_datasets_by_user_handler;
use crate::http_server::endpoints::voice_designer::voice_datasets::update_dataset::update_dataset_handler;
use crate::http_server::endpoints::voice_designer::voices::create_voice::create_voice_handler;
use crate::http_server::endpoints::voice_designer::voices::delete_voice::delete_voice_handler;
use crate::http_server::endpoints::voice_designer::voices::get_voice::get_voice_handler;
use crate::http_server::endpoints::voice_designer::voices::list_available_voices::list_available_voices_handler;
use crate::http_server::endpoints::voice_designer::voices::list_voices_by_session::list_voices_by_session_handler;
use crate::http_server::endpoints::voice_designer::voices::list_voices_by_user::list_voices_by_user_handler;
use crate::http_server::endpoints::voice_designer::voices::search_voices::search_voices;
use crate::http_server::endpoints::voice_designer::voices::update_voice::update_voice_handler;
use crate::http_server::routes::legacy_routes::beta_key_routes::add_beta_key_routes;
use crate::http_server::routes::legacy_routes::desktop_vc_app_routes::add_desktop_vc_app_routes;
use crate::http_server::routes::legacy_routes::image_studio_routes::add_image_studio_routes;
use crate::http_server::routes::legacy_routes::model_download_routes::add_model_download_routes;
use crate::http_server::routes::legacy_routes::studio_gen2_routes::add_studio_gen2_routes;
use crate::http_server::routes::legacy_routes::tts_routes::add_tts_routes;
use crate::http_server::routes::legacy_routes::voice_conversion_routes::add_voice_conversion_routes;
use crate::http_server::routes::legacy_routes::workflow_routes::add_workflow_routes;
use actix_helpers::route_builder::RouteBuilder;
use actix_http::body::MessageBody;
@@ -60,9 +86,13 @@ where
app = add_remote_download_routes(app); // /v1/remote_downloads (prev. /retrieval, aka. "generic_download_jobs")
app = add_category_routes(app); // /category
app = add_api_token_routes(app); // /api_tokens
app = add_voice_clone_request_routes(app); // /voice_clone_requests
app = add_investor_demo_routes(app); // /demo_mode // TODO: DEFINITELY TEMPORARY
app = add_flag_routes(app); // /flag
app = add_voice_designer_routes(app); // /v1/voice_designer
app = add_beta_key_routes(app); // /v1/beta_keys
app = add_model_download_routes(app);
app = add_studio_gen2_routes(app);
app = add_image_studio_routes(app);
app = add_workflow_routes(app);
app = add_engine_routes(app); // /v1/engine/...
@@ -72,6 +102,7 @@ where
// Remaining FakeYou surface area
app = add_tts_routes(app); // /tts
app = add_voice_conversion_routes(app); // /v1/voice_conversion
// ==================== Animations ====================
@@ -80,6 +111,11 @@ where
.add_post("/v1/animation/rerender/create", enqueue_rerender_animation_handler)
.into_app();
// ==================== Mocap ========================
let mut app = RouteBuilder::from_app(app)
.add_post("/v1/mocap/mocapnet/create", enqueue_mocapnet_handler)
.into_app();
// ==================== Format Conversion ====================
let mut app = RouteBuilder::from_app(app)
@@ -281,7 +317,55 @@ where
)
}
// ==================== API TOKEN ROUTES ====================
fn add_voice_clone_request_routes<T, B> (app: App<T>) -> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(web::scope("/voice_clone_requests")
.service(web::resource("/create")
.route(web::post().to(create_voice_clone_request_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/check")
.route(web::post().to(check_if_voice_clone_request_submitted_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
}
// ==================== INVESTOR DEMO MODE ====================
fn add_investor_demo_routes<T, B> (app: App<T>) -> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(web::scope("/demo_mode")
.service(web::resource("/enable")
.route(web::get().to(enable_demo_mode_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/disable")
.route(web::get().to(disable_demo_mode_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
}
// ==================== FLAG ROUTES ====================
@@ -416,4 +500,53 @@ where
}
fn add_voice_designer_routes<T,B> (app:App<T>)-> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(
web::scope("/v1/voice_designer")
.service(
web::scope("/dataset")
.route("/create", web::post().to(create_dataset_handler))
.service(web::resource("/{dataset_token}/update")
.route(web::post().to(update_dataset_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.route("/{dataset_token}", web::get().to(get_dataset_handler))
.route("/{dataset_token}/delete", web::delete().to(delete_dataset_handler))
.route("/user/{username}/list", web::get().to(list_datasets_by_user_handler))
.route("/session/list", web::get().to(list_datasets_by_session_handler))
)
.service(
web::scope("/voice")
.route("/list", web::get().to(list_available_voices_handler))
.route("/search", web::post().to(search_voices))
.route("/create", web::post().to(create_voice_handler))
.route("/{voice_token}", web::get().to(get_voice_handler))
.route("/{voice_token}/update", web::post().to(update_voice_handler))
.route("/{voice_token}/delete", web::delete().to(delete_voice_handler))
.route("/user/{username}/list", web::get().to(list_voices_by_user_handler))
.route("/session/list", web::get().to(list_voices_by_session_handler))
)
.service(
web::scope("/sample")
.route("/upload", web::post().to(upload_zs_sample_handler))
.route("/{sample_token}/delete", web::delete().to(delete_sample_handler))
.route("/dataset/{dataset_token}/list", web::get().to(list_samples_by_dataset_handler))
)
.service(
web::scope("/inference")
.route("/enqueue_tts", web::post().to(enqueue_tts_request))
.route("/enqueue_vc", web::post().to(enqueue_vc_request))
)
)
}
@@ -3,5 +3,7 @@ mod beta_key_routes;
mod desktop_vc_app_routes;
mod image_studio_routes;
mod model_download_routes;
mod studio_gen2_routes;
mod tts_routes;
mod voice_conversion_routes;
mod workflow_routes;
@@ -0,0 +1,24 @@
use crate::http_server::endpoints::studio_gen2::enqueue_studio_gen2_handler::enqueue_studio_gen2_handler;
use actix_http::body::MessageBody;
use actix_service::ServiceFactory;
use actix_web::dev::{ServiceRequest, ServiceResponse};
use actix_web::{web, App, Error, HttpResponse};
pub fn add_studio_gen2_routes<T, B> (app: App<T>) -> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(web::scope("/v1/studio_gen2")
.service(web::resource("/enqueue")
.route(web::post().to(enqueue_studio_gen2_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
}
@@ -8,6 +8,7 @@ use crate::http_server::endpoints::tts::delete_tts_model::delete_tts_model_handl
use crate::http_server::endpoints::tts::delete_tts_result::delete_tts_inference_result_handler;
use crate::http_server::endpoints::tts::edit_tts_model::edit_tts_model_handler;
use crate::http_server::endpoints::tts::edit_tts_result::edit_tts_inference_result_handler;
use crate::http_server::endpoints::tts::enqueue_infer_f5_tts_handler::enqueue_infer_f5_tts_handler::enqueue_infer_f5_tts_handler;
use crate::http_server::endpoints::tts::enqueue_infer_tts_handler::enqueue_infer_tts_handler::enqueue_infer_tts_handler;
use crate::http_server::endpoints::tts::enqueue_upload_tts_model::upload_tts_model_handler;
use crate::http_server::endpoints::tts::get_pending_tts_inference_job_count::get_pending_tts_inference_job_count_handler;
@@ -37,6 +38,10 @@ where
.route(web::post().to(enqueue_infer_tts_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(web::resource("/v1/tts/f5_inference")
.route(web::post().to(enqueue_infer_f5_tts_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
// NB(bt,2024-04-03): Older root /tts/* routes should be drained of traffic. They are used in the Twitch Streamer API though.
.service(
web::scope("/tts")
@@ -0,0 +1,41 @@
use actix_http::body::MessageBody;
use actix_service::ServiceFactory;
use actix_web::dev::{ServiceRequest, ServiceResponse};
use actix_web::error::Error;
use actix_web::{web, App, HttpResponse};
use crate::http_server::endpoints::voice_conversion::enqueue_seed_vc_inference_handler::enqueue_infer_seed_vc_handler;
use crate::http_server::endpoints::voice_conversion::enqueue_voice_conversion_inference_handler::enqueue_voice_conversion_inference_handler;
use crate::http_server::endpoints::voice_conversion::list_voice_conversion_models_handler::list_voice_conversion_models_handler;
pub fn add_voice_conversion_routes<T, B> (app: App<T>) -> App<T>
where
B: MessageBody,
T: ServiceFactory<
ServiceRequest,
Config = (),
Response = ServiceResponse<B>,
Error = Error,
InitError = (),
>,
{
app.service(
web::scope("/v1/voice_conversion")
.service(
web::resource("/inference")
.route(web::post().to(enqueue_voice_conversion_inference_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(
web::resource("/seed_vc_inference")
.route(web::post().to(enqueue_infer_seed_vc_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
.service(
web::resource("/model_list")
.route(web::get().to(list_voice_conversion_models_handler))
.route(web::head().to(|| HttpResponse::Ok()))
)
)
}
+3 -18
View File
@@ -26,7 +26,6 @@ use actix_helpers::middleware::banned_cidr_filter::banned_cidr_filter::BannedCid
use actix_helpers::middleware::banned_ip_filter::banned_ip_filter::BannedIpFilter;
use actix_helpers::middleware::disabled_endpoint_filter::disabled_endpoint_filter::DisabledEndpointFilter;
use actix_multipart::form::MultipartFormConfig;
use actix_web::HttpMessage;
use actix_web::middleware::{DefaultHeaders, Logger};
use actix_web::{middleware, web, App, HttpServer};
use billing_component::stripe::traits::internal_product_to_stripe_lookup::InternalProductToStripeLookup;
@@ -47,7 +46,6 @@ use crate::billing::stripe_internal_user_lookup_impl::StripeInternalUserLookupIm
use crate::http_server::middleware::error_alerting_middleware::error_alerting_middleware::ErrorAlertingMiddleware;
use crate::http_server::middleware::metrics_middleware::metrics_middleware::MetricsMiddleware;
use crate::http_server::middleware::pushback_filter_middleware::PushbackFilter;
use crate::http_server::middleware::trace_id_middleware::trace_id_middleware::TraceIdMiddleware;
use crate::http_server::routes::add_routes::add_routes;
use crate::http_server::web_utils::handle_multipart_error::handle_multipart_error;
use crate::startup::build_dependencies::setup_dependencies;
@@ -67,10 +65,9 @@ pub mod state;
pub mod threads;
pub mod util;
// Report cloudflare trace ID header (CF-Ray) and our request trace id in logs.
// %{trace_id}xi is resolved by `custom_request_replace` below.
// Report cloudflare trace ID header (CF-Ray) in logs
const LOG_FORMAT: &str =
"[%{HOSTNAME}e] IP=[%{X-Forwarded-For}i] %{CF-Ray}i %{trace_id}xi \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\" %T";
"[%{HOSTNAME}e] IP=[%{X-Forwarded-For}i] %{CF-Ray}i \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\" %T";
#[actix_web::main]
async fn main() -> AnyhowResult<()> {
@@ -272,21 +269,9 @@ pub async fn serve(
.wrap(BannedIpFilter::new(ip_ban_list))
.wrap(BannedCidrFilter::new(cidr_ban_set))
.wrap(Logger::new(LOG_FORMAT)
// NB: The access-log line is written after the response body completes,
// which is OUTSIDE the trace-id task-local scope — so we resolve the
// trace id from request extensions at request time instead.
.custom_request_replace("trace_id", |req| {
req.extensions()
.get::<trace_id::TraceId>()
.map(|t| t.to_string())
.unwrap_or_else(|| "-".to_string())
})
.exclude("/liveness")
.exclude("/readiness"))
.wrap(middleware::Compress::default())
// NB: Registered last => runs FIRST. Every request gets a trace id
// before any other middleware or handler executes.
.wrap(TraceIdMiddleware);
.wrap(middleware::Compress::default());
add_routes(app, server_environment)
})
@@ -147,6 +147,10 @@ pub async fn setup_dependencies(server_hostname: &str) -> AnyhowResult<SetupResu
easyenv::get_env_duration_seconds_or_default("DATABASE_TTS_CATEGORY_LIST_CACHE_TTL_SECONDS", Duration::from_secs(60))
);
let tts_queue_length_cache = SingleItemTtlCache::create_with_duration(
easyenv::get_env_duration_seconds_or_default("TTS_QUEUE_LENGTH_CACHE_TTL_SECONDS", Duration::from_secs(30))
);
let tts_model_category_assignments_cache = SingleItemTtlCache::create_with_duration(
easyenv::get_env_duration_seconds_or_default("TTS_MODEL_CATEGORY_ASSIGNMENTS_CACHE_TTL_SECONDS", Duration::from_secs(60))
);
@@ -249,7 +253,12 @@ pub async fn setup_dependencies(server_hostname: &str) -> AnyhowResult<SetupResu
},
ephemeral: EphemeralInMemoryCaches {
tts_model_list: voice_list_cache,
voice_conversion_model_list: SingleItemTtlCache::create_with_duration(
easyenv::get_env_duration_seconds_or_default(
"VOICE_CONVERSION_MODEL_LIST_CACHE_TTL_SECONDS",
Duration::from_secs(60))),
database_tts_category_list: database_tts_category_list_cache,
tts_queue_length: tts_queue_length_cache,
tts_model_category_assignments: tts_model_category_assignments_cache,
leaderboard: leaderboard_cache,
inference_queue_length: inference_queue_length_cache,
@@ -13,11 +13,14 @@ pub fn setup_static_feature_flags(paging_flags: PagingFlags) -> AnyhowResult<Sta
global_429_pushback_filter_enabled: easyenv::get_env_bool_or_default("FF_GLOBAL_429_PUSHBACK_FILTER_ENABLED", false),
disable_unified_queue_stats_endpoint: easyenv::get_env_bool_or_default("FF_DISABLE_QUEUE_STATS_ENDPOINT", false),
disable_inference_queue_length_endpoint: easyenv::get_env_bool_or_default("FF_DISABLE_INFERENCE_QUEUE_LENGTH_ENDPOINT", false),
disable_tts_queue_length_endpoint: easyenv::get_env_bool_or_default("FF_DISABLE_TTS_QUEUE_LENGTH_ENDPOINT", false),
disable_tts_model_list_endpoint: easyenv::get_env_bool_or_default("FF_DISABLE_TTS_MODEL_LIST_ENDPOINT", false),
disable_voice_conversion_model_list_endpoint: easyenv::get_env_bool_or_default("FF_DISABLE_VOICE_CONVERSION_MODEL_LIST_ENDPOINT", false),
// Refresh rates
frontend_unified_queue_stats_refresh_interval_millis: easyenv::get_env_num("FF_FRONTEND_QUEUE_STATS_REFRESH_INTERVAL_MILLIS", 15_000)?,
frontend_pending_inference_refresh_interval_millis: easyenv::get_env_num("FF_FRONTEND_PENDING_INFERENCE_REFRESH_INTERVAL_MILLIS", 15_000)?,
frontend_pending_tts_refresh_interval_millis: easyenv::get_env_num("FF_FRONTEND_PENDING_TTS_REFRESH_INTERVAL_MILLIS", 15_000)?,
// Bans
troll_ban_user_percent: easyenv::get_env_num("FF_TROLL_BANNED_USER_PERCENT", 0)?,
@@ -29,6 +32,7 @@ pub fn setup_static_feature_flags(paging_flags: PagingFlags) -> AnyhowResult<Sta
// Disable voice features
disable_tts: easyenv::get_env_bool_or_default("DISABLE_TTS", false),
disable_voice_conversion: easyenv::get_env_bool_or_default("DISABLE_VOICE_CONVERSION", false),
// Paging
paging: paging_flags,
@@ -5,6 +5,7 @@ use crate::http_server::deprecated_endpoints::leaderboard::get_leaderboard::Lead
use crate::http_server::endpoints::media_files::list::list_featured_media_files_handler::ListFeaturedMediaFilesQueryParams;
use crate::http_server::endpoints::stats::result_transformer::CacheableQueueStats;
use crate::http_server::endpoints::tts::list_tts_models::TtsModelRecordForResponse;
use crate::http_server::endpoints::voice_conversion::list_voice_conversion_models_handler::VoiceConversionModel;
use crate::http_server::user_lookup::user_session::session_utils::session_checker::SessionChecker;
use crate::http_server::web_utils::redis_rate_limiter::RedisRateLimiter;
use crate::http_server::web_utils::scoped_temp_dir_creator::ScopedTempDirCreator;
@@ -33,6 +34,7 @@ use mysql_queries::mediators::firehose_publisher::FirehosePublisher;
use mysql_queries::queries::generic_inference::web::get_pending_inference_job_count::InferenceQueueLengthResult;
use mysql_queries::queries::media_files::list::list_featured_media_files::FeaturedMediaFileListPage;
use mysql_queries::queries::model_categories::list_categories_query_builder::CategoryList;
use mysql_queries::queries::tts::tts_inference_jobs::get_pending_tts_inference_job_count::TtsQueueLengthResult;
use opaque_cursors::v2::opaque_cursor_encoder_v2::OpaqueCursorEncoderV2;
use pager::client::pager::Pager;
use redis::Client;
@@ -188,6 +190,9 @@ pub struct EphemeralInMemoryCaches {
/// Contains a list of all TTS models.
pub tts_model_list: SingleItemTtlCache<Vec<TtsModelRecordForResponse>>,
/// Contains a list of all voice conversion models.
pub voice_conversion_model_list: SingleItemTtlCache<Vec<VoiceConversionModel>>,
/// Contains a list of all TTS categories in the database
/// (before any enrichment with synthetic categories)
/// This is used in several places (list categories, computed category assignments)
@@ -207,6 +212,11 @@ pub struct EphemeralInMemoryCaches {
/// vector clock.
pub inference_queue_length: SingleItemTtlCache<InferenceQueueLengthResult>,
/// TTS queue length
/// The frontend will consult a distributed cache and use the monotonic DB time as a
/// vector clock.
pub tts_queue_length: SingleItemTtlCache<TtsQueueLengthResult>,
pub leaderboard: SingleItemTtlCache<LeaderboardInfo>,
/// Cache of featured media files
@@ -241,9 +251,15 @@ pub struct StaticFeatureFlags {
/// Disable the live `/v1/model_inference/queue_length` endpoint for all users and serve a static value instead.
pub disable_inference_queue_length_endpoint: bool,
/// Disable the live `/tts/queue_length` endpoint for all users and serve a static value instead.
pub disable_tts_queue_length_endpoint: bool,
/// Disable the live `/tts/list` endpoint for all users and serve a static value instead.
pub disable_tts_model_list_endpoint: bool,
/// Disable the live `/v1/voice_conversion/model_list` endpoint for all users and serve a static value instead.
pub disable_voice_conversion_model_list_endpoint: bool,
/// Tell the frontend client how fast to refresh their view of queue stats.
/// During an attack, we may want this to go extremely slow.
pub frontend_unified_queue_stats_refresh_interval_millis: u64,
@@ -252,6 +268,10 @@ pub struct StaticFeatureFlags {
/// During an attack, we may want this to go extremely slow.
pub frontend_pending_inference_refresh_interval_millis: u64,
/// Tell the frontend client how fast to refresh their view of the pending TTS count.
/// During an attack, we may want this to go extremely slow.
pub frontend_pending_tts_refresh_interval_millis: u64,
/// For "troll banned" users, what percentage of the time will the service misbehave?
/// This should be a number over 100.
pub troll_ban_user_percent: u8,
@@ -275,6 +295,10 @@ pub struct StaticFeatureFlags {
/// If true, respond with 429.
pub disable_tts: bool,
/// Disable voice conversion endpoints
/// If true, respond with 429.
pub disable_voice_conversion: bool,
/// Paging system flags.
pub paging: PagingFlags,
}

Some files were not shown because too many files have changed in this diff Show More