further cleanup + restore some docs

.config/hakari.toml

@@ -0,0 +1,14 @@
+hakari-package = "workspace-build-acceleration"
+
+resolver = "1"
+
+dep-format-version = "4"
+workspace-hack-line-style = "workspace-dotted"
+
+platforms = [
+    "x86_64-unknown-linux-gnu",
+    "x86_64-apple-darwin",
+    "x86_64-pc-windows-msvc",
+]
+
+exact-versions = true

_docs/dev_setup_server.md

@@ -0,0 +1,105 @@
+# ArtCraft Server
+
+ArtCraft's server is a Rust / Actix app called `storyteller-web`.
+
+You don't need to run this to develop the ArtCraft application, but it can be useful to spin up
+a development instance for adding new server functions or as your own private local copy.
+
+`storyteller-web` can function entirely within the development environment, and ArtCraft can be
+pointed to a local instance of the server.
+
+**TODO**: Concisely describe setting up server components. Note: we have
+previous docs in `old/` that may still be relevant, though they're perhaps
+slightly out of date.
+
+## Server Setup (Mac)
+
+### Install and Migrate MySQL Database
+
+```bash
+# For now, the tooling doesn't support MySQL 9.6 as it was just released in January 2026.
+# This will probably change in the future to no longer need to specify the 8.* series.
+brew install mysql@8.4
+
+# Try to connect
+mysql -uroot
+
+# If connection fails, start service, then try again.
+brew services start mysql
+mysql -uroot
+
+# If that fails, reboot and try connecting again.
+```
+
+Create the `storyteller` database table. Connect to the database:
+
+```bash
+sudo mysql -u root
+
+# If it needs a password, use argument -p and when prompted use "root" as the typical default password
+```
+
+Then run this:
+
+```mysql
+use mysql;
+CREATE DATABASE storyteller;
+CREATE USER 'storyteller'@'localhost' IDENTIFIED BY 'password';
+GRANT ALL PRIVILEGES ON storyteller.* TO 'storyteller'@'localhost';
+```
+
+Then verify access with `./script/dev_mysql_connect.sh`
+
+```bash
+cargo install diesel_cli \
+  --no-default-features \
+  --features mysql,sqlite
+```
+
+If this fails,
+
+```bash
+brew install zstd
+brew install openssl
+```
+
+Possibly,
+
+```bash
+export RUSTFLAGS="-L$(brew --prefix zstd)/lib -L$(brew --prefix openssl)/lib"
+```
+
+In older Macs, there [have been some issues with Diesel CLI](https://github.com/diesel-rs/diesel/issues/2605)
+that require a few extra dependencies to be installed:
+
+```bash
+# If you're on Mac and the above command didn't work, run the following and then retry:
+brew install libpq
+```
+
+You're ready to run the migrations:
+
+```bash
+diesel migration run
+```
+
+You might get a scary message about `"Encountered unknown type for Mysql: enum"` -- you can safely ignore this
+error if you see it in isolation. It doesn't impact the migrations whatsoever.
+
+Finally, you'll need the sqlx CLI tool to run codegen. 
+You likely won't need this now, but if you change any queries, this will be necessary:
+
+```bash
+cargo install sqlx-cli --features rustls,mysql,sqlite
+```
+
+### Install Redis (Mac)
+
+```bash
+# Install Redis
+brew install redis
+
+# Start the service:
+brew services start redis
+```
+

_docs/old/local_development_setup_guide.md

@@ -0,0 +1,395 @@
+Local Development Setup Guide (for bare metal)
+==============================================
+
+This document describes how to run the FakeYou infrastructure on your development 
+machine's bare metal (no containers).
+
+Setup instructions and development were tested and known to work on the following
+platforms:
+
+- Mac OS (Apple M2 silicon)
+- Ubuntu 22.04
+- Windows Subsystem for Linux (Ubuntu 22.04 distro) aka "WSL"
+
+The applications (binary targets) you might be interested in running include, but 
+are not limited to:
+
+- `storyteller-web`, the HTTP API server
+- `download-job`, which downloads models from the web and stores them in GCS
+- `inference-job`, which downloads models from GCS and runs inference with them
+
+Setup Instructions
+------------------
+
+### (1) Install Rust
+
+Install Rust [using this guide](https://www.rust-lang.org/learn/get-started). If it asks, 
+you'll want "stable" Rust, not "nightly" Rust. If it doesn't ask, it defaults to "stable".
+
+For Windows Subsystem for Linux, you'll want the WSL version, not the `.exe` file!
+
+### (2) Install the necessary libraries
+
+#### Mac
+
+On an up-to-date M1/M2 Mac, there should be no libraries you need to install! Any exceptions 
+to this assumption should be documented here.
+
+#### Ubuntu 22.04 and WSL:
+
+If you're on Ubuntu or WSL, you'll need to install some libraries to build the Rust code: 
+
+```bash
+# If you haven't run apt before, you'll need to fetch the package list:
+sudo apt update
+
+# Install the dependencies
+sudo apt install jq \
+  libmysqlclient-dev \
+  libsqlite3-dev \
+  pkgconf
+```
+
+(Note: jq is for combining mysql codegen outputs; pkgconf is if using openssl instead of rustls.)
+
+### (3) Install MySQL server
+
+#### Mac
+
+```bash
+# Install MySQL
+brew install mysql
+
+# Try to connect
+mysql -uroot
+```
+
+In the event of failure to connect, try the following steps:
+
+```bash
+# If connection fails, start service
+brew services start mysql
+
+# Try to connect again 
+# If this fails, reboot the machine, eg. using `sudo reboot now`
+mysql -uroot
+```
+
+#### Ubuntu 22.04 and WSL
+
+```bash
+# Install MySQL
+sudo apt install mysql-server
+
+# Try to connect 
+# The default password is typically "root".
+sudo mysql -u root -p
+````
+
+In the event of failure to connect, try the following steps:
+
+```bash
+# If it failed, you may need to start MySQL (typically only on WSL, not on native Ubuntu)
+sudo service mysql start
+
+# You may also have to change some socket file permissions (again, typically only on WSL)
+sudo chmod g+rx /var/run/mysqld
+sudo usermod -aG mysql $USER
+newgrp mysql
+
+# Try to connect again
+# The default password is typically "root".
+# If this fails, try rebooting. If it still fails, read some of the suggestions at the bottom 
+# of this README.
+sudo mysql -u root -p
+```
+
+### (4) Install a `storyteller` user and table in MySQL
+
+First, connect to MySQL:
+
+```bash
+# If the following command asks for a password, the password is typically "root"
+sudo mysql -u root -p
+```
+
+Once in MySQL, run the following to set up the database and user accounts:
+
+```mysql
+use mysql;
+CREATE DATABASE storyteller;
+CREATE USER 'storyteller'@'localhost' IDENTIFIED BY 'password';
+GRANT ALL PRIVILEGES ON storyteller.* TO 'storyteller'@'localhost';
+```
+
+Then verify access with `./dev_mysql_connect.sh`
+
+### (5) Install Diesel CLI (for MySQL migrations):
+
+In order to run MySQL database migrations, you'll need to install the migration tool we use:
+
+```bash
+cargo install diesel_cli \
+  --no-default-features \
+  --features mysql,sqlite
+```
+
+In older Macs, there [have been some issues with Diesel CLI](https://github.com/diesel-rs/diesel/issues/2605)
+that require a few extra dependencies to be installed:
+
+```bash
+# If you're on Mac and the above command didn't work, run the following and then retry:
+brew install libpq
+```
+
+### (6) Run the pending database migrations:
+
+You're ready to run the migrations:
+
+```bash
+diesel migration run
+```
+
+You might get a scary message about `"Encountered unknown type for Mysql: enum"` -- you can safely ignore this 
+error if you see it in isolation. It doesn't impact the migrations whatsoever. (See notes at the bottom of 
+the README for details.)
+
+### (7) (optional) Install the SQLx CLI (if doing database development)
+
+(You can probably skip this step unless you're a backend engineer.) 
+
+If (and only if) you're doing Rust service development that changes the MySQL queries, then you should install 
+SQLx utilities. This tool is used to cache the column types and query plans for CI, as SQLx statically verifies 
+raw MySQL queries against the schema and Rust types.
+
+We'll be using diesel to manage the migrations, but sqlx within the app to actually perform queries.
+Diesel is an immature ORM, which isn't a good tech bet, so we use sqlx as at-compile-time
+typesafe SQL. 
+
+(TODO: Reevaluate this assessment for 2023-2024 and beyond. Maybe we do want Diesel now.)
+
+```bash
+# NB(bt,2023-10-04): "--no-default-features" not working
+# cargo install sqlx-cli --no-default-features --features rustls,mysql,sqlite
+cargo install sqlx-cli --features rustls,mysql,sqlite
+```
+
+### (8) Install hosts file:
+
+(This is the same for Linux, Mac, and WSL.)
+
+If you're developing against the frontend, it'll target development domains (eg. `dev.fakeyou.com`) instead 
+of `127.0.0.1` or `localhost`. You can make your machine route domains to localhost by editing your hosts 
+file (located at `/etc/hosts`) to include the following configuration lines:
+
+```
+# 1) Edit hosts file with your editor of choice
+#
+#  - sudo vi /etc/hosts
+#  - sudo nano /etc/hosts
+#
+# 2) Then paste:
+#
+
+127.0.0.1    dev.fakeyou.com
+127.0.0.1    api.dev.fakeyou.com
+127.0.0.1    devproxy.fakeyou.com
+
+127.0.0.1    dev.storyteller.ai
+127.0.0.1    api.dev.storyteller.ai
+127.0.0.1    devproxy.storyteller.ai
+```
+
+### (9) Install Redis
+
+#### Mac:
+
+```bash
+# Install Redis
+brew install redis
+
+# Start the service:
+brew services start redis
+```
+
+#### Ubuntu 22.04 and WSL
+
+```
+# Install Redis
+sudo apt install redis
+
+# Start the service (typically only needed on WSL):
+sudo service redis-server start
+```
+
+### (10) Install Elasticsearch (optional)
+
+(You can most likely skip this step.)
+
+Mac: (TODO - haven't installed yet)
+
+WSL: (TODO - haven't installed yet)
+
+Ubuntu 22.04:
+
+Install:
+
+```bash
+curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
+
+echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
+
+sudo apt update
+sudo apt install elasticsearch
+```
+
+Configure:
+
+```text
+# Edit /etc/elasticsearch/elasticsearch.yml
+# Set the following:
+network.host: localhost
+```
+
+Run:
+
+```bash
+sudo systemctl start elasticsearch
+sudo systemctl enable elasticsearch
+```
+
+Verify:
+
+```bash
+curl -X GET 'http://localhost:9200'
+```
+
+Install Kibana (prototyping UI):
+
+```bash
+# Install
+sudo apt-get install kibana
+
+# Run
+sudo /bin/systemctl daemon-reload
+sudo /bin/systemctl enable kibana.service
+sudo systemctl start kibana.service
+
+# Access the Kibana Status Page:
+# http://localhost:5601/status
+
+# Access the query console UI:
+# Click Sidebar > Management > Dev Tools > Console
+# http://localhost:5601/
+```
+
+Extra reading on setting up Elasticsearch: 
+
+- [Guide 1](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-22-04)
+- [Guide 2](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html)
+- [Kibana](https://www.elastic.co/guide/en/kibana/current/deb.html)
+
+### (11) Run one or more applications:
+
+To start the HTTP API server,
+
+```bash
+cargo run --bin storyteller-web
+```
+
+Note that this compiles and runs the "development" binary. It's faster and easier to debug than the
+optimized "production" build. To build a fully optimized production release,
+run `cargo build --release --bin storyteller-web` . Note that this will take much longer.
+
+You may notice that this might not work right off the bat. You may need to specify some environment variables 
+and supply some development secrets.
+
+Secrets: 
+
+- Global secrets (ask Brandon for these): 
+  - Put `.env-secrets` in the root `storyteller-rust` directory.
+  
+- Per app secrets (ask Brandon for these):
+  - Put `storteller-web.development-secrets.env` in `crates/service/web/storyteller_web/config`
+  - Put `inference-job.development-secrets.env` in `crates/service/job/inference_job/config`
+  - Put `donwload-job.development-secrets.env` in `crates/service/job/download_job/config`
+
+**DO NOT CHECK THESE SECRET FILES INTO THE REPO!**
+
+Environment config: 
+
+You may need to set local development paths for your environment. You can set these in `.env-secrets` too,
+so that they won't be updated for other engineers.
+
+```
+# this should be the directory containing all of the "storyteller" projects:
+STORYTELLER_ROOT = "/home/tensor/code/storyteller" # change this
+
+# The following are entirely optional, but if you want to fine tune where each project lives, you may do so:
+STORYTELLER_FRONTEND = "/somewhere/else/storyteller-frontend"
+STORYTELLER_ML = "storyteller-ml"
+STORYTELLER_RUST = "/Users/bt/Development/storyteller-rust"
+```
+
+To download some ML models, run:
+
+```bash
+cargo run --bin download-job
+cargo run --bin tts-download-job
+```
+
+If you want to run both the HTTP API and the jobs, you'll need to run both processes in different
+terminals or tmux sessions.
+
+Note also that the configurations pointing to the ML monorepo must be set up for each application.
+Additionally, some development secrets may be needed (ask the team to share).
+
+To execute ML inference, run:
+
+```bash
+cargo run --bin inference-job
+cargo run --bin tts-inference-job
+```
+
+Again, note that the configurations pointing to the ML monorepo must be set up for each application.
+Additionally, some development secrets may be needed (ask the team to share).
+
+Seeding the Database with Some Values
+-------------------------------------
+For voice designer you might want some pre-seeded audio that will hit the google cloud bucket, it will require env-secrets ask Brandon.
+cargo run --bin dev-database-seed -- --bucket
+
+Solutions to Common Setup Problems 
+-----------------------------------
+
+### [Fix] Sqlx Error when Performing Schema Migrations
+
+You might get this error message during migration,
+
+```
+Encountered unknown type for Mysql: enum
+thread 'main' panicked at 'internal error: entered unreachable code: Mysql only supports a closed set of types.
+                         If you ever see this error message please open an issue at https://github.com/diesel-rs/diesel 
+                         containing a dump of your schema definition.', diesel_cli/src/print_schema.rs:310:17
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+```
+
+I haven't found the cause (it doesn't happen on newer installs), but the migrations appear to work regardless 
+of this error message. You can essentially ignore it.
+
+### [Fix] Can't connect to local MySQL after install on Ubuntu
+
+- If MySql in local dev can't be connected to, [reset the accounts](https://linuxconfig.org/how-to-reset-root-mysql-mariadb-password-on-ubuntu-20-04-focal-fossa-linux).
+
+### API Server
+```
+cargo run --bin docs
+```
+This setups an api server with swagger docs to show you responses and requests with each associated endpoint
+It is run at http://127.0.0.1:8989/
+To obtain the json for the postman collection go to this url http://127.0.0.1:8989/api-docs/openapi.json
+We use this: `https://github.com/juhaku/utoipa`
+The documentation example is in: 
+`crates/service/web/storyteller_web/src/http_server/endpoints/voice_designer/inference/enqueue_tts_request.rs`
+Write into the api doc to: 
+`crates/service/web/storyteller_web/src/api_doc.rs`

_docs/old/old_docs/README.md

@@ -0,0 +1,54 @@
+README
+======
+
+This directory contains a collection of (hopefully organized and up to date) documentation on 
+running the various backend pieces, development setup, production maintenance, etc. Check out 
+all the markdown documents in this directory tree for information organized by topic.
+
+Development / Setup
+-------------------
+
+- [Most of the general documents for cross-cutting setup of FakeYou.com live here.](./local_development)
+- Some per-application documents (such as for `aichatbot-sidecar`) live as READMEs under that particular 
+  application's code directory, eg. `/crates/{appCategory}/{appName}/README.md` (These are irrelevant 
+  for most folks.)
+
+Web API Overview
+----------------
+
+### FakeYou.com
+
+* [FakeYou.com](https://fakeyou.com) is powered by the following applications:
+    * HTTP APIs:
+        * `storyteller-web` - HTTP API + database monolith for FakeYou
+    * Asynchronous Jobs:
+        * `download-job` - Handle async download of ML models (RVC, SVC, etc.)
+        * `inference-job` - Run async ML model inference (RVC, SVC, SadTalker, etc.) 
+                            This will handle most model inference going forward.
+        * `tts-download-job` - (*deprecated*) Async download of Tacotron2 models only
+        * `tts-inference-job` - (*deprecated*) Async TTS inference of Tacotron2 models only
+        * `w2l-download-job` - (*deprecated*) Async download of videos and images
+        * `w2l-inference-job` - (*deprecated*) Async W2L inference
+    * Documentation Server:
+        * `docs` - this will run a local swagger server with the endpoints that you have access to, http://127.0.0.1:8989/api-docs/openapi.json use this to port everything over to the post man.
+        
+### PowerStream
+
+* [PowerStream](https://power.stream) (deprecated) is powered by the following applications:
+    * HTTP APIs:
+        * `storyteller-web` - HTTP + database monolith for FakeYou ***and*** PowerStream.
+    * Twitch API integration:
+        * `twitch-pubsub-subscriber` - Dynamically subscribes to Twitch PubSub for "active" users
+    * Websocket events:
+        * `websocket-gateway` - Websocket for users' browsers to pull down enriched Twitch events.
+
+TODO
+----
+
+Notes / TODOs:
+
+* Examples for good Actix+Sqlx Tests:
+  https://stackoverflow.com/questions/65370752/how-do-i-create-an-actix-web-server-that-accepts-both-sqlx-database-pools-and-tr
+
+* Actix/sqlx runtime compat:
+  https://github.com/launchbadge/sqlx/issues/1117#issuecomment-801237734

_docs/old/old_docs/cheat_sheet/actix.md

@@ -0,0 +1,21 @@
+development notes
+=================
+
+Actix notes
+-----------
+
+json request
+```
+async fn handler(request: web::Json<Mytype>) -> impl Responder { "whatever" }
+```
+
+form-multipart request
+```
+async fn handler(request: web::Form<Mytype>) -> impl Responder { "whatever" }
+```
+
+route parameters
+```
+#[get("/{name}")]
+async fn hello(name: web::Path<String>) -> impl Responder { format!("Hi {}", name) }
+```

_docs/old/old_docs/cheat_sheet/migrations.md

@@ -0,0 +1,43 @@
+Migrations guide
+================
+
+## Diesel Migrations
+
+To run pending migrations, execute the following:
+
+```bash
+diesel migration run
+```
+
+To revert the last 3 migrations:
+
+```bash
+diesel migration revert -n 3
+```
+
+To reset the entire database (drop, migrate), run:
+
+```bash
+diesel database reset
+```
+
+## Server Query Codegen
+
+We use SQLx instead of Diesel in the production server. It's typesafe
+SQL instead of an ORM like Diesel.
+
+SQLx connects to a database to derive type information, but obviously
+cannot do this for builds in CI. In order to cache the types, we build
+and check in a cache file (necessary for builds):
+
+```bash
+SQLX_OFFLINE=true cargo sqlx prepare
+```
+
+Now that we have multiple binaries, it's required to include all the queries in the main
+binary so we can generate the cached queries as a single target. That's then executed
+with:
+
+```bash
+SQLX_OFFLINE=true cargo sqlx prepare -- --bin storyteller-web
+```

_docs/old/old_docs/local_development/local_development_k8s.md

@@ -0,0 +1,22 @@
+# Local Kubernetes environment for standing up the APIs and their dependencies
+
+(These are Paul's notes.)
+
+- Install minikube, kubectl, docker, and musl from your OS package manager
+- `rustup target add x86_64-unknown-linux-musl`
+- `cargo build --release --bin storyteller-web --target=x86_64-unknown-linux-musl`
+- `minikube start --extra-config=apiserver.service-node-port-range=1-65535`
+- `eval $(minikube docker-env)`
+- `minikube mount /path/to/storyteller-rust/:/storyteller-rust &> /dev/null &`
+- `docker build -t storyteller-web -f localdev/Dockerfile .`
+- `kubectl apply -f localdev/kubernetes.yml`
+- Get the ip address of minikube with `minikube ip`, this is where all the services will be running on their 
+regular ports (12345 for the api, 6379 for redis, 3306 for mysql)
+- That's it, we just spun up mysql, applied database migrations, spun up redis, and spun up our api
+
+# Development tips
+- Rebuild and "deploy" (locally) with `cargo build --release --bin storyteller-web --target=x86_64-unknown-linux-musl && kubectl rollout restart storyteller-web`
+- View logs from the api with `kubectl logs  storyteller-web<TAB>`
+- Delete everything minikube (if you start running out of disk space or what-have-you) with `minikube delete`
+- See pod status with `kubectl get pods`
+- If there are errors you can get more info with `kubectl describe pods`

_docs/old/old_docs/local_development/local_development_nginx_proxy.md

@@ -0,0 +1,56 @@
+Local Development nginx proxy
+=============================
+
+It isn't necessary to use nginx for development, but it can be nice. Nginx allows for a more 
+sophisticated testing setup that makes cookie and CORS issues less of a pain.
+
+You can use nginx to serve (locally-signed) SSL, redirect traffic to different ports, or run 
+multiple copies of the app side by side.
+
+### Hosts file
+
+Set the following configs in your `/etc/hosts` file (as you would probably do even without nginx),
+
+```
+127.0.0.1    dev.fakeyou.com
+127.0.0.1    api.dev.fakeyou.com
+127.0.0.1    devproxy.fakeyou.com
+
+127.0.0.1    dev.storyteller.ai
+127.0.0.1    api.dev.storyteller.ai
+127.0.0.1    devproxy.storyteller.ai
+
+# Deprecated development hostnames :
+127.0.0.1    jungle.horse
+127.0.0.1    api.jungle.horse
+```
+
+### Nginx installation and administration
+
+Configure Nginx per the checked in Nginx configs (and instructions) in
+the directory `/_develop/localdev/nginx-http-config`.
+
+This should be straightforward, but if it isn't, contribute docs here.
+
+### [Fix] Python 3.6 on Apple M1 Mac
+
+Python3.6 isn't supported on Apple silicon, and it's not in homebrew. It can be installed with 
+[nix using Rosetta](https://stackoverflow.com/a/65980989):
+
+Download: https://nixos.org/download.html#nix-quick-install
+
+```
+nix run nixpkgs.python36 -c python
+```
+
+Install venv:
+
+```
+nix run nixpkgs.python36 -c python -m venv python
+```
+
+Install other packages on Mac that aren't used in venv:
+
+```
+python3 -m pip install --user requests gdown youtube_dl
+```

_docs/old/old_docs/production/production_builds_and_deployment.md

@@ -0,0 +1,36 @@
+production builds and deployment
+================================
+
+### Github Docker builds
+
+The repository needs to be given read access to the base docker image:
+
+https://github.com/orgs/storytold/packages/container/docker-base-images-rust-ssl/settings
+
+These instructions assume running on GCP.
+
+### Database migrations
+
+1. Set `DATABASE_URL` in `.env` to the production secrets (DO NOT COMMIT!)
+2. Run `diesel migration run`
+
+### Setting up public buckets without list permission
+
+Public buckets that deny the `list` action should use the following Role:
+
+`roles/storage.legacyObjectReader`
+
+See:
+
+* https://stackoverflow.com/a/56354633
+* https://cloud.google.com/storage/docs/access-control/making-data-public#buckets
+
+### Generating Bucket Access Key and Secret Key
+
+https://cloud.telestream.net/tutorials/how-to-setting-up-google-cloud-storage/
+
+1. Go to the GCS page
+2. Click "settings"
+3. Click "interoperability" tab
+4. (enable interoperable access if not already set)
+5. Click "create new key"

_docs/old/old_docs/production/production_database.md

@@ -0,0 +1,17 @@
+Production Database
+===================
+
+Secrets can be subbed in for `.env` and the standard diesel migration tooling will work.
+
+Dump a MySQL report
+-------------------
+
+```bash
+mysql -u storyteller \
+  -pPasswordHere \
+  -h IP_HERE \
+  -P 3306 \
+  -D storyteller \
+  -B \
+  -e "select * from voice_clone_requests;" | sed "s/'/\'/;s/\t/\",\"/g;s/^/\"/;s/$/\"/;s/\n//g" > voice_clone_report.csv
+```

_docs/old/old_docs/schema/database_schema.md

@@ -0,0 +1,21 @@
+database schema
+================
+
+(This is woefully incomplete and unmaintained.)
+
+Twitch Model
+------------
+
+- `twitch_user_id`
+  Though numeric, this should always be a string per
+  [https://discuss.dev.twitch.tv/t/bug-v5-api-returns--id-as-string-for-featured-channels/10310]
+  and 
+  [https://discuss.dev.twitch.tv/t/type-of-user-id-in-api-responses/10205]
+  
+- `twitch_channel_id`
+  This is the same as `twitch_user_id`. They're exactly 1:1. Prefer to 
+  use 'user_id' terminology.
+
+- `twitch_username`
+  Between 4 and 25 characters.
+

_docs/old/old_docs/schema/redis_schema.md

@@ -0,0 +1,27 @@
+redis schema
+============
+
+Database 0: TTS / RateLimiter
+-----------------------------
+
+TODO: Document rate limiter
+
+Public statistics on usage displayed on the website:
+
+* `ttsUseCount:{model_token}`: INCR, GET
+* `w2lUseCount:{template_token}`: INCR, GET
+  
+Temporary job progress reports to the user:
+
+These are written as a single string, such as "downloading model", "uploading model", 
+"generating preview", "executing", etc.
+
+* `ttsDownloadExtraStatus:{inference_job_token}`: SETEX, GET
+* `ttsInferenceExtraStatus:{inference_job_token}`: SETEX, GET
+* `w2lDownloadExtraStatus:{inference_job_token}`: SETEX, GET
+* `w2lInferenceExtraStatus:{inference_job_token}`: SETEX, GET
+
+Database 1: Obs / Twitch
+------------------------
+
+* WIP

crates/lib/files/filesys/src/file_size.rs

@@ -34,7 +34,7 @@ mod tests {
   pub fn test_file_size() {
     // Success case
     let path = test_file("test_data/audio/flac/zelda_ocarina_small_item.flac");
-    assert_eq!(file_size(path).unwrap(), 271925);
+    assert_eq!(file_size(path).unwrap(), 115);
 
     // Error case
     assert!(file_size("non/existing/path").is_err());

frontend/apps/artcraft/app/public/resources/pose/7.vpd

@@ -1,305 +0,0 @@
-Vocaloid Pose Data file
-
-木の人形.osm;		// 親ファイル名
-60;				// 総ポーズボーン数
-
-Bone0{全ての親
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone1{センター
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone2{上半身
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone3{首
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000286,0.002004,-0.028579,0.999590;		// Quaternion x,y,z,w
-}
-
-Bone4{頭
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.003903,-0.047796,-0.030391,0.998387;		// Quaternion x,y,z,w
-}
-
-Bone5{下半身
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000950,0.006664,-0.095038,0.995451;		// Quaternion x,y,z,w
-}
-
-Bone6{左肩P
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone7{左肩
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.003159,-0.000000,0.105260,0.994440;		// Quaternion x,y,z,w
-}
-
-Bone8{左腕
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.034463,0.034156,-0.396750,0.916644;		// Quaternion x,y,z,w
-}
-
-Bone9{左腕捩
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone10{左ひじ
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.014712,-0.013995,0.076793,0.996840;		// Quaternion x,y,z,w
-}
-
-Bone11{左手捩
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone12{左手首
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.028156,-0.026783,0.146962,0.988378;		// Quaternion x,y,z,w
-}
-
-Bone13{左親指１
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.002261,-0.000573,-0.306446,0.951885;		// Quaternion x,y,z,w
-}
-
-Bone14{左親指２
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.047385,-0.118953,0.062146,0.989819;		// Quaternion x,y,z,w
-}
-
-Bone15{左人指１
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.020755,0.017947,0.064537,0.997538;		// Quaternion x,y,z,w
-}
-
-Bone16{左人指２
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.024997,0.999687;		// Quaternion x,y,z,w
-}
-
-Bone17{左人指３
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.029995,0.999550;		// Quaternion x,y,z,w
-}
-
-Bone18{左中指１
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,-0.064954,0.997888;		// Quaternion x,y,z,w
-}
-
-Bone19{左中指２
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,-0.054972,0.998488;		// Quaternion x,y,z,w
-}
-
-Bone20{左中指３
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,0.104807,0.994492;		// Quaternion x,y,z,w
-}
-
-Bone21{左薬指１
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.232843,0.972514;		// Quaternion x,y,z,w
-}
-
-Bone22{左薬指２
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.188859,0.982004;		// Quaternion x,y,z,w
-}
-
-Bone23{左薬指３
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,0.124675,0.992197;		// Quaternion x,y,z,w
-}
-
-Bone24{左小指１
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,-0.094857,0.995490;		// Quaternion x,y,z,w
-}
-
-Bone25{左小指２
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.496880,0.867819;		// Quaternion x,y,z,w
-}
-
-Bone26{左小指３
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,0.010000,0.999949;		// Quaternion x,y,z,w
-}
-
-Bone27{右肩P
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone28{右肩
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.003159,0.000000,-0.105260,0.994440;		// Quaternion x,y,z,w
-}
-
-Bone29{右腕
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.034463,-0.034156,0.396750,0.916644;		// Quaternion x,y,z,w
-}
-
-Bone30{右腕捩
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone31{右ひじ
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.014712,0.013995,-0.076793,0.996840;		// Quaternion x,y,z,w
-}
-
-Bone32{右手捩
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone33{右手首
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.028156,0.026783,-0.146962,0.988378;		// Quaternion x,y,z,w
-}
-
-Bone34{右親指１
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.002261,0.000573,0.306446,0.951885;		// Quaternion x,y,z,w
-}
-
-Bone35{右親指２
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.047385,0.118953,-0.062146,0.989819;		// Quaternion x,y,z,w
-}
-
-Bone36{右人指１
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.020755,-0.017947,-0.064537,0.997538;		// Quaternion x,y,z,w
-}
-
-Bone37{右人指２
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.024997,0.999687;		// Quaternion x,y,z,w
-}
-
-Bone38{右人指３
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,-0.000000,-0.029995,0.999550;		// Quaternion x,y,z,w
-}
-
-Bone39{右中指１
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,0.064954,0.997888;		// Quaternion x,y,z,w
-}
-
-Bone40{右中指２
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,0.054972,0.998488;		// Quaternion x,y,z,w
-}
-
-Bone41{右中指３
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,-0.104807,0.994492;		// Quaternion x,y,z,w
-}
-
-Bone42{右薬指１
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.232843,0.972514;		// Quaternion x,y,z,w
-}
-
-Bone43{右薬指２
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.188859,0.982004;		// Quaternion x,y,z,w
-}
-
-Bone44{右薬指３
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,-0.000000,-0.124675,0.992197;		// Quaternion x,y,z,w
-}
-
-Bone45{右小指１
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,0.094857,0.995490;		// Quaternion x,y,z,w
-}
-
-Bone46{右小指２
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.496880,0.867819;		// Quaternion x,y,z,w
-}
-
-Bone47{右小指３
-  -0.000000,0.000000,0.000000;				// trans x,y,z
-  -0.000000,0.000000,-0.010000,0.999949;		// Quaternion x,y,z,w
-}
-
-Bone48{左足
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone49{左ひざ
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone50{左足首
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone51{右足
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone52{右ひざ
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone53{右足首
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone54{左つま先
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone55{右つま先
-  0.000000,0.000000,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone56{左足ＩＫ
-  -0.132786,0.336300,0.033801;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone57{右足ＩＫ
-  0.148176,0.000000,0.001482;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone58{左つま先ＩＫ
-  -0.419457,-0.807707,-0.002797;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-
-Bone59{右つま先ＩＫ
-  0.069849,-0.366887,0.000000;				// trans x,y,z
-  0.000000,0.000000,0.000000,1.000000;		// Quaternion x,y,z,w
-}
-

test_data/comfy_workflows/style_replacement_tests/5_ghibli_anime_model.json

@@ -1,29 +0,0 @@
-{
-    "id": 5,
-    "style_name": "Ghibli Anime Model",
-    "style_for_workflow": "yae_vid2vid_main_workflow_13-03_2802495.json",
-    "md5": "26c96e73ea48e3c15a20715ad08f6f55",
-    "workflow_api_name": "workflow_api_production_03-14.json",
-    "mapping_name":"mapping_one.json",
-    "modifications": {
-        "positive_prompt": "highly detailed, smooth, sharp focus, (colorful:1.2), by studio ghibli, studio ghibli film kiki’s delivery service, sharp, 8k, style anime, illustration",
-        "negative_prompt": "flare, lens flare, glare,naked, nsfw, text, logo, watermark:1.3, embedding:easynegative, embedding:badhandv4,(worst quality, low quality:1.4), lowres, blurry, monochrome, (zombie, sketch, interlocked fingers, comic)",
-        "checkpoint": "aniverse_v15Pruned.safetensors",
-        "vae": "",
-        "loras": [
-            {
-                "name": "ghibli_style_offset.safetensors",
-                "strength": 1.0
-            }
-        ],
-        "denoise_first_pass": 0.7,
-        "every_nth_frame": 2,
-        "cn_lips_strength": 0.0,
-        "cn_sparse_scribble_strength": 0.0,
-        "cn_open_pose_strength": 1.0,
-        "cn_depth_strength": 0.0,
-        "cn_line_art_realistic_strength": 0.0,
-        "cn_line_art_anime_strength": 0.7,
-        "cn_soft_edge_strength": 0.0
-    }
-}

test_data/comfy_workflows/style_replacement_tests/mapping_one.json

@@ -1,42 +0,0 @@
-{
-    "positive_prompt": "$.2061.inputs.prompt",
-    "negative_prompt": "$.2056.inputs.prompt",
-  
-    "every_nth_frame": "$.2045.inputs.value",
-    "denoise_first_pass": "$.401.inputs.value",
-  
-    "cn_lips_strength": "$.1636.inputs.value",
-    "cn_sparse_scribble_strength": "$.403.inputs.value",
-    "cn_open_pose_strength": "$.771.inputs.value",
-    "cn_depth_strength": "$.772.inputs.value",
-    "cn_line_art_realistic_strength": "$.796.inputs.value",
-    "cn_line_art_anime_strength": "$.797.inputs.value",
-    "cn_soft_edge_strength": "$.1398.inputs.value",
-  
-    "checkpoint": "$.485.inputs.ckpt_name",
-    "vae": "$.427.inputs.vae_name",
-  
-    "lora_1_name": "$.414.inputs.lora_01",
-    "lora_1_strength": "$.414.inputs.strength_01",
-    
-    "lora_2_name": "$.414.inputs.lora_02",
-    "lora_2_strength": "$.414.inputs.strength_02",
-    
-    "lora_3_name": "$.414.inputs.lora_03",
-    "lora_3_strength": "$.414.inputs.strength_03",
-    
-    "lora_4_name": "$.414.inputs.lora_04",
-    "lora_4_strength": "$.414.inputs.strength_04",
-    
-    "lora_5_name": "$.208.inputs.lora_01",
-    "lora_5_strength": "$.208.inputs.strength_01",
-    
-    "lora_6_name": "$.208.inputs.lora_02",
-    "lora_6_strength": "$.208.inputs.strength_02",
-    
-    "lora_7_name": "$.208.inputs.lora_03",
-    "lora_7_strength": "$.208.inputs.strength_03",
-    
-    "lora_8_name": "$.208.inputs.lora_04",
-    "lora_8_strength": "$.208.inputs.strength_04"
-  }

test_data/crypto/google_sign_in.2024-04-01.jwk

@@ -1,20 +0,0 @@
-{
-  "keys": [
-    {
-      "n": "y48N6JB-AKq1-Rv4SkwBADU-hp4zXHU-NcCUwxD-aS9vr4EoT9qrjoJ-YmkaEpq9Bmu1yXZZK_h_9QS3xEsO8Rc_WSvIQCJtIaDQz8hxk4lUjUQjMB4Zf9vdTmf8KdktI9tCYCbuSbLC6TegjDM9kbl9CNs3m9wSVeO_5JXJQC0Jr-Oj7Gz9stXm0Co3f7RCxrD08kLelXaAglrd5TeGjZMyViC4cw1gPaj0Cj6knDn8UlzR_WuBpzs_ies5BrbzX-yht0WfnhXpdpiGNMbpKQD04MmPdMCYq8ENF7q5_Ok7dPsVj1vHA6vFGnf7qE3smD157szsnzn0NeXIbRMnuQ",
-      "alg": "RS256",
-      "use": "sig",
-      "e": "AQAB",
-      "kty": "RSA",
-      "kid": "adf5e710edfebecbefa9a61495654d03c0b8edf8"
-    },
-    {
-      "n": "uhWRpJ3PNZaiBmq3P91A6QB0b28LeQvV-HI0TAEcN5nffQPm94w-hY2S6mThb7xXLCGHcP3bhpWl31giZJFlvzHe6db-TsPl8HSLgLIjMbMT8iYWqZPa2eodijEJrkO6SPex5jHLzSwGsoRdSfW8hFeTFQk8xtPXm7GlEEo9mFEKUAaArT9acdE8h53VR7ZkJkipiLCtx0rhySA2W4rEAcinLG3ApG709pOw6sVjA2IAQmZVYrfQ7curmFqKWL_F534kDhQJL2hMdrubhHcqCxetyi_U7WDWDkYCJ_CetjDsI0yfwB2sR01vn6LuDDo6ho8pWJcHOOvXYUnSMFAlew",
-      "use": "sig",
-      "kid": "934a5816468b95703953d14e9f15df5d09a401e4",
-      "alg": "RS256",
-      "e": "AQAB",
-      "kty": "RSA"
-    }
-  ]
-}

test_data/crypto/google_sign_in.2024-09-20.jwk

@@ -1,20 +0,0 @@
-{
-  "keys": [
-    {
-      "e": "AQAB",
-      "n": "pi22xDdK2fz5gclIbDIGghLDYiRO56eW2GUcboeVlhbAuhuT5mlEYIevkxdPOg5n6qICePZiQSxkwcYMIZyLkZhSJ2d2M6Szx2gDtnAmee6o_tWdroKu0DjqwG8pZU693oLaIjLku3IK20lTs6-2TeH-pUYMjEqiFMhn-hb7wnvH_FuPTjgz9i0rEdw_Hf3Wk6CMypaUHi31y6twrMWq1jEbdQNl50EwH-RQmQ9bs3Wm9V9t-2-_Jzg3AT0Ny4zEDU7WXgN2DevM8_FVje4IgztNy29XUkeUctHsr-431_Iu23JIy6U4Kxn36X3RlVUKEkOMpkDD3kd81JPW4Ger_w",
-      "alg": "RS256",
-      "kty": "RSA",
-      "use": "sig",
-      "kid": "b2620d5e7f132b52afe8875cdf3776c064249d04"
-    },
-    {
-      "kty": "RSA",
-      "use": "sig",
-      "e": "AQAB",
-      "n": "jPxgqe78Uy8UI0nrbys8zFQnskdLnvY9DFAKbI9Or7sPc7vhyQ-ynHWXrvrv3J3EVqcqwZSTAjiKbSbIhKRF2iXyIP5jmhS6QTUQb7D8smC89yZi6Ii-AzpH6QKvmhU7yJ1u0odMM1UDUS5bH5aL50HxxqqaQGlZ7PFOT0xrauAFW-3ONVc7_tXGMbfYRzeRrXqaONJ1B9LOconUlsBsL0U1TepINyztbwjM3NBlvEuBX0m4ZbCFznGoWmnix3FuUS4gAybOO3WYr6Zd71cKBFPfdpMMfNjWM2pf1-1O1IF8iArGbvngn8Vk5QGH3MkJDA_JgZOu9pI64LSIEKG02w",
-      "alg": "RS256",
-      "kid": "5aaff47c21d06e266cce395b2145c7c6d4730ea5"
-    }
-  ]
-}

test_data/text_files/ip_ban_example/empty_file_with_only_comments.txt

@@ -1,4 +0,0 @@
-# This file contains only comments
-
-# No IP addresses
-

test_data/text_files/ip_ban_example/local_ip_addresses_and_comments.txt

@@ -1,5 +0,0 @@
-# this is test data
-
-# local addresses
-127.0.0.1
-192.168.0.1

test_data/text_files/ip_ban_example/remote_ip_addresses_and_comments.txt

@@ -1,7 +0,0 @@
-# this is test data
-
-# remote  addresses
-4.4.4.4
-8.8.4.4
-8.8.8.8
-

test_data/text_files/troll_user_token_ban_example/empty_file_with_only_comments.txt

@@ -1,4 +0,0 @@
-# This file contains only comments
-
-# No user tokens
-

test_data/text_files/troll_user_token_ban_example/user_tokens_and_comments_1.txt

@@ -1,5 +0,0 @@
-# this is test data
-
-# example user tokens
-U:FOO
-U:BAR

test_data/text_files/troll_user_token_ban_example/user_tokens_and_comments_2.txt

@@ -1,7 +0,0 @@
-# this is test data
-
-# more example user tokens
-U:STEVE
-U:JIM
-U:BOB
-