Compare commits

...
Author SHA1 Message Date
salt d39962491f fix some warnings (mostly unnecessary mut) 2023-12-07 18:36:51 -08:00
Brandon Thomas eedbc346fe Merge pull request #194 from storytold/brandon-tts-as-media-file
Save TTS results as media files
2023-12-06 23:33:51 -05:00
Brandon Thomas 7976f68fd0 Save TTS results as media files 2023-12-06 23:31:47 -05:00
Brandon Thomas 8a6678c7f5 move workspace-hack under crates/ 2023-12-06 02:12:57 -05:00
Brandon Thomas d9b1c8c36b Merge pull request #192 from storytold/brandon-upgrade-chrono-vendored-limitation
upgrade chrono with vendored limitation
2023-12-05 21:54:36 -05:00
Brandon Thomas fc122cb996 upgrade chrono with vendored limitation 2023-12-05 21:54:06 -05:00
Brandon Thomas e0e85fed5f Merge pull request #191 from storytold/brandon-vendor-limitation
vendor `limitation` crate
2023-12-05 20:58:27 -05:00
Brandon Thomas 092745166e vendor limitation crate
The limitation crate is four years old, stuck at 0.1.1. It looks unlikey
to be updated by the maintainer.

Upgrades to chrono and other libraries have broken limitation, so we're
likely now on the hook for maintaining this.
2023-12-05 20:56:22 -05:00
Brandon Thomas ea859bfc1e Merge pull request #190 from storytold/brandon-revert-chrono-upgrade
Revert "Bump chrono from 0.4.22 to 0.4.31"
2023-12-05 20:40:37 -05:00
Brandon Thomas 92e0b1e9f4 Revert "Bump chrono from 0.4.22 to 0.4.31"
This reverts commit 4856ac28aa.
2023-12-05 20:40:09 -05:00
ArEnSc d52d3f993a Merge pull request #189 from storytold/feature/api-spec
Feature/api spec
2023-12-05 20:27:57 -05:00
Michael Chung 0033af73ce Feedback addressed! 2023-12-05 20:26:07 -05:00
Brandon Thomas 514f80b884 Merge pull request #184 from storytold/dependabot/cargo/rustix-0.38.26
Bump rustix from 0.38.25 to 0.38.26
2023-12-05 19:22:24 -05:00
Brandon Thomas 74c548c02f Merge pull request #182 from storytold/dependabot/cargo/chrono-0.4.31
Bump chrono from 0.4.22 to 0.4.31
2023-12-05 19:21:53 -05:00
Brandon Thomas 5fe09a182f Merge pull request #185 from storytold/dependabot/cargo/lettre-0.11.2
Bump lettre from 0.11.1 to 0.11.2
2023-12-05 19:21:40 -05:00
dependabot[bot] 4856ac28aa Bump chrono from 0.4.22 to 0.4.31
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.22 to 0.4.31.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.22...v0.4.31)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-06 00:19:36 +00:00
Brandon Thomas 404853e129 cleanup misleading old comment 2023-12-05 19:16:55 -05:00
Michael Chung b5fcc3cdfa Added documentation about the documentation 2023-12-05 19:14:02 -05:00
Brandon Thomas 300ac56ab7 Merge pull request #188 from storytold/brandon-media-files-global-list-fixed
global list of media files
2023-12-05 19:11:20 -05:00
Brandon Thomas e32f8094e5 global list of media files 2023-12-05 19:09:42 -05:00
Michael Chung e986a44e9d documentation system open api spec. 2023-12-05 19:00:44 -05:00
Brandon Thomas 642e17a77d Merge pull request #187 from storytold/brandon-fix-sqlx-query-builder
Fix QueryBuilder with respect to column names and type juggling
2023-12-05 18:31:13 -05:00
Brandon Thomas 36f3003576 Fix QueryBuilder with respect to column names and type juggling 2023-12-05 18:30:26 -05:00
Brandon Thomas b9f604dd18 reusable pagination types 2023-12-04 23:11:41 -05:00
Brandon Thomas e6f5ecb980 Merge pull request #186 from storytold/brandon-list-media-files-for-user
list media files for user
2023-12-04 22:02:53 -05:00
Brandon Thomas 2e3ff8a730 list media files for user 2023-12-04 21:59:39 -05:00
dependabot[bot] ac13be2900 Bump lettre from 0.11.1 to 0.11.2
Bumps [lettre](https://github.com/lettre/lettre) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/lettre/lettre/releases)
- [Changelog](https://github.com/lettre/lettre/blob/master/CHANGELOG.md)
- [Commits](https://github.com/lettre/lettre/compare/v0.11.1...v0.11.2)

---
updated-dependencies:
- dependency-name: lettre
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-04 22:30:26 +00:00
dependabot[bot] b55435d992 Bump rustix from 0.38.25 to 0.38.26
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.38.25 to 0.38.26.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.38.25...v0.38.26)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-04 22:30:06 +00:00
Brandon Thomas 28770ef736 fix query issue 2023-12-04 05:00:46 -05:00
Brandon Thomas 894084fba6 Merge pull request #181 from storytold/brandon-queue-controls
Kill inference jobs (mod endpoint)
2023-12-04 03:54:05 -05:00
144 changed files with 3993 additions and 548 deletions
Generated
+633 -213
View File
File diff suppressed because it is too large Load Diff
+3 -2
View File
@@ -86,8 +86,9 @@ members = [
"crates/service/plugins/users",
# Vendored code
"crates/vendor/limitation",
"crates/vendor/newrelic-telemetry-sdk-rust",
# Vendored code
"my-workspace-hack",
# cargo-hakari workspace hack (compilation speedup hack)
"crates/my-workspace-hack",
]
+3 -1
View File
@@ -29,7 +29,9 @@ Web API Overview
* `tts-inference-job` - (*deprecated*) Async TTS inference of Tacotron2 models only
* `w2l-download-job` - (*deprecated*) Async download of videos and images
* `w2l-inference-job` - (*deprecated*) Async W2L inference
* Documentation Server:
* `docs` - this will run a local swagger server with the endpoints that you have access to, http://127.0.0.1:8989/api-docs/openapi.json use this to port everything over to the post man.
### PowerStream
* [PowerStream](https://power.stream) (deprecated) is powered by the following applications:
@@ -380,3 +380,16 @@ of this error message. You can essentially ignore it.
### [Fix] Can't connect to local MySQL after install on Ubuntu
- If MySql in local dev can't be connected to, [reset the accounts](https://linuxconfig.org/how-to-reset-root-mysql-mariadb-password-on-ubuntu-20-04-focal-fossa-linux).
### API Server
```
cargo run --bin docs
```
This setups an api server with swagger docs to show you responses and requests with each associated endpoint
It is run at http://127.0.0.1:8989/
To obtain the json for the postman collection go to this url http://127.0.0.1:8989/api-docs/openapi.json
We use this: `https://github.com/juhaku/utoipa`
The documentation example is in:
`crates/service/web/storyteller_web/src/http_server/endpoints/voice_designer/inference/enqueue_tts_request.rs`
Write into the api doc to:
`crates/service/web/storyteller_web/src/api_doc.rs`
+1 -1
View File
@@ -17,7 +17,7 @@ path = "src/lib.rs"
# Internal
crockford = { path = "../../lib/crockford" }
errors = { path = "../../lib/errors" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# External
# (none yet)
+1 -5
View File
@@ -17,14 +17,10 @@ path = "src/lib.rs"
serde = "1.0.125"
serde_derive = "1.0.125"
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "sqlite", "runtime-tokio-native-tls", "chrono" ] }
sqlx-core = { version = "0.7.2" }
sqlx-mysql= { version = "0.7.2" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
@@ -13,10 +13,11 @@ use strum::EnumIter;
///
/// DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, Default)]
pub enum InferenceCategory {
/// Eg. SadTalker and possibly Wav2Lip
#[serde(rename = "lipsync_animation")]
#[default]
LipsyncAnimation,
#[serde(rename = "text_to_speech")]
@@ -21,6 +21,11 @@ pub enum IdCategory {
#[serde(rename = "lipsync_animation")]
LipsyncAnimationResult,
/// Results from tacotron2
/// Applies for RVC and SVC
#[serde(rename = "tts_result")]
TtsResult,
/// Results from voice conversion (which may live in the media_files table)
/// Applies for RVC and SVC
#[serde(rename = "voice_conversion")]
@@ -52,6 +57,7 @@ impl IdCategory {
match self {
Self::MediaFile => "media_file",
Self::LipsyncAnimationResult => "lipsync_animation",
Self::TtsResult => "tts_result",
Self::VoiceConversionResult => "voice_conversion",
Self::ZeroShotVoiceDataset => "zs_dataset",
Self::ZeroShotVoiceEmbedding => "zs_voice",
@@ -64,6 +70,7 @@ impl IdCategory {
match value {
"media_file" => Ok(Self::MediaFile),
"lipsync_animation" => Ok(Self::LipsyncAnimationResult),
"tts_result" => Ok(Self::TtsResult),
"voice_conversion" => Ok(Self::VoiceConversionResult),
"zs_dataset" => Ok(Self::ZeroShotVoiceDataset),
"zs_voice" => Ok(Self::ZeroShotVoiceEmbedding),
@@ -79,6 +86,7 @@ impl IdCategory {
BTreeSet::from([
Self::MediaFile,
Self::LipsyncAnimationResult,
Self::TtsResult,
Self::VoiceConversionResult,
Self::ZeroShotTtsResult,
Self::ZeroShotVoiceDataset,
@@ -100,6 +108,7 @@ mod tests {
fn test_serialization() {
assert_serialization(IdCategory::MediaFile, "media_file");
assert_serialization(IdCategory::LipsyncAnimationResult, "lipsync_animation");
assert_serialization(IdCategory::TtsResult, "tts_result");
assert_serialization(IdCategory::VoiceConversionResult, "voice_conversion");
assert_serialization(IdCategory::ZeroShotVoiceDataset, "zs_dataset");
assert_serialization(IdCategory::ZeroShotVoiceEmbedding, "zs_voice");
@@ -111,6 +120,7 @@ mod tests {
fn to_str() {
assert_eq!(IdCategory::MediaFile.to_str(), "media_file");
assert_eq!(IdCategory::LipsyncAnimationResult.to_str(), "lipsync_animation");
assert_eq!(IdCategory::TtsResult.to_str(), "tts_result");
assert_eq!(IdCategory::VoiceConversionResult.to_str(), "voice_conversion");
assert_eq!(IdCategory::ZeroShotVoiceDataset.to_str(), "zs_dataset");
assert_eq!(IdCategory::ZeroShotVoiceEmbedding.to_str(), "zs_voice");
@@ -122,6 +132,7 @@ mod tests {
fn from_str() {
assert_eq!(IdCategory::from_str("media_file").unwrap(), IdCategory::MediaFile);
assert_eq!(IdCategory::from_str("lipsync_animation").unwrap(), IdCategory::LipsyncAnimationResult);
assert_eq!(IdCategory::from_str("tts_result").unwrap(), IdCategory::TtsResult);
assert_eq!(IdCategory::from_str("voice_conversion").unwrap(), IdCategory::VoiceConversionResult);
assert_eq!(IdCategory::from_str("zs_dataset").unwrap(), IdCategory::ZeroShotVoiceDataset);
assert_eq!(IdCategory::from_str("zs_voice").unwrap(), IdCategory::ZeroShotVoiceEmbedding);
@@ -133,9 +144,10 @@ mod tests {
fn all_variants() {
// Static check
let mut variants = IdCategory::all_variants();
assert_eq!(variants.len(), 7);
assert_eq!(variants.len(), 8);
assert_eq!(variants.pop_first(), Some(IdCategory::MediaFile));
assert_eq!(variants.pop_first(), Some(IdCategory::LipsyncAnimationResult));
assert_eq!(variants.pop_first(), Some(IdCategory::TtsResult));
assert_eq!(variants.pop_first(), Some(IdCategory::VoiceConversionResult));
assert_eq!(variants.pop_first(), Some(IdCategory::ZeroShotTtsResult));
assert_eq!(variants.pop_first(), Some(IdCategory::ZeroShotVoiceDataset));
@@ -1,4 +1,5 @@
use std::collections::BTreeSet;
use serde::Deserialize;
use serde::Serialize;
#[cfg(test)]
@@ -29,6 +30,7 @@ pub enum MediaFileOriginCategory {
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
impl_enum_display_and_debug_using_to_str!(MediaFileOriginCategory);
impl_mysql_enum_coders!(MediaFileOriginCategory);
impl_mysql_from_row!(MediaFileOriginCategory);
/// NB: Legacy API for older code.
impl MediaFileOriginCategory {
@@ -24,6 +24,10 @@ pub enum MediaFileOriginModelType {
/// so-vits-svc voice conversion models
#[serde(rename = "so_vits_svc")]
SoVitsSvc,
#[serde(rename = "tacotron2")]
Tacotron2,
#[serde(rename = "vall_e_x")]
VallEX,
}
@@ -31,6 +35,7 @@ pub enum MediaFileOriginModelType {
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
impl_enum_display_and_debug_using_to_str!(MediaFileOriginModelType);
impl_mysql_enum_coders!(MediaFileOriginModelType);
impl_mysql_from_row!(MediaFileOriginModelType);
/// NB: Legacy API for older code.
impl MediaFileOriginModelType {
@@ -39,6 +44,7 @@ impl MediaFileOriginModelType {
Self::RvcV2 => "rvc_v2",
Self::SadTalker => "sad_talker",
Self::SoVitsSvc => "so_vits_svc",
Self::Tacotron2 => "tacotron2",
Self::VallEX => "vall_e_x"
}
}
@@ -48,6 +54,7 @@ impl MediaFileOriginModelType {
"rvc_v2" => Ok(Self::RvcV2),
"sad_talker" => Ok(Self::SadTalker),
"so_vits_svc" => Ok(Self::SoVitsSvc),
"tacotron2" => Ok(Self::Tacotron2),
"vall_e_x" => Ok(Self::VallEX),
_ => Err(format!("invalid value: {:?}", value)),
}
@@ -60,6 +67,7 @@ impl MediaFileOriginModelType {
Self::RvcV2,
Self::SadTalker,
Self::SoVitsSvc,
Self::Tacotron2,
Self::VallEX
])
}
@@ -83,6 +91,7 @@ mod tests {
assert_eq!(MediaFileOriginModelType::RvcV2.to_str(), "rvc_v2");
assert_eq!(MediaFileOriginModelType::SadTalker.to_str(), "sad_talker");
assert_eq!(MediaFileOriginModelType::SoVitsSvc.to_str(), "so_vits_svc");
assert_eq!(MediaFileOriginModelType::Tacotron2.to_str(), "tacotron2");
assert_eq!(MediaFileOriginModelType::VallEX.to_str(), "vall_e_x");
}
@@ -91,17 +100,19 @@ mod tests {
assert_eq!(MediaFileOriginModelType::from_str("rvc_v2").unwrap(), MediaFileOriginModelType::RvcV2);
assert_eq!(MediaFileOriginModelType::from_str("sad_talker").unwrap(), MediaFileOriginModelType::SadTalker);
assert_eq!(MediaFileOriginModelType::from_str("so_vits_svc").unwrap(), MediaFileOriginModelType::SoVitsSvc);
assert_eq!(MediaFileOriginModelType::from_str("tacotron2").unwrap(), MediaFileOriginModelType::Tacotron2);
assert_eq!(MediaFileOriginModelType::from_str("vall_e_x").unwrap(), MediaFileOriginModelType::VallEX);
assert!(MediaFileOriginModelType::from_str("foo").is_err());
assert!(MediaFileOriginModelType::from_str("foo").is_err());
}
#[test]
fn all_variants() {
let mut variants = MediaFileOriginModelType::all_variants();
assert_eq!(variants.len(), 4);
assert_eq!(variants.len(), 5);
assert_eq!(variants.pop_first(), Some(MediaFileOriginModelType::RvcV2));
assert_eq!(variants.pop_first(), Some(MediaFileOriginModelType::SadTalker));
assert_eq!(variants.pop_first(), Some(MediaFileOriginModelType::SoVitsSvc));
assert_eq!(variants.pop_first(), Some(MediaFileOriginModelType::Tacotron2));
assert_eq!(variants.pop_first(), Some(MediaFileOriginModelType::VallEX));
assert_eq!(variants.pop_first(), None);
}
@@ -21,6 +21,10 @@ pub enum MediaFileOriginProductCategory {
#[serde(rename = "face_animator")]
FaceAnimator,
/// Text to speech (Tacotron2, not voice designer / VallE-X)
#[serde(rename = "tts")]
TextToSpeech,
// TODO: This should be a temporary category until we migrate the DB to remove this default value
/// Unknown which product is attached to the file (generated the file, the file was
/// uploaded on behalf of, etc.)
@@ -39,12 +43,14 @@ pub enum MediaFileOriginProductCategory {
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
impl_enum_display_and_debug_using_to_str!(MediaFileOriginProductCategory);
impl_mysql_enum_coders!(MediaFileOriginProductCategory);
impl_mysql_from_row!(MediaFileOriginProductCategory);
/// NB: Legacy API for older code.
impl MediaFileOriginProductCategory {
pub fn to_str(&self) -> &'static str {
match self {
Self::FaceAnimator => "face_animator",
Self::TextToSpeech => "tts",
Self::Unknown => "unknown",
Self::VoiceConversion => "voice_conversion",
Self::ZeroShotVoice => "zs_voice",
@@ -54,6 +60,7 @@ impl MediaFileOriginProductCategory {
pub fn from_str(value: &str) -> Result<Self, String> {
match value {
"face_animator" => Ok(Self::FaceAnimator),
"tts" => Ok(Self::TextToSpeech),
"unknown" => Ok(Self::Unknown),
"voice_conversion" => Ok(Self::VoiceConversion),
"zs_voice" => Ok(Self::ZeroShotVoice),
@@ -66,6 +73,7 @@ impl MediaFileOriginProductCategory {
// NB: BTreeSet::from() isn't const, but not worth using LazyStatic, etc.
BTreeSet::from([
Self::FaceAnimator,
Self::TextToSpeech,
Self::Unknown,
Self::VoiceConversion,
Self::ZeroShotVoice,
@@ -84,6 +92,7 @@ mod tests {
#[test]
fn test_serialization() {
assert_serialization(MediaFileOriginProductCategory::FaceAnimator, "face_animator");
assert_serialization(MediaFileOriginProductCategory::TextToSpeech, "tts");
assert_serialization(MediaFileOriginProductCategory::Unknown, "unknown");
assert_serialization(MediaFileOriginProductCategory::VoiceConversion, "voice_conversion");
assert_serialization(MediaFileOriginProductCategory::ZeroShotVoice, "zs_voice");
@@ -92,6 +101,7 @@ mod tests {
#[test]
fn to_str() {
assert_eq!(MediaFileOriginProductCategory::FaceAnimator.to_str(), "face_animator");
assert_eq!(MediaFileOriginProductCategory::TextToSpeech.to_str(), "tts");
assert_eq!(MediaFileOriginProductCategory::Unknown.to_str(), "unknown");
assert_eq!(MediaFileOriginProductCategory::VoiceConversion.to_str(), "voice_conversion");
assert_eq!(MediaFileOriginProductCategory::ZeroShotVoice.to_str(), "zs_voice");
@@ -100,6 +110,7 @@ mod tests {
#[test]
fn from_str() {
assert_eq!(MediaFileOriginProductCategory::from_str("face_animator").unwrap(), MediaFileOriginProductCategory::FaceAnimator);
assert_eq!(MediaFileOriginProductCategory::from_str("tts").unwrap(), MediaFileOriginProductCategory::TextToSpeech);
assert_eq!(MediaFileOriginProductCategory::from_str("unknown").unwrap(), MediaFileOriginProductCategory::Unknown);
assert_eq!(MediaFileOriginProductCategory::from_str("voice_conversion").unwrap(), MediaFileOriginProductCategory::VoiceConversion);
assert_eq!(MediaFileOriginProductCategory::from_str("zs_voice").unwrap(), MediaFileOriginProductCategory::ZeroShotVoice);
@@ -108,8 +119,9 @@ mod tests {
#[test]
fn all_variants() {
let mut variants = MediaFileOriginProductCategory::all_variants();
assert_eq!(variants.len(), 4);
assert_eq!(variants.len(), 5);
assert_eq!(variants.pop_first(), Some(MediaFileOriginProductCategory::FaceAnimator));
assert_eq!(variants.pop_first(), Some(MediaFileOriginProductCategory::TextToSpeech));
assert_eq!(variants.pop_first(), Some(MediaFileOriginProductCategory::Unknown));
assert_eq!(variants.pop_first(), Some(MediaFileOriginProductCategory::VoiceConversion));
assert_eq!(variants.pop_first(), Some(MediaFileOriginProductCategory::ZeroShotVoice));
@@ -25,6 +25,7 @@ pub enum MediaFileType {
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
impl_enum_display_and_debug_using_to_str!(MediaFileType);
impl_mysql_enum_coders!(MediaFileType);
impl_mysql_from_row!(MediaFileType);
/// NB: Legacy API for older code.
impl MediaFileType {
@@ -34,6 +34,12 @@ impl_mysql_enum_coders!(JobStatusPlus);
// NB: We can derive `sqlx::Type` instead of using `impl_mysql_enum_coders`
impl Default for JobStatusPlus {
fn default() -> Self {
Self::Pending
}
}
impl JobStatusPlus {
pub fn to_str(&self) -> &'static str {
match self {
@@ -86,6 +92,11 @@ mod tests {
mod explicit_checks {
use super::*;
#[test]
fn test_default() {
assert_eq!(JobStatusPlus::default(), JobStatusPlus::Pending);
}
#[test]
fn test_serialization() {
assert_serialization(JobStatusPlus::Pending, "pending");
@@ -38,6 +38,7 @@ pub enum Visibility {
impl_enum_display_and_debug_using_to_str!(Visibility);
impl_mysql_from_row!(Visibility);
// For reference, here's what the serde implementation might be if manually written.
// This may be useful for designing composite types in the future:
+1
View File
@@ -32,3 +32,4 @@ mod macros;
pub mod by_table;
pub mod common;
pub mod traits;
@@ -0,0 +1,63 @@
/// Implement `MySqlFromRow` on a type.
macro_rules! impl_mysql_from_row {
($t:ident) => {
// Try to convert a MySQL row and named field into the value type (for non-nullable fields).
impl crate::traits::mysql_from_row::MySqlFromRow<$t> for $t {
fn try_from_mysql_row(row: &sqlx_mysql::MySqlRow, field_name: &str) -> Result<$t, sqlx::Error> {
use sqlx::{Error, Row};
// NB(bt,2023-12-05): For now only string encodings are considered.
// We may want to revisit in the future if we deal with binary data.
let value : String = row.try_get(field_name)?;
let output_type = Self::from_str(&value)
.map_err(|err| {
Error::ColumnDecode {
index: format!("mysql_from_row failure on field {}: {:?}", field_name, err),
source: format!("mysql_from_row failure on field {}: {:?}", field_name, err).into(),
}
})?;
Ok(output_type)
}
// Try to convert a MySQL row and named field into the value type (for nullable fields).
fn try_from_mysql_row_nullable(row: &sqlx_mysql::MySqlRow, field_name: &str) -> Result<Option<$t>, sqlx::Error> {
use sqlx::{Error, Row};
// NB(bt,2023-12-05): For now only string encodings are considered.
// We may want to revisit in the future if we deal with binary data.
// NB(2): Nullable fields decode as Option<T>.
let maybe_value : Option<String> = row.try_get(field_name)?;
let value = match maybe_value {
Some(value) => value,
None => return Ok(None),
};
let output_type = Self::from_str(&value)
.map_err(|err| {
Error::ColumnDecode {
index: format!("mysql_from_row failure on field {}: {:?}", field_name, err),
source: format!("mysql_from_row failure on field {}: {:?}", field_name, err).into(),
}
})?;
Ok(Some(output_type))
}
}
//// TODO: Tests.
//#[cfg(test)]
//#[test]
//fn test_name() {
// use strum::IntoEnumIterator;
// for variant in $t::iter() {
// assert_eq!(format!("{}", variant), variant.to_str());
// }
//}
}
}
+1
View File
@@ -1,3 +1,4 @@
#[macro_use] pub mod impl_enum_display_and_debug_using_to_str;
#[macro_use] pub mod impl_mysql_enum_coders;
#[macro_use] pub mod impl_mysql_from_row;
#[macro_use] pub mod impl_sqlite_enum_coders;
+1
View File
@@ -0,0 +1 @@
pub mod mysql_from_row;
@@ -0,0 +1,11 @@
use sqlx_mysql::MySqlRow;
/// This is a trait (in conjunction with the macro implementation) that makes up for sqlx's
/// QueryBuilder not playing well with FromRow implementation or derivation of named types.
pub trait MySqlFromRow<T> {
/// Attempt to parse a named column from a MySqlRow into the type.
fn try_from_mysql_row(row: &MySqlRow, field_name: &str) -> Result<T, sqlx::Error>;
fn try_from_mysql_row_nullable(row: &MySqlRow, field_name: &str) -> Result<Option<T>, sqlx::Error>;
}
+2 -2
View File
@@ -18,7 +18,7 @@ path = "src/lib.rs"
# (none)
# External
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
# Serialization
serde = "1.0.125"
@@ -26,7 +26,7 @@ serde_derive = "1.0.125"
# MySql Database
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# Test serialization
+3 -5
View File
@@ -20,13 +20,11 @@ serde_derive = "1.0.125"
# Random
rand = "0.8.5"
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
sqlx-core = { version = "0.7.2" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
utoipa = { version = "4", features = ["actix_extras"] }
[dev-dependencies]
@@ -8,7 +8,7 @@ use crate::prefixes::TokenPrefix;
/// A rare instance that is *not* a primary key.
/// These are generated as cookies to track users for a better logged out experience and for analytics.
/// We use these as indices into several columns (ML result types, uploads, etc.)
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize, Default)]
#[sqlx(transparent)]
pub struct AnonymousVisitorTrackingToken(pub String);
@@ -4,9 +4,10 @@ use serde::Deserialize;
use serde::Serialize;
use crate::prefixes::TokenPrefix;
use utoipa::ToSchema;
/// The primary key for "generic" inference jobs.
#[derive(Clone, PartialEq, Eq, sqlx::Type, Debug, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Eq, sqlx::Type, Debug, Serialize, Deserialize, Default, ToSchema)]
#[sqlx(transparent)]
pub struct InferenceJobToken(String);
+1 -1
View File
@@ -6,7 +6,7 @@ use serde::Serialize;
use crate::prefixes::TokenPrefix;
/// The primary key for users.
#[derive(Clone, PartialEq, Eq, sqlx::Type, Debug, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Eq, sqlx::Type, Debug, Serialize, Deserialize, Default)]
#[sqlx(transparent)]
pub struct UserToken(pub String);
+1 -1
View File
@@ -28,4 +28,4 @@ log = "0.4.14"
# Serialization
serde = "1.0.125"
serde_derive = "1.0.125"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -28,4 +28,4 @@ log = "0.4.14"
# Serialization
serde = "1.0.125"
serde_derive = "1.0.125"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -5
View File
@@ -36,14 +36,10 @@ clap = "2.34.0" # TODO: This is really out of date, but has conflicts when updat
# NB: We switched to Tokio from `async-std` for broader ecosystem support.
tokio = { version = "1.6.0", features = ["macros"] }
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
elasticsearch = "8.5.0-alpha.1"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -5
View File
@@ -33,12 +33,8 @@ log = "0.4.14"
# NB: We switched to Tokio from `async-std` for broader ecosystem support.
tokio = { version = "1.6.0", features = ["macros"] }
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -5
View File
@@ -33,10 +33,6 @@ strum = { version = "0.25", features = ["derive"] }
# NB: We switched to Tokio from `async-std` for broader ecosystem support.
tokio = { version = "1.6.0", features = ["macros"] }
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
elasticsearch = "8.5.0-alpha.1"
@@ -45,7 +41,7 @@ elasticsearch = "8.5.0-alpha.1"
serde = "1.0.125"
serde_derive = "1.0.125"
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -23,7 +23,7 @@ log = "0.4.20"
once_cell = "1.18.0"
rand = "0.8.5"
regex = "1.10.0"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# Async Runtime
# NB: Not using Tokio here since everything is blocking so far.
+1 -1
View File
@@ -20,7 +20,7 @@ logging = { path = "../../lib/logging" }
# External
clap = { version = "4.4.7", features = ["derive"] }
walkdir = "2.4.0"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -21,4 +21,4 @@ web_scrapers = { path = "../../lib/web_scrapers" }
# External
clap = "2.34.0" # TODO: This is really out of date, but has conflicts when updated
tokio = { version = "1.6.0", features = ["macros"] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -34,7 +34,7 @@ futures-util = "0.3.29"
# Serialization
serde = "1.0.125"
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
@@ -28,4 +28,4 @@ rust-s3 = { version = "0.33.0" }
# Runtime
# NB: We switched to Tokio from `async-std` for broader ecosystem support.
tokio = { version = "1.6.0", features = ["macros"] }
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
@@ -20,7 +20,7 @@ tokens = { path = "../../../_types/tokens" }
# External
bytes = "1.3"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
log = "0.4.14"
once_cell = "1.9.0"
@@ -22,5 +22,5 @@ subprocess_common = { path = "../../subprocess_common" }
anyhow = "1.0.75"
log = "0.4.14"
tempdir = "0.3.7"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
@@ -31,4 +31,4 @@ twitch_oauth2 = "0.6.0-rc.2"
# Serialization
serde = "1.0.125"
serde_derive = "1.0.125"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
+1 -1
View File
@@ -23,7 +23,7 @@ server_environment = { path = "../server_environment" }
dotenv = "0.15.0"
hostname = "0.3.1"
log = "0.4.14"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -22,4 +22,4 @@ anyhow = "1.0.75"
# In-memory caching
lru_time_cache = "0.11.11"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
+1 -1
View File
@@ -23,7 +23,7 @@ r2d2_redis = "0.14.0"
serde = "1.0.125"
serde_derive = "1.0.125"
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -16,7 +16,7 @@ path = "src/lib.rs"
[dependencies]
# Internal
errors = { path = "../errors" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# External
# KEEP THIS TO A MINIMUM OR NIL.
+1 -1
View File
@@ -14,7 +14,7 @@ path = "src/lib.rs"
[dependencies]
# Internal
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# External
# (None yet)
+1 -1
View File
@@ -21,7 +21,7 @@ errors = { path = "../errors" }
hmac = "0.10.0" # jwt signing
jwt = "0.13.0"
sha2 = "0.9.5" # hmac signing
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -16,7 +16,7 @@ path = "src/lib.rs"
[dependencies]
rand = "0.8.5"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+2 -2
View File
@@ -13,9 +13,9 @@ name = "datetimes"
path = "src/lib.rs"
[dependencies]
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
once_cell = "1.18.0"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# (none)
+1 -1
View File
@@ -21,4 +21,4 @@ easyenv = { path = "../../../lib/easyenv" }
# External
anyhow = "1.0.75"
url = "2.4.1"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
@@ -24,7 +24,7 @@ log = "0.4.14"
# Serialization
serde = { version = "1.0.125", features = ["derive"] }
toml = "0.5.8"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
@@ -21,7 +21,7 @@ reusable_types = { path = "../../../_types/reusable_types" }
# External
anyhow = "1.0.75"
base64 = "0.21.0"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
log = "0.4.14"
once_cell = "1.18.0"
regex = "1.5.4"
@@ -39,7 +39,7 @@ language-tags = "0.3.2" # Parsing single tags richly
serde = "1.0.125"
serde_derive = "1.0.125"
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
actix-rt = "2.9.0"
+1 -1
View File
@@ -20,7 +20,7 @@ storyteller_root = { path = "../storyteller_root" }
dotenv = "0.15.0"
env_logger = "0.10.1"
log = "0.4.14"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+2 -2
View File
@@ -23,10 +23,10 @@ elasticsearch = "8.5.0-alpha.1"
log = "0.4.14"
# Serialization
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
serde = { version = "1.0.152", features = ["derive"] }
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
@@ -27,7 +27,7 @@ pub async fn search_tts_models(
let mut response_json = search_response.json::<Value>().await?;
let mut hits = response_json.get_mut("hits")
let hits = response_json.get_mut("hits")
.map(|hits| hits.take());
let hits = hits.map(|mut hits| {
+2 -2
View File
@@ -11,5 +11,5 @@ path = "src/lib.rs"
[dependencies]
errors = { path = "../errors" }
lettre = "0.11.0"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
lettre = "0.11.2"
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -18,7 +18,7 @@ path = "src/lib.rs"
# Consider the dependencies here wisely.
# Ideally we have almost nothing as a dependency of this crate.
anyhow = "1.0.75"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -20,7 +20,7 @@ errors = { path = "../../errors" }
# External
zip = "0.6.6"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -16,7 +16,7 @@ path = "src/lib.rs"
[dependencies]
# Internal
errors = { path = "../../errors" }
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
log = "0.4.20"
# External
+1 -1
View File
@@ -22,7 +22,7 @@ bcrypt = "0.8" # password hashes
data-encoding = "2.3.2"
md-5 = "0.9.1" # DO NOT USE FOR SERIOUS CRYPTO (we use this for gravatar)
ring = "0.17.0-alpha.10"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -21,7 +21,7 @@ errors = { path = "../../errors" }
# External
image = "0.24.1"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
testing = { path = "../../testing" }
+1 -1
View File
@@ -26,7 +26,7 @@ log = "0.4.14"
# Opus is the browser's default (only) recording codec: https://blog.addpipe.com/audio-constraints-getusermedia/
symphonia = { git = "https://github.com/pdeljanov/Symphonia.git", rev = "e1a70095c4409478c1ded499fb8dd4afa540fb90", features = ["aac", "alac", "flac", "isomp4", "mkv", "mp3", "ogg", "opus", "vorbis", "wav"] }
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -16,7 +16,7 @@ path = "src/lib.rs"
[dependencies]
# External
infer = "0.11" # Mimetype detection
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -19,7 +19,7 @@ path = "src/lib.rs"
# External
uuid = { version = "0.8.2", features = ["v4"] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+2 -2
View File
@@ -24,7 +24,7 @@ redis_common = { path = "../redis_common" }
# External
anyhow = "1.0.75"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
log = "0.4.14"
r2d2_redis = "0.14.0"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -17,7 +17,7 @@ path = "src/lib.rs"
dotenv = "0.15.0"
env_logger = "0.10.1"
log = "0.4.20"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+2 -6
View File
@@ -25,7 +25,7 @@ tokens = { path = "../../_types/tokens" }
# External
anyhow = "1.0.75"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
dotenv = "0.15.0"
log = "0.4.14"
@@ -39,14 +39,10 @@ serde_derive = "1.0.125"
serde_json = "1.0.108"
toml = "0.5.8"
serial_test = "2.0.0"
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
tokio = { version = "1.6.0", features = ["full"] }
rand = "0.8"
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
@@ -167,7 +167,7 @@ FROM email_sender_jobs"#.to_string();
LIMIT ?
"#);
let mut query = sqlx::query_as::<_, AvailableEmailSenderJobRawInternal>(&query)
let query = sqlx::query_as::<_, AvailableEmailSenderJobRawInternal>(&query)
.bind(args.is_debug_worker)
.bind(args.num_records);
@@ -336,7 +336,7 @@ FROM generic_inference_jobs"#.to_string();
LIMIT ?
"#);
let mut query = sqlx::query_as::<_, AvailableInferenceJobRawInternal>(&query)
let query = sqlx::query_as::<_, AvailableInferenceJobRawInternal>(&query)
.bind(args.is_debug_worker)
.bind(args.num_records);
@@ -20,7 +20,7 @@ pub async fn mark_generic_inference_job_failure(
// statuses: "attempt_failed", "complete_failure", "dead"
let mut next_status = "attempt_failed";
let mut maybe_public_failure_reason = maybe_public_failure_reason.map(|reason| {
let maybe_public_failure_reason = maybe_public_failure_reason.map(|reason| {
let mut reason = reason.trim().to_string();
reason.truncate(512); // Max length of column is 512
reason
@@ -14,7 +14,7 @@ use tokens::tokens::users::UserToken;
use crate::helpers::boolean_converters::i8_to_bool;
#[derive(Debug)]
#[derive(Debug, Default)]
pub struct GenericInferenceJobStatus {
pub job_token: InferenceJobToken,
@@ -40,7 +40,7 @@ pub struct GenericInferenceJobStatus {
/// Details about the user's original inference request
/// (We may want to present it in the "pending" UI.)
#[derive(Debug)]
#[derive(Debug, Default)]
pub struct RequestDetails {
pub inference_category: InferenceCategory,
pub maybe_model_type: Option<String>, // TODO: Strongly type
@@ -52,7 +52,7 @@ pub struct RequestDetails {
}
/// Details about the generated result
#[derive(Debug)]
#[derive(Debug, Default)]
pub struct ResultDetails {
pub entity_type: String,
pub entity_token: String,
@@ -71,7 +71,7 @@ pub struct ResultDetails {
/// NB: DO NOT EXPOSE TO FRONTEND.
/// This is used to gate access to job termination
#[derive(Debug)]
#[derive(Debug, Default)]
pub struct UserDetails {
pub maybe_creator_user_token: Option<UserToken>,
pub maybe_creator_anonymous_visitor_token: Option<AnonymousVisitorTrackingToken>,
@@ -168,6 +168,11 @@ WHERE jobs.token = ?
}
};
Ok(Some(raw_record_to_public_result(record)))
}
/// Map the internal record type (since we query over several result tables)
fn raw_record_to_public_result(record: RawGenericInferenceJobStatus) -> GenericInferenceJobStatus {
let maybe_model_title = match record.inference_category {
InferenceCategory::LipsyncAnimation => Some("lipsync animation"),
InferenceCategory::TextToSpeech => record.maybe_tts_model_title.as_deref(),
@@ -213,7 +218,7 @@ WHERE jobs.token = ?
})
});
Ok(Some(GenericInferenceJobStatus {
GenericInferenceJobStatus {
job_token: record.job_token,
status: record.status,
attempt_count: record.attempt_count,
@@ -237,9 +242,10 @@ WHERE jobs.token = ?
is_keepalive_required: i8_to_bool(record.is_keepalive_required),
created_at: record.created_at,
updated_at: record.updated_at,
}))
}
}
#[derive(Debug, Default)]
struct RawGenericInferenceJobStatus {
pub job_token: InferenceJobToken,
@@ -281,3 +287,60 @@ struct RawGenericInferenceJobStatus {
pub maybe_successfully_completed_at: Option<DateTime<Utc>>,
}
#[cfg(test)]
mod tests {
use enums::by_table::generic_inference_jobs::inference_category::InferenceCategory;
use crate::queries::generic_inference::web::get_inference_job_status::{raw_record_to_public_result, RawGenericInferenceJobStatus};
#[test]
fn text_to_speech_as_media_file() {
// NB(bt,2023-12-06): After tts migration to media files, these are the results the job system will return
let record = RawGenericInferenceJobStatus {
inference_category: InferenceCategory::TextToSpeech,
maybe_result_entity_type: Some("media_file".to_string()), // NB: Media file record
maybe_result_entity_token: Some("m_00af018cy75pxytpb8wbdx9jqgtp0p".to_string()),
maybe_media_file_public_bucket_directory_hash: Some("3vb91yq71z5zne56saazn4qntt52yter".to_string()), // NB: Media file
maybe_media_file_public_bucket_prefix: Some("fakeyou_".to_string()),
maybe_media_file_public_bucket_extension: Some(".wav".to_string()),
maybe_tts_public_bucket_path: None, // NB: Not a tts_result!
..Default::default()
};
let record = raw_record_to_public_result(record);
assert!(record.maybe_result_details.is_some());
let result_details = record.maybe_result_details.expect("should have result details");
assert_eq!(&result_details.entity_type, "media_file");
assert_eq!(&result_details.entity_token, "m_00af018cy75pxytpb8wbdx9jqgtp0p");
assert_eq!(&result_details.public_bucket_location_or_hash, "3vb91yq71z5zne56saazn4qntt52yter");
assert_eq!(result_details.public_bucket_location_is_hash, true);
}
#[test]
fn text_to_speech_as_tts_result() {
// NB(bt,2023-12-06): Prior to tts migration to media files, these are the results the job system will return
let record = RawGenericInferenceJobStatus {
inference_category: InferenceCategory::TextToSpeech,
maybe_result_entity_type: Some("text_to_speech".to_string()), // NB: TTS record
maybe_result_entity_token: Some("TR:wefhbk4j3yc8d6zwembedbdw4ad4s".to_string()),
maybe_tts_public_bucket_path: Some("/tts_inference_output/c/5/8/vocodes_c58aeaef-84df-4478-9e7f-c64280a852e8.wav".to_string()), // NB: tts_result!
maybe_media_file_public_bucket_directory_hash: None, // NB: Not a media file!
maybe_media_file_public_bucket_prefix: None,
maybe_media_file_public_bucket_extension: None,
..Default::default()
};
let record = raw_record_to_public_result(record);
assert!(record.maybe_result_details.is_some());
let result_details = record.maybe_result_details.expect("should have result details");
assert_eq!(&result_details.entity_type, "text_to_speech");
assert_eq!(&result_details.entity_token, "TR:wefhbk4j3yc8d6zwembedbdw4ad4s");
assert_eq!(&result_details.public_bucket_location_or_hash, "/tts_inference_output/c/5/8/vocodes_c58aeaef-84df-4478-9e7f-c64280a852e8.wav");
assert_eq!(result_details.public_bucket_location_is_hash, false);
}
}
@@ -81,7 +81,7 @@ WHERE status IN(
separated.push_unseparated(") ");
if let Some(priority) = maybe_priority_or_lower {
query_builder.push(format!(" AND priority <= {priority} "));
query_builder.push(format!(" AND priority_level <= {priority} "));
}
match target {
@@ -136,7 +136,7 @@ mod tests {
let statuses = HashSet::from([KillableStatus::Failed]);
let builder = query_builder(statuses, KillableTarget::AllJobs, Some(10));
let query = cleanup_query(&builder.into_sql());
assert_eq!(query, "UPDATE generic_inference_jobs SET status='cancelled_by_system' WHERE status IN( ?) AND priority <= 10".to_string());
assert_eq!(query, "UPDATE generic_inference_jobs SET status='cancelled_by_system' WHERE status IN( ?) AND priority_level <= 10".to_string());
}
#[test]
@@ -0,0 +1,166 @@
use anyhow::anyhow;
use sqlx;
use sqlx::MySqlPool;
use enums::by_table::generic_synthetic_ids::id_category::IdCategory;
use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory;
use enums::by_table::media_files::media_file_origin_model_type::MediaFileOriginModelType;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use errors::AnyhowResult;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
use crate::queries::generic_inference::job::list_available_generic_inference_jobs::AvailableInferenceJob;
use crate::queries::generic_synthetic_ids::transactional_increment_generic_synthetic_id::transactional_increment_generic_synthetic_id;
pub struct InsertMediaFileArgs<'a> {
pub pool: &'a MySqlPool,
pub job: &'a AvailableInferenceJob,
pub maybe_mime_type: Option<&'a str>,
pub file_size_bytes: u64,
pub duration_millis: u64,
pub sha256_checksum: &'a str,
// TODO: Media duration.
//pub duration_millis: u64,
pub public_bucket_directory_hash: &'a str,
pub maybe_public_bucket_prefix: Option<&'a str>,
pub maybe_public_bucket_extension: Option<&'a str>,
pub is_on_prem: bool,
pub worker_hostname: &'a str,
pub worker_cluster: &'a str,
}
pub async fn insert_media_file_from_tacotron2(
args: InsertMediaFileArgs<'_>
) -> AnyhowResult<(MediaFileToken, u64)>
{
let result_token = MediaFileToken::generate();
let origin_model_type = MediaFileOriginModelType::Tacotron2;
let mut maybe_creator_file_synthetic_id : Option<u64> = None;
let mut maybe_creator_category_synthetic_id : Option<u64> = None;
let mut transaction = args.pool.begin().await?;
if let Some(creator_user_token) = args.job.maybe_creator_user_token.as_deref() {
let user_token = UserToken::new_from_str(creator_user_token);
let next_media_file_id = transactional_increment_generic_synthetic_id(
&user_token,
IdCategory::MediaFile,
&mut transaction
).await?;
let next_voice_conversion_id = transactional_increment_generic_synthetic_id(
&user_token,
IdCategory::TtsResult,
&mut transaction
).await?;
maybe_creator_file_synthetic_id = Some(next_media_file_id);
maybe_creator_category_synthetic_id = Some(next_voice_conversion_id);
}
let vc_model_token = args.job.maybe_model_token.as_deref();
let creator_ip_address = args.job.creator_ip_address.as_str();
let creator_set_visibility = args.job.creator_set_visibility.clone();
const ORIGIN_CATEGORY : MediaFileOriginCategory = MediaFileOriginCategory::Inference;
const ORIGIN_PRODUCT_CATEGORY : MediaFileOriginProductCategory = MediaFileOriginProductCategory::TextToSpeech;
const MEDIA_TYPE : MediaFileType = MediaFileType::Audio;
let record_id = {
let query_result = sqlx::query!(
r#"
INSERT INTO media_files
SET
token = ?,
origin_category = ?,
origin_product_category = ?,
maybe_origin_model_type = ?,
maybe_origin_model_token = ?,
media_type = ?,
maybe_mime_type = ?,
file_size_bytes = ?,
checksum_sha2 = ?,
public_bucket_directory_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
maybe_creator_user_token = ?,
maybe_creator_anonymous_visitor_token = ?,
creator_ip_address = ?,
creator_set_visibility = ?,
maybe_creator_file_synthetic_id = ?,
maybe_creator_category_synthetic_id = ?,
is_generated_on_prem = ?,
generated_by_worker = ?,
generated_by_cluster = ?
"#,
result_token.as_str(),
ORIGIN_CATEGORY.to_str(),
ORIGIN_PRODUCT_CATEGORY.to_str(),
origin_model_type.to_str(),
args.job.maybe_model_token,
MEDIA_TYPE.to_str(),
args.maybe_mime_type,
args.file_size_bytes,
args.sha256_checksum,
args.public_bucket_directory_hash,
args.maybe_public_bucket_prefix,
args.maybe_public_bucket_extension,
args.job.maybe_creator_user_token,
args.job.maybe_creator_anonymous_visitor_token,
args.job.creator_ip_address,
args.job.creator_set_visibility.to_str(),
maybe_creator_file_synthetic_id,
maybe_creator_category_synthetic_id,
args.is_on_prem,
args.worker_hostname,
args.worker_cluster
)
.execute(&mut *transaction)
.await;
let record_id = match query_result {
Ok(res) => {
res.last_insert_id()
},
Err(err) => {
// TODO: handle better
//transaction.rollback().await?;
return Err(anyhow!("Mysql error: {:?}", err));
}
};
record_id
};
transaction.commit().await?;
Ok((result_token, record_id))
}
@@ -0,0 +1,262 @@
use chrono::{DateTime, Utc};
use sqlx::{FromRow, MySql, MySqlPool, QueryBuilder, Row};
use sqlx::mysql::MySqlRow;
use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory;
use enums::by_table::media_files::media_file_origin_model_type::MediaFileOriginModelType;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use enums::traits::mysql_from_row::MySqlFromRow;
use errors::AnyhowResult;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
pub struct MediaFileListPage {
pub records: Vec<MediaFileListItem>,
pub sort_ascending: bool,
/// ID of the first record.
pub first_id: Option<i64>,
/// ID of the last record.
pub last_id: Option<i64>,
}
pub struct MediaFileListItem {
pub token: MediaFileToken,
pub origin_category: MediaFileOriginCategory,
pub origin_product_category: MediaFileOriginProductCategory,
pub maybe_origin_model_type: Option<MediaFileOriginModelType>,
pub maybe_origin_model_token: Option<String>,
pub media_type: MediaFileType,
pub public_bucket_directory_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_creator_user_token: Option<UserToken>,
pub maybe_creator_username: Option<String>,
pub maybe_creator_display_name: Option<String>,
pub maybe_creator_gravatar_hash: Option<String>,
pub creator_set_visibility: Visibility,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
#[derive(Clone, Copy)]
pub enum ViewAs {
Moderator,
NonModerator,
}
pub struct ListMediaFilesArgs<'a> {
pub limit: usize,
pub maybe_filter_media_type: Option<MediaFileType>,
pub maybe_offset: Option<usize>,
pub cursor_is_reversed: bool,
pub view_as: ViewAs,
pub mysql_pool: &'a MySqlPool,
}
pub async fn list_media_files(args: ListMediaFilesArgs<'_>) -> AnyhowResult<MediaFileListPage> {
let mut query = query_builder(
args.maybe_filter_media_type,
args.limit,
args.maybe_offset,
args.cursor_is_reversed,
args.view_as,
);
let query = query.build_query_as::<MediaFileListItemInternal>();
let results = query.fetch_all(args.mysql_pool).await?;
let first_id = results.first()
.map(|raw_result| raw_result.id);
let last_id = results.last()
.map(|raw_result| raw_result.id);
let results = results.into_iter()
.map(|record| {
MediaFileListItem {
token: record.token,
origin_category: record.origin_category,
origin_product_category: record.origin_product_category,
maybe_origin_model_type: record.maybe_origin_model_type,
maybe_origin_model_token: record.maybe_origin_model_token,
media_type: record.media_type,
public_bucket_directory_hash: record.public_bucket_directory_hash,
maybe_public_bucket_prefix: record.maybe_public_bucket_prefix,
maybe_public_bucket_extension: record.maybe_public_bucket_extension,
maybe_creator_user_token: record.maybe_creator_user_token,
maybe_creator_username: record.maybe_creator_username,
maybe_creator_display_name: record.maybe_creator_display_name,
maybe_creator_gravatar_hash: record.maybe_creator_gravatar_hash,
creator_set_visibility: record.creator_set_visibility,
created_at: record.created_at,
updated_at: record.updated_at,
}
})
.collect::<Vec<_>>();
Ok(MediaFileListPage {
records: results,
sort_ascending: !args.cursor_is_reversed,
first_id,
last_id,
})
}
fn query_builder<'a>(
maybe_filter_media_type: Option<MediaFileType>,
limit: usize,
maybe_offset: Option<usize>,
cursor_is_reversed: bool,
view_as: ViewAs,
) -> QueryBuilder<'a, MySql> {
// NB: Query cannot be statically checked by sqlx
let mut query_builder: QueryBuilder<MySql> = QueryBuilder::new(
r#"
SELECT
m.id,
m.token,
m.origin_category,
m.origin_product_category,
m.maybe_origin_model_type,
m.maybe_origin_model_token,
m.media_type,
m.public_bucket_directory_hash,
m.maybe_public_bucket_prefix,
m.maybe_public_bucket_extension,
m.maybe_creator_user_token,
u.username as maybe_creator_username,
u.display_name as maybe_creator_display_name,
u.email_gravatar_hash as maybe_creator_gravatar_hash,
m.creator_set_visibility,
m.created_at,
m.updated_at
FROM media_files AS m
LEFT OUTER JOIN users AS u
ON m.maybe_creator_user_token = u.token
WHERE m.user_deleted_at IS NULL
AND m.mod_deleted_at IS NULL
"#
);
if let Some(media_type) = maybe_filter_media_type {
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.media_file_type = ");
query_builder.push_bind(media_type.to_str());
}
match view_as {
ViewAs::Moderator => {}
ViewAs::NonModerator=> {
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.creator_set_visibility = ");
query_builder.push_bind(Visibility::Public.to_str());
}
}
if let Some(offset) = maybe_offset {
if cursor_is_reversed {
query_builder.push(format!(" AND m.id < {offset} "));
} else {
query_builder.push(format!(" AND m.id > {offset} "));
}
}
if cursor_is_reversed {
query_builder.push(" ORDER BY m.id ASC ");
} else {
query_builder.push(" ORDER BY m.id DESC ");
}
query_builder.push(format!(" LIMIT {limit} "));
query_builder
}
struct MediaFileListItemInternal {
id: i64,
token: MediaFileToken,
origin_category: MediaFileOriginCategory,
origin_product_category: MediaFileOriginProductCategory,
maybe_origin_model_type: Option<MediaFileOriginModelType>,
maybe_origin_model_token: Option<String>,
media_type: MediaFileType,
public_bucket_directory_hash: String,
maybe_public_bucket_prefix: Option<String>,
maybe_public_bucket_extension: Option<String>,
maybe_creator_user_token: Option<UserToken>,
maybe_creator_username: Option<String>,
maybe_creator_display_name: Option<String>,
maybe_creator_gravatar_hash: Option<String>,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
// NB(bt,2023-12-05): There's an issue with type hinting in the `as` clauses with QueryBuilder (or
// raw query strings) and sqlx::FromRow, regardless of whether it is derived of manually
// implemented. Perhaps this will improve in the future, but for now manually constructed queries
// cannot have type hints, eg. the following:
//
// m.token as `token: tokens::tokens::media_files::MediaFileToken`,
// m.origin_category as `origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory`,
// m.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
//
// This results in the automatic mapping not being able to be found by name (for macro derive), and
// in the manual case `row.try_get()` etc. won't have the correct column name (since the name is the
// full "as" clause).
impl FromRow<'_, MySqlRow> for MediaFileListItemInternal {
fn from_row(row: &MySqlRow) -> Result<Self, sqlx::Error> {
let maybe_creator_user_token : Option<String> = row.try_get("maybe_creator_user_token")?;
let maybe_creator_user_token = maybe_creator_user_token.map(|user_token| UserToken::new(user_token));
Ok(Self {
id: row.try_get("id")?,
token: MediaFileToken::new(row.try_get("token")?),
origin_category: MediaFileOriginCategory::try_from_mysql_row(row, "origin_category")?,
origin_product_category: MediaFileOriginProductCategory::try_from_mysql_row(row, "origin_product_category")?,
maybe_origin_model_type: MediaFileOriginModelType::try_from_mysql_row_nullable(row, "maybe_origin_model_type")?,
maybe_origin_model_token: row.try_get("maybe_origin_model_token")?,
media_type: MediaFileType::try_from_mysql_row(row, "media_type")?,
public_bucket_directory_hash: row.try_get("public_bucket_directory_hash")?,
maybe_public_bucket_prefix: row.try_get("maybe_public_bucket_prefix")?,
maybe_public_bucket_extension: row.try_get("maybe_public_bucket_extension")?,
maybe_creator_user_token,
maybe_creator_username: row.try_get("maybe_creator_username")?,
maybe_creator_display_name: row.try_get("maybe_creator_display_name")?,
maybe_creator_gravatar_hash: row.try_get("maybe_creator_gravatar_hash")?,
creator_set_visibility: Visibility::try_from_mysql_row(row, "creator_set_visibility")?,
created_at: row.try_get("created_at")?,
updated_at: row.try_get("updated_at")?,
})
}
}
@@ -0,0 +1,243 @@
use chrono::{DateTime, Utc};
use sqlx::{FromRow, MySql, MySqlPool, QueryBuilder, Row};
use sqlx::mysql::MySqlRow;
use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory;
use enums::by_table::media_files::media_file_origin_model_type::MediaFileOriginModelType;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use enums::traits::mysql_from_row::MySqlFromRow;
use errors::AnyhowResult;
use tokens::tokens::media_files::MediaFileToken;
pub struct MediaFileListPage {
pub records: Vec<MediaFileListItem>,
pub sort_ascending: bool,
/// ID of the first record.
pub first_id: Option<i64>,
/// ID of the last record.
pub last_id: Option<i64>,
}
pub struct MediaFileListItem {
pub token: MediaFileToken,
pub origin_category: MediaFileOriginCategory,
pub origin_product_category: MediaFileOriginProductCategory,
pub maybe_origin_model_type: Option<MediaFileOriginModelType>,
pub maybe_origin_model_token: Option<String>,
pub media_type: MediaFileType,
pub public_bucket_directory_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub creator_set_visibility: Visibility,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
#[derive(Clone, Copy)]
pub enum ViewAs {
Author,
Moderator,
AnotherUser,
}
pub struct ListMediaFileForUserArgs<'a> {
pub username: &'a str,
pub limit: usize,
pub maybe_filter_media_type: Option<MediaFileType>,
pub maybe_offset: Option<usize>,
pub cursor_is_reversed: bool,
pub view_as: ViewAs,
pub mysql_pool: &'a MySqlPool,
}
pub async fn list_media_files_for_user(args: ListMediaFileForUserArgs<'_>) -> AnyhowResult<MediaFileListPage> {
let mut query = query_builder(
args.maybe_filter_media_type,
args.username,
args.limit,
args.maybe_offset,
args.cursor_is_reversed,
args.view_as,
);
let query = query.build_query_as::<MediaFileListItemInternal>();
let results = query.fetch_all(args.mysql_pool).await?;
let first_id = results.first()
.map(|raw_result| raw_result.id);
let last_id = results.last()
.map(|raw_result| raw_result.id);
let results = results.into_iter()
.map(|record| {
MediaFileListItem {
token: record.token,
origin_category: record.origin_category,
origin_product_category: record.origin_product_category,
maybe_origin_model_type: record.maybe_origin_model_type,
maybe_origin_model_token: record.maybe_origin_model_token,
media_type: record.media_type,
public_bucket_directory_hash: record.public_bucket_directory_hash,
maybe_public_bucket_prefix: record.maybe_public_bucket_prefix,
maybe_public_bucket_extension: record.maybe_public_bucket_extension,
creator_set_visibility: record.creator_set_visibility,
created_at: record.created_at,
updated_at: record.updated_at,
}
})
.collect::<Vec<_>>();
Ok(MediaFileListPage {
records: results,
sort_ascending: !args.cursor_is_reversed,
first_id,
last_id,
})
}
fn query_builder<'a>(
maybe_filter_media_type: Option<MediaFileType>,
username: &'a str,
limit: usize,
maybe_offset: Option<usize>,
cursor_is_reversed: bool,
view_as: ViewAs,
) -> QueryBuilder<'a, MySql> {
// NB: Query cannot be statically checked by sqlx
let mut query_builder: QueryBuilder<MySql> = QueryBuilder::new(
r#"
SELECT
m.id,
m.token,
m.origin_category,
m.origin_product_category,
m.maybe_origin_model_type,
m.maybe_origin_model_token,
m.media_type,
m.public_bucket_directory_hash,
m.maybe_public_bucket_prefix,
m.maybe_public_bucket_extension,
m.creator_set_visibility,
m.created_at,
m.updated_at
FROM media_files AS m
LEFT OUTER JOIN users AS u
ON m.maybe_creator_user_token = u.token
WHERE m.user_deleted_at IS NULL
AND m.mod_deleted_at IS NULL
"#
);
query_builder.push(" AND u.username = ");
query_builder.push_bind(username);
if let Some(media_type) = maybe_filter_media_type {
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.media_file_type = ");
query_builder.push_bind(media_type.to_str());
}
match view_as {
ViewAs::Author => {}
ViewAs::Moderator => {}
ViewAs::AnotherUser => {
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.creator_set_visibility = ");
query_builder.push_bind(Visibility::Public.to_str());
}
}
if let Some(offset) = maybe_offset {
if cursor_is_reversed {
query_builder.push(format!(" AND m.id < {offset} "));
} else {
query_builder.push(format!(" AND m.id > {offset} "));
}
}
if cursor_is_reversed {
query_builder.push(" ORDER BY m.id ASC ");
} else {
query_builder.push(" ORDER BY m.id DESC ");
}
query_builder.push(format!(" LIMIT {limit} "));
query_builder
}
struct MediaFileListItemInternal {
id: i64,
token: MediaFileToken,
origin_category: MediaFileOriginCategory,
origin_product_category: MediaFileOriginProductCategory,
maybe_origin_model_type: Option<MediaFileOriginModelType>,
maybe_origin_model_token: Option<String>,
media_type: MediaFileType,
public_bucket_directory_hash: String,
maybe_public_bucket_prefix: Option<String>,
maybe_public_bucket_extension: Option<String>,
creator_set_visibility: Visibility,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
// NB(bt,2023-12-05): There's an issue with type hinting in the `as` clauses with QueryBuilder (or
// raw query strings) and sqlx::FromRow, regardless of whether it is derived of manually
// implemented. Perhaps this will improve in the future, but for now manually constructed queries
// cannot have type hints, eg. the following:
//
// m.token as `token: tokens::tokens::media_files::MediaFileToken`,
// m.origin_category as `origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory`,
// m.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
//
// This results in the automatic mapping not being able to be found by name (for macro derive), and
// in the manual case `row.try_get()` etc. won't have the correct column name (since the name is the
// full "as" clause).
impl FromRow<'_, MySqlRow> for MediaFileListItemInternal {
fn from_row(row: &MySqlRow) -> Result<Self, sqlx::Error> {
Ok(Self {
id: row.try_get("id")?,
token: MediaFileToken::new(row.try_get("token")?),
origin_category: MediaFileOriginCategory::try_from_mysql_row(row, "origin_category")?,
origin_product_category: MediaFileOriginProductCategory::try_from_mysql_row(row, "origin_product_category")?,
maybe_origin_model_type: MediaFileOriginModelType::try_from_mysql_row_nullable(row, "maybe_origin_model_type")?,
maybe_origin_model_token: row.try_get("maybe_origin_model_token")?,
media_type: MediaFileType::try_from_mysql_row(row, "media_type")?,
public_bucket_directory_hash: row.try_get("public_bucket_directory_hash")?,
maybe_public_bucket_prefix: row.try_get("maybe_public_bucket_prefix")?,
maybe_public_bucket_extension: row.try_get("maybe_public_bucket_extension")?,
creator_set_visibility: Visibility::try_from_mysql_row(row, "creator_set_visibility")?,
created_at: row.try_get("created_at")?,
updated_at: row.try_get("updated_at")?,
})
}
}
@@ -1,7 +1,10 @@
pub mod get_media_file;
pub mod insert_media_file_from_face_animation;
pub mod insert_media_file_from_zero_shot_tts;
pub mod insert_media_file_from_cli_tool;
pub mod insert_media_file_from_voice_conversion;
pub mod delete_media_file;
pub mod get_media_file;
pub mod insert_media_file_from_cli_tool;
pub mod insert_media_file_from_face_animation;
pub mod insert_media_file_from_tacotron2;
pub mod insert_media_file_from_voice_conversion;
pub mod insert_media_file_from_zero_shot_tts;
pub mod list_media_files;
pub mod list_media_files_for_user;
pub mod update_media_file;
@@ -14,7 +14,7 @@ pub struct UpdateMediaFileArgs<'a> {
pub async fn update_media_file(args: UpdateMediaFileArgs<'_>) -> AnyhowResult<()>{
// TODO: enforce checks for idempotency token
let mut transaction = args.mysql_pool.begin().await?;
let transaction = args.mysql_pool.begin().await?;
let query_result = sqlx::query!(
r#"
UPDATE media_files
@@ -184,7 +184,7 @@ impl ListCategoriesQueryBuilder {
query = query.bind(model_type);
}
let mut results = query.fetch_all(&mut **mysql_connection)
let results = query.fetch_all(&mut **mysql_connection)
.await?;
Ok(results)
@@ -239,7 +239,7 @@ LEFT OUTER JOIN users
let mut query = "".to_string();
if let Some(username) = self.scope_creator_user_token.as_deref() {
if let Some(_username) = self.scope_creator_user_token.as_deref() {
if !first_predicate_added {
query.push_str(" WHERE model_categories.creator_user_token = ?");
first_predicate_added = true;
@@ -248,7 +248,7 @@ LEFT OUTER JOIN users
}
}
if let Some(model_type) = self.scope_model_type.as_deref() {
if let Some(_model_type) = self.scope_model_type.as_deref() {
if !first_predicate_added {
query.push_str(" WHERE model_categories.model_type = ?");
first_predicate_added = true;
@@ -16,7 +16,7 @@ pub struct UpdateWeightArgs<'a> {
}
pub async fn update_weights(args: UpdateWeightArgs<'_>) -> AnyhowResult<()> {
let mut transaction = args.mysql_pool.begin().await?;
let transaction = args.mysql_pool.begin().await?;
let query_result = sqlx
::query!(
@@ -63,7 +63,7 @@ pub async fn list_trending_tts_models(
];
let query = query_parts.join(" UNION ");
let mut query = sqlx::query_as::<_, RawTrendingModel>(&query);
let query = sqlx::query_as::<_, RawTrendingModel>(&query);
let results = query.fetch_all(&mut **mysql_connection).await?;
@@ -159,7 +159,7 @@ impl ListAssignedTtsCategoriesQueryBuilder {
// NB: Model token is always used for scoping.
query = query.bind(&self.scope_tts_model_token);
let mut results = query.fetch_all(&mut **mysql_connection)
let results = query.fetch_all(&mut **mysql_connection)
.await?;
Ok(results)
@@ -192,7 +192,7 @@ pub async fn mark_tts_upload_job_done(
{
let status = if success { "complete_success" } else { "complete_failure" };
let query_result = sqlx::query!(
let _query_result = sqlx::query!(
r#"
UPDATE tts_model_upload_jobs
SET
@@ -268,7 +268,7 @@ LEFT OUTER JOIN users
}
}
if let Some(username) = self.scope_creator_username.as_deref() {
if let Some(_username) = self.scope_creator_username.as_deref() {
if !first_predicate_added {
query.push_str(" WHERE users.username = ?");
first_predicate_added = true;
@@ -93,7 +93,7 @@ SET
"#, query_inner_positions, query_inner_ip_addresses, user_token, rule_tokens);
// NB: This, unfortunately, cannot be statically checked.
let mut query = sqlx::query(&query);
let query = sqlx::query(&query);
let result = query.execute(mysql_pool).await;
@@ -145,7 +145,7 @@ impl TwitchOauthTokenFinder {
query = query.bind(twitch_username);
}
let mut result = query.fetch_optional(mysql_pool).await;
let result = query.fetch_optional(mysql_pool).await;
match result {
Ok(Some(record)) => {
@@ -65,10 +65,10 @@ impl ListUsersQueryBuilder {
let internal_results = self.internal_query(mysql_pool).await?;
let mut first_id = internal_results.first()
let first_id = internal_results.first()
.map(|raw_result| raw_result.user_id);
let mut last_id = internal_results.last()
let last_id = internal_results.last()
.map(|raw_result| raw_result.user_id);
let users = internal_results
@@ -106,17 +106,17 @@ impl ListUsersQueryBuilder {
) -> AnyhowResult<Vec<UserForListRaw>> {
let query = self.build_query_string();
let mut query = sqlx::query_as::<_, UserForListRaw>(&query);
let query = sqlx::query_as::<_, UserForListRaw>(&query);
let mut results = query.fetch_all(mysql_pool)
let results = query.fetch_all(mysql_pool)
.await?;
Ok(results)
}
pub fn build_query_string(&self) -> String {
let mut query_string = r#"
let query_string = r#"
SELECT
id as user_id,
token as user_token,
+1 -1
View File
@@ -19,7 +19,7 @@ errors = { path = "../errors" }
# External
bcrypt = "0.8" # password hashes
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
#hmac = "0.10.0" # jwt signing
#jwt = "0.13.0"
#sha2 = "0.9.5" # hmac signing
+1 -1
View File
@@ -27,4 +27,4 @@ serde_json = "1.0.108"
# TODO: Get rid of in favor of std::time
time = "0.3.30"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -15,7 +15,7 @@ path = "src/lib.rs"
[dependencies]
# External
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -6
View File
@@ -20,7 +20,7 @@ errors = { path = "../../lib/errors" }
tokens = { path = "../../_types/tokens" }
# External
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
dotenv = "0.15.0"
log = "0.4.14"
@@ -28,11 +28,6 @@ log = "0.4.14"
once_cell = "1.9.0"
regex = "1.5.4"
# SQlite Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
# ^^^ Granted, this applies to *MySQL* and probably not SQLite for our use case here ^^^
sqlx = { version = "0.7.2", features = [ "sqlite", "runtime-tokio-native-tls", "chrono" ] }
[dev-dependencies]
+1 -5
View File
@@ -16,12 +16,8 @@ path = "src/lib.rs"
# Internal
errors = { path = "../errors" }
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
+1 -1
View File
@@ -14,7 +14,7 @@ path = "src/lib.rs"
[dependencies]
# External
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
serial_test = "2.0.0"
+1 -1
View File
@@ -15,7 +15,7 @@ path = "src/lib.rs"
[dependencies]
# External
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# Serialization
#serde = "1.0.125"
+1 -1
View File
@@ -18,7 +18,7 @@ path = "src/lib.rs"
# Internal
errors = { path = "../errors" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
# External
# None yet
+2 -2
View File
@@ -20,7 +20,7 @@ user_input_common = { path = "../user_input_common" }
# External
anyhow = "1.0.75"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
log = "0.4.14"
once_cell = "1.18.0"
regex = "1.5.4"
@@ -29,4 +29,4 @@ unicode-segmentation = "1.10.1"
# Serialization
serde = "1.0.125"
serde_derive = "1.0.125"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -21,4 +21,4 @@ reusable_types = { path = "../../_types/reusable_types" }
# External
actix-web = "4.4.0"
anyhow = "1.0.75"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+1 -1
View File
@@ -19,4 +19,4 @@ once_cell = "1.18.0"
pulldown-cmark = "0.9.3" # Markdown
regex = "1.5.4"
unicode-segmentation = "1.10.1"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
+2 -2
View File
@@ -19,7 +19,7 @@ errors = { path = "../../lib/errors" }
# External
bytes = "1.3"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
log = "0.4.14"
once_cell = "1.18.0"
strum = { version = "0.25", features = ["derive"] } # Fancy enum iteration, etc.
@@ -35,7 +35,7 @@ reqwest = { version = "0.11", features = ["json"] } # HTTP Client
# Scraping
rss = "2.0.2"
scraper = "0.17.1"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../my-workspace-hack" }
[dev-dependencies]
# None yet
@@ -142,22 +142,22 @@ url = { version = "2.4.1", features = ["serde"] }
[target.x86_64-unknown-linux-gnu.dependencies]
openssl = { version = "0.10.59", features = ["vendored"] }
openssl-sys = { version = "0.9.95", default-features = false, features = ["vendored"] }
rustix = { version = "0.38.25", features = ["fs", "termios"] }
rustix = { version = "0.38.26", features = ["fs", "termios"] }
spin = { version = "0.9.8" }
[target.x86_64-unknown-linux-gnu.build-dependencies]
openssl = { version = "0.10.59", features = ["vendored"] }
openssl-sys = { version = "0.9.95", default-features = false, features = ["vendored"] }
rustix = { version = "0.38.25", features = ["fs", "termios"] }
rustix = { version = "0.38.26", features = ["fs", "termios"] }
socket2 = { version = "0.5.5", default-features = false, features = ["all"] }
spin = { version = "0.9.8" }
[target.x86_64-apple-darwin.dependencies]
rustix = { version = "0.38.25", features = ["fs", "termios"] }
rustix = { version = "0.38.26", features = ["fs", "termios"] }
spin = { version = "0.9.8" }
[target.x86_64-apple-darwin.build-dependencies]
rustix = { version = "0.38.25", features = ["fs", "termios"] }
rustix = { version = "0.38.26", features = ["fs", "termios"] }
socket2 = { version = "0.5.5", default-features = false, features = ["all"] }
spin = { version = "0.9.8" }
+1 -6
View File
@@ -32,7 +32,7 @@ web_scrapers = { path = "../../lib/web_scrapers" }
# External
base64 = "0.21.0"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
clap = "2.34.0" # TODO: This is really out of date, but has conflicts when updated
dotenv = "0.15.0"
log = "0.4.14"
@@ -42,11 +42,6 @@ rand = "0.8.3"
regex = "1.5.4"
strum = { version = "0.24", features = ["derive"] } # Fancy enum iteration, etc.
# SQlite Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
# ^^^ Granted, this applies to *MySQL* and probably not SQLite for our use case here ^^^
sqlx = { version = "0.7.2", features = [ "sqlite", "runtime-tokio-native-tls", "chrono" ] }
# Async Runtime
+2 -6
View File
@@ -26,16 +26,12 @@ tokens = { path = "../../../_types/tokens" }
# External
anyhow = "1.0.75"
chrono = { version = "=0.4.22", features = ["serde"] }
chrono = { version = "0.4.22", features = ["serde"] }
dotenv = "0.15.0"
hostname = "0.3.1"
log = "0.4.14"
rand = "0.8.5"
# MySql Database
# "offline" supports typesafe builds without a live database
# "rustls" doesn't appear to work with IP addresses, per:
# https://github.com/launchbadge/sqlx/issues/1095#issuecomment-798050828
sqlx = { version = "0.7.2", features = [ "mysql", "runtime-tokio", "tls-rustls", "chrono" ] }
# Runtime
@@ -46,4 +42,4 @@ tokio = { version = "1.6.0", features = ["macros"] }
serde = "1.0.125"
serde_derive = "1.0.125"
serde_json = "1.0.108"
my-workspace-hack = { version = "0.1", path = "../../../../my-workspace-hack" }
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }

Some files were not shown because too many files have changed in this diff Show More