Compare commits

...
Author SHA1 Message Date
Kasisnu afa91c032e fix typos in ratings endpoints docs 2023-12-27 16:43:07 +05:30
Kasisnu a76807ab7d document user rating endpoint 2023-12-27 16:32:09 +05:30
Kasisnu 15c1cb8e5e list weights/endpoints return public/private results based on user session 2023-12-27 16:04:10 +05:30
Kasisnu 4a69589f1d Merge pull request #247 from storytold/dec-26-mithoo
minor query builder updates for editing weights and paginatinating weights explore
2023-12-27 11:55:16 +05:30
Kasisnu ed72cf7f7f rename per_page -> page_size for consistency 2023-12-27 11:54:30 +05:30
Kasisnu ad093d3fa8 explore weights query builder offset 2023-12-26 20:02:39 +05:30
Kasisnu b505896864 update weights w/query builder 2023-12-26 16:16:47 +05:30
Brandon Thomas b680002c6b fix download job 2023-12-24 03:22:50 -05:00
Brandon Thomas 40bb91f024 fix lints 2023-12-24 02:42:58 -05:00
Brandon Thomas fa002d3fe3 Merge pull request #245 from storytold/brandon-make-model-weights-descriptions-optional
make model_weights descriptions optional
2023-12-24 02:25:12 -05:00
Brandon Thomas 02bec3655c make model_weights descriptions optional
- these will be bad to have on low-value weights, eg. zero shot.
- want to capture this change now
2023-12-24 02:24:20 -05:00
Brandon Thomas 12b7f4d25e fix broken download job query 2023-12-24 01:40:20 -05:00
Brandon Thomas 02d6f6c940 fix a class of model download failures 2023-12-24 01:24:58 -05:00
Brandon Thomas f5d7c93c22 media file scoping query bug fix 2023-12-23 23:46:03 -05:00
ArEnSc 0125eba7cc Merge pull request #244 from storytold/feature/missing-response-values
Fixed added weights type by request.
2023-12-23 23:26:49 -05:00
Michael Chung 30127e19e9 Fixed added weights type by request. 2023-12-23 23:25:43 -05:00
ArEnSc 033e42d1b1 Merge pull request #243 from storytold/feature/missing-response-values
Add missing data in response
2023-12-23 15:02:13 -05:00
Michael Chung c50a2b36aa Add missing data in response
get weight endpoint to have updated_at (the list weight has but the single weight doesnt)
profile weight list endpoint weights_category is missing from the response, i need it for rendering the cards by what type of weight it is.
2023-12-23 14:46:45 -05:00
Brandon Thomas aa0632bd41 Merge pull request #242 from storytold/dec-21-mithoo
media files pagination fix
2023-12-22 14:11:41 -05:00
Brandon Thomas 87cc49982e model weights es document 2023-12-22 12:57:50 -05:00
Brandon Thomas 853943f21a mimetype reuse 2023-12-22 11:01:21 -05:00
Kasisnu 6f9dd4d6e3 make user bookmarks pagination consistent with media/weights 2023-12-22 18:15:29 +05:30
Kasisnu 5de56cd11b make public weights & media list behave similarly wrt pagination 2023-12-22 13:14:31 +05:30
Kasisnu df16a4aa89 media files pagination fix 2023-12-22 13:06:30 +05:30
Brandon Thomas 613289f1ab update docs 2023-12-21 23:41:37 -05:00
Brandon Thomas ae49b9da7d fix image uploads 2023-12-21 23:16:15 -05:00
Brandon Thomas 3f531ae9f0 API doc, fix field name 2023-12-21 23:08:13 -05:00
Brandon Thomas 370bcf3f2d Merge pull request #241 from storytold/brandon-rename-avatar-to-cover
rename avatar to cover_image
2023-12-21 22:54:15 -05:00
Brandon Thomas 685264e20c fix seeding; codegen 2023-12-21 22:53:28 -05:00
Brandon Thomas 2c7400bf76 update code references (avatar -> cover_image) 2023-12-21 22:49:44 -05:00
Brandon Thomas fae36d485b migrations to simplify media files on a model 2023-12-21 22:26:41 -05:00
Brandon Thomas fe3107b1b0 fix lints/tests failures 2023-12-21 22:11:00 -05:00
Brandon Thomas bdde5da223 Merge pull request #240 from storytold/brandon-more-avatar-images
more "avatar" images on weights
2023-12-21 21:06:27 -05:00
Brandon Thomas 7539237ee4 more "avatar" images on weights 2023-12-21 21:05:14 -05:00
ArEnSc 49796c69e3 Merge pull request #239 from storytold/feature/weights_files
Feature/weights files
2023-12-21 20:51:39 -05:00
Brandon Thomas 268346f662 delete more easily 2023-12-21 19:40:33 -05:00
Brandon Thomas 7897ef730e refactor, fix tests 2023-12-21 19:04:56 -05:00
Brandon Thomas 43da5c91ad fix compile error 2023-12-21 18:47:28 -05:00
Brandon Thomas 71360879f3 cleanup 2023-12-21 18:43:18 -05:00
ArEnSc 7f80e106bd Merge pull request #238 from storytold/dec-21-mithoo
use public pagination for profile weights list endpoint
2023-12-21 14:30:17 -05:00
Kasisnu 58181b32cf use public pagination for profile weights list endpoint 2023-12-22 00:45:24 +05:30
ArEnSc 62c5d86584 Merge pull request #237 from storytold/dec-21-mithoo
Document user bookmarks endpoints
2023-12-21 12:34:48 -05:00
Brooks Palin cbb950cb40 Update general_setup.md
Fixed path for for storyteller-web
2023-12-21 10:30:36 -07:00
Kasisnu 64be9e49af Document user bookmarks endpoints 2023-12-21 19:52:44 +05:30
Brandon Thomas bd688703df Merge pull request #236 from storytold/brandon-creator-details
add creator details
2023-12-21 04:29:36 -05:00
Brandon Thomas c627c13e2d add creator details 2023-12-21 04:28:55 -05:00
Brandon Thomas 84104d91ac Merge pull request #235 from storytold/brandon-model-weight-avatars
model weight avatars
2023-12-21 04:13:58 -05:00
Brandon Thomas 64806135c5 model weight avatars 2023-12-21 04:13:44 -05:00
Brandon Thomas 5e48a6193a reorganize modules 2023-12-21 02:57:00 -05:00
Brandon Thomas 11c17f0a1d fix lints 2023-12-21 02:37:29 -05:00
Brandon Thomas cd91a25c3d Merge pull request #219 from storytold/dec-18-mithoo
document media files list endpoints, fix public page offset bug
2023-12-21 02:11:19 -05:00
Kasisnu c52d197717 sort user media file results by creation date 2023-12-21 12:35:44 +05:30
kasisnu 79d64c51ad fix bug with page offsets for public pagination 2023-12-21 12:35:44 +05:30
kasisnu 3e8d7be00e add remaining media files endpoints to openapispec 2023-12-21 12:35:42 +05:30
kasisnu 3a5b765e51 send favorite/comment counts with media file list endpoint 2023-12-21 12:33:04 +05:30
Brandon Thomas 6df4804e2a Merge pull request #228 from storytold/dec-19-mithoo
fail voice conversion jobs after model deleted
2023-12-21 02:02:17 -05:00
Brandon Thomas e6ac364622 Merge pull request #234 from storytold/brandon-rename-model-weights-bucket-fields
rename model_weights bucket fields: private -> public
2023-12-21 01:31:28 -05:00
Brandon Thomas 0cff684e97 rename model_weights bucket fields: private -> public 2023-12-21 01:30:10 -05:00
Brandon Thomas d698168f9e Merge pull request #233 from storytold/brandon-upload-media-file
media file upload
2023-12-21 00:40:02 -05:00
Brandon Thomas dd41c268e4 media file upload 2023-12-21 00:38:04 -05:00
Brandon Thomas d5280964bc Merge pull request #232 from storytold/brandon-fix-download-job-and-api
Fix download-job; fix Victor's API issue
2023-12-20 21:38:48 -05:00
Brandon Thomas c715d6795c Fix download-job; fix Victor's API issue 2023-12-20 21:37:48 -05:00
Brandon Thomas 4159191d68 Merge pull request #231 from storytold/brandon-tts-migration
work on tts model migration
2023-12-20 18:18:14 -05:00
Brandon Thomas 5e04cc6c2f work on tts model migration 2023-12-20 18:15:05 -05:00
Brandon Thomas c49ec5af2f Merge pull request #230 from storytold/brandon-run-rvc-svc-migration
migration cleanup
2023-12-19 23:23:20 -05:00
Brandon Thomas 44d75baeec update old record with token 2023-12-19 22:26:04 -05:00
Brandon Thomas 922e76eeb1 migration fix wip 2023-12-19 21:53:04 -05:00
Brandon Thomas 3b2f386cdb Merge pull request #229 from storytold/brandon-paths-fix
fix/cleanup some paths, fix tests
2023-12-19 20:44:35 -05:00
Brandon Thomas 61f30618b0 fix/cleanup some paths, fix tests 2023-12-19 20:43:27 -05:00
Kasisnu c116ad90d0 fail voice conversion jobs after model deleted 2023-12-19 16:45:38 +05:30
147 changed files with 4528 additions and 2861 deletions
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO model_weights\nSET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_cover_image_media_file_token = NULL,\n maybe_description_markdown = ?,\n maybe_description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n\n creator_set_visibility = ?,\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n\n public_bucket_hash = ?,\n maybe_public_bucket_prefix = ?,\n maybe_public_bucket_extension = ?,\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n\n maybe_migration_old_model_token = ?,\n version = ?,\n created_at = ?,\n updated_at = ?,\n user_deleted_at = ?,\n mod_deleted_at = ?\n\nON DUPLICATE KEY UPDATE\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_cover_image_media_file_token = NULL,\n maybe_description_markdown = ?,\n maybe_description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n public_bucket_hash = ?,\n maybe_public_bucket_prefix = ?,\n maybe_public_bucket_extension = ?,\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n maybe_migration_old_model_token = ?,\n version = ?,\n created_at = ?,\n updated_at = ?,\n user_deleted_at = ?,\n mod_deleted_at = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 43
},
"nullable": []
},
"hash": "0a392b6adb884a4bb4c08fbc2123d40c27e9fbb5c2b688b182f849b0c1cc5735"
}
@@ -1,12 +0,0 @@
{
"db_name": "MySQL",
"query": "\n INSERT INTO model_weights\n SET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_thumbnail_token = ?,\n description_markdown = ?,\n description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n maybe_last_update_user_token = ?,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n private_bucket_hash = ?,\n maybe_private_bucket_prefix = ?,\n maybe_private_bucket_extension = ?,\n cached_user_ratings_total_count = ?,\n cached_user_ratings_positive_count = ?,\n cached_user_ratings_negative_count = ?,\n maybe_cached_user_ratings_ratio = ?,\n version = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 23
},
"nullable": []
},
"hash": "274cfb1e3caca706e3dbc3d95064e0f4b354c936f8de948070854920f265cc41"
}
@@ -1,212 +0,0 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n f.token as `token: tokens::tokens::user_bookmarks::UserBookmarkToken`,\n f.entity_type as `entity_type: enums::by_table::user_bookmarks::user_bookmark_entity_type::UserBookmarkEntityType`,\n f.entity_token,\n f.user_token as `user_token: tokens::tokens::users::UserToken`,\n u.username,\n u.display_name as user_display_name,\n u.email_gravatar_hash as user_gravatar_hash,\n f.created_at,\n f.updated_at,\n f.deleted_at,\n\n media_files.media_type as `maybe_media_file_type: enums::by_table::media_files::media_file_type::MediaFileType`,\n media_files.origin_product_category as `maybe_media_file_origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory`,\n\n model_weights.title as maybe_descriptive_text_model_weight_title,\n tts_models.title as maybe_descriptive_text_tts_model_title,\n tts_results.raw_inference_text as maybe_descriptive_text_tts_result_inference_text,\n users.display_name as maybe_descriptive_text_user_display_name,\n voice_conversion_models.title as maybe_descriptive_text_voice_conversion_model_title,\n zs_voices.title as maybe_descriptive_text_zs_voice_title\n\nFROM\n user_bookmarks AS f\nJOIN users AS u\n ON f.user_token = u.token\n\nLEFT OUTER JOIN model_weights ON model_weights.token = f.entity_token\nLEFT OUTER JOIN media_files ON media_files.token = f.entity_token\nLEFT OUTER JOIN tts_models ON tts_models.token = f.entity_token\nLEFT OUTER JOIN tts_results ON tts_results.token = f.entity_token\nLEFT OUTER JOIN users ON users.token = f.entity_token\nLEFT OUTER JOIN voice_conversion_models ON voice_conversion_models.token = f.entity_token\nLEFT OUTER JOIN zs_voices ON zs_voices.token = f.entity_token\n\nWHERE\n u.username = ?\n AND f.entity_type = ?\n AND f.deleted_at IS NULL\nORDER BY f.id DESC\nLIMIT 50\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token: tokens::tokens::user_bookmarks::UserBookmarkToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "entity_type: enums::by_table::user_bookmarks::user_bookmark_entity_type::UserBookmarkEntityType",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "entity_token",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 3,
"name": "user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 4,
"name": "username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 5,
"name": "user_display_name",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 6,
"name": "user_gravatar_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 7,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 8,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 9,
"name": "deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 10,
"name": "maybe_media_file_type: enums::by_table::media_files::media_file_type::MediaFileType",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 11,
"name": "maybe_media_file_origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 12,
"name": "maybe_descriptive_text_model_weight_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 13,
"name": "maybe_descriptive_text_tts_model_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 14,
"name": "maybe_descriptive_text_tts_result_inference_text",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 15,
"name": "maybe_descriptive_text_user_display_name",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 16,
"name": "maybe_descriptive_text_voice_conversion_model_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 17,
"name": "maybe_descriptive_text_zs_voice_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
}
],
"parameters": {
"Right": 2
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
false,
true,
true,
true,
true,
true,
true,
true,
true,
true
]
},
"hash": "3a7ac7f979ac270dad1070031d0d663d0eb3d969365f6b2fd048ca41c5c455b2"
}
@@ -1,12 +0,0 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO model_weights\nSET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_thumbnail_token = NULL,\n maybe_avatar_media_file_token = NULL,\n maybe_cover_media_file_token = NULL,\n description_markdown = ?,\n description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n\n creator_set_visibility = ?,\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n\n private_bucket_hash = ?,\n maybe_private_bucket_prefix = ?,\n maybe_private_bucket_extension = ?,\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n\n maybe_migration_old_model_token = ?,\n version = ?,\n created_at = ?,\n updated_at = ?,\n user_deleted_at = ?,\n mod_deleted_at = ?\n\nON DUPLICATE KEY UPDATE\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_thumbnail_token = NULL,\n maybe_avatar_media_file_token = NULL,\n maybe_cover_media_file_token = NULL,\n description_markdown = ?,\n description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n private_bucket_hash = ?,\n maybe_private_bucket_prefix = ?,\n maybe_private_bucket_extension = ?,\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n maybe_migration_old_model_token = ?,\n version = ?,\n created_at = ?,\n updated_at = ?,\n user_deleted_at = ?,\n mod_deleted_at = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 43
},
"nullable": []
},
"hash": "4f36ec7baa9fa98cca25b5cc11f09d0eaaa2dd5797374b4c1aa103999e8e1542"
}
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO model_weights\nSET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n\n maybe_cover_image_media_file_token = NULL,\n maybe_description_markdown = NULL,\n maybe_description_rendered_html = NULL,\n\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n\n public_bucket_hash = ?,\n maybe_public_bucket_prefix = ?,\n maybe_public_bucket_extension = ?,\n\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n\n maybe_migration_old_model_token = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 15
},
"nullable": []
},
"hash": "6512f05410514927b52b8cc42f4eda0ad7a256254fd7fcf3b8a1cc429b06ad38"
}
@@ -1,12 +0,0 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO model_weights\nSET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n\n maybe_thumbnail_token = NULL,\n maybe_avatar_media_file_token = NULL,\n maybe_cover_media_file_token = NULL,\n description_markdown = NULL,\n description_rendered_html = NULL,\n\n maybe_last_update_user_token = NULL,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n\n private_bucket_hash = ?,\n maybe_private_bucket_prefix = ?,\n maybe_private_bucket_extension = ?,\n\n cached_user_ratings_total_count = 0,\n cached_user_ratings_positive_count = 0,\n cached_user_ratings_negative_count = 0,\n maybe_cached_user_ratings_ratio = 0.0,\n cached_user_ratings_last_updated_at = NOW(),\n\n maybe_migration_old_model_token = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 15
},
"nullable": []
},
"hash": "685839dd0d875409fc1f3262e234518760c3d5b10c46fb8daf96756746c01c81"
}
@@ -1,322 +0,0 @@
{
"db_name": "MySQL",
"query": "\n SELECT\n wt.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n wt.title,\n wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n wt.maybe_thumbnail_token,\n wt.description_markdown,\n wt.description_rendered_html,\n wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,\n wt.creator_ip_address,\n wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n wt.original_download_url,\n wt.original_filename,\n wt.file_size_bytes,\n wt.file_checksum_sha2,\n wt.private_bucket_hash,\n wt.maybe_private_bucket_prefix,\n wt.maybe_private_bucket_extension,\n wt.cached_user_ratings_negative_count,\n wt.cached_user_ratings_positive_count,\n wt.cached_user_ratings_total_count,\n wt.maybe_cached_user_ratings_ratio,\n wt.cached_user_ratings_last_updated_at,\n wt.version,\n wt.created_at,\n wt.updated_at,\n wt.user_deleted_at,\n wt.mod_deleted_at\n FROM model_weights as wt\n WHERE\n wt.token = ?\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token: tokens::tokens::model_weights::ModelWeightToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "title",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 2,
"name": "weights_type: enums::by_table::model_weights::weights_types::WeightsType",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 3,
"name": "weights_category: enums::by_table::model_weights::weights_category::WeightsCategory",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 4,
"name": "maybe_thumbnail_token",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 5,
"name": "description_markdown",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 6,
"name": "description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 7,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 8,
"name": "creator_ip_address",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 160
}
},
{
"ordinal": 9,
"name": "creator_set_visibility: enums::common::visibility::Visibility",
"type_info": {
"type": "String",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | ENUM",
"char_set": 224,
"max_size": 28
}
},
{
"ordinal": 10,
"name": "maybe_last_update_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 11,
"name": "original_download_url",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 2048
}
},
{
"ordinal": 12,
"name": "original_filename",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 13,
"name": "file_size_bytes",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 14,
"name": "file_checksum_sha2",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 15,
"name": "private_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 16,
"name": "maybe_private_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 17,
"name": "maybe_private_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 18,
"name": "cached_user_ratings_negative_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 19,
"name": "cached_user_ratings_positive_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 20,
"name": "cached_user_ratings_total_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 21,
"name": "maybe_cached_user_ratings_ratio",
"type_info": {
"type": "Float",
"flags": "",
"char_set": 63,
"max_size": 12
}
},
{
"ordinal": 22,
"name": "cached_user_ratings_last_updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 23,
"name": "version",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 11
}
},
{
"ordinal": 24,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 25,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 26,
"name": "user_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 27,
"name": "mod_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false,
false,
true,
false,
false,
false,
false,
false,
true,
true,
true,
false,
false,
false,
true,
true,
false,
false,
false,
true,
false,
false,
false,
false,
true,
true
]
},
"hash": "6a94791c8a8f66ee54bb367a4e8c71106bbcbbb205f54c14e3657279e83dabb6"
}
@@ -1,212 +0,0 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n f.token as `token: tokens::tokens::user_bookmarks::UserBookmarkToken`,\n f.entity_type as `entity_type: enums::by_table::user_bookmarks::user_bookmark_entity_type::UserBookmarkEntityType`,\n f.entity_token,\n f.user_token as `user_token: tokens::tokens::users::UserToken`,\n u.username,\n u.display_name as user_display_name,\n u.email_gravatar_hash as user_gravatar_hash,\n f.created_at,\n f.updated_at,\n f.deleted_at,\n\n media_files.media_type as `maybe_media_file_type: enums::by_table::media_files::media_file_type::MediaFileType`,\n media_files.origin_product_category as `maybe_media_file_origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory`,\n\n model_weights.title as maybe_descriptive_text_model_weight_title,\n tts_models.title as maybe_descriptive_text_tts_model_title,\n tts_results.raw_inference_text as maybe_descriptive_text_tts_result_inference_text,\n users.display_name as maybe_descriptive_text_user_display_name,\n voice_conversion_models.title as maybe_descriptive_text_voice_conversion_model_title,\n zs_voices.title as maybe_descriptive_text_zs_voice_title\n\nFROM\n user_bookmarks AS f\nJOIN users AS u\n ON f.user_token = u.token\n\nLEFT OUTER JOIN model_weights ON model_weights.token = f.entity_token\nLEFT OUTER JOIN media_files ON media_files.token = f.entity_token\nLEFT OUTER JOIN tts_models ON tts_models.token = f.entity_token\nLEFT OUTER JOIN tts_results ON tts_results.token = f.entity_token\nLEFT OUTER JOIN users ON users.token = f.entity_token\nLEFT OUTER JOIN voice_conversion_models ON voice_conversion_models.token = f.entity_token\nLEFT OUTER JOIN zs_voices ON zs_voices.token = f.entity_token\n\nWHERE\n u.username = ?\n AND f.deleted_at IS NULL\nORDER BY f.id DESC\nLIMIT 50\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token: tokens::tokens::user_bookmarks::UserBookmarkToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "entity_type: enums::by_table::user_bookmarks::user_bookmark_entity_type::UserBookmarkEntityType",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "entity_token",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 3,
"name": "user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 4,
"name": "username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 5,
"name": "user_display_name",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 6,
"name": "user_gravatar_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 7,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 8,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 9,
"name": "deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 10,
"name": "maybe_media_file_type: enums::by_table::media_files::media_file_type::MediaFileType",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 11,
"name": "maybe_media_file_origin_category: enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 12,
"name": "maybe_descriptive_text_model_weight_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 13,
"name": "maybe_descriptive_text_tts_model_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 14,
"name": "maybe_descriptive_text_tts_result_inference_text",
"type_info": {
"type": "Blob",
"flags": "BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 15,
"name": "maybe_descriptive_text_user_display_name",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 16,
"name": "maybe_descriptive_text_voice_conversion_model_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 17,
"name": "maybe_descriptive_text_zs_voice_title",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
false,
true,
true,
true,
true,
true,
true,
true,
true,
true
]
},
"hash": "707af35ee4ab4519f28e6d69982944666e6c7c0f24b5da79dde828ffdb2c8302"
}
@@ -1,6 +1,6 @@
{
"db_name": "MySQL",
"query": "\n SELECT\n mw.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n mw.title,\n mw.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n mw.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n mw.maybe_thumbnail_token,\n mw.description_markdown,\n mw.description_rendered_html,\n users.token as `creator_user_token: tokens::tokens::users::UserToken`,\n users.username as creator_username,\n users.display_name as creator_display_name,\n users.email_gravatar_hash as creator_email_gravatar_hash,\n mw.creator_ip_address,\n mw.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n mw.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n mw.original_download_url,\n mw.original_filename,\n mw.file_size_bytes,\n mw.file_checksum_sha2,\n mw.private_bucket_hash,\n mw.maybe_private_bucket_prefix,\n mw.maybe_private_bucket_extension,\n mw.cached_user_ratings_negative_count,\n mw.cached_user_ratings_positive_count,\n mw.cached_user_ratings_total_count,\n mw.maybe_cached_user_ratings_ratio,\n mw.cached_user_ratings_last_updated_at,\n mw.version,\n mw.created_at,\n mw.updated_at,\n mw.user_deleted_at,\n mw.mod_deleted_at\n FROM model_weights as mw\n JOIN users\n ON users.token = mw.creator_user_token\n WHERE\n users.username = ?\n AND mw.user_deleted_at IS NULL\n AND mw.mod_deleted_at IS NULL\n ",
"query": "\n SELECT\n wt.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n wt.title,\n wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n wt.maybe_description_markdown,\n wt.maybe_description_rendered_html,\n\n wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,\n users.username as creator_username,\n users.display_name as creator_display_name,\n users.email_gravatar_hash AS creator_gravatar_hash,\n\n wt.creator_ip_address,\n wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n wt.original_download_url,\n wt.original_filename,\n wt.file_size_bytes,\n wt.file_checksum_sha2,\n\n wt.public_bucket_hash,\n wt.maybe_public_bucket_prefix,\n wt.maybe_public_bucket_extension,\n\n cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,\n cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,\n cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,\n\n wt.cached_user_ratings_negative_count,\n wt.cached_user_ratings_positive_count,\n wt.cached_user_ratings_total_count,\n wt.maybe_cached_user_ratings_ratio,\n wt.cached_user_ratings_last_updated_at,\n wt.version,\n wt.created_at,\n wt.updated_at,\n wt.user_deleted_at,\n wt.mod_deleted_at\n FROM model_weights as wt\n JOIN users\n ON users.token = wt.creator_user_token\n LEFT OUTER JOIN media_files as cover_image\n ON cover_image.token = wt.maybe_cover_image_media_file_token\n WHERE\n wt.token = ?\n ",
"describe": {
"columns": [
{
@@ -45,46 +45,36 @@
},
{
"ordinal": 4,
"name": "maybe_thumbnail_token",
"name": "maybe_description_markdown",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 128
"max_size": 262140
}
},
{
"ordinal": 5,
"name": "description_markdown",
"name": "maybe_description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 6,
"name": "description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 7,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 8,
"ordinal": 7,
"name": "creator_username",
"type_info": {
"type": "VarString",
@@ -94,7 +84,7 @@
}
},
{
"ordinal": 9,
"ordinal": 8,
"name": "creator_display_name",
"type_info": {
"type": "VarString",
@@ -104,8 +94,8 @@
}
},
{
"ordinal": 10,
"name": "creator_email_gravatar_hash",
"ordinal": 9,
"name": "creator_gravatar_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
@@ -114,7 +104,7 @@
}
},
{
"ordinal": 11,
"ordinal": 10,
"name": "creator_ip_address",
"type_info": {
"type": "VarString",
@@ -124,7 +114,7 @@
}
},
{
"ordinal": 12,
"ordinal": 11,
"name": "creator_set_visibility: enums::common::visibility::Visibility",
"type_info": {
"type": "String",
@@ -134,7 +124,7 @@
}
},
{
"ordinal": 13,
"ordinal": 12,
"name": "maybe_last_update_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
@@ -144,7 +134,7 @@
}
},
{
"ordinal": 14,
"ordinal": 13,
"name": "original_download_url",
"type_info": {
"type": "VarString",
@@ -154,7 +144,7 @@
}
},
{
"ordinal": 15,
"ordinal": 14,
"name": "original_filename",
"type_info": {
"type": "VarString",
@@ -164,7 +154,7 @@
}
},
{
"ordinal": 16,
"ordinal": 15,
"name": "file_size_bytes",
"type_info": {
"type": "Long",
@@ -174,7 +164,7 @@
}
},
{
"ordinal": 17,
"ordinal": 16,
"name": "file_checksum_sha2",
"type_info": {
"type": "String",
@@ -184,8 +174,8 @@
}
},
{
"ordinal": 18,
"name": "private_bucket_hash",
"ordinal": 17,
"name": "public_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
@@ -193,9 +183,19 @@
"max_size": 128
}
},
{
"ordinal": 18,
"name": "maybe_public_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 19,
"name": "maybe_private_bucket_prefix",
"name": "maybe_public_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -205,7 +205,17 @@
},
{
"ordinal": 20,
"name": "maybe_private_bucket_extension",
"name": "maybe_cover_image_public_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 21,
"name": "maybe_cover_image_public_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -214,7 +224,17 @@
}
},
{
"ordinal": 21,
"ordinal": 22,
"name": "maybe_cover_image_public_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 23,
"name": "cached_user_ratings_negative_count",
"type_info": {
"type": "Long",
@@ -224,7 +244,7 @@
}
},
{
"ordinal": 22,
"ordinal": 24,
"name": "cached_user_ratings_positive_count",
"type_info": {
"type": "Long",
@@ -234,7 +254,7 @@
}
},
{
"ordinal": 23,
"ordinal": 25,
"name": "cached_user_ratings_total_count",
"type_info": {
"type": "Long",
@@ -244,7 +264,7 @@
}
},
{
"ordinal": 24,
"ordinal": 26,
"name": "maybe_cached_user_ratings_ratio",
"type_info": {
"type": "Float",
@@ -254,7 +274,7 @@
}
},
{
"ordinal": 25,
"ordinal": 27,
"name": "cached_user_ratings_last_updated_at",
"type_info": {
"type": "Timestamp",
@@ -264,7 +284,7 @@
}
},
{
"ordinal": 26,
"ordinal": 28,
"name": "version",
"type_info": {
"type": "Long",
@@ -274,7 +294,7 @@
}
},
{
"ordinal": 27,
"ordinal": 29,
"name": "created_at",
"type_info": {
"type": "Timestamp",
@@ -284,7 +304,7 @@
}
},
{
"ordinal": 28,
"ordinal": 30,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
@@ -294,7 +314,7 @@
}
},
{
"ordinal": 29,
"ordinal": 31,
"name": "user_deleted_at",
"type_info": {
"type": "Timestamp",
@@ -304,7 +324,7 @@
}
},
{
"ordinal": 30,
"ordinal": 32,
"name": "mod_deleted_at",
"type_info": {
"type": "Timestamp",
@@ -323,8 +343,7 @@
false,
false,
true,
false,
false,
true,
false,
false,
false,
@@ -339,6 +358,9 @@
false,
true,
true,
true,
true,
true,
false,
false,
false,
@@ -351,5 +373,5 @@
true
]
},
"hash": "a6ed8524fb5339fed17bec6611f8f56b6bdee35bac63c9eacbe21eb032a6a831"
"hash": "7d3806f46b5cc96c0e3df22454190834c0294ad34db2d8271f2e2863a3cca3a1"
}
@@ -1,12 +0,0 @@
{
"db_name": "MySQL",
"query": "\n UPDATE model_weights\n SET\n title = COALESCE(?, title),\n description_markdown = COALESCE(?, description_markdown),\n maybe_thumbnail_token = COALESCE(?, maybe_thumbnail_token),\n description_rendered_html = COALESCE(?, description_rendered_html),\n creator_set_visibility = COALESCE(?, creator_set_visibility),\n version = version + 1\n WHERE token = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "81df77a84fee48398e8a07e053b0393a0e341caf652c18c7160dd0a59247a19a"
}
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\nUPDATE voice_conversion_models\nSET\n maybe_migration_new_model_weights_token = ?\nWHERE token = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "898a2f89d1f3efcb87adaef432cc73117057e44766c330caff6ce2c9d48518fd"
}
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\n INSERT INTO model_weights\n SET\n token = ?,\n weights_type = ?,\n weights_category = ?,\n title = ?,\n maybe_description_markdown = ?,\n maybe_description_rendered_html = ?,\n creator_user_token = ?,\n creator_ip_address = ?,\n creator_set_visibility = ?,\n maybe_last_update_user_token = ?,\n original_download_url = ?,\n original_filename = ?,\n file_size_bytes = ?,\n file_checksum_sha2 = ?,\n public_bucket_hash = ?,\n maybe_public_bucket_prefix = ?,\n maybe_public_bucket_extension = ?,\n cached_user_ratings_total_count = ?,\n cached_user_ratings_positive_count = ?,\n cached_user_ratings_negative_count = ?,\n maybe_cached_user_ratings_ratio = ?,\n version = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 22
},
"nullable": []
},
"hash": "a18c5b1418d71fa9aefd3ae5c440be2afc1f9467808aa66d3b850a4d54c81e4d"
}
@@ -0,0 +1,542 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n id,\n token as `token: tokens::tokens::tts_models::TtsModelToken`,\n\n tts_model_type as `tts_model_type: enums::by_table::tts_models::tts_model_type::TtsModelType`,\n\n text_pipeline_type,\n has_self_contained_vocoder,\n has_self_contained_duration_model,\n has_self_contained_pitch_model,\n text_preprocessing_algorithm,\n\n use_default_mel_multiply_factor,\n maybe_custom_mel_multiply_factor,\n maybe_default_pretrained_vocoder,\n maybe_custom_vocoder_token as `maybe_custom_vocoder_token: tokens::tokens::vocoder_models::VocoderModelToken`,\n\n ietf_language_tag,\n ietf_primary_language_subtag,\n\n title,\n description_markdown,\n description_rendered_html,\n\n voice_name,\n actor_name,\n characteristic,\n version_string,\n\n is_front_page_featured,\n is_twitch_featured,\n\n maybe_suggested_unique_bot_command,\n\n original_download_url,\n original_filename,\n file_size_bytes,\n\n creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,\n\n creator_ip_address_creation,\n creator_ip_address_last_update,\n\n creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n\n private_bucket_hash,\n private_bucket_object_name,\n\n private_bucket_object_is_archive,\n\n user_ratings_total_count,\n user_ratings_positive_count,\n user_ratings_negative_count,\n\n is_public_listing_approved,\n is_locked_from_user_modification,\n is_locked_from_use,\n\n maybe_mod_comments,\n maybe_mod_user_token as `maybe_mod_user_token: tokens::tokens::users::UserToken`,\n\n maybe_migration_new_model_weights_token as `maybe_migration_new_model_weights_token: tokens::tokens::model_weights::ModelWeightToken`,\n\n version,\n\n created_at,\n updated_at,\n\n user_deleted_at,\n mod_deleted_at\n\nFROM tts_models\nWHERE\n id > ?\nORDER BY id ASC\nLIMIT ?\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": {
"type": "LongLong",
"flags": "NOT_NULL | PRIMARY_KEY | AUTO_INCREMENT",
"char_set": 63,
"max_size": 20
}
},
{
"ordinal": 1,
"name": "token: tokens::tokens::tts_models::TtsModelToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 2,
"name": "tts_model_type: enums::by_table::tts_models::tts_model_type::TtsModelType",
"type_info": {
"type": "String",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | ENUM",
"char_set": 224,
"max_size": 60
}
},
{
"ordinal": 3,
"name": "text_pipeline_type",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 4,
"name": "has_self_contained_vocoder",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 5,
"name": "has_self_contained_duration_model",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 6,
"name": "has_self_contained_pitch_model",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 7,
"name": "text_preprocessing_algorithm",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | ENUM",
"char_set": 224,
"max_size": 28
}
},
{
"ordinal": 8,
"name": "use_default_mel_multiply_factor",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 9,
"name": "maybe_custom_mel_multiply_factor",
"type_info": {
"type": "Double",
"flags": "",
"char_set": 63,
"max_size": 22
}
},
{
"ordinal": 10,
"name": "maybe_default_pretrained_vocoder",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 11,
"name": "maybe_custom_vocoder_token: tokens::tokens::vocoder_models::VocoderModelToken",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 12,
"name": "ietf_language_tag",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 13,
"name": "ietf_primary_language_subtag",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY",
"char_set": 224,
"max_size": 48
}
},
{
"ordinal": 14,
"name": "title",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 15,
"name": "description_markdown",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 16,
"name": "description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 17,
"name": "voice_name",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 18,
"name": "actor_name",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 19,
"name": "characteristic",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 20,
"name": "version_string",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 21,
"name": "is_front_page_featured",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 22,
"name": "is_twitch_featured",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 23,
"name": "maybe_suggested_unique_bot_command",
"type_info": {
"type": "VarString",
"flags": "UNIQUE_KEY | BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 24,
"name": "original_download_url",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 2048
}
},
{
"ordinal": 25,
"name": "original_filename",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 26,
"name": "file_size_bytes",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 27,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 28,
"name": "creator_ip_address_creation",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 160
}
},
{
"ordinal": 29,
"name": "creator_ip_address_last_update",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 160
}
},
{
"ordinal": 30,
"name": "creator_set_visibility: enums::common::visibility::Visibility",
"type_info": {
"type": "String",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | ENUM",
"char_set": 224,
"max_size": 28
}
},
{
"ordinal": 31,
"name": "private_bucket_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 32,
"name": "private_bucket_object_name",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 33,
"name": "private_bucket_object_is_archive",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 34,
"name": "user_ratings_total_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 35,
"name": "user_ratings_positive_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 36,
"name": "user_ratings_negative_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 37,
"name": "is_public_listing_approved",
"type_info": {
"type": "Tiny",
"flags": "MULTIPLE_KEY",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 38,
"name": "is_locked_from_user_modification",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL | MULTIPLE_KEY",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 39,
"name": "is_locked_from_use",
"type_info": {
"type": "Tiny",
"flags": "NOT_NULL | MULTIPLE_KEY",
"char_set": 63,
"max_size": 1
}
},
{
"ordinal": 40,
"name": "maybe_mod_comments",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 41,
"name": "maybe_mod_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "MULTIPLE_KEY | BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 42,
"name": "maybe_migration_new_model_weights_token: tokens::tokens::model_weights::ModelWeightToken",
"type_info": {
"type": "VarString",
"flags": "UNIQUE_KEY | BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 43,
"name": "version",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 11
}
},
{
"ordinal": 44,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 45,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 46,
"name": "user_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 47,
"name": "mod_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 2
},
"nullable": [
false,
false,
false,
true,
false,
false,
false,
false,
false,
true,
true,
true,
false,
false,
false,
false,
false,
true,
true,
true,
true,
false,
false,
true,
true,
true,
false,
false,
false,
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
false,
true,
true,
true,
false,
false,
false,
true,
true
]
},
"hash": "a44c637e84152864a00f0ba58838c2f3888e3061fc8b44291541840fdb706e44"
}
@@ -1,6 +1,6 @@
{
"db_name": "MySQL",
"query": "\nSELECT\n w.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n w.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n\n w.title,\n vc.has_index_file,\n\n w.private_bucket_hash,\n w.maybe_private_bucket_prefix,\n w.maybe_private_bucket_extension,\n\n w.created_at,\n w.updated_at,\n w.user_deleted_at,\n w.mod_deleted_at\n\nFROM model_weights as w\nJOIN model_weights_extension_voice_conversion_details as vc\nON vc.model_weights_token = w.token\n\nWHERE w.token = ?\n ",
"query": "\nSELECT\n w.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n w.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n\n w.title,\n vc.has_index_file,\n\n w.public_bucket_hash,\n w.maybe_public_bucket_prefix,\n w.maybe_public_bucket_extension,\n\n w.created_at,\n w.updated_at,\n w.user_deleted_at,\n w.mod_deleted_at\n\nFROM model_weights as w\nJOIN model_weights_extension_voice_conversion_details as vc\nON vc.model_weights_token = w.token\n\nWHERE w.token = ?\n ",
"describe": {
"columns": [
{
@@ -45,7 +45,7 @@
},
{
"ordinal": 4,
"name": "private_bucket_hash",
"name": "public_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
@@ -55,7 +55,7 @@
},
{
"ordinal": 5,
"name": "maybe_private_bucket_prefix",
"name": "maybe_public_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -65,7 +65,7 @@
},
{
"ordinal": 6,
"name": "maybe_private_bucket_extension",
"name": "maybe_public_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -131,5 +131,5 @@
true
]
},
"hash": "0d4c43d9055cea3ce35c71ab8ef27058075414496dc5858d57a9f06d7c076bab"
"hash": "ae34c2ce11250786962be50f167f15f6d2d023afe2bb31994d21a1007d81e0a0"
}
@@ -1,6 +1,6 @@
{
"db_name": "MySQL",
"query": "\n SELECT \n mw.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n mw.title,\n mw.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n mw.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n mw.maybe_thumbnail_token,\n mw.description_markdown,\n mw.description_rendered_html,\n users.token as `creator_user_token: tokens::tokens::users::UserToken`,\n users.username as creator_username,\n users.display_name as creator_display_name,\n users.email_gravatar_hash as creator_email_gravatar_hash,\n mw.creator_ip_address,\n mw.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n mw.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n mw.original_download_url,\n mw.original_filename,\n mw.file_size_bytes,\n mw.file_checksum_sha2,\n mw.private_bucket_hash,\n mw.maybe_private_bucket_prefix,\n mw.maybe_private_bucket_extension,\n mw.cached_user_ratings_negative_count,\n mw.cached_user_ratings_positive_count,\n mw.cached_user_ratings_total_count,\n mw.maybe_cached_user_ratings_ratio,\n mw.cached_user_ratings_last_updated_at,\n mw.version,\n mw.created_at,\n mw.updated_at,\n mw.user_deleted_at,\n mw.mod_deleted_at\n FROM model_weights as mw\n JOIN users\n ON users.token = mw.creator_user_token\n WHERE\n users.username = ?\n ",
"query": "\n SELECT\n wt.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n wt.title,\n wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n wt.maybe_description_markdown,\n wt.maybe_description_rendered_html,\n wt.creator_ip_address,\n wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n\n wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,\n users.username as creator_username,\n users.display_name as creator_display_name,\n users.email_gravatar_hash AS creator_gravatar_hash,\n\n wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n wt.original_download_url,\n wt.original_filename,\n wt.file_size_bytes,\n wt.file_checksum_sha2,\n wt.public_bucket_hash,\n wt.maybe_public_bucket_prefix,\n wt.maybe_public_bucket_extension,\n\n cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,\n cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,\n cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,\n\n wt.cached_user_ratings_negative_count,\n wt.cached_user_ratings_positive_count,\n wt.cached_user_ratings_total_count,\n wt.maybe_cached_user_ratings_ratio,\n wt.cached_user_ratings_last_updated_at,\n wt.version,\n wt.created_at,\n wt.updated_at,\n wt.user_deleted_at,\n wt.mod_deleted_at\n FROM model_weights as wt\n JOIN users\n ON users.token = wt.creator_user_token\n LEFT OUTER JOIN media_files as cover_image\n ON cover_image.token = wt.maybe_cover_image_media_file_token\n WHERE\n wt.token = ?\n AND wt.user_deleted_at IS NULL\n AND wt.mod_deleted_at IS NULL\n ",
"describe": {
"columns": [
{
@@ -45,76 +45,26 @@
},
{
"ordinal": 4,
"name": "maybe_thumbnail_token",
"name": "maybe_description_markdown",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"type": "Blob",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 128
"max_size": 262140
}
},
{
"ordinal": 5,
"name": "description_markdown",
"name": "maybe_description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"flags": "BLOB | BINARY",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 6,
"name": "description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 7,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 8,
"name": "creator_username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 9,
"name": "creator_display_name",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 10,
"name": "creator_email_gravatar_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 11,
"name": "creator_ip_address",
"type_info": {
"type": "VarString",
@@ -124,7 +74,7 @@
}
},
{
"ordinal": 12,
"ordinal": 7,
"name": "creator_set_visibility: enums::common::visibility::Visibility",
"type_info": {
"type": "String",
@@ -134,7 +84,47 @@
}
},
{
"ordinal": 13,
"ordinal": 8,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 9,
"name": "creator_username",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 10,
"name": "creator_display_name",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 80
}
},
{
"ordinal": 11,
"name": "creator_gravatar_hash",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 12,
"name": "maybe_last_update_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
@@ -144,7 +134,7 @@
}
},
{
"ordinal": 14,
"ordinal": 13,
"name": "original_download_url",
"type_info": {
"type": "VarString",
@@ -154,7 +144,7 @@
}
},
{
"ordinal": 15,
"ordinal": 14,
"name": "original_filename",
"type_info": {
"type": "VarString",
@@ -164,7 +154,7 @@
}
},
{
"ordinal": 16,
"ordinal": 15,
"name": "file_size_bytes",
"type_info": {
"type": "Long",
@@ -174,7 +164,7 @@
}
},
{
"ordinal": 17,
"ordinal": 16,
"name": "file_checksum_sha2",
"type_info": {
"type": "String",
@@ -184,8 +174,8 @@
}
},
{
"ordinal": 18,
"name": "private_bucket_hash",
"ordinal": 17,
"name": "public_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
@@ -193,9 +183,19 @@
"max_size": 128
}
},
{
"ordinal": 18,
"name": "maybe_public_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 19,
"name": "maybe_private_bucket_prefix",
"name": "maybe_public_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -205,7 +205,17 @@
},
{
"ordinal": 20,
"name": "maybe_private_bucket_extension",
"name": "maybe_cover_image_public_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 21,
"name": "maybe_cover_image_public_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
@@ -214,7 +224,17 @@
}
},
{
"ordinal": 21,
"ordinal": 22,
"name": "maybe_cover_image_public_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 23,
"name": "cached_user_ratings_negative_count",
"type_info": {
"type": "Long",
@@ -224,7 +244,7 @@
}
},
{
"ordinal": 22,
"ordinal": 24,
"name": "cached_user_ratings_positive_count",
"type_info": {
"type": "Long",
@@ -234,7 +254,7 @@
}
},
{
"ordinal": 23,
"ordinal": 25,
"name": "cached_user_ratings_total_count",
"type_info": {
"type": "Long",
@@ -244,7 +264,7 @@
}
},
{
"ordinal": 24,
"ordinal": 26,
"name": "maybe_cached_user_ratings_ratio",
"type_info": {
"type": "Float",
@@ -254,7 +274,7 @@
}
},
{
"ordinal": 25,
"ordinal": 27,
"name": "cached_user_ratings_last_updated_at",
"type_info": {
"type": "Timestamp",
@@ -264,7 +284,7 @@
}
},
{
"ordinal": 26,
"ordinal": 28,
"name": "version",
"type_info": {
"type": "Long",
@@ -274,7 +294,7 @@
}
},
{
"ordinal": 27,
"ordinal": 29,
"name": "created_at",
"type_info": {
"type": "Timestamp",
@@ -284,7 +304,7 @@
}
},
{
"ordinal": 28,
"ordinal": 30,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
@@ -294,7 +314,7 @@
}
},
{
"ordinal": 29,
"ordinal": 31,
"name": "user_deleted_at",
"type_info": {
"type": "Timestamp",
@@ -304,7 +324,7 @@
}
},
{
"ordinal": 30,
"ordinal": 32,
"name": "mod_deleted_at",
"type_info": {
"type": "Timestamp",
@@ -323,8 +343,7 @@
false,
false,
true,
false,
false,
true,
false,
false,
false,
@@ -339,6 +358,9 @@
false,
true,
true,
true,
true,
true,
false,
false,
false,
@@ -351,5 +373,5 @@
true
]
},
"hash": "3d586592422727dcc928574966bdf5081ac69fdbb856fffcbd68187fd99c4642"
"hash": "bb2c5a81ab6f38ecef7ee177a4ea36e75c5d2180a897094f940f097bbbd66060"
}
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\n UPDATE model_weights\n SET\n maybe_cover_image_media_file_token = ?\n WHERE token = ?\n LIMIT 1\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "d7b28a5b4f0220b02eacb01df030c4d971fd191cf3cb5f0211ce29604d36bca3"
}
@@ -1,322 +0,0 @@
{
"db_name": "MySQL",
"query": "\n SELECT\n wt.token as `token: tokens::tokens::model_weights::ModelWeightToken`,\n wt.title,\n wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,\n wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,\n wt.maybe_thumbnail_token,\n wt.description_markdown,\n wt.description_rendered_html,\n wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,\n wt.creator_ip_address,\n wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,\n wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,\n wt.original_download_url,\n wt.original_filename,\n wt.file_size_bytes,\n wt.file_checksum_sha2,\n wt.private_bucket_hash,\n wt.maybe_private_bucket_prefix,\n wt.maybe_private_bucket_extension,\n wt.cached_user_ratings_negative_count,\n wt.cached_user_ratings_positive_count,\n wt.cached_user_ratings_total_count,\n wt.maybe_cached_user_ratings_ratio,\n wt.cached_user_ratings_last_updated_at,\n wt.version,\n wt.created_at,\n wt.updated_at,\n wt.user_deleted_at,\n wt.mod_deleted_at\n FROM model_weights as wt\n WHERE\n wt.token = ?\n AND wt.user_deleted_at IS NULL\n AND wt.mod_deleted_at IS NULL\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "token: tokens::tokens::model_weights::ModelWeightToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | UNIQUE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 1,
"name": "title",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 2,
"name": "weights_type: enums::by_table::model_weights::weights_types::WeightsType",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 3,
"name": "weights_category: enums::by_table::model_weights::weights_category::WeightsCategory",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 4,
"name": "maybe_thumbnail_token",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 5,
"name": "description_markdown",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 6,
"name": "description_rendered_html",
"type_info": {
"type": "Blob",
"flags": "NOT_NULL | BLOB | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 262140
}
},
{
"ordinal": 7,
"name": "creator_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 8,
"name": "creator_ip_address",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 160
}
},
{
"ordinal": 9,
"name": "creator_set_visibility: enums::common::visibility::Visibility",
"type_info": {
"type": "String",
"flags": "NOT_NULL | MULTIPLE_KEY | BINARY | ENUM",
"char_set": 224,
"max_size": 28
}
},
{
"ordinal": 10,
"name": "maybe_last_update_user_token: tokens::tokens::users::UserToken",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 11,
"name": "original_download_url",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 2048
}
},
{
"ordinal": 12,
"name": "original_filename",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 1020
}
},
{
"ordinal": 13,
"name": "file_size_bytes",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 14,
"name": "file_checksum_sha2",
"type_info": {
"type": "String",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 256
}
},
{
"ordinal": 15,
"name": "private_bucket_hash",
"type_info": {
"type": "VarString",
"flags": "NOT_NULL | BINARY | NO_DEFAULT_VALUE",
"char_set": 224,
"max_size": 128
}
},
{
"ordinal": 16,
"name": "maybe_private_bucket_prefix",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 17,
"name": "maybe_private_bucket_extension",
"type_info": {
"type": "VarString",
"flags": "BINARY",
"char_set": 224,
"max_size": 64
}
},
{
"ordinal": 18,
"name": "cached_user_ratings_negative_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 19,
"name": "cached_user_ratings_positive_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 20,
"name": "cached_user_ratings_total_count",
"type_info": {
"type": "Long",
"flags": "NOT_NULL | UNSIGNED",
"char_set": 63,
"max_size": 10
}
},
{
"ordinal": 21,
"name": "maybe_cached_user_ratings_ratio",
"type_info": {
"type": "Float",
"flags": "",
"char_set": 63,
"max_size": 12
}
},
{
"ordinal": 22,
"name": "cached_user_ratings_last_updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 23,
"name": "version",
"type_info": {
"type": "Long",
"flags": "NOT_NULL",
"char_set": 63,
"max_size": 11
}
},
{
"ordinal": 24,
"name": "created_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 25,
"name": "updated_at",
"type_info": {
"type": "Timestamp",
"flags": "NOT_NULL | BINARY | TIMESTAMP | ON_UPDATE_NOW",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 26,
"name": "user_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
},
{
"ordinal": 27,
"name": "mod_deleted_at",
"type_info": {
"type": "Timestamp",
"flags": "BINARY",
"char_set": 63,
"max_size": 19
}
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false,
false,
true,
false,
false,
false,
false,
false,
true,
true,
true,
false,
false,
false,
true,
true,
false,
false,
false,
true,
false,
false,
false,
false,
true,
true
]
},
"hash": "f0b5767d52c389660d1859e628c746f6f33a9e1c25f21f66aeeb0784df768c8e"
}
@@ -0,0 +1,12 @@
{
"db_name": "MySQL",
"query": "\nINSERT INTO media_files\nSET\n token = ?,\n\n origin_category = ?,\n origin_product_category = ?,\n\n maybe_origin_model_type = NULL,\n maybe_origin_model_token = NULL,\n\n media_type = ?,\n maybe_mime_type = ?,\n file_size_bytes = ?,\n\n checksum_sha2 = ?,\n\n public_bucket_directory_hash = ?,\n maybe_public_bucket_prefix = ?,\n maybe_public_bucket_extension = ?,\n\n maybe_creator_user_token = ?,\n maybe_creator_anonymous_visitor_token = ?,\n creator_ip_address = ?,\n\n creator_set_visibility = ?,\n\n maybe_creator_file_synthetic_id = ?,\n maybe_creator_category_synthetic_id = ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 16
},
"nullable": []
},
"hash": "fed337e365f0bd36cfa543e0ce9d4076cc0bd7809b6c2cb5740151d6583a6d11"
}
Generated
+5 -3
View File
@@ -1914,6 +1914,7 @@ dependencies = [
"serde_derive",
"serde_json",
"sqlx",
"storyteller_root",
"subprocess",
"subprocess_common",
"tempdir",
@@ -2949,9 +2950,9 @@ dependencies = [
[[package]]
name = "infer"
version = "0.11.0"
version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a6c16b11a665b26aeeb9b1d7f954cdeb034be38dd00adab4f2ae921a8fee804"
checksum = "cb33622da908807a06f9513c19b3c1ad50fab3e4137d82a78107d502075aa199"
dependencies = [
"cfb 0.7.3",
]
@@ -3528,6 +3529,7 @@ dependencies = [
"easyenv",
"enums",
"errors",
"filesys",
"hashing",
"log",
"my-workspace-hack",
@@ -3563,7 +3565,7 @@ dependencies = [
name = "mimetypes"
version = "0.0.1"
dependencies = [
"infer 0.11.0",
"infer 0.15.0",
"my-workspace-hack",
]
@@ -310,7 +310,7 @@ Secrets:
- Put `.env-secrets` in the root `storyteller-rust` directory.
- Per app secrets (ask Brandon for these):
- Put `storteller-web.development-secrets.env` in `crates/service/web/storyteller-web/config`
- Put `storteller-web.development-secrets.env` in `crates/service/web/storyteller_web/config`
- Put `inference-job.development-secrets.env` in `crates/service/job/inference_job/config`
- Put `donwload-job.development-secrets.env` in `crates/service/job/download_job/config`
@@ -392,4 +392,4 @@ We use this: `https://github.com/juhaku/utoipa`
The documentation example is in:
`crates/service/web/storyteller_web/src/http_server/endpoints/voice_designer/inference/enqueue_tts_request.rs`
Write into the api doc to:
`crates/service/web/storyteller_web/src/api_doc.rs`
`crates/service/web/storyteller_web/src/api_doc.rs`
+4 -4
View File
@@ -8,14 +8,14 @@ select
id,
token,
download_type,
title,
download_url,
SUBSTRING(title, 1, 35) as title,
SUBSTRING(download_url, 1, 100) as title,
status,
attempt_count
from generic_download_jobs
where status IN ('started', 'attempt_failed', 'complete_failure', 'dead')
and download_type NOT IN ('vits', 'hifigan')
and created_at > NOW() - INTERVAL 15 DAY;
and created_at > NOW() - INTERVAL 30 DAY;
-- 2) Then restart them
@@ -32,7 +32,7 @@ where token IN (
from generic_download_jobs
where status IN ('started', 'attempt_failed', 'complete_failure', 'dead')
and download_type NOT IN ('vits', 'hifigan')
and created_at > NOW() - INTERVAL 15 DAY
and created_at > NOW() - INTERVAL 30 DAY
) as x
);
@@ -0,0 +1,12 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
RENAME COLUMN public_bucket_hash TO private_bucket_hash;
ALTER TABLE model_weights
RENAME COLUMN maybe_public_bucket_prefix TO maybe_private_bucket_prefix;
ALTER TABLE model_weights
RENAME COLUMN maybe_public_bucket_extension TO maybe_private_bucket_extension;
@@ -0,0 +1,12 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
RENAME COLUMN private_bucket_hash TO public_bucket_hash;
ALTER TABLE model_weights
RENAME COLUMN maybe_private_bucket_prefix TO maybe_public_bucket_prefix;
ALTER TABLE model_weights
RENAME COLUMN maybe_private_bucket_extension TO maybe_public_bucket_extension;
@@ -0,0 +1,6 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
RENAME COLUMN maybe_cover_image_media_file_token TO maybe_cover_media_file_token;
@@ -0,0 +1,6 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
RENAME COLUMN maybe_cover_media_file_token TO maybe_cover_image_media_file_token;
@@ -0,0 +1,13 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
ADD COLUMN maybe_thumbnail_token VARCHAR(32) DEFAULT NULL
AFTER title;
ALTER TABLE model_weights
ADD COLUMN maybe_avatar_media_file_token VARCHAR(32) DEFAULT NULL
AFTER maybe_thumbnail_token;
CREATE INDEX fk_maybe_avatar_media_file_token ON model_weights (maybe_avatar_media_file_token);
@@ -0,0 +1,7 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
-- Drop column
ALTER TABLE model_weights DROP COLUMN maybe_thumbnail_token;
ALTER TABLE model_weights DROP COLUMN maybe_avatar_media_file_token;
@@ -0,0 +1,9 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
CHANGE maybe_description_markdown description_markdown TEXT NOT NULL;
ALTER TABLE model_weights
CHANGE maybe_description_rendered_html description_rendered_html TEXT NOT NULL;
@@ -0,0 +1,9 @@
-- noinspection SqlDialectInspectionForFile
-- noinspection SqlNoDataSourceInspectionForFile
-- noinspection SqlResolveForFile
ALTER TABLE model_weights
CHANGE description_markdown maybe_description_markdown TEXT DEFAULT NULL;
ALTER TABLE model_weights
CHANGE description_rendered_html maybe_description_rendered_html TEXT DEFAULT NULL;
@@ -39,23 +39,19 @@ CREATE TABLE model_weights (
-- The "name" of the model, which might be complicated.
title VARCHAR(255) NOT NULL,
-- TODO(bt, 2023-12-14): Delete the `maybe_thumbnail_token` column later.
-- We're using other columns instead.
maybe_thumbnail_token VARCHAR(32) DEFAULT NULL,
-- The "avatar" image is a media file of type image that serves as a
-- small avatar or profile picture icon.
maybe_avatar_media_file_token VARCHAR(32) DEFAULT NULL,
-- The "cover" image is a media file of type image that covers the
-- top of the page.
maybe_cover_media_file_token VARCHAR(32) DEFAULT NULL,
-- The "cover" image is a media file of type image that severs as a small
-- example of the model. Like album art or a user profile picture / avatar.
maybe_cover_image_media_file_token VARCHAR(32) DEFAULT NULL,
-- The description of the model in markdown.
description_markdown TEXT NOT NULL,
-- This is optional, since not all weights are of the same caliber (eg. zero shot models
-- don't deserve a description).
maybe_description_markdown TEXT DEFAULT NULL,
-- Generated HTML (not user-editable).
description_rendered_html TEXT NOT NULL,
-- This is optional, since not all weights are of the same caliber (eg. zero shot
-- models don't deserve a description).
maybe_description_rendered_html TEXT DEFAULT NULL,
-- ========== CREATOR DETAILS ==========
@@ -113,22 +109,22 @@ CREATE TABLE model_weights (
-- The hash for the bucket directory that contains the original upload
-- as well as any associated other files.
-- The bucket filename for the primary file (not including the directory path) is given by:
-- `[{maybe_private_bucket_prefix}]{private_bucket_hash}[{maybe_private_bucket_extension}]`
private_bucket_hash VARCHAR(32) NOT NULL,
-- `[{maybe_public_bucket_prefix}]{public_bucket_hash}[{maybe_public_bucket_extension}]`
public_bucket_hash VARCHAR(32) NOT NULL,
-- An optional prefix on the bucket filename.
-- If present, this will be prepended to the beginning of the bucket filename to access the file.
-- The bucket filename for the primary file (not including the directory path) is given by:
-- `[{maybe_private_bucket_prefix}]{private_directory_hash}[{maybe_private_bucket_extension}]`
maybe_private_bucket_prefix VARCHAR(16) DEFAULT NULL,
-- `[{maybe_public_bucket_prefix}]{public_bucket_hash}[{maybe_public_bucket_extension}]`
maybe_public_bucket_prefix VARCHAR(16) DEFAULT NULL,
-- An optional appended extension on the bucket filename.
-- If present, this will be appended to the end of the bucket filename to access the file.
-- To allow for flexibility, this extension typically will contain a leading period if
-- the file needs it (eg ".mp4" rather than "mp4")!
-- The bucket filename for the primary file (not including the directory path) is given by:
-- `[{maybe_private_bucket_prefix}]{private_directory_hash}[{maybe_private_bucket_extension}]`
maybe_private_bucket_extension VARCHAR(16) DEFAULT NULL,
-- `[{maybe_public_bucket_prefix}]{public_bucket_hash}[{maybe_public_bucket_extension}]`
maybe_public_bucket_extension VARCHAR(16) DEFAULT NULL,
-- ========== MODERATION DETAILS ==========
@@ -194,7 +190,6 @@ CREATE TABLE model_weights (
KEY index_weights_category (weights_category),
KEY fk_creator_user_token (creator_user_token),
KEY index_creator_set_visibility (creator_set_visibility),
KEY fk_maybe_avatar_media_file_token (maybe_avatar_media_file_token),
KEY fk_maybe_cover_media_file_token (maybe_cover_media_file_token)
KEY fk_maybe_cover_media_file_token (maybe_cover_image_media_file_token)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
@@ -1,49 +0,0 @@
use std::path::PathBuf;
use errors::AnyhowResult;
use errors::bail;
#[deprecated(note = "this is a messy function and should be avoided")]
pub (crate) fn hash_to_bucket_path(file_hash: &str,
optional_root_directory: Option<&str>)
-> AnyhowResult<PathBuf>
{
if file_hash.len() < 4 {
bail!("File length is too short");
}
let root_directory = optional_root_directory
.map(|root| PathBuf::from(root.to_string()))
.unwrap_or(PathBuf::from("/"));
let file_path = format!(
"{}/{}/{}/{}",
&file_hash[0..1],
&file_hash[1..2],
&file_hash[2..3],
&file_hash,
);
Ok(root_directory.join(file_path))
}
#[cfg(test)]
mod tests {
use std::path::PathBuf;
use crate::util::hash_to_bucket_path::hash_to_bucket_path;
#[test]
fn test_bucket_without_root() {
assert_eq!(
hash_to_bucket_path("abcdefg", None).unwrap(),
PathBuf::from("/a/b/c/abcdefg"));
}
#[test]
fn test_bucket_with_root() {
assert_eq!(
hash_to_bucket_path("abcdefg", Some("/root")).unwrap(),
PathBuf::from("/root/a/b/c/abcdefg"));
}
}
@@ -1,35 +0,0 @@
use errors::anyhow;
use errors::AnyhowResult;
use crate::util::hash_to_bucket_path::hash_to_bucket_path;
#[deprecated(note = "this is a messy function and should be avoided")]
pub fn hash_to_bucket_path_string(file_hash: &str,
optional_root_directory: Option<&str>)
-> AnyhowResult<String>
{
let path = hash_to_bucket_path(file_hash, optional_root_directory)?;
let maybe_path = path.to_str().map(|s| s.to_string());
maybe_path.ok_or(anyhow!("Path could not be converted"))
}
#[cfg(test)]
mod tests {
use crate::util::hash_to_bucket_path_string::hash_to_bucket_path_string;
#[test]
fn test_bucket_without_root() {
assert_eq!(
hash_to_bucket_path_string("abcdefg", None).unwrap(),
"/a/b/c/abcdefg".to_string());
}
#[test]
fn test_bucket_with_root() {
assert_eq!(
hash_to_bucket_path_string("abcdefg", Some("/root")).unwrap(),
"/root/a/b/c/abcdefg".to_string());
}
}
@@ -5,6 +5,7 @@
/// With a million items, each directory will have ~244 items.
/// With a billion items, each directory will have ~244,140 items (impractical at this scale).
///
#[deprecated(note = "Use hashed_directory_path_long_string instead! (This function is only used by BucketPathUnifier, which should also die)")]
pub fn hashed_directory_path_short_string(file_hash: &str) -> String {
match file_hash.len() {
0 | 1 => "".to_string(),
-2
View File
@@ -1,4 +1,2 @@
pub (crate) mod hash_to_bucket_path;
pub mod hash_to_bucket_path_string;
pub mod hashed_directory_path_long_string;
pub mod hashed_directory_path_short_string;
@@ -49,6 +49,10 @@ pub enum IdCategory {
#[serde(rename = "model_weights")]
ModelWeights,
/// Files that are uploaded with no general product area they belong to. (Eg. local dev testing)
#[serde(rename = "file_upload")]
FileUpload,
}
// TODO(bt, 2022-12-21): This desperately needs MySQL integration tests!
@@ -68,6 +72,7 @@ impl IdCategory {
Self::ZeroShotTtsResult => "zs_tts_result",
Self::VideoFilterResult => "video_filter",
Self::ModelWeights => "model_weights",
Self::FileUpload => "file_upload",
}
}
@@ -82,6 +87,7 @@ impl IdCategory {
"zs_tts_result" => Ok(Self::ZeroShotTtsResult),
"video_filter" => Ok(Self::VideoFilterResult),
"model_weights" => Ok(Self::ModelWeights),
"file_upload" => Ok(Self::FileUpload),
_ => Err(format!("invalid value: {:?}", value)),
}
}
@@ -99,6 +105,7 @@ impl IdCategory {
Self::ZeroShotVoiceDataset,
Self::ZeroShotVoiceEmbedding,
Self::ModelWeights,
Self::FileUpload,
])
}
}
@@ -121,7 +128,8 @@ mod tests {
assert_serialization(IdCategory::ZeroShotVoiceDataset, "zs_dataset");
assert_serialization(IdCategory::ZeroShotVoiceEmbedding, "zs_voice");
assert_serialization(IdCategory::ZeroShotTtsResult, "zs_tts_result");
assert_serialization(IdCategory::ModelWeights, "model_weights")
assert_serialization(IdCategory::ModelWeights, "model_weights");
assert_serialization(IdCategory::FileUpload, "file_upload");
}
#[test]
@@ -135,6 +143,7 @@ mod tests {
assert_eq!(IdCategory::ZeroShotVoiceEmbedding.to_str(), "zs_voice");
assert_eq!(IdCategory::ZeroShotTtsResult.to_str(), "zs_tts_result");
assert_eq!(IdCategory::ModelWeights.to_str(), "model_weights");
assert_eq!(IdCategory::FileUpload.to_str(), "file_upload");
}
#[test]
@@ -148,13 +157,14 @@ mod tests {
assert_eq!(IdCategory::from_str("zs_voice").unwrap(), IdCategory::ZeroShotVoiceEmbedding);
assert_eq!(IdCategory::from_str("zs_tts_result").unwrap(), IdCategory::ZeroShotTtsResult);
assert_eq!(IdCategory::from_str("model_weights").unwrap(), IdCategory::ModelWeights);
assert_eq!(IdCategory::from_str("file_upload").unwrap(), IdCategory::FileUpload);
}
#[test]
fn all_variants() {
// Static check
let mut variants = IdCategory::all_variants();
assert_eq!(variants.len(), 9);
assert_eq!(variants.len(), 10);
assert_eq!(variants.pop_first(), Some(IdCategory::MediaFile));
assert_eq!(variants.pop_first(), Some(IdCategory::LipsyncAnimationResult));
assert_eq!(variants.pop_first(), Some(IdCategory::VideoFilterResult));
@@ -164,6 +174,7 @@ mod tests {
assert_eq!(variants.pop_first(), Some(IdCategory::ZeroShotVoiceDataset));
assert_eq!(variants.pop_first(), Some(IdCategory::ZeroShotVoiceEmbedding));
assert_eq!(variants.pop_first(), Some(IdCategory::ModelWeights));
assert_eq!(variants.pop_first(), Some(IdCategory::FileUpload));
assert_eq!(variants.pop_first(), None);
// Generated check
@@ -6,12 +6,13 @@ use serde::Serialize;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `media_files` table in a `VARCHAR` field.
///
/// DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
#[serde(rename_all = "snake_case")]
pub enum MediaFileOriginCategory {
/// ML model inference output - uploaded models or zero shot.
@@ -6,12 +6,13 @@ use serde::Serialize;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `media_files` table in a `VARCHAR` field.
///
/// DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
#[serde(rename_all = "snake_case")]
pub enum MediaFileOriginModelType {
/// RVC (v2) voice conversion models
@@ -4,6 +4,7 @@ use std::collections::BTreeSet;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `media_files` table in `VARCHAR(16)` field `origin_product_category`.
///
@@ -14,7 +15,7 @@ use strum::EnumIter;
///
/// DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
pub enum MediaFileOriginProductCategory {
/// Media files created by (or uploaded for) the Face Animator product.
/// The underlying model could be SadTalker, Wav2Lip, or some future model
@@ -4,12 +4,13 @@ use std::collections::BTreeSet;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `media_files` table in a `VARCHAR` field.
///
/// DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Ord, PartialOrd, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Ord, PartialOrd, Deserialize, Serialize, ToSchema)]
#[serde(rename_all = "lowercase")]
pub enum MediaFileType {
/// Audio files: wav, mp3, etc.
@@ -4,10 +4,11 @@ use serde::Serialize;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `user_bookmarks` table in a `VARCHAR(32)` field named `entity_type`.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Deserialize, Serialize, ToSchema)]
pub enum UserBookmarkEntityType {
/// User
#[serde(rename = "user")]
@@ -49,6 +50,7 @@ pub enum UserBookmarkEntityType {
// TODO(bt, 2023-01-17): This desperately needs MySQL integration tests!
impl_enum_display_and_debug_using_to_str!(UserBookmarkEntityType);
impl_mysql_enum_coders!(UserBookmarkEntityType);
impl_mysql_from_row!(UserBookmarkEntityType);
/// NB: Legacy API for older code.
impl UserBookmarkEntityType {
@@ -4,10 +4,11 @@ use serde::Serialize;
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// Used in the `user_ratings` table in a `VARCHAR(32)` field named `entity_type`.
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Deserialize, Serialize)]
#[derive(Clone, Copy, Eq, PartialEq, Hash, Deserialize, Serialize, ToSchema)]
pub enum UserRatingEntityType {
/// Media files (inference results, uploads, etc.)
#[serde(rename = "media_file")]
@@ -2,6 +2,7 @@
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
/// UserRatingValue
///
@@ -17,7 +18,7 @@ use strum::EnumIter;
/// *DO NOT CHANGE VALUES WITHOUT A MIGRATION STRATEGY!*
///
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Deserialize, Serialize, sqlx::Type)]
#[derive(Clone, Copy, Eq, PartialEq, Deserialize, Serialize, sqlx::Type, ToSchema)]
#[serde(rename_all = "lowercase")]
#[sqlx(rename_all = "lowercase")]
pub enum UserRatingValue {
+1
View File
@@ -3,3 +3,4 @@ pub mod job_status_plus;
pub mod sqlite;
pub mod visibility;
pub mod vocoder_type;
pub mod view_as;
+153
View File
@@ -0,0 +1,153 @@
#[cfg(test)]
use strum::EnumCount;
#[cfg(test)]
use strum::EnumIter;
use utoipa::ToSchema;
#[cfg_attr(test, derive(EnumIter, EnumCount))]
#[derive(Clone, Copy, Eq, PartialEq, Deserialize, Serialize, sqlx::Type,ToSchema)]
#[serde(rename_all = "snake_case")]
#[sqlx(rename_all = "snake_case")]
pub enum ViewAs {
/// Public entities are able to be listed in public lists.
/// It does not mean that they necessarily will be (eg. they could be "mod unapproved" or deleted).
Author,
/// Hidden entities are not shown in public lists, but the URL to them may be given out freely.
/// They are available to non-logged-in users as long as they have the URL.
Moderator,
/// Private entities should only be available to the creator, a list of approved users, and
/// website moderation staff.
AnotherUser,
}
impl_enum_display_and_debug_using_to_str!(ViewAs);
impl_mysql_from_row!(ViewAs);
impl Default for ViewAs {
fn default() -> Self { Self::AnotherUser }
}
/// NB: Legacy API for older code.
impl ViewAs {
pub fn to_str(&self) -> &'static str {
match self {
Self::Author => "author",
Self::Moderator => "moderator",
Self::AnotherUser => "another_user",
}
}
pub fn from_str(value: &str) -> Result<Self, String> {
match value {
"author" => Ok(Self::Author),
"moderator" => Ok(Self::Moderator),
"another_user" => Ok(Self::AnotherUser),
_ => Err(format!("invalid value: {:?}", value)),
}
}
}
#[cfg(test)]
mod tests {
use crate::common::view_as::ViewAs;
use crate::test_helpers::assert_serialization;
#[test]
fn test_default() {
assert_eq!(ViewAs::default(), ViewAs::AnotherUser);
}
#[test]
fn test_serialization() {
assert_serialization(ViewAs::Author, "author");
assert_serialization(ViewAs::Moderator, "moderator");
assert_serialization(ViewAs::AnotherUser, "another_user");
}
#[test]
fn test_to_str() {
assert_eq!(ViewAs::Author.to_str(), "author");
assert_eq!(ViewAs::Moderator.to_str(), "moderator");
assert_eq!(ViewAs::AnotherUser.to_str(), "another_user");
}
#[test]
fn test_from_str() {
assert_eq!(ViewAs::from_str("author").unwrap(), ViewAs::Author);
assert_eq!(ViewAs::from_str("moderator").unwrap(), ViewAs::Moderator);
assert_eq!(ViewAs::from_str("another_user").unwrap(), ViewAs::AnotherUser);
assert!(ViewAs::from_str("foo").is_err());
}
mod traits {
use crate::common::view_as::ViewAs;
#[test]
fn display() {
let view_as = ViewAs::Moderator;
assert_eq!(format!("{}", view_as), "moderator".to_string());
}
#[test]
fn debug() {
let view_as = ViewAs::Moderator;
assert_eq!(format!("{:?}", view_as), "moderator".to_string());
}
}
#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct CompositeType {
view_as: ViewAs,
string: String,
}
mod serde_serialization {
use crate::common::view_as::ViewAs;
use super::CompositeType;
#[test]
fn serialize() {
let expected = "\"author\"".to_string(); // NB: Quoted
assert_eq!(expected, toml::to_string(&ViewAs::Author).unwrap());
// Just to show this serializes the same as a string
assert_eq!(expected, toml::to_string("author").unwrap());
}
#[test]
fn nested_serialize() {
let value = CompositeType { view_as: ViewAs::Moderator, string: "bar".to_string() };
let expected = r#"{"view_as":"moderator","string":"bar"}"#.to_string();
assert_eq!(expected, serde_json::to_string(&value).unwrap());
}
}
mod serde_deserialization {
use crate::common::view_as::ViewAs;
use super::CompositeType;
#[test]
fn deserialize() {
let payload = "\"another_user\""; // NB: Quoted
let value: ViewAs = serde_json::from_str(payload).unwrap();
assert_eq!(value, ViewAs::AnotherUser);
}
#[test]
fn nested_deserialize() {
let payload = r#"{"view_as":"another_user","string":"bar"}"#.to_string();
let expected = CompositeType {
view_as: ViewAs::AnotherUser,
string: "bar".to_string(),
};
assert_eq!(expected, serde_json::from_str::<CompositeType>(&payload).unwrap());
}
}
}
@@ -2,11 +2,12 @@ use std::fmt::Debug;
use serde::Deserialize;
use serde::Serialize;
use utoipa::ToSchema;
use crate::prefixes::TokenPrefix;
/// The primary key for Audit Logs.
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize, ToSchema)]
#[sqlx(transparent)]
pub struct MediaFileToken(pub String);
@@ -2,11 +2,12 @@ use std::fmt::Debug;
use serde::Deserialize;
use serde::Serialize;
use utoipa::ToSchema;
use crate::prefixes::TokenPrefix;
/// The primary key for Audit Logs.
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize)]
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash, sqlx::Type, Debug, Serialize, Deserialize, ToSchema)]
#[sqlx(transparent)]
pub struct UserBookmarkToken(pub String);
@@ -9,7 +9,7 @@ use enums::by_table::model_weights::{
};
use enums::common::visibility::Visibility;
use errors::{anyhow, AnyhowResult};
use mysql_queries::queries::model_weights::create_weight::{create_weight, CreateModelWeightsArgs};
use mysql_queries::queries::model_weights::create::create_weight::{create_weight, CreateModelWeightsArgs};
use mysql_queries::queries::users::user::get_user_token_by_username::get_user_token_by_username;
use tokens::tokens::model_weights::ModelWeightToken;
use tokens::tokens::users::UserToken;
@@ -111,8 +111,7 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
let mut description_rendered_html;
let mut original_filename;
let mut original_download_url;
let mut thumbnail_token;
let mut private_bucket_hash:String = "".to_string();
let mut private_bucket_prefix;
let mut private_bucket_extension;
@@ -140,9 +139,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("gura_gwar{}.safetensors", i);
original_download_url = format!("www.google.ca");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = "";
// private_bucket_hash = format!("bucket_hash{}", i);
private_bucket_prefix = format!("_fake");
private_bucket_extension = format!("rvcV2");
@@ -171,9 +167,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("filename{}.txt", i);
original_download_url = format!("https://civitai.com/api/download/models/149193?type=Model&format=SafeTensor&size=pruned&fp=fp16");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = "";
// private_bucket_hash = format!("bucket_hash{}", i);
private_bucket_prefix = format!("_fake");
private_bucket_extension = format!("tt2");
@@ -197,9 +190,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("filename{}.txt", i);
original_download_url = format!("https://civitai.com/api/download/models/149193?type=Model&format=SafeTensor&size=pruned&fp=fp16");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = ""; //sd1_5_image_token;
// private_bucket_hash = format!("bucket_hash{}", i);
private_bucket_prefix = format!("_fake");
private_bucket_extension = format!("sd15");
@@ -245,9 +235,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("filename{}.txt", i);
original_download_url = format!("https://civitai.com/api/download/models/149193?type=Model&format=SafeTensor&size=pruned&fp=fp16");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = "";
cached_user_ratings_total_count = i;
cached_user_ratings_positive_count = i;
cached_user_ratings_negative_count = i;
@@ -273,9 +260,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("filename{}.txt", i);
original_download_url = format!("https://civitai.com/api/download/models/149193?type=Model&format=SafeTensor&size=pruned&fp=fp16");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = ""; // lora_image_token;
// private_bucket_hash = format!("bucket_hash{}", i);
private_bucket_prefix = format!("_fake");
private_bucket_extension = format!("loRA");
@@ -303,9 +287,6 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename = format!("gura_gwar{}.safetensors", i);
original_download_url = format!("www.google.ca");
// NOTE TO SELF this will be a join for later on for now they can use this.
thumbnail_token = "";
// private_bucket_hash = format!("bucket_hash{}", i);
private_bucket_prefix = format!("_fake");
private_bucket_extension = format!("rvcV2");
@@ -324,9 +305,8 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
weights_type: weights_types, // replace with actual WeightsType
weights_category, // replace with actual WeightsCategory
title,
maybe_thumbnail_token: Some(thumbnail_token.to_string()),
description_markdown: description,
description_rendered_html: description_rendered_html.to_string(),
maybe_description_markdown: Some(description),
maybe_description_rendered_html: Some(description_rendered_html.to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -335,9 +315,9 @@ pub async fn seed_weights_for_paging(mysql_pool: &Pool<MySql>, user_token: UserT
original_filename: Some(original_filename),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some(private_bucket_prefix),
maybe_private_bucket_extension: Some(private_bucket_extension),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some(private_bucket_prefix),
maybe_public_bucket_extension: Some(private_bucket_extension),
cached_user_ratings_total_count,
cached_user_ratings_positive_count,
cached_user_ratings_negative_count,
@@ -370,9 +350,8 @@ pub async fn seed_weights_for_user_token(
weights_type: WeightsType::RvcV2, // replace with actual WeightsType
weights_category: WeightsCategory::VoiceConversion, // replace with actual WeightsCategory
title: "Title 1".to_string(),
maybe_thumbnail_token: Some("Thumbnail 1".to_string()),
description_markdown: "Description 1".to_string(),
description_rendered_html: "<p>Description 1</p>".to_string(),
maybe_description_markdown: Some("Description 1".to_string()),
maybe_description_rendered_html: Some("<p>Description 1</p>".to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -381,9 +360,9 @@ pub async fn seed_weights_for_user_token(
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -396,9 +375,8 @@ pub async fn seed_weights_for_user_token(
weights_type: WeightsType::HifiganTacotron2, // replace with actual WeightsType
weights_category: WeightsCategory::TextToSpeech, // replace with actual WeightsCategory
title: "Title 2".to_string(),
maybe_thumbnail_token: Some("Thumbnail 2".to_string()),
description_markdown: "Description 2".to_string(),
description_rendered_html: "<p>Description 2</p>".to_string(),
maybe_description_markdown: Some("Description 2".to_string()),
maybe_description_rendered_html: Some("<p>Description 2</p>".to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "292.268.2.2",
creator_set_visibility: Visibility::Public,
@@ -407,9 +385,9 @@ pub async fn seed_weights_for_user_token(
original_filename: Some("filename2.txt".to_string()),
file_size_bytes: 2024,
file_checksum_sha2: "checksum2".to_string(),
private_bucket_hash: "bucket_hash2".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("tt2".to_string()),
public_bucket_hash: "bucket_hash2".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("tt2".to_string()),
cached_user_ratings_total_count: 20,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 2,
@@ -422,9 +400,8 @@ pub async fn seed_weights_for_user_token(
weights_type: WeightsType::StableDiffusion15, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 3".to_string(),
maybe_thumbnail_token: Some("Thumbnail 3".to_string()),
description_markdown: "Description 3".to_string(),
description_rendered_html: "<p>Description 3</p>".to_string(),
maybe_description_markdown: Some("Description 3".to_string()),
maybe_description_rendered_html: Some("<p>Description 3</p>".to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "392.368.3.3",
creator_set_visibility: Visibility::Public,
@@ -433,9 +410,9 @@ pub async fn seed_weights_for_user_token(
original_filename: Some("filename3.txt".to_string()),
file_size_bytes: 3024,
file_checksum_sha2: "checksum3".to_string(),
private_bucket_hash: "bucket_hash3".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("safetensors".to_string()),
public_bucket_hash: "bucket_hash3".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("safetensors".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -448,9 +425,8 @@ pub async fn seed_weights_for_user_token(
weights_type: WeightsType::LoRA, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 4".to_string(),
maybe_thumbnail_token: Some("Thumbnail 4".to_string()),
description_markdown: "Description 4".to_string(),
description_rendered_html: "<p>Description 4</p>".to_string(),
maybe_description_markdown: Some("Description 4".to_string()),
maybe_description_rendered_html: Some("<p>Description 4</p>".to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -459,9 +435,9 @@ pub async fn seed_weights_for_user_token(
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum4".to_string(),
private_bucket_hash: "bucket_hash4".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("LoRA".to_string()),
public_bucket_hash: "bucket_hash4".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("LoRA".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -474,9 +450,8 @@ pub async fn seed_weights_for_user_token(
weights_type: WeightsType::LoRA, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 5".to_string(),
maybe_thumbnail_token: Some("Thumbnail 4".to_string()),
description_markdown: "Description 4".to_string(),
description_rendered_html: "<p>Description 4</p>".to_string(),
maybe_description_markdown: Some("Description 4".to_string()),
maybe_description_rendered_html: Some("<p>Description 4</p>".to_string()),
creator_user_token: Some(&user_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -485,9 +460,9 @@ pub async fn seed_weights_for_user_token(
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum4".to_string(),
private_bucket_hash: "bucket_hash4".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("LoRA".to_string()),
public_bucket_hash: "bucket_hash4".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("LoRA".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -526,9 +501,8 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
weights_type: WeightsType::RvcV2, // replace with actual WeightsType
weights_category: WeightsCategory::VoiceConversion, // replace with actual WeightsCategory
title: "Title 1".to_string(),
maybe_thumbnail_token: Some("Thumbnail 1".to_string()),
description_markdown: "Description 1".to_string(),
description_rendered_html: "<p>Description 1</p>".to_string(),
maybe_description_markdown: Some("Description 1".to_string()),
maybe_description_rendered_html: Some("<p>Description 1</p>".to_string()),
creator_user_token: Some(&creator_token1), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -537,9 +511,9 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -552,9 +526,8 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
weights_type: WeightsType::HifiganTacotron2, // replace with actual WeightsType
weights_category: WeightsCategory::TextToSpeech, // replace with actual WeightsCategory
title: "Title 2".to_string(),
maybe_thumbnail_token: Some("Thumbnail 2".to_string()),
description_markdown: "Description 2".to_string(),
description_rendered_html: "<p>Description 2</p>".to_string(),
maybe_description_markdown: Some("Description 2".to_string()),
maybe_description_rendered_html: Some("<p>Description 2</p>".to_string()),
creator_user_token: Some(&creator_token2), // replace with actual UserToken
creator_ip_address: "292.268.2.2",
creator_set_visibility: Visibility::Public,
@@ -563,9 +536,9 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
original_filename: Some("filename2.txt".to_string()),
file_size_bytes: 2024,
file_checksum_sha2: "checksum2".to_string(),
private_bucket_hash: "bucket_hash2".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("tt2".to_string()),
public_bucket_hash: "bucket_hash2".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("tt2".to_string()),
cached_user_ratings_total_count: 20,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 2,
@@ -578,9 +551,8 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
weights_type: WeightsType::StableDiffusion15, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 3".to_string(),
maybe_thumbnail_token: Some("Thumbnail 3".to_string()),
description_markdown: "Description 3".to_string(),
description_rendered_html: "<p>Description 3</p>".to_string(),
maybe_description_markdown: Some("Description 3".to_string()),
maybe_description_rendered_html: Some("<p>Description 3</p>".to_string()),
creator_user_token: Some(&creator_token3), // replace with actual UserToken
creator_ip_address: "392.368.3.3",
creator_set_visibility: Visibility::Public,
@@ -589,9 +561,9 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
original_filename: Some("filename3.txt".to_string()),
file_size_bytes: 3024,
file_checksum_sha2: "checksum3".to_string(),
private_bucket_hash: "bucket_hash3".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("safetensors".to_string()),
public_bucket_hash: "bucket_hash3".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("safetensors".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -604,9 +576,8 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
weights_type: WeightsType::LoRA, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 4".to_string(),
maybe_thumbnail_token: Some("Thumbnail 4".to_string()),
description_markdown: "Description 4".to_string(),
description_rendered_html: "<p>Description 4</p>".to_string(),
maybe_description_markdown: Some("Description 4".to_string()),
maybe_description_rendered_html: Some("<p>Description 4</p>".to_string()),
creator_user_token: Some(&creator_token4), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -615,9 +586,9 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum4".to_string(),
private_bucket_hash: "bucket_hash4".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("LoRA".to_string()),
public_bucket_hash: "bucket_hash4".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("LoRA".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -630,9 +601,8 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
weights_type: WeightsType::LoRA, // replace with actual WeightsType
weights_category: WeightsCategory::ImageGeneration, // replace with actual WeightsCategory
title: "Title 5".to_string(),
maybe_thumbnail_token: Some("Thumbnail 5".to_string()),
description_markdown: "Description 5".to_string(),
description_rendered_html: "<p>Description 5</p>".to_string(),
maybe_description_markdown: Some("Description 5".to_string()),
maybe_description_rendered_html: Some("<p>Description 5</p>".to_string()),
creator_user_token: Some(&creator_token5), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -641,9 +611,9 @@ pub async fn original_seed_weights(mysql_pool: &Pool<MySql>,user_token: UserToke
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1025,
file_checksum_sha2: "checksum5".to_string(),
private_bucket_hash: "bucket_hash5".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("LoRA".to_string()),
public_bucket_hash: "bucket_hash5".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("LoRA".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -15,8 +15,8 @@ use filesys::path_to_string::path_to_string;
use hashing::sha256::sha256_hash_file::sha256_hash_file;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mimetypes::mimetype_for_file::get_mimetype_for_file;
use mysql_queries::queries::media_files::create::insert_media_file_from_cli_tool::{insert_media_file_from_cli_tool, InsertArgs};
use mysql_queries::queries::media_files::get_media_file::get_media_file;
use mysql_queries::queries::media_files::insert_media_file_from_cli_tool::{insert_media_file_from_cli_tool, InsertArgs};
use storyteller_root::get_seed_tool_data_root;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
@@ -15,7 +15,7 @@ use filesys::file_size::file_size;
use filesys::path_to_string::path_to_string;
use hashing::sha256::sha256_hash_file::sha256_hash_file;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mysql_queries::queries::model_weights::create_weight::{
use mysql_queries::queries::model_weights::create::create_weight::{
create_weight,
CreateModelWeightsArgs,
};
@@ -92,7 +92,6 @@ pub async fn seed_weights_files(
let description_rendered_html = "This is a description rendered html";
let original_filename = "model.safetensors";
let original_download_url = "https://example.com/model.safetensors";
let thumbnail_token = format!("{}", n);
// let public_bucket_hash: String = weights_file_token.to_string();
// let public_bucket_prefix = weights_type.to_string();
@@ -130,9 +129,8 @@ pub async fn seed_weights_files(
weights_type, // Assuming weights_type is defined elsewhere
weights_category, // Assuming weights_category is defined elsewhere
title: title.to_string(),
maybe_thumbnail_token: Some(thumbnail_token),
description_markdown: description.to_string(),
description_rendered_html: description_rendered_html.to_string(),
maybe_description_markdown: Some(description.to_string()),
maybe_description_rendered_html: Some(description_rendered_html.to_string()),
creator_user_token: Some(&user_token),
creator_ip_address: "127.0.0.1", // Assuming the IP address is localhost
creator_set_visibility: Visibility::Public, // Assuming the visibility is public
@@ -141,9 +139,9 @@ pub async fn seed_weights_files(
original_filename: Some(original_filename.to_string()),
file_size_bytes: file_size_bytes as i32, // Assuming the file size is 0
file_checksum_sha2: sha256_checksum.to_string(), // Assuming the file checksum is an empty string
private_bucket_hash: bucket_hash,
maybe_private_bucket_prefix: Some(maybe_bucket_prefix.to_string()),
maybe_private_bucket_extension: Some(maybe_bucket_extension.to_string()),
public_bucket_hash: bucket_hash,
maybe_public_bucket_prefix: Some(maybe_bucket_prefix.to_string()),
maybe_public_bucket_extension: Some(maybe_bucket_extension.to_string()),
cached_user_ratings_total_count,
cached_user_ratings_positive_count,
cached_user_ratings_negative_count,
+1 -1
View File
@@ -11,7 +11,7 @@ use enums::common::visibility::Visibility;
use errors::{AnyhowResult, bail};
use filesys::file_read_bytes::file_read_bytes;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use mysql_queries::queries::media_files::insert_media_file_from_cli_tool::{insert_media_file_from_cli_tool, InsertArgs};
use mysql_queries::queries::media_files::create::insert_media_file_from_cli_tool::{insert_media_file_from_cli_tool, InsertArgs};
use crate::cli_args::parse_cli_args;
+1
View File
@@ -20,6 +20,7 @@ config = { path = "../../lib/deprecated/config" }
easyenv = { path = "../../lib/easyenv" }
enums = { path = "../../_types/enums" }
errors = { path = "../../lib/errors" }
filesys = { path = "../../lib/files/filesys" }
hashing = { path = "../../lib/files/hashing" }
mysql_queries = { path = "../../lib/mysql_queries" }
storyteller_root = { path = "../../lib/storyteller_root" }
@@ -4,6 +4,8 @@ use buckets::public::weight_files::bucket_file_path::WeightFileBucketPath;
use cloud_storage::bucket_path_unifier::BucketPathUnifier;
use enums::by_table::voice_conversion_models::voice_conversion_model_type::VoiceConversionModelType;
use errors::{anyhow, AnyhowResult};
use filesys::safe_delete_temp_directory::safe_delete_temp_directory;
use filesys::safe_delete_temp_file::safe_delete_temp_file;
use hashing::sha256::sha256_hash_file::sha256_hash_file;
use mysql_queries::queries::model_weights::migration::upsert_model_weight_from_voice_conversion_model::CopiedFileData;
use mysql_queries::queries::voice_conversion::migration::list_whole_voice_conversion_models_using_cursor::WholeVoiceConversionModelRecord;
@@ -43,11 +45,14 @@ async fn copy_model(model: &WholeVoiceConversionModelRecord, deps: &Deps) -> Any
VoiceConversionModelType::SoftVc => return Err(anyhow!("we never built softvc models")),
};
deps.bucket_development_public.upload_filename_with_content_type(
deps.bucket_production_public.upload_filename_with_content_type(
&new_model_bucket_path.get_full_object_path_str(),
&model_temp_fs_path,
"application/octet-stream").await?;
safe_delete_temp_file(&model_temp_fs_path);
safe_delete_temp_directory(&temp_dir);
Ok(CopiedFileData {
bucket_path: new_model_bucket_path,
file_sha_hash: file_checksum,
@@ -68,10 +73,13 @@ async fn copy_index_file(model: &WholeVoiceConversionModelRecord, deps: &Deps, b
let new_model_bucket_path =
WeightFileBucketPath::rvc_index_file_from_object_hash(bucket_path.get_object_hash());
deps.bucket_development_public.upload_filename_with_content_type(
deps.bucket_production_public.upload_filename_with_content_type(
&new_model_bucket_path.get_full_object_path_str(),
&model_temp_fs_path,
"application/octet-stream").await?;
safe_delete_temp_file(&model_temp_fs_path);
safe_delete_temp_directory(&temp_dir);
Ok(()) // NB: We don't care about the path of the index file.
}
@@ -1,9 +1,15 @@
use std::thread;
use std::time::Duration;
use log::{info, warn};
use buckets::public::weight_files::bucket_file_path::WeightFileBucketPath;
use enums::by_table::model_weights::weights_types::WeightsType;
use errors::AnyhowResult;
use mysql_queries::queries::model_weights::migration::upsert_model_weight_from_voice_conversion_model::upsert_model_weight_from_voice_conversion_model;
use mysql_queries::queries::voice_conversion::migration::list_whole_voice_conversion_models_using_cursor::list_whole_voice_conversion_models_using_cursor;
use mysql_queries::queries::model_weights::get_weight::get_weight_by_token;
use mysql_queries::queries::model_weights::migration::upsert_model_weight_from_voice_conversion_model::{CopiedFileData, upsert_model_weight_from_voice_conversion_model};
use mysql_queries::queries::voice_conversion::migration::list_whole_voice_conversion_models_using_cursor::{list_whole_voice_conversion_models_using_cursor, WholeVoiceConversionModelRecord};
use tokens::tokens::model_weights::ModelWeightToken;
use crate::deps::Deps;
use crate::migrations::voice_conversion_to_weights::copy_cloud_files::copy_cloud_files;
@@ -15,17 +21,19 @@ pub async fn migrate_voice_conversion_to_weights(deps: &Deps) -> AnyhowResult<()
let mut cursor = 0;
loop {
info!("Querying {PAGE_SIZE} models at cursor = {cursor}");
let results
= list_whole_voice_conversion_models_using_cursor(&deps.mysql_production, PAGE_SIZE, cursor).await?;
if results.is_empty() {
warn!("No more results found; exiting.");
break;
}
for result in results.iter() {
println!("result: {:?}\n\n", result);
let copied_data = copy_cloud_files(result, &deps).await?;
upsert_model_weight_from_voice_conversion_model(result, &deps.mysql_development, &copied_data).await?;
println!("\n\nmigrating over result {:?} : {:?} ...", result.token, result);
copy_model_record_and_bucket_files(result, &deps).await?;
}
if let Some(last_id) = results.last().map(|result| result.id) {
@@ -37,3 +45,48 @@ pub async fn migrate_voice_conversion_to_weights(deps: &Deps) -> AnyhowResult<()
Ok(())
}
async fn copy_model_record_and_bucket_files(record: &WholeVoiceConversionModelRecord, deps: &Deps) -> AnyhowResult<()> {
let mut maybe_copied_data = None;
match &record.maybe_migration_new_model_weights_token {
None => {
info!("Copying bucket files for record {:?} ...", record.token);
let result = copy_cloud_files(record, &deps).await?;
maybe_copied_data = Some(result);
}
Some(token) => {
info!("Existing migrated record; skipping cloud bucket copy...");
maybe_copied_data = lookup_bucket_details_from_record(token, &deps).await?;
}
}
if let Some(copied_data) = &maybe_copied_data {
info!("Upserting records...");
upsert_model_weight_from_voice_conversion_model(record, &deps.mysql_production, copied_data).await?;
}
Ok(())
}
/// Look up the bucket storage details from a previously-migrated record
async fn lookup_bucket_details_from_record(model_weight_token: &ModelWeightToken, deps: &Deps) -> AnyhowResult<Option<CopiedFileData>> {
let maybe_model_weight_record =
get_weight_by_token(model_weight_token, true, &deps.mysql_production).await?;
let model_weight_record = match maybe_model_weight_record {
None => return Ok(None),
Some(record) => record,
};
let bucket_path = match model_weight_record.weights_type {
WeightsType::RvcV2 => WeightFileBucketPath::rvc_model_file_from_object_hash(&model_weight_record.public_bucket_hash),
WeightsType::SoVitsSvc => WeightFileBucketPath::svc_model_file_from_object_hash(&model_weight_record.public_bucket_hash),
_ => return Ok(None), // NB: Stable Diffusion, etc. aren't valid for migration
};
Ok(Some(CopiedFileData {
bucket_path,
file_sha_hash: model_weight_record.file_checksum_sha2,
}))
}
@@ -114,7 +114,7 @@ impl BucketClient {
if code != 200 {
let body = String::from_utf8_lossy(body_bytes);
warn!("upload body: {}", body);
return Err(anyhow!("upload failed: {}", code));
Err(anyhow!("upload failed: {}", code))
} else {
info!("upload success: {}", code);
Ok(())
@@ -5,7 +5,6 @@ use async_trait::async_trait;
use errors::AnyhowResult;
use crate::bucket_client::BucketClient;
use crate::remote_file_manager::file_meta_data::FileMetaData;
#[async_trait]
pub trait BucketOrchestrationCore {
@@ -85,11 +84,11 @@ impl BucketOrchestration {
public_bucket_name: String,
private_bucket_name: String) -> Self {
Self {
access_key: access_key,
secret_key: secret_key,
region_name: region_name,
public_bucket_name: public_bucket_name,
private_bucket_name: private_bucket_name,
access_key,
secret_key,
region_name,
public_bucket_name,
private_bucket_name,
}
}
@@ -7,18 +7,18 @@ const REMOTE_FILE_DIRECTORY: &str = "/implement_google_cloud_storage_bucket_name
pub trait FileDescriptor {
// By default a file belongs in the public bucket will help us figureout which bucket to use.
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file
// e.g requires! a period .safetensors .bin .jpg
fn get_suffix(&self)->String {
return "implement".to_string();
"implement".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, loRA, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "implement".to_string();
"implement".to_string()
}
// This will be ensure that the right bucket is picked
@@ -32,10 +32,10 @@ pub struct FileBucketDirectory {
let full_remote_cloud_file_path = format!("{}/{}", remote_cloud_base_directory , file_name);
Self {
file_object_hash: entropy.to_string(),
remote_cloud_base_directory: remote_cloud_base_directory,
full_remote_cloud_file_path: full_remote_cloud_file_path,
file_name:file_name,
file_descriptor:file_descriptor
remote_cloud_base_directory,
full_remote_cloud_file_path,
file_name,
file_descriptor
}
}
@@ -53,10 +53,10 @@ pub struct FileBucketDirectory {
let full_remote_cloud_file_path = format!("{}/{}", remote_cloud_base_directory , file_name);
Self {
file_object_hash: entropy,
remote_cloud_base_directory: remote_cloud_base_directory,
full_remote_cloud_file_path: full_remote_cloud_file_path,
file_name:file_name,
file_descriptor:file_descriptor
remote_cloud_base_directory,
full_remote_cloud_file_path,
file_name,
file_descriptor
}
}
@@ -1,7 +1,6 @@
use crate::remote_file_manager::weights_descriptor::{self};
use super::file_descriptor::FileDescriptor;
use crate::remote_file_manager::weights_descriptor::{WeightsLoRADescriptor, WeightsSD15Descriptor, WeightsSDXLDescriptor, self};
pub struct RemoteCloudBucketDetails {
pub object_hash: String,
@@ -12,9 +11,9 @@ pub struct RemoteCloudBucketDetails {
impl RemoteCloudBucketDetails {
pub fn new(object_hash: String, prefix: String, suffix: String) -> Self {
Self {
object_hash: object_hash,
prefix: prefix,
suffix: suffix
object_hash,
prefix,
suffix
}
}
pub fn get_object_hash(&self) -> &str {
@@ -49,7 +48,6 @@ impl RemoteCloudBucketDetails {
#[cfg(test)]
mod tests {
use super::*;
#[test]
pub fn test() {
@@ -1,21 +1,15 @@
use crate::remote_file_manager::file_directory::FileBucketDirectory;
use crate::remote_file_manager::remote_cloud_bucket_details::RemoteCloudBucketDetails;
use super::bucket_orchestration;
use super::file_descriptor::{FileDescriptor};
use super::file_meta_data::FileMetaData;
use errors:: AnyhowResult;
use errors::AnyhowResult;
use filesys::file_read_bytes::file_read_bytes;
use filesys::file_size::file_size;
use hashing::sha256::sha256_hash_file::sha256_hash_file;
use mimetypes::mimetype_for_bytes::get_mimetype_for_bytes;
use crate::remote_file_manager::bucket_orchestration::{BucketOrchestrationCore};
use crate::remote_file_manager::bucket_orchestration::BucketOrchestrationCore;
use crate::remote_file_manager::file_directory::FileBucketDirectory;
use crate::remote_file_manager::remote_cloud_bucket_details::RemoteCloudBucketDetails;
use super::file_descriptor::FileDescriptor;
use super::file_meta_data::FileMetaData;
struct RemoteCloudFileClient {
bucket_orchestration_client: Box<dyn BucketOrchestrationCore>
@@ -24,7 +18,7 @@ struct RemoteCloudFileClient {
impl RemoteCloudFileClient {
pub fn new(bucket_orchestration_client: Box<dyn BucketOrchestrationCore>) -> Self {
Self {
bucket_orchestration_client: bucket_orchestration_client
bucket_orchestration_client
}
}
@@ -73,8 +67,8 @@ impl RemoteCloudFileClient {
let mimetype: &str = get_mimetype_for_bytes(&bytes).unwrap_or("application/octet-stream");
Ok(FileMetaData {
file_size_bytes: file_size_bytes,
sha256_checksum: sha256_checksum,
file_size_bytes,
sha256_checksum,
mimetype: mimetype.to_string()
})
}
@@ -85,8 +79,9 @@ mod tests {
use async_trait::async_trait;
use env_logger;
use errors::AnyhowResult;
use crate::remote_file_manager::bucket_orchestration::BucketOrchestrationCore;
use crate::remote_file_manager::weights_descriptor::{WeightsLoRADescriptor, WeightsSD15Descriptor, WeightsSDXLDescriptor};
use crate::remote_file_manager::weights_descriptor::WeightsLoRADescriptor;
struct BucketOrchestrationMock {}
@@ -127,7 +122,6 @@ mod tests {
}
#[tokio::test]
async fn remote_file_manager_descriptor_test() {
use super::*;
let remote_cloud_bucket_details = RemoteCloudBucketDetails::new("object_hash".to_string(), "loRA".to_string(), "safetensors".to_string());
@@ -138,6 +132,8 @@ mod tests {
assert_eq!(file_descriptor.is_public(), true);
}
#[ignore]
#[tokio::test]
async fn remote_file_manager_download_existing_file() {
use super::*;
@@ -1,23 +1,23 @@
use super::file_descriptor::FileDescriptor;
pub struct WeightsLoRADescriptor {}
pub struct WeightsLoRADescriptor;
const REMOTE_FILE_DIRECTORY: &str = "/weights";
impl FileDescriptor for WeightsLoRADescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "safetensors".to_string();
"safetensors".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, loRA, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "loRA".to_string();
"loRA".to_string()
}
// This will be ensure that the right bucket is picked
@@ -26,22 +26,22 @@ impl FileDescriptor for WeightsLoRADescriptor {
}
}
pub struct WeightsSD15Descriptor {}
pub struct WeightsSD15Descriptor;
impl FileDescriptor for WeightsSD15Descriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "safetensors".to_string();
"safetensors".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "sd15".to_string();
"sd15".to_string()
}
// This will be ensure that the right bucket is picked
@@ -51,22 +51,22 @@ impl FileDescriptor for WeightsSD15Descriptor {
}
pub struct WeightsSDXLDescriptor {}
pub struct WeightsSDXLDescriptor;
impl FileDescriptor for WeightsSDXLDescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "safetensors".to_string();
"safetensors".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "sdxl".to_string();
"sdxl".to_string()
}
// This will be ensure that the right bucket is picked
@@ -75,22 +75,22 @@ impl FileDescriptor for WeightsSDXLDescriptor {
}
}
pub struct WeightsSVCDescriptor {}
pub struct WeightsSVCDescriptor;
impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsSVCDescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "safetensors".to_string();
"safetensors".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "svc".to_string();
"svc".to_string()
}
// This will be ensure that the right bucket is picked
@@ -99,22 +99,22 @@ impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsS
}
}
pub struct WeightsRVCDescriptor {}
pub struct WeightsRVCDescriptor;
impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsRVCDescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "pth".to_string();
"pth".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "rvc".to_string();
"rvc".to_string()
}
// This will be ensure that the right bucket is picked
@@ -123,22 +123,22 @@ impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsR
}
}
pub struct WeightsRVCIndexDescriptor {}
pub struct WeightsRVCIndexDescriptor;
impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsRVCIndexDescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "index".to_string();
"index".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "rvc".to_string();
"rvc".to_string()
}
// This will be ensure that the right bucket is picked
@@ -147,22 +147,22 @@ impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsR
}
}
pub struct WeightsVallePromptDescriptor {}
pub struct WeightsVallePromptDescriptor;
impl FileDescriptor for crate::remote_file_manager::weights_descriptor::WeightsVallePromptDescriptor {
fn remote_directory_path(&self) -> &str {
return REMOTE_FILE_DIRECTORY;
REMOTE_FILE_DIRECTORY
}
// this will be the type of file peroid is handled by the file formatter
// e.g safetensors bin jpg
fn get_suffix(&self)->String {
return "safetensors".to_string();
"safetensors".to_string()
}
// This will be the prefix of the media type or the weights type.
// name of the weights or the name of the media type
// vall-e_prompt, SD15, sd15, sdxl when implmenting add to the end
fn get_prefix(&self)->String {
return "valle_prompt".to_string();
"valle_prompt".to_string()
}
// This will be ensure that the right bucket is picked
@@ -73,7 +73,7 @@ impl GoogleDriveDownloadCommand {
info!("Downloading {} to: {}", download_url, temp_filename);
let mut command = format!("{} --url {} --output_filename {}",
let mut command = format!("{} --url \"{}\" --output_filename {}",
&self.download_script,
download_url,
&temp_filename);
@@ -10,6 +10,14 @@ pub fn is_bad_download_url(url: &str) -> anyhow::Result<bool> {
return Ok(true);
}
let looks_malicious = url.contains("\"")
|| url.contains("\'")
|| url.contains("\\");
if looks_malicious {
return Ok(true);
}
let url = Url::parse(url)?;
match url.host() {
@@ -23,6 +31,8 @@ pub fn is_bad_download_url(url: &str) -> anyhow::Result<bool> {
|| domain.contains("youtu.be")
|| domain.contains("youtube.com");
println!("domain: {:?}", domain);
if bad_host {
return Ok(true)
}
@@ -39,15 +49,31 @@ mod tests {
#[test]
fn bad_download_url() {
assert_eq!(is_bad_download_url("").unwrap(), true);
assert_eq!(is_bad_download_url(" ").unwrap(), true);
assert_eq!(is_bad_download_url("https://vm.tiktok.com/ZMNYjT7Xy/?k=1 ").unwrap(), true); // NB: We get lots of these
assert_eq!(is_bad_download_url("https://m.youtube.com/watch?v=HY-vzGBiAZo").unwrap(), true); // NB: We get lots of these
assert_eq!(true, is_bad_download_url("").unwrap());
assert_eq!(true, is_bad_download_url(" ").unwrap());
assert_eq!(true, is_bad_download_url("https://vm.tiktok.com/ZMNYjT7Xy/?k=1 ").unwrap()); // NB: We get lots of these
assert_eq!(true, is_bad_download_url("https://m.youtube.com/watch?v=HY-vzGBiAZo").unwrap()); // NB: We get lots of these
}
#[test]
fn attacker_urls() {
// NB: These URLs may break escaping
assert_eq!(true, is_bad_download_url("https://huggingface.co/\"").unwrap());
assert_eq!(true, is_bad_download_url("https://huggingface.co/\"/bin/bash").unwrap());
assert_eq!(true, is_bad_download_url("https://huggingface.co/\\\'").unwrap());
assert_eq!(true, is_bad_download_url("https://huggingface.co/\\\"").unwrap());
}
#[test]
fn good_tts_model_url() {
assert_eq!(is_bad_download_url("https://drive.google.com/file/d/1-1kEoX4HGCwJm4R9cZhSVByWmUoQVGVm/view").unwrap(), false);
assert_eq!(is_bad_download_url("https://drive.google.com/file/d/1SofQhvSkDY-vi_zuBfHivBbJo4CqhJeH/view?usp=sharing").unwrap(), false);
assert_eq!(false, is_bad_download_url("https://drive.google.com/file/d/1-1kEoX4HGCwJm4R9cZhSVByWmUoQVGVm/view").unwrap());
assert_eq!(false, is_bad_download_url("https://drive.google.com/file/d/1SofQhvSkDY-vi_zuBfHivBbJo4CqhJeH/view?usp=sharing").unwrap());
}
#[test]
fn tricky_download_urls() {
// NB: These are URLs that have caused problems
assert_eq!(false, is_bad_download_url("https://huggingface.co/aaalby/nmixx/resolve/main/kyujin(1).zip").unwrap());
assert_eq!(false, is_bad_download_url("https://huggingface.co/datasets/Roscall/Elvis70s/resolve/main/Elvis%2070s%20RVC.zip?download=true").unwrap());
}
}
@@ -1 +1,2 @@
pub mod tts_model_document;
pub mod tts_model_document;
mod model_weight_document;
@@ -0,0 +1,65 @@
use std::path::PathBuf;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use enums::by_table::model_weights::weights_category::WeightsCategory;
use enums::by_table::model_weights::weights_types::WeightsType;
use enums::common::visibility::Visibility;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::model_weights::ModelWeightToken;
use tokens::tokens::users::UserToken;
use crate::traits::document::Document;
/// The current name for the index.
/// We may need to perform migrations in the future
pub const MODEL_WEIGHT_INDEX: &str = "model_weights_v1";
#[derive(Serialize, Deserialize, Debug)]
pub struct ModelWeightDocument {
pub token: ModelWeightToken,
pub creator_set_visibility: Visibility,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
pub maybe_cover_image_media_file_token: Option<MediaFileToken>,
pub maybe_cover_image_media_bucket_path: Option<String>,
//pub description_markdown: String,
//pub description_markdown_html: String,
// TODO(bt,2023-12-22): Populate these fields
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
pub cached_user_ratings_ratio: f32,
pub cached_user_ratings_last_updated_at: DateTime<Utc>,
pub creator_user_token: UserToken,
pub creator_username: String,
pub creator_display_name: String,
pub creator_gravatar_hash: String,
// Fields only used for TTS models and voice conversion models.
pub maybe_ietf_language_tag: Option<String>,
pub maybe_ietf_primary_language_subtag: Option<String>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
impl Document for ModelWeightDocument {
fn get_document_id(&self) -> String {
self.token.to_string()
}
fn get_document_path(&self) -> PathBuf {
let document_id = self.get_document_id();
PathBuf::from(format!("/{MODEL_WEIGHT_INDEX}/_doc/{document_id}"))
}
}
+1 -1
View File
@@ -18,7 +18,7 @@ path = "src/lib.rs"
my-workspace-hack = { version = "0.1", path = "../../../my-workspace-hack" }
# External
infer = "0.11" # Mimetype detection
infer = "0.15.0" # Mimetype detection
[dev-dependencies]
@@ -4,16 +4,23 @@
pub fn mimetype_to_extension(mimetype: &str) -> Option<&'static str> {
// TODO(bt,2023-11-17): maybe make a bidirectional map
match mimetype {
// Image
"image/gif" => Some("gif"),
"image/jpeg" => Some("jpg"),
"image/png" => Some("png"),
"image/webp" => Some("webp"),
// Audio
"audio/aac" => Some("aac"),
"audio/m4a" => Some("m4a"),
"audio/mpeg" => Some("mp3"),
"audio/ogg" => Some("ogg"),
"audio/opus" => Some("opus"),
"audio/x-flac" => Some("flac"),
"audio/wav" => Some("wav"),
"audio/x-flac" => Some("flac"),
"audio/x-wav" => Some("wav"),
// Video
"video/mp4" /* .mp4 */ => Some("mp4"),
"video/quicktime" /* .mov */ => Some("mov"),
"video/webm" => Some("webm"), // TODO: Browsers send this for audio?
_ => None
}
@@ -35,5 +42,8 @@ mod test {
assert_eq!(mimetype_to_extension("audio/mpeg"), Some("mp3"));
assert_eq!(mimetype_to_extension("audio/wav"), Some("wav"));
assert_eq!(mimetype_to_extension("audio/x-wav"), Some("wav"));
assert_eq!(mimetype_to_extension("image/png"), Some("png"));
assert_eq!(mimetype_to_extension("image/jpeg"), Some("jpg"));
assert_eq!(mimetype_to_extension("video/mp4"), Some("mp4"));
}
}
@@ -6,28 +6,54 @@ use buckets::public::weight_files::bucket_file_path::WeightFileBucketPath;
use cloud_storage::bucket_path_unifier::BucketPathUnifier;
use enums::by_table::model_weights::weights_types::WeightsType;
use enums::by_table::voice_conversion_models::voice_conversion_model_type::VoiceConversionModelType;
use errors::AnyhowResult;
use jobs_common::semi_persistent_cache_dir::SemiPersistentCacheDir;
use mysql_queries::queries::model_weights::inference::get_model_weight_for_voice_conversion_inference::{get_model_weight_for_voice_conversion_inference, ModelWeightForVoiceConversionInference};
use mysql_queries::queries::voice_conversion::inference::get_voice_conversion_model_for_inference::{get_voice_conversion_model_for_inference, VoiceConversionModelForInference};
use mysql_queries::queries::model_weights::inference::get_model_weight_for_voice_conversion_inference::{get_model_weight_for_voice_conversion_inference, ModelWeightError, ModelWeightForVoiceConversionInference};
use mysql_queries::queries::voice_conversion::inference::get_voice_conversion_model_for_inference::{get_voice_conversion_model_for_inference, VoiceConversionModelForInference, VoiceConversionModelForInferenceError};
use tokens::tokens::model_weights::ModelWeightToken;
#[derive(Clone, Debug)]
pub enum VcModelError {
ModelDeleted,
DatabaseError { reason: String },
}
impl std::fmt::Display for VcModelError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
VcModelError::ModelDeleted => write!(f, "ModelDeleted"),
VcModelError::DatabaseError { reason} =>
write!(f, "Database error: {:?}", reason),
}
}
}
impl std::error::Error for VcModelError {}
/// Query model info
/// Depending on the token prefix, this does either a modern or legacy lookup.
pub async fn query_vc_model_for_migration(model_token: &str, mysql_pool: &MySqlPool) -> AnyhowResult<Option<VcModel>> {
pub async fn query_vc_model_for_migration(model_token: &str, mysql_pool: &MySqlPool) -> Result<Option<VcModel>, VcModelError> {
// NB: This is temporary migration code as we switch from the `voice_conversion_models` table to the `model_weights` table.
if model_token.starts_with(ModelWeightToken::token_prefix()) {
let model_weights_token = ModelWeightToken::new_from_str(model_token);
let maybe_model = get_model_weight_for_voice_conversion_inference(
&mysql_pool, &model_weights_token).await?;
&mysql_pool, &model_weights_token).await.map_err(|err|
match err {
ModelWeightError::ModelDeleted => VcModelError::ModelDeleted,
ModelWeightError::DatabaseError { reason } => VcModelError::DatabaseError { reason },
}
)?;
Ok(maybe_model
.map(|model| VcModel::ModelWeight(model)))
} else {
let maybe_vc_model = get_voice_conversion_model_for_inference(
&mysql_pool, model_token).await?;
&mysql_pool, model_token).await.map_err(|err|
match err {
VoiceConversionModelForInferenceError::ModelDeleted => VcModelError::ModelDeleted,
VoiceConversionModelForInferenceError::DatabaseError { reason } => VcModelError::DatabaseError { reason },
})?;
Ok(maybe_vc_model
.map(|model| VcModel::LegacyVoiceConversion(model)))
@@ -361,7 +387,7 @@ mod tests {
model.weights_type = WeightsType::RvcV2;
model.has_index_file = true;
let path = VcModel::ModelWeight(model).get_index_file_cloud_bucket_path(&BucketPathUnifier::default_paths());
assert_eq!(path, Some(PathBuf::from("/weights/h/a/s/hash/hash.rvc_index")));
assert_eq!(path, Some(PathBuf::from("/weights/h/a/s/hash/rvc_hash.index")));
}
}
}
@@ -52,7 +52,7 @@ SELECT
id,
token,
creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,
creator_user_token,
creator_ip_address,
creator_set_visibility,
@@ -101,7 +101,7 @@ WHERE
id: GenericDownloadJobId(record.id),
download_job_token: DownloadJobToken::new(record.token),
creator_ip_address: record.creator_ip_address,
creator_user_token: record.creator_user_token,
creator_user_token: UserToken::new_from_str(&record.creator_user_token),
// NB: Failure case for parsing visibility - default to private
creator_set_visibility: Visibility::from_str(&record.creator_set_visibility)
.unwrap_or(Visibility::Private),
@@ -144,7 +144,7 @@ struct AvailableDownloadJobRawInternal {
pub id: i64,
pub token: String,
pub creator_user_token: UserToken,
pub creator_user_token: String,
pub creator_ip_address: String,
pub creator_set_visibility: String,
@@ -0,0 +1,158 @@
use anyhow::anyhow;
use sqlx;
use sqlx::MySqlPool;
use enums::by_table::generic_synthetic_ids::id_category::IdCategory;
use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCategory;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::anonymous_visitor_tracking::AnonymousVisitorTrackingToken;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::users::UserToken;
use crate::queries::generic_synthetic_ids::transactional_increment_generic_synthetic_id::transactional_increment_generic_synthetic_id;
pub enum UploadType {
Filesystem,
DeviceCaptureApi,
}
pub struct InsertMediaFileFromUploadArgs<'a> {
pub pool: &'a MySqlPool,
pub maybe_creator_user_token: Option<&'a UserToken>,
pub maybe_creator_anonymous_visitor_token: Option<&'a AnonymousVisitorTrackingToken>,
pub creator_ip_address: &'a str,
pub creator_set_visibility: Visibility,
pub upload_type: UploadType,
pub media_file_type: MediaFileType,
pub maybe_mime_type: Option<&'a str>,
pub file_size_bytes: u64,
pub duration_millis: u64,
pub sha256_checksum: &'a str,
// TODO: Media duration.
//pub duration_millis: u64,
pub public_bucket_directory_hash: &'a str,
pub maybe_public_bucket_prefix: Option<&'a str>,
pub maybe_public_bucket_extension: Option<&'a str>,
}
pub async fn insert_media_file_from_file_upload(
args: InsertMediaFileFromUploadArgs<'_>
) -> AnyhowResult<(MediaFileToken, u64)>
{
let token = MediaFileToken::generate();
let origin_category = match args.upload_type {
UploadType::Filesystem => MediaFileOriginCategory::Upload,
UploadType::DeviceCaptureApi => MediaFileOriginCategory::DeviceApi,
};
let mut maybe_creator_file_synthetic_id : Option<u64> = None;
let mut maybe_creator_category_synthetic_id : Option<u64> = None;
let mut transaction = args.pool.begin().await?;
if let Some(user_token) = args.maybe_creator_user_token.as_deref() {
let next_media_file_id = transactional_increment_generic_synthetic_id(
&user_token,
IdCategory::MediaFile,
&mut transaction
).await?;
let next_voice_conversion_id = transactional_increment_generic_synthetic_id(
&user_token,
IdCategory::FileUpload,
&mut transaction
).await?;
maybe_creator_file_synthetic_id = Some(next_media_file_id);
maybe_creator_category_synthetic_id = Some(next_voice_conversion_id);
}
const ORIGIN_PRODUCT_CATEGORY : MediaFileOriginProductCategory = MediaFileOriginProductCategory::Unknown;
let record_id = {
let query_result = sqlx::query!(
r#"
INSERT INTO media_files
SET
token = ?,
origin_category = ?,
origin_product_category = ?,
maybe_origin_model_type = NULL,
maybe_origin_model_token = NULL,
media_type = ?,
maybe_mime_type = ?,
file_size_bytes = ?,
checksum_sha2 = ?,
public_bucket_directory_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
maybe_creator_user_token = ?,
maybe_creator_anonymous_visitor_token = ?,
creator_ip_address = ?,
creator_set_visibility = ?,
maybe_creator_file_synthetic_id = ?,
maybe_creator_category_synthetic_id = ?
"#,
token.as_str(),
origin_category.to_str(),
ORIGIN_PRODUCT_CATEGORY.to_str(),
args.media_file_type.to_str(),
args.maybe_mime_type,
args.file_size_bytes,
args.sha256_checksum,
args.public_bucket_directory_hash,
args.maybe_public_bucket_prefix,
args.maybe_public_bucket_extension,
args.maybe_creator_user_token,
args.maybe_creator_anonymous_visitor_token,
args.creator_ip_address,
args.creator_set_visibility.to_str(),
maybe_creator_file_synthetic_id,
maybe_creator_category_synthetic_id,
)
.execute(&mut *transaction)
.await;
let record_id = match query_result {
Ok(res) => {
res.last_insert_id()
},
Err(err) => {
// TODO: handle better
//transaction.rollback().await?;
return Err(anyhow!("Mysql error: {:?}", err));
}
};
record_id
};
transaction.commit().await?;
Ok((token, record_id))
}
@@ -0,0 +1,7 @@
pub mod insert_media_file_from_cli_tool;
pub mod insert_media_file_from_face_animation;
pub mod insert_media_file_from_file_upload;
pub mod insert_media_file_from_rerender;
pub mod insert_media_file_from_tacotron2;
pub mod insert_media_file_from_voice_conversion;
pub mod insert_media_file_from_zero_shot_tts;
@@ -6,6 +6,7 @@ use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCat
use enums::by_table::media_files::media_file_origin_model_type::MediaFileOriginModelType;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::view_as::ViewAs;
use enums::common::visibility::Visibility;
use enums::traits::mysql_from_row::MySqlFromRow;
use errors::AnyhowResult;
@@ -44,21 +45,19 @@ pub struct MediaFileListItem {
pub creator_set_visibility: Visibility,
pub comment_count: u64,
pub favorite_count: u64,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
#[derive(Clone, Copy)]
pub enum ViewAs {
Moderator,
NonModerator,
}
pub struct ListMediaFilesArgs<'a> {
pub limit: usize,
pub maybe_filter_media_type: Option<MediaFileType>,
pub maybe_offset: Option<usize>,
pub cursor_is_reversed: bool,
pub sort_ascending: bool,
pub view_as: ViewAs,
pub mysql_pool: &'a MySqlPool,
}
@@ -70,6 +69,7 @@ pub async fn list_media_files(args: ListMediaFilesArgs<'_>) -> AnyhowResult<Medi
args.limit,
args.maybe_offset,
args.cursor_is_reversed,
args.sort_ascending,
args.view_as,
);
@@ -100,6 +100,8 @@ pub async fn list_media_files(args: ListMediaFilesArgs<'_>) -> AnyhowResult<Medi
maybe_creator_display_name: record.maybe_creator_display_name,
maybe_creator_gravatar_hash: record.maybe_creator_gravatar_hash,
creator_set_visibility: record.creator_set_visibility,
comment_count: record.comment_count as u64,
favorite_count: record.favorite_count as u64,
created_at: record.created_at,
updated_at: record.updated_at,
}
@@ -119,9 +121,11 @@ fn query_builder<'a>(
limit: usize,
maybe_offset: Option<usize>,
cursor_is_reversed: bool,
sort_ascending: bool,
view_as: ViewAs,
) -> QueryBuilder<'a, MySql> {
let mut sort_ascending = sort_ascending;
// NB: Query cannot be statically checked by sqlx
let mut query_builder: QueryBuilder<MySql> = QueryBuilder::new(
r#"
@@ -148,28 +152,48 @@ SELECT
m.creator_set_visibility,
m.created_at,
m.updated_at
m.updated_at,
COUNT(f.id) as favorite_count,
COUNT(c.id) as comment_count
FROM media_files AS m
LEFT OUTER JOIN users AS u
ON m.maybe_creator_user_token = u.token
WHERE m.user_deleted_at IS NULL
AND m.mod_deleted_at IS NULL
LEFT OUTER JOIN favorites as f
ON f.entity_type = 'media_file' AND f.entity_token = m.token
LEFT OUTER JOIN comments as c
ON c.entity_type = 'media_file' AND c.entity_token = c.token
"#
);
let mut first_predicate_added = false;
if let Some(media_type) = maybe_filter_media_type {
if !first_predicate_added {
query_builder.push(" WHERE ");
first_predicate_added = true;
} else {
query_builder.push(" AND ");
}
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.media_file_type = ");
query_builder.push(" m.media_type = ");
query_builder.push_bind(media_type.to_str());
}
match view_as {
ViewAs::Moderator => {}
ViewAs::NonModerator=> {
ViewAs::Author | ViewAs::AnotherUser => {
if !first_predicate_added {
query_builder.push(" WHERE ");
first_predicate_added = true;
} else {
query_builder.push(" AND ");
}
query_builder.push(" m.user_deleted_at IS NULL AND m.mod_deleted_at IS NULL ");
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
@@ -179,14 +203,36 @@ AND m.mod_deleted_at IS NULL
}
if let Some(offset) = maybe_offset {
if cursor_is_reversed {
query_builder.push(format!(" AND m.id < {offset} "));
if !first_predicate_added {
query_builder.push(" WHERE ");
first_predicate_added = true;
} else {
query_builder.push(format!(" AND m.id > {offset} "));
query_builder.push(" AND ");
}
if sort_ascending {
if cursor_is_reversed {
// NB: We're searching backwards.
query_builder.push(" AND m.id < ");
sort_ascending = !sort_ascending;
} else {
query_builder.push(" AND m.id > ");
}
} else {
if cursor_is_reversed {
// NB: We're searching backwards.
query_builder.push(" AND m.id > ");
sort_ascending = !sort_ascending;
} else {
query_builder.push(" AND m.id < ");
}
}
query_builder.push_bind(offset as i64);
}
if cursor_is_reversed {
query_builder.push(" GROUP BY m.id");
if sort_ascending {
query_builder.push(" ORDER BY m.id ASC ");
} else {
query_builder.push(" ORDER BY m.id DESC ");
@@ -218,6 +264,9 @@ struct MediaFileListItemInternal {
creator_set_visibility: Visibility,
comment_count: i64,
favorite_count: i64,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
@@ -257,6 +306,8 @@ impl FromRow<'_, MySqlRow> for MediaFileListItemInternal {
creator_set_visibility: Visibility::try_from_mysql_row(row, "creator_set_visibility")?,
created_at: row.try_get("created_at")?,
updated_at: row.try_get("updated_at")?,
comment_count: row.try_get("comment_count")?,
favorite_count: row.try_get("favorite_count")?,
})
}
}
@@ -6,6 +6,7 @@ use enums::by_table::media_files::media_file_origin_category::MediaFileOriginCat
use enums::by_table::media_files::media_file_origin_model_type::MediaFileOriginModelType;
use enums::by_table::media_files::media_file_origin_product_category::MediaFileOriginProductCategory;
use enums::by_table::media_files::media_file_type::MediaFileType;
use enums::common::view_as::ViewAs;
use enums::common::visibility::Visibility;
use enums::traits::mysql_from_row::MySqlFromRow;
use errors::AnyhowResult;
@@ -39,25 +40,18 @@ pub struct MediaFileListItem {
pub updated_at: DateTime<Utc>,
}
#[derive(Clone, Copy)]
pub enum ViewAs {
Author,
Moderator,
AnotherUser,
}
pub struct ListMediaFileForUserArgs<'a> {
pub username: &'a str,
pub maybe_filter_media_type: Option<MediaFileType>,
pub page_size: usize,
pub page_index: usize,
pub cursor_is_reversed: bool,
pub sort_ascending: bool,
pub view_as: ViewAs,
pub mysql_pool: &'a MySqlPool,
}
pub async fn list_media_files_for_user(args: ListMediaFileForUserArgs<'_>) -> AnyhowResult<MediaFileListPage> {
/// Let's figure out how many results we could have returned total
let count_fields = select_total_count_field();
let mut count_query_builder = query_builder(
@@ -66,16 +60,13 @@ pub async fn list_media_files_for_user(args: ListMediaFileForUserArgs<'_>) -> An
false,
0,
0,
args.cursor_is_reversed,
args.sort_ascending,
args.view_as,
count_fields.as_str(),
);
let row_count_query = count_query_builder.build_query_scalar::<i64>();
let row_count_result = row_count_query.fetch_one(args.mysql_pool).await?;
/// Figure out limit start and end based on page size and indexes as requested
let limit_start = args.page_size * args.page_index;
let limit_end = limit_start + args.page_size;
/// Now fetch the actual results with all the fields
let result_fields = select_result_fields();
@@ -83,9 +74,9 @@ pub async fn list_media_files_for_user(args: ListMediaFileForUserArgs<'_>) -> An
args.maybe_filter_media_type,
args.username,
true,
limit_start,
limit_end,
args.cursor_is_reversed,
args.page_index,
args.page_size,
args.sort_ascending,
args.view_as,
result_fields.as_str(),
);
@@ -115,7 +106,7 @@ pub async fn list_media_files_for_user(args: ListMediaFileForUserArgs<'_>) -> An
Ok(MediaFileListPage {
records: results,
sort_ascending: !args.cursor_is_reversed,
sort_ascending: args.sort_ascending,
current_page: args.page_index,
total_page_count: number_of_pages,
})
@@ -157,9 +148,9 @@ fn query_builder<'a>(
maybe_filter_media_type: Option<MediaFileType>,
username: &'a str,
enforce_limits: bool,
limit_start: usize,
limit_end: usize,
cursor_is_reversed: bool,
page_index: usize,
page_size: usize,
sort_ascending: bool,
view_as: ViewAs,
select_fields: &'a str,
) -> QueryBuilder<'a, MySql> {
@@ -172,27 +163,27 @@ SELECT
FROM media_files AS m
LEFT OUTER JOIN users AS u
ON m.maybe_creator_user_token = u.token
WHERE m.user_deleted_at IS NULL
AND m.mod_deleted_at IS NULL
"#
));
query_builder.push(" AND u.username = ");
query_builder.push(" WHERE u.username = ");
query_builder.push_bind(username);
if let Some(media_type) = maybe_filter_media_type {
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
query_builder.push(" AND m.media_file_type = ");
query_builder.push(" AND m.media_type = ");
query_builder.push_bind(media_type.to_str());
}
match view_as {
ViewAs::Author => {}
ViewAs::Author => {
query_builder.push(" AND m.user_deleted_at IS NULL AND m.mod_deleted_at IS NULL ");
}
ViewAs::Moderator => {}
ViewAs::AnotherUser => {
query_builder.push(" AND m.user_deleted_at IS NULL AND m.mod_deleted_at IS NULL ");
// FIXME: Binding shouldn't require to_str().
// Otherwise, it's calling the Display trait on the raw type which is resulting in an
// incorrect binding and runtime error.
@@ -201,18 +192,15 @@ WHERE m.user_deleted_at IS NULL
}
}
if cursor_is_reversed {
query_builder.push(" ORDER BY m.id ASC ");
if sort_ascending {
query_builder.push(" ORDER BY m.created_at ASC ");
} else {
query_builder.push(" ORDER BY m.id DESC ");
query_builder.push(" ORDER BY m.created_at DESC ");
}
if enforce_limits {
if cursor_is_reversed {
query_builder.push(format!(" LIMIT {limit_start}, {limit_end} "));
} else {
query_builder.push(format!(" LIMIT {limit_end}, {limit_start} "));
}
let offset = page_index * page_size;
query_builder.push(format!(" LIMIT {page_size} OFFSET {offset} "));
}
query_builder
@@ -0,0 +1,3 @@
pub mod list_media_files;
pub mod list_media_files_by_tokens;
pub mod list_media_files_for_user;
@@ -1,12 +1,5 @@
pub mod create;
pub mod delete_media_file;
pub mod get_media_file;
pub mod insert_media_file_from_cli_tool;
pub mod insert_media_file_from_face_animation;
pub mod insert_media_file_from_rerender;
pub mod insert_media_file_from_tacotron2;
pub mod insert_media_file_from_voice_conversion;
pub mod insert_media_file_from_zero_shot_tts;
pub mod list_media_files;
pub mod list_media_files_by_tokens;
pub mod list_media_files_for_user;
pub mod list;
pub mod update_media_file;
@@ -63,11 +63,9 @@ SET
weights_category = ?,
title = ?,
maybe_thumbnail_token = NULL,
maybe_avatar_media_file_token = NULL,
maybe_cover_media_file_token = NULL,
description_markdown = NULL,
description_rendered_html = NULL,
maybe_cover_image_media_file_token = NULL,
maybe_description_markdown = NULL,
maybe_description_rendered_html = NULL,
maybe_last_update_user_token = NULL,
original_download_url = ?,
@@ -79,9 +77,9 @@ SET
creator_ip_address = ?,
creator_set_visibility = ?,
private_bucket_hash = ?,
maybe_private_bucket_prefix = ?,
maybe_private_bucket_extension = ?,
public_bucket_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
cached_user_ratings_total_count = 0,
cached_user_ratings_positive_count = 0,
@@ -13,9 +13,8 @@ pub struct CreateModelWeightsArgs<'a> {
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub maybe_description_markdown: Option<String>,
pub maybe_description_rendered_html: Option<String>,
pub creator_user_token: Option<&'a UserToken>,
pub creator_ip_address: &'a str,
pub creator_set_visibility: Visibility,
@@ -24,9 +23,9 @@ pub struct CreateModelWeightsArgs<'a> {
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
@@ -47,9 +46,8 @@ pub async fn create_weight(args: CreateModelWeightsArgs<'_>) -> AnyhowResult<Mod
weights_type = ?,
weights_category = ?,
title = ?,
maybe_thumbnail_token = ?,
description_markdown = ?,
description_rendered_html = ?,
maybe_description_markdown = ?,
maybe_description_rendered_html = ?,
creator_user_token = ?,
creator_ip_address = ?,
creator_set_visibility = ?,
@@ -58,9 +56,9 @@ pub async fn create_weight(args: CreateModelWeightsArgs<'_>) -> AnyhowResult<Mod
original_filename = ?,
file_size_bytes = ?,
file_checksum_sha2 = ?,
private_bucket_hash = ?,
maybe_private_bucket_prefix = ?,
maybe_private_bucket_extension = ?,
public_bucket_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
cached_user_ratings_total_count = ?,
cached_user_ratings_positive_count = ?,
cached_user_ratings_negative_count = ?,
@@ -71,9 +69,8 @@ pub async fn create_weight(args: CreateModelWeightsArgs<'_>) -> AnyhowResult<Mod
args.weights_type.to_str(),
args.weights_category.to_str(),
args.title,
args.maybe_thumbnail_token,
args.description_markdown,
args.description_rendered_html,
args.maybe_description_markdown,
args.maybe_description_rendered_html,
args.creator_user_token.as_deref(),
args.creator_ip_address,
args.creator_set_visibility.to_str(),
@@ -82,9 +79,9 @@ pub async fn create_weight(args: CreateModelWeightsArgs<'_>) -> AnyhowResult<Mod
args.original_filename,
args.file_size_bytes,
args.file_checksum_sha2,
args.private_bucket_hash,
args.maybe_private_bucket_prefix,
args.maybe_private_bucket_extension,
args.public_bucket_hash,
args.maybe_public_bucket_prefix,
args.maybe_public_bucket_extension,
args.cached_user_ratings_total_count,
args.cached_user_ratings_positive_count,
args.cached_user_ratings_negative_count,
@@ -1 +1,2 @@
pub mod create_model_weight_from_voice_conversion_download_job;
pub mod create_model_weight_from_voice_conversion_download_job;
pub mod create_weight;
@@ -0,0 +1,2 @@
pub mod set_model_weight_cover_image;
pub mod update_weight;
@@ -0,0 +1,32 @@
use sqlx::MySqlPool;
use errors::AnyhowResult;
use tokens::tokens::media_files::MediaFileToken;
use tokens::tokens::model_weights::ModelWeightToken;
pub struct UpdateArgs<'a> {
pub model_weight_token: &'a ModelWeightToken,
pub maybe_cover_image_media_file_token: Option<&'a MediaFileToken>,
//pub update_ip_address: &'a str,
pub mysql_pool: &'a MySqlPool
}
pub async fn set_model_weight_cover_image(args: UpdateArgs<'_>) -> AnyhowResult<()>{
let transaction = args.mysql_pool.begin().await?;
let query_result = sqlx::query!(
r#"
UPDATE model_weights
SET
maybe_cover_image_media_file_token = ?
WHERE token = ?
LIMIT 1
"#,
args.maybe_cover_image_media_file_token,
args.model_weight_token,
).execute(args.mysql_pool).await;
transaction.commit().await?;
Ok(())
}
@@ -0,0 +1,103 @@
use anyhow::anyhow;
use sqlx::{MySql, MySqlPool, QueryBuilder};
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::model_weights::ModelWeightToken;
pub struct UpdateWeightArgs<'a> {
pub weight_token: &'a ModelWeightToken,
pub title: Option<&'a str>,
pub maybe_description_markdown: Option<&'a str>,
pub maybe_description_rendered_html: Option<&'a str>,
pub creator_set_visibility: Option<&'a Visibility>,
pub weights_type: Option<String>,
pub weights_category: Option<String>,
pub mysql_pool: &'a MySqlPool,
}
pub async fn update_weights(args: UpdateWeightArgs<'_>) -> AnyhowResult<()> {
if args.title.is_none()
&& args.maybe_description_markdown.is_none()
&& args.maybe_description_rendered_html.is_none()
&& args.creator_set_visibility.is_none()
&& args.weights_type.is_none()
&& args.weights_category.is_none(){
return Err(anyhow!("No fields to update"));
}
let mut query_builder: QueryBuilder<MySql> = QueryBuilder::new(
r#"
UPDATE model_weights
SET
"#
);
let mut separated = query_builder.separated(", ");
if let Some(title) = args.title {
separated.push("title = ");
separated.push_bind_unseparated(title);
}
if let Some(maybe_description_markdown) = args.maybe_description_markdown {
separated.push("maybe_description_markdown = ");
separated.push_bind_unseparated(maybe_description_markdown);
}
if let Some(maybe_description_rendered_html) = args.maybe_description_rendered_html {
separated.push("maybe_description_rendered_html = ");
separated.push_bind_unseparated(maybe_description_rendered_html);
}
if let Some(creator_set_visibility) = args.creator_set_visibility {
separated.push("creator_set_visibility = ");
separated.push_bind_unseparated(creator_set_visibility.to_str());
}
if let Some(weights_type) = args.weights_type {
separated.push("weights_type = ");
separated.push_bind_unseparated(weights_type);
}
if let Some(weights_category) = args.weights_category {
separated.push("weights_category = ");
separated.push_bind_unseparated(weights_category);
}
separated.push("version = version + 1");
separated.push_unseparated(" WHERE token = ");
separated.push_bind_unseparated(args.weight_token.as_str());
separated.push_unseparated(" LIMIT 1");
let transaction = args.mysql_pool.begin().await?;
let query_result = query_builder.build().execute(args.mysql_pool).await;
transaction.commit().await?;
match query_result {
Ok(_) => Ok(()),
Err(err) => {
Err(anyhow!("weights update error: {:?}", err))
}
}
}
#[cfg(test)]
mod tests {
// Template
use sqlx::mysql::MySqlPoolOptions;
use tokio;
use config::shared_constants::DEFAULT_MYSQL_CONNECTION_STRING;
use errors::AnyhowResult;
#[ignore]
#[tokio::test]
async fn test_update_weights() -> AnyhowResult<()> {
let db_connection_string = DEFAULT_MYSQL_CONNECTION_STRING;
let pool = MySqlPoolOptions::new()
.max_connections(3)
.connect(&db_connection_string)
.await?;
Ok(())
}
}
@@ -1,40 +1,48 @@
use container_common::anyhow_result::AnyhowResult;
use sqlx::{ MySqlPool, MySql };
use sqlx::pool::PoolConnection;
use anyhow::anyhow;
use log::{ error };
use chrono::{DateTime, Utc};
use log::error;
use sqlx::{MySql, MySqlPool};
use sqlx::pool::PoolConnection;
use enums::by_table::model_weights::weights_types::WeightsType;
use container_common::anyhow_result::AnyhowResult;
use enums::by_table::model_weights::weights_category::WeightsCategory;
use enums::by_table::model_weights::weights_types::WeightsType;
use enums::common::visibility::Visibility;
use tokens::tokens::{ users::UserToken, model_weights::ModelWeightToken };
use chrono::{ DateTime, Utc };
use tokens::tokens::{model_weights::ModelWeightToken, users::UserToken};
// Notes ensure that Enums have sqlx::Type
// Notes ensure that Enums have sqlx::Type
// 'weights_type: enums::by_table::model_weights::weights_types::WeightsType' use this to map
// Retrieved Model Weight can be constrained to the fields that are needed
pub struct RetrivedModelWeight {
pub struct RetrievedModelWeight {
pub token: ModelWeightToken,
pub title: String,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub maybe_description_markdown: Option<String>,
pub maybe_description_rendered_html: Option<String>,
pub creator_user_token: UserToken,
pub creator_username: String,
pub creator_display_name: String,
pub creator_gravatar_hash: String,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>, // TODO crossproduct rename these to public thx ^-^
pub maybe_private_bucket_extension: Option<String>,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_negative_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_total_count: u32,
@@ -51,7 +59,7 @@ pub async fn get_weight_by_token(
weight_token: &ModelWeightToken,
can_see_deleted: bool,
mysql_pool: &MySqlPool
) -> AnyhowResult<Option<RetrivedModelWeight>> {
) -> AnyhowResult<Option<RetrievedModelWeight>> {
let mut connection = mysql_pool.acquire().await?;
get_weights_by_token_with_connection(weight_token, can_see_deleted, &mut connection).await
}
@@ -60,7 +68,7 @@ pub async fn get_weights_by_token_with_connection(
weight_token: &ModelWeightToken,
can_see_deleted: bool,
mysql_connection: &mut PoolConnection<MySql>
) -> AnyhowResult<Option<RetrivedModelWeight>> {
) -> AnyhowResult<Option<RetrievedModelWeight>> {
let maybe_result = if can_see_deleted {
select_include_deleted(weight_token, mysql_connection).await
} else {
@@ -81,15 +89,17 @@ pub async fn get_weights_by_token_with_connection(
// unwrap the result
Ok(
Some(RetrivedModelWeight {
Some(RetrievedModelWeight {
token: record.token,
title: record.title,
weights_type: record.weights_type,
weights_category: record.weights_category,
maybe_thumbnail_token: record.maybe_thumbnail_token,
description_markdown: record.description_markdown,
description_rendered_html: record.description_rendered_html,
maybe_description_markdown: record.maybe_description_markdown,
maybe_description_rendered_html: record.maybe_description_rendered_html,
creator_user_token: record.creator_user_token,
creator_username: record.creator_username,
creator_display_name: record.creator_display_name,
creator_gravatar_hash: record.creator_gravatar_hash,
creator_ip_address: record.creator_ip_address,
creator_set_visibility: record.creator_set_visibility,
maybe_last_update_user_token: record.maybe_last_update_user_token,
@@ -97,9 +107,12 @@ pub async fn get_weights_by_token_with_connection(
original_filename: record.original_filename,
file_size_bytes: record.file_size_bytes,
file_checksum_sha2: record.file_checksum_sha2,
private_bucket_hash: record.private_bucket_hash,
maybe_private_bucket_prefix: record.maybe_private_bucket_prefix,
maybe_private_bucket_extension: record.maybe_private_bucket_extension,
public_bucket_hash: record.public_bucket_hash,
maybe_public_bucket_prefix: record.maybe_public_bucket_prefix,
maybe_public_bucket_extension: record.maybe_public_bucket_extension,
maybe_cover_image_public_bucket_hash: record.maybe_cover_image_public_bucket_hash,
maybe_cover_image_public_bucket_prefix: record.maybe_cover_image_public_bucket_prefix,
maybe_cover_image_public_bucket_extension: record.maybe_cover_image_public_bucket_extension,
cached_user_ratings_negative_count: record.cached_user_ratings_negative_count,
cached_user_ratings_positive_count: record.cached_user_ratings_positive_count,
cached_user_ratings_total_count: record.cached_user_ratings_total_count,
@@ -127,10 +140,14 @@ async fn select_include_deleted(
wt.title,
wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,
wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,
wt.maybe_thumbnail_token,
wt.description_markdown,
wt.description_rendered_html,
wt.maybe_description_markdown,
wt.maybe_description_rendered_html,
wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash AS creator_gravatar_hash,
wt.creator_ip_address,
wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,
@@ -138,9 +155,15 @@ async fn select_include_deleted(
wt.original_filename,
wt.file_size_bytes,
wt.file_checksum_sha2,
wt.private_bucket_hash,
wt.maybe_private_bucket_prefix,
wt.maybe_private_bucket_extension,
wt.public_bucket_hash,
wt.maybe_public_bucket_prefix,
wt.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
wt.cached_user_ratings_negative_count,
wt.cached_user_ratings_positive_count,
wt.cached_user_ratings_total_count,
@@ -152,6 +175,10 @@ async fn select_include_deleted(
wt.user_deleted_at,
wt.mod_deleted_at
FROM model_weights as wt
JOIN users
ON users.token = wt.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = wt.maybe_cover_image_media_file_token
WHERE
wt.token = ?
"#,
@@ -175,20 +202,29 @@ async fn select_without_deleted(
wt.title,
wt.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,
wt.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,
wt.maybe_thumbnail_token,
wt.description_markdown,
wt.description_rendered_html,
wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,
wt.maybe_description_markdown,
wt.maybe_description_rendered_html,
wt.creator_ip_address,
wt.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
wt.creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash AS creator_gravatar_hash,
wt.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,
wt.original_download_url,
wt.original_filename,
wt.file_size_bytes,
wt.file_checksum_sha2,
wt.private_bucket_hash,
wt.maybe_private_bucket_prefix,
wt.maybe_private_bucket_extension,
wt.public_bucket_hash,
wt.maybe_public_bucket_prefix,
wt.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
wt.cached_user_ratings_negative_count,
wt.cached_user_ratings_positive_count,
wt.cached_user_ratings_total_count,
@@ -200,6 +236,10 @@ async fn select_without_deleted(
wt.user_deleted_at,
wt.mod_deleted_at
FROM model_weights as wt
JOIN users
ON users.token = wt.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = wt.maybe_cover_image_media_file_token
WHERE
wt.token = ?
AND wt.user_deleted_at IS NULL
@@ -217,10 +257,14 @@ pub struct RawWeight {
pub title: String,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub maybe_description_markdown: Option<String>,
pub maybe_description_rendered_html: Option<String>,
pub creator_user_token: UserToken,
pub creator_username: String,
pub creator_display_name: String,
pub creator_gravatar_hash: String,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
@@ -228,9 +272,14 @@ pub struct RawWeight {
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_negative_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_total_count: u32,
@@ -65,9 +65,9 @@ SELECT
w.title,
vc.has_index_file,
w.private_bucket_hash,
w.maybe_private_bucket_prefix,
w.maybe_private_bucket_extension,
w.public_bucket_hash,
w.maybe_public_bucket_prefix,
w.maybe_public_bucket_extension,
w.created_at,
w.updated_at,
@@ -111,9 +111,9 @@ WHERE w.token = ?
weights_type: model.weights_type,
title: model.title,
has_index_file: i8_to_bool(model.has_index_file),
public_bucket_hash: model.private_bucket_hash,
maybe_public_bucket_prefix: model.maybe_private_bucket_prefix,
maybe_public_bucket_extension: model.maybe_private_bucket_extension,
public_bucket_hash: model.public_bucket_hash,
maybe_public_bucket_prefix: model.maybe_public_bucket_prefix,
maybe_public_bucket_extension: model.maybe_public_bucket_extension,
created_at: model.created_at,
updated_at: model.updated_at,
mod_deleted_at: model.mod_deleted_at,
@@ -129,9 +129,9 @@ struct InternalRecord {
has_index_file: i8,
private_bucket_hash: String,
maybe_private_bucket_prefix: Option<String>,
maybe_private_bucket_extension: Option<String>,
public_bucket_hash: String,
maybe_public_bucket_prefix: Option<String>,
maybe_public_bucket_extension: Option<String>,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
@@ -20,16 +20,18 @@ pub struct WeightsByTokensRecord {
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub creator_user_token: UserToken,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
@@ -71,14 +73,16 @@ async fn get_raw_weights_by_tokens(
mw.title,
mw.weights_type,
mw.weights_category,
mw.maybe_thumbnail_token,
users.token as creator_user_token,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash as creator_email_gravatar_hash,
mw.private_bucket_hash,
mw.maybe_private_bucket_prefix,
mw.maybe_private_bucket_extension,
mw.public_bucket_hash,
mw.maybe_public_bucket_prefix,
mw.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
mw.cached_user_ratings_negative_count,
mw.cached_user_ratings_positive_count,
mw.cached_user_ratings_total_count,
@@ -91,6 +95,8 @@ async fn get_raw_weights_by_tokens(
FROM model_weights as mw
JOIN users
ON users.token = mw.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = mw.maybe_cover_image_media_file_token
WHERE
mw.creator_set_visibility = "public"
AND mw.token IN (
@@ -104,14 +110,16 @@ async fn get_raw_weights_by_tokens(
mw.title,
mw.weights_type,
mw.weights_category,
mw.maybe_thumbnail_token,
users.token as creator_user_token,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash as creator_email_gravatar_hash,
mw.private_bucket_hash,
mw.maybe_private_bucket_prefix,
mw.maybe_private_bucket_extension,
mw.public_bucket_hash,
mw.maybe_public_bucket_prefix,
mw.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
mw.cached_user_ratings_negative_count,
mw.cached_user_ratings_positive_count,
mw.cached_user_ratings_total_count,
@@ -124,6 +132,8 @@ async fn get_raw_weights_by_tokens(
FROM model_weights as mw
JOIN users
ON users.token = mw.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = mw.maybe_cover_image_media_file_token
WHERE
mw.creator_set_visibility = "public"
AND mw.user_deleted_at IS NULL
@@ -157,34 +167,37 @@ fn token_predicate(tokens: &[ModelWeightToken]) -> String {
fn map_to_weights(dataset:Vec<RawWeightJoinUser>) -> Vec<WeightsByTokensRecord> {
let weights: Vec<WeightsByTokensRecord> = dataset
.into_iter()
.map(|dataset: RawWeightJoinUser| {
.map(|weight| {
WeightsByTokensRecord {
token: dataset.token,
title: dataset.title,
weights_type: dataset.weights_type,
weights_category: dataset.weights_category,
maybe_thumbnail_token: dataset.maybe_thumbnail_token,
token: weight.token,
title: weight.title,
weights_type: weight.weights_type,
weights_category: weight.weights_category,
creator_user_token: dataset.creator_user_token,
creator_username:dataset.creator_username,
creator_display_name:dataset.creator_display_name,
creator_email_gravatar_hash:dataset.creator_email_gravatar_hash,
creator_user_token: weight.creator_user_token,
creator_username: weight.creator_username,
creator_display_name: weight.creator_display_name,
creator_email_gravatar_hash: weight.creator_email_gravatar_hash,
private_bucket_hash: dataset.private_bucket_hash,
maybe_private_bucket_prefix: dataset.maybe_private_bucket_prefix,
maybe_private_bucket_extension: dataset.maybe_private_bucket_extension,
public_bucket_hash: weight.public_bucket_hash,
maybe_public_bucket_prefix: weight.maybe_public_bucket_prefix,
maybe_public_bucket_extension: weight.maybe_public_bucket_extension,
cached_user_ratings_negative_count: dataset.cached_user_ratings_negative_count,
cached_user_ratings_positive_count: dataset.cached_user_ratings_positive_count,
cached_user_ratings_total_count: dataset.cached_user_ratings_total_count,
maybe_cover_image_public_bucket_hash: weight.maybe_cover_image_public_bucket_hash,
maybe_cover_image_public_bucket_prefix: weight.maybe_cover_image_public_bucket_prefix,
maybe_cover_image_public_bucket_extension: weight.maybe_cover_image_public_bucket_extension,
maybe_cached_user_ratings_ratio: dataset.maybe_cached_user_ratings_ratio,
cached_user_ratings_last_updated_at: dataset.cached_user_ratings_last_updated_at,
cached_user_ratings_negative_count: weight.cached_user_ratings_negative_count,
cached_user_ratings_positive_count: weight.cached_user_ratings_positive_count,
cached_user_ratings_total_count: weight.cached_user_ratings_total_count,
created_at: dataset.created_at,
updated_at: dataset.updated_at,
user_deleted_at: dataset.user_deleted_at,
mod_deleted_at: dataset.mod_deleted_at,
maybe_cached_user_ratings_ratio: weight.maybe_cached_user_ratings_ratio,
cached_user_ratings_last_updated_at: weight.cached_user_ratings_last_updated_at,
created_at: weight.created_at,
updated_at: weight.updated_at,
user_deleted_at: weight.user_deleted_at,
mod_deleted_at: weight.mod_deleted_at,
}
}).collect();
@@ -200,17 +213,19 @@ fn map_to_weights(dataset:Vec<RawWeightJoinUser>) -> Vec<WeightsByTokensRecord>
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub creator_user_token: UserToken,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
@@ -242,14 +257,16 @@ impl FromRow<'_, MySqlRow> for RawWeightJoinUser {
weights_type: WeightsType::try_from_mysql_row(row, "weights_type")?,
weights_category: WeightsCategory::try_from_mysql_row(row, "weights_category")?,
title: row.try_get("title")?,
maybe_thumbnail_token: row.try_get("maybe_thumbnail_token")?,
creator_user_token: UserToken::new_from_str(row.try_get("creator_user_token")?),
creator_username: row.try_get("creator_username")?,
creator_display_name: row.try_get("creator_display_name")?,
creator_email_gravatar_hash: row.try_get("creator_email_gravatar_hash")?,
private_bucket_hash: row.try_get("private_bucket_hash")?,
maybe_private_bucket_prefix: row.try_get("maybe_private_bucket_prefix")?,
maybe_private_bucket_extension: row.try_get("maybe_private_bucket_extension")?,
public_bucket_hash: row.try_get("public_bucket_hash")?,
maybe_public_bucket_prefix: row.try_get("maybe_public_bucket_prefix")?,
maybe_public_bucket_extension: row.try_get("maybe_public_bucket_extension")?,
maybe_cover_image_public_bucket_hash: row.try_get("maybe_cover_image_public_bucket_hash")?,
maybe_cover_image_public_bucket_prefix: row.try_get("maybe_cover_image_public_bucket_prefix")?,
maybe_cover_image_public_bucket_extension: row.try_get("maybe_cover_image_public_bucket_extension")?,
cached_user_ratings_total_count: row.try_get("cached_user_ratings_total_count")?,
cached_user_ratings_positive_count: row.try_get("cached_user_ratings_positive_count")?,
cached_user_ratings_negative_count: row.try_get("cached_user_ratings_negative_count")?,
@@ -0,0 +1,371 @@
use chrono::{DateTime, Utc};
use sqlx::{FromRow, MySql, MySqlPool, QueryBuilder, Row};
use sqlx::mysql::MySqlRow;
use enums::by_table::model_weights::{
weights_category::WeightsCategory,
weights_types::WeightsType,
};
use enums::common::view_as::ViewAs;
use enums::common::visibility::Visibility;
use enums::traits::mysql_from_row::MySqlFromRow;
use errors::AnyhowResult;
use tokens::tokens::model_weights::ModelWeightToken;
use tokens::tokens::users::UserToken;
pub struct WeightsForUserListPage {
pub records: Vec<WeightsJoinUserRecord>,
pub sort_ascending: bool,
pub current_page: usize,
pub total_page_count: usize,
}
#[derive(Serialize)]
pub struct WeightsJoinUserRecord {
pub token: ModelWeightToken,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
// TODO(bt,2023-12-24): These aren't really appropriate for a list endpoint.
// Hopefully we don't break the frontend by omitting these.
//pub maybe_description_markdown: String,
//pub maybe_description_rendered_html: String,
pub creator_user_token: UserToken,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
pub maybe_cached_user_ratings_ratio: Option<f32>,
pub cached_user_ratings_last_updated_at: DateTime<Utc>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
}
pub struct ListWeightsForUserArgs<'a> {
pub creator_username: &'a str,
pub page_size: usize,
pub page_index: usize,
pub sort_ascending: bool,
pub view_as: ViewAs,
pub mysql_pool: &'a MySqlPool,
}
pub async fn list_weights_by_creator_username(args: ListWeightsForUserArgs<'_>) -> AnyhowResult<WeightsForUserListPage> {
let count_fields = select_total_count_field();
let mut count_query_builder = query_builder(
args.creator_username,
false,
0,
0,
args.sort_ascending,
count_fields.as_str(),
args.view_as,
);
let row_count_query = count_query_builder.build_query_scalar::<i64>();
let row_count_result = row_count_query.fetch_one(args.mysql_pool).await?;
/// Now fetch the actual results with all the fields
let result_fields = select_result_fields();
let mut query = query_builder(
args.creator_username,
true,
args.page_index,
args.page_size,
args.sort_ascending,
result_fields.as_str(),
args.view_as,
);
let query = query.build_query_as::<RawWeightJoinUser>();
let results = query.fetch_all(args.mysql_pool).await?;
let number_of_pages = (row_count_result / args.page_size as i64) as usize;
let weights_records: Vec<WeightsJoinUserRecord> = map_to_weights(results).await;
Ok(WeightsForUserListPage {
records: weights_records,
sort_ascending: args.sort_ascending,
current_page: args.page_index,
total_page_count: number_of_pages,
})
}
fn select_total_count_field() -> String {
r#"
COUNT(mw.id) AS total_count
"#
.to_string()
}
fn select_result_fields() -> String {
r#"
mw.token,
mw.title,
mw.weights_type,
mw.weights_category,
u.token as creator_user_token,
u.username as creator_username,
u.display_name as creator_display_name,
u.email_gravatar_hash as creator_email_gravatar_hash,
mw.creator_ip_address,
mw.creator_set_visibility,
mw.maybe_last_update_user_token,
mw.original_download_url,
mw.original_filename,
mw.file_size_bytes,
mw.file_checksum_sha2,
mw.public_bucket_hash,
mw.maybe_public_bucket_prefix,
mw.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
mw.cached_user_ratings_negative_count,
mw.cached_user_ratings_positive_count,
mw.cached_user_ratings_total_count,
mw.maybe_cached_user_ratings_ratio,
mw.cached_user_ratings_last_updated_at,
mw.version,
mw.created_at,
mw.updated_at,
mw.user_deleted_at,
mw.mod_deleted_at
"#.to_string()
}
fn query_builder<'a>(
username: &'a str,
enforce_limits: bool,
page_index: usize,
page_size: usize,
sort_ascending: bool,
select_fields: &'a str,
view_as: ViewAs,
) -> QueryBuilder<'a, MySql> {
// NB: Query cannot be statically checked by sqlx
let mut query_builder: QueryBuilder<MySql> = QueryBuilder::new(
format!(r#"
SELECT
{select_fields}
FROM model_weights as mw
JOIN users as u
ON u.token = mw.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = mw.maybe_cover_image_media_file_token
"#
));
query_builder.push(" WHERE u.username = ");
query_builder.push_bind(username);
match view_as {
ViewAs::Author => {
query_builder.push(" AND mw.user_deleted_at IS NULL AND mw.mod_deleted_at IS NULL ");
}
ViewAs::Moderator => {}
ViewAs::AnotherUser => {
query_builder.push(" AND mw.user_deleted_at IS NULL AND mw.mod_deleted_at IS NULL ");
query_builder.push(" AND mw.creator_set_visibility = ");
query_builder.push_bind(Visibility::Public.to_str());
}
}
if sort_ascending {
query_builder.push(" ORDER BY mw.created_at ASC ");
} else {
query_builder.push(" ORDER BY mw.created_at DESC ");
}
if enforce_limits {
let offset = page_index * page_size;
query_builder.push(format!(" LIMIT {page_size} OFFSET {offset} "));
}
query_builder
}
async fn map_to_weights(dataset:Vec<RawWeightJoinUser>) -> Vec<WeightsJoinUserRecord> {
let weights: Vec<WeightsJoinUserRecord> = dataset
.into_iter()
.map(|weight: RawWeightJoinUser| {
WeightsJoinUserRecord {
token: weight.token,
title: weight.title,
weights_type: weight.weights_type,
weights_category: weight.weights_category,
//maybe_description_markdown: weight.maybe_description_markdown,
//maybe_description_rendered_html: weight.maybe_description_rendered_html,
creator_user_token: weight.creator_user_token,
creator_ip_address: weight.creator_ip_address,
creator_set_visibility: weight.creator_set_visibility,
maybe_last_update_user_token: weight.maybe_last_update_user_token,
original_download_url: weight.original_download_url,
original_filename: weight.original_filename,
file_size_bytes: weight.file_size_bytes,
file_checksum_sha2: weight.file_checksum_sha2,
public_bucket_hash: weight.public_bucket_hash,
maybe_public_bucket_prefix: weight.maybe_public_bucket_prefix,
maybe_public_bucket_extension: weight.maybe_public_bucket_extension,
maybe_cover_image_public_bucket_hash: weight.maybe_cover_image_public_bucket_hash,
maybe_cover_image_public_bucket_prefix: weight.maybe_cover_image_public_bucket_prefix,
maybe_cover_image_public_bucket_extension: weight.maybe_cover_image_public_bucket_extension,
cached_user_ratings_negative_count: weight.cached_user_ratings_negative_count,
cached_user_ratings_positive_count: weight.cached_user_ratings_positive_count,
cached_user_ratings_total_count: weight.cached_user_ratings_total_count,
maybe_cached_user_ratings_ratio: weight.maybe_cached_user_ratings_ratio,
cached_user_ratings_last_updated_at: weight.cached_user_ratings_last_updated_at,
version: weight.version,
created_at: weight.created_at,
updated_at: weight.updated_at,
user_deleted_at: weight.user_deleted_at,
mod_deleted_at: weight.mod_deleted_at,
creator_username: weight.creator_username,
creator_display_name: weight.creator_display_name,
creator_email_gravatar_hash: weight.creator_email_gravatar_hash,
}
}).collect();
weights
}
pub struct RawWeightJoinUser {
pub token: ModelWeightToken,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
// TODO(bt,2023-12-24): These aren't really appropriate for a list endpoint.
// Hopefully we don't break the frontend by omitting these.
//pub description_markdown: String,
//pub description_rendered_html: String,
pub creator_user_token: UserToken,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
pub maybe_cached_user_ratings_ratio: Option<f32>,
pub cached_user_ratings_last_updated_at: DateTime<Utc>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
}
impl FromRow<'_, MySqlRow> for RawWeightJoinUser {
fn from_row(row: &MySqlRow) -> Result<Self, sqlx::Error> {
Ok(Self {
token: row.try_get("token")?,
weights_type: row.try_get("weights_type")?,
weights_category: row.try_get("weights_category")?,
title: row.try_get("title")?,
// TODO(bt,2023-12-24): These aren't really appropriate for a list endpoint.
// Hopefully we don't break the frontend by omitting these.
//maybe_description_markdown: row.try_get("maybe_description_markdown")?,
//maybe_description_rendered_html: row.try_get("maybe_description_rendered_html")?,
creator_user_token: row.try_get("creator_user_token")?,
creator_ip_address: row.try_get("creator_ip_address")?,
creator_set_visibility: Visibility::try_from_mysql_row(row, "creator_set_visibility")?,
maybe_last_update_user_token: row.try_get("maybe_last_update_user_token")?,
original_download_url: row.try_get("original_download_url")?,
original_filename: row.try_get("original_filename")?,
file_size_bytes: row.try_get("file_size_bytes")?,
file_checksum_sha2: row.try_get("file_checksum_sha2")?,
public_bucket_hash: row.try_get("public_bucket_hash")?,
maybe_public_bucket_prefix: row.try_get("maybe_public_bucket_prefix")?,
maybe_public_bucket_extension: row.try_get("maybe_public_bucket_extension")?,
maybe_cover_image_public_bucket_hash: row.try_get("maybe_cover_image_public_bucket_hash")?,
maybe_cover_image_public_bucket_prefix: row.try_get("maybe_cover_image_public_bucket_prefix")?,
maybe_cover_image_public_bucket_extension: row.try_get("maybe_cover_image_public_bucket_extension")?,
cached_user_ratings_total_count: row.try_get("cached_user_ratings_total_count")?,
cached_user_ratings_positive_count: row.try_get("cached_user_ratings_positive_count")?,
cached_user_ratings_negative_count: row.try_get("cached_user_ratings_negative_count")?,
maybe_cached_user_ratings_ratio: row.try_get("maybe_cached_user_ratings_ratio")?,
cached_user_ratings_last_updated_at: row.try_get("cached_user_ratings_last_updated_at")?,
version: row.try_get("version")?,
created_at: row.try_get("created_at")?,
updated_at: row.try_get("updated_at")?,
user_deleted_at: row.try_get("user_deleted_at")?,
mod_deleted_at: row.try_get("mod_deleted_at")?,
creator_username: row.try_get("creator_username")?,
creator_display_name: row.try_get("creator_display_name")?,
creator_email_gravatar_hash: row.try_get("creator_email_gravatar_hash")?,
})
}
}
@@ -2,20 +2,18 @@
#![forbid(unused_mut)]
#![forbid(unused_variables)]
use chrono::{ DateTime, Utc };
use chrono::{DateTime, Utc};
use sqlx::MySqlPool;
use config::shared_constants::DEFAULT_MYSQL_QUERY_RESULT_PAGE_SIZE;
use enums::by_table::model_weights::{
weights_category::WeightsCategory,
weights_types::WeightsType,
};
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::users::UserToken;
use tokens::tokens::model_weights::ModelWeightToken;
use enums::by_table::model_weights::{
weights_types::WeightsType,
weights_category::WeightsCategory,
};
use tokens::tokens::users::UserToken;
#[derive(Serialize)]
pub struct WeightsPage {
@@ -38,10 +36,8 @@ pub struct WeightJoinUser {
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub maybe_description_markdown: Option<String>,
pub maybe_description_rendered_html: Option<String>,
pub creator_user_token: UserToken,
pub creator_ip_address: String,
@@ -55,10 +51,13 @@ pub struct WeightJoinUser {
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
@@ -163,13 +162,15 @@ impl ListWeightsQueryBuilder {
mysql_pool: &MySqlPool
) -> AnyhowResult<Vec<RawWeightJoinUser>> {
let query = self.build_query_string();
//println!("HERE! query: {}", query);
//panic!("query: {:?}", query);
let mut query = sqlx::query_as::<_, RawWeightJoinUser>(&query);
// NB: The following bindings must match the order of the query builder !!
if let Some(offset) = self.offset {
query = query.bind(offset);
}
if let Some(username) = self.scope_creator_username.as_deref() {
query = query.bind(username);
}
@@ -184,9 +185,6 @@ impl ListWeightsQueryBuilder {
query = query.bind(self.limit);
if let Some(offset) = self.offset {
query = query.bind(offset);
}
let mut results = query.fetch_all(mysql_pool).await?;
@@ -219,9 +217,8 @@ impl ListWeightsQueryBuilder {
record.weights_category.as_str()
).unwrap(),
title: record.title,
maybe_thumbnail_token: record.maybe_thumbnail_token,
description_markdown: record.description_markdown,
description_rendered_html: record.description_rendered_html,
maybe_description_markdown: record.maybe_description_markdown,
maybe_description_rendered_html: record.maybe_description_rendered_html,
creator_user_token: UserToken::new_from_str(&record.creator_user_token),
creator_ip_address: record.creator_ip_address,
creator_set_visibility: Visibility::from_str(
@@ -234,9 +231,12 @@ impl ListWeightsQueryBuilder {
original_filename: record.original_filename,
file_size_bytes: record.file_size_bytes,
file_checksum_sha2: record.file_checksum_sha2,
private_bucket_hash: record.private_bucket_hash,
maybe_private_bucket_prefix: record.maybe_private_bucket_prefix,
maybe_private_bucket_extension: record.maybe_private_bucket_extension,
public_bucket_hash: record.public_bucket_hash,
maybe_public_bucket_prefix: record.maybe_public_bucket_prefix,
maybe_public_bucket_extension: record.maybe_public_bucket_extension,
maybe_cover_image_public_bucket_hash: record.maybe_cover_image_public_bucket_hash,
maybe_cover_image_public_bucket_prefix: record.maybe_cover_image_public_bucket_prefix,
maybe_cover_image_public_bucket_extension: record.maybe_cover_image_public_bucket_extension,
cached_user_ratings_total_count: record.cached_user_ratings_total_count,
cached_user_ratings_positive_count: record.cached_user_ratings_positive_count,
cached_user_ratings_negative_count: record.cached_user_ratings_negative_count,
@@ -271,9 +271,8 @@ impl ListWeightsQueryBuilder {
model_weights.weights_type,
model_weights.weights_category,
model_weights.title,
model_weights.maybe_thumbnail_token,
model_weights.description_markdown,
model_weights.description_rendered_html,
model_weights.maybe_description_markdown,
model_weights.maybe_description_rendered_html,
model_weights.creator_user_token,
model_weights.creator_ip_address,
model_weights.creator_set_visibility,
@@ -282,9 +281,12 @@ impl ListWeightsQueryBuilder {
model_weights.original_filename,
model_weights.file_size_bytes,
model_weights.file_checksum_sha2,
model_weights.private_bucket_hash,
model_weights.maybe_private_bucket_prefix,
model_weights.maybe_private_bucket_extension,
model_weights.public_bucket_hash,
model_weights.maybe_public_bucket_prefix,
model_weights.maybe_public_bucket_extension,
cover_image.public_bucket_directory_hash as maybe_cover_image_public_bucket_hash,
cover_image.maybe_public_bucket_prefix as maybe_cover_image_public_bucket_prefix,
cover_image.maybe_public_bucket_extension as maybe_cover_image_public_bucket_extension,
model_weights.cached_user_ratings_total_count,
model_weights.cached_user_ratings_positive_count,
model_weights.cached_user_ratings_negative_count,
@@ -301,6 +303,8 @@ impl ListWeightsQueryBuilder {
FROM model_weights
JOIN users
ON users.token = model_weights.creator_user_token
LEFT OUTER JOIN media_files as cover_image
ON cover_image.token = model_weights.maybe_cover_image_media_file_token
"#.to_string();
query.push_str(&self.build_predicates());
@@ -326,18 +330,18 @@ impl ListWeightsQueryBuilder {
if sort_ascending {
if self.cursor_is_reversed {
// NB: We're searching backwards.
query.push_str(" model_weights.id");
query.push_str(" model_weights.id < ? ");
sort_ascending = !sort_ascending;
} else {
query.push_str(" model_weights.id");
query.push_str(" model_weights.id > ? ");
}
} else {
if self.cursor_is_reversed {
// NB: We're searching backwards.
query.push_str(" model_weights.id");
query.push_str(" model_weights.id > ? ");
sort_ascending = !sort_ascending;
} else {
query.push_str(" model_weights.id");
query.push_str(" model_weights.id < ? ");
}
}
}
@@ -406,10 +410,6 @@ impl ListWeightsQueryBuilder {
query.push_str(" LIMIT ?");
}
if let Some(_) = self.offset {
query.push_str(" OFFSET ?");
}
query
}
}
@@ -425,10 +425,8 @@ struct RawWeightJoinUser {
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub maybe_description_markdown: Option<String>,
pub maybe_description_rendered_html: Option<String>,
pub creator_user_token: String,
pub creator_ip_address: String,
@@ -442,10 +440,13 @@ struct RawWeightJoinUser {
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub public_bucket_hash: String,
pub maybe_public_bucket_prefix: Option<String>,
pub maybe_public_bucket_extension: Option<String>,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub maybe_cover_image_public_bucket_hash: Option<String>,
pub maybe_cover_image_public_bucket_prefix: Option<String>,
pub maybe_cover_image_public_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
@@ -468,12 +469,13 @@ struct RawWeightJoinUser {
#[cfg(test)]
mod tests {
use crate::queries::model_weights::list_weights_query_builder::ListWeightsQueryBuilder;
use enums::by_table::model_weights::{
weights_types::WeightsType,
weights_category::WeightsCategory,
weights_types::WeightsType,
};
use crate::queries::model_weights::list::list_weights_query_builder::ListWeightsQueryBuilder;
#[test]
fn predicates_without_scoping() {
let query_builder = ListWeightsQueryBuilder::new();
@@ -1 +1,4 @@
pub mod list_model_weights_for_voice_conversion;
pub mod list_model_weights_for_voice_conversion;
pub mod list_weights_by_tokens;
pub mod list_weights_by_user;
pub mod list_weights_query_builder;
@@ -1,276 +0,0 @@
use chrono::{DateTime, Utc};
use sqlx::{Acquire, MySqlConnection, MySqlPool};
use enums::by_table::model_weights::{
weights_category::WeightsCategory,
weights_types::WeightsType,
};
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::model_weights::ModelWeightToken;
use tokens::tokens::users::UserToken;
#[derive(Serialize)]
pub struct WeightsJoinUserRecord {
pub token: ModelWeightToken,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub creator_user_token: UserToken,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
pub maybe_cached_user_ratings_ratio: Option<f32>,
pub cached_user_ratings_last_updated_at: DateTime<Utc>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
}
pub async fn list_weights_by_creator_username(
mysql_pool: &MySqlPool,
creator_username: &str,
can_see_deleted: bool
) -> AnyhowResult<Vec<WeightsJoinUserRecord>> {
let mut connection = mysql_pool.acquire().await?;
let raw_weights: Vec<RawWeightJoinUser> = get_raw_weights_by_creator_username(&mut connection, creator_username, can_see_deleted).await?;
let weights_records: Vec<WeightsJoinUserRecord> = map_to_weights(raw_weights).await;
Ok(weights_records)
}
async fn get_raw_weights_by_creator_username(
connection: &mut MySqlConnection,
creator_username: &str,
can_see_deleted: bool
) -> AnyhowResult<Vec<RawWeightJoinUser>> {
let connection = connection.acquire().await?;
if can_see_deleted {
let raw_weights: Vec<RawWeightJoinUser> = sqlx::query_as!(
RawWeightJoinUser,
r#"
SELECT
mw.token as `token: tokens::tokens::model_weights::ModelWeightToken`,
mw.title,
mw.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,
mw.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,
mw.maybe_thumbnail_token,
mw.description_markdown,
mw.description_rendered_html,
users.token as `creator_user_token: tokens::tokens::users::UserToken`,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash as creator_email_gravatar_hash,
mw.creator_ip_address,
mw.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
mw.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,
mw.original_download_url,
mw.original_filename,
mw.file_size_bytes,
mw.file_checksum_sha2,
mw.private_bucket_hash,
mw.maybe_private_bucket_prefix,
mw.maybe_private_bucket_extension,
mw.cached_user_ratings_negative_count,
mw.cached_user_ratings_positive_count,
mw.cached_user_ratings_total_count,
mw.maybe_cached_user_ratings_ratio,
mw.cached_user_ratings_last_updated_at,
mw.version,
mw.created_at,
mw.updated_at,
mw.user_deleted_at,
mw.mod_deleted_at
FROM model_weights as mw
JOIN users
ON users.token = mw.creator_user_token
WHERE
users.username = ?
"#,
creator_username).fetch_all(connection)
.await?;
Ok(raw_weights)
} else {
let raw_weights: Vec<RawWeightJoinUser> = sqlx::query_as!(
RawWeightJoinUser,
r#"
SELECT
mw.token as `token: tokens::tokens::model_weights::ModelWeightToken`,
mw.title,
mw.weights_type as `weights_type: enums::by_table::model_weights::weights_types::WeightsType`,
mw.weights_category as `weights_category: enums::by_table::model_weights::weights_category::WeightsCategory`,
mw.maybe_thumbnail_token,
mw.description_markdown,
mw.description_rendered_html,
users.token as `creator_user_token: tokens::tokens::users::UserToken`,
users.username as creator_username,
users.display_name as creator_display_name,
users.email_gravatar_hash as creator_email_gravatar_hash,
mw.creator_ip_address,
mw.creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
mw.maybe_last_update_user_token as `maybe_last_update_user_token: tokens::tokens::users::UserToken`,
mw.original_download_url,
mw.original_filename,
mw.file_size_bytes,
mw.file_checksum_sha2,
mw.private_bucket_hash,
mw.maybe_private_bucket_prefix,
mw.maybe_private_bucket_extension,
mw.cached_user_ratings_negative_count,
mw.cached_user_ratings_positive_count,
mw.cached_user_ratings_total_count,
mw.maybe_cached_user_ratings_ratio,
mw.cached_user_ratings_last_updated_at,
mw.version,
mw.created_at,
mw.updated_at,
mw.user_deleted_at,
mw.mod_deleted_at
FROM model_weights as mw
JOIN users
ON users.token = mw.creator_user_token
WHERE
users.username = ?
AND mw.user_deleted_at IS NULL
AND mw.mod_deleted_at IS NULL
"#,
creator_username).fetch_all(connection).await?;
Ok(raw_weights)
}
}
async fn map_to_weights(dataset:Vec<RawWeightJoinUser>) -> Vec<WeightsJoinUserRecord> {
let weights: Vec<WeightsJoinUserRecord> = dataset
.into_iter()
.map(|dataset: RawWeightJoinUser| {
WeightsJoinUserRecord {
token: dataset.token,
title: dataset.title,
weights_type: dataset.weights_type,
weights_category: dataset.weights_category,
maybe_thumbnail_token: dataset.maybe_thumbnail_token,
description_markdown: dataset.description_markdown,
description_rendered_html: dataset.description_rendered_html,
creator_user_token: dataset.creator_user_token,
creator_ip_address: dataset.creator_ip_address,
creator_set_visibility: dataset.creator_set_visibility,
maybe_last_update_user_token: dataset.maybe_last_update_user_token,
original_download_url: dataset.original_download_url,
original_filename: dataset.original_filename,
file_size_bytes: dataset.file_size_bytes,
file_checksum_sha2: dataset.file_checksum_sha2,
private_bucket_hash: dataset.private_bucket_hash,
maybe_private_bucket_prefix: dataset.maybe_private_bucket_prefix,
maybe_private_bucket_extension: dataset.maybe_private_bucket_extension,
cached_user_ratings_negative_count: dataset.cached_user_ratings_negative_count,
cached_user_ratings_positive_count: dataset.cached_user_ratings_positive_count,
cached_user_ratings_total_count: dataset.cached_user_ratings_total_count,
maybe_cached_user_ratings_ratio: dataset.maybe_cached_user_ratings_ratio,
cached_user_ratings_last_updated_at: dataset.cached_user_ratings_last_updated_at,
version: dataset.version,
created_at: dataset.created_at,
updated_at: dataset.updated_at,
user_deleted_at: dataset.user_deleted_at,
mod_deleted_at: dataset.mod_deleted_at,
creator_username:dataset.creator_username,
creator_display_name:dataset.creator_display_name,
creator_email_gravatar_hash:dataset.creator_email_gravatar_hash
}
}).collect();
weights
}
#[derive(Serialize)]
pub struct RawWeightJoinUser {
pub token: ModelWeightToken,
pub weights_type: WeightsType,
pub weights_category: WeightsCategory,
pub title: String,
pub maybe_thumbnail_token: Option<String>,
pub description_markdown: String,
pub description_rendered_html: String,
pub creator_user_token: UserToken,
pub creator_ip_address: String,
pub creator_set_visibility: Visibility,
pub maybe_last_update_user_token: Option<UserToken>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub file_checksum_sha2: String,
pub private_bucket_hash: String,
pub maybe_private_bucket_prefix: Option<String>,
pub maybe_private_bucket_extension: Option<String>,
pub cached_user_ratings_total_count: u32,
pub cached_user_ratings_positive_count: u32,
pub cached_user_ratings_negative_count: u32,
pub maybe_cached_user_ratings_ratio: Option<f32>,
pub cached_user_ratings_last_updated_at: DateTime<Utc>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
pub creator_username: String,
pub creator_display_name: String,
pub creator_email_gravatar_hash: String,
}
@@ -32,8 +32,7 @@ pub async fn upsert_model_weight_from_voice_conversion_model(
upsert_model_weights_record(record, &model_weight_token, copied_data, &mut transaction).await?;
upsert_model_weights_extension_record(record, &model_weight_token, &mut transaction).await?;
// TODO: Don't update source record with new token *unless* we're backfilling the same database.
// update_original_record(record, &model_weight_token, &mut transaction).await?;
update_original_record(record, &model_weight_token, &mut transaction).await?;
transaction.commit().await?;
@@ -64,9 +63,8 @@ pub async fn upsert_model_weights_record(
// NB: Not setting a few fields (for now)
// maybe_last_update_user_token - seems like bad design
// TODO(bt): OBVIOUSLY COPY THE BUCKET FILES !
// TODO(bt): file checksum
// TODO(bt): rename maybe_private_bucket_extension to maybe_private_bucket_suffix (!!!)
// TODO(bt): rename maybe_public_bucket_extension to maybe_public_bucket_suffix (!!!)
// TODO(bt): do we need model_weights.ip_address_last_update without audit logs?
// TODO(bt): rename creator_ip_address to ip_address_creation (and add ip_address_last_update)
// TODO(bt): Check model_weights column integer types - signed vs unsigned
@@ -78,11 +76,9 @@ SET
weights_type = ?,
weights_category = ?,
title = ?,
maybe_thumbnail_token = NULL,
maybe_avatar_media_file_token = NULL,
maybe_cover_media_file_token = NULL,
description_markdown = ?,
description_rendered_html = ?,
maybe_cover_image_media_file_token = NULL,
maybe_description_markdown = ?,
maybe_description_rendered_html = ?,
creator_user_token = ?,
creator_ip_address = ?,
@@ -93,9 +89,9 @@ SET
file_size_bytes = ?,
file_checksum_sha2 = ?,
private_bucket_hash = ?,
maybe_private_bucket_prefix = ?,
maybe_private_bucket_extension = ?,
public_bucket_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
cached_user_ratings_total_count = 0,
cached_user_ratings_positive_count = 0,
cached_user_ratings_negative_count = 0,
@@ -113,11 +109,9 @@ ON DUPLICATE KEY UPDATE
weights_type = ?,
weights_category = ?,
title = ?,
maybe_thumbnail_token = NULL,
maybe_avatar_media_file_token = NULL,
maybe_cover_media_file_token = NULL,
description_markdown = ?,
description_rendered_html = ?,
maybe_cover_image_media_file_token = NULL,
maybe_description_markdown = ?,
maybe_description_rendered_html = ?,
creator_user_token = ?,
creator_ip_address = ?,
creator_set_visibility = ?,
@@ -126,9 +120,9 @@ ON DUPLICATE KEY UPDATE
original_filename = ?,
file_size_bytes = ?,
file_checksum_sha2 = ?,
private_bucket_hash = ?,
maybe_private_bucket_prefix = ?,
maybe_private_bucket_extension = ?,
public_bucket_hash = ?,
maybe_public_bucket_prefix = ?,
maybe_public_bucket_extension = ?,
cached_user_ratings_total_count = 0,
cached_user_ratings_positive_count = 0,
cached_user_ratings_negative_count = 0,
@@ -227,6 +221,18 @@ pub async fn update_original_record(
model_weight_token: &ModelWeightToken,
transaction: &mut Transaction<'_, MySql>
) -> AnyhowResult<()> {
let query = sqlx::query!(
r#"
UPDATE voice_conversion_models
SET
maybe_migration_new_model_weights_token = ?
WHERE token = ?
"#,
model_weight_token,
record.token,
);
let _r = query.execute(&mut **transaction).await?;
Ok(())
}
@@ -1,13 +1,9 @@
pub mod create;
pub mod create_weight;
pub mod delete_weights;
pub mod edit;
pub mod get_weight;
pub mod inference;
pub mod list;
pub mod list_weights_by_tokens;
pub mod list_weights_by_user;
pub mod list_weights_query_builder;
pub mod migration;
pub mod model_weight_info_lite;
pub mod tests;
pub mod update_weight;
@@ -4,9 +4,8 @@ CREATE TABLE model_weights (
weights_type VARCHAR(32) NOT NULL,
weights_category VARCHAR(32) NOT NULL,
title VARCHAR(255) NOT NULL,
maybe_thumbnail_token VARCHAR(32) DEFAULT NULL,
description_markdown TEXT NOT NULL,
description_rendered_html TEXT NOT NULL,
maybe_description_markdown TEXT DEFAULT NULL,
maybe_description_rendered_html TEXT DEFAULT NULL,
creator_user_token VARCHAR(32) NOT NULL,
creator_ip_address VARCHAR(40) NOT NULL,
creator_set_visibility ENUM(
@@ -19,9 +18,9 @@ CREATE TABLE model_weights (
original_filename VARCHAR(255) DEFAULT NULL,
file_size_bytes INT(10) NOT NULL DEFAULT 0,
file_checksum_sha2 CHAR(64) NOT NULL,
private_bucket_hash VARCHAR(32) NOT NULL,
maybe_private_bucket_prefix VARCHAR(16) DEFAULT NULL,
maybe_private_bucket_extension VARCHAR(16) DEFAULT NULL,
public_bucket_hash VARCHAR(32) NOT NULL,
maybe_public_bucket_prefix VARCHAR(16) DEFAULT NULL,
maybe_public_bucket_extension VARCHAR(16) DEFAULT NULL,
cached_user_ratings_total_count INT(10) UNSIGNED NOT NULL DEFAULT 0,
cached_user_ratings_positive_count INT(10) UNSIGNED NOT NULL DEFAULT 0,
cached_user_ratings_negative_count INT(10) UNSIGNED NOT NULL DEFAULT 0,
@@ -1,10 +1,11 @@
#[ignore]
#[cfg(test)]
mod tests {
use anyhow::Ok;
use rand::Rng;
use serial_test::serial;
use sqlx::MySqlPool;
use sqlx::mysql::MySqlPoolOptions;
use sqlx::MySqlPool;
use tokio;
use config::shared_constants::DEFAULT_MYSQL_CONNECTION_STRING;
@@ -17,8 +18,8 @@ mod tests {
use enums::common::visibility::Visibility;
use tokens::tokens::{model_weights::ModelWeightToken, users::UserToken};
use crate::queries::model_weights::create_weight::create_weight;
use crate::queries::model_weights::create_weight::CreateModelWeightsArgs;
use crate::queries::model_weights::create::create_weight::create_weight;
use crate::queries::model_weights::create::create_weight::CreateModelWeightsArgs;
use crate::queries::model_weights::delete_weights::{
delete_weights_as_mod,
delete_weights_as_user,
@@ -26,8 +27,8 @@ mod tests {
undelete_weights_as_user,
};
use crate::queries::model_weights::get_weight::get_weight_by_token;
use crate::queries::model_weights::list_weights_by_user::list_weights_by_creator_username;
use crate::queries::model_weights::list_weights_query_builder::ListWeightsQueryBuilder;
use crate::queries::model_weights::list::list_weights_by_user::{list_weights_by_creator_username, ListWeightsForUserArgs};
use crate::queries::model_weights::list::list_weights_query_builder::ListWeightsQueryBuilder;
use crate::queries::users::user::get_user_token_by_username::get_user_token_by_username;
async fn setup() -> sqlx::Pool<sqlx::MySql> {
@@ -65,9 +66,8 @@ mod tests {
weights_type: WeightsType::RvcV2, // replace with actual WeightsType
weights_category: WeightsCategory::VoiceConversion, // replace with actual WeightsCategory
title: "Title 1".to_string(),
maybe_thumbnail_token: Some("Thumbnail 1".to_string()),
description_markdown: "Description 1".to_string(),
description_rendered_html: "<p>Description 1</p>".to_string(),
maybe_description_markdown: Some("Description 1".to_string()),
maybe_description_rendered_html: Some("<p>Description 1</p>".to_string()),
creator_user_token: Some(&creator_token1), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -76,9 +76,9 @@ mod tests {
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -98,9 +98,8 @@ mod tests {
assert_eq!(result.title, "Title 1".to_string());
assert_eq!(result.weights_type, WeightsType::RvcV2);
assert_eq!(result.weights_category, WeightsCategory::VoiceConversion);
assert_eq!(result.maybe_thumbnail_token, Some("Thumbnail 1".to_string()));
assert_eq!(result.description_markdown, "Description 1".to_string());
assert_eq!(result.description_rendered_html, "<p>Description 1</p>".to_string());
assert_eq!(result.maybe_description_markdown, Some("Description 1".to_string()));
assert_eq!(result.maybe_description_rendered_html, Some("<p>Description 1</p>".to_string()));
assert_eq!(result.creator_user_token, creator_token1);
assert_eq!(result.creator_ip_address, "192.168.1.1".to_string());
assert_eq!(result.creator_set_visibility, Visibility::Public);
@@ -112,9 +111,9 @@ mod tests {
assert_eq!(result.original_filename, Some("filename1.txt".to_string()));
assert_eq!(result.file_size_bytes, 1024);
assert_eq!(result.file_checksum_sha2, "checksum1".to_string());
assert_eq!(result.private_bucket_hash, "bucket_hash1".to_string());
assert_eq!(result.maybe_private_bucket_prefix, Some("_fake".to_string()));
assert_eq!(result.maybe_private_bucket_extension, Some("rvc".to_string()));
assert_eq!(result.public_bucket_hash, "bucket_hash1".to_string());
assert_eq!(result.maybe_public_bucket_prefix, Some("_fake".to_string()));
assert_eq!(result.maybe_public_bucket_extension, Some("rvc".to_string()));
assert_eq!(result.cached_user_ratings_total_count, 10);
assert_eq!(result.cached_user_ratings_positive_count, 9);
assert_eq!(result.cached_user_ratings_negative_count, 1);
@@ -140,9 +139,8 @@ mod tests {
weights_type: WeightsType::RvcV2, // replace with actual WeightsType
weights_category: WeightsCategory::VoiceConversion, // replace with actual WeightsCategory
title: "Title 1".to_string(),
maybe_thumbnail_token: Some("Thumbnail 1".to_string()),
description_markdown: "Description 1".to_string(),
description_rendered_html: "<p>Description 1</p>".to_string(),
maybe_description_markdown: Some("Description 1".to_string()),
maybe_description_rendered_html: Some("<p>Description 1</p>".to_string()),
creator_user_token: Some(&creator_token1), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -151,9 +149,9 @@ mod tests {
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -217,9 +215,8 @@ mod tests {
weights_type: WeightsType::RvcV2, // replace with actual WeightsType
weights_category: WeightsCategory::VoiceConversion, // replace with actual WeightsCategory
title: "Title 1".to_string(),
maybe_thumbnail_token: Some("Thumbnail 1".to_string()),
description_markdown: "Description 1".to_string(),
description_rendered_html: "<p>Description 1</p>".to_string(),
maybe_description_markdown: Some("Description 1".to_string()),
maybe_description_rendered_html: Some("<p>Description 1</p>".to_string()),
creator_user_token: Some(&creator_token1), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -228,9 +225,9 @@ mod tests {
original_filename: Some("filename1.txt".to_string()),
file_size_bytes: 1024,
file_checksum_sha2: "checksum1".to_string(),
private_bucket_hash: "bucket_hash1".to_string(),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: "bucket_hash1".to_string(),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -290,16 +287,19 @@ mod tests {
&creator_username
).await?;
let weights_by_username = list_weights_by_creator_username(
&pool,
&creator_username,
can_see_deleted
).await?;
for weight in weights_by_username.iter() {
let weights_by_username = list_weights_by_creator_username(ListWeightsForUserArgs {
creator_username: &creator_username,
page_size: 0,
page_index: 0,
sort_ascending: false,
can_see_deleted,
mysql_pool: &pool,
}).await?;
for weight in weights_by_username.records.iter() {
// print weight
println!("weight: {:?}", weight.creator_username);
}
assert_eq!(weights_by_username.len(), 5);
assert_eq!(weights_by_username.records.len(), 5);
Ok(())
}
@@ -326,12 +326,11 @@ mod tests {
let args = CreateModelWeightsArgs {
token: &model_weight_token, // replace with actual ModelWeightToken
weights_type: weights_type, // replace with actual WeightsType
weights_category: weights_category, // replace with actual WeightsCategory
weights_type, // replace with actual WeightsType
weights_category, // replace with actual WeightsCategory
title: format!("Title {}", i),
maybe_thumbnail_token: Some(format!("Thumbnail {}", i)),
description_markdown: format!("Description {}", i),
description_rendered_html: format!("<p>Description {}</p>", i),
maybe_description_markdown: Some(format!("Description {}", i)),
maybe_description_rendered_html: Some(format!("<p>Description {}</p>", i)),
creator_user_token: Some(&creator_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -340,9 +339,9 @@ mod tests {
original_filename: Some(format!("filename {}.txt", i)),
file_size_bytes: 1024,
file_checksum_sha2: format!("checksum{}", i),
private_bucket_hash: format!("bucket_hash{}", i),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: format!("bucket_hash{}", i),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -569,12 +568,11 @@ mod tests {
let args = CreateModelWeightsArgs {
token: &model_weight_token, // replace with actual ModelWeightToken
weights_type: weights_type, // replace with actual WeightsType
weights_category: weights_category, // replace with actual WeightsCategory
weights_type, // replace with actual WeightsType
weights_category, // replace with actual WeightsCategory
title: format!("Title {}", i),
maybe_thumbnail_token: Some(format!("Thumbnail {}", i)),
description_markdown: format!("Description {}", i),
description_rendered_html: format!("<p>Description {}</p>", i),
maybe_description_markdown: Some(format!("Description {}", i)),
maybe_description_rendered_html: Some(format!("<p>Description {}</p>", i)),
creator_user_token: Some(&creator_token), // replace with actual UserToken
creator_ip_address: "192.168.1.1",
creator_set_visibility: Visibility::Public,
@@ -583,9 +581,9 @@ mod tests {
original_filename: Some(format!("filename {}.txt", i)),
file_size_bytes: 1024,
file_checksum_sha2: format!("checksum{}", i),
private_bucket_hash: format!("bucket_hash{}", i),
maybe_private_bucket_prefix: Some("_fake".to_string()),
maybe_private_bucket_extension: Some("rvc".to_string()),
public_bucket_hash: format!("bucket_hash{}", i),
maybe_public_bucket_prefix: Some("_fake".to_string()),
maybe_public_bucket_extension: Some("rvc".to_string()),
cached_user_ratings_total_count: 10,
cached_user_ratings_positive_count: 9,
cached_user_ratings_negative_count: 1,
@@ -1,76 +0,0 @@
use anyhow::anyhow;
use sqlx::MySqlPool;
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::model_weights::ModelWeightToken;
pub struct UpdateWeightArgs<'a> {
pub weight_token: &'a ModelWeightToken,
pub title: Option<&'a str>,
pub maybe_thumbnail_token: Option<&'a str>,
pub description_markdown: Option<&'a str>,
pub description_rendered_html: Option<&'a str>,
pub creator_set_visibility: Option<&'a Visibility>,
pub weights_type: Option<String>,
pub weights_category: Option<String>,
pub mysql_pool: &'a MySqlPool,
}
pub async fn update_weights(args: UpdateWeightArgs<'_>) -> AnyhowResult<()> {
let transaction = args.mysql_pool.begin().await?;
let query_result = sqlx::query!(
r#"
UPDATE model_weights
SET
title = COALESCE(?, title),
description_markdown = COALESCE(?, description_markdown),
maybe_thumbnail_token = COALESCE(?, maybe_thumbnail_token),
description_rendered_html = COALESCE(?, description_rendered_html),
creator_set_visibility = COALESCE(?, creator_set_visibility),
version = version + 1
WHERE token = ?
"#,
args.title.as_deref(),
args.description_markdown.as_deref(),
args.maybe_thumbnail_token.as_deref(),
args.description_rendered_html.as_deref(),
args.creator_set_visibility.as_deref(),
args.weight_token.as_str()
)
.execute(args.mysql_pool).await;
transaction.commit().await?;
match query_result {
Ok(_) => Ok(()),
Err(err) => {
Err(anyhow!("weights update error: {:?}", err))
}
}
}
#[cfg(test)]
mod tests {
// Template
use sqlx::mysql::MySqlPoolOptions;
use tokio;
use config::shared_constants::DEFAULT_MYSQL_CONNECTION_STRING;
use errors::AnyhowResult;
#[tokio::test]
async fn test_update_weights() -> AnyhowResult<()> {
let db_connection_string = DEFAULT_MYSQL_CONNECTION_STRING;
let pool = MySqlPoolOptions::new()
.max_connections(3)
.connect(&db_connection_string)
.await?;
Ok(())
}
}
@@ -0,0 +1,339 @@
use anyhow::anyhow;
use chrono::{DateTime, Utc};
use log::warn;
use sqlx::{MySql, MySqlPool};
use sqlx::pool::PoolConnection;
use enums::by_table::tts_models::tts_model_type::TtsModelType;
use enums::common::visibility::Visibility;
use errors::AnyhowResult;
use tokens::tokens::model_weights::ModelWeightToken;
use tokens::tokens::tts_models::TtsModelToken;
use tokens::tokens::users::UserToken;
use tokens::tokens::vocoder_models::VocoderModelToken;
use crate::helpers::boolean_converters::{i8_to_bool, nullable_i8_to_optional_bool};
/// This is meant to be the entire table
#[derive(Debug)]
pub struct WholeTtsModelRecord {
pub id: i64,
pub token: TtsModelToken,
pub tts_model_type: TtsModelType,
pub text_pipeline_type: Option<String>,
pub has_self_contained_vocoder: bool,
pub has_self_contained_duration_model: bool,
pub has_self_contained_pitch_model: bool,
pub text_preprocessing_algorithm: Option<String>,
pub use_default_mel_multiply_factor: bool,
pub maybe_custom_mel_multiply_factor: Option<f64>,
pub maybe_default_pretrained_vocoder: Option<String>,
pub maybe_custom_vocoder_token: Option<VocoderModelToken>,
pub ietf_language_tag: String,
pub ietf_primary_language_subtag: String,
pub title: String,
pub description_markdown: String,
pub description_rendered_html: String,
pub voice_name: Option<String>,
pub actor_name: Option<String>,
pub characteristic: Option<String>,
pub version_string: Option<String>,
pub is_front_page_featured: bool,
pub is_twitch_featured: bool,
pub maybe_suggested_unique_bot_command: Option<String>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub creator_user_token: UserToken,
pub creator_ip_address_creation: String,
pub creator_ip_address_last_update: String,
pub creator_set_visibility: Visibility,
pub private_bucket_hash: String,
pub private_bucket_object_name: String,
pub private_bucket_object_is_archive: bool,
pub user_ratings_total_count: u32,
pub user_ratings_positive_count: u32,
pub user_ratings_negative_count: u32,
pub is_public_listing_approved: Option<bool>,
pub is_locked_from_user_modification: bool,
pub is_locked_from_use: bool,
pub maybe_mod_comments: Option<String>,
pub maybe_mod_user_token: Option<UserToken>,
pub maybe_migration_new_model_weights_token: Option<ModelWeightToken>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
}
pub async fn list_whole_tts_models_using_cursor(
mysql_pool: &MySqlPool,
page_size: u64,
cursor: u64,
) -> AnyhowResult<Vec<WholeTtsModelRecord>> {
let mut connection = mysql_pool.acquire().await?;
let maybe_models
= list_whole_voice_conversion_models(&mut connection, page_size, cursor)
.await;
let models : Vec<RawRecord> = match maybe_models {
Ok(models) => models,
Err(err) => {
match err {
RowNotFound => {
return Ok(Vec::new());
},
_ => {
warn!("vc model list query error: {:?}", err);
return Err(anyhow!("vc model list query error"));
}
}
}
};
Ok(models.into_iter()
.map(|model| {
WholeTtsModelRecord {
id: model.id,
token: model.token,
tts_model_type: model.tts_model_type,
text_pipeline_type: model.text_pipeline_type,
has_self_contained_vocoder: i8_to_bool(model.has_self_contained_vocoder),
has_self_contained_duration_model: i8_to_bool(model.has_self_contained_duration_model),
has_self_contained_pitch_model: i8_to_bool(model.has_self_contained_pitch_model),
text_preprocessing_algorithm: model.text_preprocessing_algorithm,
use_default_mel_multiply_factor: i8_to_bool(model.use_default_mel_multiply_factor),
maybe_custom_mel_multiply_factor: model.maybe_custom_mel_multiply_factor,
maybe_default_pretrained_vocoder: model.maybe_default_pretrained_vocoder,
title: model.title,
description_markdown: model.description_markdown,
description_rendered_html: model.description_rendered_html,
voice_name: model.voice_name,
actor_name: model.actor_name,
characteristic: model.characteristic,
ietf_language_tag: model.ietf_language_tag,
ietf_primary_language_subtag: model.ietf_primary_language_subtag,
is_front_page_featured: i8_to_bool(model.is_front_page_featured),
is_twitch_featured: i8_to_bool(model.is_twitch_featured),
maybe_suggested_unique_bot_command: model.maybe_suggested_unique_bot_command,
original_download_url: model.original_download_url,
original_filename: model.original_filename,
file_size_bytes: model.file_size_bytes,
creator_user_token: model.creator_user_token,
creator_ip_address_creation: model.creator_ip_address_creation,
creator_ip_address_last_update: model.creator_ip_address_last_update,
creator_set_visibility: model.creator_set_visibility,
private_bucket_hash: model.private_bucket_hash,
private_bucket_object_name: model.private_bucket_object_name,
private_bucket_object_is_archive: i8_to_bool(model.private_bucket_object_is_archive),
user_ratings_total_count: model.user_ratings_total_count,
user_ratings_positive_count: model.user_ratings_positive_count,
user_ratings_negative_count: model.user_ratings_negative_count,
is_public_listing_approved: nullable_i8_to_optional_bool(model.is_public_listing_approved),
is_locked_from_user_modification: i8_to_bool(model.is_locked_from_user_modification),
is_locked_from_use: i8_to_bool(model.is_locked_from_use),
maybe_mod_comments: model.maybe_mod_comments,
maybe_mod_user_token: model.maybe_mod_user_token,
maybe_migration_new_model_weights_token: model.maybe_migration_new_model_weights_token,
version: model.version,
created_at: model.created_at,
updated_at: model.updated_at,
user_deleted_at: model.user_deleted_at,
mod_deleted_at: model.mod_deleted_at,
maybe_custom_vocoder_token: model.maybe_custom_vocoder_token,
version_string: model.version_string
}
})
.collect::<Vec<WholeTtsModelRecord>>())
}
async fn list_whole_voice_conversion_models(
mysql_connection: &mut PoolConnection<MySql>,
page_size: u64,
cursor: u64,
) -> AnyhowResult<Vec<RawRecord>> {
Ok(sqlx::query_as!(
RawRecord,
r#"
SELECT
id,
token as `token: tokens::tokens::tts_models::TtsModelToken`,
tts_model_type as `tts_model_type: enums::by_table::tts_models::tts_model_type::TtsModelType`,
text_pipeline_type,
has_self_contained_vocoder,
has_self_contained_duration_model,
has_self_contained_pitch_model,
text_preprocessing_algorithm,
use_default_mel_multiply_factor,
maybe_custom_mel_multiply_factor,
maybe_default_pretrained_vocoder,
maybe_custom_vocoder_token as `maybe_custom_vocoder_token: tokens::tokens::vocoder_models::VocoderModelToken`,
ietf_language_tag,
ietf_primary_language_subtag,
title,
description_markdown,
description_rendered_html,
voice_name,
actor_name,
characteristic,
version_string,
is_front_page_featured,
is_twitch_featured,
maybe_suggested_unique_bot_command,
original_download_url,
original_filename,
file_size_bytes,
creator_user_token as `creator_user_token: tokens::tokens::users::UserToken`,
creator_ip_address_creation,
creator_ip_address_last_update,
creator_set_visibility as `creator_set_visibility: enums::common::visibility::Visibility`,
private_bucket_hash,
private_bucket_object_name,
private_bucket_object_is_archive,
user_ratings_total_count,
user_ratings_positive_count,
user_ratings_negative_count,
is_public_listing_approved,
is_locked_from_user_modification,
is_locked_from_use,
maybe_mod_comments,
maybe_mod_user_token as `maybe_mod_user_token: tokens::tokens::users::UserToken`,
maybe_migration_new_model_weights_token as `maybe_migration_new_model_weights_token: tokens::tokens::model_weights::ModelWeightToken`,
version,
created_at,
updated_at,
user_deleted_at,
mod_deleted_at
FROM tts_models
WHERE
id > ?
ORDER BY id ASC
LIMIT ?
"#,
cursor,
page_size
)
.fetch_all(&mut **mysql_connection)
.await?)
}
struct RawRecord {
pub id: i64,
pub token: TtsModelToken,
pub tts_model_type: TtsModelType,
pub text_pipeline_type: Option<String>,
pub has_self_contained_vocoder: i8, // bool
pub has_self_contained_duration_model: i8, // bool
pub has_self_contained_pitch_model: i8, // bool
pub text_preprocessing_algorithm: Option<String>,
pub use_default_mel_multiply_factor: i8, // bool
pub maybe_custom_mel_multiply_factor: Option<f64>,
pub maybe_default_pretrained_vocoder: Option<String>,
pub maybe_custom_vocoder_token: Option<VocoderModelToken>,
pub ietf_language_tag: String,
pub ietf_primary_language_subtag: String,
pub title: String,
pub description_markdown: String,
pub description_rendered_html: String,
pub voice_name: Option<String>,
pub actor_name: Option<String>,
pub characteristic: Option<String>,
pub version_string: Option<String>,
pub is_front_page_featured: i8, // bool
pub is_twitch_featured: i8, // bool
pub maybe_suggested_unique_bot_command: Option<String>,
pub original_download_url: Option<String>,
pub original_filename: Option<String>,
pub file_size_bytes: i32,
pub creator_user_token: UserToken,
pub creator_ip_address_creation: String,
pub creator_ip_address_last_update: String,
pub creator_set_visibility: Visibility,
pub private_bucket_hash: String,
pub private_bucket_object_name: String,
pub private_bucket_object_is_archive: i8, // bool
pub user_ratings_total_count: u32,
pub user_ratings_positive_count: u32,
pub user_ratings_negative_count: u32,
pub is_public_listing_approved: Option<i8>, // bool
pub is_locked_from_user_modification: i8, // bool
pub is_locked_from_use: i8, // bool
pub maybe_mod_comments: Option<String>,
pub maybe_mod_user_token: Option<UserToken>,
pub maybe_migration_new_model_weights_token: Option<ModelWeightToken>,
pub version: i32,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub user_deleted_at: Option<DateTime<Utc>>,
pub mod_deleted_at: Option<DateTime<Utc>>,
}

Some files were not shown because too many files have changed in this diff Show More