mirror of
https://github.com/storytold/artcraft.git
synced 2026-10-23 18:19:30 -04:00
Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bcf3300b13 | ||
|
|
f423f1ccb8 | ||
|
|
7a3135c020 | ||
|
|
dfcf827a41 | ||
|
|
6ee6f25c86 |
Binary file not shown.
|
After Width: | Height: | Size: 2.0 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.8 MiB |
@@ -1,9 +1,16 @@
|
||||
[
|
||||
{
|
||||
"slug": "world-models-for-film",
|
||||
"title": "3D and World Models for Film",
|
||||
"description": "How to use World Models and 3D Rendering to Get Consistent Film Settings",
|
||||
"date": "2026-01-30",
|
||||
"body": "\nTest"
|
||||
},
|
||||
{
|
||||
"slug": "welcome",
|
||||
"title": "Welcome to the ArtCraft Blog",
|
||||
"description": "We are excited to launch our new blog where we will share updates, tutorials, and behind-the-scenes content.",
|
||||
"date": "2026-01-16",
|
||||
"body": "\n# Hello World!\n\nWelcome to the official **ArtCraft Blog**. This is where we'll be posting about:\n\n- **New Features**: Be the first to know about the latest tools and capabilities added to ArtCraft.\n- **Tutorials**: Learn how to get the most out of our AI generation and editing suite.\n- **Community Showcases**: Highlighting amazing work created by our users.\n- **Development Updates**: Insights into what we are building next.\n\n## Stay Tuned\n\nWe have a lot of exciting things in the pipeline. Make sure to check back regularly!\n\nHappy Crafting!\nThe ArtCraft Team\n"
|
||||
"body": "\n# Hello World!\n\nWelcome to the official **ArtCraft Blog**. This is where we'll be posting about:\n\n- **New Features**: Be the first to know about the latest tools and capabilities added to ArtCraft.\n- **Tutorials**: Learn how to get the most out of our AI generation and editing suite.\n- **Community Showcases**: Highlighting amazing work created by our users.\n- **Development Updates**: Insights into what we are building next.\n\n## Stay Tuned\n\nWe have a lot of exciting things in the pipeline. Make sure to check back regularly!\n\nHappy Crafting!\n\nThe ArtCraft Team\n"
|
||||
}
|
||||
]
|
||||
@@ -18,4 +18,5 @@ Welcome to the official **ArtCraft Blog**. This is where we'll be posting about:
|
||||
We have a lot of exciting things in the pipeline. Make sure to check back regularly!
|
||||
|
||||
Happy Crafting!
|
||||
|
||||
The ArtCraft Team
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
title: World Models for Consistent AI Filmmaking
|
||||
abstract: How to use World Models and 3D Rendering to Get Consistent AI Film Locations
|
||||
date: 2026-01-30
|
||||
---
|
||||
|
||||
# The "AI Film Trailer" Era
|
||||
|
||||
You've seen a lot of these:
|
||||
|
||||
@youtube(pGn-1BKo3nY)
|
||||
|
||||
For the past year, it been relatively easy to make montages, "clip shows", and trailers
|
||||
like this.
|
||||
|
||||
Good stories have characters that occupy real spaces. They have room to live, breathe, and interact.
|
||||
|
||||
You don't just fast track everyone through a journey from point A to B and have that be the end of things [1].
|
||||
Characters need to spend time in places, to explore interpersonal relationships, to overcome challenges.
|
||||
They need places to think, talk, fight and argue, process emotions, grow, and sometimes do absolutely nothing
|
||||
at all.
|
||||
|
||||
Stories need locations. They're a first class citizen in storytelling.
|
||||
|
||||
# Location and Blocking
|
||||
|
||||
We're all filmmakers at ArtCraft. The old school, photons-on-glass kind.
|
||||
|
||||
One of the biggest problems we've had over the past few years with AI video is giving our characters room to occupy.
|
||||
Setting them up on set, precisely controlling what's on screen, and maintaining consistency with their location.
|
||||
The majority of AI video right now looks like a series of disconnected clips, which results in
|
||||
AI film having that "film trailer" montage feel to it. Lots of jumping around, very little
|
||||
continuity.
|
||||
|
||||
We've figured out how to overcome this problem. And in the spirit of "Show, Don't Tell",
|
||||
let me show you what we've discovered. You'll intuit how this technique works immediately:
|
||||
|
||||
@youtube(wJCJYdGdpHg)
|
||||
|
||||
# Moving the Camera is Important
|
||||
|
||||
In *"film language"*, the camera's relationship to the characters conveys meaning. Sometimes we want to give the
|
||||
viewer a warm and friendly perspective, or perhaps frame the relationship between two different characters and
|
||||
imply a certain power dynamic.
|
||||
|
||||
Sometimes it's just important to show the location so the viewer can settle in. So the setting feels lived in and
|
||||
the viewer is immersed in it alongside our characters.
|
||||
|
||||
# Text is a Coarse Grained Representation
|
||||
|
||||
Here's a simple Nano Banana Pro prompt:
|
||||
|
||||
> Sam Altman in a room. He sits at a desk, looking towards a desktop computer on the left.
|
||||
> The room is dimly lit. The glow of a computer terminal, sitting in front of him bathes the room in blue light.
|
||||
> Behind him, there is a wall with a poster. There’s an AC/DC poster on the left, and a window on the right.
|
||||
> Through the window, there’s a view of moonlit trees.
|
||||
|
||||
And the results:
|
||||
|
||||

|
||||
|
||||
What happened to the location here? Why are none of these the same? Why are the locations of the poster,
|
||||
computer, window, and so forth so imprecise?
|
||||
|
||||
Moreover, how do we adjust this? Iteratively prompting?
|
||||
|
||||
You already know the answer: text is a poor serialization of the physical world. Text-to-Image is coarse
|
||||
grained. It lacks the ability to precisely convey physical attributes and spatial relationships. You can try,
|
||||
but you'll only get so much precision, and it'll never match the vision in your head.
|
||||
|
||||
# Images are Incomplete
|
||||
|
||||
There are multiple ways to move a camera in an image-to-image workflow. You can use a ControlNet, a LoRA, or even
|
||||
ask an instructive model nicely:
|
||||
|
||||
> Rotate the camera behind Sam Altman to show him from behind. Show the desk, computer screen,
|
||||
> and wall behind the desk. Show the door to the room.
|
||||
|
||||

|
||||
|
||||
While the lighting and scene elements remain somewhat consistent, we're still ultimately left with the same
|
||||
issues we had with text-to-image. We can iterate on editing with words, but it's slow, tedious, and still
|
||||
imprecise.
|
||||
|
||||
(Why does the desk block the door? What is it even trying to do with that poster?)
|
||||
|
||||
# 3D to the Rescue
|
||||
|
||||
One of the best tools for consistency is 3D. You can position your characters and props in a "3D set", and then move the "camera" to any angle, maintaining strong consistency throughout.
|
||||
|
||||

|
||||
|
||||
Pick a "smart" model like GPT Image 1.5 or Nano Banana Pro to convert this previz into a photorealistic render (or any style - anime, sci-fi, whatever):
|
||||
|
||||

|
||||
|
||||
And after Nano Banana Pro lighting adjustments and upscaling (and showing it the previz again to restore the missing desk pad), we get this:
|
||||
|
||||

|
||||
|
||||
It's a great starting composition before calling "action" with a video model. (I'd adjust the framing, but I wanted to show under the desk as well as bookcase features.)
|
||||
|
||||
|
||||
# 3D Kit Bashing and Greyboxing
|
||||
|
||||
You can use 3D kits to provide blocking and layout. You can also use 3D primitive shapes to create a "greybox" of your scene.
|
||||
|
||||
# Object Generation for Props
|
||||
|
||||
You can turn images into 3D prop objects using models such as Hunyuan 3D.
|
||||
|
||||
# World Models Make This Easy
|
||||
|
||||
The real star of this workflow is image-to-Gaussian Splat models, such as World Labs' Marble or Apple's Sharp. You can very quickly create a pleasing image of an intracate set in MidJourney, edit it in Nano Banana, then turn it into a fully navigable 3D scene.
|
||||
|
||||
|
||||
# ArtCraft: Free and Open 3D Filmmaking
|
||||
|
||||
ArtCraft in its entirety is available on [Github (please star us!)](https://github.com/storytold/artcraft)
|
||||
|
||||
We have [downloads for Windows and Mac](/download), with Linux support coming soon.
|
||||
|
||||
You can pay us for image and video compute, or you can bring your own keys and subscriptions
|
||||
without needing to pay us a dime. We offer the ability to log in with MidJourney, Grok,
|
||||
OpenAI/Sora, and several other providers. We'll be adding FAL, Replicate, and Google Gemini
|
||||
shortly, and local GPU support is coming soon (subscribe to stay updated).
|
||||
|
||||
— Brandon Thomas
|
||||
|
||||
[1] Unless you're Game of Thrones...
|
||||
@@ -5,6 +5,7 @@ export const markdownToHtml = (
|
||||
const lines = markdown.replace(/\r\n/g, "\n").split("\n");
|
||||
const htmlParts: string[] = [];
|
||||
let inList = false;
|
||||
let blockquoteLines: string[] = [];
|
||||
|
||||
const closeListIfOpen = () => {
|
||||
if (inList) {
|
||||
@@ -144,13 +145,45 @@ export const markdownToHtml = (
|
||||
return out;
|
||||
};
|
||||
|
||||
// Accumulate paragraph lines - consecutive non-blank, non-block lines form a single paragraph
|
||||
let paragraphLines: string[] = [];
|
||||
|
||||
const flushParagraph = () => {
|
||||
if (paragraphLines.length > 0) {
|
||||
const text = paragraphLines.map(renderInline).join(" ");
|
||||
htmlParts.push(`<p>${text}</p>`);
|
||||
paragraphLines = [];
|
||||
}
|
||||
};
|
||||
|
||||
const flushBlockquote = () => {
|
||||
if (blockquoteLines.length > 0) {
|
||||
const content = blockquoteLines.join(" ");
|
||||
htmlParts.push(`<blockquote>${renderInline(content)}</blockquote>`);
|
||||
blockquoteLines = [];
|
||||
}
|
||||
};
|
||||
|
||||
for (const raw of lines) {
|
||||
const line = raw.trimEnd();
|
||||
const ltrim = line.replace(/^\s+/, "");
|
||||
|
||||
// Blockquote: > text
|
||||
const blockquoteMatch = line.match(/^>\s?(.*)$/);
|
||||
if (blockquoteMatch) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
blockquoteLines.push(blockquoteMatch[1]);
|
||||
continue;
|
||||
}
|
||||
|
||||
// If we were in a blockquote but this line isn't, flush it
|
||||
flushBlockquote();
|
||||
|
||||
// YouTube embed: @youtube(VIDEO_ID or URL)
|
||||
const youtubeMatch = ltrim.match(/^@youtube\(([^)]+)\)\s*$/);
|
||||
if (youtubeMatch) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
const videoId = extractYouTubeId(youtubeMatch[1]);
|
||||
if (videoId) {
|
||||
@@ -164,6 +197,7 @@ export const markdownToHtml = (
|
||||
// Video embed: @video(url)
|
||||
const videoMatch = ltrim.match(/^@video\(([^)]+)\)\s*$/);
|
||||
if (videoMatch) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
htmlParts.push(renderVideoEmbed(videoMatch[1]));
|
||||
continue;
|
||||
@@ -174,6 +208,7 @@ export const markdownToHtml = (
|
||||
/^@(image|gif)\(([^,)]+)(?:,\s*([^)]+))?\)\s*$/,
|
||||
);
|
||||
if (imageMatch) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
const url = imageMatch[2].trim();
|
||||
const size = imageMatch[3]?.trim() || "";
|
||||
@@ -184,6 +219,7 @@ export const markdownToHtml = (
|
||||
// ATX headings: support #..###### with optional space after hashes
|
||||
const heading = ltrim.match(/^(#{1,6})\s*(.*)$/);
|
||||
if (heading) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
const level = Math.min(heading[1].length, 6);
|
||||
const text = renderInline(heading[2] || "");
|
||||
@@ -193,6 +229,7 @@ export const markdownToHtml = (
|
||||
// Standalone image/video line: 
|
||||
const imgMatch = line.match(/^!\[([^\]]*)\]\(([^)]+)\)\s*$/);
|
||||
if (imgMatch) {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
const alt = String(imgMatch[1] || "").replace(/"/g, """);
|
||||
const [url, size] = imgMatch[2].split("|");
|
||||
@@ -207,6 +244,7 @@ export const markdownToHtml = (
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith("- ")) {
|
||||
flushParagraph();
|
||||
if (!inList) {
|
||||
inList = true;
|
||||
htmlParts.push("<ul>");
|
||||
@@ -215,14 +253,18 @@ export const markdownToHtml = (
|
||||
continue;
|
||||
}
|
||||
if (line.trim() === "") {
|
||||
flushParagraph();
|
||||
closeListIfOpen();
|
||||
htmlParts.push("");
|
||||
continue;
|
||||
}
|
||||
// Regular text line - accumulate for paragraph
|
||||
closeListIfOpen();
|
||||
htmlParts.push(`<p>${renderInline(line)}</p>`);
|
||||
paragraphLines.push(line);
|
||||
}
|
||||
|
||||
flushParagraph();
|
||||
flushBlockquote();
|
||||
closeListIfOpen();
|
||||
return htmlParts.join("\n");
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user