WEBVTT
1
00:00:00.180 --> 00:00:04.059
The weakest area with AI is still, like, wide shots, lots of detail.
2
00:00:04.260 --> 00:00:14.420
And if you're generating at 720 and there's a lotta detail in that shot, it's shrunk down, it's small, and then if you upres it, it doesn't necessarily know what that is, and sometimes you just don't really get sharp details.
3
00:00:14.520 --> 00:00:27.680
[static] All right. Welcome back to Denoised. Addy, how you doing? How you doing, sir? I'm good. All right. Busy week, but, uh, all's good. [laughs] Lot, lotta updates, man.
4
00:00:27.740 --> 00:00:39.840
Lot of stuff came out, and then is about to come out. A lot of new little things. Well, this one's a big one. Corridor Key 2.0- Yeah... is released. And we talked about Corridor Key one. I've used the first version.
5
00:00:40.480 --> 00:00:50.020
It takes a little tweaking, but I mean, it's great. And what it does is you give it green screen or blue screen flare, uh, footage, and it pulls a key out of it. Yeah, absolutely. Did a really good job first time around.
6
00:00:50.660 --> 00:01:02.700
Yeah. Uh, new one, even better. It's, uh, it's... I heard, I saw somewhere it's, like, 100 times [laughs] better, like some weird number. But no, it does, it does look really good.
7
00:01:02.800 --> 00:01:09.500
And, uh, I mean, personally, I think you and I are both fans of Corridor Crew, like the channel [laughs] on YouTube. Yeah, they do great stuff. Yeah.
8
00:01:09.540 --> 00:01:15.480
And I used to actually work, uh, in the Arts District, like, like walking distance from their studio. Oh, from where they are. So, like, I used- Oh, cool...
9
00:01:15.520 --> 00:01:18.640
to see them once in a while walking around, uh, which is awesome, too.
10
00:01:19.380 --> 00:01:31.640
I think there's a lot of under-the-hood AI neural network stuff going on with this keyer that you're not gonna get with a traditional keyer, like, let's say in Resolve pers- per se, you know? No.
11
00:01:31.660 --> 00:01:44.520
And they, their video does a good breakdown of, like, first showing the clips. And this clip's probably one of my favorite, uh, demo clips of someone holding on the green, going out of focus. Yeah. Walking back.
12
00:01:44.540 --> 00:01:55.240
And the glass of water. Yeah, the glass of water- And the glass of water... is so crazy. Yeah. And it pulls a perfect key out of that. Uh, but this video goes into depth of how they trained 2.0.
13
00:01:55.280 --> 00:02:01.180
And basically, I mean, the gist was, like, 1.0 was sorta trained on each frame. And it's like, "Here's a frame. Figure out what the colors are, what the key is."
14
00:02:01.820 --> 00:02:12.120
2.0 now has temporal awareness, and so can look at the entire video clip and make better- It can go, uh-... uh, key decisions... backwards in time, forwards in time. Yeah. Yeah. That's, that, that's how you gotta do it.
15
00:02:12.180 --> 00:02:19.680
Yep. And they br- and they re- recruited some neural network, uh, experts and stuff, so it's a really great video that gets into depth of how they built and trained this.
16
00:02:19.740 --> 00:02:30.640
And trained it on local hardware, uh, Puget Systems, uh, local hardware and stuff that they have. Which is also- I-... cool and, you know, not needing a big render farm and, uh, uh, cloud compute network.
17
00:02:30.860 --> 00:02:44.640
That's fantastic. I actually downloaded it yesterday. I bought the $50 license 'cause I was just so curious, and it, it's affordable enough. And, uh, I ran a couple of videos through it. It was really fast on my, um...
18
00:02:45.120 --> 00:03:02.840
I think I have a 4090 RTX card. Like, 150 frame video took, like, a minute to process. Okay. Yeah. Uh, 1080- 1080 or 4K? Um, 4K. And it's also optimized for CUDA, I'm guessing. So it wants the NVIDIA GPUs. Yeah.
19
00:03:02.860 --> 00:03:12.620
And it does have a Mac beta, which is good. I mean, I, I... You're gonna get better performance out of an NVIDIA card, but it is good that at least there's an option if you do have Mac. Yeah. And they have- Um...
20
00:03:12.720 --> 00:03:20.940
a standalone application now. So, like, i- if you don't wanna use the plugin in Resolve or Premiere or whatnot, then you can just download the standalone app. Yeah.
21
00:03:21.000 --> 00:03:28.840
I mean, also to clarify, because the first one came out and it was sort of this open source model that you could do- run and download on your own hardware. It, it...
22
00:03:29.080 --> 00:03:39.080
You just had to sorta know what you were doing, and then the community built a bunch of plugins and stuff. Yeah. 2.0 is still free. The weights and stuff are still free and open source to download and use if you want to.
23
00:03:39.680 --> 00:03:47.980
But also they have this nice app or plugin for pretty much all of the apps, uh, and a much nicer user interface. Mm-hmm. $50 one-time fee.
24
00:03:48.440 --> 00:03:55.500
Uh, which also honestly, if I [laughs] would probably just pay that even if I was gonna download the weights- Exactly... because- Just to support them... just to support the project. Yeah, yeah. You know?
25
00:03:55.500 --> 00:04:06.200
No, this is- I think that's a very reasonable price. Yeah. Yeah. It's like $300 for Resolve Studio. It's like a no-brainer, you know? [laughs] Yeah. Actually- Yeah... Resolve Studio is on sale right now for, like, 250.
26
00:04:06.280 --> 00:04:16.700
Oh. Uh, Blackmagic was putting that on a sale. Yeah, so. Yeah. Christmas gift- If you were in the market for Studio... for the video nerd. Yeah. [laughs] One-time fee, uh, 250. Res- Resolve is...
27
00:04:17.260 --> 00:04:25.080
Yeah, Resolve's a good deal. Anyways, Corridor Key is also a very good deal. Yeah. And, and these guys do it right. Like, uh, they're, they're building it for the artist and, uh- Mm-hmm...
28
00:04:25.080 --> 00:04:37.280
with workflows, professional VFX workflows in mind. Um, I even saw some other VFX artists, uh, use this Corridor Key 2 and then comment on it, and most of the comments have been really positive.
29
00:04:37.320 --> 00:04:48.740
Like, "Oh, yeah, this is much better." Yeah, yeah, especially movement, crazy hair stuff. Ha- Um- Yeah, skin highlights and despill stuff- Mm-hmm... I think they really nailed as well. And we're focusing on green screen.
30
00:04:48.780 --> 00:05:03.220
It does work on green and blue screen. I'm gonna take a guess. Corridor Key 3.0, gray or no screen. You do love the gray screen, Joey. That's your color. You should wear a gray shirt every day. Yeah.
31
00:05:03.240 --> 00:05:15.320
Every one thing, I'll just put gray behind me. Uh, gray, gray is the new... Gray is the, uh, the mascot flag of AI hybrid productions now. I think s- Um- I think you're right. With some, like, uh, tape and markers on it.
32
00:05:15.560 --> 00:05:19.280
Yeah. Yeah, some feature. You need some features- The problem is-... on it... if you have gray
33
00:05:20.540 --> 00:05:27.740
and you, the, [laughs] you're not getting the best results out of it, there's limited options, uh, versus you shoot it on green or blue.
34
00:05:28.560 --> 00:05:38.260
The AI can work with it, but also you have stuff like Corridor Key a- and just also traditional compositing where it's like you know there is a workflow. There, that a, a, a system that can pull keys from that- Yeah...
35
00:05:38.560 --> 00:05:46.780
just in case- Absolutely. I mean-... you're having issues... if any of the guys, Niko and them, are [laughs] watching this video, please add gray in there for the next one. I'm sure gray is coming.
36
00:05:46.840 --> 00:05:49.760
Or just no screen, you know? Just say- No screen... it's messy stuff. Yeah.
37
00:05:49.840 --> 00:05:58.840
I mean, look, I'm sure they're running depth anything or, like, some crazy high-performing depth model in there, and that's why they're getting the, the really clean keys, right?
38
00:05:58.940 --> 00:06:07.220
Even though there is variations in the green color, like you have dark green, light green, no green, they're still able to key it out really well. Uh, and I- Mm-hmm...
39
00:06:07.280 --> 00:06:17.160
I think 'cause their depth pulling is working so well. Maybe they're upscaling the depth map and then bringing it back in. I'm not sure what they're doing, but it, it seems to be working.
40
00:06:17.592 --> 00:06:34.112
Also, if it's not doing that or you have a tricky key or stuff where it's, like, not perfectly lit, I've had it where I had Claude kind of process and sort of do two passes on a clip, and it's sort of like overcrank settings on one to do, like, good hair and then at, like, undercrank settings to get a cleaner key on the feet.
41
00:06:34.492 --> 00:06:41.532
The agents are very good if you just kind of describe what you're trying to do and give it visuals. It can also work out workflows with- Yeah...
42
00:06:41.552 --> 00:06:50.932
multiple processing I'm just thinking, like, the other thing they probably did is they took a lot of pre-green screen and artist-corrected post-green screen data, right?
43
00:06:50.992 --> 00:06:57.572
So, like, an artist goes in and they touch up on skin tones, they remove spill, and all that, and remove highlights.
44
00:06:57.792 --> 00:07:08.492
And, um, if you have enough pairing of that, like pre- and post-green screen, um, let's say, I don't know, 100 hours or so, like you could start to deep tune a model on it pretty well.
45
00:07:08.972 --> 00:07:13.552
I think in the first one, I remember he was saying, I believe that they did a lot of synthetic data.
46
00:07:13.672 --> 00:07:24.132
Like he used MetaHumans and used that to do, here's green, MetaHuman on green, here it is keyed, and used that to create the training data. Right. Right. I would imagine, yeah, a lot of...
47
00:07:24.172 --> 00:07:37.752
Like, if, if an artist goes in and they really put their artistry onto removing the green, then that artistry remains in the data, which imprints itself onto the model. Yeah. This is another great update from them.
48
00:07:37.772 --> 00:07:39.192
Awesome. All right, you want to talk about code?
49
00:07:39.292 --> 00:07:56.212
[laughs] So I have seen a couple of things pop up on X that was basically really elaborate 3D worlds that, uh, Lian Lin has built a few of them, and then it's revealed that they were made entirely with code. Okay.
50
00:07:56.432 --> 00:08:09.452
So I think that means it's, um, it's like a blender MCP where you're saying draw X- No... rectangle this size. No, absolutely nothing. No, the whole thing, the blender, the whole thing is rendered in code.
51
00:08:09.492 --> 00:08:23.212
But how, Joey, but how? The whole thing is rendered in Three.js, which is- Oh... uh, the web version of shapes- Yeah... and shaders and code, uh- Right... to render 3D in a browser. So the...
52
00:08:23.272 --> 00:08:31.792
Yeah, this has nothing to do with like any, like controlling blender. This is all purely- Purely on the web... raw. Like on the web. On the fly real time. Yeah.
53
00:08:31.872 --> 00:08:48.292
I don't know enough about the history of Three.js, but I think, or using it, but I'm guessing it would be clunky to use in the past, but to do something this detailed with agents is now feasible. That's crazy.
54
00:08:48.692 --> 00:08:56.292
I don't know, yeah, I don't know enough about Three.js and sort of web real time stuff. Um, if any of our viewers know what's going on here, please chime in. But
55
00:08:57.392 --> 00:09:08.212
I don't even think there's vertices and shaders here, right? Uh, it's all Three.js of what it does. Here is the prompt. Uh, and he's also pretty open about it. He has, uh, some repos with everything he built.
56
00:09:08.812 --> 00:09:19.412
Uh, the prompt- Oh, yeah... he gave it reference images. Um- Okay, it is generated in, so it's generated in situ, if you will, like the mesh and textures and everything. Okay.
57
00:09:19.672 --> 00:09:23.992
Here, build me a massive ultra detail 3D world in Three.js based on the nine reference images I attached.
58
00:09:24.152 --> 00:09:39.032
Rules, zero external assets, everything has to be generated in code, match references as closely as possible, detail. Wow. Uh- So he's not even cutting and pasting like a single tree, [laughs] you know?
59
00:09:39.092 --> 00:09:47.272
It's just all- No, he ga- Yeah... he generated the reference images, and then he gave it to it, and then he said, "Go make this." Go. Oh yeah, and then there's the code there. Jeez.
60
00:09:47.312 --> 00:10:00.612
Here's a, a more detailed view of, of everything. Uh- It, it looks like, uh-... took 45 hours to make... the quality is already like a, you know, 2000s video game. Like it's- He could, he should rebuild Myst.
61
00:10:01.092 --> 00:10:09.672
That's what I was thinking. Yeah, exactly. [laughs] It, it reminded me of Myst, but like, yeah, it's already there at that level with just like nine images and all pure code.
62
00:10:10.352 --> 00:10:29.192
Lower fidelity versions of this have really come into play to have these sort of browser-based 3D blocking editors, which are also very useful to make like a 3D, work in a rough 3D space online, create your shot, block out your shot, use that to feed it to one of the video models, and get your camera control, and then send it.
63
00:10:29.552 --> 00:10:42.732
This is taking that to another level. But I, this would... This level of detail is more fun [both laughing] and like an experiment. I'm curious how much further the detail could go with this. Yeah.
64
00:10:43.072 --> 00:10:49.592
You're not fighting photorealism, you know- Mm-hmm... you're not trying to attain this like, uh, unbelievable level of photorealism.
65
00:10:49.672 --> 00:11:02.432
It, it's stylized enough to just have fun with and become maybe a playground for a future game or any sort of RPG-like thing. Yeah. Or I don't know if this could run real time on the browser. I'm also curious about that.
66
00:11:02.472 --> 00:11:15.212
I th- I think it's more like rendered and he made these shots, but like if you could make a detailed world like this that is explorable. And also it's on a laptop GPU, like Radeon 780M.
67
00:11:15.832 --> 00:11:27.072
Imagine if this could be built real time for like a 5090 RTX, right? Like- Yeah. Now, he also did this other one, it was like an alpine lake, but this one is, there is no Three.js.
68
00:11:27.212 --> 00:11:41.232
This is purely Python and GLSL, which I'm not sure what that is. Oh, yeah, Graphics, uh, Shading Language. Yeah, that builds, um, shaders, um, in code. So it's like mathematical equivalent of traditional shaders.
69
00:11:41.772 --> 00:11:57.272
That's crazy. I remember we had to build a GLSL shader for work to replicate fur back in 2012 or something like that, you know, pre-AI obviously, and it was so difficult to do.
70
00:11:57.552 --> 00:12:09.372
We had to like bring in a PhD who understood GLSL and mathematics at that level to be able to just write something that mimicked fur, 'cause, you know, fur is so heavy- Yeah... it uses geometry. Yeah.
71
00:12:09.392 --> 00:12:16.892
So this GLSL shader mimicked fur without any of the geometry. Um, the way that it did it was really, really, uh, tricky.
72
00:12:17.212 --> 00:12:31.260
So it knew the camera normal, it knew the angle of the camera, and it put up these 2D cards at exactly the right angle in layers so that it looked like fur, and then as the camera angle would change, they would just like kinda move the 2D-...
73
00:12:31.320 --> 00:12:39.240
cards in that direction. So we did that, and we thought it was just such groundbreaking and we're like, "We should go get a patent or something." Like, "This is crazy."
74
00:12:39.240 --> 00:12:49.200
[laughs] And now it's, it's just so commoditized, like this is, this is just something that, you know, Claude runs on autopilot essentially. Yeah.
75
00:12:49.280 --> 00:13:12.320
I think that's what I was trying to sort of in- ineloquently say, or the point I tried to make before was what other coding things or things that were built before that kind of worked and had like a s- specific solution but were just really cumbersome to like implement and build out and do anything detailed with, but the foundation is there of this code, but now what's possible with it because of agents, what will this unlock?
76
00:13:12.580 --> 00:13:23.960
Exactly. And to sort of add to the nuance here, there is nothing generative here. Like, we're not doing- Mm... any diffusion. There's no video being generated, uh, images being generated.
77
00:13:24.840 --> 00:13:34.060
It's purely an LLM coding in a way that far exceeds the normal person's capabilities, at least in the immediate near term world. Yeah.
78
00:13:34.140 --> 00:13:43.840
I mean, maybe, you know, you see those elaborate, um, Minecraft creations where it's like, "I built a fully functional like mechanical clock in Minecraft that like took me 20, like 22 years." 20,000 hours.
79
00:13:43.900 --> 00:13:45.000
Yeah, like- Yeah. Yeah.
80
00:13:45.100 --> 00:14:00.900
And it's like sure, I'm sure someone could have spent like a [laughs] really long project coding the, uh, uh, fake Athens, but the fact that, you know, it was very un- un- unrealistic, but the, um, ability to do it now, uh, and what this unlocks- Yeah...
81
00:14:01.020 --> 00:14:10.480
is, is, is gonna be interesting. I'm just blown away, and, uh, the one thing that i- is, is like my inevitable question is like, "Well now what do we do [laughs] with it?"
82
00:14:10.560 --> 00:14:19.400
One thing I wanted to show, though, he did post, he posts a lot of data, which is also nice, but, um, the Alpine one, which is the one that was just Python and the other thing that y- you talked about- Mm-hmm...
83
00:14:20.300 --> 00:14:31.420
the API cost to make it, the total cost was about a little under $3,000. [laughs] Oh, boy. [laughs] In- It's not cheap... when computed. [laughs] So this wasn't... Uh, he was working using Claude.
84
00:14:31.480 --> 00:14:40.100
I mean, obviously once stuff gets distilled and- Uh... newer stuff comes out, then these get less expensive, then that's less expensive. But- Yeah, I wonder where the $3,000 came from.
85
00:14:40.160 --> 00:14:45.900
Like, is it [laughs] his, uh, savings account, or is it like work or, you know, 'cause that, that is a big chunk of change. Yeah.
86
00:14:45.900 --> 00:15:03.020
Or I don't know if he computed it, uh, 'cause like obviously all the subscriptions are subsidized, so I don't know if, because it says cost at API prices, I'm gonna guess this was like a let me use my plan, but if I didn't have a plan and I needed to, you know, I was enterprising running this, what would the equivalent be in buying this through API costs?
87
00:15:03.280 --> 00:15:11.320
I'm gonna guess he u- used a plan, like a Max plan and- Yeah... 'cause the amount you get from Max plan versus what the actual API costs, it's like heavily subsidized. Is the
88
00:15:12.460 --> 00:15:19.660
API cost in general going up faster than inflation? Do you know that? Do you know if that's true? I have no idea. Like- Don't know...
89
00:15:19.700 --> 00:15:29.240
I wonder if, uh, like 100 hours of Claude time in 2025 is less expensive than 100 hours of Claude time in 2026 on the same model.
90
00:15:29.740 --> 00:15:37.460
If you're doing the same model now to before, I mean, every time there's a new model, the old models get deprecated, and then they're usually cheaper. Okay, so then they're, they're...
91
00:15:37.740 --> 00:15:48.140
Okay, maybe that's one of- Like if you were trying to run- Yeah... or they just get, or they just get, um, end of lifed. Yeah. Yeah. So like let's say f- Opus 4.8, 100 hours in 2025 was like $100 let's say, right?
92
00:15:49.000 --> 00:15:59.460
Would Opus 5.5, 100 hours be $100? Uh, wait, sorry. So they're going to Opus 4... Oh, 4.8 was 100 bucks. Would it 5.5 be the same? Yeah. Doubt. I think s- I think they're roughly...
93
00:15:59.500 --> 00:16:05.140
I think whatever the, like, equivalent model, I think they've roughly stayed the same price, but I'm not 100% sure. Mm-hmm. Mm-hmm.
94
00:16:05.580 --> 00:16:14.680
I saw a recent, uh, news article that a lot of companies are cutting back on AI spending now because things have just- Yeah... gotten, uh, really out of control with spending.
95
00:16:14.760 --> 00:16:23.000
Yeah, if you're buying these API costs- Yeah... directly, and it's like the cost of the API, uh, AI versus, you know, just a person to do it is not, not saving much. Yeah. Interesting.
96
00:16:23.240 --> 00:16:33.740
This is all playing out in real time. All right. Model updates. Kling 4. Kling 4. Sorry. Kling 4 is released. I had to do that. I got excited. [laughs] Kling's great. Yeah, Kling 3 is great, and it's kind of been
97
00:16:34.820 --> 00:16:40.880
kind of quiet. Cdance 2 came out and sort of took a lot of the, the- Oh, yeah... but, uh- Yeah... now 4 is out. Yeah, and it took a while because- It was-...
98
00:16:40.940 --> 00:16:50.999
I remember when we did the Kling 03 test back in, uh, like end of last year, uh, I remember you, I used to do the AI roundup. You put a cowboy hat on me and all that. Like that- [laughs]...
99
00:16:51.040 --> 00:16:58.960
that looked okay for back then, like for a video-to-video model. That was, I think, the very first model specifically built for video to video.
100
00:16:59.400 --> 00:17:07.620
It was one of the first video to video, one of the first, I wanna say one of the kind of first reference- Yes... multi, uh, multimodal- Yeah... input kind of models.
101
00:17:07.620 --> 00:17:18.400
'Cause it was that, and then, uh, Wan 2.2 Animate was kind of that, and then Google Omni, and then Cdance 2.5, and now they're back at it. Now they're back. So 4.
102
00:17:18.760 --> 00:17:29.420
Um, main things, I mean, look, uh, honestly, it's a little bit of a catch-up to get to like Cdance 2 parity. Resolution-wise, I don't think all of these are out yet- Mm-hmm...
103
00:17:29.820 --> 00:17:41.940
or at least not on the fall API, but it will be able to do up to 4K. It will also be able to do up to 4K 10-bit HDR outputs. Ooh, that's interesting. Yes. Okay. Yes. Um- And look, and look, look...
104
00:17:42.100 --> 00:17:53.640
input-wise- The aspect ratio, 21:9, cinematic aspect. That's hard to do. Yeah. Two-channel stereo output. I'm kind of curious if that unlocks, if that, what that means, what that does. I mean, this- Input or output...
105
00:17:53.680 --> 00:18:04.840
this got like a Netflix ready thing already. [laughs] Like- Netflix approved. [laughs] Uh, and then input-wise, kind of what you would see on, on Cdance.
106
00:18:04.920 --> 00:18:15.880
Uh, 10 image input, video input, up to five videos with a combined duration of up to 30 seconds, which I think's about the same of what Cdance will take. Uh, audio input, you can give a voice reference- Mm-hmm...
107
00:18:15.920 --> 00:18:17.120
so for character speaking.
108
00:18:17.340 --> 00:18:33.080
You know, I would love to see, 'cause Kling also had the separate specific model, uh, motion transfer that was specifically designed to take a human performance input and map it onto a, uh, AI character- Mm-hmm
109
00:18:33.980 --> 00:18:39.900
I guess you can kind of do it with these models now, but I felt like that one was just good because it was like, it does this one thing and it does it really well.
110
00:18:40.360 --> 00:18:52.060
And I wonder if they're gonna do another version of that model or roll in a feature into this that does it. Is it out on Fal right now, Kling 4? Kling 4 is out on Fal. Oh, dude, I'm testing right after this recording.
111
00:18:52.060 --> 00:19:05.400
[laughs] I would say also the other kind of... You know, we're kind of comparing it and it, it definitely is on par feature-wise with Cdance, but the main thing is pricing, and it's way less- Way cheaper?...
112
00:19:05.600 --> 00:19:13.160
than- Oh, that, that's crazy... Cdance. Okay, let's see. So are you looking at about- So this is 720p, audio on, per second.
113
00:19:13.200 --> 00:19:19.560
And this is on Fal, and also Fal's Cdance rates are pretty high compared to where you could get it elsewhere, but it's just a good baseline.
114
00:19:19.780 --> 00:19:31.480
Um, Kling 4.0, uh, about 15 sec- 15 cents a second, whereas Cdance 2.0 is about 30 cents and, um, Cdance 2.5 is 47 cents- Yeah, I mean-... a second... I'm looking at the total.
115
00:19:31.540 --> 00:19:46.200
Like, you're getting 30 seconds at 1080p for half the price, whereas, you know... So it's really a quarter of the cost of Cdance 2.5. No, this co- this column. This is max.
116
00:19:46.260 --> 00:19:55.420
This is just the max resolu- the max settings it can do. Oh, okay. Oh, I was looking at, uh- Uh, this is price... 10, a 10-second column. So, like, uh, if you look- Oh, the 10- Yeah. Okay.
117
00:19:55.480 --> 00:20:07.920
Yeah, the 10-second is, uh, what the cost is at 10 seconds. So Kling 4 is about- It's like half... 50 for 10 seconds. And then it's half of Cdance 2, and it's- Like one third of Cdance 2.5... a third of Cdance 2.5.
118
00:20:08.340 --> 00:20:16.680
How do they do it? I mean, are they losing money on it? Like, what do you, what do you think? Or is it just a better model? Or I don't either. They just figure out how to get the quality at,
119
00:20:17.860 --> 00:20:31.280
uh, with less compute or less cost or- Yeah. Did you see that, uh, is it Huawei? Uh, Huawei is now building GPUs that rival NVIDIA GPUs, like data center level GPUs. Neural processing units. Neural processing units.
120
00:20:31.360 --> 00:20:39.780
Yeah. Yeah. So, like, they're really meant to go into, like, data centers and, um- Okay... obviously, that's gonna be cheaper for a Chinese, um,
121
00:20:40.740 --> 00:20:50.760
AI company to buy Huawei GPUs, put it into a data center in China than to, like, buy NVIDIA GPUs and try to use inference elsewhere or y- you know.
122
00:20:51.060 --> 00:21:00.620
So maybe there's something going on there that we're not privy to, and I'm sure we'll find out about later. Yeah. And yeah, how they can get this quality and put it out at this price.
123
00:21:00.760 --> 00:21:09.800
I just hope, like, you know, for, for the, like the sake of the planet, um, like we get to better and more efficient models 'cause, like, we, we just can't have,
124
00:21:10.900 --> 00:21:22.360
like, one video take, you know, five GPUs for 10 minutes or whatever. Like, that's just too much compute for, like, one video. So I hope, like, the efficiency and everything come very soon.
125
00:21:22.680 --> 00:21:35.840
I think that will, 'cause, you know, as g- going back to your comment about, uh, with the AI costs and pe- um, companies going back or cutting down on the AI costs, uh, and then also the AI companies needing to find a way to actually make money- Make sense of, yeah...
126
00:21:35.900 --> 00:21:45.200
and not have these- Make dollars and make sense. Yeah... so expensive. I mean, I see it as like everything now, yeah, it's to do the state of the art, it's expensive and to have this compute.
127
00:21:45.240 --> 00:21:52.700
But then every time there was like a new state-of-the-art model that comes out, eventually whatever was like the top benchmark then gets...
128
00:21:53.360 --> 00:22:04.860
New models come out, that one is still very capable and but it gets cheaper as new stuff comes out. And then the less expensive model or the, what was the state of the art, you know, Opus 5.5 in a year from now,
129
00:22:06.000 --> 00:22:17.200
maybe it's even deprecated 'cause it's like not even [laughs] have so many new things out that it's not good, but Opus 5.5 is very good, and it will eventually get cheaper once they, uh, you know, figure out how to make it more efficient.
130
00:22:17.460 --> 00:22:17.620
Yeah.
131
00:22:17.660 --> 00:22:31.720
I almost, I almost look at it like the way cars have evolved, high performance cars have evolved, at least in, in, you know, uh, from the '60s on, like the really crazy V8s and V12s were incredibly gas guzzling, right?
132
00:22:31.880 --> 00:22:43.560
And so it just didn't make economical sense to have them. And now that those engines are able to like get 20 miles, 30 miles a gallon, now everybody has V8s in their cars, like big engines in their cars.
133
00:22:43.940 --> 00:22:53.180
So I think like the AI models right now are very, um, resource intensive because it's new. But I think over time that that problem will hopefully solve itself.
134
00:22:53.680 --> 00:23:03.920
It's an economic problem, so I think it'll have to be solved if this is gonna be around. Yeah. Uh, I'm looking at some of their sample clips they posted on the site. Um, I would say honestly, these are
135
00:23:04.880 --> 00:23:15.500
not the best examples. Um, uh, th- this still feels very synthetic. Yeah. And I know it's streaming- Like the, the ship is not at scale [laughs] and- I know it's streaming through you. Uh- Yeah...
136
00:23:15.540 --> 00:23:23.800
so I can't see the best quality I'm looking at. Um, the close-up looks good. Uh, close-ups with AI look really good. It's very good at the detail and getting close-ups.
137
00:23:23.880 --> 00:23:34.400
The wider you get, that's where things always seem to fall apart. I mean, like every time I look at a synthetic human's face, I'm always like, "Man, we never got there with Unreal Engine and MetaHumans." Mm-hmm.
138
00:23:34.420 --> 00:23:45.040
Like as much as we tried, we, we were just so deep in uncanny valley, but AI is just getting so close to being completely real. Yeah. And it seems- Yeah...
139
00:23:45.060 --> 00:23:52.460
like this 15-second clip we're watching that's sort of like an action sequence is a one-shot kind of deal, which is impressive. Uh- Mm-hmm...
140
00:23:52.660 --> 00:24:02.600
but you know, I mean, I think, I think the big thing is it's like on par with maybe Cdance 2, but- Yeah... way cheaper. Okay. Why don't we do a test- Yeah... while, since it's out, out today? Okay.
141
00:24:02.760 --> 00:24:13.560
I'll run a couple of tests with Fal and, uh, with Cdance and, uh- This looks, uh-... Kling 4. This look, this look like from Hacks, but it's not Hacks. Ooh, I like the handheld camera. That's a nice touch.
142
00:24:14.620 --> 00:24:24.860
A good- And actually-... good destruction there... this is better 'cause one thing I noticed with Kling 3 a lot was, um, handheld cams and camera pans. It would have this very weird synthetic motion blur,
143
00:24:25.860 --> 00:24:33.780
and this pan definitely feels, uh, like a more natural- Natural motion blur. Yeah... yeah. Yeah. Kling always had this weird-
144
00:24:35.080 --> 00:24:44.260
I actually noticed that on the, on the arrow, uh, example that you showed earlier- Yeah, I did have-... the motion blur on the arrow looked pretty good, yeah. Yeah. Even it came through even on the, on the call here.
145
00:24:44.480 --> 00:24:53.280
Yeah. The other thing I was gonna mention, did you hear anything about NCM or New Concept Model? No, what's that? Okay. You can Google it.
146
00:24:53.380 --> 00:25:06.360
So, yeah, remember, uh, that I often bring up Yann LeCun and JePA, which is a new type of architecture that's not an LLM. Yeah. Uh, Nec- Next Concept Modeling. Next Concept Modeling, yes.
147
00:25:06.580 --> 00:25:14.760
So this is, uh, another type of architecture that is emerging, and it's supposed to be way, way more efficient.
148
00:25:14.820 --> 00:25:27.480
So yeah, you look at the numbers there, generating text to 10 to 31 times faster, seeing four to five times fewer training tokens. So not only is it cheaper to train, but it's cheaper to do inference on. Okay.
149
00:25:27.480 --> 00:25:41.860
It's not a large language model. It's using layers. Uh, I don't know what that means, but, like, this is an upcoming thing that is, um, hopefully gonna sort of put an ease on the, the resource intensive nature of AI.
150
00:25:42.700 --> 00:25:50.420
Okay, cool. Go check it out. And it- Yeah... but is, is this more... We said it's not an LLM, but is this more around training or building- It-... language models?
151
00:25:50.820 --> 00:26:00.440
It is, it is a language model, it's just not the way that... Like, it's not a transformer model- Uh-huh... 'cause that's what LLMs are. Yeah. So this is a new type of architecture, yeah. Okay, cool. Interesting.
152
00:26:00.500 --> 00:26:13.340
I'm kind of also ki- interesting how, you know, like we saw JeV and sort of a different type of, um, model. Yeah. I did a little bit more digging on JeV, um, and uh, I could see its usefulness at scale.
153
00:26:13.400 --> 00:26:25.400
Like, if you have 100 JeVs or 1,000 JeVs working together, you could really start to build incredible logic around it, 'cause it, it's, it's a really simple gating mechanism. Mm-hmm. So, like,
154
00:26:26.500 --> 00:26:37.140
if you're looking at a picture of Joey, right? Uh, JeV is gonna be like, "That's a, that's an adult male. Pass. That's not a child. Pass," right? Like, so it's gonna just do very basic things like that.
155
00:26:37.200 --> 00:26:48.120
It's not gonna be as sophisticated as, like, a VLM or an LLM. No, it's not that, but its advantage is its speed. Speed, right. Speed and cost. Right. It's extremely fast and extremely cheap. Yeah. Exactly.
156
00:26:48.260 --> 00:26:57.360
So the types of stuff where you'd just be... If you gave a photo to ChatGPT and are like, "What is this?" It would- Yeah... eat up a lot of tokens to figure out not a child. That's it. Yeah. Yeah.
157
00:26:57.880 --> 00:27:10.240
So I could see it ve- being very useful for, like, Internet of Things and, like, um, stuff that doesn't really need high processing. Mm-hmm. But it's also fuzzy, unstructured data. Yes. That, like, you know, could...
158
00:27:10.300 --> 00:27:23.360
is not an easy, uh, pass-fail thing to, to give to a traditional codeine. Exactly. All right, other new updates, quick roundup. Ideogram has their first vector model. Ooh, nice. Oh, vector models finally. Wow. Yeah.
159
00:27:23.740 --> 00:27:32.020
I, I actually... I thought they had something before, but I guess not. Um, yeah, no, they're, they're... You... I think the previously you can go from, uh, pixel- Or they can convert...
160
00:27:32.120 --> 00:27:42.680
space to vector space as, as a- Okay... like a conversion. Okay. So you're still generating in pixel space, but this looks like a native vector model- Yeah... so it's generating in vectors. That's great.
161
00:27:42.740 --> 00:27:49.020
Yeah, Ideogram's definitely been focused on, on the graphic design more world. Graphics side, yeah. For sure. Yeah. Um, so- Yeah... yeah, that's a good update.
162
00:27:49.560 --> 00:27:54.060
When I was, um, trying to tinker with this stuff about a year ago, having...
163
00:27:54.240 --> 00:28:06.220
Everybody's asking for, like, a vector model, vector model, and we were able to build some pix- pixel-to-vector conversions, and it was terrible. Was not good. [laughs] So this, this hopefully solves that.
164
00:28:06.660 --> 00:28:19.540
Uh, other update, File has another custom version of a post-trained model, so they took Flux.3 and then post-trained it to improve the quality, and I believe also speed. Uh, so they got Flux.3 Max.
165
00:28:20.060 --> 00:28:27.280
Flux.3, the video model, right? Flux.3, the video model. Yeah. Um- Flux.3's not bad, actually. Yeah. Yeah. And so this is post-trained.
166
00:28:27.300 --> 00:28:40.660
I mean, so in their speed chart, basically Flux.3, the original, would take about two minutes a generation. Their version will take, uh, 10 seconds- Oh, wow... which is a huge leap. That's fast.
167
00:28:40.800 --> 00:28:46.400
And according to the ranking, the quality is also higher. Um- Okay... but, uh, yeah.
168
00:28:46.460 --> 00:28:57.940
I mean, this is really interesting what File's been doing, where they've been taking these models, um, and then making their own custom versions on top of it, and that definitely is also smart to give them, uh, something unique.
169
00:28:57.960 --> 00:29:05.880
The guy's so sweaty. Look at this armpit, armpit stain. Yeah. Yeah, that's a nice touch. Uh, w- so here's my question to you. Like,
170
00:29:07.080 --> 00:29:19.440
if the File version of Flux.3 is better in both speed and quality, why would somebody use the original Flux? Well, you could download the open weights of Flux and use it on your own hardware bill.
171
00:29:19.500 --> 00:29:31.460
I mean, you could do what they did, um, on your own. I'm guessing, like, that there is some licensing play that they- Ah... you know, like if you run it on File, that, like, Black Forest Labs gets a cut.
172
00:29:31.660 --> 00:29:36.520
I mean, 'cause, uh, y- you know, it's... You could download the weights, but, like, their licensing stuff's usually...
173
00:29:37.480 --> 00:29:47.340
I'm thinking, I'm thinking of, um, LTX, where it's like if s- sort of the epic model where it's like, yeah, free to use if you make under a certain amount, but then if you make over a certain amount, like, talk to us about licensing.
174
00:29:47.800 --> 00:30:00.440
I'm guessing it's something similar to that. I don't know for sure, but I would assume there's a cut that Black Forest Labs gets. Yeah. S- they seem to be doing okay with, like, groundbreaking models and stuff. Yeah.
175
00:30:00.460 --> 00:30:07.460
They're another great, uh, AI studio that's, you know, punching above its weight with quality- 100%, yeah... not being a- With, like, a small team. Yeah. Yeah.
176
00:30:07.740 --> 00:30:18.680
And then last one, and thank you for Byte+ for posting this before we started recording. CDance 2.5 is- Ooh... uh, 4K is rolling out. That, that is the big Rolls-Royce right now.
177
00:30:18.740 --> 00:30:31.880
Like, that just dropped, and everybody's like, "Oh, dang." You need to pay a Rolls-Royce amount [laughs] for every generation. Yeah. Uh, it doesn't have pricing posted yet, but I mean- Yeah... 2.5 at 1080, I...
178
00:30:32.440 --> 00:30:45.316
on File is, like, $7 a generation for, like, 10 seconds- Okay... 15 seconds, so. So quadruple that. Yes, maybe 20, maybe- So $30? Yeah, yeah... 20 to $30 a generation.
179
00:30:45.636 --> 00:31:01.676
What we need an- is better pipelines to work at 720, and then once you're like, "Yeah, this is it-" Commit... upres it. Yes. But not- Yeah... as a new generation, have everything locked in.
180
00:31:01.896 --> 00:31:15.436
There's a way, and maybe I could demo, uh, the workflow next time, but, um, we found a pretty decent way to do this using depth maps, where you like, if you had your original footage, you, like, turn it into a depth map, and then you give that plus your original reference images
181
00:31:16.396 --> 00:31:26.856
into, like, the 4K output. And so you get... Depends on the shot, but you get a pretty good, consistent result that's 4K, but that still looks like your original clip- So the 4K-... it had the same movement...
182
00:31:26.856 --> 00:31:37.756
is being conditioned from the depth map of the 720p? Yeah. 'Cause my question was, like, how do you do seed locking so you're getting the exact same result? Yeah, I've tried with these models.
183
00:31:37.836 --> 00:31:46.836
Maybe the, you know, by, uh, CDance will be better, but it's like I've tried where it's like, yeah, if you took the seed you ran on 720 and you ran the same exact seed and same inputs on the 4K, it still is- It's still different.
184
00:31:47.116 --> 00:31:58.056
Yeah, because there's- It's different... yeah, because for every one 720p pixel, you now have to fill eight or 10, whatever- Yeah... 4K pixel. So it's fundamentally a different problem for it to solve.
185
00:31:59.336 --> 00:32:10.196
Yeah, I, I, I mean, that's where, like, upscalers, they make sense as, like, a finishing pass, right? So you do all of this work on getting the right thing at 720p, and then you commit to the upscale later.
186
00:32:10.936 --> 00:32:20.256
But then why would you use CDance 2.5 4K to just mess around? Like, that's a very expensive messing around. It's a lot of [laughs] it's a very pricey experiment.
187
00:32:20.356 --> 00:32:30.736
I have seen people online where they just swear by a workflow of, like, just strictly working and generating in 720, and then using, you know, Topaz or one of the other upres models to finish.
188
00:32:31.176 --> 00:32:39.476
I think there is an advantage. I think it de- look, it depends on the shot. And, uh, the weakest area with AI is still, like, wide shots, lots of detail.
189
00:32:40.076 --> 00:32:51.055
And if you're generating at 720 and there's a lot of detail in that shot, it's, it's shrunk down, it's small, and then if you upres it, it doesn't necessarily know what that is, and sometimes you just don't really get sharp details.
190
00:32:51.476 --> 00:33:05.576
So I think purely generating 4K, the best use case would be, like, just very detailed shots, crowds, buildings with lots of lines, uh, stuff that just- Mm... gets noisy and confused.
191
00:33:05.696 --> 00:33:13.136
Like, if you have a wide establishing shot of, I don't know, 1970s New York City. 1970s New York- [laughs]... with a bunch of people walking down the street- Yeah... with the taxi cabs going by.
192
00:33:13.156 --> 00:33:22.616
You need pixels on those guys in the back, yeah. Yeah. Yeah. And it's like if you had a 720 and then you tried to upres it- Right... it just doesn't always know what those pixels should be. And if you're purely...
193
00:33:22.676 --> 00:33:33.376
If you're giving this 4K reference input images and you're giving this all 4K inputs and you're going out to 4K- Mm... I, I, I think you will get a better result than- No, you're absolutely right... 720 to upres.
194
00:33:33.416 --> 00:33:40.656
Dude, you, this totally makes sense. Um, but if you're doing a close-up, if you're doing, like, close-ups of faces and stuff, I don't think you're gonna have... You're just gonna burn a lot of money.
195
00:33:40.676 --> 00:33:50.496
I don't think there's any benefit in, um, purely generating in 4K. Yeah, so, like, you pull the lever when you need to kind of a thing, and it's nice to have in your back pocket. Yeah, yeah.
196
00:33:50.896 --> 00:33:59.616
It, it's also, like, a, I think a little bit of, like, pushing their weight around, like, "Look at us. We got 4K." 4K. Yeah, huh. Do you have 4K? Do you have 4K? We got 4K. You...
197
00:33:59.616 --> 00:34:10.736
[laughs] We will give you an entire data center to spin up your clip, 'cause it's gonna need- It's got a center [laughs]... to generate your CDance 2.5 4K clip. Yeah. Okay. Well, I mean, it's all progress.
198
00:34:10.856 --> 00:34:12.736
Let's see where it goes. We'll see where it goes.
199
00:34:13.036 --> 00:34:23.856
Links to everything we talked about at denoisepodcast.com If there's anything that is incorrect, it's just our opinion, but we would love to hear about it on the YouTube comments. So hit us up on the comments.
200
00:34:24.016 --> 00:34:33.056
We did have some questions, uh, uh, throughout the episode, so you're, if you're more of an expert on this stuff, we'd love to hear it and get educated. Yeah, if you're a Three.js expert, let us know- Yes...
201
00:34:33.076 --> 00:34:38.976
in the comments what you think. [laughs] Uh, all right, thanks everyone. We will catch you in the next episode.
1
00:00:00.180 --> 00:00:04.059
The weakest area with AI is still, like, wide shots, lots of detail.
2
00:00:04.260 --> 00:00:14.420
And if you're generating at 720 and there's a lotta detail in that shot, it's shrunk down, it's small, and then if you upres it, it doesn't necessarily know what that is, and sometimes you just don't really get sharp details.
3
00:00:14.520 --> 00:00:27.680
[static] All right. Welcome back to Denoised. Addy, how you doing? How you doing, sir? I'm good. All right. Busy week, but, uh, all's good. [laughs] Lot, lotta updates, man.
4
00:00:27.740 --> 00:00:39.840
Lot of stuff came out, and then is about to come out. A lot of new little things. Well, this one's a big one. Corridor Key 2.0- Yeah... is released. And we talked about Corridor Key one. I've used the first version.
5
00:00:40.480 --> 00:00:50.020
It takes a little tweaking, but I mean, it's great. And what it does is you give it green screen or blue screen flare, uh, footage, and it pulls a key out of it. Yeah, absolutely. Did a really good job first time around.
6
00:00:50.660 --> 00:01:02.700
Yeah. Uh, new one, even better. It's, uh, it's... I heard, I saw somewhere it's, like, 100 times [laughs] better, like some weird number. But no, it does, it does look really good.
7
00:01:02.800 --> 00:01:09.500
And, uh, I mean, personally, I think you and I are both fans of Corridor Crew, like the channel [laughs] on YouTube. Yeah, they do great stuff. Yeah.
8
00:01:09.540 --> 00:01:15.480
And I used to actually work, uh, in the Arts District, like, like walking distance from their studio. Oh, from where they are. So, like, I used- Oh, cool...
9
00:01:15.520 --> 00:01:18.640
to see them once in a while walking around, uh, which is awesome, too.
10
00:01:19.380 --> 00:01:31.640
I think there's a lot of under-the-hood AI neural network stuff going on with this keyer that you're not gonna get with a traditional keyer, like, let's say in Resolve pers- per se, you know? No.
11
00:01:31.660 --> 00:01:44.520
And they, their video does a good breakdown of, like, first showing the clips. And this clip's probably one of my favorite, uh, demo clips of someone holding on the green, going out of focus. Yeah. Walking back.
12
00:01:44.540 --> 00:01:55.240
And the glass of water. Yeah, the glass of water- And the glass of water... is so crazy. Yeah. And it pulls a perfect key out of that. Uh, but this video goes into depth of how they trained 2.0.
13
00:01:55.280 --> 00:02:01.180
And basically, I mean, the gist was, like, 1.0 was sorta trained on each frame. And it's like, "Here's a frame. Figure out what the colors are, what the key is."
14
00:02:01.820 --> 00:02:12.120
2.0 now has temporal awareness, and so can look at the entire video clip and make better- It can go, uh-... uh, key decisions... backwards in time, forwards in time. Yeah. Yeah. That's, that, that's how you gotta do it.
15
00:02:12.180 --> 00:02:19.680
Yep. And they br- and they re- recruited some neural network, uh, experts and stuff, so it's a really great video that gets into depth of how they built and trained this.
16
00:02:19.740 --> 00:02:30.640
And trained it on local hardware, uh, Puget Systems, uh, local hardware and stuff that they have. Which is also- I-... cool and, you know, not needing a big render farm and, uh, uh, cloud compute network.
17
00:02:30.860 --> 00:02:44.640
That's fantastic. I actually downloaded it yesterday. I bought the $50 license 'cause I was just so curious, and it, it's affordable enough. And, uh, I ran a couple of videos through it. It was really fast on my, um...
18
00:02:45.120 --> 00:03:02.840
I think I have a 4090 RTX card. Like, 150 frame video took, like, a minute to process. Okay. Yeah. Uh, 1080- 1080 or 4K? Um, 4K. And it's also optimized for CUDA, I'm guessing. So it wants the NVIDIA GPUs. Yeah.
19
00:03:02.860 --> 00:03:12.620
And it does have a Mac beta, which is good. I mean, I, I... You're gonna get better performance out of an NVIDIA card, but it is good that at least there's an option if you do have Mac. Yeah. And they have- Um...
20
00:03:12.720 --> 00:03:20.940
a standalone application now. So, like, i- if you don't wanna use the plugin in Resolve or Premiere or whatnot, then you can just download the standalone app. Yeah.
21
00:03:21.000 --> 00:03:28.840
I mean, also to clarify, because the first one came out and it was sort of this open source model that you could do- run and download on your own hardware. It, it...
22
00:03:29.080 --> 00:03:39.080
You just had to sorta know what you were doing, and then the community built a bunch of plugins and stuff. Yeah. 2.0 is still free. The weights and stuff are still free and open source to download and use if you want to.
23
00:03:39.680 --> 00:03:47.980
But also they have this nice app or plugin for pretty much all of the apps, uh, and a much nicer user interface. Mm-hmm. $50 one-time fee.
24
00:03:48.440 --> 00:03:55.500
Uh, which also honestly, if I [laughs] would probably just pay that even if I was gonna download the weights- Exactly... because- Just to support them... just to support the project. Yeah, yeah. You know?
25
00:03:55.500 --> 00:04:06.200
No, this is- I think that's a very reasonable price. Yeah. Yeah. It's like $300 for Resolve Studio. It's like a no-brainer, you know? [laughs] Yeah. Actually- Yeah... Resolve Studio is on sale right now for, like, 250.
26
00:04:06.280 --> 00:04:16.700
Oh. Uh, Blackmagic was putting that on a sale. Yeah, so. Yeah. Christmas gift- If you were in the market for Studio... for the video nerd. Yeah. [laughs] One-time fee, uh, 250. Res- Resolve is...
27
00:04:17.260 --> 00:04:25.080
Yeah, Resolve's a good deal. Anyways, Corridor Key is also a very good deal. Yeah. And, and these guys do it right. Like, uh, they're, they're building it for the artist and, uh- Mm-hmm...
28
00:04:25.080 --> 00:04:37.280
with workflows, professional VFX workflows in mind. Um, I even saw some other VFX artists, uh, use this Corridor Key 2 and then comment on it, and most of the comments have been really positive.
29
00:04:37.320 --> 00:04:48.740
Like, "Oh, yeah, this is much better." Yeah, yeah, especially movement, crazy hair stuff. Ha- Um- Yeah, skin highlights and despill stuff- Mm-hmm... I think they really nailed as well. And we're focusing on green screen.
30
00:04:48.780 --> 00:05:03.220
It does work on green and blue screen. I'm gonna take a guess. Corridor Key 3.0, gray or no screen. You do love the gray screen, Joey. That's your color. You should wear a gray shirt every day. Yeah.
31
00:05:03.240 --> 00:05:15.320
Every one thing, I'll just put gray behind me. Uh, gray, gray is the new... Gray is the, uh, the mascot flag of AI hybrid productions now. I think s- Um- I think you're right. With some, like, uh, tape and markers on it.
32
00:05:15.560 --> 00:05:19.280
Yeah. Yeah, some feature. You need some features- The problem is-... on it... if you have gray
33
00:05:20.540 --> 00:05:27.740
and you, the, [laughs] you're not getting the best results out of it, there's limited options, uh, versus you shoot it on green or blue.
34
00:05:28.560 --> 00:05:38.260
The AI can work with it, but also you have stuff like Corridor Key a- and just also traditional compositing where it's like you know there is a workflow. There, that a, a, a system that can pull keys from that- Yeah...
35
00:05:38.560 --> 00:05:46.780
just in case- Absolutely. I mean-... you're having issues... if any of the guys, Niko and them, are [laughs] watching this video, please add gray in there for the next one. I'm sure gray is coming.
36
00:05:46.840 --> 00:05:49.760
Or just no screen, you know? Just say- No screen... it's messy stuff. Yeah.
37
00:05:49.840 --> 00:05:58.840
I mean, look, I'm sure they're running depth anything or, like, some crazy high-performing depth model in there, and that's why they're getting the, the really clean keys, right?
38
00:05:58.940 --> 00:06:07.220
Even though there is variations in the green color, like you have dark green, light green, no green, they're still able to key it out really well. Uh, and I- Mm-hmm...
39
00:06:07.280 --> 00:06:17.160
I think 'cause their depth pulling is working so well. Maybe they're upscaling the depth map and then bringing it back in. I'm not sure what they're doing, but it, it seems to be working.
40
00:06:17.592 --> 00:06:34.112
Also, if it's not doing that or you have a tricky key or stuff where it's, like, not perfectly lit, I've had it where I had Claude kind of process and sort of do two passes on a clip, and it's sort of like overcrank settings on one to do, like, good hair and then at, like, undercrank settings to get a cleaner key on the feet.
41
00:06:34.492 --> 00:06:41.532
The agents are very good if you just kind of describe what you're trying to do and give it visuals. It can also work out workflows with- Yeah...
42
00:06:41.552 --> 00:06:50.932
multiple processing I'm just thinking, like, the other thing they probably did is they took a lot of pre-green screen and artist-corrected post-green screen data, right?
43
00:06:50.992 --> 00:06:57.572
So, like, an artist goes in and they touch up on skin tones, they remove spill, and all that, and remove highlights.
44
00:06:57.792 --> 00:07:08.492
And, um, if you have enough pairing of that, like pre- and post-green screen, um, let's say, I don't know, 100 hours or so, like you could start to deep tune a model on it pretty well.
45
00:07:08.972 --> 00:07:13.552
I think in the first one, I remember he was saying, I believe that they did a lot of synthetic data.
46
00:07:13.672 --> 00:07:24.132
Like he used MetaHumans and used that to do, here's green, MetaHuman on green, here it is keyed, and used that to create the training data. Right. Right. I would imagine, yeah, a lot of...
47
00:07:24.172 --> 00:07:37.752
Like, if, if an artist goes in and they really put their artistry onto removing the green, then that artistry remains in the data, which imprints itself onto the model. Yeah. This is another great update from them.
48
00:07:37.772 --> 00:07:39.192
Awesome. All right, you want to talk about code?
49
00:07:39.292 --> 00:07:56.212
[laughs] So I have seen a couple of things pop up on X that was basically really elaborate 3D worlds that, uh, Lian Lin has built a few of them, and then it's revealed that they were made entirely with code. Okay.
50
00:07:56.432 --> 00:08:09.452
So I think that means it's, um, it's like a blender MCP where you're saying draw X- No... rectangle this size. No, absolutely nothing. No, the whole thing, the blender, the whole thing is rendered in code.
51
00:08:09.492 --> 00:08:23.212
But how, Joey, but how? The whole thing is rendered in Three.js, which is- Oh... uh, the web version of shapes- Yeah... and shaders and code, uh- Right... to render 3D in a browser. So the...
52
00:08:23.272 --> 00:08:31.792
Yeah, this has nothing to do with like any, like controlling blender. This is all purely- Purely on the web... raw. Like on the web. On the fly real time. Yeah.
53
00:08:31.872 --> 00:08:48.292
I don't know enough about the history of Three.js, but I think, or using it, but I'm guessing it would be clunky to use in the past, but to do something this detailed with agents is now feasible. That's crazy.
54
00:08:48.692 --> 00:08:56.292
I don't know, yeah, I don't know enough about Three.js and sort of web real time stuff. Um, if any of our viewers know what's going on here, please chime in. But
55
00:08:57.392 --> 00:09:08.212
I don't even think there's vertices and shaders here, right? Uh, it's all Three.js of what it does. Here is the prompt. Uh, and he's also pretty open about it. He has, uh, some repos with everything he built.
56
00:09:08.812 --> 00:09:19.412
Uh, the prompt- Oh, yeah... he gave it reference images. Um- Okay, it is generated in, so it's generated in situ, if you will, like the mesh and textures and everything. Okay.
57
00:09:19.672 --> 00:09:23.992
Here, build me a massive ultra detail 3D world in Three.js based on the nine reference images I attached.
58
00:09:24.152 --> 00:09:39.032
Rules, zero external assets, everything has to be generated in code, match references as closely as possible, detail. Wow. Uh- So he's not even cutting and pasting like a single tree, [laughs] you know?
59
00:09:39.092 --> 00:09:47.272
It's just all- No, he ga- Yeah... he generated the reference images, and then he gave it to it, and then he said, "Go make this." Go. Oh yeah, and then there's the code there. Jeez.
60
00:09:47.312 --> 00:10:00.612
Here's a, a more detailed view of, of everything. Uh- It, it looks like, uh-... took 45 hours to make... the quality is already like a, you know, 2000s video game. Like it's- He could, he should rebuild Myst.
61
00:10:01.092 --> 00:10:09.672
That's what I was thinking. Yeah, exactly. [laughs] It, it reminded me of Myst, but like, yeah, it's already there at that level with just like nine images and all pure code.
62
00:10:10.352 --> 00:10:29.192
Lower fidelity versions of this have really come into play to have these sort of browser-based 3D blocking editors, which are also very useful to make like a 3D, work in a rough 3D space online, create your shot, block out your shot, use that to feed it to one of the video models, and get your camera control, and then send it.
63
00:10:29.552 --> 00:10:42.732
This is taking that to another level. But I, this would... This level of detail is more fun [both laughing] and like an experiment. I'm curious how much further the detail could go with this. Yeah.
64
00:10:43.072 --> 00:10:49.592
You're not fighting photorealism, you know- Mm-hmm... you're not trying to attain this like, uh, unbelievable level of photorealism.
65
00:10:49.672 --> 00:11:02.432
It, it's stylized enough to just have fun with and become maybe a playground for a future game or any sort of RPG-like thing. Yeah. Or I don't know if this could run real time on the browser. I'm also curious about that.
66
00:11:02.472 --> 00:11:15.212
I th- I think it's more like rendered and he made these shots, but like if you could make a detailed world like this that is explorable. And also it's on a laptop GPU, like Radeon 780M.
67
00:11:15.832 --> 00:11:27.072
Imagine if this could be built real time for like a 5090 RTX, right? Like- Yeah. Now, he also did this other one, it was like an alpine lake, but this one is, there is no Three.js.
68
00:11:27.212 --> 00:11:41.232
This is purely Python and GLSL, which I'm not sure what that is. Oh, yeah, Graphics, uh, Shading Language. Yeah, that builds, um, shaders, um, in code. So it's like mathematical equivalent of traditional shaders.
69
00:11:41.772 --> 00:11:57.272
That's crazy. I remember we had to build a GLSL shader for work to replicate fur back in 2012 or something like that, you know, pre-AI obviously, and it was so difficult to do.
70
00:11:57.552 --> 00:12:09.372
We had to like bring in a PhD who understood GLSL and mathematics at that level to be able to just write something that mimicked fur, 'cause, you know, fur is so heavy- Yeah... it uses geometry. Yeah.
71
00:12:09.392 --> 00:12:16.892
So this GLSL shader mimicked fur without any of the geometry. Um, the way that it did it was really, really, uh, tricky.
72
00:12:17.212 --> 00:12:31.260
So it knew the camera normal, it knew the angle of the camera, and it put up these 2D cards at exactly the right angle in layers so that it looked like fur, and then as the camera angle would change, they would just like kinda move the 2D-...
73
00:12:31.320 --> 00:12:39.240
cards in that direction. So we did that, and we thought it was just such groundbreaking and we're like, "We should go get a patent or something." Like, "This is crazy."
74
00:12:39.240 --> 00:12:49.200
[laughs] And now it's, it's just so commoditized, like this is, this is just something that, you know, Claude runs on autopilot essentially. Yeah.
75
00:12:49.280 --> 00:13:12.320
I think that's what I was trying to sort of in- ineloquently say, or the point I tried to make before was what other coding things or things that were built before that kind of worked and had like a s- specific solution but were just really cumbersome to like implement and build out and do anything detailed with, but the foundation is there of this code, but now what's possible with it because of agents, what will this unlock?
76
00:13:12.580 --> 00:13:23.960
Exactly. And to sort of add to the nuance here, there is nothing generative here. Like, we're not doing- Mm... any diffusion. There's no video being generated, uh, images being generated.
77
00:13:24.840 --> 00:13:34.060
It's purely an LLM coding in a way that far exceeds the normal person's capabilities, at least in the immediate near term world. Yeah.
78
00:13:34.140 --> 00:13:43.840
I mean, maybe, you know, you see those elaborate, um, Minecraft creations where it's like, "I built a fully functional like mechanical clock in Minecraft that like took me 20, like 22 years." 20,000 hours.
79
00:13:43.900 --> 00:13:45.000
Yeah, like- Yeah. Yeah.
80
00:13:45.100 --> 00:14:00.900
And it's like sure, I'm sure someone could have spent like a [laughs] really long project coding the, uh, uh, fake Athens, but the fact that, you know, it was very un- un- unrealistic, but the, um, ability to do it now, uh, and what this unlocks- Yeah...
81
00:14:01.020 --> 00:14:10.480
is, is, is gonna be interesting. I'm just blown away, and, uh, the one thing that i- is, is like my inevitable question is like, "Well now what do we do [laughs] with it?"
82
00:14:10.560 --> 00:14:19.400
One thing I wanted to show, though, he did post, he posts a lot of data, which is also nice, but, um, the Alpine one, which is the one that was just Python and the other thing that y- you talked about- Mm-hmm...
83
00:14:20.300 --> 00:14:31.420
the API cost to make it, the total cost was about a little under $3,000. [laughs] Oh, boy. [laughs] In- It's not cheap... when computed. [laughs] So this wasn't... Uh, he was working using Claude.
84
00:14:31.480 --> 00:14:40.100
I mean, obviously once stuff gets distilled and- Uh... newer stuff comes out, then these get less expensive, then that's less expensive. But- Yeah, I wonder where the $3,000 came from.
85
00:14:40.160 --> 00:14:45.900
Like, is it [laughs] his, uh, savings account, or is it like work or, you know, 'cause that, that is a big chunk of change. Yeah.
86
00:14:45.900 --> 00:15:03.020
Or I don't know if he computed it, uh, 'cause like obviously all the subscriptions are subsidized, so I don't know if, because it says cost at API prices, I'm gonna guess this was like a let me use my plan, but if I didn't have a plan and I needed to, you know, I was enterprising running this, what would the equivalent be in buying this through API costs?
87
00:15:03.280 --> 00:15:11.320
I'm gonna guess he u- used a plan, like a Max plan and- Yeah... 'cause the amount you get from Max plan versus what the actual API costs, it's like heavily subsidized. Is the
88
00:15:12.460 --> 00:15:19.660
API cost in general going up faster than inflation? Do you know that? Do you know if that's true? I have no idea. Like- Don't know...
89
00:15:19.700 --> 00:15:29.240
I wonder if, uh, like 100 hours of Claude time in 2025 is less expensive than 100 hours of Claude time in 2026 on the same model.
90
00:15:29.740 --> 00:15:37.460
If you're doing the same model now to before, I mean, every time there's a new model, the old models get deprecated, and then they're usually cheaper. Okay, so then they're, they're...
91
00:15:37.740 --> 00:15:48.140
Okay, maybe that's one of- Like if you were trying to run- Yeah... or they just get, or they just get, um, end of lifed. Yeah. Yeah. So like let's say f- Opus 4.8, 100 hours in 2025 was like $100 let's say, right?
92
00:15:49.000 --> 00:15:59.460
Would Opus 5.5, 100 hours be $100? Uh, wait, sorry. So they're going to Opus 4... Oh, 4.8 was 100 bucks. Would it 5.5 be the same? Yeah. Doubt. I think s- I think they're roughly...
93
00:15:59.500 --> 00:16:05.140
I think whatever the, like, equivalent model, I think they've roughly stayed the same price, but I'm not 100% sure. Mm-hmm. Mm-hmm.
94
00:16:05.580 --> 00:16:14.680
I saw a recent, uh, news article that a lot of companies are cutting back on AI spending now because things have just- Yeah... gotten, uh, really out of control with spending.
95
00:16:14.760 --> 00:16:23.000
Yeah, if you're buying these API costs- Yeah... directly, and it's like the cost of the API, uh, AI versus, you know, just a person to do it is not, not saving much. Yeah. Interesting.
96
00:16:23.240 --> 00:16:33.740
This is all playing out in real time. All right. Model updates. Kling 4. Kling 4. Sorry. Kling 4 is released. I had to do that. I got excited. [laughs] Kling's great. Yeah, Kling 3 is great, and it's kind of been
97
00:16:34.820 --> 00:16:40.880
kind of quiet. Cdance 2 came out and sort of took a lot of the, the- Oh, yeah... but, uh- Yeah... now 4 is out. Yeah, and it took a while because- It was-...
98
00:16:40.940 --> 00:16:50.999
I remember when we did the Kling 03 test back in, uh, like end of last year, uh, I remember you, I used to do the AI roundup. You put a cowboy hat on me and all that. Like that- [laughs]...
99
00:16:51.040 --> 00:16:58.960
that looked okay for back then, like for a video-to-video model. That was, I think, the very first model specifically built for video to video.
100
00:16:59.400 --> 00:17:07.620
It was one of the first video to video, one of the first, I wanna say one of the kind of first reference- Yes... multi, uh, multimodal- Yeah... input kind of models.
101
00:17:07.620 --> 00:17:18.400
'Cause it was that, and then, uh, Wan 2.2 Animate was kind of that, and then Google Omni, and then Cdance 2.5, and now they're back at it. Now they're back. So 4.
102
00:17:18.760 --> 00:17:29.420
Um, main things, I mean, look, uh, honestly, it's a little bit of a catch-up to get to like Cdance 2 parity. Resolution-wise, I don't think all of these are out yet- Mm-hmm...
103
00:17:29.820 --> 00:17:41.940
or at least not on the fall API, but it will be able to do up to 4K. It will also be able to do up to 4K 10-bit HDR outputs. Ooh, that's interesting. Yes. Okay. Yes. Um- And look, and look, look...
104
00:17:42.100 --> 00:17:53.640
input-wise- The aspect ratio, 21:9, cinematic aspect. That's hard to do. Yeah. Two-channel stereo output. I'm kind of curious if that unlocks, if that, what that means, what that does. I mean, this- Input or output...
105
00:17:53.680 --> 00:18:04.840
this got like a Netflix ready thing already. [laughs] Like- Netflix approved. [laughs] Uh, and then input-wise, kind of what you would see on, on Cdance.
106
00:18:04.920 --> 00:18:15.880
Uh, 10 image input, video input, up to five videos with a combined duration of up to 30 seconds, which I think's about the same of what Cdance will take. Uh, audio input, you can give a voice reference- Mm-hmm...
107
00:18:15.920 --> 00:18:17.120
so for character speaking.
108
00:18:17.340 --> 00:18:33.080
You know, I would love to see, 'cause Kling also had the separate specific model, uh, motion transfer that was specifically designed to take a human performance input and map it onto a, uh, AI character- Mm-hmm
109
00:18:33.980 --> 00:18:39.900
I guess you can kind of do it with these models now, but I felt like that one was just good because it was like, it does this one thing and it does it really well.
110
00:18:40.360 --> 00:18:52.060
And I wonder if they're gonna do another version of that model or roll in a feature into this that does it. Is it out on Fal right now, Kling 4? Kling 4 is out on Fal. Oh, dude, I'm testing right after this recording.
111
00:18:52.060 --> 00:19:05.400
[laughs] I would say also the other kind of... You know, we're kind of comparing it and it, it definitely is on par feature-wise with Cdance, but the main thing is pricing, and it's way less- Way cheaper?...
112
00:19:05.600 --> 00:19:13.160
than- Oh, that, that's crazy... Cdance. Okay, let's see. So are you looking at about- So this is 720p, audio on, per second.
113
00:19:13.200 --> 00:19:19.560
And this is on Fal, and also Fal's Cdance rates are pretty high compared to where you could get it elsewhere, but it's just a good baseline.
114
00:19:19.780 --> 00:19:31.480
Um, Kling 4.0, uh, about 15 sec- 15 cents a second, whereas Cdance 2.0 is about 30 cents and, um, Cdance 2.5 is 47 cents- Yeah, I mean-... a second... I'm looking at the total.
115
00:19:31.540 --> 00:19:46.200
Like, you're getting 30 seconds at 1080p for half the price, whereas, you know... So it's really a quarter of the cost of Cdance 2.5. No, this co- this column. This is max.
116
00:19:46.260 --> 00:19:55.420
This is just the max resolu- the max settings it can do. Oh, okay. Oh, I was looking at, uh- Uh, this is price... 10, a 10-second column. So, like, uh, if you look- Oh, the 10- Yeah. Okay.
117
00:19:55.480 --> 00:20:07.920
Yeah, the 10-second is, uh, what the cost is at 10 seconds. So Kling 4 is about- It's like half... 50 for 10 seconds. And then it's half of Cdance 2, and it's- Like one third of Cdance 2.5... a third of Cdance 2.5.
118
00:20:08.340 --> 00:20:16.680
How do they do it? I mean, are they losing money on it? Like, what do you, what do you think? Or is it just a better model? Or I don't either. They just figure out how to get the quality at,
119
00:20:17.860 --> 00:20:31.280
uh, with less compute or less cost or- Yeah. Did you see that, uh, is it Huawei? Uh, Huawei is now building GPUs that rival NVIDIA GPUs, like data center level GPUs. Neural processing units. Neural processing units.
120
00:20:31.360 --> 00:20:39.780
Yeah. Yeah. So, like, they're really meant to go into, like, data centers and, um- Okay... obviously, that's gonna be cheaper for a Chinese, um,
121
00:20:40.740 --> 00:20:50.760
AI company to buy Huawei GPUs, put it into a data center in China than to, like, buy NVIDIA GPUs and try to use inference elsewhere or y- you know.
122
00:20:51.060 --> 00:21:00.620
So maybe there's something going on there that we're not privy to, and I'm sure we'll find out about later. Yeah. And yeah, how they can get this quality and put it out at this price.
123
00:21:00.760 --> 00:21:09.800
I just hope, like, you know, for, for the, like the sake of the planet, um, like we get to better and more efficient models 'cause, like, we, we just can't have,
124
00:21:10.900 --> 00:21:22.360
like, one video take, you know, five GPUs for 10 minutes or whatever. Like, that's just too much compute for, like, one video. So I hope, like, the efficiency and everything come very soon.
125
00:21:22.680 --> 00:21:35.840
I think that will, 'cause, you know, as g- going back to your comment about, uh, with the AI costs and pe- um, companies going back or cutting down on the AI costs, uh, and then also the AI companies needing to find a way to actually make money- Make sense of, yeah...
126
00:21:35.900 --> 00:21:45.200
and not have these- Make dollars and make sense. Yeah... so expensive. I mean, I see it as like everything now, yeah, it's to do the state of the art, it's expensive and to have this compute.
127
00:21:45.240 --> 00:21:52.700
But then every time there was like a new state-of-the-art model that comes out, eventually whatever was like the top benchmark then gets...
128
00:21:53.360 --> 00:22:04.860
New models come out, that one is still very capable and but it gets cheaper as new stuff comes out. And then the less expensive model or the, what was the state of the art, you know, Opus 5.5 in a year from now,
129
00:22:06.000 --> 00:22:17.200
maybe it's even deprecated 'cause it's like not even [laughs] have so many new things out that it's not good, but Opus 5.5 is very good, and it will eventually get cheaper once they, uh, you know, figure out how to make it more efficient.
130
00:22:17.460 --> 00:22:17.620
Yeah.
131
00:22:17.660 --> 00:22:31.720
I almost, I almost look at it like the way cars have evolved, high performance cars have evolved, at least in, in, you know, uh, from the '60s on, like the really crazy V8s and V12s were incredibly gas guzzling, right?
132
00:22:31.880 --> 00:22:43.560
And so it just didn't make economical sense to have them. And now that those engines are able to like get 20 miles, 30 miles a gallon, now everybody has V8s in their cars, like big engines in their cars.
133
00:22:43.940 --> 00:22:53.180
So I think like the AI models right now are very, um, resource intensive because it's new. But I think over time that that problem will hopefully solve itself.
134
00:22:53.680 --> 00:23:03.920
It's an economic problem, so I think it'll have to be solved if this is gonna be around. Yeah. Uh, I'm looking at some of their sample clips they posted on the site. Um, I would say honestly, these are
135
00:23:04.880 --> 00:23:15.500
not the best examples. Um, uh, th- this still feels very synthetic. Yeah. And I know it's streaming- Like the, the ship is not at scale [laughs] and- I know it's streaming through you. Uh- Yeah...
136
00:23:15.540 --> 00:23:23.800
so I can't see the best quality I'm looking at. Um, the close-up looks good. Uh, close-ups with AI look really good. It's very good at the detail and getting close-ups.
137
00:23:23.880 --> 00:23:34.400
The wider you get, that's where things always seem to fall apart. I mean, like every time I look at a synthetic human's face, I'm always like, "Man, we never got there with Unreal Engine and MetaHumans." Mm-hmm.
138
00:23:34.420 --> 00:23:45.040
Like as much as we tried, we, we were just so deep in uncanny valley, but AI is just getting so close to being completely real. Yeah. And it seems- Yeah...
139
00:23:45.060 --> 00:23:52.460
like this 15-second clip we're watching that's sort of like an action sequence is a one-shot kind of deal, which is impressive. Uh- Mm-hmm...
140
00:23:52.660 --> 00:24:02.600
but you know, I mean, I think, I think the big thing is it's like on par with maybe Cdance 2, but- Yeah... way cheaper. Okay. Why don't we do a test- Yeah... while, since it's out, out today? Okay.
141
00:24:02.760 --> 00:24:13.560
I'll run a couple of tests with Fal and, uh, with Cdance and, uh- This looks, uh-... Kling 4. This look, this look like from Hacks, but it's not Hacks. Ooh, I like the handheld camera. That's a nice touch.
142
00:24:14.620 --> 00:24:24.860
A good- And actually-... good destruction there... this is better 'cause one thing I noticed with Kling 3 a lot was, um, handheld cams and camera pans. It would have this very weird synthetic motion blur,
143
00:24:25.860 --> 00:24:33.780
and this pan definitely feels, uh, like a more natural- Natural motion blur. Yeah... yeah. Yeah. Kling always had this weird-
144
00:24:35.080 --> 00:24:44.260
I actually noticed that on the, on the arrow, uh, example that you showed earlier- Yeah, I did have-... the motion blur on the arrow looked pretty good, yeah. Yeah. Even it came through even on the, on the call here.
145
00:24:44.480 --> 00:24:53.280
Yeah. The other thing I was gonna mention, did you hear anything about NCM or New Concept Model? No, what's that? Okay. You can Google it.
146
00:24:53.380 --> 00:25:06.360
So, yeah, remember, uh, that I often bring up Yann LeCun and JePA, which is a new type of architecture that's not an LLM. Yeah. Uh, Nec- Next Concept Modeling. Next Concept Modeling, yes.
147
00:25:06.580 --> 00:25:14.760
So this is, uh, another type of architecture that is emerging, and it's supposed to be way, way more efficient.
148
00:25:14.820 --> 00:25:27.480
So yeah, you look at the numbers there, generating text to 10 to 31 times faster, seeing four to five times fewer training tokens. So not only is it cheaper to train, but it's cheaper to do inference on. Okay.
149
00:25:27.480 --> 00:25:41.860
It's not a large language model. It's using layers. Uh, I don't know what that means, but, like, this is an upcoming thing that is, um, hopefully gonna sort of put an ease on the, the resource intensive nature of AI.
150
00:25:42.700 --> 00:25:50.420
Okay, cool. Go check it out. And it- Yeah... but is, is this more... We said it's not an LLM, but is this more around training or building- It-... language models?
151
00:25:50.820 --> 00:26:00.440
It is, it is a language model, it's just not the way that... Like, it's not a transformer model- Uh-huh... 'cause that's what LLMs are. Yeah. So this is a new type of architecture, yeah. Okay, cool. Interesting.
152
00:26:00.500 --> 00:26:13.340
I'm kind of also ki- interesting how, you know, like we saw JeV and sort of a different type of, um, model. Yeah. I did a little bit more digging on JeV, um, and uh, I could see its usefulness at scale.
153
00:26:13.400 --> 00:26:25.400
Like, if you have 100 JeVs or 1,000 JeVs working together, you could really start to build incredible logic around it, 'cause it, it's, it's a really simple gating mechanism. Mm-hmm. So, like,
154
00:26:26.500 --> 00:26:37.140
if you're looking at a picture of Joey, right? Uh, JeV is gonna be like, "That's a, that's an adult male. Pass. That's not a child. Pass," right? Like, so it's gonna just do very basic things like that.
155
00:26:37.200 --> 00:26:48.120
It's not gonna be as sophisticated as, like, a VLM or an LLM. No, it's not that, but its advantage is its speed. Speed, right. Speed and cost. Right. It's extremely fast and extremely cheap. Yeah. Exactly.
156
00:26:48.260 --> 00:26:57.360
So the types of stuff where you'd just be... If you gave a photo to ChatGPT and are like, "What is this?" It would- Yeah... eat up a lot of tokens to figure out not a child. That's it. Yeah. Yeah.
157
00:26:57.880 --> 00:27:10.240
So I could see it ve- being very useful for, like, Internet of Things and, like, um, stuff that doesn't really need high processing. Mm-hmm. But it's also fuzzy, unstructured data. Yes. That, like, you know, could...
158
00:27:10.300 --> 00:27:23.360
is not an easy, uh, pass-fail thing to, to give to a traditional codeine. Exactly. All right, other new updates, quick roundup. Ideogram has their first vector model. Ooh, nice. Oh, vector models finally. Wow. Yeah.
159
00:27:23.740 --> 00:27:32.020
I, I actually... I thought they had something before, but I guess not. Um, yeah, no, they're, they're... You... I think the previously you can go from, uh, pixel- Or they can convert...
160
00:27:32.120 --> 00:27:42.680
space to vector space as, as a- Okay... like a conversion. Okay. So you're still generating in pixel space, but this looks like a native vector model- Yeah... so it's generating in vectors. That's great.
161
00:27:42.740 --> 00:27:49.020
Yeah, Ideogram's definitely been focused on, on the graphic design more world. Graphics side, yeah. For sure. Yeah. Um, so- Yeah... yeah, that's a good update.
162
00:27:49.560 --> 00:27:54.060
When I was, um, trying to tinker with this stuff about a year ago, having...
163
00:27:54.240 --> 00:28:06.220
Everybody's asking for, like, a vector model, vector model, and we were able to build some pix- pixel-to-vector conversions, and it was terrible. Was not good. [laughs] So this, this hopefully solves that.
164
00:28:06.660 --> 00:28:19.540
Uh, other update, File has another custom version of a post-trained model, so they took Flux.3 and then post-trained it to improve the quality, and I believe also speed. Uh, so they got Flux.3 Max.
165
00:28:20.060 --> 00:28:27.280
Flux.3, the video model, right? Flux.3, the video model. Yeah. Um- Flux.3's not bad, actually. Yeah. Yeah. And so this is post-trained.
166
00:28:27.300 --> 00:28:40.660
I mean, so in their speed chart, basically Flux.3, the original, would take about two minutes a generation. Their version will take, uh, 10 seconds- Oh, wow... which is a huge leap. That's fast.
167
00:28:40.800 --> 00:28:46.400
And according to the ranking, the quality is also higher. Um- Okay... but, uh, yeah.
168
00:28:46.460 --> 00:28:57.940
I mean, this is really interesting what File's been doing, where they've been taking these models, um, and then making their own custom versions on top of it, and that definitely is also smart to give them, uh, something unique.
169
00:28:57.960 --> 00:29:05.880
The guy's so sweaty. Look at this armpit, armpit stain. Yeah. Yeah, that's a nice touch. Uh, w- so here's my question to you. Like,
170
00:29:07.080 --> 00:29:19.440
if the File version of Flux.3 is better in both speed and quality, why would somebody use the original Flux? Well, you could download the open weights of Flux and use it on your own hardware bill.
171
00:29:19.500 --> 00:29:31.460
I mean, you could do what they did, um, on your own. I'm guessing, like, that there is some licensing play that they- Ah... you know, like if you run it on File, that, like, Black Forest Labs gets a cut.
172
00:29:31.660 --> 00:29:36.520
I mean, 'cause, uh, y- you know, it's... You could download the weights, but, like, their licensing stuff's usually...
173
00:29:37.480 --> 00:29:47.340
I'm thinking, I'm thinking of, um, LTX, where it's like if s- sort of the epic model where it's like, yeah, free to use if you make under a certain amount, but then if you make over a certain amount, like, talk to us about licensing.
174
00:29:47.800 --> 00:30:00.440
I'm guessing it's something similar to that. I don't know for sure, but I would assume there's a cut that Black Forest Labs gets. Yeah. S- they seem to be doing okay with, like, groundbreaking models and stuff. Yeah.
175
00:30:00.460 --> 00:30:07.460
They're another great, uh, AI studio that's, you know, punching above its weight with quality- 100%, yeah... not being a- With, like, a small team. Yeah. Yeah.
176
00:30:07.740 --> 00:30:18.680
And then last one, and thank you for Byte+ for posting this before we started recording. CDance 2.5 is- Ooh... uh, 4K is rolling out. That, that is the big Rolls-Royce right now.
177
00:30:18.740 --> 00:30:31.880
Like, that just dropped, and everybody's like, "Oh, dang." You need to pay a Rolls-Royce amount [laughs] for every generation. Yeah. Uh, it doesn't have pricing posted yet, but I mean- Yeah... 2.5 at 1080, I...
178
00:30:32.440 --> 00:30:45.316
on File is, like, $7 a generation for, like, 10 seconds- Okay... 15 seconds, so. So quadruple that. Yes, maybe 20, maybe- So $30? Yeah, yeah... 20 to $30 a generation.
179
00:30:45.636 --> 00:31:01.676
What we need an- is better pipelines to work at 720, and then once you're like, "Yeah, this is it-" Commit... upres it. Yes. But not- Yeah... as a new generation, have everything locked in.
180
00:31:01.896 --> 00:31:15.436
There's a way, and maybe I could demo, uh, the workflow next time, but, um, we found a pretty decent way to do this using depth maps, where you like, if you had your original footage, you, like, turn it into a depth map, and then you give that plus your original reference images
181
00:31:16.396 --> 00:31:26.856
into, like, the 4K output. And so you get... Depends on the shot, but you get a pretty good, consistent result that's 4K, but that still looks like your original clip- So the 4K-... it had the same movement...
182
00:31:26.856 --> 00:31:37.756
is being conditioned from the depth map of the 720p? Yeah. 'Cause my question was, like, how do you do seed locking so you're getting the exact same result? Yeah, I've tried with these models.
183
00:31:37.836 --> 00:31:46.836
Maybe the, you know, by, uh, CDance will be better, but it's like I've tried where it's like, yeah, if you took the seed you ran on 720 and you ran the same exact seed and same inputs on the 4K, it still is- It's still different.
184
00:31:47.116 --> 00:31:58.056
Yeah, because there's- It's different... yeah, because for every one 720p pixel, you now have to fill eight or 10, whatever- Yeah... 4K pixel. So it's fundamentally a different problem for it to solve.
185
00:31:59.336 --> 00:32:10.196
Yeah, I, I, I mean, that's where, like, upscalers, they make sense as, like, a finishing pass, right? So you do all of this work on getting the right thing at 720p, and then you commit to the upscale later.
186
00:32:10.936 --> 00:32:20.256
But then why would you use CDance 2.5 4K to just mess around? Like, that's a very expensive messing around. It's a lot of [laughs] it's a very pricey experiment.
187
00:32:20.356 --> 00:32:30.736
I have seen people online where they just swear by a workflow of, like, just strictly working and generating in 720, and then using, you know, Topaz or one of the other upres models to finish.
188
00:32:31.176 --> 00:32:39.476
I think there is an advantage. I think it de- look, it depends on the shot. And, uh, the weakest area with AI is still, like, wide shots, lots of detail.
189
00:32:40.076 --> 00:32:51.055
And if you're generating at 720 and there's a lot of detail in that shot, it's, it's shrunk down, it's small, and then if you upres it, it doesn't necessarily know what that is, and sometimes you just don't really get sharp details.
190
00:32:51.476 --> 00:33:05.576
So I think purely generating 4K, the best use case would be, like, just very detailed shots, crowds, buildings with lots of lines, uh, stuff that just- Mm... gets noisy and confused.
191
00:33:05.696 --> 00:33:13.136
Like, if you have a wide establishing shot of, I don't know, 1970s New York City. 1970s New York- [laughs]... with a bunch of people walking down the street- Yeah... with the taxi cabs going by.
192
00:33:13.156 --> 00:33:22.616
You need pixels on those guys in the back, yeah. Yeah. Yeah. And it's like if you had a 720 and then you tried to upres it- Right... it just doesn't always know what those pixels should be. And if you're purely...
193
00:33:22.676 --> 00:33:33.376
If you're giving this 4K reference input images and you're giving this all 4K inputs and you're going out to 4K- Mm... I, I, I think you will get a better result than- No, you're absolutely right... 720 to upres.
194
00:33:33.416 --> 00:33:40.656
Dude, you, this totally makes sense. Um, but if you're doing a close-up, if you're doing, like, close-ups of faces and stuff, I don't think you're gonna have... You're just gonna burn a lot of money.
195
00:33:40.676 --> 00:33:50.496
I don't think there's any benefit in, um, purely generating in 4K. Yeah, so, like, you pull the lever when you need to kind of a thing, and it's nice to have in your back pocket. Yeah, yeah.
196
00:33:50.896 --> 00:33:59.616
It, it's also, like, a, I think a little bit of, like, pushing their weight around, like, "Look at us. We got 4K." 4K. Yeah, huh. Do you have 4K? Do you have 4K? We got 4K. You...
197
00:33:59.616 --> 00:34:10.736
[laughs] We will give you an entire data center to spin up your clip, 'cause it's gonna need- It's got a center [laughs]... to generate your CDance 2.5 4K clip. Yeah. Okay. Well, I mean, it's all progress.
198
00:34:10.856 --> 00:34:12.736
Let's see where it goes. We'll see where it goes.
199
00:34:13.036 --> 00:34:23.856
Links to everything we talked about at denoisepodcast.com If there's anything that is incorrect, it's just our opinion, but we would love to hear about it on the YouTube comments. So hit us up on the comments.
200
00:34:24.016 --> 00:34:33.056
We did have some questions, uh, uh, throughout the episode, so you're, if you're more of an expert on this stuff, we'd love to hear it and get educated. Yeah, if you're a Three.js expert, let us know- Yes...
201
00:34:33.076 --> 00:34:38.976
in the comments what you think. [laughs] Uh, all right, thanks everyone. We will catch you in the next episode.
