SF Compute: Commoditizing Compute to solve the GPU Bubble forever

WEBVTT
1
00:00:00.080 --> 00:00:12.880
[ on-hold music] Hey, everyone. Welcome to the Living Space Podcast. This is Alessio, partner and CTO of Decibel, and I'm joined by my co-host Swyx, founder of Small AI.
2
00:00:13.180 --> 00:00:20.740
Hey, and today we're so excited to be finally in the studio with Evan Conrad from SF Compute. Welcome. Hello. How goes it? [laughs] How are we doing?
3
00:00:20.900 --> 00:00:37.660
Uh, I've been fortunate enough to be your friend before you're famous, and also we've hung out at, like, various social things, so it's, it's really cool to see that SF Compute, like, is, is coming into its own thing and it's, uh, you know, it's, it's a significant presence- [laughs] At least in the San Francisco community, which of course it's in the name, so you couldn't help but be.
4
00:00:38.120 --> 00:00:48.660
I- indeed. Indeed. Uh, I think we have a long way to go, but yeah, thanks. Of course. Yeah. Yeah. One way I w- was thinking about kicking off this conversation is we will likely release this right after CoreWeave IPO.
5
00:00:48.880 --> 00:00:58.500
Uh-huh. And, and, uh, I was watching... I was looking, doing some research on you. You did a talk at The Curve. Yeah. I think I may have been viewer number seventy. [exhales] It was a great talk. Uh, thank you.
6
00:00:58.540 --> 00:01:00.760
More people should go see it, Evan Conrad at The Curve.
7
00:01:01.120 --> 00:01:10.360
But we're-- we have, like, three orders of magnitude more people, and I just wanted to, to highlight, like, what is your analysis of what CoreWeave did that, that went so right for them?
8
00:01:11.080 --> 00:01:15.200
Sell locked-in long-term contracts and don't really do much short-term at all.
9
00:01:15.640 --> 00:01:30.640
I think, like, a lot of people had this assumption that GPUs would work a lot like CPUs, and the, like, standard business model of any sort of CPU cloud is you buy commodity hardware, then you layer on services, um, that are mostly software, and that gives you high margins.
10
00:01:30.920 --> 00:01:34.840
Um, and pretty much all your value comes from those services, not really the underlying compute in any capacity.
11
00:01:35.240 --> 00:01:40.700
And because it's commodity hardware and it's not actually that expensive, most of that can be, um, sort of on-demand compute.
12
00:01:40.740 --> 00:01:49.700
And while you do want locked-in contracts for folks, it's mostly just to sort of de-risk your sit-situation. It helps you plan revenue because you don't know if people are gonna scale up or down.
13
00:01:50.180 --> 00:01:55.540
But fundamentally, people are, like, buying hourly, and that's how your business is structured. Um, and you're gonna make fifty percent margins or higher.
14
00:01:56.160 --> 00:02:03.980
This, like, doesn't really work, um, in GPUs, and the reason why it doesn't work is because you end up with, like, super price-sensitive customers.
15
00:02:04.560 --> 00:02:09.900
And that isn't because necessarily it's just way more expensive, though that's totally the case.
16
00:02:10.280 --> 00:02:16.840
So in a CPU cloud, you might have, like, you know, let's say if you had a million dollars of hardware, in GPUs you have a billion dollars of hardware.
17
00:02:17.300 --> 00:02:21.620
And so your customers are buying, um, at much higher volumes, um, than you'd otherwise expect.
18
00:02:21.640 --> 00:02:34.360
And it's also smaller customers who are buying at higher am-amounts of volume, so relative to the, what they're spending in general. But in GPUs in particular, your customer cares about the scaling law behind it.
19
00:02:35.100 --> 00:02:43.880
So if you take, like, Gusto, for example, or Rippling or an HR service like this, when they're buying from an AWS or a GCP, um, they're buying CPUs and they're running web servers.
20
00:02:44.400 --> 00:02:55.220
Those web servers, they kind of buy up to the capacity that they need. They buy enough, like, CPUs, and then they don't buy any more. Like, they don't buy any more at all.
21
00:02:55.420 --> 00:03:01.460
And it's- Yeah, you have a chart that goes like this and then flattens. Correct. And it, and it's, like, a complete flat. It's not even, like, an incremental tiny amount.
22
00:03:01.500 --> 00:03:10.860
It's not like you could just, like, turn on some more nodes and then suddenly, you know, they would make a incremental amount of mo-money more. Like, Gusto isn't gonna make, like, you know, five percent more money.
23
00:03:10.920 --> 00:03:15.600
They're gonna make zero, like, literally zero money from every incremental GP or CPU after a certain point.
24
00:03:15.960 --> 00:03:23.140
This is not the case for anyone who is training models, and it's not the case for anyone who's doing test time inference or, like, inference that has, uh, scales at test time.
25
00:03:23.580 --> 00:03:30.540
Because, like, y-you, your scaling laws mean that you may have some diminishing returns, um, but your, there's always returns.
26
00:03:30.920 --> 00:03:36.600
Adding GPUs always means your model does actually get better, um, and that actually does translate into revenue for you.
27
00:03:36.940 --> 00:03:44.540
And then for test time inference, you actually can just, like, run the inference longer and get a better performance. Or maybe you can run more customers, um, faster and then charge for that.
28
00:03:44.880 --> 00:03:47.900
It actually does translate into revenue. Every incremental GPU translates to revenue.
29
00:03:48.300 --> 00:04:02.060
And what that means from the customer's perspective is you've got, like, a flat budget and you're trying to max the amount of GPUs you have for that budget, and that's very distinctly different than, like, where a Gusto or a Rippling might think, where they think, "Oh, we need this amount of CPUs.
30
00:04:02.540 --> 00:04:08.080
How do we, you know, reduce our amount of money that we're spending on this to get the same amount of CPUs?"
31
00:04:08.600 --> 00:04:18.720
But what that translates to is customers who are spending in really high volume, but also customers who are super price sensitive, who don't give a shit... Can I swear on this? Can I swear? Yeah, yeah.
32
00:04:18.760 --> 00:04:27.660
Um, who, who don't give a shit at all about your software because a ten percent difference in a billion dollars of hardware is, like, a hundred million dollars of value for you.
33
00:04:28.080 --> 00:04:38.380
So if you have a ten percent margin increase because you have great software, um, on your billion dollars, the customers are that price sensitive. They will immediately switch off, um, if they can.
34
00:04:38.760 --> 00:04:47.300
Because why wouldn't you? You would just take that hundred million dollars, you'd spend fifty million dollars on hiring a software engineering team to replicate anything that you possibly could.
35
00:04:47.800 --> 00:04:57.280
So that means that the best way to make money in GPUs [chuckles] was to do basically exactly what CoreWeave did, um, which is go out and sign only long-term contracts.
36
00:04:57.320 --> 00:05:11.440
Pretty much ignore the bottom end of the market completely, and then maximize your long-term contracts with customers who are, um, who don't have credit risk, who won't sue you if, um, or are unlikely to sue you for, like, frivolous reasons.
37
00:05:12.020 --> 00:05:19.720
And then because they don't have credit risk and they won't sue you for frivolous reasons, you can go back to your lender and you can say, "Look, this is a really low-risk situation for us to do.
38
00:05:20.140 --> 00:05:26.680
Um, you should give me prime, like prime interest rate. You should give me the lowest cost of capital you possibly can." And when you do that, you just, like, make tons of money.
39
00:05:26.960 --> 00:05:37.460
Um, the problem that I think lots of people are gonna talk about with CoreWeave is it doesn't really look like a cloud provider financially. It, it also doesn't really look like a software company financially.
40
00:05:37.720 --> 00:05:48.900
It's a bank. It's a bank. It's, it's a real estate company. Uh, and it's very hard to not be that. The problem of that that people have tricked themselves into is thinking that CoreWeave is a bad business.
41
00:05:49.000 --> 00:05:57.360
I don't think CoreWeave is an explicitly a bad business. Um, there's a bunch of people... There's kinda like two versions of the CoreWeave take at the moment. There's, "Oh my God, CoreWeave amazing.
42
00:05:57.380 --> 00:06:00.620
Uh, CoreWeave is this great cloud, new cloud provider, competitive with the hyperscalers."
43
00:06:01.000 --> 00:06:07.780
And to some extent, this is true from a structural perspective, like they are indeed a real sort of thing against the cloud providers in this particular category.
44
00:06:08.320 --> 00:06:16.948
And the other take is, "Oh my gosh, CoreWeave is this horrible business," and so on and blah, blah, blah. And I think it's just like- A set of perception or perspective.
45
00:06:17.428 --> 00:06:24.668
If you think CoreWeave's business is supposed to look like the traditional cloud providers, you're gonna be really upset to learn that GPUs don't look like that at all.
46
00:06:24.768 --> 00:06:27.088
And in fact, for the hyperscalers, it doesn't look like this either.
47
00:06:27.508 --> 00:06:35.948
My intuition is that the hyperscalers are probably gonna lose a lot of money, and they know they're gonna lose a lot of money, um, on reselling Nvidia GPUs at least. Hyperscalers, spe- I wanna, uh- Yeah...
48
00:06:35.968 --> 00:06:45.008
Microsoft, AWS, Google. Correct, yeah. Okay. The Microsoft, AWS, and Google, um- Does Google resell? I mean, Google has TPUs, but- Google has TPUs, but I think they-- you can also get H100s from them and so on.
49
00:06:45.688 --> 00:06:54.068
But there are, like, two ways they can make money. One is by selling to small customers who aren't actually buying in any serious volume.
50
00:06:54.448 --> 00:07:00.868
They're testing around, they're playing around, and if they get big, they're immediately going to do one of two things.
51
00:07:01.328 --> 00:07:10.648
They're gonna ask you for, um, a discount because they're not gonna pay your crazy sort of margin that you have locked into your business, because for CPUs, you need that.
52
00:07:11.068 --> 00:07:16.128
They're gonna pay your, your massive per-hour price, and so they want you to sign a long-term contract.
53
00:07:16.648 --> 00:07:27.088
And so that's your other way that you can make money is you can, um, you can basically do exactly what CoreWeave does, which is have them pay as much as possible up front and lock in the contract for a long time, or you can have small customers.
54
00:07:27.568 --> 00:07:42.848
But the problem is that, like, for a hyperscaler, the GPUs to sell on the low margins relative to what your, um, your other business, your CPUs are, is a worse business [chuckles] than what you are currently doing.
55
00:07:43.648 --> 00:07:53.248
'Cause, like, you could have spent the same money on those GPUs, and you could have trained model, and you could have made a model on top of it and then turned that into a product and had high margins from your product.
56
00:07:53.668 --> 00:07:59.168
Or you could have taken that same money and you could have competed with Nvidia, um, and you could have cut into their margin instead.
57
00:07:59.588 --> 00:08:14.008
But just simply reselling Nvidia GPUs doesn't work like your CPU business where you're able to capture high margins from big customers and so on, and then they never leave you because your customers aren't actually price sensitive, and so, um, they won't switch off if your prices are a little higher.
58
00:08:14.568 --> 00:08:25.788
Um- You actually had a really nice chart, again on that talk, of this two by two- Sure... of like where you want to be, and you also had some hot takes on who's making money and who isn't. Sure.
59
00:08:25.828 --> 00:08:33.908
So CoreWeave locked up long-term contracts. Get that. Yes. Maybe share your mental framework. Just verbally describe it 'cause we're trying to help the audio listeners as well. Sure.
60
00:08:33.948 --> 00:08:43.768
People can look up the chart if they want to. Sure. Okay, so this is a graph of interest rates, and on the y-axis is a probability you're able to sell your GPUs from zero to one. Mm.
61
00:08:44.028 --> 00:08:54.308
And on the x-axis, it's how much they'll depreciate in cost from zero to one. Uh, yeah. And then you had iso cost curves or iso interest rate curves. Yeah. So there's...
62
00:08:54.508 --> 00:09:05.948
They, they kinda shape in a sort of concave fashion. Yeah. Um, the lowest interest rates, uh, enable the most aggressive form of this cost curve. Yeah.
63
00:09:05.968 --> 00:09:18.268
And the, the higher interest rates go, the more you have to push out to the top right. Yeah. And then you had some analysis of where every player sits in this, including CoreWeave, but also Together and Modal- Yeah...
64
00:09:18.328 --> 00:09:21.688
and all these other guys. I thought that was super insightful, so I just wanted you to- Thank you... elaborate.
65
00:09:22.228 --> 00:09:29.848
Basically, it's like a graph of risk, um, and the genres of places where you can be and what the risk is associated with that.
66
00:09:30.388 --> 00:09:40.308
The, like, optimal thing for you to do, if you can, is to lock in long-term contracts that are paid all up front or in with a situation in which you trust the other party to pay you over time.
67
00:09:40.748 --> 00:09:48.348
So if you're, you know, selling to Microsoft or something, or OpenAI. Which are together seventy-seven percent of the revenue of CoreWeave. [chuckles] Yeah.
68
00:09:48.388 --> 00:09:58.708
So if, if you're doing that, um, that's a great business to be in because your interest rate that you can pitch for is really low because no one thinks Microsoft is gonna default.
69
00:09:59.128 --> 00:10:14.148
And, like, maybe OpenAI will default, but the backing by Microsoft kinda helps you, and I think there's enough, like, generally it looks like OpenAI is winning, that you can make a-- It's just a much better case than if you're selling to the pre-seed startup that just raised thirty million dollars or something pre-revenue.
70
00:10:14.568 --> 00:10:19.468
It's, like, way easier to make the case that the OpenAI is not gonna default than the pre-seed startup.
71
00:10:19.968 --> 00:10:29.888
And so the optimal place to be is selling to the maximally low-risk customer for as long as possible, and then you never have to worry about depreciation, um, and you make lots of money.
72
00:10:30.408 --> 00:10:36.107
Um, the less good place to be is you could sell long-term contracts to people who might default on you.
73
00:10:36.628 --> 00:10:45.048
And then, um, if you're not bringing it to the present, so you're not, like, saying, "Hey, you have to pay us all up front," then you're in this, like, more risky territory. Um- So this is the top left of the chart?
74
00:10:45.488 --> 00:10:53.208
Uh, if I have the chart right, maybe. Large, large contracts paid over time. Yeah, large contracts paid over time is, like, top left. So it's, uh, m-more risky, but you could still probably get away with it.
75
00:10:53.628 --> 00:10:58.428
And then the other opportunity is that you could sell short-term contracts, um, for really high prices.
76
00:10:58.928 --> 00:11:06.008
And so lots of people tried that too because this is actually closer to the original business model that people thought would work in cloud providers.
77
00:11:06.368 --> 00:11:14.148
For CPUs, it works for, for CPUs, but it doesn't really work for GPUs. And I don't think people were trying this because they were thinking about the risk associated with it.
78
00:11:14.228 --> 00:11:22.808
I think a lot of people who just come from a software background have not really thought about, like, cogs or margins or inventory risk or th-things that y-you have to worry about in the physical world.
79
00:11:23.428 --> 00:11:27.108
And I think they were just, like, copy-pasting the same business model onto CPUs.
80
00:11:27.168 --> 00:11:45.468
And also, I remember fundraising, uh, like a few years ago, and I know based on, like, what we knew other people were saying who were in a very similar business to us versus what we were saying, and we know that our pitch was way worse at the time 'cause in the beginning of SF Compute, um, we, we looked very similar to, um, pretty much every other GPU cloud.
81
00:11:45.928 --> 00:11:56.028
Not on purpose, but sort of accidentally. And I know that the correct pitch to give to an investor was, "We will look like a traditional CPU cloud with high margins, um, and we'll sell to everyone."
82
00:11:56.528 --> 00:12:00.648
And that is a bad business model because your customers are price sensitive.
83
00:12:01.048 --> 00:12:14.728
And so what happens is if you sell at high prices, which is the price that you would need to sell at in order to de-risk your loss on the depreciation curve, and specifically what I mean by that is, like, let's say you're selling at, like, five dollars an hour and you're paying a dollar fifty an hour for the GPU under the hood.
84
00:12:14.808 --> 00:12:20.508
Um, it's a little bit different than that, but, you know, nice numbers. Five dollars an hour, dollar fifty, um, an hour. Great. Excellent.
85
00:12:20.828 --> 00:12:28.028
Well, you're charging a really high price per GPU hour because over time the price will go down and you'll get competed out.
86
00:12:28.348 --> 00:12:39.928
Um, and what you need is to make sure that you never go under or if you do go under your underlying costs, you've made so much money in the f-first part of it that the later end of it, like, doesn't matter because from the whole structure of the deal you've made money.
87
00:12:40.568 --> 00:12:52.144
The problem is that just- You think that you're gonna be able to retain your customers with software, and actually what happens is your customers are super price sensitive and push you down, and push you down, and push you down, and push you down, um, that they don't care about your software at all.
88
00:12:52.564 --> 00:13:03.784
And then the other problem that you have is you have, um, really big players like the hyperscalers who are looking to win the market, and they have way more money than you, and they can push down on margin much better than you can.
89
00:13:04.084 --> 00:13:13.844
And so if they have to, and they do- they don't necessarily all the time, um, I think they actually keep probably a higher margin, but if they needed to, they could totally just, like, wreck your margin at any point, um, and push you down.
90
00:13:14.143 --> 00:13:26.604
Which meant that that quadrant over there where you're charging a high price, um, and just to make up for the risk, completely got destroyed, like, did not work at all for many places because of the price sensitivity and because people could just shove you down.
91
00:13:27.004 --> 00:13:34.984
Instead, that pushed everybody up to the top right-hand corner of that, which is selling short-term contracts for low prices, um- Paid over time. Yeah...
92
00:13:35.224 --> 00:13:51.324
paid over time, um, which is the worst place to be in, um, the worst financial place to be in because it has the highest interest rate, um, which means that your, um, your costs go up at the same time your, uh, your incoming cash goes down and squeezes your margins and squeezes your margins.
93
00:13:51.884 --> 00:13:58.044
The nice thing for, like, a CoreWeave is that most of their business is over on the, on the other sides of those quadrants that the ones that survived.
94
00:13:58.544 --> 00:14:07.664
The only remaining question I have with CoreWeave, and I promise I get to SF Compute, and I promise this is relevant to SF Compute in general 'cause the framework is important, right? Sure. To understand the company.
95
00:14:08.224 --> 00:14:15.944
So why didn't Nvidia or Microsoft, both of which have more money than CoreWeave- Yeah... do CoreWeave? Right? Why didn't they do CoreWeave?
96
00:14:16.264 --> 00:14:29.204
Why have this middleman when either Nvidia or Microsoft have more money than God, and they could have done an internal CoreWeave, which is e- effectively like a self-funding vehicle, like a financial instrument.
97
00:14:29.504 --> 00:14:36.764
Why, why does there have to be a third party? Your question is like, why didn't, um- Nvidia... Microsoft- Or why didn't either, either one of those... Nvidia just do CoreWeave?
98
00:14:36.804 --> 00:14:48.284
Why didn't they just set up their own cloud provider? Yeah. Um, I, I think, and I don't know, and so correct me if I'm wrong, and, um, lots of people will, uh, have different opinions here or, or... I mean, not opinions.
99
00:14:48.324 --> 00:14:56.264
They'll have actual facts that differ from my facts. Not-- Those aren't opinions. Those are actually indeed different differences of reality. Is that Nvidia doesn't wanna compete with their customers.
100
00:14:56.604 --> 00:14:59.064
They make a large amount of money by selling to existing clouds.
101
00:14:59.504 --> 00:15:07.704
If they launched their own CoreWeave, then it would make it much harder for them to sell to the hyperscalers, and so they have a complex relationship with there. So not great for them.
102
00:15:08.084 --> 00:15:14.124
Second is that, at least for a while, I think they were dealing with antitrust concerns or fears, um- [chuckles]... that if they're going through...
103
00:15:14.464 --> 00:15:21.884
If they own too much layers of the stack, I could imagine that could be a problem for them. I don't know if that's actually true, but that's where my mind would go, I guess. Mostly, I think it's the first one.
104
00:15:21.944 --> 00:15:28.364
Um, it's that- Yeah... they, they would be competing directly with their primary customers. Or... Then Microsoft coulda done it, right? Like, that's the other question. Yeah.
105
00:15:28.424 --> 00:15:34.284
So Microsoft, um, didn't do it, and my guess is that Nvidia doesn't, doesn't want Microsoft to do it.
106
00:15:34.504 --> 00:15:45.364
And so they would limit the capacity because, um, from Nvidia's perspective, both they don't wanna necessarily launch their own cloud provider 'cause it's competing with their customers, but also they don't want only one customer, um, or only a few customers.
107
00:15:45.564 --> 00:15:55.064
It's really bad for, um, Nvidia if you have customer concentration, and Microsoft and, um, Google and, um, Amazon, like Oracle too, like buy up your entire supply.
108
00:15:55.264 --> 00:16:02.044
Um, and then you have four or five customers or so, um, who pretty much get to set prices. And- Monopsony. Yeah, a monopsony.
109
00:16:02.084 --> 00:16:08.984
And so the optimal thing for you is a diverse set of customers, um, who all are willing to pay at whatever price, because if you don't, somebody else will.
110
00:16:09.504 --> 00:16:16.064
And so it's really optimal for Nvidia to have lots of other customers who are all competing against each other. Great. Yeah. Just wanted to establish that.
111
00:16:16.124 --> 00:16:19.444
It's unintuitive for people who have never thought about it, and you think about it all day long. Yeah.
112
00:16:19.624 --> 00:16:30.744
[laughs] The last thing I'll, I'll call out from the talk, which is kinda cool, and then I, I promise we'll get to, uh, SF Compute, is why will, uh, DigitalOcean and Together lose money on their clusters?
113
00:16:31.244 --> 00:16:38.024
Why will DigitalOcean [chuckles] and Together lose money on their clusters? Um, I'm gonna start by clarifying that all of these businesses are excellent and fantastic.
114
00:16:38.184 --> 00:16:43.924
Um, that Together and DigitalOcean and Lambda, I think, are wonderful businesses who do, um, like, build excellent products.
115
00:16:44.384 --> 00:16:49.804
Um, but my general intuition, um, is that if you try to couple the software and the hardware together, you're gonna lose money.
116
00:16:50.344 --> 00:17:06.844
Um, that if you go out and you buy a long-term contract from, from someone and then you layer on services, or you buy the hardware yourself and you spin it up and you get a bunch of debt, you're gonna run into the same problem that everybody else did, the pr- same problem we did, same problem the hyperscalers are doing, which is you cannot, um, add software and make high margins like a cloud provider can.
117
00:17:07.024 --> 00:17:16.884
Um, you can pitch that into investors and it'll totally make sense, and it's like the ch- correct play in, in CPUs, but there isn't software you could make to make this occur.
118
00:17:17.044 --> 00:17:20.804
Like, if you are spending a billion dollars on hardware, you need to make a billion dollars of software.
119
00:17:21.064 --> 00:17:30.484
There isn't a billion dollars of software that you can realistically make, and if you do, you're gonna look like SAP. Um, and like, um, not, that's not a knock on SAP. SAP makes a fuck ton of money, right?
120
00:17:30.804 --> 00:17:43.884
So there just aren't that many pieces of software that you could make, that you can realistically sell like a billion dollars of software, and you're probably not gonna do it to price-sensitive customers who are spending their entire budget already on compute.
121
00:17:44.124 --> 00:17:55.104
They don't have any more money to give you. It's a very hard proposition to do. And so many parties have been trying to do this, like buy their own compute, um, because that's what a traditional cloud does.
122
00:17:55.864 --> 00:18:04.144
It doesn't really work for them. You know that meme where there's, um, like the Grim Reaper and he's like knocking on the door? Mm-hmm. And then he keeps knocking on the next door?
123
00:18:04.504 --> 00:18:15.424
Um, we have just seen door after door after door of the Grim Reaper, um, comes by and the economic realities of the compute market, um, come knocking.
124
00:18:15.764 --> 00:18:27.324
And so the thing we encourage folks to do is if you are thinking about buying a big GPU cluster and you are gonna layer on software on top, don't. There are so many dead bodies in the wake there.
125
00:18:27.344 --> 00:18:38.324
[chuckles] We would recommend not doing that. And we, as SF Compute, our entire business is structured to help you not do that. It's helped, um, disintegrate these. The GPU clouds are fantastic real estate businesses.
126
00:18:38.484 --> 00:18:47.704
If you treat them like real estate businesses, you will make a lot of money. Um, the cloud pr- services you can ta- make on that, all the software You wanna make on that? You can do that fantastically.
127
00:18:48.044 --> 00:18:57.684
Um, if you don't own the underlying hardware, if you mix these businesses together, you get shot in the head. Um, but if you combi-- if you split them, um, and that's what the market does, it helps you split them.
128
00:18:57.724 --> 00:19:05.884
It allows you to buy like layer on services, but just buy from the market. You can make lots of money. So companies like Modal who don't own the underlying compute, like they don't own it.
129
00:19:06.644 --> 00:19:14.644
Lots of money, fantastic product. And then, um, companies like CoreWeave, um, who are functionally like really, really good real estate businesses. Lots of money, fantastic product.
130
00:19:14.744 --> 00:19:25.144
Like, but if you combine them, you die. That's the economic reality of compute. I think it also, it also splits into training versus inference, uh- Sure... which different kinds of workloads. Yeah, yeah. True.
131
00:19:25.144 --> 00:19:29.464
Um- And then, um, yeah, one comment about the price sensitivity thing before we leave this topic.
132
00:19:29.964 --> 00:19:40.044
I wanna credit Martin Casado for m- coining or naming this thing, which is like, you know, you, you said, you said this thing about like you, y- you don't have room for a ten percent margin on GPUs- Yep... for software.
133
00:19:40.264 --> 00:19:46.324
Yep. And Martin actually played it out further. It's, it's the first one I ever saw doing this a- at large enough runs.
134
00:19:46.384 --> 00:19:55.144
So let's say GPT-4 and o1 both had total training costs of like a five hundred million dollars- Yeah... is, is the rough estimate. When you get the five billion dollar runs- Yes...
135
00:19:55.164 --> 00:20:06.804
when you get the fifty billion dollar runs, um, it is actually makes sense to build your own chips. Like to, for OpenAI to get into chip design, which is-- it's so funny to like, I would make an ASIC for this run. Yeah.
136
00:20:06.824 --> 00:20:18.404
[laughing] Um, maybe. Um, I, I think a caveat of that that is not super well thought about is that only works if you're really confident. Yeah. It only works if you really know which chip you're gonna do.
137
00:20:18.924 --> 00:20:26.324
If you don't, then it's a little harder. So it makes... in my head, it makes more sense for inference, um, where you've already established it. But for training, um, there's so much like experimentation.
138
00:20:26.344 --> 00:20:35.664
You need generality, yeah. Yeah. Yeah. The generality is much more useful. In some sense, you know, Google is like six generations into the CPUs and- Yeah. Yeah. Okay, cool. Um, maybe we should go into SF Compute now.
139
00:20:35.664 --> 00:20:42.424
[laughs] Sure, yeah. Um, yeah, so you kinda talked about the different providers. Why did you decide to go with this approach?
140
00:20:42.524 --> 00:20:53.864
And maybe talk a bit about how the market dynamics have evolved since you started the company. So originally, we were not doing this at all. Um, we were definitely, like, forced into this to some extent.
141
00:20:54.484 --> 00:20:59.644
SF Compute started because, uh, we wanted to go train models for music, um, and audio in general.
142
00:20:59.704 --> 00:21:07.224
We, we were gonna do a sort of generic audio model, and at some points, and then we were gonna do a music model at some point. So it was early company. We didn't really spec down on a particular thing.
143
00:21:07.624 --> 00:21:13.924
But yeah, we were gonna do a music model and audio model. First thing that you do when you start any AI lab is you go out and you buy a big cluster.
144
00:21:14.384 --> 00:21:20.324
The thing we had seen everybody else do was they went out, and they raised a really big round, and then they would get stuck.
145
00:21:20.604 --> 00:21:31.584
Um, because if you raise the amount of money that you need to train a model initially, like, you know, the fifty million dollar pre-seed, pre-revenue, um, your valuation is so high or you get diluted so much, um, that you can't raise the next round.
146
00:21:32.024 --> 00:21:43.104
Um, and that's a very big ask to make. And also, I don't know, I, I felt like we just felt like we couldn't do it. We probably could have, in retrospect, but, um, I think, one, we didn't really feel like we could do it.
147
00:21:43.344 --> 00:21:47.744
Two, it felt like if we did, we would've been stuck later on. We didn't wanna raise a big round.
148
00:21:48.384 --> 00:22:01.744
And so instead, we thought surely by now, um, w- we would be able to just go out to any provider and buy like a traditional CPU cloud w- would sell, offer you, and just buy, like, on demand, um, or by, like, a month or so on.
149
00:22:01.884 --> 00:22:11.824
And this worked for, like, small incremental things, and I think this is where we were basing it off. We just, like, assumed we could go to, like, Lambda, um, or something and, like, buy thousands of, at the time, A100s.
150
00:22:12.364 --> 00:22:21.184
And this just, like, was not at all the case. So we started doing all the sales calls with people, and, uh, we said, "Okay, well, can we just get like month to month? Can we get like one month of compute or so on?"
151
00:22:21.684 --> 00:22:30.564
Everyone told us at the time, "No, you need to have a year-long contract or longer, or you're out of luck. Sorry." And at the time, we were just, like, pissed off.
152
00:22:30.604 --> 00:22:32.284
We're like, "Why won't nobody sell us a month at a time?"
153
00:22:32.624 --> 00:22:41.624
Nowadays, we totally understand why, because it's the same economic reason, 'cause if you-- if they had sold us the month to month or so on, um, and we canceled or so on, they would have massive risk on that.
154
00:22:42.104 --> 00:22:51.784
And so the optimal thing to do was to only to just, like, completely abandon this section of the market. We didn't like that. [laughs] Um, so our plan was we were gonna buy a year-long contract anyway.
155
00:22:52.024 --> 00:22:59.084
We would use a month, and then we would sublease the other eleven months. And we were locked in for a year, but we only had to pay on every individual month.
156
00:22:59.404 --> 00:23:07.904
And so we did this, but then immediately we said, "Oh, shit, now we have a cloud provider, not a, like, training models company- Right... not an AI lab."
157
00:23:08.004 --> 00:23:14.264
Um, because every thirty days, we owed about five hundred thousand dollars or so, and we had about five hundred thousand dollars in the bank.
158
00:23:14.624 --> 00:23:21.484
Um, so that meant that every single month, if we did not sell out our cluster, uh, we would just go bankrupt. So that's what we did for the first year of the company.
159
00:23:21.924 --> 00:23:26.344
And when you're in that position, y- you try to think how in the world do you get out of that position.
160
00:23:26.904 --> 00:23:33.344
What that transitioned to is, okay, well, we tend to be pretty good at, like, selling this cluster every month 'cause we haven't died yet.
161
00:23:33.724 --> 00:23:41.284
And so what we should do is we should go basically be like this broker for other people, and we will be more like a GPU real estate, uh, or like a GPU realtor.
162
00:23:41.784 --> 00:23:56.504
And so we started doing that for a while, where we would go to other people who had-- who was trying to sell, like, a year-long contract with somebody, and we'd go to another person who, like, maybe this person wanted six months and somebody else wanted six months or something, and we'd, like, combine all these people together, um, to make the deal happen.
163
00:23:56.944 --> 00:24:01.024
And we'd organize these, like, one-off bespoke deals, um, that looked like...
164
00:24:01.504 --> 00:24:09.064
Basically, it ended up with us taking a bunch of customers, us signing with a vendor, taking some cut, and then us operating the cluster for people, typically with bare metal.
165
00:24:09.404 --> 00:24:18.244
And so we were doing this, but this was definitely like a, "Oh, shit. Oh, shit. Oh, shit, how do we get out of our current situation?" Um, and less of a, um, like a strategic plan of any sort.
166
00:24:18.604 --> 00:24:27.664
But while we were doing this, since, like, the beginning of the company, um, we had been thinking about how to buy GPU clusters, how to sell them effectively, because we'd seen every part of it.
167
00:24:27.964 --> 00:24:35.824
Um, and what we ended up with was, like, a book of everybody who's trying to buy and everyone who's trying to sell because we were these, like, GPU brokers.
168
00:24:36.304 --> 00:24:45.324
And so that turned into what is today SF Compute, uh, which is a compute market, which we think we are the functionally the most liquid GPU market of any capacity.
169
00:24:45.764 --> 00:24:53.264
Honestly, I think we're the only thing that actually is like a real market, that there's like bids and asks, and there's, um, like a, like a trading engine that combines everything and so on.
170
00:24:53.744 --> 00:25:02.328
I think we're the only place where you can do things that a market should be able to do, like You can go on SF Compute today and you can get thousands of H100s for an hour if you want.
171
00:25:02.608 --> 00:25:05.928
And that's because there is a price for thousands of GPUs for an hour.
172
00:25:05.968 --> 00:25:16.948
That is, like, not a thing you can reasonably do on kind of any other cloud provider because nobody should realistically sell you thousands of GPUs for an hour. Um, they should sell it to you for a year or so on.
173
00:25:17.328 --> 00:25:24.428
But one of the nice things about a market is that you can buy the year on SF Compute, but then if you need to sell back, you can sell back as well.
174
00:25:24.548 --> 00:25:31.408
And that opens up all these little pockets of liquidity where somebody who's just trying to buy for a little bit of time some burst capacity... So people don't normally buy for an hour.
175
00:25:31.428 --> 00:25:38.788
That's not, like, actually a realistic thing, but it's, like, the range. Somebody who wants-- who is like us, um, who needed to buy for a month, can actually buy for a month.
176
00:25:38.828 --> 00:25:45.848
They can, like, place the order and there is actually a price for that, and it typically comes from somebody else who's selling back, um, somebody who bought a longer term contract and is, like, they bought for some period of time.
177
00:25:45.888 --> 00:25:54.068
They, their code doesn't work, and now they need to, like, um, sell off a little bit. What are the utilization rates at which a market like this works?
178
00:25:54.328 --> 00:26:00.028
What do you see the usual GPU utilization rate and, like, at what point- So-... does the market get saturated?
179
00:26:00.348 --> 00:26:10.328
Assuming there are not, like, hardware problems or software problems, the utilization rate is, like, near 100% because the price dips until the utilization is 100%.
180
00:26:10.808 --> 00:26:17.928
So the price actually has to dip quite a lot in order for the utilization not to be. That's not always the case because you just have logistical problems. Um, like,
181
00:26:18.848 --> 00:26:28.148
you get a cluster and parts of the InfiniBand fabric are broken, and there's, like, um, some issue with some switch somewhere, and so you have to take some portion of the cluster offline or, you know, stuff like this.
182
00:26:28.188 --> 00:26:37.388
Like, there's just underlying physical realities of the clusters. But nominally, um, we have better utilization than basically anybody because, um... But that's on utilization of the cluster.
183
00:26:37.848 --> 00:26:48.728
Like, that doesn't necessarily translate into, um... Well, I, I actually do think we have much better overall money made for our underlying vendors than kind of anybody else. We work with the other GPU clouds.
184
00:26:49.048 --> 00:26:59.008
Um, and the basic pitch to the other GPU clouds is, one, we're still your broker, so we can, we can find you the long-term contracts that are at the prices that you want. But meanwhile, your cluster is idle.
185
00:26:59.448 --> 00:27:05.208
And for that, we can increase your utilization and get you more money because we can sell that idle cluster for you.
186
00:27:05.228 --> 00:27:10.988
And then the moment, um, we find the longer, the bigger customer and they come on, you can kick off those people and then go to the other ones.
187
00:27:11.348 --> 00:27:19.928
You get kind of the mix of like sell your cluster at whatever price you can get on the market, and then sell your cluster at the big price that you wanna do for a locked-in long-term contract, which is your ideal business model.
188
00:27:20.408 --> 00:27:29.548
And then the benefit of the whole thing being on the market is you can pitch your customer that they can cancel their long-term contract, which is not a thing that you can reasonably do if you are just the GPU cloud.
189
00:27:29.628 --> 00:27:37.508
If you're just the GPU cloud, you can never cancel your contract because that introduces so much risk that you would otherwise, like, not get your cheap cost capital or whatever.
190
00:27:37.908 --> 00:27:53.188
But if you're selling it through the market or you're selling it with us, then you can say, "Hey, look, you can cancel for a fee," and that fee is the difference between the price of the market and then the price, um, that they paid at, which means that they canceled and you have the ability to offer that flexibility, but you don't have to take the risk of it.
191
00:27:53.328 --> 00:28:04.828
The money's already there and, like, you got paid, but it's just being sold to somebody else. One of our top pieces from last year was talking about the H100 glut- Sure... from all the,
192
00:28:05.808 --> 00:28:15.728
uh, long-term contracts that were not being fully utilized and being put onto the market. Yeah. Uh, you have on here dollar, a dollar per hour- Yeah... uh, contracts as well as it goes up to two.
193
00:28:16.088 --> 00:28:24.008
Actually, I think you were involved. You, you, you were obliquely quoted in that article. I think you remember. Yes, I remember. Because this was hidden. Well, we hid your name- [laughs]...
194
00:28:24.088 --> 00:28:34.308
uh, but then you were like, "Yeah, it's us." Yeah. Could you talk about the supply and demand of H100s? Was that just a normal cycle? Was that like a super cycle because of all the VC funding that went in in 2003?
195
00:28:34.728 --> 00:28:41.428
What was that? Like, GPU prices have come down. Yeah. G- GPU prices have come down. Um- Some, some part of that is normal dep- depreciation cycle.
196
00:28:41.508 --> 00:28:48.728
Some part of that is just there were a lot of f- uh, startups that bought GPUs and never used them, and now they're lending it out and therefore you exist.
197
00:28:49.368 --> 00:28:56.008
There's a lot of, like, various theories as to why this happened. I dislike all of them because they're all kind of like...
198
00:28:56.628 --> 00:29:04.688
They're often said with really high confidence, and I think just the market's much more complicated than that. Of course. And so, uh, everything I'm gonna say is, like, very hedged.
199
00:29:05.248 --> 00:29:21.088
But there was a series of, like, places where a bunch of the orders were placed and people were pitching to their customers and their investors and just the broader market, um, that they would arrive on time, and that is not how the world works.
200
00:29:21.308 --> 00:29:30.188
Um, and because there was such a really quick build-out of things, you would end up with bottlenecks in the supply chain somewhere that has nothing to do with necessarily the chip.
201
00:29:30.588 --> 00:29:37.748
Um, it's like the InfiniBand cables or the NICs or, like, whatever, or you need a bunch of, like, generators or you're, you don't have data center space or like...
202
00:29:37.888 --> 00:29:43.928
But, like, there's always some bottleneck somewhere else. And so a lot of the clusters didn't come online within the period of time.
203
00:29:44.328 --> 00:29:47.928
But then all the bottlenecks got sorted out and then they all came online [chuckles] all at the same time.
204
00:29:48.228 --> 00:29:59.888
So I think you saw a shortage because supply chain hard, and then you saw a increase, uh, or like a, a, a glut, um, because supply chain eventually figured itself out.
205
00:30:00.488 --> 00:30:09.508
And specifically, people over-ordered in order to get the allocations that they wanted. Then they, they got the allocations, and then they went under. They... Yeah, whatever, right? There was just a lot of shenanigans.
206
00:30:09.888 --> 00:30:16.988
A caveat of this is every time you say somebody, like, over-ordered, um, there is this assumption that the problem was, like, the demand went down. Uh-huh.
207
00:30:17.088 --> 00:30:25.648
Um, and I don't think that's the case at all, and so I wanna clarify that. Um, it definitely seems like there's more demand for GPUs than there ever was. It's just that there is also more supply.
208
00:30:26.268 --> 00:30:36.588
So at the moment, I think there is still functionally a glut, but the difference that I think is happening is mostly the test time inference stuff, that you just need way more chips for that than you did before.
209
00:30:36.988 --> 00:30:44.348
And so whenever you make a statement about the current market, um, people sort of take your words and then they assume that you're making a statement about the future market.
210
00:30:44.788 --> 00:30:49.628
And so if you say there's a glut now, people will continue to think there's a glut. But I, I think what is happening at the moment...
211
00:30:50.048 --> 00:31:00.468
My general prediction is that, like, by the winter, um, we will be back towards shortage, but then also, um, this very much depends on the rollout of future chips. Um, and that comes with its own...
212
00:31:00.988 --> 00:31:08.628
I think I'm trying to give you like a, a good Uh, here's Evan's forecast. Um- Okay... um, but I don't know if my forecast is very- You don't have to.
213
00:31:08.968 --> 00:31:18.328
Nobody's gonna hold you to it, but, like, I, I think people want to know what, what's true and what's not, and there's a lot of vague speculations from people who are not that close to the market, actually, and you are.
214
00:31:18.728 --> 00:31:27.128
So- I think I'm a close to the market, but also a vague speculator. Yeah. [laughs] Um, like, I think there are a lot of really highly confident speculators, and I am indeed a vague speculator.
215
00:31:27.308 --> 00:31:36.768
I think I have more information than a lot of other people, and this makes me more vague of a spectator, um, because I feel less certain or less confident, um, than I think a lot of other people do.
216
00:31:37.368 --> 00:31:46.408
The thing I do feel reasonably confident about saying is that, um, the test time inference is probably going to quite significantly, um, expand the amount of compute that was used for inference.
217
00:31:46.928 --> 00:31:59.088
So a caveat of this is, like, um, p-pretty much all the inference demand is in a few companies. A good example is, like, lots of bio and pharma, um, was using, um, H100s, training sort of the, the bio models of sorts.
218
00:31:59.508 --> 00:32:05.608
And they would come along, and they would buy, you know, thousands of H100s for training, and then just, like, not a lot of stuff for inference.
219
00:32:05.748 --> 00:32:10.568
Not in any-- not relative to, like, an OpenAI or an Anthropic or someone because they, like, don't have a consumer product.
220
00:32:10.988 --> 00:32:15.128
Their inference event, if they can do it right, there's really, like, only one inference event that matters.
221
00:32:15.348 --> 00:32:24.168
Um, and obviously I think they're gonna run in batch, and they're not gonna literally just run one inference event. Um, but, like, the one that produces the drug is the important one, right?
222
00:32:24.228 --> 00:32:31.008
[laughs] And I'm dumb, and I don't know anything about biology, so I could be completely wrong here. But my understanding is that's kind of the gist is you wanna- I can check that for you.
223
00:32:31.108 --> 00:32:39.748
You can check, check that for me. Check that for me. But my, my understanding is, like, the one that produces the sequence that is the drug that, um, you know, cures cancer or whatever, that's, that's the important deal.
224
00:32:40.228 --> 00:32:44.588
But, like, a lot of models look like this, um, where they're sort of more enterprise-y use cases or they're...
225
00:32:45.028 --> 00:32:53.748
So prior to something that looks like test time inference, you got lots and lots of demand for training and then pretty much entirely fell off, um, for inference.
226
00:32:54.328 --> 00:33:06.308
And I, I think, like, we looked at, like, OpenRadar, for example. The entirety of OpenRadar that was not Anthropic or Google, uh, like, or Gemini or OpenAI or something. It was like ten H100 nodes or something like that.
227
00:33:06.468 --> 00:33:16.768
It's just, like, not that much. Um, it's, like, not that many GPUs actually to service that entire demand. But that's, like, a really sizable portion of the sort of open source market.
228
00:33:17.148 --> 00:33:25.888
But the actual amount of compute needed for it was not that much. Um, but the... If you imagine, like, what an OpenAI needs, um, for, um, like GPT-4, it's, like, tremendously big.
229
00:33:26.088 --> 00:33:37.648
But that's because it's a consumer product that has almost all the inference demand. Yeah, that's a message we've had. Uh, roughly open source AI compared to closed AI is, like, five percent.
230
00:33:37.908 --> 00:33:49.968
Yeah, it's, like, super small. It's super small. [laughs] It's super small. Super small. Um, but test time inference changes that quite significantly. Um, so I will expect that to increase our, uh, overall demand.
231
00:33:50.548 --> 00:34:01.308
But my question on whether or not that actually affects your compute price is entirely based on, um, how quickly do we roll out the next chips? Um, like- The way that you burst is different for- Yeah... test time.
232
00:34:01.748 --> 00:34:11.568
Any thoughts on the third part of the market, which is the more peer-to-peer distributed, some are, like, crypto-enabled, like hyperbolic, Prime Intellect and all of that? Where, where do those fit?
233
00:34:11.748 --> 00:34:20.628
Like, do you see a lot of people will wanna participate in a peer-to-peer market, or just because of the capital requirements- Um... at the end of the day, it doesn't really matter? I'm, like,
234
00:34:21.568 --> 00:34:32.468
wildly skeptical of these, to be frankly. Um- The dream is, like, like, sitting at home, right? I got this 5090. F- nobody has 5090s. Uh, 4090 sitting at home. I can rent it out.
235
00:34:32.988 --> 00:34:42.168
Yeah, like, I just don't really think this is gonna ever be more efficient than a fully interconnected cluster with InfiniBand or, you know, whatever the sort of next spec might be.
236
00:34:42.808 --> 00:34:50.828
Like, I could be completely wrong, but speed of light is really hard to beat, and regardless of whatever you're using, you just, like, can't get around that physical limitation.
237
00:34:50.868 --> 00:34:58.728
And so you could, like, imagine a decentralized market that still has a lot of places where there's, um, like, co-location, but then you would get something that looks like SF Compute.
238
00:34:58.748 --> 00:35:06.988
And so that's what we're-- that's what we do. That's why we take our, our general take is, like, on SF Compute, you're not buying from, like, random people. You're, you're buying from the other GPU clouds functionally.
239
00:35:07.268 --> 00:35:14.668
Um, you're buying from data centers that are the same genre of people that you would work with already, and you can specify, "Oh, I want all these nodes to be co-located."
240
00:35:15.168 --> 00:35:27.168
And I don't think you're really gonna get around that. And I think I buy crypto for the purposes of, like, transferring money. Like, the financial system is, like, quite painful, um, and so on.
241
00:35:28.048 --> 00:35:35.228
I can understand the, um, uses of it to sort of incentivize an initial market or try to get around the cold start problem.
242
00:35:35.768 --> 00:35:39.248
We've been able to get around the cold start problem just fine, um, so it didn't actually need that at all.
243
00:35:39.788 --> 00:35:49.868
What I do think is totally possible is you could launch a token, and then you could, like, subsidize, um, the compute prices for a bit, but, like, maybe that will help you. Um- I think that's what Nus is doing.
244
00:35:49.888 --> 00:35:57.168
[laughs] Yeah, I, I think there's lots of people who are trying to do things like this, but at some point that runs out. So I would be, be... I think generally agree.
245
00:35:57.688 --> 00:36:03.448
I think the, the only thread in that model is very fine-grained mixture of experts that can be- Yeah...
246
00:36:03.468 --> 00:36:12.928
like the, like, algorithms can shift to adapt to hardware realities, and the hardware reality is like, okay, it's annoying to do large co-located clusters.
247
00:36:13.408 --> 00:36:19.228
Then we'll just redesign attention or whatever in our architecture to distribute it more. Yeah.
248
00:36:19.248 --> 00:36:25.488
And there was a little bit buzz, like, of block attention last year that, uh, Strong Compute, uh, made a, made a big push on.
249
00:36:26.008 --> 00:36:34.308
But I think, like, you know, in, in a world where we have two hundred mix, uh, experts in MOE model, it starts to be a little bit better. Like, I don't disagree with this.
250
00:36:34.668 --> 00:36:44.408
I can imagine the world in which you have, like, in which you've redesigned it to be more parallelizable, um, like, across space. But- [laughs]...
251
00:36:44.448 --> 00:36:50.808
um, assuming without that, your hardware limitation is your speed of light limitation. Um, and that's a very hard one to get around.
252
00:36:51.368 --> 00:36:58.948
Any customers or, like, stories that you wanna shout out of, like, maybe things that wouldn't have been economically viable, like others?
253
00:36:59.068 --> 00:37:13.308
I know there's some sensitivity on, on that, but- My, my favorites are grad students, are folks who are trying to do things that would normally otherwise require the scale of a big lab, and the grad students are, like, the worst possible customer- Mm-hmm...
254
00:37:13.368 --> 00:37:22.488
for the traditional GPU clouds because they will immediately turn, um, if you sell them a thing because they're gonna graduate and, like, not gonna go anywhere. Or they're not, they're not gonna, like- Yeah, yeah...
255
00:37:22.508 --> 00:37:28.188
that project isn't continuing, uh, to spend lots of money. Like, sometimes it does, but not, um, if you're, like, working with the university or you're working with a lab of some sort.
256
00:37:28.568 --> 00:37:40.156
But a, a lot of times it's just like- The ability for us to offer, like, big burst capacity, I think is lovely and wonderful, and it's, like, one of my favorite things to do because all those folks look like we did.
257
00:37:40.476 --> 00:37:54.656
Um, and I have a special place in my heart for young hackers and young grad students and researchers who are trying to do the same genre thing that we are doing for the same reason I have a special place in my heart for, like, the startups- Mm-hmm...
258
00:37:54.676 --> 00:38:02.716
um, the people who are just actively trying to compete on the same scale, um, but can't afford it time-wise, but can afford it, um, you know- Money, yeah... spike-wise.
259
00:38:03.176 --> 00:38:10.396
Yeah, I liked your example of, like, I have a grant of 100K and, um, it's expiring. Yes. I gotta- Cut. Cut... you know, spend it on that. Yeah.
260
00:38:10.436 --> 00:38:17.116
That's, that's, that's really beautiful, and I, you know, I hope interesting. Has there been interesting work coming out of that? Any- anything you wanna mention? Yeah.
261
00:38:17.256 --> 00:38:24.916
So from, like, a startup perspective, um, like Standard Intelligence and Phind, um, P-H-I-N-D, um- Yep. We've had them on the pod, yeah. Yeah. Um- Michael's great...
262
00:38:24.996 --> 00:38:29.976
and then from grad students' perspective, um, we worked a lot with, like, the Schmidt Futures grantees of various sorts.
263
00:38:30.416 --> 00:38:38.596
My fear is if I, um, talk about their research, I will be completely wrong to a sort of almost insulting degree 'cause I am very dumb, but yeah.
264
00:38:39.076 --> 00:38:50.536
I think one thing that's maybe also relevant, startups and GPUs-wise, is there was a brief moment where it kind of made sense that VCs provided GPU clusters, and obviously you worked at AI Grant- Yeah...
265
00:38:50.556 --> 00:38:53.236
which set up Andromeda, which is supposedly a $100 million cluster.
266
00:38:53.516 --> 00:39:05.876
Yeah, I can explain why that's the case or why anybody would think that would be smart because I remember before any of that happened, we were asking for it to happen. Yeah. And the general reason is, uh, credit risk.
267
00:39:06.336 --> 00:39:14.636
Again, it's a bank. I have higher- Yeah... I have lower risk than you. I do the credit transformation. I take your risk onto my balance sheet. Correct. Exactly. If you wanted to go...
268
00:39:14.676 --> 00:39:22.976
For a while, if you wanted to go set up a GPU cluster, you had to be the one that actually bought the hardware and racked and stacked it, like, co-located it somewhere with someone.
269
00:39:23.496 --> 00:39:30.116
Functionally, it was, like, on your balance sheet, which meant you had to get a loan, and you cannot get a loan for, like, $50 million as a startup, like, not really.
270
00:39:30.636 --> 00:39:36.916
Uh, you can get, like, venture debt and stuff, but, like, it's, it's, like, very, very difficult to get a loan of any serious price for that.
271
00:39:37.296 --> 00:39:49.636
But it's, like, not that difficult to get a loan for $50 million if you already have a fund or you already have, like, like, a billion dollars under asset somewhere, um, or, like, you personally can, like, do a personal guarantee for it or something.
272
00:39:49.696 --> 00:39:55.176
Um, like, there's... If you have a lot of money, it is way easier for you to get a loan than if you don't have a lot of money.
273
00:39:55.656 --> 00:40:08.156
And so the hack of a VC or some capital partner offering equity for, um, compute is always some arbitrage on the credit risk. Um- That's amazing. Yeah. That's a hack. You should do that [laughs].
274
00:40:08.696 --> 00:40:11.436
I don't think people should do it right now. I think the market has, like...
275
00:40:11.656 --> 00:40:20.416
I think it made sense at the time, and it was helpful and useful for the people who did it at the time, but I think it was a one-time arbitrage because now there are lots of other sources that can do it.
276
00:40:20.476 --> 00:40:27.696
Um, and also, I think, like, it made sense when no one else was doing it and you were the only person who was doing it. Yeah. But now it's like, it's an arbitrage that gets competed down. Sure.
277
00:40:27.756 --> 00:40:37.096
Um, so I don't know if it's, like, super effective. I wouldn't totally recommend it. Like, it's great that Andromeda did it. Um, but the marginal increase of somebody else doing it is, like, not super helpful.
278
00:40:37.276 --> 00:40:46.956
I don't think that many people have followed in their footsteps. I think maybe Andreessen did it. Um- Yeah... that's it. Um, uh, I think just because pretty much all the value, like, flows to Andromeda [laughs].
279
00:40:47.036 --> 00:40:53.296
Um, like, I think the- What? That cannot be true. I, I think you had to do- How many, how many companies are in AI Grant? Um- Like 50.
280
00:40:53.776 --> 00:41:03.256
I, my understanding of Andromeda is it works with all the NFDG companies, um, or, like, several of the NFDG companies. Um, but I might be wrong about that again. Um, uh, you know, something, something.
281
00:41:03.316 --> 00:41:12.836
Um, Nat, don't kill me. Um- [laughs]... uh, I, I could be completely wrong. But the... But no, I, I think Andromeda was, like, an excellent idea to do at the right time, um, in which it occurred. Um, and- It's perfect.
282
00:41:13.136 --> 00:41:22.556
His timing is impeccable... timing, yeah. Uh, Nat and Daniel are, like... I mean, there is lots of people who are, like- Seer? Yeah, Seer, like S-E-E-R. Um- Oh, Seers. Like Seers of the- Oh, Seers the, the mall...
283
00:41:22.556 --> 00:41:35.736
the valley. Um- Yeah... they for years and years before any of the, like, ChatGPT moment or anything, they had fully understood what was gonna happen. Um- [laughs]... like, way, way before. Like, AI Grant is, like,
284
00:41:36.916 --> 00:41:45.656
like, five years old, six years old or something like that, seven years old when I... when it, like, first launched or something. Depends where you start. Uh, the non- nonprofit version.
285
00:41:45.756 --> 00:41:50.856
Yeah, the nonprofit version was, like- Yeah... like, happening for a while, I think. It's going on for quite a bit of time.
286
00:41:50.976 --> 00:42:00.896
And then, like, Nat and Daniel are, like, the early investors in a lot of the sort of early AI labs of various sorts. They've been doing this for a bit. I was looking at your pricing yesterday.
287
00:42:00.996 --> 00:42:10.436
We were kind of talking about it before, and there's this weird thing where one week is more expensive of both one day and one month. Oh, yeah. Um, what are, like, some of the market pricing dynamics?
288
00:42:10.496 --> 00:42:18.796
What are things that, that... Like, this, to somebody that is not in the business, this looks really weird. Um, but I'm curious, like, if you have an explanation for it that looks normal to you. Yeah.
289
00:42:18.836 --> 00:42:28.836
So the, the simple answer is preemptible pricing is cheaper than non-preemptible pricing, and the same economic principle is the reason why that's the case right now. That's not entirely true on SF Compute.
290
00:42:29.036 --> 00:42:33.396
SF Compute doesn't really have the concept of preemptible. Instead, what it has is very short reservations.
291
00:42:33.876 --> 00:42:42.256
So, you know, you go to a traditional cloud provider and you can say, "Hey, I want a reserve contract for a year." We will let you do a reserve contract for one hour, which is the part of SFC.
292
00:42:42.616 --> 00:42:51.616
Um, but what you can do is you can just buy every single hour continuously, um, and you're reserving just for that hour, and then the next hour you reserve just for that next hour, and this is obviously, like, a built-in.
293
00:42:51.656 --> 00:42:59.716
This is, like, an automation that you can use. But what you're seeing when you see the cheap price is you're seeing somebody who's buying the next hour, but maybe not necessarily buying the hour after that.
294
00:42:59.956 --> 00:43:03.296
So if the price goes up too much, uh, they might not get that next hour.
295
00:43:03.756 --> 00:43:11.476
And the underlying part of this, of where that's coming from in the market, is you can imagine, like, day-old milk or, like, milk that's about to be old.
296
00:43:11.836 --> 00:43:19.356
It might drop its price until it's expired, um, because nobody wants to buy the milk that's in the past, or maybe you can't legally sell it. Compute is the same way.
297
00:43:19.836 --> 00:43:29.016
No, you can't sell a block of compute that is not, that is in the past. And so what you should do in the market, and what people do do, is they take [laughs]
298
00:43:29.076 --> 00:43:37.816
they take a, um, a block of compute, and then they drop it and drop it and drop it and drop it into a floor price right before it's about to expire, and they keep dropping it until it clears.
299
00:43:38.216 --> 00:43:50.096
And so anything that is idle drops until some point. So if you go and you s- on the website and you set that That chart to like a week from now, what you'll see is much more normal looking, um, sort of- Curves... curves.
300
00:43:50.516 --> 00:43:58.896
But if you say, "Oh, I wanna start right now," that immediate instant, here's the compute that I want right now, um, is the-- it's functionally the preemptible price.
301
00:43:59.436 --> 00:44:11.696
It's where most people are getting the best compute or, like, the best compute prices from. The caveat of that, um, is you can do really fun stuff on SFC if you want. So, um, 'cause it's not actually preemptible.
302
00:44:11.756 --> 00:44:20.356
It's, it's reserved, um, but only reserved for an hour, which means that the optimal way to use SF Compute is to just buy on the market price, but set a limit price that is much higher.
303
00:44:20.596 --> 00:44:28.976
So you can set a limit price for, like, four dollars and say, "Oh, if the market ever happens to spike up to four dollars, uh, then don't buy. I don't wanna buy at, at that price for that hour."
304
00:44:29.616 --> 00:44:40.136
But otherwise, just buy at the cheapest price, and if you're comfortable with that, of the volatility of it, you're actually gonna get, like, really good prices, like close to a dollar an hour or so on, um, sometimes down to, like, eighty cents or whatever.
305
00:44:40.456 --> 00:44:44.696
Um- You, you said four, though. Yeah, so that's the thing. You wanna lower the limit? Right. So four is your max price.
306
00:44:44.756 --> 00:44:51.876
Four is, like, where you basically wanna, like, pull the plug and say, "Don't do it," because the actual average price is not, uh-- or like the, you know, the, the preemptible price doesn't actually look like that.
307
00:44:52.236 --> 00:45:01.596
So what you're doing when you're saying four is always, always, always [chuckles] give me this compute. Like, continue to buy every hour. Don't preempt me. Don't kick me off, and I want this compute. Okay, okay.
308
00:45:01.676 --> 00:45:04.796
Um, and just buy at the preemptible price, but never kick me off.
309
00:45:05.116 --> 00:45:13.716
The only times in which you get kicked off is if there is a big price spike and, you know, let's say one day out of the year there's, like, a four dollar an hour price because of some w-weird fluke or something.
310
00:45:14.096 --> 00:45:19.296
If there are other periods of time you're actually getting a much lower price, then you- It makes sense... it makes sense.
311
00:45:19.356 --> 00:45:28.236
Your, your average cost that you're actually paying is way better, and your trade-off here is you don't literally know what price you're gonna get, so it's volatile.
312
00:45:28.716 --> 00:45:37.936
But, uh, your actual average historically has been, like everyone who's done this, has gotten wildly better prices. And this is, like, one of the clever things you can do with the market.
313
00:45:37.996 --> 00:45:45.476
If you're willing to make those trade-offs, um, you can get a lot of really good prices. Um, you can also do a bunch of other things like, um, you can only buy at night, for example, so the price goes down at night.
314
00:45:45.576 --> 00:45:50.616
Um, and so you can say, "Oh, I want to only buy, you know, if the price is lower than ninety cents."
315
00:45:51.076 --> 00:46:01.096
And so if you have some long-running job, you can make it only run on ninety cents, um, then you recover back and so on. But- Yeah. So what you can, uh, kind of create as, like, a spot inst is what other...
316
00:46:01.596 --> 00:46:09.676
the CPU world has. Yes. Is, uh... But you've created man- uh, a system where you can kinda manufacture the exact profile that you want. Exactly.
317
00:46:09.756 --> 00:46:18.016
Uh, that is not just whatever the hyperscaler has offered you, which is usually just one thing. Correct. SF Compute is, like, the, um... It's like the power tool of GPU financing.
318
00:46:18.036 --> 00:46:22.756
The underlying primitives of, like, hourly compute is, is there. Correct. Uh, yeah, it's pretty interesting.
319
00:46:22.976 --> 00:46:31.816
I've often asked OpenAI, so, like, you know, all these guys, um, uh, Claude as well, they do batch, um, uh, APIs. Yep. So it's, it's half off of whatever your thing is. Yeah.
320
00:46:31.876 --> 00:46:40.856
And the only contract w- is will return in twenty-four hours. Sure. Right? And I was like, twenty-four hours is good, but sometimes I want one hour. I want four hours. I want something.
321
00:46:41.076 --> 00:46:49.656
And so based off of SF Compute's system, you can actually kinda create that kinda guarantee- Totally... that would be like, uh, you know, not twenty-four, but- Mm-hmm...
322
00:46:49.716 --> 00:46:59.176
within eight hours, within four hours, like the, the work, half of a workday- Yes... I can return your result to you. And if your latency requirements are, like, that low, actually, like, it's, it's fine. Yes.
323
00:46:59.216 --> 00:47:07.136
And- Correct. Yeah. You can carve out that. You can financially engineer that on SFC. Yeah. Yeah. I mean, and, and I think to me, that unlocks a lot of agent use cases that I want. Mm.
324
00:47:07.216 --> 00:47:12.276
Which is like, yeah, it works in the background, but I don't want you to take a day. Yeah. Take- Correct... take a couple hours or something. Yeah.
325
00:47:12.836 --> 00:47:21.576
This touches a lot of my, like, background because I used to be a f- a derivatives trader. Yeah. And what... This is a forward market. Yeah. A, a future is forward market, whatever you call it. Not a future.
326
00:47:22.076 --> 00:47:28.016
Very explicitly not a future. Not yet a futures, yes. Uh, yeah. We can talk about that one. Yeah. Uh, but I don't know if you have any other points to talk about.
327
00:47:28.276 --> 00:47:36.696
So you recognize that you are a, uh, you know, a, a marketplace, and you've hired, I, I met, um, Alex Epstein at your, uh- Yeah... launch event. Mm-hmm.
328
00:47:36.776 --> 00:47:47.396
And, uh, you, like, you're, you're building out the financialization of GPUs. Yeah. So part of that's legal. Mm-hmm. [chuckles] Part of that is, like, listing on an exchange. Yep. Or maybe you're the exchange.
329
00:47:47.456 --> 00:47:54.036
I don't know how that works. But just, like, talk to me about that. Like, uh, from the legal, the standardization, the, like, where is this all headed?
330
00:47:54.376 --> 00:47:58.936
You know, uh, is this like a full listed on the Chicago Mercantile Exchange or whatever?
331
00:47:59.496 --> 00:48:07.336
What we're trying to do is create an underlying spot market that gives you an index price that you can use, and then with that index price, you can create a cash-settled future.
332
00:48:07.736 --> 00:48:16.596
And with a cash-settled future, you can go back to the data centers, um, and you can say, "Lock in your price now and de-risk your entire position," um, which lets you get cheaper cost of capital and so on.
333
00:48:17.036 --> 00:48:26.916
And that, we think, will improve the entire industry because the marginal cost of compute is the risk, as shown by that graph in basically every part of this conversation.
334
00:48:27.316 --> 00:48:35.596
It's risk that causes the, um, the price to be all sorts of funky, and we think a future is the correct solution to this. So that's the, that's the eventual goal.
335
00:48:36.096 --> 00:48:43.556
Right now you have to make the, um, the underlying spot market in order to make this occur, and then to make the spot market work, you actually have to solve a lot of technology problems.
336
00:48:43.836 --> 00:48:52.236
You really cannot make a spot market work if you don't run the clusters, if you don't, um, have control over them, if you don't know how to audit them, because these are supercomputers, not soybeans.
337
00:48:52.696 --> 00:49:00.876
They have to work, um, in a way that, like, it's just a lot simpler to l-deliver a soybean than it is to deliver the- I don't know. Talk to the soybean guys. Sure. You know.
338
00:49:00.936 --> 00:49:08.576
Yeah, but you, you have to have a delivery mechanism. Your delivery mechanism, like somebody somewhere has to actually get the compute at some point, and it actually has to work, um, and it is really complicated.
339
00:49:09.036 --> 00:49:18.276
And so that is the other part of our business, that we go and we build, um, a bare metal infrastructure stack that goes... And then also we do, um, auditing of all the clusters.
340
00:49:18.616 --> 00:49:26.096
You sort of de-risk the technical perspective, and that allows you to eventually de-risk the financial perspective, and that is kind of the pitch of SF Compute. Um. Yeah.
341
00:49:26.456 --> 00:49:32.756
I'll, I'll double-click on the auditing on the clusters. Yep. Uh, this is something I've had conversations with Yitay on. He started Reka and, um- Yeah...
342
00:49:32.816 --> 00:49:39.340
and, and I think it's a-- he had a blog post which kinda shone the light a little bit on how Unreliable some clusters are versus others Correct.
343
00:49:39.520 --> 00:49:46.820
Yep And sometimes you kind of have to season them and age them a little bit to find, like, the bad cards. Correct. You have to burn 'em in. Yep. So what do you do to audit them?
344
00:49:47.220 --> 00:49:54.000
There's, like, a burn-in process, a suite of tests, and then active checking and passive checking. Burn-in process is where you typically run Linpack.
345
00:49:54.080 --> 00:50:01.700
Linpack is this thing that, like, a bunch of linear algebra equations that you're, you're stress testing the GPUs. This is a proprietary thing that you wrote? No, no, no. Lin- Linpack, you- Oh, is it? Okay... you...
346
00:50:01.720 --> 00:50:09.600
Linpack is, like, the most common- Okay. Sorry... form of burn-in. If you just type in burn-in, um, typically when people say burn-in, they literally just mean Linpack. There's like an NVIDIA reference version of this.
347
00:50:09.840 --> 00:50:17.440
Um- And again, NVIDIA could run this before they ship, but now the customers have to do it. It's, it's annoying. You're not just checking for the, the GPU itself.
348
00:50:17.500 --> 00:50:22.840
You're checking, like, the whole component, all the hardware, and, um- It's an integration test... its location. It's an integration test. Yeah. Yeah.
349
00:50:22.900 --> 00:50:31.180
So what you're doing when you're running Linpack or burn-in in general is you're stress testing the GPUs for some period of time, 48 hours, for example, maybe seven th-...
350
00:50:31.400 --> 00:50:37.140
uh, days or so on, and you're just trying to kill all the dead GPUs or any components in the system that are broken.
351
00:50:37.720 --> 00:50:49.560
And we've had experiences where we ran Linpack on a cluster and it browns out, like, you know, sort of comes offline when you run Linpack. This is a pretty good sign, um, that maybe there is a problem with this cluster.
352
00:50:49.860 --> 00:50:53.940
And so Linpack is, like, the most common sort of standard test.
353
00:50:54.260 --> 00:51:06.940
But then beyond that, um, what you do is we have, like, a series of performance tests, um, that replicate a much more realistic environment as well that we run just assuming if Linpack works at all, then you run the next set of tests.
354
00:51:07.400 --> 00:51:13.280
And then while the GPUs are in operation, you're also going through and you're doing active tests and passive tests.
355
00:51:13.320 --> 00:51:17.780
Passive tests are things that are running in the background while somebody else is running, while, like, some other workload is running.
356
00:51:18.140 --> 00:51:21.240
And active tests are during, like, idle periods, you're running some period of t-...
357
00:51:21.440 --> 00:51:30.880
um, you're running some sort of check that would otherwise sort of interrupt something, and then the active tests will take something offline, basically, or a passive check might mark it to get taken offline later, um, and so on.
358
00:51:30.900 --> 00:51:42.600
And then the thing that we are working on that we have working partially but not entirely, um, is automated refunds, which is basically, like, um- Yep... is the case that the hardware- They go... yeah... breaks so much.
359
00:51:42.660 --> 00:51:46.659
Um, and there's only so much that we can do, and it is the effect of pretty much the entire industry.
360
00:51:46.719 --> 00:52:06.940
So a pretty common thing that I think happens to kind of everybody in the space is a customer comes online, they experience your cluster, and your cluster, uh, has the same problem that, like, any cluster has, or it's, I mean, a different problem every time, but they experience one of the problems of HPC.
361
00:52:07.360 --> 00:52:16.520
And then, um, their experience is bad, and you have to, like, negotiate a refund or some other thing like this. Um- It's always case by case and, like- Yeah... a lot of people just eat the cost. Correct.
362
00:52:16.700 --> 00:52:24.820
So one of the nice things about a market that we can do as we get bigger, and have been doing as we get bigger, is we can immediately give you something else, and then also we can automatically refund you.
363
00:52:25.240 --> 00:52:34.180
And you're still gonna experience it. Like, the hardware problems aren't going away until the underlying vendors fix things, but honestly, I don't think that's likely because you're always pushing the limits of HPC.
364
00:52:34.240 --> 00:52:36.900
This is the case of trying to build a supercomputer. But
365
00:52:37.840 --> 00:52:46.080
that's one of the nice things that we can do is we can switch you out for somebody else somewhere and then, um, uh, automatically refund you or prorate or whatever the correct- Yeah, yeah... move is. Yeah.
366
00:52:46.140 --> 00:52:53.120
Uh, one of the things that you tell in, in this conversation with me was, like, you know, you know a provider is good when they guarantee automatic refunds. Yep.
367
00:52:53.600 --> 00:52:59.340
Um- Which doesn't happen, but [laughs] Yeah, that's, that's in our contract with all the underlying cloud providers. Um, so- You built it in already. Yeah.
368
00:52:59.380 --> 00:53:03.000
So we, we have, um, uh, a quite strict SLA that we pass on to you. Yeah.
369
00:53:03.040 --> 00:53:11.980
The reason why I'm, like, hedging on this is because we have some amount of active checks, we have some amount of passive checks, um, there are always new genres of bullshit. Mm-hmm.
370
00:53:12.020 --> 00:53:19.400
And the new genres of bullshit might cause a customer to have a bad experience, um, and the active or passive checks, um, didn't catch it, and so then it's a manual process after that.
371
00:53:19.480 --> 00:53:30.380
Um, then we have, like, a literal thing on our website that you can just say [laughs] "Hey, some hardware problem, please tell us," um, and then we will go and resolve it for you. Um- Well, how...
372
00:53:30.420 --> 00:53:34.180
I mean, cards don't change generation to generation. What, what, what is a new genre of bullshit?
373
00:53:34.480 --> 00:53:45.040
If every component piece in the cluster has maybe, like, a one in 100 chance of failing, or maybe a one in 1,000 chance of failing, or maybe a one in 10,000 chance of failing- You discover them... you discover them.
374
00:53:45.080 --> 00:53:54.979
So there's ones that, um, like, maybe nobody saw, maybe you didn't see, or maybe it only matters for this one cluster with this motherboard in this particular data center, um, or something.
375
00:53:55.180 --> 00:54:04.580
There's new interactions that otherwise don't happen. Most problems are really common, and you can adapt to them. Like, like, a GPU falls off a bus is, like, one of the most common things that can happen.
376
00:54:05.120 --> 00:54:11.520
So it's not SF Compute's job to go fix those things. No, it totally is- You just-... to some extent. Totally is to some extent. So we, we operate the cluster.
377
00:54:11.580 --> 00:54:16.520
So unlike a reseller, which is what we were doing before, in almost all cases, we have BMC access.
378
00:54:16.660 --> 00:54:23.060
So if on your laptop there's, like, the button in the top right-hand corner that you can hold down to, like, re-image the machine- Mm-hmm...
379
00:54:23.160 --> 00:54:33.680
there's a similar thing in, like, Server X that you, is like this other box that kind of plugs in, and it basically lets you reset the machine from outside, and it's like remote, it's a remote hand sort of thing.
380
00:54:34.080 --> 00:54:43.640
So we ask for this, and we get this from a lot of, um, our vendors, uh, which means we have quite a lot of ability to solve problems for customers, um, in a way that you might not actually get from a reseller.
381
00:54:44.260 --> 00:54:52.300
Oftentimes, we are the person who's debugging your cluster. Um, for most customers that we work with, we have Slack channel. Our entire engineering team, um, gets put in the Slack channel.
382
00:54:52.660 --> 00:54:58.680
If there was a problem at 2:00 AM, we are the ones who are debugging your problem at 2:00 AM. Not always the case, um, because we don't physically run the- Mm-hmm...
383
00:54:58.720 --> 00:55:10.200
the hardware cluster or, like, the, the data center itself, but most problems are solvable through this. So that's the auditing side. Yeah. The other side is, I think of it as standardization or whatever you call it.
384
00:55:10.560 --> 00:55:21.240
Beyond auditing- Yeah... the other part of the work is kind of standardizing the commodity contracts. Yeah. Yeah. So there's, there's two ways that we do that. Um, one is that you set, like, a this or better list.
385
00:55:21.340 --> 00:55:27.820
So you set, like, a spec list. Um- Yeah... and you say, "Oh, you're gonna get..." Uh, like a common variability is the amount of storage on the cluster.
386
00:55:28.240 --> 00:55:32.940
And so you'll say like, "Oh, you're gonna get X or better," and there's some guarantee minimum, and sometimes you might get more.
387
00:55:33.460 --> 00:55:42.012
And then we're working on a persistent storage layer that might sort of abstract a lot of this way, but Mostly it's that, and then there's like a whitelist of motherboards and various genres of things.
388
00:55:42.432 --> 00:55:49.692
But the other part is, uh, we run the clusters from bare metal up, and so we make a thing, uh, that's this like... It's a UEFI shim.
389
00:55:49.772 --> 00:55:59.952
Um, and if you're not familiar with what UEFI is, a UEFI is like the sort of- Firmware... modern version of BIOS. Yeah. Um, modern meaning it's been around for like forever, but um, you know, BIOS is like really old.
390
00:55:59.992 --> 00:56:07.172
It's like this old IBM thing. And, uh, you can write code that exists at the UEFI layer, and again, when you hear UEFI, you should think BIOS.
391
00:56:07.672 --> 00:56:16.252
And it, uh, does the same sort of thing as a PXE boot, um, but in environments in which PXE boot doesn't necessarily always work for us. So we, um...
392
00:56:16.372 --> 00:56:26.872
It basically sits at your, um, your BIOS, it downloads an image, boots into an image that's like custom for the user, and then on top of that image, we can throw Kubernetes on it, we can throw, um, VMs on it or whatever you want.
393
00:56:27.252 --> 00:56:31.232
And at some point we'll probably like do more stuff with that, but that's functionally what we can do.
394
00:56:31.392 --> 00:56:39.072
Um, the nice thing though is that because you control from that layer, you can easily image a entire cluster, you can make it all the same, you can run your performance tests, it's all automated. Um,
395
00:56:40.212 --> 00:56:50.532
so much nicer than what we used to do. [laughs] Yeah. Yeah, I mean, that, that is a very important work. I, I think like for me, the, you know, as a trader- Yeah... I need standard contracts. Correct. Yeah.
396
00:56:50.552 --> 00:57:02.852
And so there basically needs to be the safe of a- Right... GPU. Yes. What we functionally do is we have a market under the hood that is focused on the buyer and the seller, um, and it's optimized for them.
397
00:57:03.252 --> 00:57:11.812
And then beyond that, for a trader, um, you can standardize around a certain si- like segment of it, and you can trade on that contract. Um, that's the, that's the goal that we're trying to get to.
398
00:57:12.192 --> 00:57:15.892
But you start by making something that works really well for buyers and really well for sellers.
399
00:57:16.292 --> 00:57:22.072
For those who are not familiar with derivatives markets, I, I can go ahead and say this because the point of being cash-settled, which is something that you mentioned- Yep...
400
00:57:22.072 --> 00:57:27.232
which I think people might miss, is that you don't have to take physical delivery of the GPUs. Right.
401
00:57:27.572 --> 00:57:38.452
And, uh, that, so it's a pure financial instrument, which actually does mean that almost for certain there will be more volume on SFC's marketplace than actually change hands in GPU terms.
402
00:57:38.692 --> 00:57:44.992
To be super clear, we are not a derivatives market. This doesn't happen yet. Yeah. We are not a derivatives market. Um, we may in the future, um, work to create a cash-settled future.
403
00:57:45.272 --> 00:57:47.871
We are not currently a derivatives market, we are an online spot market. Yeah.
404
00:57:47.972 --> 00:57:58.272
Um- I just think like people, normies get really upset when they're like, then they learn things like, oh, like derivatives on mortgages- Yeah... are like 12 times larger than the mortgages themselves. Yes.
405
00:57:58.272 --> 00:58:00.652
[laughs] Um, yeah, no, I, I, um...
406
00:58:01.012 --> 00:58:14.412
A, a common thing that people have talked to us about, or like a fear or concern I think people have is like, "Oh, you're financializing compute and this will like cause various problems of sorts." Subprime crisis. Yeah.
407
00:58:14.492 --> 00:58:23.112
Um, a-and I think- [laughs] So first, I think part of this is just because crypto caused a lot of people to think about finance in a like very degen way, for the right word.
408
00:58:23.392 --> 00:58:31.752
Um, and then before that, um, the sort of 2008, 2009 crisis, um, caused people to think about it also in sort of like a degen-y way. And this is very much not our mindset.
409
00:58:32.052 --> 00:58:38.872
The reason to create a derivative at all, or the reason to create a future at all is a risk reduction thing. Um, that's what futures do.
410
00:58:39.112 --> 00:58:46.272
The reason why a farmer wants a future is because they have no idea what the weather is going to do, and they don't want to be on the hook, um, for...
411
00:58:46.832 --> 00:58:56.112
Like, they have small margins, and if things go wrong, they really, really wanna have a locked in price, um, so that way they can like continue to exist for the next year. Data centers are the same way.
412
00:58:56.592 --> 00:59:05.212
The way that they solve it today is you go out and you sign long-term contracts with your customers. What that does for you is it means your business is de-risked.
413
00:59:05.552 --> 00:59:07.532
Um, you don't have to worry about the revenue for the next year.
414
00:59:08.112 --> 00:59:13.872
But that means that the customer now has to worry about what they're gonna do with all this compute, and if they don't optimally use it, and so on and so on.
415
00:59:14.152 --> 00:59:19.872
And that just pushes everything onto the startups, who then in turn push it onto VCs.
416
00:59:20.272 --> 00:59:30.572
And so what the VCs are forced to do in order to invest in AI is they have to go and write big, giant valuations, like pre-revenue at ridiculous multiples.
417
00:59:30.872 --> 00:59:38.192
So what you've done by not having a future is you've inflated the venture capital market, and that is a bubble that's totally [laughs] gonna pop at some point.
418
00:59:38.232 --> 00:59:49.192
Like, a lot of the companies are not gonna work and the valuations are not gonna work. And what's gonna happen is a lot of these funds aren't going to return back to their LPs, and that affects the broader market.
419
00:59:49.532 --> 00:59:58.032
The way that you solve that, the way that you add security to the entire economic system in this chain, is you add a future. That's how we did it in lots of other markets.
420
00:59:58.251 --> 01:00:08.752
It doesn't have to be this like, oh my gosh, we're gonna like speculate on GB prices and like, whatever. No. The whole point of SF Compute is to reduce the risk, reduce the technical risk, reduce the financial risk.
421
01:00:09.192 --> 01:00:16.232
Let's just chill out a little bit. [laughs] There's so much other random shit. It's supercomputers, there's AGI, whatever. No, let's just like chill the fuck out.
422
01:00:16.332 --> 01:00:22.392
I mean, also like Dan is going, raising like at a $30 billion- Yeah... valuation for Ilia. You know, like- Yeah.
423
01:00:22.412 --> 01:00:33.992
[laughs] If, if everybody else in all of AI is like pushing the hype and the extreme, everything we've been trying to do is go the other way. [laughs] Like, whole website is just like a fucking single page.
424
01:00:34.372 --> 01:00:46.332
Um, like the e- entire brand is just like, what if we were like calm in nature? And then everything that we do as the product is just calm. What if we, what if we were the opposite force of the big hype-y extreme thing?
425
01:00:46.992 --> 01:00:55.232
W- what if we just like chilled things out? And part of that was because we, in the beginning, were at the whim of the [laughs] the hype-y nature.
426
01:00:55.272 --> 01:01:04.352
Like, our entire origin is every 30 days, if we don't sell out, we're gonna go crazy, um, and just completely bankrupt the company. And so everybody in the company is just like, "What if we just chilled out?"
427
01:01:04.412 --> 01:01:13.132
[laughs] What if every- Mm. What if we stop for a bit? [laughs] This is the first time I've ever heard derivatives are the way to chill out. Yes. [laughs] No, futures are the way to chill out.
428
01:01:13.172 --> 01:01:19.832
Fut- [laughs] Futures are the way to chill out the entire industry. [laughs] And, um, we wouldn't be doing this if it wasn't that case. [laughs] I like that.
429
01:01:20.532 --> 01:01:28.832
Um- And you have a very nice brand with a, you know- Yeah, you mentioned the website... clear sky. Sure. You know? We have to ask about the website. Yeah. What was the inspiration behind it?
430
01:01:29.032 --> 01:01:36.572
Why did you not go the black neon more cool thing and go the more nature route? Yeah. Um, I don't think I really am a black neon sort of person.
431
01:01:36.632 --> 01:01:41.112
I say this wearing black pants and I thought I was wearing a black shirt, but apparently I'm not. So, um- [laughs]...
432
01:01:41.152 --> 01:01:52.716
the actual, the actual thing was a lot of companies do this thing where they Their website, you go to there, and it's like a magical experience, and like everything is extreme and amazing and incredible.
433
01:01:52.976 --> 01:01:57.256
And then you go to the product and it's like some SaaS app or something. Mm-hmm. Um, and it's like not actually that exciting.
434
01:01:57.656 --> 01:02:08.676
And that expectation of being like really, really good and then the fall off, the drop of not being really, really good, was something that from a product perspective I never wanted to happen.
435
01:02:08.976 --> 01:02:15.676
Especially 'cause in the beginning, like our product was really bad. And so I don't want to set the expectation that it's gonna be like an amazing experience.
436
01:02:15.776 --> 01:02:24.476
I want to set the expectation that it's going to be like a good price, um, for short-term bursts. And so what we did instead is we set the thing to be really low.
437
01:02:24.536 --> 01:02:30.916
You set your expectations really low, and then you get a supercomputer, [laughs] um, for like millions of dollars cheaper than you would've otherwise gotten your supercomputer.
438
01:02:30.956 --> 01:02:38.176
[laughs] And so you have the opposite expectation. You have like really low expectations that, uh, are like mi- are met higher, and I think that's like the correct way to do things.
439
01:02:38.496 --> 01:02:48.856
But also, I think we were just like so sick of hype and excitement and, um, I just like really wanna like not do that. Sure. It- it's weird, like by, by being anti-hype, you have created hype.
440
01:02:49.136 --> 01:02:52.776
Like, I would say like the- [laughs] Yeah, I hate that... the, the, the, the vibe's very immaculate. You can't help yourself. You know? Yeah.
441
01:02:52.796 --> 01:02:59.496
Like, you just, you go to, like, at the, the bay, the Cow Trade, you just put up like, like a banner that just, just says SF Compute. True.
442
01:02:59.536 --> 01:03:08.076
That banner was created about five minutes before we had to actually put something up- Yeah... like before the deadline was there. Um- You opened up Microsoft Word and you did some Serif. Yep. What, what is the font?
443
01:03:08.136 --> 01:03:15.356
Exactly. It's, I don't know. Yeah, that was, um- [laughs] Indeed. [laughs] Um, the... Yeah, I think every time we tried to do...
444
01:03:15.596 --> 01:03:21.676
The, the only caveat to this, the only caveat that we ever violate this rule with, uh, is when we're pitching San Francisco.
445
01:03:22.136 --> 01:03:28.256
I think San Francisco is amazing, so sometimes you will see these, like, advertisements, um, that from- You mean, you mean the city? Yeah, the city. Um, so if...
446
01:03:28.516 --> 01:03:37.436
There's a part of San Francisco Compute's brand which are these beautiful, like, images- Yeah... of SF or various SF things, and I am the complete opposite about this.
447
01:03:37.716 --> 01:03:48.876
I am such a San Francisco promoter that any time we talk about the city, I want to show the city from the, like, eyes that we have, which is mostly just gorgeous, beautiful area with nature.
448
01:03:48.916 --> 01:03:55.456
Like, a lot of people think about San Francisco and they think about like- Tenderloin... the tech industry or they think... Yeah, or the Tenderloin or something, like grind culture or something.
449
01:03:55.856 --> 01:04:07.296
And no, like I think about like the fog, um, and just like the gorgeous view over the bridge, and just the fact that there is this, like, massive amount of optimism in the city. And it's...
450
01:04:07.616 --> 01:04:12.896
The backdrop of that optimism is the most beautiful countryside in all of the world.
451
01:04:13.316 --> 01:04:21.076
And so any time we talk about SF, you will see like or like we have a billboard somewhere that's just like local friendly supercomputer whatever, and then the backdrop is, like, beautiful and amazing.
452
01:04:21.496 --> 01:04:28.516
And that's because to some extent we're pitching the city and the people here, and I think the people in the city here are actually really amazing. And so you get to earn the brand. Mm-hmm.
453
01:04:28.616 --> 01:04:38.596
Um, [laughs] 'cause the expectations are met. [laughs] Whereas I think on our own product, I'm typically want it to be better, and so I set the brand a lot lower. Um, and then the expectations are higher.
454
01:04:38.896 --> 01:04:46.176
Um, and you still meet the expectations, but you, you set them a little lower. Um- I know, a- are you the designer? I, I know you have an artistic side. Um, so, uh, I was in the beginning.
455
01:04:46.476 --> 01:04:56.996
So, uh, I, I'm like a figurative artist, so I, I draw people. Um, but we've worked with a design firm. Aerofoil was really excellent with us, and then, um, nowadays though, John Pham, um- Oh... head of design.
456
01:04:56.996 --> 01:05:08.996
Um- Yeah, from Vercel. Yeah. John is unbelievably amazing. Um, I think the amount of care and craft and attention to detail that he puts into just everything is so cool. [laughs] Yeah.
457
01:05:09.076 --> 01:05:17.196
Um, like if you go on our buy page right now, you go to sfcompute.com/buy, there is an Easter egg there that will... You should find. Ooh. I almost don't wanna spoil it- Yeah...
458
01:05:17.216 --> 01:05:23.416
but you should go find- Yeah, go look for it... that Easter egg. If you just, like, hover the mouse around the thing in the top right-hand corner- Yeah... um, you will, you will find it.
459
01:05:23.696 --> 01:05:37.096
Um- Yeah, tweet at Evan if you find it. And then the, um, other person is, um, Ethan Anderson, our COO, who, um, has this RISD design background. And so, uh, he's, like, uh, he used to be sort of industrial designery.
460
01:05:37.156 --> 01:05:46.356
Uh, I'm probably gonna say that wrong. He's probably not an actual industrial designer, but design background, same. So I think between me and John and, uh, Ethan, um, I think we- The source of the vibes.
461
01:05:46.436 --> 01:05:49.856
I had to- The source of the vibes... I had to ask. Yeah. Okay. So we're gonna zoom out a little bit.
462
01:05:49.936 --> 01:05:58.156
One of the last things I wanted to ask you was actually, like, I remember, I think the first time I met you was in like kinda Celo- Mm-hmm... and you were working on your email startup. Oh, yeah, yeah.
463
01:05:58.196 --> 01:06:08.256
And I have a favorite pet topic of mine. We were here with Dharmesh yesterday talking about someone build an agent that reads my emails. Yeah. And you did, and I think I actually paid for the first one.
464
01:06:08.296 --> 01:06:16.396
You were, you were so excited in the early GPT-3 days, three days. I was like... You were like, uh, "I'm building the most expensive startup ever." Yeah. [laughs] Uh, and then it's like- It's so expensive... like...
465
01:06:16.636 --> 01:06:24.316
Anyway, so the, the point being, what, what I'm trying to get to is you are a very smart guy. You built email. You, you didn't like it. You pivoted away. I've seen other...
466
01:06:24.336 --> 01:06:34.656
Like, every year there's someone who is like, "I will crack email." Yeah. "And I'll..." And then, and then they give up. Yeah. What is so hard about email? I didn't pivot away because the product or the idea was bad.
467
01:06:34.936 --> 01:06:42.736
I pivoted away because I was super burnt out. Yeah. Um, I did a startup for like four years, um, and the first thing didn't work out. Is this Room Service? Uh, yeah, this is Room Service.
468
01:06:42.936 --> 01:06:52.376
So my startup before this, uh, originally started as Quirq, which was like a mental health app, but then Quirq had the same problems that basically every mental health app has, which is like your retention goes to zero if you work it in any capacity.
469
01:06:52.916 --> 01:07:00.196
And so, um, switched and then said, "Okay, well, I will do something that's closer to my actual background," was a distri- like a distributed systems company, uh, called Room Service.
470
01:07:00.536 --> 01:07:06.836
Room Service went for about nine months and then sort of had the same problem that I think every other competitor Room Service has, which is mostly people build it in-house.
471
01:07:07.156 --> 01:07:12.496
And so then I went back to our investors at the time, which was Nat and Daniel, and specifically Daniel told me that I should go start the Ocean.
472
01:07:12.936 --> 01:07:22.876
Um, and you know, I will find something else to do and just throw shit at the wall. And then I think, um, I think it was Gustaf at YC. Maybe, it was probably actually Dalton Caldwell.
473
01:07:23.056 --> 01:07:32.896
Dalton Caldwell, um, like just said, "Don't die." Um, like, "You can just keep doing things and don't die." And so I think I just got it in my head- That you should just, like, keep trying things and not die.
474
01:07:33.356 --> 01:07:44.676
And I really, really, really did not want to die and didn't really know what to do, and so I just threw out, like, 40 products with the assumption that if you just keep trying things, uh, you won't die.
475
01:07:44.896 --> 01:07:54.756
This is actually not the most ideal thing to do. You actually should totally just pick a thing and go with it. But my brain wasn't set on, like, "Oh, I should do this particular thing." It was set on not die.
476
01:07:55.316 --> 01:08:08.916
And so I just kept going for a very long time, for, like, four years, and by the end of it, I think I was just super burnt out and I, um, was gonna do the email thing with one co-founder, and then they quit, and then I was gonna do the email thing with another co-founder, and then they fell in love and decided to go get married and you know all that.
477
01:08:09.276 --> 01:08:16.496
Okay, so it wasn't that email is intractable. Yeah. I'm just trying to figure out, like- Yeah... look, is there something ba- like, j- is, is this the graveyard of ideas, right?
478
01:08:16.576 --> 01:08:24.916
Everyone wants to do email and then nobody does because something. And um- I think it's just hard to make an email client. I think it's hard to make an email client that, um, is...
479
01:08:25.696 --> 01:08:36.616
Uh, it's a competitive space in which there are lots of things. I do think that the better version of that is something that looks closer to what Intercom is doing, and Intercom obviously existed beforehand.
480
01:08:37.016 --> 01:08:43.376
So you can think about, like, any product, like should you be doing it or should somebody else in the industry who already has the existing customer set do it?
481
01:08:43.696 --> 01:08:53.216
And I think Intercom has pretty much very successfully done... Like, they already had the position to do it. Um, like, what, what do you actually need, um, the AI to write your emails for? Like, it...
482
01:08:53.576 --> 01:09:03.116
Most people don't need this, um, but what, who does need this is, like, support use cases is pretty much there, and the people who are best able to execute on this is totally Intercom. So, um, like, props to Owen.
483
01:09:03.176 --> 01:09:09.776
Um, I think that was, like, completely the correct move. Yeah. So yeah, closing thoughts and- Closing thought, call to actions. Yes. Like, are, are you- You're hiring. Yeah. Oh, yeah, we are.
484
01:09:09.856 --> 01:09:16.476
Uh, we are hiring for two roles as of this recording. I don't know, maybe this will change and we'll h- be hiring for different roles, so go to, go to the website or whatever.
485
01:09:16.956 --> 01:09:25.036
But, uh, the first role, um, is for traditional systems engineering. This is, like, low-level systems or low-level Linux-y people. Um- Rust. Yeah, so all Rust.
486
01:09:25.336 --> 01:09:33.036
Most all of our code base is in Rust, but n- we're not necessarily just looking for, like, Rust engineers. Um, we're specifically looking for, like, l- Linux-y people.
487
01:09:33.316 --> 01:09:44.756
The sort of pitch is you get to work on supercomputers. You get to work on one of the few places in supercomputers that I think has a pretty good business model and is, like, um, a, um, like, a, a working thing.
488
01:09:45.236 --> 01:09:55.576
And, uh, people generally seem to think that our, our vibe at SF Compute is very nice. Um, the, um... We have just an unbelievably excellent team, I think, nowadays. Our CTO is Eric Park.
489
01:09:55.616 --> 01:10:05.236
He's, uh, the co-founder of Voltage Park, which is one of the other, um, GPU clouds. Um, and he is quite possibly the sweetest man I've ever met. Um, he is extremely chill and also just extremely earnest and kind.
490
01:10:05.656 --> 01:10:16.596
And the rest of the team kind of feels that energy very strongly. And then the other role that we're hiring for, um, is financial systems engineering, which I really should learn what...
491
01:10:16.656 --> 01:10:23.316
It's not systems engineering, but, um, we should really find a better name for this role. It's basically a fintech engineer. That it's...
492
01:10:23.496 --> 01:10:35.576
We have the same problems as traditional fintech does, um, and that's, like, we have a ledger, we have recording requirements and all that stuff. This role is responsible for the not lose all the money, um, Cole.
493
01:10:35.616 --> 01:10:42.116
Like, we've got a whole bunch of money flowing through us. There is a bunch of stuff that you need to do in order to not lose all that money.
494
01:10:42.516 --> 01:10:57.776
And then the actual outcome of that work, besides not just losing all the money, which is very important, is that you end up with better prices for the vendors and better prices for the buyers, and this means that your grad student who is making the cancer cure or whatever and needs to be able to buy, like, 100K of compute to, like, scale up really big actually can do so.
495
01:10:58.136 --> 01:11:07.896
And that's, I think, the, like, this is part of the reason to work at SFC is that your, the things you do actually matter in a way that doesn't necessarily always at all the companies.
496
01:11:08.336 --> 01:11:19.796
Functionally, we run supercomputers, like, not soybeans or, I don't know. [laughs] Um, it's a very cool place to work because your outcomes of what you do, um, have real deal impact- Yeah...
497
01:11:19.856 --> 01:11:26.876
in a way that you don't always get when you're doing SaaS. Um- Uh, excellent pitch. I- Mm-hmm... I bet you've done that a lot, but, uh, it's, it's nice to hear it for the first time.
498
01:11:26.936 --> 01:11:36.096
I was gonna say, like, uh, you know, have you looked into TigerBeetle, uh, the, uh, dual-entry accounting database? Uh, we have though- That seems to be the thing if you wanna make systems that don't lose money. Yes.
499
01:11:36.576 --> 01:11:44.756
Systems that don't lose money, um, there are lots of other things you have to do. Um- Yeah... like, you have to make things in a format that your accountants can read and then- Yeah... get audited and so on.
500
01:11:44.816 --> 01:12:01.196
Um, it's not purely just the, um... Yeah, it's not purely just the tech. Cool. Yeah. Awesome. Thank you so much. Yeah, of course. It's been- Thank you so much for having me. [upbeat music]