Pełna transkrypcja odcinka Data Centers: Think Small

when I left General Motors I'm just thinking like how I came to this point and working on the data centers. When I left General Motors what I left with is little bit disappointed disappointment because we were trying to do data. We were trying to work with data. We were trying to do AI first initiatives But we were always the cost centers, you know, getting to the computation, getting to the storage, it's still elite. So it's expensive, you cannot scale those works. and I started to study how we can change the way that we are looking at the data centers and how we can actually look at the energy aspects. And That was just before OpenAI cracked and you know Open Pan Pandora's box opened. So I started to study data centers and I started to study data centers from the perspective that could be smaller and at the same time distributed as a large data center. Now Robert: Okay. Meltem Ballan, Concrete Engine: it's it's very relevant, but it was not relevant back in twenty Twenty one, twenty three, what not? Robert: Yeah. Well d data's always been the driver of you know, the more data that you have, say verified or specified data that you have, being able to solve problems. Again, you can't solve problems with just random stuff, but you can be able to solve problems with data that actually means something that can be validated. so you've got this crazy idea about concrete engine and you kinda got it started where it There's opportunity for and I'll call it like a remodel, right? It's it's you there's spaces out there that have power. they have facility, they have maybe land where they have a spot. And we're we're needing to scale this so that we can, you know, modularly look at what a data center means. Now, and that's something you and I didn't get to talk about when we kind of had this conversation about this. topic, but you know, there's a couple of different types of data centers, right? There's there's ones that just hold data for folks and then there's some that are utilized for the for the models. I and I kinda wanna know where you're at in that in that spectrum. Meltem Ballan, Concrete Engine: Yeah, I mean when we talk about the data centers in the general sense or traditional data centers. They are like I d they are basically AWS, Google, Microsoft. They do everything from hosting. So they are hosting our podcast right now and they do networking and they do obviously they sell GPUs as well. So those Robert: Mm-hmm. Meltem Ballan, Concrete Engine: are traditional generalized data centers. And the GPU as a service or the AI aspects or data aspects of their business is just the part of it. in that model when you're doing some AI or inference work your data and computation might be or might not be in the same location. So they they put the data someplace else, the training happens elsewhere, and then it comes together and then you get the you get the result. For those centers though you need big lens. You need lots of energy. I wrote an article two

years ago and people were discussing about five gigawatts of data centers. And that is the energy that entire New York City uses in the peak time. So it's it's huge. it is a beta we start competing against the data centers with our resources. Am I gonna get the lights on? Am I gonna have the heat? AC we cannot live without AC in Texas. Robert: yeah, right, yeah. Meltem Ballan, Concrete Engine: So those are those are huge data centers. And initially some AI inference or computation companies like the CoreView, Crusoe, they started in a containerized or smaller model, but then you start you start starming your money. So if you're if you raise a lot of money from the venture capital or outside capital, you have a big push and pressure from your investors to scale fast and bring turn that money into billions. If you raise a two hundred million dollar Robert: Mm-hmm. Meltem Ballan, Concrete Engine: your master expects you to bring a billion within three years, five years. Robert: Okay. Meltem Ballan, Concrete Engine: and they they fell in they fell into that trap, but at the same time open AI, later on Anthropic, and now we have Groc, we have some Lama models from Meta, they all started to compete to get they biggest co biggest competition and they they already resort those capacities. And there is another company coming scale. When you go to their website is all resort. And when you look at those resorts, those resorts are all the big companies, five big companies. So the the data center builders And then the inference builders like OpenAI, Anthropic, they are sharing the same resources. So they started to become private, hyperscaler. Private means they only store for AI. So they just those are computation the GPUs filled in the rooms and then they just run Constantly they crunch the numbers. They don't even have storage. The storage are always on the chips so they just get do one thing, they just do the L tr L inference and some persons do the training but mostly inference work. Robert: Okay. So they've got all this basically the hyperscalers are kinda taking all the air out of the room. Right? Meltem Ballan, Concrete Engine: Yes, yes. Hyperscaler took the air on out of the room and the big scalers, the open AI, those companies, took all the resources. So if you wanna build your own AI model, let's say from scratch, not using the rags from OpenAI or Anthropic whatnot, you're not gonna be able to get resources that you need in the given environment, it's very scarce. Robert: So if you wanna do big compute, l like say you're doing medical research or whatever, you know, something like that. but you're not utilizing one of those models, let's say you're trying to do your own research, you're saying there's no room for them to do that work? Meltem Ballan, Concrete Engine: If they wanna scale the the resources are so limited. You have some high performance compu it's called high performance compute facilities by research institutes or the universities. So UT Austin has one, Purdue University has one, Sandia Research Center ha has one, five to ten in the United States. You have to wait. So you're a researcher, you request to do your research on those resources and you have to wait. And yes, yes. Robert: It's like the old days again, right? Remember remember you had the used to have to wait to be able to get on the servers? Like I remember like the big, you know, computer rooms at Purdue. I I you know, graduated Meltem Ballan, Concrete Engine: Yes. Robert: Purdue, yes. but I remember, you know, I had sometimes you had to wait, you know, like, there's not you know, I gotta you gotta wait to be able to log on. Yeah, you couldn't there wasn't enough room for everybody. Meltem Ballan, Concrete Engine: The same, the same. we work with a lot of professors. We don't have a lot of resources, obviously we are a tiny startup, but we work with the professors. The they even though they have the research money, they don't have the resources ready. they go they go try to buy resources elsewhere as well from the hyperscaler perspective, but they don't have the knowledge of building the orchestra this the software layer and that that their grants or money is not designed for that. And now you know there is a mismatch. We are saying AI is going to take us to do you know cutting edge technologies. We are gonna solve the issues with the cancer. We are But then we don't have those resources because we are using all that resources for Chat GPT, chatting with the computers, creating creating beautiful documents and whatnot, but all those resources are used there. And innovation goes as far as what you have. I mean, most people do wipe coding or using the rags, whatnot, but it is It is limited to what they developed and it will be limited to what they developed unless you take an open and look at each piece of those algorithms what what is it about? I think that is coming to another dangerous question because we heard Sam Altman saying, Hey, some of our rogue algorithms ran away and stole some algorithms from Hugging Face because Hugging Phase is the best. And then a couple weeks later, Hugging Phase got bought by another sorry? Nvidia, yes. Robert: NVIDIA. NVIDIA just bot hugging face. Yeah, yeah. Meltem Ballan, Concrete Engine: And now where are we left? All those open source algorithms are becoming closed source. And you might start to notice that Tokens are becoming expensive. All those algorithms are going to be paid closed source. This will bring us to another bottleneck where the innovation is gonna be questioned. Robert: I get I get it. So we're at a we're at another we've we create bottlenecks for ourselves in the in the aims of dollars and cents. Let's talk a little bit more about concrete engine and your ideas around it to kind of solve these bottlenecks for these for these these very things. We need people to be able to do their stuff, right? And and and important work. So let's talk a little bit about, you

know, concrete engine and and your ideas. And that. Meltem Ballan, Concrete Engine: Yeah, when I left Journal Motors I was a little angry. So I went to my I went to my Zen in Purdue University. Robert: Yeah. Meltem Ballan, Concrete Engine: so I was in Purdue for a while and I was actually talking to farmers, I was talking to landowners, I was talking to people that you would think that they are far from the technology. They They might be, but they are the business people. So they all want to be part of these developments, the AI and data data centers. Then I came up twenty twenty four in May I quit my job in Purdue, I quit everything else, and then I came up with this idea that we can build distributed AI factories. And then We can make that work like one big AI data center or work them separate in an air gapped environment that data and AI are not affected by anything else. So my partner, our CTO, Nathan Hansen, was as crazy as me and then joined me. We started to we started our first data center. Initially we decided we are gonna use cheap GPUs and cheap connectivity to build the training backbone. So we we brought the concept of performance performance per dollar. So we built that. It took us two years to build the backbone, the softwares and the resilience tests. so it was a great learning process for us. Then couple months ago we realized the commodity we we we look at the computation as a commodity. The data center is a commodity in the sense where they call it new cloud right now there are different names but the GPU as a service is a commodity. But then we realize it's not true. The innovation is in the st way that you're looking at it. Because what you look and then you start looking at the economies and people, the LinkedIn, people are starting, go fund me and they are all very, very educated people. They're all technologists. They can build these kind of initiatives themselves. So we said, let's go back and then talk to the language that everybody is talking. They they are looking for GPUs. Why don't we talk to landlords and build data centers less than a megawatt or you know, more than a and you know, nothing more than ten megawatt and distribute and put them together. And this will allow some of those technologists who are currently looking for jobs to manage those little data centers distributed in different places and then landlords to leverage their lands they became they became empty because those technologists lost their jobs or working from home. So that is what we started two months ago. So we have several land partners and to everybody's surprise land partners are so willing to do search and upgrades and to pay the improvement fees on their lands and they are very open to different kind of agreements with with the lands. so we're very excited on that. And we are also finding a lot of companies smaller companies who are start resources to to come to this this mix and then we are integrating those two world and then running running the new cloud ourselves. So this is kind of we started from what everybody understands Robert: Mm-hmm. Meltem Ballan, Concrete Engine: and then we will take to the other stage that We might not even need those expensive GPs or competition. So we will come back to that discussion again so that this will really democratize the competition and AI for everybody. And hopefully this is not gonna be the bottleneck of the third winter. Robert: Right. Well, it's a the idea of it seems like we have there's resources out there, kind of untapped resources. construction will be a big part of this because we're gonna have to do some work to get those untapped resources kind of up and running, right? Meltem Ballan, Concrete Engine: Yeah, I mean there are a lot of like from the construction perspective, we see a lot of innovation gonna happen because you know in the data center world you talk about the redundancies, there's the SLAs where you have to you cannot lose the power more than three Rs and when you say that everybody thinks about aisles and aisles and UPS But there are different ways of doing that and there are different ways of cutting the cost. So I mean, I have numbers here. I talked a couple of companies who are in the data center business and they are telling me they need seven million dollars to build a megawatt data center facilities insight. But then When you bring those innovation and different heads together, I mean there are a lot of things that you can actually cut the cost. You can take a let's say you can take a container and a group of people can insulate those containers and then you can take an old generator and then you can schedule that g that generator to die, you know, gracefully so that because you don't wanna do those computation anyways. Like this Monday we lost open AI. The world lost open AI. Who got upset with that? Nobody. If you lose something for twenty minutes, you're not gonna lose it. And there are a lot of different ways to improve on that, to cut the cost. to make make sure that a megawatt or half a megawatt facility is going to be you know successful. and then secondly what we are doing I mean we are doing the so inside that we're the new neo neo cloud and there are a lot of ways to put the GPUs and networking together to improve the competition there as well so we work with the fabric, we work with the orchestration in those. And these are all like accepted as commodity. But there are ways. Innovation is cutting the cutting the cost. Innovation is making sure that all these different mechanisms work efficiently and integrated with with limited budgets. Then everybody can buy GPUs for an amount that they can afford. Robert: So I'm just thinking here, you say you worked a lot with, you know, agriculture and you know, the lot of still a lot of science behind agriculture, lots of the things that need to be developed behind agriculture in order to again feed the world. lots of things that need to happen in

healthcare. what else have you guys been thinking about? Because I when I think about this, again, part of this is already built. Yeah, I think when you and I talked, we were talking about there was, you know, in the farming communities, you know, you've got kind of grain. elevators and there's, you know, three phase power already there. They've got some land kind of around it, those kind of things. You could maybe set up, you know, modular systems there. O obviously you worry about cooling. But again, this is where kind of construction comes in thinking about I I think when people think about data centers, especially in the construction side of the world, they're like, this is a massive tilt wall building. It's got all kinds of mission critical systems and all these kind of things. But you're thinking about this differently. It's like take that scale and kind of pair it down with power that's already there on the grid, that's been allocated. there's there there's land to, you know, build something on or put modular pieces on, whether it's a container. obviously you can figure out the cooling of it. So there's work to be done in that space and and really nobody's doing it. Meltem Ballan, Concrete Engine: Yes, I mean when you look at it now, you know, I was actually chatting with Grogue yesterday. It is it is so funny. and I'm like, who's doing this? and then the answer was everybody is trying to turn fifty megawatt of a land in the in the desert to a data center right now because the hyperscalers are all filled up. But again they are using the same kind of methodology that I can technology. but anything smaller than that, nobody is still doing it. I think the the one of the reasons is that we as Americans we want the bigger, we want the better. And look when you look at the smaller it it it looks like it is not as shiny or it is not as good as the other one. Like for example when you tell somebody, hey I'm selling AMD GPUs in the GPU world, it's not it's not gonna be good unless you have the Nvidia. Now you know they don't even look for H two hundred for the cutting edge and then now it becomes B three hundreds and then they are talking about n next generation GPUs. These are kind of power hungry GPUs and it it's hard to really fit in the smaller environments. Robert: Mm-hmm. Again, I come from the constructor world, right? And I don't necessarily always need to have if I'm going to, you know, bust up some concrete somewhere, you know. I don't necessarily know if I need to bring in, you know, the biggest concrete saws and all these different teams when I'm just busting up a three foot area of concrete. And this is my analogy, right? I could probably do that with a with a jackhammer if I if I needed to for a timely fix. And I'm I'm not having to bring in a team and all this. I'm again I'm just doing a small I'm doing small jobs. and I I like the way that this sounds for me because I it sounds logical. It's like not everybody is trying to solve all

these, you know, they're not they're not running these models that are trying to, you know, be super cutting edge and at the very, very forefront. What they're trying to do is solve specific issues. They're they're getting very focused on something and they're trying to develop an AI that will help them, some AI technology that will actually help them solve that problem. It Meltem Ballan, Concrete Engine: Definitely so for example we've been talking to that world too or we call them off takers. talking to off takers and some of them are okay to work with the three hundred GPUs, five hundred GPUs, so they usually start from three hundred GPUs, five hundred GPUs. Those are kind of research loads. From the three hundred GPU you can actually take a or five hundred GPUs we're talking about B B three hundred in this context. You can actually take a big office building and they have three phase. They have everything ready and then you can either put the containers or you can actually re retrofit this. So you don't even need the container. You can retrofit it. And Robert: And this and this is a big thing. So, like, this is a huge idea here. And this is why I liked it because you stop and think about it. People working from home, not a lot of folks are back in the office anymore. there's a lot of real estate out there, especially commercial real estate. People are like, what are we going to do with all these offices now that we've got remote work with people can work from their laptop? This is a great idea. You've got empty floors and a building. And you utilize that for that kind of space and then we just figure out how to, you know, kind of cool it down. And but, you know, if we're not using those big monster energy suckers, right, they're they're not gonna be as hot. Meltem Ballan, Concrete Engine: Yeah, I mean we're starting to work with you tomorrow, because we we do have our CTO. As I mentioned, he came from Rackspace and he was a founding engineer, so he knew how to turn the shopping malls into a data center. And you can do it if you do it in a shopping mall, you can do it in a smaller scale as well. So I mean you really I mean, you really need to look what is behind the scenes that everybody is seeing right now. And when you look at that, I mean I'm happy to share it here because it's it it is almost like I'm competing maybe other people th I'm not alone in this, other people doing this, but I also understand that there are a lot of need, there are a lot of use cases for everything. You can build smart cities, critical infrastructure with this. You can do what is in front of us right now, the those inferences selling the tokens. But then you know, as soon as you pass this foggy matter with all those LMs, you'll have a lot of different applications and you will be ready for those special vertically integrated applications coming. Then everybody is going to have jobs because construction workers, even given the environment, the data center environment, you're gonna be there maybe a year, two years, then it is done. What are you gonna do? So it's the for the construction workers it's gonna be more democratized more construction workers are going to get jobs without leaving their homes, not going to the distant places and technologies are going to have places to innovate. I mean we are looking at some of the lands where there is the energy, let's say five to ten megawatt of energy. I will only be using for five megawatt, sixteen racks, maybe twenty five thousand square foot. But those places that you're looking at is like five acres, ten acres. Then you can actually turn that into a living research center. So you know you can build a dope park. We can build people coming and having their coffee because the the sense that we think these data centers they're not gonna be buzzing like the big ones. Robert: Right, right. It's kind of the the kind of the epicenter. We build it where we've got power, we've got land, but we don't create this monolith that that just shun shuns people away. We create a a center that provides locally, I would say. I like the idea of locally because I do believe that's how it has to position itself that actually solves problems in that specific area. And then we're not trying to solve the world's problems from this. We're trying to solve our local. We've got our local You know, kind of AI s hub, right? I I can see it that way too. Meltem Ballan, Concrete Engine: Definitely, I mean going back to the farm farmers and agriculture, that's where we I started. So basically precision agriculture, it requires a lot of data. It requires a lot of competition. The biggest enemy that farmers see is John Deere of the world. And that they have to use the John Deere of the world and then they always argue who owns the data. And if they get their if they get the power of the John Deere is gonna be like soon enough is gonna be a technology company, the trucks are gonna be irrelevant, but if you can give the farmer power of the data and understanding of the data, they will monetize because the farmers that I know are the best business people. They they don't farm like olden days they know that they have to they have to farm make money, they have to do the business right. And now they are understanding the the power of the data. I mean they are they have agronomists coming and telling them what to do, what kind of pet pesticides thing or what kind of chemicals that they should be buying, but they they are much better in technology than the those agronomists coming to their lands and they Robert: That's correct. Meltem Ballan, Concrete Engine: started to realize that. And Robert: It's like the difference between builders and designers. You know, we have plenty of people have come to me with a design idea on a on a build and I have had to say, I'm sorry, but that does not work here. And here are the reasons why. And a farmer is the same thing. I'm sorry, that's not gonna work here.

They know their land, they know the soil. They they I mean, it's just it's just logical, you know. They're closest to it. They're gonna have the most intelligence and and need help with utilizing their own data in their space to make better solutions. Meltem Ballan, Concrete Engine: Yes, I mean that is how I started. That is like why don't you own your data? Give me some energy, why don't you own your data? They were obviously skeptical. I'm not there yet. But hopefully we will we will convince one of some of them to build this in their cope. That will I mean that will show a different layer of the economics and I think it will be very, very beneficial for everybody. Robert: Well, I do too, and that's exactly the reason why you're on here today. I want construction to understand there is a different model that we have to start looking at, which is probably far more scalable than what our current model is. there's it's I I always look at the world when I say if I'm going to make improvements, they have to be incremental. And this kind of bite size, like, hey, we need to let's perfect this little model because it it impacts the most people in a positive way. It doesn't impact the most people in a negative way. It's it's we've got, you know, this power, this infrastructure, we've got these areas unused, we've got things that we need to do. I look at construction, I say, construction people, please look at it differently, you know. let's look at this differently. Let's not say we have to be large tilt wall contractors that figure out how to build mission critical systems that are gargantuan. Obviously, those things are going to be out there, but you know, let's think about this model as well because quite honestly, this seems like just a bit more logical to me to say we really need to perfect how to build these smaller areas and build it for the locality so that it works with the community. and works for the community that it's in, which is one of the biggest talking points in in AI right now is how do we, you know, you hear all these hyperscalers and people talking about how do we fix our messaging because we totally screwed up, right? And we need to fix our messaging because now I I've got to figure out a way to sell you this massive data center. And I say, no, I don't think that's a good idea. Let's talk to Concrete Engine. And figure out how we can build something that actually helps that community. Meltem Ballan, Concrete Engine: The I mean definitely that's the way and one of the I think bottlenecks or the stopping points everybody thinks that it's the price of the power. Yes, price of the power is a is a bottleneck. But at the same time if you cut so the price of the power would be high, but if you cut the cost of building those data centers and then cut the costs from other layers the price of the power is not gonna be a big problem because you're actually building your retrofitting on top of something that you know and you can actually adjust that so it's not always the price of the power just pull and push it on that direction but also price of the construction where when you cut it you will be cutting that price from different different angles because hyperscalers maybe get the power so cheap but they spend a lot of money on the construction and running a data center requires a lot of overheads. I I mean they don't r not the human resources because if you have five people running a big huge data centers with that say one gigawatt, you don't need five to ten people to run the entire thing. Robert: Yeah. There's not a lot of jobs. Yeah, and you're right. There's a lot of lines on the balance sheet that need to be utilized, right? when it comes down to the end of the end of the day. But I think it comes back to there needs to be some thought about how data and the data center itself actually gives back to that local community more than just, we're gonna add thirty jobs, you know, over time or gonna a bunch of construction jobs at the beginning and then that they're all gonna go away. I don't know if that's you know, obviously that's the current model which basically says this is how we think we need to do it. My thing is is that who can benefit from that data center that's in that local area, again, I think about hospitals, I think about farmers, I think about even construction companies utilizing localized AI. you know, stop and think about that as well. You know, how How can they utilize that that space, be a part of that so that community benefits from these from these data centers? Yeah. I think it's I think you have a fantastic idea. I really do. I'm I'm hoping it goes great for you. But that's why I wanted to have you on here. Cause I want construction to understand that that's this model that you're that you may be outside of. There may be some construction executives out there that might be outside of this model. There's another thing working that's probably going to I think I think be a big part of it, which is this kind of modular style of data computing, you know, AI infrastructure that we're not even s touching just yet, but you're kinda on the front end of it. Meltem Ballan, Concrete Engine: Yeah, I mean when we set mo when we say modular everybody thinks about containers. It is true we started the containers, but it could be a lot of different things. The enclosing of everything, you know, putting everything together in a safe environment, that requires a lot of innovation. And that is I think that is where the design light construction comes in place and then you know how you re retrofit this in your land. Are you gonna mean you can bury underground, right? That could be an innovation. Lots of lots of different ways of thinking. And we just started and I don't think that we're gonna finish it, but hopefully we will be able to start and we will be able to get the right partners because you know the Construction is one end of it, the land, the the power and then what we do. And then you're also inviting some of the investors in this model because without money, without the construction, without the energy, the computation cannot happen. I mean that's Robert: That's right. Meltem Ballan, Concrete Engine: like that's where the right formula is. when I was first studying the startups, they told me right time, right place and right money, which is true. You know, we have to be in the right place, right time and right money. And we can actually then we can move so fast to show this. with the I think social media it everybody moves so fast. The attention spams spans are so short. And that's what we are trying to catch here. Now the attention span is where we are at and then if we catch that spin, I think we will grow. Robert: That is true. Well, like I said, I think I think you're on the right track here with this. I wanted to have you on the program to talk about this. It's it's a fascinating idea. Definitely wanna stay in touch with you. Meltum, thank you for being with me today and hope everybody in the contact exec enjoys this conversation. And if they do, I wanted you to get response back because I think they're gonna want to hear some more about what you have to say about some of these ideas. So we'll hope to have you back sometime soon. Meltem Ballan, Concrete Engine: Amazing, amazing. It was great to be here on a Friday and I look forward to working with you, Robert. Bye.