The Enterprise Archival Roadmap

Marketing@Archon: looking at Archival as a serious enterprise community, trying to connect almost every one of their strategic enterprise systems into an Archival. An application which probably is running for 10 plus years or should start looking for an Archival solution. It is always not like a lift and shift. There is always a scope where you can reduce what is needed to be archived. That way you will be able to, again, push off a lot of your technical debt. You should have one unified enterprise archive solution, which will probably help you to reduce all your technical depth, all your data management, and still be able to scale. You cannot tell every organization in the world to go start a large enterprise archive, spend the power of their budget on this. Where is it very relevant? According to you, what is the ideal use case for a customer who should be archiving? Hello and welcome all to another session of The Archivist. And I have here with me Mr. Harsha, who heads our operation at Platform 3. And we have an interesting topic for discussion today. We are going to talk to you about what are the challenges that large scale enterprise archiving deployments typically face, the opportunities, the success, and what benefits do large scale enterprise operations carry. Now it's an interesting topic because we are seeing quite a few large organizations, enterprises. looking at archival as a serious enterprise capability, trying to connect almost every one of their strategic enterprise systems into an archive. And it's a lot of investment, let's not kid ourselves, right? It's serious money to connect all your systems, identify data that needs to be archived, bringing that data on a periodic basis, daily, weekly, monthly basis, and then collecting them, managing their lifecycle and purging the data. It's the entire process. is difficult to set up and takes a lot of effort to manage and successfully maintain. So it's definitely an interesting topic and I'm very excited to talk about it today. And Harsha, why don't you start by explaining what it takes for a large enterprise to archive data? What are the typical steps, a summary of what systems are archived or what do you call as a large archival deployment? Thank you, Karka. So ⁓ let's start with who needs to archive. So anybody who has an application which probably is running for 10 plus years will start or should start looking for an archival solution. Having said this, we are seeing a lot of traction. People are now realizing the importance of data archiving. And they do take a lot of... Steps to get into a place where they could probably set up an enterprise archive. But one of the key challenges becomes just pops up as planning for point solutions. So that is what is one of the key things that we'll probably also talk about in this session. But in an ideal situation, the archival solution should suffice all the archival requirements of that particular enterprise, right? Otherwise, there is no point. You will still probably end up with archival solution, one, two, three, four, five. That's not the ideal situation. You should have one unified enterprise archival solution, which will probably help you to reduce all your technical depth, all your data management, and still be able to scale, right? So it's not just a point in time solution that you are planning to. that solution should be able to help your archival requirements now and in the future. So what that means is ⁓ choice of archival solution becomes very important. So now that customers are understanding the importance of archival, And now that more and more people are starting to think about archive, it is imperative that they take a decision to go with the enterprise-wide archive rather going on specific point solution for their archival solutions. And whenever we talk about large enterprises, we have seen basically there could be like 10s, 20s, if not hundreds of technically different applications that are used by the enterprise. which actually complicates the things, right? So it is not managed by one team. It is not managed by one owner, but that could be number of applications into so many application owners, technical owners. So what that means is the system itself is so ⁓ complicated that it becomes very hard to initiate a archival program, but. Once the initial period and once the initial pain is taken and steps are taken to set up an enterprise RKL solution. Most of the customers feel very happy that they have taken those kind of steps. And now they are able to reduce their technical footprint and they also are able to reduce their technical. So it's really a snowball effect, right? Like starting and making the ball roll is very difficult, once it starts rolling, once it gaining traction, it becomes massive, right? But where does... a large enterprise archiving really value? mean, you cannot tell every organization in the world to go start a large enterprise archiving, spend a part of their budget on this. Where is it very relevant? According to you, what is the ideal use case or a customer who should be archiving all their data? So there can be multiple use cases, right? So one common use case can be places where there are lot of acquisitions and there are the places where Right, ⁓ &A is massive. Also there are application modernization or probably from modernization which probably could lead into application duplication. So you'll have, let's say, we have seen customers who use the same product but like five versions of the same product and they're still managing all the five versions. They know that it's not the efficient way of doing things but still they keep doing because of the lights has to be on. The data is there so the lights have to be on. Lights has to be on, yeah. So but if they start thinking that the data is available in an archival solution, they can probably get rid of all those multiple versions. And that could be multiple reasons why they were not able to get rid of these version tie-up, right? So that could be multiple reasons, business reasons as well. But there has to be ⁓ a thought process which actually thinks that archival is the need of the hire. Very true, very true, very true. Now, when you talk about the challenges, right? One of the biggest challenges, like you said, is the people challenge. Availability of the resources, availability of SMEs to tell what to archive, how to archive. How have you solved that problem in ADS? So it's very simple. So there are ⁓ out of the box features that are available within products like Archon Data Store, where we can easily leverage and understand what is there within the product. I mean, what is there within the data? What lays beneath? to understand what actually is in with that particular application, you will be able to take a very informed decision that what needs to be done on what type of data set. So in one of our interactions with one of our clients, they had a need to archive a very large data set. So they were talking in multi... terabyte volumes when we started engaging with them, once we started more discovery with them, we found out that probably not all the data needs the same level of treatment in respect to archiving the data. So probably started at a multi terabyte and finally they ended up in few terabytes. So that is one exercise probably will need to be done in order. So can I take it that a lot of data doesn't need to be archived, can also be pushed? Definitely, yes. So, what that means is ⁓ One of the key requirements for ⁓ compliant archive is to add to regulatory and compliance requirements. So it's always recommended that we follow those guidelines stipulated by the government and all the legal entities. So as long as we are there within that ⁓ guidelines, are always good. So that will always help. it is always not like a lift and shift. There is always a scope where you can reduce what is needed to be archived. That way you will be able to again ⁓ push off lot of your technical depth. Very true, very true. ⁓ so the challenge itself, we talked about the people challenges, we talked about the challenges in the process. What about the challenges with the data itself? See, today defining the data is difficult. ⁓ 90 % of the time when we are dealing with large scale archiving, at least what I have seen is customers saying, I know I want to archive the data. I don't know how I'm going to access the data. If discovery comes, what can I do to access the data? You need to figure it out. We have seen it so many times, right? And it's very classic, very commonplace for customers to think that way. And we have solved it over the years through analyzer to building the metadata. See the concept of archival itself only works when you have a solid metadata that removes the dependency on the SME. So essentially archival is transferring the risk from people to the product. Right. And if we can do that without the help of the customer that is what I think. creates the biggest winner at an enterprise, right? Yeah. So that's one of the key challenges that we always face. ⁓ People know that they want the data, but they don't know what type of data is needed by when, I mean, at what point in time. They probably cannot define what is needed or what is sufficient versus what is not sufficient, right? So that becomes a very ⁓ tricky challenge, wherein a lot of the time the solution supports you in terms of making ⁓ of what is there within the data, right? So what we call the term as metadata. So we will leverage the metadata in such a way that we are able to understand by looking at the metadata, what, where are my data fields, where, which thing refers to which other item, right? So those kinds of things ⁓ in a solution like Archon data store will help us to probably get hold of the data in a very rapid manner. Very true, very true. See, there is also the problem of scale, right? Like, typically in large scale enterprise archiving deployments, we now have an ongoing operation where we have to do several petabytes a day of ingestion, right? The problem of scale of dealing with large volume data is that there is a constraint in type. If you are saying that you are going to analyze 10 petabyte of data in a... 10 years, by the 10th year they will be in 200 petabits. How long are we going to take to analyze that? So how do we manage time and the scale of data that needs to be handled? What's your solution to that? So my choice is we have a scalable model, which probably helps us to scale out whenever needed. And while we are talking about large ⁓ applications, what that means is we'll have a very narrow window wherein we will be able to go get access and access the data. what that means is we'll have a very short runway wherein We are supposed to do lot of analysis and get a lot of understanding on the data, which probably will have to be taken into ⁓ as we archive the data. So that is the key. Very true. Now, the benefit side of things, why would a large enterprise, a large enterprise, more than a billion dollars with the money to spend to manage the infrastructure, spend on archiving. Why would they do that? There could be multiple reasons and factors that probably will push enterprise to do this. So one is ever changing technology. So we do have almost version updates for any type of system that handles data, for example, database, for example. So that could be multiple updates of database, which means that to catch up with the speed of those updates, there is always consistent effort, time, material and money needed to keep it up and running, right? So that is one thing that will always be problematic in the long run. The second is understanding of the data, right? So people will ⁓ not have a lot of details on what type of data resets where, right? So that kind of things probably again could be a challenge. And third is availability of resource, right? So that is, ⁓ mean, nowadays it's always, we know that there are no, there is not going to be a lot of support, meaning ⁓ And not a lot of people left who has the knowledge of the data. Very true. Of the data source. So these kind of challenges always travel along with us. And these can be... Tackled in multiple ways, one of the best way to do this is do a good round of discovery on the data and on the application. Try and ⁓ understand the data model on how the data was ⁓ put into the system and how efficiently we will be able to get the same on the archival site. That way, it becomes very easy for us to manage on the long run. Once the source application is gone, we are still able to get the data in the way that we were able to do it in the ⁓ with a real application. true, very true. doing all this, what do enterprises get? There is significant cost, time, effort in trying to do this. There are obvious benefits, right? There will be like, ⁓ yeah, getting rid of the system, compliance. I'm not talking about soft benefits. What are the hard benefits that they can take? So, ⁓ I mean, benefits I can keep talking for days, right? So those are always available with an archival solution. Key benefit is getting rid of technical debt. So people will be able to ⁓ not needed to manage different versions of same application, for example, or any legacy application. Or non-strategic systems. They all can be cloud. And in many enterprise situations, are ⁓ cloud initiatives. So that can also be supported. We will be able to modernize their ⁓ hard infrastructure, which is... spanned across different stories of building, probably to a cloud-based and more efficiently managed infrastructure. And those kinds of things can be taken as... See, also the hard dollar saving itself, right? I have seen enterprises run a cost center, say millions doing archival to reduce their operational footprint. So that is the hard dollar saving aspect to it too, right? Anyway, let's see, I want to talk about one last thing before we conclude, which is the risk associated with archiving in the large enterprise. What are the typical, the biggest challenges that you have seen? So risk is definitely in there. People cannot just archive and sit there, but ⁓ There are multiple ways that can be leveraged and risk can be mitigated. So it's not going to be a risk that I'll carry on forever. One of the key risk is business users complaining that I'm not able to access or get hold of the data that I want to like I am able to do it today. And in any archival situation on an archival scenario, is not that you are recreating any of your applications. That is never the point. It is how efficiently you are able to manage the data, hold the data, and also unsuffice the requirements that the business users have. So the customer request or internal or external customer request or how these get handled and how efficiently you are able to store the data in an archival solution and how efficiently you able to get it back whenever needed for any audit or legal purposes. So those are definitely going to be very essential. Wonderful, wonderful, Akshay. So why don't we conclude by talking about your most recent experience with the large enterprise. What, how you felt, what were the challenges or how easy was it for you to navigate through and deploy? So there is no one challenge which I can probably talk about. Each customer interaction or each potential customer interaction is always a new challenge to us, right? So even though we have very good backing of a product, a solution, which probably is battle tested for years, still customer's problem keep coming in different shapes and Anything new? The recent one that we were talking about is they don't have any runway but they are planning to accomplish a lot of things. So they have a hard end date which is always fixed. But on the other side, run, more people run, right? So to get to the end point so that we are able to pull the load, right? So that is one unique situation we are into, but. That's not actually unique, man. Every big one I've seen, I mean, very typical, but still exciting, right? It's what keeps us happy. If we don't have a challenge, then what are we going after? Anyways. So guys, thanks for listening. And I think this is going to be useful to any enterprise architect who needs to look at how they want to execute archiving. Hope we had some useful information for you and thank you all. Thank you.