Tool Qualification with Oscar Slotosch

DSLs help shorten development time and reduce errors. This is true for many domains, including safety-critical ones. Yet, as they say, with great power comes great responsibility. What process do you need to follow to use DSLs in safety-critical contexts? How can you make sure that errors in your tools do not harm or kill people? We invited tool qualification expert Oscar Slotosch to find out.

Highlights

“Tools are always helping to improve things, but they are also the source of risks, additional risks. If you have a model from which you generate code, then those code generators can also introduce new errors even if they are quite helpful. The same holds of course for DSL. If you have a DSL that enables you to do cool things, then those cool things can go wrong (…).  You have to discuss and analyze the risks.”

 “If the tool is classified as being critical, then it has to be qualified somehow and there are different qualification methods around depending on different risk classes (…).  If you have a tool that needs qualification, then you have to test it intensively.”

“That’s the benefit for tool providers: the tool user has something like a driver’s license, and the tool provider can provide qualification kits or any other support to make his tool usable in a new market.”

“The risk analysis is done in two steps. The first step is to define whether the tool has an impact on the safety of the developed product (…). The second step is then if there is an impact, we need to think of all potential tool errors. A potential tool error is not a real or a known tool error, that’s everything that can go wrong in the tool. And that’s typically very hard to find out.”

“There needs to be a classification report and there needs to be a qualification report, and of course also a bit like a safety manual that says to the user, “Okay, you have qualified version X of the tool and then for this and this feature, you have to use this.” Maybe also the known bugs need to be considered for this version. Things like that go into the qualification process as well. It’s not only testing things.”

Once we build a good Qualification kit, and building a good Qualification kit also means verifying and validating that is satisfying the requirements, then we know that this is correct. We have a lot of support we can offer people to simplify their work in this field.

Transcript

Sergej: Welcome to Beyond Parsing. This podcast is dedicated to language engineering, creating custom languages, and using them to develop domain-specific software tools.

Federico: We are your hosts, Federico and Sergej, two language engineering consultants.

Sergej: Today we want to talk about tool qualification and we have with us Oscar Slotosch from the company named Validas. Oscar, hello.

Oscar: Hello, Sergej. Nice to be within your show. Thank you for having me.

Sergej: Thank you for coming. Can you tell us a bit about Validas and what you do?

Oscar: Yes. Validas is specialized and that’s the only company worldwide that’s specialized on tool and library qualification, which is in the field of safety, functional safety. I’m the CEO. It’s called Vorstand, I’m leading the company and my purpose is here to acquire new projects and to be, well, let’s say, the interface to the outer world. We have Peter here with us at Validas. He’s interface with Validas and the team. We both are driving Validas: me from the outside and he’s for the inside. Of course, we do work together very well.

Federico: Today, we are interested in discussing tool qualification because of this relation with DSLs. In particular, in our experience as language engineers, we have seen that DSLs can bring strong benefits when we are working in a safety critical environment. This is mainly because DSLs permit to reduce errors by writing less code, by raising the level of obstruction, so that you can match the domain level. By doing this, we reduce the number of errors, and we make the code more testable.

Federico: However, by introducing DSLs, we introduce a new component and a new possible source of problems because basically, if our DSL solution doesn’t work, some parts could lead to bugs in the resulting application. It’s a little bit like having problems in your compiler. Your compiler has some problems, all your applications that you write could potentially have a problem. This is why we think it’s important to qualify your tools and your experience can be very valuable. Did I get this right about the role of tool qualification?

Oscar: Yes. That’s exactly, you framed it in the right way. Tools are always helping to improve things, but they are also the source of risks, additional risks. If you have a model from which you generate code, then those code generators can also introduce new errors even if this is quite helpful. The same of course for DSL. If you have a DSL that enables you to do cool things, then those cool things can go wrong.

Oscar: Well, it is not necessary to qualify every cool thing, every cool tool, but at least, and that’s a general principle in safety, you have to discuss and analyze the risks. Is it a dangerous tool or dangerous DSL or is it not so dangerous? Well, this is called tool classification. Of course, if a tool is classified to be critical, then it needs to be qualified or it can be qualified in order to use it with confidence and use it safely.

Oscar: Well, and the different safety standards are all a bit in the same way. If you want to build a safe system, you need to have safe hardware, safe software, and safe development and testing tools. Well, that’s how they break down the safety of the system and the safety of different aspects.

Sergej: Well, before we talk more about tool qualification, you say that if the tool is classified as critical, it can be qualified. Are there alternatives or is it something I have to do with my tool to use in a critical project?

Oscar: Well, that depends on the classification. If the tool is classified as being critical, then it has to be qualified somehow and there are different qualification methods around depending on different risk classes. In a very high critical risk class or let’s say the highest in automotive is called ASIL D, which was the highest automotive software level. If you have a tool that needs qualification, then you have to test it intensively.

Oscar: But if it’s a lower risk class, let’s say A or B, you can also argue “confidence from use” as a qualification method. Both of these are qualification methods. If a tool is critical, then it needs to be qualified and that’s in all standards, the same procedures, even if they have slightly different qualification methods and classifications of course.

Sergej: But qualification is not just testing, right? You say you have to argue that the tool is somehow safe to use. If I have never heard about tool qualification, what can I imagine this is?

Oscar: That’s a good question. It’s a process. At the end, it’s important that the tool user is using the tool with confidence. This can be achieved in two ways. Either using it carefully, and then verifying that the tool did everything what it did correctly. Then we speak of an uncritical tool, but the user has to do all these verification activities. Then that’s an uncritical tool and you typically write something like a tool safety manual and deliver it to the tool user and say, “Okay, you may use this tool, unqualified, but be careful and follow the safety manual.” And at the end, also document that you have followed the safety manual. If there’s, for example, “You have to review the generated code”, then the people have to document, “Well today is the 23rd of March and we did this review and we found this and this or we found nothing.” But at the end, you have to document that you used your tool carefully enough, then it doesn’t need to be qualified.

Oscar: If it’s classified as critical then you qualify it, let’s say by executing some test cases, and you get a tool qualification report that documents the environment, which you have been using, and it documents the test cases that you have executed, and it documents that everything was successful, and it documents that it is up to ASIL D, possibly, and a lot of things are documented there.

Oscar: Then you get the safety manual just saying, “Okay, use the software version we have qualified,” which is of course far easier to follow than to use the tool carefully and doing many, many steps. In tool qualification, the essential documents are a classification report and, if you decide to qualify it, it’s a qualification report and of course you should have something like a safety manual to follow. Those are the main things of tool qualification.

Sergej: You’re saying depending on my case, I will need to qualify my tool or not. To understand if I need to qualify my tools, is it something that I can do myself? There are some rules, there are some guidance, or should I just contact a company like yours and having a specialist help me in doing this classification in order to understand if I need the qualification or not?

Oscar: Yes, of course, we as Validas offer you free strategy talks and we can discuss this with every tool user or every tool provider that’s interested in discussing with us. But the idea of course is, the responsibility for the tool qualification is always on the tool user side, so the tool user has to use the tools with confidence. Well and what can a tool provider do? Well, he can assist him because if we want to use a tool as a tool user with confidence, we have to document the use cases.

Oscar: Let’s say, which features of the compiler we are using and which inputs and outputs we are compiling, and that is the documentation. The next step is then to analyze the risks. What can go wrong? And of course, if you develop a compiler or DSL or other tool, we know what can go wrong principally and we can support the tool user in doing this classification.

Oscar: That’s what we can offer them. Typically, this is called a qualification kit. A qualification kit is nothing defined in the safety standards, but the qualification kit is everything that helps the user to qualify the tool or at least to classify and use it correctly. That’s a typical thing, a qualification kit. That’s by the way something people pay for. If you are good, you sell them the tool and in addition, you sell them a qualification kit that helps them to use the tool with confidence. Of course, this is something we can support you or other tool providers in building such qualification kits that you can sell to the users to use the tools safely.

Federico: Can you help me understand more clearly the benefits of going through tool qualification? So far, I understand that there are different benefits. The first one would be that I am compliant with the rules, right, first of all? It’s like I get the driver’s license.

Oscar: That’s a very good analogy with the driver’s license. I like that. I like that one.

Federico: But then, are there other benefits like feeling more confident, like knowing that my tools work better? Can you help me understand better the benefits for going through tool qualification?

Oscar: Yes, there are of course two aspects to consider. One is the tool user that will then be safety compliant. The tool user will get the tool qualification report or the safety manual, the confidence he needs. That’s like a driver’s license. Then he can build a product for the safety-relevant domain. That’s for him like driving with a driver’s license. Without a driver’s license, he can’t drive in the safety area.

Oscar: That’s important for the tool user, that he can have this “license” and of course, some safety experts and users are very critical and say, “Oh, I’m responsible for the product and if something goes wrong…”, and by the way, we have seen cars crashing due to tool errors. That’s really nothing trivial. Tools really impact the safety of modern cars. Also, we have seen people spending millions for recalling cars due to tool errors.

Oscar: This is something that they want to avoid, they want to feel safe. If you have a qualification or a quantifiable tool, then they prefer to use it. Klaus Lambertz, which was the CEO of company Verifysoft, built a qualification kit for his coverage tool with us. His feedback was on the exhibition fair, “Oscar, that’s really strange. There are people stopping at our booth that we never knew to exist at all.”

Oscar: That’s like an entry ticket for a new domain for him as a tool provider. That’s the benefit for tool providers. They get new customers in the safety area. I think that’s a big benefit. So, one is the tool user that has something like a driver’s license, and the other is the tool provider that can provide qualification kits or any other support to make his tool usable in a new market.

Federico: Okay. As a tool provider, I may want to qualify my tools because this makes it easier or possible to sell to people using it in a safety critical environment?

Oscar: Well, there is one thing. You’re right, but at the end, the user has to use the tool with confidence. Let’s say if they’re using Windows development platform and you have only been testing on the Linux platform, then you might not know whether it’s working on his side. It’s always, the tool has to be qualified in the environment of the user. You cannot quantify the tool for the user. This only works in very rare cases. You can argue, well we are using Java and there is a Java version and if this Java version is used on Windows or Linux, it doesn’t make a difference where we run the tool.

Oscar: Then you have a very good argument to say, “Okay, we can qualify the tool here with Java version 1.8 and then we know on the user side with Java version 1.8, it will work.” But usually the tool qualification has to be performed on the customer side and that’s why we build qualification kits that can execute and test the tool at the customer side.

Oscar: He will get something like a test automation unit within our tool qualification kit and then he just gets also the test cases and then he just clicks an installer, click, click, click, next, next, next, and then the tests start to run. This can take an hour or two or just 15 minutes depending on what features he’s using and how many test cases we supply. Then we automatically generate the test report and the tool qualification report, showing that everything was qualified correctly. That’s exactly what the standard requires, that it’s tested in the environment of the user.

Federico: Okay. If I understand correctly, if I built my tool, I cannot do the qualification myself because I need to do that for the specific client. However, I can buy this qualification kit, that then is run by every single client, so he can do the qualification on the precise environment on the client. Is this correct?

Oscar: Well, almost correct. You’re right that the qualification has to be at the customer side, but this does not mean that you cannot do it on your side. You just need an argument that running the test cases on your side and the customer side does not make a difference. If this is the case you can have, we call this a prequalified tool, which means a tool that is qualified for some given environment.

Oscar: For example, also a compiler can be a prequalified tool. If you have a certain target and a given set of standard options, maybe two sets or 10 sets, then you can run all those 10 different option sets on the hardware and then you have a prequalified tool and you can go to the customer and say, “Here’s our tool. If you’re using this hardware and one of those 10 different configurations or option sets, then we have qualified it already for you.”

Oscar: Of course, you are allowed to qualify a tool for the customer. But sometimes this doesn’t … For example, typical compilers have so many targets they support and so many options that it is almost impossible for a tool provider to prequalify all targets and all option combinations. Therefore, either they pick only the most relevant ones or they build a Q kit that the user can, on his side, select the hardware and select the options and run test cases.

Sergej: I want to ask a more specific question. So far, we have talked about qualification of tools in general, but can we talk more about the risks or qualification specifically to DSLs or code generators? Can you talk more about the risks involved in qualifying those?

Oscar: Yes, but let me first explain to you how this risk analysis is done. The risk analysis is done in two steps. The first step is to define whether the tool has an impact to the safety of the developed product. The impact can be to introduce an error or to not detect an error. Also testing tools can have an impact. I guess that having a DSL is something that is very constructive and a bit like generating code, then you can, from a correct program, generate the wrong code. You can introduce an error.

Oscar: There is an impact. That’s the first step. The second step is then if there is an impact, we need to think of all potential tool errors. That’s strange. A potential tool error is not a real or a known tool error, that’s everything that can go wrong in the tool.

Oscar: And that’s typically very hard to find out. If you say to me, “Hey, I’ve built here a nice DSL tool for you and I don’t know what can go wrong within it.” That’s where I need your support and you can help customers saying, “Okay, we could do this and this and this wrong.” Well, we come to this back later. Once we assume we have all potential tool errors, then we need to think of measures how we can detect them.

Oscar: In case this, for example, tool crashes. Tool crash is a potential tool error, then it does not produce an output. I as a user will have a high detection probability to see, okay there was no output, then the tool has had a problem. This is very easy to detect and I have a high detection probability and this is an uncritical error, as we say.

Oscar: But if there is another error that [the tool] might be generating just wrong, but compiling code. This is really a mean error that I might not have a good chance to detect. Then we have a low detection probability, which results a higher critical tool and then I need to qualify it. That’s a bit of a thing what we can discuss for DSLs now. What kind of potential errors there could be and how can we detect them?

Oscar: Usually our basic assumption is, everything that a tool is producing can go wrong. If you have a tool with three different outputs, let’s say a linker producing an executable, a mapping file, and a log file, then we have at least three potential errors. The executable can be wrong, the map file can be wrong, and the log file can be wrong. That’s a good starting point.

Oscar: For example, if the log file is not used further then we say, “Okay, the log file has no impact and we don’t need to care about this potential error.” But for the executable and the mapping file, we need to figure out how we could detect this. Maybe by testing the generated software like manual written one, I’m not sure. That’s a bit where we can start discussing whether DSLs are critical tools and need qualification or not. Which is a very interesting discussion that I’d like to do this with you because you’re the expert about DSLs.

Sergej: I guess this will depend on the actual usage of the DSL and what code is generated using the DSL. If it’s for example, just user interface, or if it’s some critical application logic or some infrastructure, boilerplate code, right?

Oscar: Yes. If you generate only test cases, then if the test cases are failing, you have a high probability if they are wrongly generated.

Sergej: Well, but maybe if I generate test cases that always succeed by mistake, then it’s also a problem.

Oscar: But that’s exactly what we need to do, to think about what can go wrong in the test case. For example, in the case we generate test cases in, of course it can all be wrong or they can be incomplete. Let’s say we have a model with three states and we want to test all three states and the test case will only test two states of it. Then we have something and how can we detect this, whether it’s correct or not?

Sergej: Okay. The first step in qualifying a DSL or a code generator would basically be to list all the possible things or all the possible risks, all the possible events that can go wrong, right? Then the second step would be to think about how we can detect if this went wrong. Okay, can we somehow be sure that we listed all the things that could go wrong, that we didn’t forget something? Is it even possible?

Oscar: Yes, that’s what I was saying previously. We start with the outputs of the tool, so of course there can be so many errors and you can have a bug tracker and everything, a new bug, a new bug, you never know whether you have now all bugs recorded or not. But if you look to the outputs of a tool and the tool producing three outputs, then there can be at most three different errors or classes of errors. At least we have something to start with.

Oscar: Usually, we do some basic safety principles. When we look, for example, to the executable, and executable being a test program, then we ask ourselves, is it too early, too late, too much, too little? We have different criteria, different standard questions to ask and, what does it mean, “too little” test? Well, of course maybe there might be dropped test cases or missing test cases and from that, we can make a good, a very systematic way to derive the potential errors.

Oscar: By the way, we are using a model based tool, our toolchain analyzer, to support those analyses. We model the toolchain or we model the tools, we model the inputs, outputs, and then we can just press a button and then for every output there will be something we call a default error generated and we can refine it, split it, or using other creation techniques, derivation techniques, black box and white box models, for example, to come to a systematic error model. You are right. This is very important.

Sergej: I guess this is where your experience would be valuable, right? Because you know what kinds of things could go wrong and what questions to ask.

Oscar: Well, I think the experience in this case is something we share at least, so we have some experience what can go wrong. Where maybe my experience comes in is, how can we argue to detect it? So then you know the process and safety relevant engineering. For example, if you generate code, and in the safety relevant software development, all code has to be tested, every single line of the code has to be tested. Until you’re doing something very mean like compiler, which sometimes from one line of code generates many, many instructions, even branching instruction and other things, by loop unrolling, unfolding, all those very powerful techniques, then you can rely on testing in many cases.

Oscar: A compiler, you cannot rely on testing, because only if you’ll test all object code level statements, then we can say, “Okay, now we have tested really everything and we know there is no compiler error anymore in the code.” But again, this writing into the safety manual, “You may use our compiler provided that you do a 100% object code coverage,” is something that most users won’t like.

Oscar: Therefore they come to us and qualify their compiler. That’s something we provide help. In this case, also tool qualification saves money because testing all object codes down to the level of machine statements is something very expensive ,and if you qualify the tool in the correct way, then you can save a lot of money.

Sergej: You can then argue that I can write my test cases in C or in a higher level language instead of writing them in assembly. Right?

Oscar: Yes. Or at least measuring code coverage in assembly, which is painful enough. You might still write your test cases in C, but you need to ensure that they cover 100% assembly code coverage if you don’t have qualified tools.

Sergej: Okay. The whole process is then first I figure out what can go wrong. Second, I figure out how I could detect it. Third, I write the test cases for that, right?

Oscar: Yes. For the test cases, you need only for those errors that you have not high probability. Your test cases do not need to show that your tool never crashes because there, we have a high detection probability, and it’s quite hard to show by testing that the tool never crashes. But we need to show that those critical errors where we don’t have a high detection probability, that those are not part in the product and that the DSL will work for those safely.

Sergej: Okay. Then I take all those tests, package them in a nice installer, so that the user can run it and press next, next, next, and they will all be run. Is this all there is to qualification or is there more?

Oscar: Well, of course there’s more. There is the documentation of the qualification. This means that there needs to be a classification report and there needs to be a qualification report, and of course also a bit like a safety manual that says to the user, “Okay, you have qualified version X of the tool and then for this and this feature, you have to use this.” Maybe also the known bugs need to be considered for this version. Things like that go into the qualification process as well. It’s not only testing things.

Federico: Let me ask you, when should they start this tool qualification process? Is it something that I should start doing during the design phase or it is something that I should start doing once I finish my tool?

Oscar: Oh, that’s a very, very good question. From the tool provider point of view, you need to start tool qualification once you want to enter in the safety domain. That’s a bit up to you to decide. If you say we have so many customers that don’t care about safety, we don’t need a Q kit, so that’s okay, we don’t need it. Once you decide, “Hey, more and more of our customers are building autonomous driving features and flying software and whatever,” then you might want to have a qualification kit.

Oscar: That’s a bit your decision. But from the tool user side, there are very rigorous decisions. Tool qualification, many people do it very late, which they shouldn’t, because tool qualification is part of safety planning. When you want to build a safe system, you make a safety concept, a safety plan saying, “Okay, we are going to satisfy all the requirements like this and this and this, and we are using those tools.”

Oscar: This is part of safety planning and tool qualification goes into the safety planning phase. Once you have qualified the tool, then you can use them to build the product safely. In principle, the safety standards say, “You have to do the qualification. Well, while you are selecting the tools and while you are planning your product.”

Oscar: By the way, one thing we didn’t talk about is the compliance. Well, the safety standards are consisting of many, many requirements, their goals. Most requirements have IDs. For example, in ISO 26262, the tool quantification is in part eight, chapter 11. Then there are maybe 50 or 80 requirements about how to qualify a tool. But in total, there are thousands of requirements or maybe 1000 or 2000, I don’t know exactly, that the user has to comply with and he has to argue how he’s compliant with that. We call this the compliance report. Saying, “We are compliant to this requirement because we do this and this.”

Oscar: Of course, when we build the qualification kit, we are compliant to this section eight 11 of ISO 26262 and we do have even the TÜV certificate that we are compliant with those requirements because you might ask, as Marc Hoag asked me last week in his podcast, “Where do you know that you are qualified?” Which was a nice question to get asked.

Oscar: We said, “Yes, we have qualified our processes with TÜV.” Once we build a good Q kit, and building a good Q kit also means verifying and validating that is satisfying the requirements, then we know that this is correct. That is a bit of documentation and formalities which is normal in the safety related fields and which we have with our frameworks, a lot of patterns and documents and generators that help people to get easy in this field.

Federico: I’m a really not an expert in these standards, so maybe I would ask a stupid question, but are these standards international standards, European standards, or standards that are typically specific to a single country?

Oscar: No, they are worldwide international standards. ISO is the International Standardization Organization and you can worldwide buy those standards. ISO 26262, means standard number 26262 from ISO International Standardization Organization. There are worldwide committees that are building those standards and they are worldwide available. Well, a bit there is a difference. In the US, they don’t follow ISO 26262 as strictly as we do here in Europe.

Oscar: In Europe, ISO 26262 is the most relevant standard for software in the automotive industry. While in the US, yes, ISO 26262 … I always say, why don’t they need this? Because it’s like a driver’s license. Are they allowed to drive without a license? Well, I don’t know, but maybe they have their own standards, which are more in the US, so there are those standards as well.

Oscar: I think the ISO 26262 is an international standard. To be honest, there are no regulations, for example, even in Germany where you have a lot of things to do. You have to have crash tests and you have to have those diesel tests, which obviously some of our German OEMs didn’t pass very well. There are a lot of things that you need to do before bringing a car on the street. And some of those are not satisfied by some foreign cars. Not every car may drive on German streets, so they don’t get it through these assessments.

Oscar: Nevertheless, functional safety is not part of those preconditions. You may release a car saying just it’s safe and then don’t care about ISO 26262. Just saying ,”We have developed it safely,” or whatever. But if something happens, so if there’s a crash, there will be some lawyers checking why the car is crashing and if the software was involved, like an air bag or whatever, then the law says, you have to have developed your car according to state of the art.

Oscar: Well, what does state of the art? ISO 26262 is state of the art, and therefore, even if it’s not really mandatory to introduce a car, it’s mandatory to have it once something happens. Therefore, I always said it’s like a driver’s license because you can drive without the license. You can just start the engine and drive. But if you get stopped by police or even if a crash happens, then without the license it will be very hard.

Federico: Yeah. Yeah. To me, it seems like insurance. If I don’t run into an accident, okay, you don’t have it, you’re fine. But if I run in an accident and I have to pay a huge amount, if I have this insurance, it covers me from responsibility.

Oscar: Yes. For example, the Uber case. I’m not sure if you know this. The first person in 2018, which was killed by a self-driving car because the safety driver didn’t pay enough attention. What happens? Uber was accused and the main point of accusing was not to have killed a person, but was not to have a safety plan, not following a safety plan. Not having a driver’s license. This was the main point of accusation they had to face. Not having killed somebody, but not having a driver’s license. This was really the main point and I’m not sure how this story will end.

Sergej: I guess the argument is that if they had the safety plan, they wouldn’t have killed anybody, right?

Oscar: Well, that’s something that people might argue, but if you are driving without the license, you might not even kill somebody. You are just not allowed even if you can drive. If then something happens, then this is a double issue.

Sergej: Okay. I think it’s almost time to wrap up this episode. Oscar, is there anything we should have asked you, but didn’t?

Oscar: Oh, there’s so much more to tell about tool qualification. Maybe people want to follow or test our podcast about tool and library qualification. This might be interesting to hear more about.

Sergej: Right. Where can people find your podcast and in general, where can people find more about you and your company and your work?

Oscar: Our podcasts can be found in every iTunes or Spotify or any podcast application or of course, in our web page from Validas.de. There, we are also reachable. There’s a webpage explaining many things we have talked today.

Sergej: Okay. Thank you very much for coming Oscar.

Federico: Thank you.

Oscar: Thank you Sergej for having me and have a nice day and a healthy time.

Sergej: Thank you.

Federico: Thank you. Goodbye.

Sergej: Thank you for listening to this episode of Beyond Parsing. We are your hosts, Federico and Sergej, two language engineering consultants.

Federico: If you want to learn more about domain-specific language and language engineering, visit Beyondparsing.com.

The post Tool Qualification with Oscar Slotosch appeared first on Beyond Parsing.