Software tool qualification is a process that more and more safety-critical developers have to conduct. Let’s go through the basics of whether and why you need to validate your software tools, and how to go about the qualification process in practice!
As digitalization transforms even the rigid landscape of safety-critical technology, the robustness and reliability of software content is becoming a central question. Whatever the extent they rely on software, the correct functioning of safety-critical products is absolutely crucial, as human lives may be at stake.
It is no wonder, then, that growing scrutiny is placed on not only the products themselves and their delivery processes, but also on the software tools used in the development of all that safety-critical software embedded in these products. Such tools (including code compilers, ALM platforms, issue tracking tools, etc) can directly impact the quality, safety, and reliability of safety-critical end products. Therefore, more and more regulations require the qualification of the software development tools used in the delivery of these technologies.
Software tool validation
In essence, the goal of qualifying software tools is to determine a level of confidence in the use of these solutions in the delivery of safety-critical technology. This process is crucial in ensuring that development processes are in compliance with relevant standards and regulations.
In this article, we’re taking the example of ISO 26262 which governs automotive systems development processes, and also calls for tool qualification in Volume 8. While the exact method of tool qualification may not be generalized, it’s true that more and more regulations in other industries call for the vetting and validation of software tools. Understanding the general principles of this process can benefit developers operating in other industries, too.
The qualification of software tools is far from straightforward. Developers use a variety of tools across all stages of the product lifecycle (including commercial, open-source, as well as homegrown software solutions), most of them interconnected. In order to carry out a rigorous qualification, there needs to be a clear understanding of the intended use, workflows, artifacts, inputs and outputs, etc of each of these tools.
Related webinar:
codeBeamer ALM TÜV Certification: Simplify Tool Qualification for Safety-critical Development
So first off, it makes sense to determine just which tools a developer needs to qualify, and which ones are OK to use without any qualification.
What tools need to be qualified?
In order to answer that question, you’ll start with asking yourself: “is it possible that this tool will be responsible for any kind of error in my final end product?”. Make sure you also take into account "the possibility that the malfunctioning software tool can ..fail to detect errors in a safety-related item….“ (ISO 26262-8, 11.2). In other words, the fault doesn’t have to be caused by the tool itself: if it merely fails to detect the error, that alone renders it a subject of qualification.
If your answer is yes, then you have a high confidence in the tool and consequently you’ll need to quality it before using it.
Of course, it’s not all that simple – for instance, ISO 26262 defines 3 levels of tool confidence. For the lowest level (TCL1) no qualification is needed, while TCL3 means a high confidence level which requires you to demonstrate the reliability of the tool via a qualification procedure. So what factors determine tool confidence levels, and thus, the rigour of qualification you’ll need to apply?
Software tool analysis and qualification process
The path towards qualifying your software tools will generally follow these steps:
1) Planning
The process of tool qualification begins with planning. You’ll plan and prepare a list of all the software development tools you’ll be using in your project. Then, you’ll begin tool classification analysis.
2) Tool classification analysis
Classifying your software tools corresponding to the confidence levels that your relevant regulation (in this case, ISO 26262) outlines is based on a multi-step process:
You’ll start with analyzing tool impact, e.g. whether your tool can introduce errors or fail to detect them in the end product. You can simply split tools into two categories. From a qualification perspective, you’ll disregard tools that do not impact product quality.
Then, you will analyze the tools that you have found can impact the quality of your end product based on tool error detection. In this step you’ll be looking at the likelihood of detecting any errors that the tool causes in the process of product delivery. If you’re sure any errors will be discovered, you won’t be needing a high confidence in the tool’s correct functioning. If, however, you find that malfunctions may go unnoticed, you’ll need to have a high level of confidence that the tool behaves and functions as it should.
Based on tool impact and detection, you’ll assign a TCL (Tool Confidence Level) to each of your software tools, where TCL 1 means low confidence, while TCL 2 and 3 mean medium and high confidence levels respectively.
3) Performing tool qualification
As per ISO 26262, the following methods are available for the actual process of tool qualification:
- 1a. Increased confidence from use
- 1b. Evaluation of the development process
- 1c. Validation of the software tool
- 1d. Development in compliance with a safety standard
You can rely on increased confidence from use if you have used the tool in previous projects (1a). Alternatively, you can also evaluate the process of tool development (1b), but this may not be feasible if you’re using a COTS (commercial off-the-shelf) product as your vendor is not likely to let you in on the nitty-gritty details of their product development efforts. In such cases, the vendor can substitute this by acquiring a certification by a certification authority such as TÜV.
For instance, if you’re using Intland Software’s products, you can rely on our certifications to help you tackle tool qualification & validation procedures. codeBeamer ALM is a TÜV Trusted Tool to support software development as per the requirements of ISO 26262:2011 and IEC 61508:2010.
Related reading: codeBeamer Receives TÜV “Trusted Tool” Certification
If none of these procedures work, you’ll need to carry out the validation of the software tool yourself (1c), meaning that you will test the product’s functioning against your intended use case. That is, all your intended use cases: you’ll need to carefully map what features of the tool you’ll be using in your project, and test the correct functioning of each and every one of those features. Naturally, this requires immense amounts of costly manual work.
If your chosen vendor provides a validation kit for the tool, you can greatly reduce the costs of this process. Intland Software offers Validation Kits to simplify and accelerate tool qualification in various industries.
Interested in accelerating ALM validation? Learn more about our Validation Kits!
1d is generally not feasible, as the tools used in the delivery of safety-critical embedded solutions are not developed according to a safety standard.
Overall, the qualification of software tools isn’t rocket science – but it’s a lengthy, effort-intensive, and costly process. Make sure you select development tools whose vendors offer you all the help they can to save valuable resources: look for certified tools or at least solutions that come with validation kits to simplify qualification!
