Statement: Standard ReliaSoft Software Validation & QA Practices

 

ReliaSoft often receives questions/requests from customers/third parties regarding software validation (usually related to FDA requirements). This document was created to address such requests.

---

 

General Statement

All of ReliaSoft's shrink-wrap products are thoroughly tested and completely validated before commercial release. Our strict validation process, along with our extensive documentation on both the use of the software and the underlying mathematics, ensures (with a very high probability) that all results provided by ReliaSoft applications are valid and correct.

 

ReliaSoft's validation and quality assurance (QA) procedures were developed independently of the Food and Drug Administration (FDA) validation requirements. Although we believe that ReliaSoft's validation testing is of a much broader nature and far more intensive than the FDA requirements, it is of course the responsibility of each organization to determine whether the use of ReliaSoft's software will be in compliance with any regulatory guidelines they may be subject to. We expect that this document will provide the majority of the information about ReliaSoft's procedures that will be required for you to conduct your evaluation.

 

End User Independent Validation

With all standard shrink-wrap products, ReliaSoft provides extensive documentation in the form of user's guides, help files and theoretical textbooks. The purpose of the documentation is to present the methodologies used (equations and formulations), provide examples and familiarize the end user with the underlying theory.

 

An end user can do the following:

 

• Independently confirm that the application behaves as described in these guides.
• Use the provided textbooks to perform additional independent validation of the software results, either by using the formulations given or by using the many examples and solutions provided within these textbooks (many of which are from published papers).

Additionally, comparisons between ReliaSoft's numerical results and published results (by other authors) are also provided with the printed documentation.

 

Some of these textbooks are posted on ReliaSoft's public wiki. Specifically, the following references are available:

 

• Weibull++
  • Life Data Analysis Reference
  • Experiment Design and Analysis Reference
• ALTA
  • Accelerated Life Testing Reference
• RGA
  • Reliability Growth and Repairable System Analysis Reference
• BlockSim
  • System Reliability Reference

The following sections present an overview of the requirements for software development at ReliaSoft.

 

• Theoretical Formulation and Theory Development
• Code Development
• Source Code Management
• QA and Testing Procedures Overview
• Unit Testing
• Integration Testing
• System Testing
• Installation and Environmental Testing
• Results Validation Testing
• Custom Systems
• Software Testing (QA) Summary Table
• Issue Tracking and Resolution Process
• Software Deployment

---

Theoretical Formulation and Theory Development

All theory and formulations created or implemented by ReliaSoft must be theoretically sound and correct, accepted by academia and industry experts, and also must be thoroughly validated before becoming part of a standard software product (SSP).

 

Theory and formulations that are not created by ReliaSoft must meet the following standards:

 

• Must have been published in reputable academic journals or textbooks and must have had sufficient time for peer review.
• Must have been re-derived and validated by ReliaSoft scientists to assure correctness of the original work.
• All assumptions made during the formulation must be scrutinized, well documented and understood.
• Must be reviewed and approved by ReliaSoft’s theoretical review board and by at least one other independent academic expert on the subject.

Theory and formulations that are developed by a ReliaSoft scientist must meet the following standards:

 

• Must document all assumptions made during the formulation.
• Must be reviewed and approved by ReliaSoft’s theoretical review board and at least two other independent academic experts on the subject.
• If release of the formulation is not considered to be a competitive advantage, then it should be published in reputable academic journals for peer review. If a competitive advantage exists, then it should be published after the product release either in journals or in ReliaSoft textbooks.

---

Code Development

All products and product components developed by ReliaSoft must adhere to the industry-wide accepted standards for developing Windows-based object-oriented applications, including industry standards for graphical user interfaces (GUIs). Multiple authors, including Microsoft and Microsoft Press publications, offer a wide range of articles and books that detail accepted practices. Additionally, such code development must adhere to the guidelines for software development per ReliaSoft's internal document "Coding Practices and Procedures for SSP Software," and to current practices, methods and documents posted in the Development section of ReliaSoft’s intranet.

Microsoft guidelines for developing Windows applications (as detailed in several documents, such as "Application Specification for Microsoft® Windows® 2000") are adhered to. Deviation, if any, from these guidelines must be approved by ReliaSoft’s technical review board and reasons for the deviation must be documented.

Periodic design reviews are conducted with members of the code development team, technical writers and quality assurance. In addition, theoretical and technical review boards are conducted.

---

Source Code Management

The guidelines implemented by ReliaSoft for source code management are as follows:

• All source code, including modifications to code, is tracked and controlled through the use of third party source control software.  Under no circumstances may code not tracked through source control  be introduced into the application.
• All product builds are versioned using a standard Major.Minor.Revision numbering scheme (e.g., 3.1.2).
• All compiled code (components) used in ReliaSoft applications are controlled through ReliaSoft’s configuration and version control processes. All components are capable (through an event or property) of identifying their version and build to a host application.

---

 

QA and Testing Procedures Overview

ReliaSoft has always adhered to the highest quality and reliability standards for all of its software products and services. The quality assurance (QA) and testing procedures for all ReliaSoft products, including custom software, are based on a scaled agile framework which is facilitated using CA Agile Central. The scaled agile process involves detailed and comprehensive testing efforts by multiple individuals and teams over time specific iterations; thorough documentation of all issues identified during each iteration; and independent validation of all methods, theory and calculated results.

Unit Testing

These tests include low level testing at 'unit' and 'integration' levels. They generally test libraries etc. and are therefore performed by developers. Their purpose is to ensure that the software produces the correct results (e.g. 2+2 = 4). They must be completed before System Tests can be concluded. This unit testing ensures that components and models function as intended and also ensures robustness. This testing is the responsibility of the individual developers and it involves a test-analyze-and-fix (TAAF) process.

Integration Testing

Integration testing involves testers looking for bugs within the relationships and interfaces between a pair of components or group of components. (An example is how Weibull++ is integrated with ALTA.) Integration testing is not required if the application is made up of individual utilities that do not share data or invoke one another. However, if the software uses API, shares data or passes control from one component to another, then integration testing becomes an important method to verify that the components are working together properly. Integration testing requires that the tester has a firm understanding of how the components are intended to work together.

System Testing

Once all modules/components have been tested, compiled and linked without any errors or warnings, wider testing is implemented on the complete application or system. Formal logs of system testing activities are kept in our Bug Logging and Tracking system - CA Agile Central. In general, any and all issues found are corrected as they are identified and testing resumes immediately with the corrected version to assure that the fix did not create any regression issues. Special emphasis is given to the component or issue that was corrected. This is repeated throughout each iteration of the development phase.

Installation and Environmental Testing

The last phase of testing is installation and configuration testing. This is done in-house on multiple test systems including most versions of Windows® (e.g., Windows 7 & 8.1, Windows 10). When appropriate, feedback is also obtained from outside testers. In the case of software that will reach foreign markets, foreign OS (operating system) testing is also performed (e.g., Chinese Windows, Korean Windows).

Results Validation Testing

In parallel with the software functionality testing, multiple engineers, scientists, and statisticians working independently perform testing and validation of all calculated results. Validated results are thoroughly documented and many of the examples are then illustrated in textbooks that are released with the software. Any issues found are corrected, re-tested and re-validated using multiple data scenarios.

Release Decision

When ReliaSoft releases a product, it has been thoroughly tested and ReliaSoft is highly confident of its quality and reliability. After all issues have been resolved and validated, a release decision is made. This decision is based on the number and frequency of issues found. Before software release can commence, the Release Manager must ensure that all test documentation has been signed-off (and approval for release has been given) by the Test Manager.

Concluding Remarks

In the case of custom systems, we expect the client to test the product to assure that the system is functioning as intended in the client environment. For systems that incorporate a non-ReliaSoft database to support functionality and calculations, the client is responsible for taking steps to maintain the integrity of the data stored in the database. An overview of the testing procedures employed by ReliaSoft is presented in the next table.

---

Software Testing (QA) Summary Table

 

Testing in each phase is cyclical utilizing a test-analyze-and-fix (TAAF) process. All issues after the initial development phase are communicated, documented and resolved utilizing CA Agile Central.

---

Issue Tracking and Resolution Process

CA Agile Central tracks all issues encountered for each product from the development phase and throughout its life cycle. This integrated system forms the basis of our QA procedures. All issues are logged into CA Agile Central as incidents and are tracked from creation to resolution. Each defect is logged for the appropriate team to address. Defects can include suggestions from customers, employees, bugs found during testing, etc. Once the defect has been addressed by development, it must be tested by the QA group to ensure that the resolution employed correctly addresses the issue and does not introduce any additional regression issues. CA Agile Central facilitates teamwork between developers, theoretical experts, technical writers, QA and management to create a reliable and robust application.

The next figure shows one of the defect reporting interfaces within CA Agile Central.

---

Software Deployment

A software application is not deemed ready for deployment until the release candidate has been accepted by ReliaSoft's QA group. The QA group must be satisfied with the current state of the application based on the results of the testing.

Once a product is ready for deployment:

• A master is built for subsequent duplication.
• The source control system is updated to include the final configuration for the product and source code for the build and is then frozen.
• Backups of the source code, build components and machine configuration used for the release build are made, and component versions are documented (including non-ReliaSoft components). Storage and location of redundant backups is done according to ReliaSoft’s Disaster Recovery Protocols.
• The machine used for the build is then isolated and frozen and may not be used for any purpose other than subsequent builds (revisions) of the same version of the application.