|
Technical Support
|
Statement Regarding Standard Software Validation & QA Practices
Technical Support Document # RS30001
Last Edited: 10/28/2011
Product(s): All officially released standard software
ReliaSoft often receives questions/requests from customers/third parties regarding software validation (usually related to FDA requirements). This document was created to address such requests.
General Statement
All of ReliaSoft's shrink-wrap products are thoroughly tested and completely validated before commercial release. Our strict validation process, along with our extensive documentation on both the use of the software and the underlying mathematics, ensures (with a very high probability) that all results provided by ReliaSoft applications are valid and correct.
ReliaSoft's validation and quality assurance (QA) procedures were developed independently of the Food and Drug Administration (FDA) validation requirements. Although we believe that ReliaSoft's validation testing is of a much broader nature and far more intensive than the FDA requirements, it is of course the responsibility of each organization to determine whether the use of ReliaSoft's software will be in compliance with any regulatory guidelines they may be subject to. We expect that this document will provide the majority of the information about ReliaSoft's procedures that will be required for you to conduct your evaluation.
ReliaSoft is a Microsoft® Gold Certified Partner.
End User Independent Validation
With all standard shrink-wrap products, ReliaSoft provides extensive documentation in the form of user's guides, help files and theoretical textbooks. The purpose of the documentation is to present the methodologies used (equations and formulations), provide examples and familiarize the end user with the underlying theory.
An end user can do the following:
- Independently confirm that the application behaves as described in these guides.
- Use the provided textbooks to perform additional independent validation of the software results, either by using the formulations given or by using the many examples and solutions provided within these textbooks (many of which are from published papers).
Additionally, comparisons between ReliaSoft's numerical results and published results (by other authors) are also provided with the printed documentation.
Some of these textbooks are posted on ReliaSoft's public, free reliability engineering website at www.weibull.com. Specifically, the following references are available:
- Weibull++®
- Life Data Analysis Reference (Rev. 2005, 580 pages)
- ALTA®
- Accelerated Life Testing Reference (Rev. 2007, 430 pages)
- RGA®
- Reliability Growth Reference (Rev. 2010, 637 pages)
- BlockSim®
- System Reliability Reference (Rev. 2007, 487 pages)
- DOE++®
- Experiment Design and Analysis Reference (Rev. 2008, 438 pages)
The following sections present an overview of the requirements for software development at ReliaSoft.
- Theoretical Formulation and Theory Development
- Code Development
- Source Code Management
- QA and Testing Procedures Overview
- Unit Testing
- Integration Testing
- System Testing
- Release Decision and Quantitative Reliability Growth Monitoring
- Installation and Environmental Testing
- Results Validation Testing
- Custom Systems
- Software Testing (QA) Summary Table
- Issue Tracking and Resolution Process
- Software Deployment
- Independent Software Evaluations
Theoretical Formulation and Theory Development [Top]
All theory and formulations created or implemented by ReliaSoft Corporation must be theoretically sound and correct, accepted by academia and industry experts, and also must be thoroughly validated before becoming part of a standard software product (SSP).
Theory and formulations that are not created by ReliaSoft must meet the following standards:
- Must have been published in reputable academic journals or textbooks and must have had sufficient time for peer review.
- Must have been re-derived and validated by ReliaSoft scientists to assure correctness of the original work.
- All assumptions made during the formulation must be scrutinized, well documented and understood.
- Must be reviewed and approved by ReliaSoft’s theoretical review board and by at least one other independent academic expert on the subject.
Theory and formulations that are developed by a ReliaSoft scientist must meet the following standards:
- Must document all assumptions made during the formulation.
- Must be reviewed and approved by ReliaSoft’s theoretical review board and at least two other independent academic experts on the subject.
- If release of the formulation is not considered to be a competitive advantage, then it should be published in reputable academic journals for peer review. If a competitive advantage exists, then it should be published after the product release either in journals or in ReliaSoft textbooks.
Code Development [Top]
All products and product components developed by ReliaSoft must adhere to the industry-wide accepted standards for developing Windows-based object-oriented applications, including industry standards for graphical user interfaces (GUIs). Multiple authors, including Microsoft and Microsoft Press publications, offer a wide range of articles and books that detail accepted practices. Additionally, such code development must adhere to the guidelines for software development per ReliaSoft's internal document "Coding Practices and Procedures for SSP Software," and to current practices, methods and documents posted in the Development section of ReliaSoft’s intranet.
Microsoft guidelines for developing Windows applications (as detailed in several documents, such as "Application Specification for Microsoft® Windows® 2000") are adhered to. Deviation, if any, from these guidelines must be approved by ReliaSoft’s technical review board and reasons for the deviation must be documented.
Periodic design reviews are conducted with members of the code development team, technical writers and quality assurance. In addition, theoretical and technical review boards are conducted.
Source Code Management [Top]
The guidelines implemented by ReliaSoft for source code management are as follows:
- All source code, including modifications to code, is tracked and controlled through the use of third party source control software. Under no circumstances may code not tracked through source control be introduced into the application.
- All product builds are versioned using a standard Major.Minor.Revision numbering scheme (e.g. 3.1.2).
- All compiled code (components) used in ReliaSoft applications are controlled through ReliaSoft’s configuration and version control processes. All components are capable (through an event or property) of identifying their version and build to a host application.
QA and Testing Procedures Overview [Top]
ReliaSoft has always adhered to the highest quality and reliability standards for all of its software products and services. The quality assurance (QA) and testing procedures for all ReliaSoft products, including custom software, are based on ReliaSoft's commercial standard software product (SSP) practices and methods. The procedures involve detailed and comprehensive testing efforts by multiple individuals and teams over time; thorough documentation of all issues identified during the general testing phase; and independent validation of all methods, theory and calculated results.
Unit Testing [Top]
Most of the low-level testing (i.e. unit testing) is done during development and, at the lowest level, when code is written or changed. Testing is also done at higher-level modules as the product is assembled. This unit testing ensures that components and models function as intended and also ensures robustness. This testing is the responsibility of the individual developers and it involves a test-analyze-and-fix (TAAF) process.
Integration Testing [Top]
Integration testing involves testers looking for bugs within the relationships and interfaces between a pair of components or group of components. (An example is how Weibull++ is integrated into ALTA.) Integration testing is not required if the application is made up of individual utilities that do not share data or invoke one another. However, if the software uses API, shares data or passes control from one component to another then integration testing becomes an important method to verify that the components are working together properly. Integration testing requires that the tester have a firm understanding of how the components are intended to work together.
System Testing [Top]
Once all modules/components have been tested, compiled and linked without any errors or warnings, wider testing is implemented on the complete application or system. Formal logs of system testing activities are kept in ReliaSoft's internal XFRACAS® system (a state-of-the-art, web-based incident/issue management system developed by ReliaSoft). In general, any and all issues found are corrected as they are identified and testing resumes immediately with the corrected version to assure that the fix did not create any new issues. Special emphasis is given to the component or issue that was corrected. This is repeated through Alpha.X, Beta.X and Release Candidate.X phases.
Release Decision and Quantitative Reliability Growth Monitoring [Top]
After all issues have been resolved, a release decision is made. This decision is based on the number and frequency of issues found, as well as formal reliability growth analysis (using ReliaSoft's own RGA reliability growth analysis software) and target reliability goals.
Installation and Environmental Testing [Top]
The last phase of testing is installation and configuration testing. This is done in-house on multiple test systems including most versions of Windows® (e.g., XP, Vista, Windows 7). When appropriate, feedback is also obtained from outside testers. In the case of software that will reach foreign markets, foreign OS (operating system) testing is also performed (e.g., Chinese Windows, Korean Windows).
Results Validation Testing [Top]
In parallel with the software functionality testing, multiple engineers, scientists, and statisticians working independently perform testing and validation of all calculated results. Validated results are thoroughly documented and many of the examples are then illustrated in textbooks that are released with the software. Any issues found are corrected, re-tested and re-validated using multiple data scenarios.
Concluding Remarks [Top]
When ReliaSoft releases a product, it has been thoroughly tested and ReliaSoft is highly confident of its quality and reliability. In the case of custom systems, we expect the client to test the product to assure that the system is functioning as intended in the client environment. For systems that incorporate a non-ReliaSoft database to support functionality and calculations, the client is responsible for taking steps to maintain the integrity of the data stored in the database. An overview of the testing procedures employed by ReliaSoft is presented in the next table.
Software Testing (QA) Summary Table [Top]
|
Testing
|
Testing Type |
Responsibility |
Testing Details |
|
Development Phase |
Unit Testing Integration Testing |
Development QA Theoretical |
Perform ongoing unit testing during development. This will involve testing individual components as they are developed. At this stage, individual components will be tested for correctness of calculations as well as interaction with other components. |
|
General Testing Phase |
Process Testing Integration Testing System Testing |
QA Development Theoretical Documentation Alpha/Beta Sites |
Validate process results for correctness. Track the occurrence and resolution of all anomalous issues. Test the performance of components when they are fully integrated. |
|
Installation Testing Phase |
Installation and Environmental Testing |
QA Development Alpha/Beta Sites |
Test the system internally/externally under various installation environments. For custom systems, work with the client to test the system at the deployment location. |
|
Calculation Validation Phase |
Calculation Validation |
Theoretical |
Validate a representative sample of system calculations by performing identical calculations independently and comparing the results. |
|
Release Phase |
Final Testing |
QA Development Theoretical Documentation Alpha/Beta Sites |
Unit, process, integration and functionality, usability and deployment testing. |
Testing in each phase is cyclical utilizing a test-analyze-and-fix (TAAF) process. All issues after the initial development phase are communicated, documented and resolved utilizing XFRACAS.
Issue Tracking and Resolution Process [Top]
ReliaSoft’s internal XFRACAS system tracks all issues encountered for each product from the development phase and throughout its life cycle. This integrated system forms the basis of our QA procedures. It was designed by ReliaSoft and is currently being marketed as a solution for other companies that have the same stringent quality tracking requirements that we do.
All issues are logged into XFRACAS as incidents and are tracked from creation to resolution. Each incident is logged for the appropriate software entity. Each software product has its own entity designation, which allows ReliaSoft to maintain an organized structure for all of our software products. Incidents can include suggestions from customers or ReliaSoft employees, bugs found during testing, etc. Once the incident has been addressed, it must be tested by the QA group to ensure that the resolution employed correctly addresses the issue and does not introduce any additional problems into the software. XFRACAS facilitates teamwork between developers, theoretical experts, technical writers, QA and management to create a reliable and robust application. In addition, the software product scheduling and completion dates can be more closely monitored.
The next figure shows one of the incident reporting interfaces within XFRACAS.
![XFRACAS New Incident Tracking Utility [Click to Enlarge]](images/XFRACAS_sm.gif)
Software Deployment [Top]
A software application is not deemed ready for deployment until the release candidate has been accepted by ReliaSoft's QA group. The QA group must be satisfied with the current state of the application based on the results of the testing.
Once a product is ready for deployment:
- A master is built for subsequent duplication.
- The source control system is updated to include the final configuration for the product and source code for the build and is then frozen.
- Backups of the source code, build components and machine configuration used for the release build are made, and component versions are documented (including non-ReliaSoft components). Storage and location of redundant backups is done according to ReliaSoft’s Disaster Recovery Protocols.
- The machine used for the build is then isolated and frozen and may not be used for any purpose other than subsequent builds (revisions) of the same version of the application.
Independent Software Evaluations [Top]
ReliaSoft maintains validation records for all current versions of commercially released applications, including detailed records of all modifications and issues found and addressed during the testing phase and subsequent deployment. These records are treated as "Proprietary and Confidential" and are not made available to the general public. In special circumstances, ReliaSoft is willing to make such records available to clients or other third parties under the following conditions (and at ReliaSoft's sole discretion):
- Review of such records is at ReliaSoft's Corporate Headquarters (Tucson, AZ) and in the presence of ReliaSoft personnel.
- If needed (depending on the nature/amount of the disclosed information) a "Non-Disclosure Agreement" is executed.
- Depending on the amount of time required, a fee may be charged to cover ReliaSoft's costs.
