Theoretical Formulation and Theory Development    [Top]

All theory and formulations created or implemented by ReliaSoft Corporation must be theoretically sound/correct and accepted by academia and industry experts and also must be thoroughly validated before becoming part of a standard software package (SSP).

Theory and formulations that are not created by ReliaSoft must meet the following standards:

• Must have been published in reputable academic journals or textbooks and must have had sufficient time for peer review. 

• Must have been re-derived and validated by ReliaSoft scientists to assure correctness of the original work.

• All assumptions made during the formulation must be scrutinized, well documented and understood.

• Must be reviewed and approved by ReliaSoft’s theoretical review board and by at least one other independent academic expert on the subject.

Theory and formulations that are developed by a ReliaSoft scientist must meet the following standards:

• Must document all assumptions made during the formulation. 

• Must be reviewed and approved by ReliaSoft’s theoretical review board and at least two other independent academic experts on the subject.

• If release of the formulation is not considered to be a competitive advantage, then it should be published in reputable academic journals for peer review or, if a competitive advantage exists, then it should be published after the product release either in journals or in ReliaSoft textbooks.

Code Development  [Top]

All products and product components developed by ReliaSoft must adhere to the industry-wide accepted standards for developing Windows-based object-oriented applications, including industry standards for graphical user interfaces (GUIs). Multiple authors, including Microsoft and Microsoft press publications, offer a wide range of articles and books that detail accepted practices. Additionally, such code development must adhere to the guidelines for software development per ReliaSoft's internal document "Coding Practices and Procedures for SSP Software," and to current practices, methods and documents as posted in the Development section of ReliaSoft’s Intranet.

Microsoft guidelines for developing Windows applications (as detailed in several documents, such as "Application Specification for Microsoft® Windows® 2000") are adhered to. Deviation, if any, from these guidelines must be approved by ReliaSoft’s technical review board and reasons for the deviation must be documented.

Periodic design reviews are conducted with members of the code development team, technical writers and quality assurance. In addition, theoretical and technical review boards are conducted.

Source Code Management   [Top]

The guidelines implemented by ReliaSoft for source code management are as follows:

• All source code, including modifications to code, is tracked and controlled through the use of third party source control software.  Under no circumstances may code not tracked through source control  be introduced into the application.

• All product builds are versioned using a standard Major.Minor.Revision (e.g. 3.1.2) numbering scheme.

• All compiled code (components) used in ReliaSoft applications are controlled through ReliaSoft’s configuration and version control processes. All components are capable (through an event or property) of identifying their version and build to a host application.

QA and Testing Procedures Overview   [Top]

ReliaSoft has always adhered to the highest quality and reliability standards for all of its software products and services. The Quality Assurance (QA) and testing procedures for all ReliaSoft products, including custom software, are based on ReliaSoft's commercial Standard Software Products (SSP) practices and methods. The procedures involve detailed and comprehensive testing efforts by multiple personnel/teams over time; thorough documentation of all issues identified during the general testing phase; and independent validation of all methods, theory and calculated results.

Unit Testing  [Top]

Most of the low-level testing (i.e. unit testing) is done during development and, at the lowest level, when code is written or changed. Testing is also done at higher-level modules as the product is assembled. This unit testing ensures that components and models function as intended and also ensures robustness. This testing is the responsibility of the individual developers and it involves a test-analyze-and-fix (TAAF) process.

Integration Testing  [Top]

Integration testing involves testers looking for bugs within the relationships and interfaces between a pair of components or group of components. An example is how Weibull++ is integrated into ALTA. Integration testing is not required if the application is made up of individual utilities that do not share data or invoke one another. However, if the software uses API, shares data or passes control from one component to another then integration testing becomes an important method to verify that the components are working together properly. Integration testing requires that the tester have a firm understanding of how the components are intended to work together.

System Testing   [Top]

Once all modules/components have been tested, compiled and linked without any errors or warnings, wider testing is implemented on the complete application or system. Formal logs of application testing activities are kept in ReliaSoft's internal XFRACAS (an internal system that tracks all issues found on a product, along with their formal resolution). In general, any and all issues found are corrected as they are identified and testing resumes immediately with the corrected version to assure that the fix did not create any new issues. Special emphasis is given to the component or issue that was corrected. This is repeated through Alpha.X, Beta.X and Release Candidate.X phases.

Release Decision and Quantitative Reliability Growth Monitoring   [Top]

After all issues have been resolved, a release decision is made. This decision is based on the number and frequency of issues found and on formal reliability growth analysis (using reliability methods, including ReliaSoft's own RGA reliability growth and repairable systems analysis software) and target reliability goals.

Installation and Environmental Testing   [Top]

The last phase of testing is installation testing and configuration. This is done in-house on multiple test systems including most versions of Windows (e.g. 95/98/NT/2000/Me). When appropriate, feedback from outside testers is also obtained. In the case of software that will reach foreign markets, foreign OS (operating system) testing (e.g. Chinese Windows, Korean Windows, etc.) is also performed.

Results Validation Testing  [Top]

In parallel with the software functionality testing, multiple engineers, scientists, and statisticians working independently perform testing and independent validation of all calculated results. Validated results are thoroughly documented and many of the examples are then illustrated in textbooks that are released with the software. Any issues found are corrected, re-tested and re-validated using multiple data scenarios.

Custom Systems   [Top]

When ReliaSoft releases a product, it has been thoroughly tested and ReliaSoft is highly confident of its quality and reliability. In the case of custom systems, we expect the client to test the product to assure that the system is functioning as intended in the client environment. For systems that incorporate a non-ReliaSoft database to support functionality and calculations, the client is responsible for taking steps to maintain the integrity of the data stored in the database. An overview of the testing procedures employed by ReliaSoft is presented in the next table.

Software Testing (QA) Summary Table  [Top]

Testing in each phase is cyclical utilizing a test-analyze-and-fix (TAAF) process. Problems are communicated, documented and resolved utilizing XFRACAS, a state-of-the-art incident/issue management system developed by ReliaSoft. All issues after the initial development phase are managed through the system.