Code Reviews and Testing Third Party Software

An in-depth study of software audit is beyond the scope of this book, it is however necessary to touch on the subject. In earlier chapters testing methodologies that also relate to software have been described. These range from the black box test commonly used when code is unavailable (such as in the case of third-party software reviews and reviews package software) through to white box and crystal box assessments. In the latter, all code is available and tested.

It is not essential that the auditor understands the intricacies of coding. Rather, it is sufficient to understand how the various testing approaches function and to have sufficient understanding to be able to work with the test engineer who has designed the test cases associated with software in order to be able to understand their work. In particular, the auditor should be able to understand the reports produced by the test engineer.

We shall quickly rehash the types of software audit and review before going further. At the extremes these are:
Black box testing
Black box software testing does not require any understanding of internal behavior. No access to code is available, but rather the response to input is validated. UML diagrams may be available in some instances and in this case a test of functionality will be matched to the functional requirements in the specification. In any event, input will be matched to output to test for expected or unexpected behavior. Some of the various testing methods include:

• Equivalence partitioning,
• Boundary value analysis,
• All-pairs testing,
• Fuzzing,
• Model-based testing, and
• Traceability matrix.

White box testing
This type of testing includes access to the internal data structures. At the extreme (crystal box tests), the tester has access to all code, algorithms and design notes. White box testing will include tests to ensure predefined criteria have been met. Some examples of this form and testing include:

• Static code testing,
• Mutation testing,
• Completeness testing,
• Fault injection testing, and
• Lexical code analysis.

Testing in Combination
The most effective means of testing software comes from combination of methods being deployed together. Unfortunately, access to code is not always available. In cases of packaged software and many third-party products, access to the code is restricted. Access to code is also effective in increasing the capabilities of the traditional black box test (commonly called a grey box test when code is available to conduct the test using black box test methods).
Correcting a software problem after the event is far more expensive than stopping it before it goes into production release. It is often stated that post-release fixes are in the order of hundreds of times more expensive to fix then when compared to correcting the issue in code and requirements reviews.

When auditing software is necessary to consider the following aspects of development associated with the code:

• Software Quality

Correctness,

Completeness,

Integrity,

• Capability,
• Reliability,
• Efficiency,
• Portability,
• Maintainability,
• Compatibility, and
• Usability.

Test engineers will generally develop metrics to report on each of these aspects of software development.

The various Levels of Testing
Unit testing
Unit testing focuses on individual software modules (the components of the software). Each module is tested individually in order to validate the software implementation component by component. An example would be the testing of individual classes associated within an object-oriented development environment.

Integration testing
Integration testing is designed to uncover defects in the interfaces and interaction amid the integrated software modules. This form of testing starts with individual modules and joins them to form progressively larger associative groups. Each phase works on larger groupings until the software architecture is tested as an entire system.

Acceptance testing
Acceptance testing is conducted by the end-user. The goal is to decide whether or not to accept the final software product. Acceptance testing may be conducted between development phases.

Regression testing
Regression testing is a process where a previously conducted test is a rerun on the software. This type of testing is conducted in order to ensure that prior defects have not been reintroduced or regressed into the code. This type of testing is frequently automated.
Some specific types of regression testing include sanity testing (this is a check for unexpected and unforeseen behavior) and smoke testing (which is a test to ensure that the product provides basic functionality).

Test Cycles
There are many ways of engineering software. Each of these comes with its own test methodologies. One of the more common ones is the Software Development Life Cycle (SDLC). Some of the common foes is involved with testing include many phases of the project that are analogous to many other audit processes.

Requirements analysis
The first stage of testing generally starts with the creation of a document detailing what is necessary. In this phase both developers and testers will work together to determine what tests may be conducted.

Test planning
This phase includes the creation of a strategy and the scope of the testing. Like an audit, system testing should be conducted as a project. Some areas to consider include:

1. The creation of a test strategy,
2. The formulation of a test plan, and
3. The creation of a test bed or other testing system.

Test development
The development phase of testing involves the creation of a number of test procedures based on the requirements derived in the preceding stages. Some of the steps involved with this phase of testing include:

1. The development of test procedures,
2. The creation test scenarios,
3. Creating test cases and populating simulated data,
4. The creation of test programs and scripts and possibly the sourcing of third-party testing software (such as the static analysis platforms by Fortify).

Test execution
The test execution phase involves the actual testing of the software based on the processors decided above. Any errors or defects in the code would then be reported to the development team.

Test reporting
Test metrics that were developed in the preceding stages coupled with data concerning errors and defects and possibly recommendations for improvement. This will also include recommendations whether the software needs further testing before being released.
Retesting the defects

Defects may be the result of either errors in the code or the test process itself. It is necessary to ensure that any defects that are a result of the testing process are rectified. Defects may or may not be corrected. Many defects do not have a security related consequence and could be left for future software versions.