One solution to this problem is to use white-box testing in addition to black-box testing. White-box testing strategies include designing tests such that every line of source code is executed at least once, or requiring every function to be individually tested.
Very few white-box tests can be done without modifying the program, changing values to force different execution paths, or to generate a full range of inputs to test a particular function. Traditionally, this modification has been done using interactive debuggers, or by actually changing the source code. While this may be adequate for small programs, it does not scale well to larger applications. Traditional debuggers greatly affect the timing, sometimes enough so that a large application will not run without major modifications. Changing the source code is also unwieldy on a large program that runs in a test bed environment.
There are a number of testing tools that let you perform white-box testing on executables, without modifying the source and without incurring the high overhead of an interactive debugger. Advantages of this approach include:
- These tools speed testing and debugging because there is no need to wait for test support code to be inserted into the program. In commercial software development, with many developers and a separate department in charge of testing and integration, this can save significant time.
- Best use can be made of a test environment. For example, running a 747 simulator requires a specific hardware setup. If each tester changes the loaded software on the simulator configuration, then each test takes much longer to set up and each testing scenario is much more error prone. A 747 test environment is expensive to maintain. The faster testing can occur, the cheaper the cost of the testing process.
- Source code may not be available for all of the software. It is common to use third-party products or software delivered from another organization as part of any commercial software effort. These typically do not ship with source code and certainly do not ship with the ability to rebuild them after changing the source code.
- It is better to test the actual executable that will be delivered, rather than a special testing executable. For one thing, it eliminates extra error-prone steps in the testing process. It also lets the testing scenario be more easily repeated on demand, rather than the single shot effort that typically occurs with source code modification.
Most white-box testing tools change the executable in one way or another, or check for certain classes of failures. Rational's Purify (#-Link-Snipped-#), for instance, modifies the compiled code so each load or store initiates a check of the memory location against specific criteria. BugTrapper from Mutek (<a href="https://www.mutek.com/" target="_blank" rel="nofollow noopener noreferrer">MUTEK – Card Dispensers, Card Readers & Card Dispenser Kiosks</a>) alters the executable to record all of the system calls. Applied Microsystems LiveCODE (https:// #-Link-Snipped-#) adds trace commands to embedded systems so execution paths can be analyzed. Aprobe from OC Systems (my company; #-Link-Snipped-#) allows you to modify the executable in whatever way you might choose.
Typically, white-box testing tools such as these provide a high-level programming interface for writing code that patches the executable and performs a specific function or obtains a particular type of information. Other features might include patches that execute as part of the application, at full machine speeds, which makes them noninvasive and useful for experimentation. Also, the tools might be able to automatically mangle/demangle names so users can use source code names even though the executable contains mangled object code names.
To illustrate, I'll present examples that use code for Aprobe. Aprobe uses ANSI C (with a few keywords added) as the base language in which the patches are specified, so it is straightforward to read/write code.
The aprobe.h include file contains a number of functions useful for writing patches for testing, debugging, and benchmarking. All of these functions have descriptive names and start with "ap_." These functions are designed to support the testing process. While adding arbitrary source code to an executable is advantageous, much of the power of such tools lies in these support functions.
One thing to note is the shortness of the code sequences. With this type of testing tool, most of the patches are extremely short -- much shorter than the corresponding source code modifications.