Download Paper - STS

5 Evaluation
warnings we can remove for each program with all the options enabled for the forward
analysis can be seen in Figure 2 in the appendix.
The interesting part is that the -sub option only removes additional warnings in
combination with -path and -sat. The reason for that is that we have to insert a few
predicates into our flow data using the guards. After we transfer these predicates to the
newly assigned variable, we still have to apply the linear constraint solver to prove that
the fault cannot occur in a run-time error free execution. To do so, we need all three
options enabled. Further, we can again reduce the number of programs where we cannot
remove any warnings from 71 to 26.
Table 5.4 shows the result of the benchmark with the backward analysis. Unfortunately, for the second test suite the backwards analysis worsens the result even more.
In four cases, we remove 135 warnings less, which also includes the case without any
options. However, in this test suite, we have a few programs that contribute a lot to this
reduction. Listing 1, from the appendix, shows a piece of code annotated by the rte
plug-in. In this example, we take one warning and push it into 10 branches. Thereby, we
insert 10 warnings from 1 original warning. Pushing one warning into so many branches
has a big impact on the result of the backward analysis.
Original Warnings
Warnings Added
Warnings Removed
-backwards
2533
729
-path
2817
1013
7211
1804
-path -sub
2817
1013
-path -sat
3631
1827
Options
Removed - Added
10.11%
2533
14.05%
2735
14.05%
2735
25.34%
3667
Relatively Reduced
-sub
729
10.11%
-sat
931
12.91%
-sat -sub
931
12.91%
-path -sat -sub
1863
25.84%
Table 5.4: Benchmark of the IARPA STONESOUP Phase 1 - Memory Corruption
for C Version 1.0 Test Suite from NIST with the Backward Analysis
Figure 5.4 shows how many warnings we could remove for each program. Using the
backward analysis, we end up with more warnings than we started in a few cases. These
are often programs where we could not remove more than one or two warnings using
only the forward analysis. If we then push a warning into many branches, we end up
with more warnings than we started with. The program to which the piece of code from
Listing 1 belongs is one of the −42% bars.
The backward analysis performs the worst with the -sat options and the combination
of -sat and -sub options. We cannot benefit from these options since we push many
warnings not far up from the origin. Having two warnings like first the predicate x < 4
and then x < 5, the -sat option removes the second warning. The backward analysis
pushes both of the warnings up to the last assignment of x, which is not necessarily
in a branch. Thereby, we insert one warning to remove one warning. However, now
66