FuzzBench: brain13 report

(experiment incomplete/still running...)

Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

No aggregate ranking as the data contains a single benchmark.

Median relative code-coverages on each benchmark

Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
trial_relative_coverage = trial_coverage / experiment_max_coverage

	app_frozen	app_wrs_rp	app_wrs_rf_rp	app_rf_u_wrs
FuzzerMedian	97.96	91.23	89.95	86.08
FuzzerMean	97.96	91.23	89.95	86.08
libxml2-v2.9.2	97.96	91.23	89.95	86.08

Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
Green background = highest relative median coverage.
Blue gradient background = greater than 95% relative median coverage.

libxml2-v2.9.2 summary

Ranking by median reached code coverage

Mean code coverage growth over time

* The error bands show the 95% confidence interval around the mean code coverage.

Sample statistics and statistical significance (code coverage)

Code coverage sample statistics

		count	mean	std	min	25%	median	75%	max
fuzzer	time
app_frozen	86400	5.0	12827.2	486.488129	12052.0	12683.0	13031.0	13068.0	13302.0
app_wrs_rp	86400	5.0	12016.4	261.696962	11600.0	11926.0	12135.0	12172.0	12249.0
app_wrs_rf_rp	86400	5.0	11872.8	332.020632	11488.0	11559.0	11965.0	12144.0	12208.0
app_rf_u_wrs	86400	5.0	11488.2	177.339223	11315.0	11409.0	11451.0	11481.0	11785.0

Vargha-Delaney A12 measure

The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.

Mann-Whitney U test

The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: d620aa9b152ac11314eab2aa64910679398dfbea

To reproduce this experiment run the following commands in your FuzzBench repo:


            # Check out the right commit. 

            git checkout d620aa9b152ac11314eab2aa64910679398dfbea 

            # Download the internal config file. 

            curl https://storage.googleapis.com/brain13/config/experiment.yaml >  /tmp/experiment-config.yaml

            make install-dependencies 

            # Launch the experiment using paramters from the internal config file. 

            PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

(None,)

experiment summary
libxml2-v2.9.2
experiment data