Grading Methodology for Run-Time Errors

A score of 0 is given when the error was not detected.

A score of 1 is given for error messages with the correct error name.

A score of 2 is given for error messages with the correct error name and line number where the error occurred but not the file name where the error occurred.

A score of 3 is given for error messages with the correct error name, line number and the name of the file where the error occurred.

A score of 4 is given for error messages which contain the information for a score of 3 but less information than needed for a score of 5.

A score of 5 is given in all cases when the error message contains all the information needed for the quick fixing of the error.


Results

You can sort the tables below by clicking on the table headers with one click sorting from best to worst and another click to sort from worst to best.

FORTRAN results

OS/Compiler/ Runtime Tool Array Index out of Bound Uninitialized Variables Subprogram Call Errors Pointer Errors Floating Point Errors String Errors Allocation Deallocation Errors Memory Leaks Input Output Errors Fortran95 Specific Errors Array Conformance Errors Average
IBM AIX/IBM XLF 2.64 0.03 0.00 3.00 2.86 2.33 0.89 0.00 1.00 2.88 1.00 1.51
Linux/NAGWare Fortran 95 3.90 3.27 2.50 4.00 0.71 2.50 2.52 0.00 1.78 3.82 1.00 2.36
SunOS/i86pc/Sun Fortran 95 3.00 0.02 0.00 3.03 0.00 3.00 2.67 0.00 2.17 3.00 3.00 1.81
SunOS/i86pc/Sun Fortran 95/bcheck 3.10 0.87 1.00 3.09 0.29 3.00 2.67 1.00 2.17 3.00 3.00 2.11
SunOS/sparc/Sun Fortran 95 3.00 0.02 0.00 3.03 0.00 3.00 2.67 0.00 2.17 2.82 3.00 1.79
SunOS/sparc/Sun Fortran 95/bcheck 3.05 0.98 1.00 3.03 0.29 3.00 2.78 1.00 2.17 2.88 3.00 2.11
Linux/Intel Fortran 7.1 1.86 0.44 1.67 2.00 0.00 2.00 2.00 0.00 1.44 1.94 0.42 1.25
Linux/Intel Fortran 9.0 3.45 0.12 0.17 3.56 0.57 0.00 1.54 0.00 2.06 2.82 0.42 1.34
Cray XT4/ PGI pgf90 3.85 0.15 0.00 4.00 0.00 0.00 0.39 0.00 2.44 3.82 0.42 1.37
Cray X1 UNICOS/mp/Cray Fortran 3.51 0.40 2.00 4.00 0.71 4.00 2.94 0.00 2.89 3.94 3.00 2.49
Cray X2/ Cray Fortran 3.76 0.40 2.00 4.00 0.00 4.00 0.89 0.00 1.00 3.94 3.00 2.07
Cray XT4/PathScale Fortran 1.89 0.00 0.00 2.96 0.00 0.00 0.67 0.00 0.61 0.71 0.00 0.62
Linux/GNU Fortran 2.82 0.01 0.00 3.00 0.00 0.00 1.04 0.00 2.33 2.88 1.00 1.19

C results
OS/Compiler/ Runtime Tool Array Index out of Bound Uninitialized Variables Subprogram Call Errors Pointer Errors Floating Point Errors String Errors Allocation Deallocation Errors Memory Leaks Input Output Errors C99 Specific Errors Average
IBM AIX/IBM XLC 0.04 0.00 0.00 0.04 0.00 0.18 0.64 0.20 0.00 0.10 0.12
SunOS/i86pc/Sun C 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
SunOS/i86pc/Sun C/bcheck 0.02 0.00 0.00 0.06 0.00 0.00 0.16 1.13 0.00 0.00 0.14
SunOS/sparc/Sun C 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
SunOS/sparc/Sun C/bcheck 1.19 0.61 0.95 1.38 0.00 0.75 2.68 1.93 0.30 1.00 1.08
Linux/Intel C++ 7.1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Linux/Intel C++ 9.0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Cray XT4/ PGI pgcc 1.94 0.00 0.00 0.02 0.00 0.05 0.44 0.07 0.20 1.54 0.43
Cray X1 UNICOS/mp/ Cray C++ 2.63 0.22 0.32 0.03 0.67 0.00 0.08 0.27 0.20 0.25 0.47
Cray X2/ Cray C++ 2.64 0.01 0.00 0.02 0.00 0.05 0.60 0.33 0.20 0.00 0.39
Linux/Insure++ 3.53 3.66 0.32 3.16 0.17 3.00 3.80 3.80 3.20 2.85 2.75
Cray XT4/ PathScale C 0.01 0.00 0.00 0.02 0.00 0.05 0.44 0.07 0.20 0.04 0.08
Linux/GNU C 0.01 0.00 0.00 0.02 0.00 0.05 0.44 0.07 0.20 0.04 0.08

C++ results
OS/Compiler/ Runtime Tool Array Index out of Bound Uninitialized Variables Pointer Errors Floating Point Errors String Errors Allocation Deallocation Errors Memory Leaks Input Output Errors Average
IBM AIX/IBM XLC 0.04 0.00 0.10 0.00 0.18 0.38 0.10 0.00 0.10
SunOS/i86pc/Sun C++ 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
SunOS/i86pc/Sun C++/bcheck 0.02 0.00 0.02 0.00 0.00 0.25 1.17 0.00 0.18
SunOS/sparc/Sun C++ 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
SunOS/sparc/Sun C++/bcheck 1.26 1.00 1.56 0.71 1.30 1.92 1.57 1.00 1.29
Linux/Intel C++ 7.1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Linux/Intel C++ 9.0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Cray XT4/ PGI pgCC 1.76 0.00 0.03 0.00 0.05 0.31 0.05 0.07 0.28
UNICOS/mp/Cray C++ 2.91 0.34 0.04 0.65 0.00 0.05 0.14 0.07 0.53
Cray X2/ Cray C++ 2.89 0.00 0.23 0.00 0.05 0.38 0.17 0.07 0.47
Linux/Insure++ 3.44 3.57 2.80 0.17 3.00 3.80 3.80 3.20 2.97
Cray XT4/ PathScale C++ 0.01 0.00 0.03 0.00 0.05 0.28 0.05 0.07 0.06
Linux/GNU C++ 0.01 0.00 0.03 0.00 0.05 0.31 0.05 0.07 0.06

.

.

.

.

.

.

Computer info:
DataStar
IBM p-Series
University of California, San Diego
Compiler:
IBM(R) XL Fortran Enterprise Edition V9.1o
Compiler options used:
-O0 -qsuffix=f=f90 -qcheck -g -qsource -qsigtrap=xl__trce -qflttrap=enable:zerodivide:invalid:overflow
Computer info:
Pentium III processor series
RedHat Linux 9.0
Iowa State University
Compiler:
NAGWare Fortran 95 compiler Release 5.0
Compiler options used:
-C=all -C=undefined -gline
Computer info:
Cray XT4 (Opteron)
Cray 2.1-CLE
Compiler:
PathScale(TM) EKOPath C/C++ compiler, Version 3.2
Compiler options used:
-g -trapuv -Wall -TENV:simd_umask=OFF:simd_omask=OFF:
simd_imask=OFF:simd_zmask=OFF:
simd_dmask=OFF:simd_pmask=OFF
Computer info:
Cray XT4 (Opteron)
Cray 2.1-CLE
Compiler:
PathScale(TM) EKOPath C/C++ compiler, Version 3.2
Compiler options used:
-g -trapuv -Wall -TENV:simd_umask=OFF:simd_omask=OFF:
simd_imask=OFF:simd_zmask=OFF:
simd_dmask=OFF:simd_pmask=OFF
Computer info:
Opteron
Linux 2.6.9-34.ELsmp
Iowa State University
Compiler:
PathScale(TM) EKOPath Fortran 77/90/95 compiler, Version 2.4
Compiler options used:
-C -g -trapuv -Wall -TENV:simd_umask=OFF:simd_omask=OFF:
simd_imask=OFF:simd_zmask=OFF:
simd_dmask=OFF:simd_pmask=OFF
Computer info:
Opteron
Linux 2.6.18-128.1.10.el5 #1 SMP
Iowa State University
Compiler:
GNU Fortran compiler gcc4-gfortran, Version 4.1.2
Compiler options used:
-frange-check -O0 -fbounds-check -g
-ffpe-trap=invalid,zero,overflow
-fdiagnostics-show-location=every-line
Computer info:
Opteron
Linux 2.6.18-128.1.10.el5 #1 SMP
Iowa State University
Compiler:
GNU C compiler gcc, Version 4.1.2
Compiler options used:
-O0 -g -fbounds-check -ftrapv
Computer info:
Opteron
Linux 2.6.18-128.1.10.el5 #1 SMP
Iowa State University
Compiler:
GNU C++ compiler gcc-c++, Version 4.1.2
Compiler options used:
-O0 -g -fbounds-check -ftrapv
.
Computer info:
DataStar
IBM p-Series
University of California, San Diego
San Diego Supercomputer Center
Compiler:
IBM(R) XL C/C++ Enterprise Edition V7.0
Compiler options used:
-O0 -qheapdebug -qinfo -qcheck -g -qsource -qflttrap=enable:zerodivide:invalid:overflow
Computer info:
DataStar
IBM p-Series
University of California, San Diego
San Diego Supercomputer Center
Compiler:
IBM(R) XL C/C++ Enterprise Edition V7.0
Compiler options used:
-O0 -qheapdebug -qinfo -qrtti -qcheck -g -qsource -qflttrap=enable:zerodivide:invalid:overflow
Computer info:
Xeon
Linux 2.4.18-14smp
Iowa State University
Compiler:
Intel(R) C++ Compiler for 32-bit applications, Version 7.1 Build 20030307Z
Compiler options used:
-O0 -C -g
Computer info:
Xeon
Linux 2.4.18-14smp
Iowa State University
Compiler:
Intel(R) C++ Compiler for 32-bit applications, Version 7.1 Build 20030307Z
Compiler options used:
-O0 -C -g
Computer info:
Xeon
Linux 2.4.18-14smp
Iowa State University
Compiler:
Intel(R) Fortran Compiler for 32-bit applications, Version 7.1 Build 20030307Z
Compiler options used:
-O0 -C -g
Computer info:
Xeon
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-g
Tool:
Insure++ 7.0
Computer info:
Xeon
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-O0 -C -g -traceback
Computer info:
Xeon
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-g
Tool:
Insure++ 7.0
Computer info:
Xeon
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) C++ Compiler Version 9.0
Compiler options used:
-O0 -C -g -traceback
Computer info:
Xeon
Linux 2.6.9 ELsmp
Iowa State University
Compiler:
Intel(R) Fortran Compiler Version 9.0
Compiler options used:
-O0 -C -g -traceback
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun C 5.7 2005/01/07
Compiler options used:
-g -xs -ftrap=%all -fnonstd -xcheck=%all
Tool:
bcheck
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun C++ 5.7 2005/01/07
Compiler options used:
-g -xs -ftrap=%all -fnonstd -xcheck=%all
Tool:
bcheck
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun Fortran 95 8.1 2005/01/07
Compiler options used:
-C -g -xs -ftrap=%all -fnonstd -xcheck=%all
Tool:
bcheck
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun C 5.7 2005/01/07
Compiler options used:
-g -xs -ftrap=%all -fnonstd -xcheck=%all
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun C++ 5.7 2005/01/07
Compiler options used:
-g -xs -ftrap=%all -fnonstd -xcheck=%all
Computer info:
Ultra SparcIII
SunOS 5.9 sparc
Iowa State University
Compiler:
Sun Fortran 95 8.1 2005/01/07
Compiler options used:
-C -g -xs -ftrap=%all -fnonstd -xcheck=%all
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun C 5.8 2005/10/13
Compiler options used:
-g -xs -ftrap=%all -fnonstd
Tool:
bcheck
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun C 5.8 2005/10/13
Compiler options used:
-g -xs -ftrap=%all -fnonstd
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun C++ 5.8 2005/10/13
Compiler options used:
-g -xs -ftrap=%all -fnonstd
Tool:
bcheck
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun C++ 5.8 2005/10/13
Compiler options used:
-g -xs -ftrap=%all -fnonstd
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun Fortran 95 8.2 2005/10/13
Compiler options used:
-C -g -xs -ftrap=%all -fnonstd -xcheck=%all
Tool:
bcheck
Computer info:
Opteron
SunOS 5.10 i86pc
Iowa State University
Compiler:
Sun Fortran 95 8.2 2005/10/13
Compiler options used:
-C -g -xs -ftrap=%all -fnonstd -xcheck=%all
Computer info:
Cray X1
UNICOS/mp 3.0.55
Compiler:
Cray C++ : Version 5.5.0.5
Compiler options used:
-O0 -C -g -h bounds -h indef -h conform
Computer info:
Cray X2
Cray 2.1-CLE
Compiler:
Cray C++ : Version 6.0.0.7
Compiler options used:
-O0 -C -g -h bounds -h indef -h conform
Computer info:
Cray X1
UNICOS/mp 3.0.55
Compiler:
Cray C++ : Version 5.5.0.5
Compiler options used:
-O0 -C -g -h bounds -h indef -h new_for_init -h conform
Computer info:
Cray X2
Cray 2.1-CLE
Compiler:
Cray C++ : Version 6.0.0.7
Compiler options used:
-O0 -C -g -h bounds -h indef -h new_for_init -h conform
Computer info:
Cray X1
UNICOS/mp 3.0.55
Compiler:
Cray Fortran : Version 5.5.0.5
Compiler options used:
-O0 -e D
Computer info:
Cray X2
Cray 2.1-CLE
Compiler:
Cray Fortran : Version 6.0.0.7
Compiler options used:
-O0 -e D
Computer info:
Cray XT4 (Opteron)
Cray 2.1-CLE
Compiler:
pgcc 8.0-4 64-bit target on x86-64 Linux
Compiler options used:
-O0 -C -g -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Msignextend
Computer info:
Cray XT4 (Opteron)
Cray 2.1-CLE
Compiler:
pgCC 8.0-4 64-bit target on x86-64 Linux
Compiler options used:
-O0 -C -g -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Msignextend
Computer info:
Cray XT4 (Opteron)
Cray 2.1-CLE
Compiler:
pgf90 8.0-4 64-bit target on x86-64 Linux
Compiler options used:
-O0 -C -g -Ktrap=fp -Mbounds -Mchkfpstk -Mchkstk -Mchkptr