Local detector evaluation
Overview:
This page is dedicated to evaluate the performance of local detectors (scale-invariant, affine-invariant) on non-planar scenes as described in the paper "A novel performance evaluation method of local detectors on non-planar scenes" by Fraundorfer and Bischof (PDF).
The evaluation framework will read the detector output produced by the various local detectors available at the webpage of Krystian Mikolajczyk. The evaluation framework comes with the ground truth for two non-planar test scenes. The evaluation tool will compute the repeatability score and two variants of the matching score as described in the above mentioned publication.
Evaluation framework:
The evaluation tool "eval_run.exe" is a simple command line tool. The input images and groundtruth are specified by parameters to the evaluation tool. The syntax is as follows:
Syntax: "eval_run detectorfile1 detectorfile2 densematchingpoints trifocaltensor image1 image2 transferredpoints [outimage1] [outimage2]" [overlap]
Parameters:
-detectorfile1: File with detected interest regions (from Mikolajczyk binaries)
-detectorfile2: File with detected interest regions (from Mikolajczyk binaries)
-densematchingpoints: File with dense point correspondences between image1 and intermediate image (Groundtruth)
-trifocaltensor: File with trifocal tensor between image1 and image2 (Groundtruth)
-image1: Input image (source to detectorfile1)
-image2: Input image (source to detectorfile2)
-transferredpoints: File with point correspondences projected to image 2 (Groundtruth)
-outimage1: [optional] Output filename for image1 showing interest regions (corresponding and not corresponding)
-outimage2: [optional] Output filename for image2 showing interest regions (corresponding and not corresponding)
-overlap: [optional] Value for maximal allowed overlap error (0..1), default: 0.5
Example: "eval_run gruppe1.01.mser gruppe1.03.mser homPts12_dense.txt T12_03.txt gruppe1.01.bmp gruppe1.03.bmp points03_dense.txt"
The program will produce a simple text output containing the repeatability score and matching score and some additional information looking as follows:
Overlap Threshold: 0.5
group01.haraff <-> group03.haraff ::: Repeat.: 0.509106, Dev.:(0.509106,0.509106), Abs. Repeat.: 615, Match.: 0.365894 (0.718699), Abs. Match.: 442, notC: 0, AvgDens: 1.094463
The evaluation of the whole scene can easily be scripted using a batch file. The evaluation framework contains a demo batch file called "run_evaluation.bat". After downloading and unzipping just call "run_evaluation.bat" from the command line.
Download evaluation framework:
The evaluation framework contains all the images, geometric groundtruth, executables and an Excel spreadsheet with graph templates as used in the paper. The executable will run under Win2K/XP.
Dependencies: To run the evaluation tool Matlab 6.5 has to be installed. The tool will need Matlab DLL's from the path "matlabroot\bin\win32". If you don't have Matlab 6.5 installed you might run "mglinstaller.exe" contained in the ZIP-file which installs the Matalb Run-Time-Library.
Download image sequences:
Here are the image sequences only.
