15th International Conference on Pattern Recognition

Algorithm Performance Contest

The 15th International Conference on Pattern Recognition is sponsored by the International Association for Pattern Recognition and will be held in Barcelona, Spain, in September 2000. This contest, which is organized by Prof. Robert M. Haralick at the Intelligent Systems Laboratory of the University of Washington as a part of the conference, involves the running and evaluation of computer vision and pattern recognition techniques on different data sets with known groundtruth.

The contest currently includes five areas; binary shape recognition, multivariate data classification, symbol recognition, image flow, and vehicle detection. There is a package available for each area. Each package contains either real images with manual groundtruth or programs to generate data sets of ideal as well as noisy images with known groundtruth. They also contain programs to evaluate the results of an algorithm according to the given groundtruth. These evaluation criteria include the generation of confusion matrices, computation of the misdetection and false alarm rates and other performance measures suitable for the problem. Some of these packages also include code for algorithms that were developed in our laboratory. These algorithms are not claimed to be the best algorithms available in the literature but are included in case you would like to make comparisons.

Detailed information about each package is given in the links below. Each package contains descriptions for the parameters used and also some example parameter files. You can use any set of parameters to generate test images for your algorithm development. Please use the default parameters supplied in each package to generate data for the final experiments. Then, please run your algorithms on these data sets and report your results to us. A submission to the contest should include the names and affiliations of the participants, a one-paragraph summary of the algorithm used, a one-paragraph summary of the experimental results, at least one reference to an already published paper or a publicly accessible technical report, and possibly a link to detailed online information. Each package below includes code and detailed information for performance evaluation. The submission for experimental results should be in the form of the final output (e.g. confusion matrices, misdetection and false alarm rates) obtained by running these codes. The deadline for contest participation is May 10, 2000. The submitted results will be summarized in a paper that will be published in the conference proceedings.

• Image Generation and Shape Recognition Package by the Intelligent Systems Laboratory at the University of Washington.
  This package is intended to provide a test data set with known groundtruth to evaluate binary shape recognition algorithms. It includes code for generation of primitives and shape prototypes as the groundtruth model set, and perturbed images containing translated and scaled prototypes as the test data set. Another program generates a confusion matrix and computes a success score from the groundtruth and recognition results. A sample shape recognition program that uses recursive mathematical morphology to recognize the shapes is also included.
• Multivariate Data Generation Package by the Intelligent Systems Laboratory at the University of Washington.
  This program generates random vectors in different classes in a high dimensional space. The generated data can be used to evaluate the classifiers. An example classifier that was developed at the Multimedia Communications and Visualization Laboratory of the University of Missouri-Columbia is also available. The false alarm rate and the total classification error rate are output as the classification results.
• Symbol Recognition Package by the Intelligent Systems Laboratory at the University of Washington.
  This package is intended to provide a test data set with known groundtruth to evaluate binary symbol recognition algorithms. The symbol library consists of electrical symbols as the model set and noisy versions of randomly translated and scaled symbols as the test data set. Code to generate a confusion matrix from the groundtruth and recognition results is also included.
• Image Flow Package by the Intelligent Systems Laboratory at the University of Washington.
  This package includes synthetic and real image sequences and their optic flow groundtruth generation for performance evaluation in terms of false alarm rate, misdetection rate and average error vector magnitude. It also includes code and documentation for the image flow estimation techniques developed at our lab.
• Vehicle Detector Evaluation Package by the Intelligent Systems Laboratory at the University of Washington.
  This package provides a moderately large sized data set of ground-truthed real imagery to be used by researchers in developing and evaluating the performance of vehicle detection algorithms in aerial images. The package contains both the image data and the ground-truth information about the vehicles in the images. Code and documentation for accessing the ground-truth information, a vehicle detection algorithm, the performance evaluation software which compares the algorithm output against the ground-truth to give the performance measures are also provided.

If you want to submit the results of your algorithms on the data supplied above or if you want to make your data with groundtruth available in this contest, please send a mail including a brief description and a link to your package to aksoy@isl.ee.washington.edu.

Update 4 (May 18, 2000): Final paper is available.
Update 3 (April 23, 2000): Submission details are updated.
Update 2 (May 14, 1999): An example classifier is added to the Multivariate Data Generation Package.
Update 1 (April 29, 1999): The Vehicle Detector Evaluation Package is included.

Copyright 2000, Intelligent Systems Laboratory,
University of Washington.
Last updated on May 18, 2000 by Selim Aksoy.
Please send your comments to aksoy@isl.ee.washington.edu.

[ ICPR'2000 Contest | ICPR 2000 Home Page | Official IAPR Home Page | IAPR Home Page at UW ISL ]

This page has been visited times.