Massive Benchmarking of Vision Algorithms Using Synthetic Video Variants

Keywords: video analysis, software testing, variability modeling, big data, image analysis, statistics

Contact: Mathieu Acher ( or Benoit Baudry (


Numerous vision algorithms for video analysis have been and will be developed, each one focusing on a specific task (e.g., segmentation, registration, compression, object recognition, tracking).
Even for one category of algorithm, many alternatives exist; a one-size-fits-all solution capable of handling the diversity of scenarios and signal qualities is hardly conceivable. An algorithm may excel in tracking a vehicle in a desert but completely fails when the luminance is low or some distractors are present in the video scene.
Another algorithm may be less performant in a normal situation, but more robust when luminance is changed and distractors are added.
The diversity of situations poses a challenge for both developers and users of video algorithms. For consumers (users), the practical problem is to determine what algorithms are likely to fail or excel in certain conditions before the actual deployment in realistic settings. For providers (developers), how to have confidence and produce evidence that the algorithms are capable of handling certain situations?

A fundamental challenge, which we aim at addressing in this work, consists in systematically testing these algorithms in different situations.
We have developed ViViD [1, 2], a tool for the synthesis of video variants with a wide range of characteristics. We can vary luminance, add distractors or objects, and so on. ViViD has been developed in the context of an industrial project involving consumers and providers of video processing algorithms.

We are now at the point of being able to synthesize thousands of videos and teras of data.

The goal of the PhD thesis is to exploit synthetic video variants with a wide range of characteristics to test/benchmark the algorithms. Specifically we want to perform large-scale experiments on thousands of videos and dozens of algorithms (e.g., tracking algorithms). The PhD shall address the following research questions: Can we determine and even predict some qualities of vision algorithms (e.g., performance, reliability) with synthetic video variants and dynamic code analysis? Can we compare competing algorithms? Can the video variants reveal bugs in vision algorithms?
The work will be conducted in close collaboration with InPixal, a company that is developing vision algorithms and is part of ViViD. The company is already using the generator and the methodology, but numerous improvements are expected to mature the systematic exploitation of the results.

Contact: Mathieu Acher ( or Benoit Baudry (


  • José A. Galindo, Mauricio Alferez, Mathieu Acher, Benoit Baudry, and David Benavides.
    A Variability-based Testing Approach for Synthesizing Video Sequences (2014).
    In International Symposium on Software Testing and Analysis (ISSTA’14)
  • Mathieu Acher, Mauricio Alferez, José A. Galindo, Pierre Romenteau, and Benoit Baudry.
    ViViD: A Variability-Based Tool for Synthesizing Video Sequences (2014).
    In 18th International Software Product Line Conference (SPLC’14)

Working Environment

The candidate will work at INRIA in the DIVERSE team (workplace: Université Rennes 1, Campus de Beaulieu, 35000 Rennes, France).
The candidate will also work at InPixal (very close to the Campus de Beaulieu).
The contract is for 36 months.
The monthly net salary is around 1800 euros (net); the salary depends on the previous experience of the candidate and subject to negotiation.
DIVERSE’s research is in the area of software engineering, focusing on the management of diversity in the construction of software intensive systems.
The team is actively involved in European, French and industrial projects and is composed of 8 faculty members, 18 PhD students, 5 postdocs and 4 engineers.