Continuous Video Quality Assessment Experiments
“Quality is in the Eye of the Beholder”

GitHub Repository

ContiVQAExp is a software tool for designing, conducting and analyzing subjective video quality assessment experiments. The special feature of this tool is the possibility of a continuous collection of a subject’s assessment data for generic video quality characteristics during video playback. In this way, we are able to perform analyzes that provide detailed insights into the relationships between the implementation of video quality characteristics and their effects on a viewer’s quality assessment.

Overview of ContiVQAExp

The screenshots show the central views of ContiVQAExp for designing, conducting, and analyzing subjective video quality assessment experiments. Below, each screenshot is shown with explanations of the associated functions.

Manage Experiments

ContiVQAExp – Manage Experiments

ContiVQAExp supports the management of experiments and their associated subjects to facilitate the experimenter’s handling of the respective data sets. In addition, all collected data (demographics, continuous video quality assessments, results of the predefined assessment form on video quality, and internal analysis results) can be exported all as CSV files for subsequent analysis in other software tools.

Design an Experiment

ContiVQAExp – Design an Experiment

ContiVQAExp supports the design of video quality assessment experiments. An experiment includes a name, the video to be examined, a customizable set of demographics, a predefined assessment form for video quality, and a customizable input method for the continuous data collection during video playback. At the moment, the ContiVQAExp supports a slider as well as radio buttons as an input method. These input methods are completely customizable concerning the labels, the scale, and the number of radio buttons (2 – 10).

Collect Demographics

ContiVQAExp – Collect Demographics

ContiVQA supports the collection of demographics at the beginning of an experiment session with a subject. These demographics can be defined as part of the experiment design and can be retrieved and exported in ContiVQA at any time.

Conduct Continuous Video Quality Assessment

ContiVQAExp – Conduct Continuous Video Quality Assessment

In ContiVQAExp, a subject can continuously assess his or her current perception of a quality characteristic under investigation during video playback by the defined input method, in this case, a slider. In case of problems, such as distractions, ContiVQAExp supports the repetition of the continuous assessment either completely or from a certain point in the video.

Enter Predefined Video Quality Assessment Form

ContiVQAExp – Enter Predefined Video Quality Assessment Form

ContiVQAExp offers experimenters the possibility to use a predefined vision video quality assessment form after the continuous video quality assessment. Based on their proposed vision video quality model, Karras et al. developed this assessment form to collect the subject’s assessment of the overall quality and each quality characteristic of the quality model for vision videos.

Analyze Assessment Data of Single Subject

ContiVQAExp – Analyze Assessment Data of Single Subject

ContiVQAExp provides experimenters with the ability to examine the collected data immediately after the continuous video quality assessment of an experiment session with a subject. As part of this analysis, experimenters can navigate to specific parts of the video by interacting with the graph. Experimenters can take snapshots of the particular video parts and supplement them with their own comments or comments from the subject to document the findings and insights.

Analyze all Assessment Data

ContiVQAExp – Analyze all Assessment Data

ContiVQAExp also provides experimenters with the ability to examine the collected data of all subjects. As part of this analysis, experimenters can navigate to specific parts of the video by interacting with the graph. Experimenters can take snapshots of the particular video parts and supplement them with their comments to document the findings and insights. In addition, ContiVQAExp provides different aggregated analyzes using the mean, median, and standard deviation.