Contrast Signal-to-Noise Ratio (CSNR)

Introduction

Contrast Signal-to-Noise Ratio refers to a camera's ability to separate contrasts by comparing the mean contrast to its standard deviation. In other words, it describes how well the camera reproduces and separates various objects in a scene under varying conditions. The IEEE-P2020 working group* developed this metric to articulate better image quality key performance indicators (KPIs) crucial to automotive and autonomous driving camera systems.

IEEE-P2020 came together when it became clear that an industry standard was required to assess the performance of automotive and autonomous driving systems, especially when consumer safety is at stake. During development, the group realized that new KPIs needed to be established to reflect the dynamic nature of autonomous driving. We can no longer rely on traditional camera tests using a standard light source and test target. We need KPIs to drill down and see how a camera performs in a constantly changing environment, such as where an autonomous vehicle is expected to operate.

These new KPIs focus on how well an automotive system can identify and separate objects within the scene and make necessary adjustments to the vehicle. KPIs include contrast transfer accuracy (CTA), modulated light mitigation probability (MMP), high dynamic range (HDR), and contrast signal-to-noise ratio (CSNR).

The difference between CSNR and CTA

CSNR and CTA are similar in examining the camera system's ability to detect and measure contrast in the scene. For example, the contrast of an object forms to the scene's background. CTA evaluates the accuracy of the system's contrast recording, while CSNR examines patch separation.

CTA is the probability that a measured contrast will be within a defined threshold. The evaluator determines thresholds based on the camera system's exact requirements and confidence intervals (IEEE-P2020 recommends Michaelson as the default).¹ After testing, we can determine where contrast reproduction is substantial and where it falls off compared to the established acceptable threshold.

CSNR, meanwhile, is not constrained by pre-determined values and provides a more general look at contrast reproduction by evaluating patch separation without saturation. High CSNR values indicate good contrast separation, while low values indicate poor patch separation and contrast detection.¹

Both CTA and CSNR are crucial KPIs for understanding the performance of an autonomous camera system.

CSNR vs. Contrast-to-Noise Ratio (CNR)

Contrast-to-noise ratio (CNR) is utilized within the P2020 metric to describe dynamic range testing. It is typically measured using a test target with grayscale patches. These patches have different contrast values with similar signal separations that might not shift with intensity changes.² Therefore, this measurement can challenge camera systems to interpret and separate objects in varying light intensities. For example, an automated driving application must be able to differentiate objects in low-light scenes to detect objects and make necessary adjustments correctly. The classic use case is a pedestrian walking across a dark street. Can the camera differentiate between the human and, e.g., a median, road sign, etc.?

This challenge made it crucial to have KPIs such as CSNR and CNR that describe a more dynamic scenario. CSNR uses the standard deviation of the contrast distribution to describe the noise and to separate contrasts in the scene from the noise.

CSNR and Signal-to-Noise Ratio (SNR)

CSNR and SNR (signal-to-noise ratio) are similar in that a high value indicates a good patch separation compared to noise level, while low values convey the opposite. Where they differ is in how the KPI should be measured. SNR values can typically be derived from traditional test procedures using grayscale test targets and standard light sources. Again, however, these tests do not replicate real-world scenes in which, e.g., autonomous driving systems must operate. These environments are very dynamic and require unique testing methods to establish a real-world setting.

They differ when a reduction in contrast leads to a reduction in noise and does not impact the mean value (or signal). As a result, the SNR gets improved by an effect that is detrimental to automotive and other machine vision applications. CSNR, meanwhile, is not calculated by dividing the mean by noise but instead by the contrast between two defined patches.

How to test for CSNR

During the development of the IEEE-P2020 standard, it became clear that new test methods and equipment would be needed to test the newly established KPIs for automotive camera systems accurately. As many of our engineers are a part of the P2020 working group, we began to develop a dynamic system capable of testing multiple P2020 KPIs in the comfort of a test lab.

Versatile Light System (VLS)

The Versatile Light System (VLS) offers a solution for multiple IEEE-P2020 measurements, including CSNR measurements. The solution consists of Vega light sources that use DC technology to create a high-stability light source. Combining these light sources with specifically designed gray step test targets creates an excellent solution for CSNR measurements and applications with very short exposure times. It can perform spatial and temporal recording measurements as described in the standard.

The VLS is a system of up to six Vega devices connected to an individual movable arm. All Vegas are linked via the arms to a tripod, offering high adaptability. The flexibility of the arms allows each Vega to be manually moved around to quickly generate randomized setups and angles during testing, allowing you to create a dynamic test scene. The VLS has control software enabling you to create test sequences and increase or decrease the brightness for CSNR measurements. In addition, the VLS evaluation software allows you to evaluate your test results.

We utilize the VLS in our iQ-Lab when testing systems for their CSNR, CTA, and MMP performance.

Conclusion

Contrast Signal-to-Noise Ratio (CSNR) refers to a camera's ability to separate contrasts by comparing the mean contrast to its standard deviation. In other words, it describes how well the camera reproduces and separates various objects in a scene under varying conditions. The IEEE-P2020 working group established this KPI to account for the dynamic nature of autonomous driving, and other automotive-grade cameras are expected to operate successfully.

While traditional contrast-to-noise ratios (CNR) and signal-to-noise (SNR) measurements are often adequate to determine noise separation performance, they rely on standard grayscale test charts in a fixed testing setup. They thus cannot account for the full scope of a dynamic scene where camera systems are performing life-and-death adjustments.

As a result, our team has developed the Versatile Light System (VLS), based on the IEEE-P2020 standard, to create a more dynamic test scene in the comfort of a test lab. We use the VLS in our iQ-Lab when testing the P2020 KPIs, including CSNR, contrast transfer accuracy (CTA), and modulated light mitigation probability (MMP).

*Disclaimer: The opinions contained within the IEEE-P2020 working group solely represent the views of this working group and do not necessarily represent the position of either the IEEE or the IEEE Standards Association.