VCX testing
The mission of the our test lab has always been to provide our customers with transparent and objective results when evaluating their camera systems. Similarly, the VCX organization aims to provide objective image quality testing for mobile phone cameras and webcam systems so that consumers can make a more informed decision when purchasing a smartphone or webcam based on a final "score."
This score is derived from subjective studies done by the VCX-Forum during the development of both VCX profiles. The objective scoring system was created using these studies to match the final score with the overall user perception. As a result, the correlation between the VCX score and the user experience is established independent of the tester's influence.
To ensure full transparency, it is crucial for the test labs performing the camera tests and evaluations to follow the same guidelines. The VCX-Forum provides a white paper (request to download from the VCX website) that outlines the test procedures.
We are a trusted VCX testing lab for VCX-PhoneCam testing (we are currently in the process of becoming a certified test lab for VCX-WebCam testing). Using our test equipment, we have set up our lab per the specifications outlined by the VCX white paper.
Additionally, as a trusted VCX-PhoneCam lab, we play an essential role in developing future standard versions. We regularly run validation tests for a selected pool of test devices to assist the VCX-Forum in finalizing future test protocols.
For more information on the VCX-Forum and the mobile phone/webcam standards, please visit their website at https://vcx-forum.org/.
Each smartphone submitted for testing undergoes an independent evaluation and receives a numeric score. The final score is the weighted sum of the performance of each image quality factor.
When performing a VCX smartphone test, we evaluate the quality of the selfie and primary camera and combine their results for the final score. The primary camera is assessed for its image and video quality and performance, while the selfie camera is tested for its image quality.
Mobile camera testing scene
The primary camera's image quality, both the image and video quality, is tested under three conditions. These conditions are as follows:
- Bright-light – measured with a brightness of 2000 lux (D55) representing daylight.
- Mid-light – measured at 250 lux (neutral LED), representing standard indoor lighting without direct sunlight.
- Low-light – measured at 10 lux (warm LED). Low light is the most challenging situation for a camera. As a result, VCX also tests low-light performance by capturing images with decreasing illumination to gauge the lowest light level a camera can still perform adequately.
The primary camera's image quality is additionally tested under the following conditions to investigate further use cases:
- Flash – evaluated in a low-light scene with 10 lux with the flash on.
- Extended Low Light – evaluated in extra low-light scenes with 10 lux, 7.5 lux, 5 lux, 3 lux, 2 lux, and 1 lux without the flash.
- Zoom – evaluated with 4x zoom at each light condition (bright, mid, low). We first use the optical zoom for devices with optical zoom and add digital zoom if needed to achieve a 4x zoom.
For each condition mentioned above, VCX assesses various image quality factors, including resolution, texture loss, sharpening, color, visual noise, dynamic range, shading, and distortion.
Each factor is measured using the TE42-LL multipurpose test chart, except for the dynamic range. We illuminate the chart with LED-powered iQ-Flatlights. Using iQ-LED technology allows us to generate custom spectra and thus more accurately create the various light conditions required by VCX. For dynamic range measurements of the primary camera, we use the transparent grayscale step chart TE269 in combination with LE7, a uniform lightbox that also uses iQ-LED technology.
For the performance testing, VCX aims to simulate real-life cases that can be considered challenging for the camera system. The primary camera performance test consists of the following sub-tests:
- Stabilization – evaluates the image stabilization with a simulated handshake. It is tested using the STEVE (Stabilization Evaluation Equipment).
- Timing – evaluates the auto-focus performance and the frame rate. It is tested using the TE42-LL Timing chart with built-in LED-Panels and the iQ-Near Focus to trigger the system's auto-focus.
The results of all the individual tests are combined using the scoring system and the weightings provided by VCX to generate the cameras' final score.
VCX webcam testing uses the same principles as VCX mobile phone tests. These tests emphasize face-present and face-moving scenes, auto-exposure, white balance, noise, and spatial frequency performance. The final VCX score provides the level of camera imaging performance. A high VCX score means a high level of image quality. This test is designed for both standalone and built-in webcams.
Webcam testing scene
When performing a VCX webcam test, we evaluate the camera's performance in six image quality focus areas. The areas are as follows:
- Noise – evaluating the visual noise in luminance and color channels according to ISO 15739.
- Contrast Response, Dynamic Range, Exposure – evaluating the system's performance in preserving high-dynamic scenes, brightness, and exposure accuracy based on a target condition.
- Spatial Frequency Response – testing resolution, texture loss, and sharpening performance of the system.
- Color Reproduction – analyzing how accurate the color reproduction of the system is, with emphasis on skin tone reproduction.
- Image Uniformity – testing the luminance and color shading effects.
- Frame Rate – testing whether the frame capture frequency is fast enough.
We test the camera under seven different setups to measure the KPIs listed above. These setups can be listed as:
- Auto-Exposure and Color – measured with the TE295 chart and the mannequin heads (Alexis and Richard) across a broad range of illuminants generated with the iQ-Flatlights.
- Auto-Exposure and Auto-White Balance Sequences – measured with the TE295 chart and the mannequin heads (Alexis and Richard). Predefined light sequences are generated with iQ-Flatlights to trigger different exposure and white balance responses.
- Head Turn – measured with the TE295 chart and the moving mannequin heads (Alexis and Richard) to evaluate the effect of face presence on color reproduction and exposure accuracy. The iQ-Headturner solution is used to generate and control the movement of the heads.
- Challenge – measured with the TE295 chart, the mannequin heads (Alexis and Richard), and a TV in the background to realize HDR scenes. A predefined sequence of slides consisting of unicolor and different textures is played in the background to challenge the auto-exposure and auto-white balance of the system.
- Timing – measured using an LED-Panel placed under the TE295 chart.
- OECF – measured with the transparent TE264X-V2-D280 chart and LE7 as the light source.
- Shading – measured with TE255-D280 diffuser chart and a LE7 as the light source.
Analysis of VCX-PhoneCam solely relies on iQ-Analyzer V6, and the evaluatin of VCX-WebCam is done with both iQ-Analyzer V6 and a Matlab tool implemented by Intel. We are in the process of getting the iQ-Analyzer-X ready for the webcam analysis.
The next version of the VCX-PhoneCam standard is currently under development, and the iQ-Lab plays a crucial role. This version will contain a more extensive video performance test, ultra-wide camera testing, and zoom performance testing. In addition, the score metric will be adjusted using data collected from subjective study surveys from over 1000 participants. We will provide more information and a potential release date as it becomes available.