Overview
Intel® Edge Software Device Qualification (Intel® ESDQ) for Intel® Edge AI Box allows customers to run an Intel-provided test suite at the target system, aiming to enable partners to qualify on their platform.
The following information is specific to the Intel® ESDQ for Intel® Edge AI Box package. For documentation on the Intel® ESDQ CLI binary, refer to Intel® Edge Software Device Qualification (Intel® ESDQ) CLI Overview.
Select Configure & Download to download Intel® ESDQ for Intel® Edge AI Box from the Intel® Edge AI Box page and refer to Get Started for the installation steps.
Target System Requirements
- 11th, 12th, or 13th generation Embedded Intel® Core™ processors
- 12th generation Intel® Core™ Desktop processors with Intel® Arc™ A380 Graphics
- Intel Atom® Processor X Series, (formerly Alder Lake-N)
- Intel® Processor N-series (formerly Alder Lake-N)
- Operating System:
- Ubuntu* 22.04.2 or higher
- At least 80 GB of disk space
- At least 8 GB of memory
- Direct Internet access
Ensure you have sudo access to the system and a stable Internet connection.
How It Works
The AI Box Test Module in the Intel® Edge AI Box interacts with the Intel® ESDQ CLI through a common test module interface (TMI) layer, which is part of the Intel® ESDQ binary.
The selected components from the download options will be validated via the automated test suite.
Test results will be stored in the output folder. Intel® ESDQ generates a complete test report in HTML format and detailed logs packaged as one ZIP file, which you can email to the Intel® ESDQ support team at edge.software.device.qualification@intel.com.
AI Box Test Module
The AI Box test module is the validation framework for Intel® Edge AI Box. This module validates the installation of software packages and measures the performance of the platform using the following benchmarks:
- OpenVINO™ based neural network model benchmarks
- Video analytics pipeline benchmarks
- Video decode performance benchmark
- Memory benchmarks
- GPU AI frequency measurement
OpenVINO Benchmark
The following neural network models are benchmarked using the OpenVINO™ benchmark tool. Both latency and throughput are measured. Benchmark results are included in the ESDQ report.
- resent-50-tf
- ssdlite-mobilenet-v2
- yolo-v3-tiny
- yolo-v3
- yolo-v4
- efficientnet-b0
- yolo-v5 n/s/m
- yolo-v8 n/s/m
Video Analytic Benchmark
The video analytic benchmark is based on a six-stage pipeline that implements a vehicle detection and attribute classification use case. The pipeline consists of decode, object detection, object tracking, object classification, watermark, and encode stages. The models used are downloaded from OpenVINO™ Model Zoo. You can run up to 25 streams of videos simultaneously.
Video Decode and Composite Benchmark
Using the platform's GPU, the video decode and composite benchmark measures the average frame rate to decode multiple 1080p video streams simultaneously and tile them into a multi-view layout. The benchmark covers both H.264 and H.265 encoded video streams. The composited stream is re-encoded with the corresponding codec and saved as output for visual inspection. For integrated GPU, 16 video streams are decoded; 36 video streams are used for discrete GPU.
Memory Benchmark
The memory benchmark measures the sustained memory bandwidth based on STREAM.
GPU AI Frequency Measurement
The GPU inference frequency benchmark was designed to stress the GPU for an extended period. The benchmark records the GPU frequency while it runs an inference workload using the OpenVINO™ Benchmark Tool.
Get Started
The AI Box Test Module and the Intel® ESDQ CLI tool are installed as part of the Intel® Edge AI Box development package. As a result, many of the steps are common with the instructions given in the Get Started section of Intel® Edge AI Box.
NOTE: The screenshots may show a package version number different from the current release. See the Release Notes for information on the current release.
Prerequisite
Perform a clean installation of Ubuntu 22.04.2 or higher on the target system.
Download and Install Intel® ESDQ for Intel® Edge AI Box
-
Select Configure & Download to download the Intel® Edge AI Box package.
Configure & Download - Choose the required configuration based on your test system environment. The following figure shows the recommended selection.
- Transfer the downloaded package to the target Ubuntu* system and unzip:
unzip intel_edge_ai_box.zip
- Go to the intel_edge_ai_box/ directory:
cd intel_edge_ai_box
- Change permission of the executable edgesoftware file:
chmod 755 edgesoftware
- Install the Intel® Edge AI Box package:
./edgesoftware install
- When prompted, enter the Product Key. You can enter the Product Key mentioned in the email from Intel confirming your download (or the Product Key you copied in Step 3).
Note for People’s Republic of China (PRC) Network:
- If you are connecting from the PRC network, the following prompt will appear during the bmra base installation:
- Type Yes, and the installation script will replace certain download sources. This will avoid download failure in the PRC network.
- Then, while installing the AI Box Test Module, the following prompt will appear:
- Type Yes to replace certain GitHub* links.
- When prompted for the BECOME password, enter your Linux* account password.
- When prompted to reboot the machine, press Enter. Ensure to save your work before rebooting.
- After rebooting, resume the installation:
cd intel_edge_aibox
./edgesoftware install
- After the ESDQ is installed, you will be prompted for the password. Enter the password to proceed.
- When the installation is complete, you will see the message “Installation of package complete” and the installation status for each module.
- Reboot the system:
sudo reboot
Run the Application
The installed Intel® Edge AI Box package was configured for the Platform Qualification environment, and the benchmarks were run using this environment.
For the complete Intel® ESDQ CLI, refer to Intel® ESDQ CLI Overview. To find the available Intel® Edge AI Box tests, run the following command:
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-h"
You have the option to run an individual test or all tests together. The results from each test will be collated in the HTML report.
Run Full ESDQ
Run the following commands to execute all the Intel® Edge AI Box tests and generate the full report.
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r all"
Run OpenVINO Benchmark
The OpenVINO benchmark measures the performance of commonly used neural network models on the platform. The following models are supported:
- resent-50
- ssdlite-mobilenet-v2
- yolo-v3-tiny
- yolo-v3
- yolo-v4
- efficientnet-b0
- yolo-v5 n/s/m
- yolo-v8 n/s/m
The following OVRunner runner commands benchmark all the models using dGPU for 180 seconds.
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r OVRunner -d dGPU -t 180"
OVRunner parameters:
- -d {CPU, GPU, iGPU, dGPU} specifies the device type.
- -t <seconds> specifies the benchmark duration in seconds.
- -m {resnet-50-tf, ssdlite_mobilenet_v2, yolo-v3-tiny-tf, yolo-v4-tf, efficientnet-b0, yolo-v3-tf, yolo-v5n, yolo-v5s, yolo-v5m, yolo-v8n, yolo-v8s, yolo-v8m} specifies model. Do not include this parameter to run all models.
The following is the example report:
Run Video Analytic Benchmark
The video analytic benchmark is based on a six-stage pipeline that implements a vehicle detection and attribute classification use case using Intel® DLStreamer.
Run the following DLRunner command:
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r DLRunner -d GPU [-s 4/8/16/25]"
DLRunner Parameters:
-
-d {CPU, GPU, iGPU, dGPU} specifies the device type.
- -s {4,8,16,25} specifies number streams. Do not specify this parameter to run all.
The following is the example report:
Run Memory Benchmark
The memory benchmark measures the sustained memory bandwidth based on STREAM. To measure memory bandwidth for media processing, invoke the MemBenchmark runner:
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r MemBenchmark"
After the MemBenchmark runner completes, the location of the test report is displayed. The following is the example report:
Run GPU AI Frequency Measurement
The GPU inference frequency benchmark was designed to stress the GPU for an extended period. The benchmark records the GPU frequency while it runs an inference workload using the OpenVINO™ Benchmark Tool.
The following FreqRunner runner command measures GPU frequency for 10 seconds.
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r FreqRunner -d GPU -t 10"
FreqRunner parameters:
- -d {CPU, GPU, iGPU, dGPU} specifies the device type.
- -t <seconds> specifies the benchmark duration in seconds.
The following is an example plot:
Run Video Decode and Composite Benchmark
The video decode and composite benchmark measure the average frame rate to decode multiple 1080p video streams simultaneously and tile them into a multi-view layout using the platform’s integrated GPU.
The following VMRunner runner command invokes this benchmark for H.264 decode.
cd intel_edge_aibox/Intel_Edge_AI_Box_3.1
esdq --verbose module run aibox --arg "-r VWRunner -d GPU -c h264"
VMRunner Parameters:
- -d {GPU, iGPU, dGPU} specifies device type.
- -c {h264,h265} specifies codec type. Do not specify this parameter to run the benchmark for h264 and h265.
The following is the example report:
Sample Full ESDQ Report
Known Issues
[AIBOX-254]: VDBoxes are not 100% utilized in video decode and composite benchmarks.
[AIBOX-283]: The xpumanager outputs less data in kernel 6.2+ than in 5.19+.
[AIBOX-293]: Report.zip is empty when you generate the report for the first time to generate the report.
[AIBOX-296]: Failed to install Intel® Edge AI Box version 3.1 over an earlier Intel® Video AI Box version 3.0.
Troubleshooting
As the system is being automatically updated in the background, you may encounter the following errors:
To resolve this issue, reboot and manually update the system. Then, rerun the installation:
sudo reboot
sudo apt update
./edgesoftware install
Refer to the Troubleshooting section of the Intel® Edge AI Box package.
If you’re unable to resolve your issues, contact the Support Forum.
Release Notes
Current Version: 3.1
New in This Release
- Changed proxy URL for PRC network.
- Fixed issues related to dependencies version updates.
Version: 3.0
New in This Release
- Updated ESDQ infrastructure to version v11.0.0
- Built Benchmarks and executed them in Video Analytics Base Library containers.
- Added support for discrete GPU.
- Added memory benchmark.
Version: 2.6
New in This Release
- Updated the test module to support Intel® Distribution of OpenVINO™ toolkit 2022.2 Samples.
- Added tests for Intel® Distribution of OpenVINO™ toolkit in a container.
- Added video decode 16 channel 1080p 4x4 video wall tests.
- Included GPU Inference Frequency Plot test.
Version: 2.5
New in This Release
- Updated the test module to support Intel® Distribution of OpenVINO™ toolkit 2022.1 Samples.
- Removed SVET, Intel® Media SDK and Yolo tests.
- Included the functionality test of Pipeline and Inference Benchmark requirements.
Version: 2.0
New in This Release
- Integrated Intel® Distribution of OpenVINO™ toolkit 2021.4.2.
- Included the functionality test of Azure IoT PnP Bridge, Intel® Distribution of OpenVINO™ toolkit TensorFlow Bridge, and Amazon Web Services Greengrass.
Known Issues
- Smart Video and AI Workload Reference Implementation test requires the user to feed in the root password. However, when Intel® ESDQ is run on a graphical interface, the system switches to console mode during SVET execution, and the control to Intel® ESDQ terminal that displays the password prompt is lost. This results in a hang situation. Log in to the system using the SSH terminal and execute the Intel® ESDQ tests to work around this issue.
Version: 1.2
New in This Release
- Latency and Throughput information included in the HTML report.
- Included the functionality test of Smart Video and AI Workload Reference Implementation.
Version 1.0
New in This Release
- Initial features for recommended configuration.
Known Issues
- If yolo-v3-tf.xml is not downloaded, then Latency and Throughput metrics will be null in the Intel® ESDQ HTML report page.