Benchmarking & Performance Analysis

Benchmarking & Performance Analysis


 * Purpose of this document is to put together details for platform benchmarking and performance to assist engineers who want to measure performance and analyze it further to improve it and to compare it with other platforms
 * This document covers 4 aspects of overall performance CPU, Memory access, I/O and Graphics
 * Since Quadrant is most commonly used benchmarking app It is used here as an use case
 * Since this has to be prepared in short span of time the details are abstracted in this draft
 * The details here are based on ES 2.1 Omap4 EMU blaze board with 27.10.2 release

Benchmarking Apps and Tests


 * There are multiple apps and tests available on the net and each test try to use specific method to measure the performance in a specific way
 * Since there are multiple things which impacts the performance (e.g. pm, accelerators,..) it’s heard to conclude the performance based on specific app or test
 * In general it’s recommended and it’s better to use multiple benchmarking apps and tests to measure the performance which will give us fair picture
 * Since Quadrant apps is commonly used in android world this document uses this app as an use case

CPU Benchmarking using Quadrant


 * Before running test case make sure governor set to performance mode else you will see lower numbers as PM might scale down the frequency while running the tests
 * Make sure you run this test multiple times (your numbers may be lower some times, at start in particular)
 * If you are using the latest release check for stability of release, it might possible that you get lower numbers than previous release because of stability reasons
 * Make sure your CPU is clocked at highest rate possible

Memory Benchmarking using Quadrant


 * Before running test case make sure governor set to performance mode else you will see lower numbers as PM will scale down the frequency while running the tests
 * Make sure you run this test multiple times (your numbers may be lower some times, at start in particular)
 * If you are using the latest release check for stability of release, it might possible that you get lower numbers than previous release
 * Make sure your CPU, Bus and memory is clocked at highest rate possible
 * Make sure you have all the memory access optimizations in place
 * If you are comparing your numbers with other platforms then it good to understand CPU, Bus and memory clock rate for other platform for fair comparison

I/O Benchmarking using Quadrant


 * Before running test case make sure governor set to performance mode else you will see lower numbers as PM will scale down the frequency while running the test
 * Make sure you run this test multiple times (your numbers may be lower some times, at start in particular)
 * If you are using the latest release check for stability of release, it might possible that you might get lower numbers than previous release because of stability reasons
 * Make sure your CPU, Bus and Memory is clocked at highest rate possible
 * If you are comparing bench mark numbers with other platforms then check for other paltform CPU, Bus and Memory clock rate for fair comparison
 * I/O benchmark numbers have major dependency on file system
 * If you are comparing I/O numbers with other platforms then understand what file system other platforms are using
 * Some of the platform are not using standard ext file system but using their own file systems
 * Some of the platform are using customized ext file system, some times new ext file system that doesn’t come with standard android
 * There are also many customizations available in the market to improve the file system I/O speed
 * Some of them comes with stability and data loss risk (e.g. data loss when battery failed or discharged or removed)

2D Benchmarking using Quadrant 


 * Before running test case make sure governor is set to performance mode else you will see lower numbers as PM will scale down the frequency while running the tests
 * Make sure you run this test multiple times (your numbers may be lower some times, at start in particular)
 * If you are using the latest release check for stability of release, it might possible that you might get lower numbers than previous release because of stability reasons
 * Make sure your CPU, Bus, Memory and GPU is clocked at highest rate possible
 * If you are comparing bench mark numbers with other platforms then check for other platform CPU, Bus, Memory and GPU clock rate for fair comparison
 * There are multiple tests within 2D tests, each test try to measure different capability and runs differently (on ARM and accelerator) so one needs to compare test results separately for each test
 * If you find fps is 60 and not going beyond, it might possible that Vsync is limiting fps. You might want to run the test with Vsync unlocked (this is just for benchmarking purpose, in real world you are always going to be Vsync locked)
 * 2D benchmarking numbers have dependency on multiple things
 * Make sure neon is enabled in your system and specific SKIA functions are accelerated using neon
 * If you are comparing 2D numbers with other platforms then understand what acceleration other platforms are supporting
 * Some of the platform have some more additional SKIA functions implemented using neon acceleration
 * Some of the platforms are using using additional accelerators like Arm Mali and some of the platforms are supporting additional 2D acceleration in their GPU or some times a separate 2D accelerator itself
 * And also keep in mind that there is always tread of bwtn performance and power

3D Benchmarking using Quadrant


 * Before running test case make sure governor is set to performance mode else you will see lower numbers as PM will scale down the frequency while running the tests
 * Make sure you run this test multiple times (your numbers may be lower some times, at start in particular)
 * If you are using the latest release then check for stability of release, it might possible that you might get lower numbers than previous release because of stability reasons
 * Make sure your CPU, Bus, Memory and GPU is clocked at highest rate possible
 * If you are comparing bench mark numbers with other platforms then check for other platform CPU, Bus, Memory and GPU clock rate for fair comparison
 * There are multiple tests within 3D tests, each test try to measure different capability and runs differently (some of them are on ARM and others on GPU) so you need to compare test results separately for each test
 * If you find fps is 60 and not going beyond, it might possible that Vsync is limiting fps. You might want to run the test with Vsync unlocked
 * 3D benchmarking numbers have dependency on multiple things, since part of test runs on ARM it’s speed and configuration matters (clk, neon and other accelerators also matters)
 * If you are comparing 3D numbers with other platform then it’s good to understand what accelerator other platforms are supporting and what clk it runs on
 * Last but not least keep in mind that there is always tread of bwtn performance and power