Battery power profiles and performance

One question that I had for a long time was do the power profiles on a laptop have any real impact?. While working on this blog’s engine, I had to compile large Haskell libraries (e.g., pandoc) and I got some empirical proof that there is some impact. Hence, the question now turned into how much is performance affected when switching from “performance” to “power saver” mode?. Let’s answer this in this article.

Designing the experiments 🔗

I am using NixOs – which is Linux –, within a GNOME environment. This allows me to select between 3 different power profiles: “performance”, “balanced” and “power saver”. Thus, we need to compare across all these modes.

Most (if not all) modern laptops come with multicore CPUs, to perform multithreading. The performance of this could also be affected by battery power profile, hence I will test with single thread programs as well as programs that run on as many threads as the system can run at the same time.

In general, programs can be divided into 2 categories: compute-bound programs – which spend most of their time running code on the CPU, with data from cache –, and I/O-bound programs – which spend most of their time waiting for a peripheral device. There are variations in between, this is not a binary decision but a spectrum. But for the purposes of this study, we’ll only focus on a program that does a lot of computation and a program that read a large number of files from the disk. At some point in the future, I might decide to replicate these experiments with programs that use a GPU as well as programs that access the internet, etc.

For the I/O bound scenario, we need to take into account the operating system’s cache. Hence, I will run this experiment both with a cold filesystem cache (i.e., after a reboot) and with a warm one (i.e., running the same program multiple times in a row before performing an actual measurement).

Finally, to study the impact of battery level, we will repeat the measurements at different battery charge levels, both plugged-in and unplugged.

These are the general things we need to consider for this study. Let’s now write the programs for the test scenarios.

Programs under test 🔗

As discussed, we want one suite of programs that are CPU-bound and one suite of programs that are disk-bound. Each suite contains a version that runs single threaded and a version that runs multi-threaded. I will post the code without much comments, but I also want the change from single-thread to multi-thread to be minimal.

Matrix multiplication 🔗

For the CPU-bound program, I wrote a very naive matrix multiplication in C. Thus, the programs in this suite are named mm.., where the last 2 letters relate to whether the program is for a single core (mmsc) or multicore (mmmc).

The single-threaded program is:

// mmsc.c
#include <stdio.h>
#include <stdlib.h>

#ifndef SZ
#define SZ 10
#endif

int main(void) {
	int sz = SZ * SZ;
	long long *a = calloc(sz, sizeof(*a));
	for (int i = 0; i < sz; i++) a[i] = 1;
	a[1] = 0;

	long long *b = calloc(sz, sizeof(*b));
	for (int i = 0; i < SZ; i++)
		for (int j = 0; j < SZ; j++)
			for (int k = 0; k < SZ; k++)
				b[i * SZ + j] += a[i * SZ + k] * a[k * SZ + j];

	long long sa = 0;
	for (int i = 0; i < SZ; i++)
		for (int j = 0; j < SZ; j++)
			sa += b[i * SZ + j];

	long long sd = 0;
	for (int i = 0; i < SZ; i++)
		sd += b[i * SZ + i];

	long long esa = (long long)(SZ * SZ) * SZ - 2 * SZ;
	long long esd = SZ * SZ - 2;
	printf("%lld = %lld || %lld = %lld\n", sa, esa, sd, esd);

	free(b);
	free(a);
	return 0;
}

I am basically computing \(B = A^2\) where \(A\) is a matrix of size SZ * SZ filled with 1, except one element being set to 0:

\[A = \begin{pmatrix} 1 & 0 & 1 & \ldots & 1 \\ 1 & 1 & 1 & \ldots & 1 \\ 1 & 1 & 1 & \ldots & 1 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & 1 & 1 & \ldots & 1 \end{pmatrix}\]

To check that the multiplication is correct, I compute the sum of all elements and the sum of the elements on the diagonal. It is easy to prove (by induction) that these sums are:

\[\begin{align} s_a &=& \sum_{i=0}^n\sum_{j=0}^n{B_{ij}} &=& n^3 - 2n\\ s_d &=& \sum_{i=0}^n {B_{ii}} &=& n^2 - 2 \end{align}\]

Hence, the end of the program is printing the value of the sum as computed by iterating over the resulting matrix as well as those computed from the formulas above.

The multicore program uses OpenMP for minimal code changes:

diff --git a/mmsc.c b/mmmc.c
index 7531933..4209e09 100644
--- a/mmsc.c
+++ b/mmmc.c
@@ -12,10 +12,12 @@ int main(void) {
        a[1] = 0;

        long long *b = calloc(sz, sizeof(*b));
+#pragma omp parallel for
        for (int i = 0; i < SZ; i++)
                for (int j = 0; j < SZ; j++)
                        for (int k = 0; k < SZ; k++)
                                b[i * SZ + j] += a[i * SZ + k] * a[k * SZ + j];
+#pragma omp barrier

        long long sa = 0;
        for (int i = 0; i < SZ; i++)

The size parameter is given by a #define as this would allow me to change the matrix size to have a runtime that is large enough but also not require me to spend a full day with all the experiments. I settled for a size of 2048, so the compilation of this suite is as follows:

gcc -Wall -Wextra -O3          -DSZ=2048 mmsc.c  -o mmsc
gcc -Wall -Wextra -O3 -fopenmp -DSZ=2048 mmmc.c  -o mmmc

File summarization 🔗

For the disk bound program, I wrote a C++ program that would read all files under a directory and its subdirectories and count the number of occurrences of the 'a' and 'b' characters. The naming scheme is fs.. (from file system), followed by sc/mc as above.

The single core directory walk program is:

#include <deque>
#include <filesystem>
#include <fstream>
#include <iostream>

struct PathEntry {
	std::string path;
	bool isFile;

	PathEntry(std::string path_, bool isFile_) : path(path_), isFile(isFile_) {}
	PathEntry() : path(""), isFile(false) {}
};

class Processor {
    public:
	Processor(std::string rootDir);
	~Processor();

	void Run();
	void PrintStats();
    private:
	void ProcessFile(std::string path);
	void ProcessDirectory(std::string path);

	int na_, nb_;
	std::deque<PathEntry> queue_;
};

Processor::Processor(std::string rootDir) : na_(0), nb_(0) {
	queue_.emplace_back(rootDir, false);
}

Processor::~Processor() {
	queue_.clear();
}

void Processor::Run() {
	while (!queue_.empty()) {
		PathEntry entry = queue_.front();
                queue_.pop_front();
		if (entry.isFile) {
			ProcessFile(entry.path);
		} else {
			ProcessDirectory(entry.path);
		}
	}
}

void Processor::PrintStats() {
	std::cout << na_ << " " << nb_ << "\n";
}

void Processor::ProcessFile(std::string path) {
	std::fstream file(path, std::ios::in);
	if (!file.is_open()) {
		std::cerr << "Failed to open " << path << "\n";
		return;
	}

	std::string line;
	while (std::getline(file, line)) {
		for (const auto& c :  line) {
			if (c == 'a') { na_++; }
			if (c == 'b') { nb_++; }
		}
	}

	file.close();
}

void Processor::ProcessDirectory(std::string path) {
	for (const auto& entry : std::filesystem::directory_iterator(path)) {
		queue_.emplace_back(entry.path().string(), !entry.is_directory());
	}
}

int main(int argc, char **argv) {
	if (argc < 2) {
		fprintf(stderr, "Need path to a directory");
		exit(EXIT_FAILURE);
	}

	Processor p{argv[1]};
	p.Run();
	p.PrintStats();
}

I am using std::filesystem utilities to walk the directory, putting every entry into a queue of entries to process later (breadth-first search). If an entry from the queue is a file, I just count the number of occurrences of 'a'/'b' characters. It is trivial to extend to count other occurrences, or to compute a checksum, or anything else, but I want to do minimal processing to ensure this program is as much disk-bound as possible.

For the parallel program, I need to use a thread pool, to ensure that the number of executing threads is exactly the number of cores my system has:

diff --git a/fssc.cc b/fsmc.cc
index c9f4bbc..d7872f0 100644
--- a/fssc.cc
+++ b/fsmc.cc
@@ -1,7 +1,11 @@
+#include <condition_variable>
 #include <deque>
 #include <filesystem>
 #include <fstream>
 #include <iostream>
+#include <mutex>
+#include <thread>
+#include <vector>

 struct PathEntry {
        std::string path;
@@ -19,15 +23,24 @@ class Processor {
        void Run();
        void PrintStats();
     private:
+       void ProcessLoop();
        void ProcessFile(std::string path);
        void ProcessDirectory(std::string path);

+       std::mutex mn_;
        int na_, nb_;
+
+       std::mutex mq_;
+       std::condition_variable cv_;
        std::deque<PathEntry> queue_;
+       int nf_;
+
+       std::vector<std::thread> threads_;
 };

 Processor::Processor(std::string rootDir) : na_(0), nb_(0) {
        queue_.emplace_back(rootDir, false);
+       nf_ = 1;
 }

 Processor::~Processor() {
@@ -35,21 +48,48 @@ Processor::~Processor() {
 }

 void Processor::Run() {
-       while (!queue_.empty()) {
-               PathEntry entry = queue_.front();
-                queue_.pop_front();
+       const int num_threads = std::thread::hardware_concurrency();
+
+       threads_.resize(num_threads);
+       for (int i = 0; i < num_threads; i++) {
+               threads_.emplace_back(&Processor::ProcessLoop, this);
+       }
+
+       for (auto& thread : threads_) {
+               if (thread.joinable()) {
+                   thread.join();
+               }
+       }
+       threads_.clear();
+}
+
+void Processor::ProcessLoop() {
+       while (true) {
+               PathEntry entry;
+               {
+                       std::unique_lock<std::mutex> lock(mq_);
+                       cv_.wait(lock, [this] { return !queue_.empty() || !nf_; });
+                       if (!nf_) { break; }
+                       entry = queue_.front();
+                       queue_.pop_front();
+               }
                if (entry.isFile) {
                        ProcessFile(entry.path);
                } else {
                        ProcessDirectory(entry.path);
                }
+               {
+                       std::unique_lock<std::mutex> lock(mq_);
+                       nf_--;
+                       cv_.notify_all();
+               }
        }
 }
 
 void Processor::ProcessFile(std::string path) {
        std::fstream file(path, std::ios::in);
        if (!file.is_open()) {
@@ -57,20 +97,27 @@ void Processor::ProcessFile(std::string path) {
                return;
        }
 
+       int na = 0, nb = 0;
        std::string line;
        while (std::getline(file, line)) {
                for (const auto& c :  line) {
-                       if (c == 'a') { na_++; }
-                       if (c == 'b') { nb_++; }
+                       if (c == 'a') { na++; }
+                       if (c == 'b') { nb++; }
                }
        }
 
        file.close();
+
+       std::unique_lock<std::mutex> lock(mn_);
+       na_ += na;
+       nb_ += nb;
 }
 
 void Processor::ProcessDirectory(std::string path) {
        for (const auto& entry : std::filesystem::directory_iterator(path)) {
+               std::unique_lock<std::mutex> lock(mq_);
                queue_.emplace_back(entry.path().string(), !entry.is_directory());
+               nf_++;
        }
 }

The diff is quite large, but it can be summarized as adding a few more includes, adding synchronization primitives and threading constructs, moving the processing code from Run into ProcessLoop, and small changes to the main processing functions to ensure minimal locking (i.e., in the ProcessFile function we first compute the character occurrences into local variables and then merge these to the class members only at the end).

Compilation is trivial here too:

g++ -Wall -Wextra -O3 fssc.cc -o fssc
g++ -Wall -Wextra -O3 fsmc.cc -o fsmc

Note: Initially, I was planning to write this suite in C, too. However, handling error cases from the POSIX API made the control flow very obfuscated and I had some race conditions. Switching to C++ allowed me to simplify the control flow as well as run the code under sanitizers to detect issues:

g++ -Wall -Wextra -O -g -fsanitize=thread fsmc.cc -o fsmc -ltsan
g++ -Wall -Wextra -O -g -fsanitize=address fsmc.cc -o fsmc -lasan
g++ -Wall -Wextra -O -g -fsanitize=leak fsmc.cc -o fsmc -llsan

Experimental results 🔗

We have 4 programs: mmsc, mmmc, fssc and fsmc. The mm.. ones are CPU-bound, the fs.. ones are I/O-bound. The ..sc ones are single-threaded whereas the ..mc ones run on every available CPU core.

The I/O-bound programs need to run both immediately after a reboot (cold filesystem cache) and after 4 repeated runs. That is, for the warm filesystem cache I will report the runtime after the fifth run.

For the I/O-bound programs, I am using a clone of the TensorFlow repository at commit 7099ad591878eb549574ef5e985180588eed9aca. In fact, I am creating 5 copies of the same checkout:

tf/
├── 1
│   └── ...tf_checkout...
├── 2
│   └── ...tf_checkout...
├── 3
│   └── ...tf_checkout...
├── 4
│   └── ...tf_checkout...
└── 5
    └── ...tf_checkout...

For each experiment, we run 5 trials recording the running time as given by /usr/env/bin time. I am picking 5 trials as that makes it easier to recover the quartiles from the data as the 2nd, 3rd and 4th sorted measurements. This will make it easy to get box plots to summarize the results. Next number of trials where this property would show up would be 11 experiments, but that is more than double and will take much too long to get all the data for that case. The results will also be published in table format.

Since battery levels might also impact performance, I am considering 3 different scenarios: battery charged at least 90%, battery between 40% and 60% charge and battery charge less than 15%. For each of these, I am measuring the time with the charger plugged in and with the laptop unplugged.

These programs are run without anything else running on the system, except a user session in the GNOME graphical environment and a GNOME terminal tab from which these start. I will be playing Zelda while these run :)

With these details, it is time to run all experiments and record the results. Next subsection has all data tables, but you can jump to the graphs one if you don’t want to see all the tiny details.

Table of results 🔗

We have a total of 540 experiments to run, as described below, grouped by type of experiment. In all of the tables below, the times are given in seconds.

CPU bound experiments 🔗

We group the tables based on power level. First, for battery charge at least 90% we have the following results:

For battery charge around 50% (between 40% and 60%), we have the following results:

Finally, for battery charge below 15%, we have the following results:

Disk bound experiments 🔗

Here, we have one additional dimension to consider, as discussed above: cold vs warm CPU cache. Since it is easier to run the experiment 5 times after a reboot and record the first and 5th running time, the table headers are changed.

First, battery level at at least 90%, but laptop being unplugged:

Next, battery level is still at least 90%, but laptop is now plugged in:

Battery level between 40% and 60%, unplugged:

Battery level around 50%, plugged in:

Low battery mode (<15%), unplugged:

Low batter mode but charging:

Graphs and summaries 🔗

Looking at the data we observe that battery level and charging status are not influencing performance. Up to measurement error, the data in all other dimensions matches regardless of how charged my laptop’s battery is or whether I am currently charging it or not. The only impact is that battery saver kicks in when battery level goes below a threshold (10%), at which point power mode is switched to power save mode. However, this can be switched back to performance if so desired.

Since the data is independent of battery status, I will combine all matching measurements. This means that I could have also ran only 3 rounds of each experiment, since I would determine the quartiles from the entire data, integrating over battery status. That would have been a nearly 40% reduction in experimentation time, but it would have required knowing the future or changing the experiment as data was arriving (or, a better experimental design where I would not set up to study all cartesian combinations from the start – the field of experimental design is something I want to touch in a separate (set of) article(s)).

If we combine all the data, we have the following summary table where all values are in seconds. The experiments are grouped based on binary, disk cache status (for disk-bound experiments only) and power mode:

There are 3 main categories of results: the CPU-bound experiments, and the 2 disk-bound ones, with the 2 conditions on the filesystem caches. It turns out that these caches are quite efficient!

For the CPU-bound experiments, we first see that the multi-core one did not result in the expected speed improvement, in any of the cases (I was running on 8 cores, but the gains are around 2x). This is because we only implemented a naive parallelism scheme, maybe I’ll improve on this in another article and possibly repeat the experiment. The performance impact of switching from performance battery mode to the other 2 are quite severe: 39% / 63% additional time when switching to balanced and 109% / 78% when switching to power-save.

For the disk-bound runtimes, under a cold filesystem cache, the changes in power profiles are not significant: we can approximate them to be around 7% compared to performance mode, regardless of whether we run in single threaded mode or with maximum concurrency, regardless of whether we switch to balanced mode or to power-save.

On the case when the filesystem cache is completely warmed up, so we can expect most reads to be served from it, we have significant performance impact when using power-save mode: 212% more time in the single threaded case and 242% more time in the multi threaded one! The balanced mode is not giving a significant change in behavior.

If the disk cache is warm, the runtime is extremely small, to the point where I am having doubts about whether running the code with multiple copies of TensorFlow repository (or multiple large repositories) might show different statistics. I might repeat this experiment later too.

Alternatively, if we look closer at the error bars in the disk-bound experiments, we can also state that battery power profile has no impact on the runtime of reading from disk. The observed differences are due to the small amount of CPU processing that we need to perform, plus HDD noise, and operating system tasks. If the filesystem cache is warm, then the ratio of time spent running on CPU vs waiting for I/O is larger, so the impact of the power-save mode is more visible.

I could devise a new experiment to test this (for another article), but reading documentation shows that only the CPU should be affected. This means that I also don’t need to run another suite of experiments to analyze networking impact. But, I might need to test how GPU acceleration is impacted. Will leave this for another time.