Multi-Processing Performance Benchmarks¶
This section contains a performance comparison of single- and multiprocess-based calculation methods used in Time Series.
The %%timeit cell magic runs the cell content multiple times and prints statistics on those multiple runs. This reduces factors such as garbage collection pauses which influence the runtime performance and can be used to verify performance assumptions.
from multiprocessing import Pool, cpu_count
from numpy import ma
from pathlib import Path
import rasterio as r
test_files = list(Path('resources/tempelhofer_feld/ndvi').glob('*.tif'))
print(f'Number of files: {len(test_files)}')
Number of files: 30
The performance is tested with the following function call:
def average(file_path):
with r.open(file_path) as src:
data = src.read(1, masked=True)
return file_path, ma.average(data)
The function receives a string or pathlib.Path, reads the data using rasterio, and calculates the average value inside the data using the ma.average function provided by numpy.
The benchmark is performed on 4 CPUs:
cpu_count()
4
Single File¶
In a Single Process¶
%%timeit
average(test_files[0])
4.35 ms ± 24.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
With Multiprocessing¶
%%timeit
with Pool() as pool:
averages = [avg for avg in pool.map(average, test_files[:1])]
187 ms ± 5.09 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
All Files in Folder¶
In a Single Process¶
%%timeit
averages = list(map(average, test_files))
131 ms ± 366 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
With Multiprocessing¶
%%timeit
with Pool() as pool:
averages = list(pool.map(average, test_files))
248 ms ± 10 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
All Files Multiple Times¶
In a Single Process¶
%%timeit
averages = list(map(average, test_files * 5))
654 ms ± 952 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
With Multiprocessing¶
%%timeit
with Pool() as pool:
averages = list(pool.map(average, test_files * 5))
434 ms ± 16.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Result¶
Multiprocessing comes with an overhead, which is significant enough to rule it out when processing a single element. When the list of elements is sufficiently large, there is a slight reduction in processing time, that, even with a higher standard deviation, manages to be faster than the single-process version.
Averaging the masked array is an operation that can be implemented to scale in \(O(N)\) with the size of the input array. The time reduction should be even higher for more complex tasks.