Python Multiprocessing with output to file

Photo by Clément Hélardot on Unsplash

Python multiprocessing module comes into mind whenever we can split a big problem into multiple smaller chunks and run them in parallel. Recently I encountered a scenario where I need to write the parallelized results into an output file, and the direct approach of adding the writing-to-file logic into the worker function doesn’t work as expected due to some concurrency issues when multiple workers access the output…