Question about multiprocessing and print

asked 2022-08-24 10:48:45 +0200

lijr07 gravatar image

I have a question about printing and multiprocessing.

I have a function ll_permsWithNumericalMutation which generates data and I would like to print the data to a file line by line.

I used multiprocessing to increase the speed. In the function

if __name__ == '__main__':

A part of it is

N_CPU = multiprocessing.cpu_count()
        with Pool(processes=N_CPU) as p: 
            bb = p.starmap(ll_permsWithNumericalMutation,[(lls[i], typ, rank, max_column, n, repeat,minn,maxn,NRandomM,CL1,CL2,fp1,fp2,b6,b_noncrossing) for i in range(len(lls))])

I print the data while they are generated inside the function ll_permsWithNumericalMutation. The problem is that some data are printed more than one time (different parameters for ll_permsWithNumericalMutation can generate the same data). Another problem is that although I put


when I print data inside ll_permsWithNumericalMutation, it can happen that a line is printed, but another line is printed before \n is printed because of the multiprocessing.

I can solve the problem by using a dictionary to store the data and then print the data only in

if __name__ == '__main__':

But the data are very large and it will take very long time. Is there some way to print in the function ll_permsWithNumericalMutation but without the above problem? Thank you very much.

edit retag flag offensive close merge delete


The issue with same data generated multiple times has nothing do with printing and needs to be addressed by modification of the algorithm.

As for writing \n - don't do that in the separate statement but rather include \n in the same write statement along with your data.

Max Alekseyev gravatar imageMax Alekseyev ( 2022-08-24 16:33:20 +0200 )edit

@max, thank you very much! So for example I can write F1.write( str(r)+'\n')? I will try that.

lijr07 gravatar imagelijr07 ( 2022-08-24 17:16:49 +0200 )edit

ot just F1.write(r, '\n') or F1.write(f'{r}\n')

Max Alekseyev gravatar imageMax Alekseyev ( 2022-08-24 18:34:51 +0200 )edit

@max, thank you very much!

lijr07 gravatar imagelijr07 ( 2022-08-24 19:36:11 +0200 )edit