Question about multiprocessing and print

asked 2 years ago

lijr07 gravatar image

I have a question about printing and multiprocessing.

I have a function ll_permsWithNumericalMutation which generates data and I would like to print the data to a file line by line.

I used multiprocessing to increase the speed. In the function

if __name__ == '__main__':

A part of it is

N_CPU = multiprocessing.cpu_count()
        with Pool(processes=N_CPU) as p: 
            bb = p.starmap(ll_permsWithNumericalMutation,[(lls[i], typ, rank, max_column, n, repeat,minn,maxn,NRandomM,CL1,CL2,fp1,fp2,b6,b_noncrossing) for i in range(len(lls))])

I print the data while they are generated inside the function ll_permsWithNumericalMutation. The problem is that some data are printed more than one time (different parameters for ll_permsWithNumericalMutation can generate the same data). Another problem is that although I put

F1.write('\n')

when I print data inside ll_permsWithNumericalMutation, it can happen that a line is printed, but another line is printed before \n is printed because of the multiprocessing.

I can solve the problem by using a dictionary to store the data and then print the data only in

if __name__ == '__main__':

But the data are very large and it will take very long time. Is there some way to print in the function ll_permsWithNumericalMutation but without the above problem? Thank you very much.

Preview: (hide)

Comments

The issue with same data generated multiple times has nothing do with printing and needs to be addressed by modification of the algorithm.

As for writing \n - don't do that in the separate statement but rather include \n in the same write statement along with your data.

Max Alekseyev gravatar imageMax Alekseyev ( 2 years ago )

@max, thank you very much! So for example I can write F1.write( str(r)+'\n')? I will try that.

lijr07 gravatar imagelijr07 ( 2 years ago )

ot just F1.write(r, '\n') or F1.write(f'{r}\n')

Max Alekseyev gravatar imageMax Alekseyev ( 2 years ago )

@max, thank you very much!

lijr07 gravatar imagelijr07 ( 2 years ago )