I have a question about printing and multiprocessing.
I have a function ll_permsWithNumericalMutation which generates data and I would like to print the data to a file line by line.
I used multiprocessing to increase the speed. In the function
if __name__ == '__main__':
A part of it is
N_CPU = multiprocessing.cpu_count()
with Pool(processes=N_CPU) as p:
bb = p.starmap(ll_permsWithNumericalMutation,[(lls[i], typ, rank, max_column, n, repeat,minn,maxn,NRandomM,CL1,CL2,fp1,fp2,b6,b_noncrossing) for i in range(len(lls))])
I print the data while they are generated inside the function ll_permsWithNumericalMutation. The problem is that some data are printed more than one time (different parameters for ll_permsWithNumericalMutation can generate the same data). Another problem is that although I put
F1.write('\n')
when I print data inside ll_permsWithNumericalMutation, it can happen that a line is printed, but another line is printed before \n is printed because of the multiprocessing.
I can solve the problem by using a dictionary to store the data and then print the data only in
if __name__ == '__main__':
But the data are very large and it will take very long time. Is there some way to print in the function ll_permsWithNumericalMutation but without the above problem? Thank you very much.
