Problem in memory usage when loading file (Memory leak?)

asked 0 years ago

nur_hamid gravatar image

I need to get coefficients of some polynomials save in my disk. However, the memory usage is increasing in each iteration although garbage collector is used. Here is my program. The memory looks increasing in the command pol = load(f"{foldername}pp{2}.sobj")

def get_co(filename, Mon):
     pol = load(filename)
     co = [pol.coefficient(m) for m in Mon]
     del pol
     gc.collect()
    return co


def load_monomials(n):
    foldername = f"mult/p{n}/"
    diname = f"mon/di{n}"
    os.makedirs(diname, exist_ok=True)  # Ensure directory exists

    k = 0
    filename = f"{foldername}pp{k}.sobj"

    # Take set of monomials with different coefficients
    Mon = set()
    # Call the polynomials of index 2 (we want random, but not 0 actually)
    pol = load(f"{foldername}pp{2}.sobj")

    print("Taking the monomials with different coefficients")
    Co = set()
    for mon in pol.monomials():
        coe = pol.coefficient(mon)
        if coe not in Co:
            Co.add(coe)
            Mon.add(mon)
    print("The length of monomials is", len(Mon))

    while os.path.exists(filename):
        print(f"Loading {filename}...")
        coname = f"{diname}/co{k}.sobj"       

        co = get_co(filename, Mon)
        save(co, coname)

        # Explicitly dereference
        del co
        gc.collect()

        print(f"Memory usage: {get_memory_usage()} MB, GC stats: {gc.get_stats()}")
        k += 1
        filename = f"{foldername}pp{k}.sobj"

load_monomials(160)
Preview: (hide)

Comments

Can you create a minimal example illustrating the problem? (Perhaps, loading the same polynomial in the loop.) Then the issue can be reported at https://github.com/sagemath/sage/issues

Max Alekseyev gravatar imageMax Alekseyev ( 0 years ago )

For instance:

k=0
filename = f"pol{k}.sobj"
while os.path.exists(filename):
    pol = load(filename)
    k += 1
    filename = f"pol{k}.sobj"

Even loading the file only, the memory usage keep increasing.

nur_hamid gravatar imagenur_hamid ( 9 hours ago )

Please report the issue at https://github.com/sagemath/sage/issues

As a workaround, you may try to add @fork decorator to your get_co function:

@fork
def get_co(filename, Mon):
...
Max Alekseyev gravatar imageMax Alekseyev ( 26 mins ago )