Embedded Python creates a large number of memory leaks when trying to load a non-existent module

[GShabanov]

Bug report

Bug description:

baseName -- This module does not exist; it was renamed for testing to .py.tmp
After Finalize(); completes and the application is closed, a large number of leaks are created. This behavior is not observed if the module exists.

Sample code

    PyGILState_STATE state = PyGILState_Ensure();

    //
    // check script
    //
    LogMessage(_T("[*] Finding script %s.py...\n"), baseName);

    PyObject* pModuleName;
#ifdef _UNICODE
    pModuleName = PyUnicode_FromWideChar(baseName, -1);
#else  // _UNICODE
    pModuleName = PyUnicode_FromString(baseName);
#endif // _UNICODE

    if (pModuleName)
    {

        pModule = PyImport_GetModule(pModuleName);


        if (pModule)
        {
            Py_DECREF(pModule);

            pModule = PyImport_ReloadModule(pModuleName);
        }
        else
        {

            pModule = PyImport_Import(pModuleName);
        }


        Py_DECREF(pModuleName);
    }

    if (!pModule)
    {
        LogMessage(_T("[-] failed to find module: %s.py\n"), baseName);
        PyGILState_Release(state);
        return false;
    }

---

CPython versions tested on:

3.14

Operating systems tested on:

Windows

[StanFromIreland]

Modified repro to compile

#include <Python.h>
#include <stdio.h>

int main() {
    Py_Initialize();
    PyGILState_STATE state = PyGILState_Ensure();

    //
    // check script
    //
    const char* baseName = "nonexist";
    PyObject* pModule = NULL;

    printf("[*] Finding script %s.py...\n", baseName);

    PyObject* pModuleName;
    pModuleName = PyUnicode_FromString(baseName);

    if (pModuleName)
    {
        pModule = PyImport_GetModule(pModuleName);

        if (pModule)
        {
            Py_DECREF(pModuleName);
            pModule = PyImport_ReloadModule(pModuleName);
        }
        else
        {
            pModule = PyImport_Import(pModuleName);
        }

        Py_DECREF(pModuleName);
    }

    if (!pModule)
    {
        printf("[-] failed to find module: %s.py\n", baseName);
        PyGILState_Release(state);
        Py_Finalize();
        return 1;
    }

    Py_DECREF(pModule);

    PyGILState_Release(state);
    Py_Finalize();
    return 0;
}

On main, valgrind reports:

==132208== Memcheck, a memory error detector
==132208== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==132208== Using Valgrind-3.25.1 and LibVEX; rerun with -h for copyright info
==132208== Command: ./repro
==132208== 
[*] Finding script nonexist.py...
[-] failed to find module: nonexist.py
Exception ignored on threading shutdown:
ModuleNotFoundError: No module named 'nonexist'
==132208== 
==132208== HEAP SUMMARY:
==132208==     in use at exit: 394,652 bytes in 9 blocks
==132208==   total heap usage: 2,474 allocs, 2,465 frees, 2,779,108 bytes allocated
==132208== 
==132208== 32 bytes in 1 blocks are still reachable in loss record 1 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x420C02: _PyMem_RawMalloc (obmalloc.c:63)
==132208==    by 0x420C02: _PyMem_DefaultRawMalloc (obmalloc.c:390)
==132208==    by 0x420C02: _PyMem_DefaultRawWcsdup (obmalloc.c:435)
==132208==    by 0x6D782B: _PyPathConfig_UpdateGlobal (pathconfig.c:152)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 44 bytes in 1 blocks are still reachable in loss record 2 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x420C02: _PyMem_RawMalloc (obmalloc.c:63)
==132208==    by 0x420C02: _PyMem_DefaultRawMalloc (obmalloc.c:390)
==132208==    by 0x420C02: _PyMem_DefaultRawWcsdup (obmalloc.c:435)
==132208==    by 0x6D7795: _PyPathConfig_UpdateGlobal (pathconfig.c:149)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 44 bytes in 1 blocks are still reachable in loss record 3 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x420C02: _PyMem_RawMalloc (obmalloc.c:63)
==132208==    by 0x420C02: _PyMem_DefaultRawMalloc (obmalloc.c:390)
==132208==    by 0x420C02: _PyMem_DefaultRawWcsdup (obmalloc.c:435)
==132208==    by 0x6D77C7: _PyPathConfig_UpdateGlobal (pathconfig.c:150)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 68 bytes in 1 blocks are still reachable in loss record 4 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x420C02: _PyMem_RawMalloc (obmalloc.c:63)
==132208==    by 0x420C02: _PyMem_DefaultRawMalloc (obmalloc.c:390)
==132208==    by 0x420C02: _PyMem_DefaultRawWcsdup (obmalloc.c:435)
==132208==    by 0x6D788F: _PyPathConfig_UpdateGlobal (pathconfig.c:154)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 104 bytes in 1 blocks are still reachable in loss record 5 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x420C02: _PyMem_RawMalloc (obmalloc.c:63)
==132208==    by 0x420C02: _PyMem_DefaultRawMalloc (obmalloc.c:390)
==132208==    by 0x420C02: _PyMem_DefaultRawWcsdup (obmalloc.c:435)
==132208==    by 0x6D77F9: _PyPathConfig_UpdateGlobal (pathconfig.c:151)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 376 bytes in 1 blocks are still reachable in loss record 6 of 9
==132208==    at 0x483EB26: malloc (vg_replace_malloc.c:446)
==132208==    by 0x6D7922: _PyPathConfig_UpdateGlobal (pathconfig.c:171)
==132208==    by 0x533610: interpreter_update_config (pylifecycle.c:413)
==132208==    by 0x533610: init_interp_main (pylifecycle.c:1204)
==132208==    by 0x5361AB: pyinit_main (pylifecycle.c:1413)
==132208==    by 0x5361AB: Py_InitializeFromConfig (pylifecycle.c:1444)
==132208==    by 0x5361AB: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 768 bytes in 1 blocks are still reachable in loss record 7 of 9
==132208==    at 0x4846607: realloc (vg_replace_malloc.c:1801)
==132208==    by 0x42284E: PyMem_RawRealloc (obmalloc.c:1018)
==132208==    by 0x42284E: new_arena (obmalloc.c:1939)
==132208==    by 0x42284E: allocate_from_new_pool (obmalloc.c:2153)
==132208==    by 0x422B90: pymalloc_alloc (obmalloc.c:2317)
==132208==    by 0x422B90: _PyObject_Malloc (obmalloc.c:2328)
==132208==    by 0x4FF554: _PyObject_MallocWithType (pycore_object_alloc.h:46)
==132208==    by 0x4FF554: gc_alloc (gc.c:2291)
==132208==    by 0x4FF554: _PyObject_GC_New (gc.c:2311)
==132208==    by 0x675869: new_dict (dictobject.c:875)
==132208==    by 0x675869: PyDict_New (dictobject.c:973)
==132208==    by 0x4A0746: init_interned_dict (unicodeobject.c:351)
==132208==    by 0x4A0746: _PyUnicode_InitGlobalObjects (unicodeobject.c:15825)
==132208==    by 0x532A84: pycore_init_global_objects (pylifecycle.c:689)
==132208==    by 0x532A84: pycore_interp_init (pylifecycle.c:873)
==132208==    by 0x5344A3: pyinit_config (pylifecycle.c:954)
==132208==    by 0x5344A3: pyinit_core.constprop.0 (pylifecycle.c:1117)
==132208==    by 0x53612F: Py_InitializeFromConfig (pylifecycle.c:1437)
==132208==    by 0x53612F: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 131,072 bytes in 1 blocks are still reachable in loss record 8 of 9
==132208==    at 0x48463F3: calloc (vg_replace_malloc.c:1675)
==132208==    by 0x407B02: PyMem_RawCalloc (obmalloc.c:1009)
==132208==    by 0x407B02: arena_map_get (obmalloc.c:1797)
==132208==    by 0x407B02: arena_map_mark_used (obmalloc.c:1840)
==132208==    by 0x4225AE: new_arena (obmalloc.c:1973)
==132208==    by 0x4225AE: allocate_from_new_pool (obmalloc.c:2153)
==132208==    by 0x422B90: pymalloc_alloc (obmalloc.c:2317)
==132208==    by 0x422B90: _PyObject_Malloc (obmalloc.c:2328)
==132208==    by 0x4FF554: _PyObject_MallocWithType (pycore_object_alloc.h:46)
==132208==    by 0x4FF554: gc_alloc (gc.c:2291)
==132208==    by 0x4FF554: _PyObject_GC_New (gc.c:2311)
==132208==    by 0x675869: new_dict (dictobject.c:875)
==132208==    by 0x675869: PyDict_New (dictobject.c:973)
==132208==    by 0x4A0746: init_interned_dict (unicodeobject.c:351)
==132208==    by 0x4A0746: _PyUnicode_InitGlobalObjects (unicodeobject.c:15825)
==132208==    by 0x532A84: pycore_init_global_objects (pylifecycle.c:689)
==132208==    by 0x532A84: pycore_interp_init (pylifecycle.c:873)
==132208==    by 0x5344A3: pyinit_config (pylifecycle.c:954)
==132208==    by 0x5344A3: pyinit_core.constprop.0 (pylifecycle.c:1117)
==132208==    by 0x53612F: Py_InitializeFromConfig (pylifecycle.c:1437)
==132208==    by 0x53612F: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== 262,144 bytes in 1 blocks are still reachable in loss record 9 of 9
==132208==    at 0x48463F3: calloc (vg_replace_malloc.c:1675)
==132208==    by 0x407BCE: PyMem_RawCalloc (obmalloc.c:1009)
==132208==    by 0x407BCE: arena_map_get (obmalloc.c:1785)
==132208==    by 0x407BCE: arena_map_mark_used (obmalloc.c:1840)
==132208==    by 0x4225AE: new_arena (obmalloc.c:1973)
==132208==    by 0x4225AE: allocate_from_new_pool (obmalloc.c:2153)
==132208==    by 0x422B90: pymalloc_alloc (obmalloc.c:2317)
==132208==    by 0x422B90: _PyObject_Malloc (obmalloc.c:2328)
==132208==    by 0x4FF554: _PyObject_MallocWithType (pycore_object_alloc.h:46)
==132208==    by 0x4FF554: gc_alloc (gc.c:2291)
==132208==    by 0x4FF554: _PyObject_GC_New (gc.c:2311)
==132208==    by 0x675869: new_dict (dictobject.c:875)
==132208==    by 0x675869: PyDict_New (dictobject.c:973)
==132208==    by 0x4A0746: init_interned_dict (unicodeobject.c:351)
==132208==    by 0x4A0746: _PyUnicode_InitGlobalObjects (unicodeobject.c:15825)
==132208==    by 0x532A84: pycore_init_global_objects (pylifecycle.c:689)
==132208==    by 0x532A84: pycore_interp_init (pylifecycle.c:873)
==132208==    by 0x5344A3: pyinit_config (pylifecycle.c:954)
==132208==    by 0x5344A3: pyinit_core.constprop.0 (pylifecycle.c:1117)
==132208==    by 0x53612F: Py_InitializeFromConfig (pylifecycle.c:1437)
==132208==    by 0x53612F: Py_InitializeEx (pylifecycle.c:1475)
==132208==    by 0x402312: main (in /tmp/repro)
==132208== 
==132208== LEAK SUMMARY:
==132208==    definitely lost: 0 bytes in 0 blocks
==132208==    indirectly lost: 0 bytes in 0 blocks
==132208==      possibly lost: 0 bytes in 0 blocks
==132208==    still reachable: 394,652 bytes in 9 blocks
==132208==         suppressed: 0 bytes in 0 blocks
==132208== 
==132208== For lists of detected and suppressed errors, rerun with: -s
==132208== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

3.13 report, features leak

output.txt

[sergey-miryanov]

@StanFromIreland it seems that you should call pModuleReloaded = PyImport_ReloadModule(pModule); instead of pModule = PyImport_ReloadModule(pModuleName);. And also store previous old pModule to properly clean it.

Edited: But this branch not taken by this test. Sorry.

[StanFromIreland]

But this branch not taken by this test.

Yes, I was trying to not modify the provided sample, with minimal changes needed for it to compile on Linux.

[GShabanov]

Sorry for my mistake in the code. I fixed it mostly. However, it does not affect the final result, since we assume that we do not reload the module, but load a new one.

        if (pModule)
        {
            Py_DECREF(pModule); // here

            pModule = PyImport_ReloadModule(pModuleName);
        }

A little explanation of my task: I am working on a robotic device simulator (C++); the mathematical scripts had to be taken out separately, so I took Python as a basis. The task was to reload the module with updated code when someone changes the file. In this case, the option was tested when the main application cannot find this module (script). I noticed that after this, the application reports a large number of memory leaks.

[ZeroIntensity]

To my knowledge, Python just doesn't care about freeing all memory at shutdown, because the process is about to end anyway. Running the following through valgrind is showing several "leaks":

#include <Python.h>

int main() {
    Py_Initialize();
    Py_Finalize();
    return 0;
}

==16153== HEAP SUMMARY:
==16153==     in use at exit: 394,508 bytes in 9 blocks
==16153==   total heap usage: 2,635 allocs, 2,626 frees, 2,524,547 bytes allocated
==16153== 
==16153== LEAK SUMMARY:
==16153==    definitely lost: 0 bytes in 0 blocks
==16153==    indirectly lost: 0 bytes in 0 blocks
==16153==      possibly lost: 0 bytes in 0 blocks
==16153==    still reachable: 394,508 bytes in 9 blocks
==16153==         suppressed: 0 bytes in 0 blocks
==16153== Reachable blocks (those to which a pointer was found) are not shown.

Is this causing a problem in practice? The process should always be at the end of its life when it calls Py_Finalize, so the memory should be immediately released anyway.

[GShabanov]

First, why doesn't such a problem occur if the module was found?
==131533==
==131533== HEAP SUMMARY:
==131533== in use at exit: 165,219 bytes in 3,063 blocks
==131533== total heap usage: 37,114 allocs, 34,051 frees, 5,199,977 bytes allocated

Second, the process that Python embeds does not necessarily terminate after Py_Finalize. In my case, it continues to run. That is, the memory will continue to hang regardless of whether Python will be used further.
And third, are you saying that leaking memory or objects is normal?

There is quite a lot of memory leaking; we can check if it is growing. I can offer such a test.

#include <Python.h>

int main()
{
     int iterations;

     Py_Initialize();
     PyGILState_STATE state = PyGILState_Ensure();
     PyObject* pModuleName;

     for (iterations = 0; iterations < 1000; iterations++) 
     {
        pModuleName = PyUnicode_FromString("badname");

        if (pModuleName)
        {
             pModule = PyImport_Import(pModuleName);
             assert(!pModule);
             Py_DECREF(pModuleName);
        }
    }

    PyGILState_Release(state);
    Py_Finalize();
    return 0;
}

[ZeroIntensity]

First, why doesn't such a problem occur if the module was found?

It does, that's what I'm trying to say. Leak checkers will complain about Python becuase it doesn't free some of its state in Py_Finalize.

Second, the process that Python embeds does not necessarily terminate after Py_Finalize. In my case, it continues to run.

Yeah, that doesn't really work without leaks. Getting this supported will likely require a larger proposal.

There is quite a lot of memory leaking; we can check if it is growing. I can offer such a test.

You never call PyGILState_Release. I'm pretty sure that will cause the thread state to leak, so I guess there could be state accumulated there that is never freed. FWIW, you don't actually need to call PyGILState_Ensure here at all; Py_Finalize will initialize the thread state for you.

I couldn't reproduce any memory accumulation with PyImport_Import failures on 3.13+.

[GShabanov]

First, why doesn't such a problem occur if the module was found?

It does, that's what I'm trying to say. Leak checkers will complain about Python becuase it doesn't free some of its state in Py_Finalize.

As I understand this problem, perhaps not completely. If the module is found immediately, then a full scan does not occur. Just in the process of the full scan, these entities are created, which then wait for atexit (possibly). But this fact needs to be checked.

Second, the process that Python embeds does not necessarily terminate after Py_Finalize. In my case, it continues to run.

Yeah, that doesn't really work without leaks. Getting this supported will likely require a larger proposal.

I think I need a lot more time to get into Python's life cycle. In principle, this corresponds to the ideology of open source products: if you want to change something, take part.

There is quite a lot of memory leaking; we can check if it is growing. I can offer such a test.

You never call PyGILState_Release. I'm pretty sure that will cause the thread state to leak, so I guess there could be state accumulated there that is never freed. FWIW, you don't actually need to call PyGILState_Ensure here at all; Py_Finalize will initialize the thread state for you.

Sorry, I corrected my example. I wrote it here as an example (I didn't run it). Yes, it is really necessary to release GIL PyGILState_Release there

I couldn't reproduce any memory accumulation with PyImport_Import failures on 3.13+.

This is an interesting observation, since in that case it is a bug that was introduced between 3.13 and 3.14