`Can't initialize heap: [0x000001e7] Attempt to access invalid address` random segfault on Windown x86

[mvorisek]

Description

CI report: https://github.com/php/php-src/actions/runs/3160522498/jobs/5145079204#step:8:110

here is commit /w Windows CI to reproduce: 37498f9

it seems there is some core problem not specific to the failed test, as later absolutely unrelated push https://github.com/php/php-src/compare/37498f95b5b26d47a03b74b0dcc062d545195a0c..3adb2737355882b3d808cf53630d304374d5bf7e did not produce it. and in another pushes I saw another failed tests like in #8643 and #8450...

PHP Version

PHP 8.0 and probably 8.1+ too

Operating System

Windows x86/32 bit

[mvorisek]

related to #8643 and https://bugs.php.net/bug.php?id=77194 (confirmed in https://bugs.php.net/bug.php?id=77194#1548144531)

To reproduce, please run couple hundreds of full tests iterations on Windows /w x86 build locally and you should be able to reproduce it, in CI, it is reproducible after a few pushes /w random test failures.

[TysonAndre]

I've seen similar spurious issues in Phan's for a while, using https://ci.appveyor.com/project/TysonAndre/phan/branch/v5/job/1f74k3nsurgxpdgb when php is running in paratest

• I guess I can switch away from x86 to work around this, it's just a coincidence that the appveyor file I based Phan's appveyor setup on was using x86, and x64 is far, far more common

.phan/plugins\RedundantAssignmentPlugin.php:1 PhanNativePHPSyntaxCheckPlugin means that https://github.com/phan/phan/blob/v5/.phan/plugins/InvokePHPNativeSyntaxCheckPlugin.php was running php.exe --syntax-check on the file RedundantAssignmentPlugin

• This is essentially making multiple calls to php --syntax-check, up to max_incomplete_processes concurrently - knowing that the issue can also be reproduced with just --syntax-check may be of use in trying to diagnose or reproduce the issue

  (though php is also doing a lot of other things)
  (This is also using the x86 32-bit build for reasons I forget.)
  (https://github.com/phan/phan/blob/v5/.appveyor.yml - I believe it doesn't use opcache, but I guess the tests of php-src)

• Many failures in the build history are related to InvokePHPNativeSyntaxCheckPlugin, probably because it calls php.exe extremely often, so it has the largest chance of encountering those failures

Can't initialize heap: [0x000001e7] Attempt to access invalid address
░░░░░  54 / 619 (  8%) 409MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 108 / 619 ( 17%) 426MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 162 / 619 ( 26%) 436MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 216 / 619 ( 35%) 449MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 270 / 619 ( 43%) 456MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ [324](https://ci.appveyor.com/project/TysonAndre/phan/branch/v5/job/1f74k3nsurgxpdgb#L324) / 619 ( 52%) 458MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 378 / 619 ( 61%) 461MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 432 / 619 ( 69%) 462MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 486 / 619 ( 78%) 463MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 540 / 619 ( 87%) 466MB
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 594 / 619 ( 96%) 470MB
░░░░░░░░░░░░░░░░░░░░░░░░░                              619 / 619 (100%) 471MB
.phan/plugins\RedundantAssignmentPlugin.php:1 PhanNativePHPSyntaxCheckPlugin Saw error or notice for php --syntax-check: "No output was detected. Is 'c:\\\\projects\\\\php\\\\php.exe' a relative or absolute path to an executable PHP binary?"

[TysonAndre]

It looks like initializing the memory manager ("Can't initialize heap", zend_mm_init) is failing. It's attempting to allocate a 2MB chunk (ZEND_MM_CHUNK_SIZE) but failing.

If php was compiled for windows with ZEND_MM_ERROR it looks like it would output the error message with the failure reason for VirtualAlloc

• Guesses: Not enough memory in appveyor (really unlikely if it's failing on the 2MB)? Or is the heap somehow fragmented?
• Maybe the fact that hundreds of php processes have been started up has caused available memory to be fragmented

https://learn.microsoft.com/en-us/windows/win32/memory/large-page-support This sounds somewhat related, but I'm unfamiliar with win32 programming

When writing applications that use large-page memory, keep the following considerations in mind:

• Large-page memory regions may be difficult to obtain after the system has been running for a long time because the physical space for each large page must be contiguous, but the memory may have become fragmented. Allocating large pages under these conditions can significantly affect system performance. Therefore, applications should avoid making repeated large-page allocations and instead allocate all large pages one time, at startup.

// Zend/zend_alloc.c
static zend_mm_heap *zend_mm_init(void)
{
	zend_mm_chunk *chunk = (zend_mm_chunk*)zend_mm_chunk_alloc_int(ZEND_MM_CHUNK_SIZE, ZEND_MM_CHUNK_SIZE);
	zend_mm_heap *heap;

	if (UNEXPECTED(chunk == NULL)) {
#if ZEND_MM_ERROR
#ifdef _WIN32
		stderr_last_error("Can't initialize heap");
#else
		fprintf(stderr, "\nCan't initialize heap: [%d] %s\n", errno, strerror(errno));
#endif
#endif
		return NULL;
	}

// Zend/zend_alloc.c
static void *zend_mm_mmap(size_t size)
{
#ifdef _WIN32
	void *ptr = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);

	if (ptr == NULL) {
#if ZEND_MM_ERROR
		stderr_last_error("VirtualAlloc() failed");
#endif
		return NULL;
	}
	return ptr;
#else

[TysonAndre]

If the issue is just in VirtualAlloc and is actually related to starting and stopping thousands of php processes in a short time period, causing virtual memory to be fragmented (not actually sure), then a possible workaround for applications that aren't php-src to in x86(32-bit builds) is to SET USE_ZEND_ALLOC=0 before running php to use malloc/free instead of the Zend memory manager, which will perform regular size malloc allocations rather than allocating blocks of size 2 MB.

[mvorisek]

@TysonAndre thank you very much for looking into it!

Guesses: Not enough memory in appveyor (really unlikely if it's failing on the 2MB)? Or is the heap somehow fragmented?

I would expect consequent tests then to fail. Currently the failure is more or less random and next test is (p>99%) passing.

The fragmentation can be closely related, as in 32-bit address space it is much easier to create some collision than in 64-bit space. Maybe there is some bug also in 64-bit builds, but they are only much more rare.

I did some more reseach:

https://forum.exetools.com/showthread.php?t=8963 - are we sure we do not increment the address over the alloved range?

VirtualAlloc(NULL, ... is maybe fine, but VirtualAlloc(addr, ... can create collisions easily (in 64-bit address space however less likely)

"mremp Windows" - https://stackoverflow.com/questions/17197615/no-mremap-for-windows#38502927 can this help?

to test is memory remap is a problem, is there any easy way to compile php without zend_mm_mmap_fixed completely?

or can we reserve (but not commit) like 1GB address space for php mremp on php start at least?

based on https://stackoverflow.com/questions/31672069/is-the-mem-reserve-flag-of-virtualalloc-really-useful-today and https://stackoverflow.com/questions/21368429/error-code-487-error-invalid-address-when-using-virtualallocex we should always reserve a memory

more links (for later, if the above does not help):

https://www.youtube.com/watch?v=MPnsPlDJbhI - can someone with Windows check?

using VirtualQuery https://forums.codeguru.com/showthread.php?258460-ERROR_INVALID_ADDRESS-with-VirtualAllocEx-and-Win2K it should be possible to check the problem before even VirtualAlloc call

https://stackoverflow.com/questions/55119066/virtualalloc-failed

golang/go#35465

https://bugzilla.mozilla.org/show_bug.cgi?id=1662560