PyPy link error when limited_api is set

[mattip]

Describe the bug
Setting the limite_api argument to extension_module changes the link library for windows to python3.lib (rather than a version-specific python39.lib), but this is not shipped with PyPy.
To Reproduce

This came up in numpy/numpy#25531, where the complete meson build contains two extensions. Here is what I needed to do for the limited api extension:

# py does not export the is_pypy method
if not py.get_variable('SOABI').startswith('pypy')
  py.extension_module(
      'limited_api',
      'limited_api.pyx',
      install: false,
      c_args: [
        '-DNPY_NO_DEPRECATED_API=0',  # Cython still uses old NumPy C API
        # Require 1.25+ to test datetime additions
        '-DNPY_TARGET_VERSION=NPY_2_0_API_VERSION',
        '-DCYTHON_LIMITED_API=1',
      ],
      include_directories: [npy_include_path],
      limited_api: '3.7',
  )
endif

where limited_api.pyx is

#cython: language_level=3

"""
Make sure cython can compile in limited API mode (see meson.build)
"""

cdef extern from "numpy/arrayobject.h":
    pass
cdef extern from "numpy/arrayscalars.h":
    pass

Expected behavior
I expected the build to succeed, it failed. Also, it would be nice to have a cleaner way to detect PyPy.

system parameters

• Is this a cross build or just a plain native build (for the same computer)? plain
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) windows
• what Python version are you using e.g. 3.8.0 3.9, 3.10
• what meson --version latest
• what ninja --version if it's a Ninja build

[rgommers]

# py does not export the is_pypy method

Also, it would be nice to have a cleaner way to detect PyPy.

@eli-schwartz gave it a thumbs-up in mesonbuild/meson-python#556 (reply in thread). Looks like the method name to use here is implementation, and use values from platform.python_implementation() in the Python stdlib (probably all-lowercase though, to align with the rest of Meson - see https://mesonbuild.com/Reference-tables.html). So the code above would use:

if py.implementation() != 'pypy'

[rgommers]

Setting the limite_api argument to extension_module changes the link library for windows to python3.lib (rather than a version-specific python39.lib), but this is not shipped with PyPy.

To clarify: what do you think the right solution is? It seems like this is mostly a problem on the PyPy side, right? Continuing to link to python39.lib seems wrong.

[mattip]

Until PyPy fixes that glaring error, detection of the implemenation to override or skip the limited_api option seems like the best course.

[rgommers]

Wouldn't it be better to get a clear error message when the user tries this? Something like "Use of the limited_api keyword is not supported with PyPy, since PyPy does not support compiling against the limited API yet. See pypy#1234 for more details."

If there's an automatic override, the user will get wheels tagged with abi3 that are not going to actually work across minor versions.

[mattip]

The override I was thinking of would force limited_abi to be false on PyPy, which would produce non-limited-api wheels AFAICT. An error would indeed be better. The relevant PyPy issue is https://github.com/pypy/pypy/issues/3397

[eli-schwartz]

The good news is that when originally designing the feature, I asked the author to include an override option. The intent was that some users, in particular those who are building locally, know that they won't be installing the same wheel on multiple python versions and are thus happy to compile against the faster ABI. However it also allows users who know they are on PyPy to observe that the build fails, and manually set that option to make the build succeed. No need to edit build files.

It's better than nothing and it works today, so PyPy users aren't left completely out in the cold.

What we should do is force that on PyPy as well. I don't think an error message is good news here. The keyword as it stands means that a module asserts "I support the limited API, so use it for me unless you think it's a bad idea for some reason".

Perhaps we could log a message saying that it's been automatically disabled due to missing support.

[rgommers]

The keyword as it stands means that a module asserts "I support the limited API, so use it for me unless you think it's a bad idea for some reason".

Those semantics do make sense to me. I'm worried about a bad interaction with the limited-api setting in meson-python though: https://meson-python.readthedocs.io/en/latest/reference/pyproject-settings.html?highlight=limited#cmdoption-arg-tool.meson-python.limited-api

It looks to me like that code will skip all checks for PyPy (because it's not possible, it generates the same filenames as for a regular build) and return abi3 for the ABI tag.

So perhaps the solution to that is for meson-python to unconditionally raise an exception if that flag is set to true with PyPy?

[eli-schwartz]

... shouldn't meson-python unconditionally set it to false rather than heuristically guessing that it works?

[rgommers]

Both are defensible I think. If the user knows that they're not using anything outside of the limited api, they may be able to produce wheels with the stable ABI tag that do work. If meson-python always errors out, they'd have to manually rename wheels with abi3.

Switching to false automatically seems worse than an error perhaps - the semantics are "produce a wheel targeting the stable ABI", not "try to target the stable ABI, and give me a minor version specific wheel otherwise".

I'm not sure what is best - it's really a PyPy issue, and between meson and meson-python we get to pick from several sub-optimal choices here.

[mattip]

There are a few problems:

• PyPy does not properly remove non-limited-API interfaces, it will expose way too much even when LIMITED_API is defined
• PyPy does not know how to load an abi3 shared object even if one is created. It needs a naming convention. The issue goes into more detail, but the major sticking point is that CPython abi3 shared objects are not portable between implementations. So PyPy would need something else.

Give that, I think the proper design decision for now is to either silently use the non-limited API even when the limited API is requested on PyPy (if the intent is "attempt to use the limited API"), or to raise an error (if the intent is "use the limited API").

In any case, having a py.implementation() attribute would allow users to avoid this. I would imagine it is more a problem for tests (like in NumPy) than an actual build problem, so they could skip testing the limited API for PyPy if they had a way to detect it.

[eli-schwartz]

It looks like more options than it actually is, due to all the moving parts. Actual options:

What to produce?

• a non abi3 wheel
• a broken wheel with invalid contents
• an abi3 wheel
• an error message

Option 2 is actually not an option, so call that 3 options.

Option 3 is the CPython happy path.

Options 1 and 4 are the PyPy choices. Which one is better?

• Option 1 is best if the user is locally building from source to install "a project", and treating wheels as a strange internal implementation quirk of python
• Option 4 is semantically useful, in the specific case of publishing a binary artifactory -- the number one example being https://pypi.org -- and only because degrading to building a non-abi3 wheel when you assumed adding limited_api would let you reduce your cibuildwheel matrix would result in a debugging session while you tried to figure out why you have a perfectly good wheel that works great, but only supports a fraction of the version matrix you wanted to support. In the end you still have to select option 1 explicitly.

It's not obvious to me that breaking by default and requiring users to pass -Csetup-args=... to their pip install invocations is worth it just to notify cibuildwheel users that they will have an incomplete set of wheels if they don't also pass it.

But this is why I suggested it would be better to warn than to error.