Should a local file always be re-installed on 'pip install' (with or without '--upgrade')?

[gaborbernat]

Without the resolver pip always reinstalled source distributions that it got as a direct command-line argument. Such is the case of tox, that builds the sdist, and then invokes pip to install it. In a pip with resolver world now, pip seems to no longer do this but rather first check the version number to see if it's not already installed:

  Getting requirements to build wheel ... done
    Created temporary directory: /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j
    Running command /tmp/magic/.tox/test/bin/python /tmp/magic/.tox/test/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/tmpqp55wn5c
    running dist_info
    creating /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info
    writing /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/dependency_links.txt
    writing entry points to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/entry_points.txt
    writing requirements to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/requires.txt
    writing top-level names to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/top_level.txt
    writing manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    writing manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    creating '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.dist-info'
    Preparing wheel metadata ... done
  Source in /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-build-h298ahlt has version 1.4.1, which satisfies requirement magic==1.4.1 from file:///tmp/magic/.tox/.tmp/package/1/magic-1.4.1.tar.gz
  Removed magic==1.4.1 from file:///tmp/magic/.tox/.tmp/package/1/magic-1.4.1.tar.gz from build tracker '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-tracker-5awzmelt'
Removed build tracker: '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-tracker-5awzmelt'

For local development projects, the version number though is highly untrustworthy, and the records/their contents are different even if the version number is the same. You generally only change versions at releases. We should either always re-install sdists/wheels passed in as paths or ensure that the content of the installed and passed are byte identical...

PS. tox/users could pass --force-reinstall to force the operation I guess, but this would be a breaking feature from POV of pip users.

[sbidoul]

This relates to #5780 and the upgrade strategies for direct URLs.

[uranusjr]

Agreed. More specifically, the old resolver uses different logic for PEP 508 and legacy requirements on whether it should re-install a URL requirement. The new one tries to be consistent, but that consistent logic needs refining. I think the conclusion in #5780 is to

• Always re-install if the URL changes (possible with PEP 610)
• Use the existing installation if the URL does not change, and the package version matches
• Re-install if the package version changes, even if the URL is the same

I’m not sure what parts of the URL PEP 610 records; if it contains the query string part, tools like tox should be able to force package re-installation with general cache-busting techniques.

[sbidoul]

And let's not forget editables, which are always re-installed :)

[gaborbernat]

I’m not sure what parts of the URL PEP 610 records; if it contains the query string part, tools like tox should be able to force package re-installation with general cache-busting techniques.

Can you detail how this would look?

Use the existing installation if the URL does not change, and the package version matches

Note this use falls under this section which is not a solution for this issue and users will be surprised by this change in behaviour.

[uranusjr]

The most widely-used generic cache busting approach simply appends a hash to the query string, so you’d do something like

pip install "package_name @ file://path/to/file.whl?tox=1234abcd"

where the tox= value is machine generated and changes every time (e.g. build timestamp, hash, or UUID). This would make the URL change, without actually changing the thing it points to.

[gaborbernat]

That sounds something a tool could do (though all tooling would need to change how to install packages), but not that useful for humans 🤔

[uranusjr]

My understanding is that pip doesn’t (never did?) position itself as a development tool, and is expected to be used in conjuction of other tooling to work well in such settings.

---

pip’s current behaviour (always re-installs bare URLs, but never re-installs PEP 508 URL specs) is entirely arbitrary without justification. My understanding is that the situation is caused by historic oversight, and should be classified as a bug. But it is also not obvious which behaviour is “corerct”, and there are two camps of people wanting the opposite approach. The resolution in #5780 is most reasonable middle ground (which actually makes sense!) that I know. I am fully aware that it would break someone’s workflow no matter what we do, but I still believe it’s something that should be done.

[gaborbernat]

How would maintaining status quo break peoples workflow? 😊

[pradyunsg]

always re-installs bare URLs

I've not read this issue yet, but this behaviour is definitely intentional: #4764 + #536

never re-installs PEP 508 URL specs

🤷