Unable to install latest version of `flake8` and `Sphinx` together

[rnag]

Describe the bug

I created an issue under flake8 for this here, but got shot down as soon as I opened it: PyCQA/flake8#1564

In any case, I copied the build output here:

$ make init
Collecting flake8==4.0.1
  Using cached flake8-4.0.1-py2.py3-none-any.whl (64 kB)
Collecting Sphinx==4.4.0
  Using cached Sphinx-4.4.0-py3-none-any.whl (3.1 MB)

ERROR: Cannot install -r requirements-dev.txt (line 5), -r requirements-dev.txt (line 6) and -r requirements-dev.txt (line 8) because these package versions have conflicting dependencies.

The conflict is caused by:
    flake8 4.0.1 depends on importlib-metadata<4.3; python_version < "3.8"
    tox 3.24.5 depends on importlib-metadata>=0.12; python_version < "3.8"
    sphinx 4.4.0 depends on importlib-metadata>=4.4; python_version < "3.10"

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
make: *** [Makefile:31: init] Error 

I feel like the blame is more on flake8 side, because they restrict to importlib-metadata<4.3 for Python 3.8 and below. I don't understand the rationale behind this, as it restricts using the latest available version, however I'm open to any suggestions or possible solutions for handling this. For clarification I use pyup to get updated version of new dependencies automatically, and I am running into tons of conflicts recently between latest versions of flake8 and Sphinx.

I think it could be resolved if either of them relaxed the requirements on importlib-metadata slightly:

importlib-metadata>=4.2; python_version < "3.8"

I'm of course definitely convinced that flake8 is not updating this requirement any time soon, so I guess i'm wondering at the next best steps for me. Ideally, how can I use pyup to ensure the dev dependencies for packages are at the latest versions, and still keep my sanity when it comes to "dependency hell" if you will, in such a scenario? I am open to any inputs or suggestions here.

How to Reproduce

On Python 3.7 or earlier, install latest versions of both:

$ pip install flake8==4.0.1 Sphinx==4.4.0

Expected behavior

No response

Your project

https://pyup.io/repos/github/rnag/dataclass-wizard/

Screenshots

No response

OS

Linux/Mac/Win 11

Python version

3.7.3

Sphinx version

4.4.0

Sphinx extensions

No response

Extra tools

No response

Additional context

No response

[tk0miya]

The dependency was added in #10007.

@takluyver Is there any reason to pin importlib-metadata-4.4.0 or newer? Is it possible to make it relaxed?

[takluyver]

importlib_metadata 4.4 is the backport of importlib.metadata from Python 3.10, and when I did that PR, I used some APIs new in 3.10 (because the alternatives were already deprecated). It might be possible to relax it a bit, but it's not clear to me exactly which versions would work.

I think that both Sphinx and flake8 are primarily applications rather than libraries, so they both get fairly free reign to choose their dependencies and not worry if that means you can't install them in the same environment. It's maybe less true for Sphinx with all its extensions, though.

[cblegare]

Adding on top of @takluyver comment, as a workaround, it might be possible for you @rnag to use both applications in separate virtualenvs. Tools like tox and nox make this quite efficient.