diff --git a/.vscode/cspell.json b/.vscode/cspell.json index d579a7e86e51..da07611bb847 100644 --- a/.vscode/cspell.json +++ b/.vscode/cspell.json @@ -416,6 +416,12 @@ "pytestmarkparametrize" ] }, + { + "filename": "doc/dev/pylint_checking.md", + "words": [ + "pylintrc" + ] + }, { "filename": "tools/azure-sdk-tools/devtools_testutils/proxy_startup.py", "words": [ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index eafc195fdce5..4132bf40c7f7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,13 +11,13 @@ If you want to contribute to a file that is generated (header contains `Code gen The Azure SDK team's Python CI leverages the tool `tox` to distribute tests to virtual environments, handle test dependency installation, and coordinate tooling reporting during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing. -[A Brief Overview of Tox](https://tox.readthedocs.io/en/latest/) +[A Brief Overview of Tox](https://tox.wiki/en/latest/) #### A Monorepo and Tox in Harmony Traditionally, the `tox.ini` file for a package sits _alongside the setup.py_ in source code. The `azure-sdk-for-python` necessarily does not adhere to this policy. There are over one-hundred packages contained here-in. That's a lot of `tox.ini` files to maintain! -Instead, the CI system leverages an tox plugin called `tox-monorepo`. This plugin allows `tox` to act as if the `tox.ini` is located in whatever directory you executed tox in! +Instead, the CI system leverages the `--root` argument which is new to `tox4`. The `--root` argument allows `tox` to act as if the `tox.ini` is located in whatever directory you specify! #### Tox Environments @@ -32,26 +32,50 @@ Internally `tox` leverages `virtualenv` to create each test environment's virtua This means that once the `tox` workflow is in place, all tests will be executed _within a virtual environment._ -To see the default environments from a specific `tox.ini` file, use the command `tox -l` in the same directory as the file itself. - -> sdk-for-python/eng/tox> tox -l - -``` - -whl -sdist - +You can use the command `tox list` to list all the environments provided by a `tox.ini` file. You can either use that command in the +same directory as the file itself, or use the `--conf` argument to specify the path to it directly. + + +Sample output of `tox list`: + +``` +sdk-for-python/eng/tox> tox list +default environments: +whl -> Builds a wheel and runs tests +sdist -> Builds a source distribution and runs tests + +additional environments: +pylint -> Lints a package with a pinned version of pylint +next-pylint -> Lints a package with pylint (version 2.15.8) +mypy -> Typechecks a package with mypy (version 1.0.0) +next-mypy -> Typechecks a package with the latest version of mypy +pyright -> Typechecks a package with pyright (version 1.1.287) +next-pyright -> Typechecks a package with the latest version of static type-checker pyright +verifytypes -> Verifies the "type completeness" of a package with pyright +whl_no_aio -> Builds a wheel without aio and runs tests +develop -> Tests a package +sphinx -> Builds a package's documentation with sphinx +depends -> Ensures all modules in a target package can be successfully imported +verifywhl -> Verify directories included in whl and contents in manifest file +verifysdist -> Verify directories included in sdist and contents in manifest file. Also ensures that py.typed configuration is correct within the setup.py +devtest -> Tests a package against dependencies installed from a dev index +latestdependency -> Tests a package against the released, upper-bound versions of its azure dependencies +mindependency -> Tests a package against the released, lower-bound versions of its azure dependencies +apistub -> Generate an api stub of a package ( for https://apiview.dev ) +bandit -> Runs bandit, a tool to find common security issues, against a package +samples -> Runs a package's samples +breaking -> Runs the breaking changes checker against a package ``` -Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `pylint` and `mypy` environments. - ### Example Usage of the common Azure SDK For Python `tox.ini` Basic usage of `tox` within this monorepo is: -1. `pip install tox<4 tox-monorepo` -2. `cd` to target package folder -3. run `tox -c path/to/tox.ini` +1. `pip install tox<5` +2. Run `tox run -e ENV_NAME -c path/to/tox.ini --root path/to/python_package` + * **Note**: You can use environment variables to provide defaults for tox config values + * With `TOX_CONFIG_FILE` set to the absolute path of `tox.ini`, you can avoid needing `-c path/to/tox.ini` in your tox invocations + * With `TOX_ROOT_DIR` set to the absolute path to your python package, you can avoid needing `--root path/to/python_package` The common `tox.ini` location is `eng/tox/tox.ini` within the repository. @@ -59,13 +83,11 @@ If at any time you want to blow away the tox created virtual environments and st #### Example `azure-core` mypy -1. `cd` to `sdk/core/azure-core` -2. Run `tox -e mypy -c ../../../eng/tox/tox.ini` +1. Run `tox run -e mypy -c ../../../eng/tox/tox.ini --root sdk/core/azure-core` #### Example `azure-storage-blob` tests -1. `cd` to `sdk/storage/azure-storage-blob` -2. Execute `tox -c ../../../eng/tox/tox.ini` +2. Execute `tox run -c ../../../eng/tox/tox.ini --root sdk/storage/azure-storage-blob` Note that we didn't provide an `environment` argument for this example. Reason here is that the _default_ environment selected by our common `tox.ini` file is one that runs `pytest`. @@ -75,7 +97,7 @@ Used for test execution across the spectrum of all the platforms we want to supp * Installs the wheel, runs tests using the wheel ``` -\> tox -e whl -c +\> tox run -e whl -c --root ``` @@ -87,7 +109,7 @@ Used for the local dev loop. ``` -\> tox -e sdist -c +\> tox run -e sdist -c --root ``` @@ -95,7 +117,7 @@ Used for the local dev loop. Pylint install and run. ``` -\> tox -e pylint -c +\> tox run -e pylint -c --root ``` @@ -103,14 +125,14 @@ Pylint install and run. Mypy install and run. ``` -\> tox -e mypy -c +\> tox run -e mypy -c --root ``` #### `sphinx` environment Generate sphinx doc for this package. ``` -\> tox -e sphinx -c +\> tox run -e sphinx -c --root ``` ### Custom Pytest Arguments @@ -120,7 +142,7 @@ Generate sphinx doc for this package. [Tox Documentation on Positional Arguments](https://tox.wiki/en/latest/config.html#substitutions-for-positional-arguments-in-commands) **Example: Invoke tox, breaking into the debugger on failure** -`tox -e whl -c ../../../eng/tox/tox.ini -- --pdb` +`tox run -e whl -c --root -- --pdb` ### Performance Testing @@ -150,7 +172,7 @@ a. cd to package root folder b. run tox environment devtest ``` -\> tox -e devtest -c +\> tox run -e devtest -c --root ``` This tox test( devtest) will fail if installed dependent packages are not dev build version. diff --git a/doc/dev/dev_setup.md b/doc/dev/dev_setup.md index 4f2171f49123..c324f23ec794 100644 --- a/doc/dev/dev_setup.md +++ b/doc/dev/dev_setup.md @@ -33,11 +33,11 @@ or execute the various commands available in the toolbox. 4. Setup your development environment - Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox], [Tox monorepo][tox_monorepo] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`: + Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`: ``` azure-sdk-for-python> cd sdk/formrecognizer/azure-ai-formrecognizer azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -r dev_requirements.txt - azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox tox-monorepo + azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox<5 azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -e . ``` @@ -58,6 +58,5 @@ Follow the instructions in the [Perform one-time test proxy setup][proxy_setup] [python_39]: https://www.microsoft.com/p/python-39/9p7qfqmjrfp7 [proxy_setup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#perform-one-time-test-proxy-setup [tests]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md -[tox]: https://tox.readthedocs.io/en/latest/ -[tox_monorepo]: https://pypi.org/project/tox-monorepo/ +[tox]: https://tox.wiki/en/latest/ [virtual_environment]: https://docs.python.org/3/tutorial/venv.html diff --git a/doc/dev/pylint_checking.md b/doc/dev/pylint_checking.md index ccc5494b59ed..8fa389cfb904 100644 --- a/doc/dev/pylint_checking.md +++ b/doc/dev/pylint_checking.md @@ -24,7 +24,7 @@ In the Azure SDK for Python repository, in addition to the standard pylint libra One way to run pylint is to run at the package level with tox: - .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox -e pylint -c ../../../eng/tox/tox.ini + .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox run -e pylint -c ../../../eng/tox/tox.ini --root . If you don't want to use tox, you can also install and run pylint on its own: @@ -58,9 +58,9 @@ In addition to being a part of the CI, the custom pylint checkers are also integ There is now a new step on the CI pipeline called `Run Pylint Next`. This is merely a duplicate of the `Run Pylint` step with the exception that `Run Pylint Next` uses the latest version of pylint and the latest version of the custom pylint checkers. -This next-pylint enviornment can also be run locally through tox: +This next-pylint environment can also be run locally through tox: - tox -e next-pylint -c ../../../eng/tox/tox.ini + tox run -e next-pylint -c ../../../eng/tox/tox.ini --root The errors generated by the `Run Pylint Next` step will not break your weekly test pipelines, but make sure to fix the warnings so that your client library is up to date for the next pylint release. diff --git a/doc/dev/sample_guide.md b/doc/dev/sample_guide.md index 056c02579ec8..0f3dc138b602 100644 --- a/doc/dev/sample_guide.md +++ b/doc/dev/sample_guide.md @@ -52,13 +52,13 @@ The given `START`/`END` keywords can be used in a [sphinx literalinclude][sphinx [Literalinclude example][literalinclude] The rendered code snippets are sensitive to the indentation in the sample file. Adjust the `dedent` accordingly to ensure the sample is captured accurately and not accidentally trimmed. -You can preview how published reference documentation will look by running [tox][tox]: `tox -e sphinx -c ../../../eng/tox/tox.ini`. +You can preview how published reference documentation will look by running [tox][tox]: `tox run -e sphinx -c ../../../eng/tox/tox.ini --root `. ## Test run samples in CI live tests Per the [Python guidelines][snippet_guidelines], sample code and snippets should be test run in CI to ensure they remain functional. Samples should be run in the package's live test pipeline which is scheduled to run daily. To ensure samples do get tested as part of regular CI runs, add these [lines][live_tests] to the package's tests.yml. -You can test this CI step locally first, by utilizing [tox][tox] and running `tox -e samples -c ../../../eng/tox/tox.ini` at the package-level. +You can test this CI step locally first, by utilizing [tox][tox] and running `tox run -e samples -c ../../../eng/tox/tox.ini --root `. The `Test Samples` step in CI will rely on the resources provisioned and environment variables used for running the package's tests. diff --git a/doc/dev/static_type_checking.md b/doc/dev/static_type_checking.md index 25a9c7a3a449..db76343d996d 100644 --- a/doc/dev/static_type_checking.md +++ b/doc/dev/static_type_checking.md @@ -175,9 +175,9 @@ version of the type checker ships. All client libraries in the Python SDK repo a The easiest way to install and run the type checkers locally is with [tox](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#tox). This reproduces the exact type checking -environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox` and `tox-monorepo`: +environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox`: -`pip install tox tox-monorepo` +`pip install tox<5` ### Run mypy @@ -185,7 +185,7 @@ mypy is currently pinned to version [1.0.0](https://pypi.org/project/mypy/1.0.0/ To run mypy on your library, run the tox mypy env at the package level: -`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e mypy -c ../../../eng/tox/tox.ini` +`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e mypy -c ../../../eng/tox/tox.ini --root .` If you don't want to use `tox` you can also install and run mypy on its own: @@ -217,7 +217,7 @@ Note that pyright requires that node is installed. The command-line [wrapper pac To run pyright on your library, run the tox pyright env at the package level: -`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e pyright -c ../../../eng/tox/tox.ini` +`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e pyright -c ../../../eng/tox/tox.ini --root .` If you don't want to use `tox` you can also install and run pyright on its own: @@ -249,7 +249,7 @@ from the code in the PR vs. the latest release on PyPi. To run verifytypes on your library, run the tox verifytypes env at the package level: -`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e verifytypes -c ../../../eng/tox/tox.ini` +`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e verifytypes -c ../../../eng/tox/tox.ini --root .` If you don't want to use `tox` you can also install and run pyright/verifytypes on its own: diff --git a/doc/dev/tests.md b/doc/dev/tests.md index 5f22aa6581af..e3fd49ea03be 100644 --- a/doc/dev/tests.md +++ b/doc/dev/tests.md @@ -123,20 +123,20 @@ If you have print statements in your tests for debugging you can add the `-s` fl ## Tox -The Python SDK uses the [tox project](https://tox.readthedocs.io/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment. -`(env) > pip install tox tox-monorepo`. +The Python SDK uses the [tox project](https://tox.wiki/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment. +`(env) > pip install tox<5`. To run a tox command from your directory use the following commands: ```cmd -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sphinx -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pylint -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e mypy -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pyright -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e verifytypes -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e whl -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sdist -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e samples -(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e apistub +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sphinx -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pylint -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e mypy -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pyright -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e verifytypes -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e whl -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sdist -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e samples -c ../../../eng/tox/tox.ini --root . +(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e apistub -c ../../../eng/tox/tox.ini --root . ``` A quick description of the five commands above: * sphinx: documentation generation using the inline comments written in our code diff --git a/doc/eng_sys_checks.md b/doc/eng_sys_checks.md index c22c12e4d76c..f2f892c65114 100644 --- a/doc/eng_sys_checks.md +++ b/doc/eng_sys_checks.md @@ -169,14 +169,14 @@ Analyze job in both nightly CI and pull request validation pipeline runs a set o [`MyPy`](https://pypi.org/project/mypy/) is a static analysis tool that runs type checking of python package. Following are the steps to run `MyPy` locally for a specific package: 1. Go to root of the package -2. Execute following command: `tox -e mypy -c ../../../eng/tox/tox.ini` +2. Execute following command: `tox run -e mypy -c ../../../eng/tox/tox.ini --root .` ### Pyright [`Pyright`](https://github.com/microsoft/pyright/) is a static analysis tool that runs type checking of python package. Following are the steps to run `pyright` locally for a specific package: 1. Go to root of the package -2. Execute following command: `tox -e pyright -c ../../../eng/tox/tox.ini` +2. Execute following command: `tox run -e pyright -c ../../../eng/tox/tox.ini --root .` ### Verifytypes @@ -184,7 +184,7 @@ Analyze job in both nightly CI and pull request validation pipeline runs a set o [`Verifytypes`](https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#verifying-type-completeness) is a feature of pyright that checks the type completeness of a python package. Following are the steps to run `verifytypes` locally for a specific package: 1. Go to root of the package -2. Execute following command: `tox -e verifytypes -c ../../../eng/tox/tox.ini` +2. Execute following command: `tox run -e verifytypes -c ../../../eng/tox/tox.ini --root .` ### Pylint @@ -192,14 +192,14 @@ Analyze job in both nightly CI and pull request validation pipeline runs a set o [`Pylint`](https://pypi.org/project/pylint/) is a static analysis tool to run lint checking, it is automatically run on all PRs. Following are the steps to run `pylint` locally for a specific package. 1. Go to root of the package. -2. Execute following command: `tox -e pylint -c ../../../eng/tox/tox.ini` +2. Execute following command: `tox run -e pylint -c ../../../eng/tox/tox.ini --root .` ### Bandit `Bandit` is static security analysis tool. This check is triggered for all Azure SDK package as part of analyze job. Following are the steps to `Bandit` tool locally for a specific package. 1. Got to package root directory. -2. Execute command: `tox -e bandit -c ../../../eng/tox/tox.ini` +2. Execute command: `tox run -e bandit -c ../../../eng/tox/tox.ini --root .` ### ApiStubGen @@ -255,7 +255,7 @@ This test installs wheel of the package being tested and runs all tests cases in 1. Go to package root folder on a command line 2. Run following command - ``tox -e whl -c ../../../eng/tox/tox.ini`` + `tox run -e whl -c ../../../eng/tox/tox.ini --root .` #### sdist @@ -263,7 +263,7 @@ This test installs sdist of the package being tested and runs all tests cases in 1. Go to package root folder on a command line 2. Run following command - ``tox -e sdist -c ../../../eng/tox/tox.ini`` + `tox run -e sdist -c ../../../eng/tox/tox.ini --root .` #### depends @@ -273,7 +273,7 @@ Following is the command to run this test environment locally. 1. Go to package root folder on a command line 2. Run following command - ``tox -e sdist -c ../../../eng/tox/tox.ini`` + `tox run -e sdist -c ../../../eng/tox/tox.ini --root .` ## Nightly CI Checks @@ -321,7 +321,7 @@ Note: Any dependency mentioned only in dev_requirements are not considered to id Tox name of this test is `latestdependency` and steps to manually run this test locally is as follows. 1. Go to package root. For e.g azure-storage-blob or azure-identity -2. Run command `Tox –e latestdependency –c ../../../tox/tox.ini` +2. Run command `tox run -e latestdependency -c ../../../tox/tox.ini --root .` #### Minimum Dependency Test @@ -338,7 +338,7 @@ Tox name of this test is `mindependency` and steps to manually run this test loc 1. Go to package root. For e.g azure-storage-blob or azure-identity 2. Run following command -`Tox –e mindependency –c ../../../tox/tox.ini` +`tox run -e mindependency -c ../../../tox/tox.ini --root .` #### Regression Test diff --git a/eng/ci_tools.txt b/eng/ci_tools.txt index 641ce899429c..b3348ed08ff7 100644 --- a/eng/ci_tools.txt +++ b/eng/ci_tools.txt @@ -1,28 +1,20 @@ # requirements leveraged by ci tools cryptography<4 -setuptools==44.1.0; python_version == '2.7' setuptools==67.6.0; python_version >= '3.5' -virtualenv==20.13.2 +virtualenv==20.23.0 wheel==0.37.0 Jinja2==2.11.2 -packaging==20.4 -tox==3.25.1 -tox-monorepo==0.1.2 +packaging==23.1 +tox==4.5.0 twine==3.1.1; python_version >= '3.6' pathlib2==2.3.5 doc-warden==0.7.2 -# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed -coverage==4.5.4 beautifulsoup4==4.9.1 pkginfo==1.5.0.1 pip==20.3.3 wrapt==1.12.1; python_version <= '3.10' wrapt==1.14.1; python_version >= '3.11' -markupsafe==2.0.1; python_version > '2.7' -markupsafe==1.1.1; python_version == '2.7' - -# python 3.10 pinned packages -cffi==1.15.0rc2; python_version >= '3.10' +markupsafe==2.0.1 # locking packages defined as deps from azure-sdk-tools or azure-devtools pytoml==0.1.21 @@ -31,10 +23,9 @@ json-delta==2.0 ConfigArgParse==1.2.3 six==1.14.0 pyyaml==5.3.1 -pytest==4.6.9; python_version == '2.7' -pytest==5.4.2; python_version >= '3.5' and python_version <= '3.9' -pytest==6.2.4; python_version >= '3.10' -pytest-cov==2.8.1 +pytest==7.3.1 +pytest-cov==4.0.0 +coverage==7.2.5 # local dev packages ./tools/azure-devtools diff --git a/eng/pipelines/templates/jobs/tests-nightly-python.yml b/eng/pipelines/templates/jobs/tests-nightly-python.yml index 1f61661382a8..b3e5663c0b1a 100644 --- a/eng/pipelines/templates/jobs/tests-nightly-python.yml +++ b/eng/pipelines/templates/jobs/tests-nightly-python.yml @@ -22,7 +22,7 @@ jobs: - script: | python -m pip freeze python -m pip --version - python -m pip install setuptools==58.3.0 wheel==0.37.0 tox==3.24.3 tox-monorepo==0.1.2 packaging==21.0 requests + python -m pip install setuptools==58.3.0 wheel==0.37.0 tox==4.5.0 packaging==23.1 requests python -m pip install $(Build.SourcesDirectory)/tools/azure-sdk-tools[build] displayName: Install Dependencies @@ -67,7 +67,7 @@ jobs: sudo apt-get install build-essential -y python -m pip freeze python -m pip --version - python -m pip install setuptools==58.3.0 wheel==0.37.0 tox==3.24.3 tox-monorepo==0.1.2 packaging==21.0 requests + python -m pip install setuptools==58.3.0 wheel==0.37.0 tox==4.5.0 packaging==23.1 requests python -m pip install $(Build.SourcesDirectory)/tools/azure-sdk-tools[build] displayName: Install Dependencies @@ -131,7 +131,7 @@ jobs: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py python3 -m pip install setuptools==67.6.0 wheel - python3 -m pip install tox tox-monorepo packaging twine beautifulsoup4 + python3 -m pip install tox packaging twine beautifulsoup4 python3 --version cd $(Build.SourcesDirectory) python3 ./scripts/devops_tasks/setup_execute_tests.py "$(BuildTargetingString)" --junitxml="junit/test_results_38.xml" --toxenv="whl" --filter-type="None" diff --git a/eng/pipelines/templates/jobs/update_pr.yml b/eng/pipelines/templates/jobs/update_pr.yml index 6e459bdfc86a..7515af893551 100644 --- a/eng/pipelines/templates/jobs/update_pr.yml +++ b/eng/pipelines/templates/jobs/update_pr.yml @@ -23,7 +23,7 @@ steps: python3 -m pip install -e "git+https://github.com/Azure/azure-sdk-for-python#subdirectory=tools/azure-sdk-tools&egg=azure-sdk-tools" displayName: 'Install Azure SDK tools' -- script: python3 -m packaging_tools.update_pr -v --pr-number $(System.PullRequest.PullRequestNumber) --repo $(Build.Repository.Name) +- script: python3 -m packaging_tools.update_pr -v --pr-number $(System.PullRequest.PullRequestNumber) --repo $(Build.Repository.Name) displayName: 'Update packaging of PR' env: GH_TOKEN: $(azuresdk-github-pat) diff --git a/eng/pipelines/templates/steps/build-test.yml b/eng/pipelines/templates/steps/build-test.yml index 9cadcb2cbdc6..9195c6126771 100644 --- a/eng/pipelines/templates/steps/build-test.yml +++ b/eng/pipelines/templates/steps/build-test.yml @@ -89,7 +89,7 @@ steps: CustomCondition: '${{ parameters.RunCoverage }}' - pwsh: | - python -m pip install coverage==4.5.4 + python -m pip install coverage==7.2.5 python scripts/devops_tasks/create_coverage.py displayName: Create Coverage Report condition: and(succeeded(), ${{ parameters.RunCoverage }}) diff --git a/eng/pipelines/templates/steps/use-python-version.yml b/eng/pipelines/templates/steps/use-python-version.yml index 9520e4f1a8f0..1a2fe4130e0b 100644 --- a/eng/pipelines/templates/steps/use-python-version.yml +++ b/eng/pipelines/templates/steps/use-python-version.yml @@ -9,7 +9,7 @@ steps: versionSpec: 3.8 - pwsh: | - python -m pip install packaging==20.4 + python -m pip install packaging==23.1 displayName: Prep Environment # select the appropriate version from manifest if present diff --git a/eng/regression_test_tools.txt b/eng/regression_test_tools.txt index 4cf5407b65d7..8a11aca7c7fa 100644 --- a/eng/regression_test_tools.txt +++ b/eng/regression_test_tools.txt @@ -1,15 +1,17 @@ pip==20.2 +cryptography<4 # requirements leveraged by ci for testing -pytest==4.6.9; python_version == '2.7' -pytest==5.4.2; python_version >= '3.5' -pytest-asyncio==0.12.0; python_version >= '3.5' -pytest-cov==2.8.1 +pytest==7.3.1 +pytest-asyncio==0.21.0 +pytest-cov==4.0.0 pytest-custom-exit-code==0.3.0 -pytest-xdist==1.32.0 -# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed -coverage==4.5.4 +pytest-xdist==3.2.1 +coverage==7.2.5 bandit==1.6.2 +protobuf==3.17.3; python_version == '2.7' +chardet>=2.0,<5.0 +cmarkgfm<0.7.0 # locking packages defined as deps from azure-sdk-tools or azure-devtools pytoml==0.1.21 @@ -19,15 +21,17 @@ json-delta==2.0 ConfigArgParse==1.2.3 six==1.14.0 pyyaml==5.3.1 -packaging==20.4 -wheel==0.34.2 +packaging==23.1 Jinja2==2.11.2 markupsafe==2.0.1; python_version > '2.7' markupsafe==1.1.1; python_version == '2.7' +wrapt==1.12.1; python_version <= '3.10' +wrapt==1.14.1; python_version >= '3.11' # Locking pylint and required packages pylint==1.8.4; python_version < '3.4' -pylint==2.5.2; python_version >= '3.4' +pylint==2.9.3; python_version >= '3.6' and python_version <= '3.10' +pylint==2.14.5; python_version >= '3.11' # python-dotenv python-dotenv==0.15.0 \ No newline at end of file diff --git a/eng/regression_tools.txt b/eng/regression_tools.txt index 6d60efad586e..c4d7965ea55e 100644 --- a/eng/regression_tools.txt +++ b/eng/regression_tools.txt @@ -1,26 +1,20 @@ # requirements leveraged by ci tools cryptography==3.3.2 -setuptools==44.1.0; python_version == '2.7' setuptools==67.6.0; python_version >= '3.5' -virtualenv==20.0.23 -wheel==0.34.2 +virtualenv==20.23.0 +wheel==0.37.0 Jinja2==2.11.2 -packaging==20.4 -tox==3.25.1 -tox-monorepo==0.1.2 -twine==1.15.0; python_version == '2.7' or python_version == '3.5' +packaging==23.1 +tox==4.5.0 twine==3.1.1; python_version >= '3.6' pathlib2==2.3.5 -readme-renderer[md]==25.0 -doc-warden==0.7.1 -# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed -coverage==4.5.4 +doc-warden==0.7.2 beautifulsoup4==4.9.1 pkginfo==1.5.0.1 -pip==20.2 -black==21.6b0; python_version >= '3.6' -markupsafe==2.0.1; python_version > '2.7' -markupsafe==1.1.1; python_version == '2.7' +pip==20.3.3 +wrapt==1.12.1; python_version <= '3.10' +wrapt==1.14.1; python_version >= '3.11' +markupsafe==2.0.1 # locking packages defined as deps from azure-sdk-tools or azure-devtools pytoml==0.1.21 @@ -29,10 +23,10 @@ json-delta==2.0 ConfigArgParse==1.2.3 six==1.14.0 pyyaml==5.3.1 -pytest==5.4.2; python_version >= '3.5' -pytest==4.6.9; python_version == '2.7' -pytest-cov==2.8.1 +pytest==7.3.1 +pytest-cov==4.0.0 +coverage==7.2.5 # local dev packages ./tools/azure-devtools -./tools/azure-sdk-tools +./tools/azure-sdk-tools[build] diff --git a/eng/test_tools.txt b/eng/test_tools.txt index 0183a41989f0..ab47e7beb313 100644 --- a/eng/test_tools.txt +++ b/eng/test_tools.txt @@ -1,15 +1,12 @@ cryptography<4 # requirements leveraged by ci for testing -pytest==4.6.9; python_version == '2.7' -pytest==5.4.2; python_version >= '3.5' and python_version <= '3.9' -pytest==6.2.4; python_version >= '3.10' -pytest-asyncio==0.12.0; python_version >= '3.5' -pytest-cov==2.8.1 +pytest==7.3.1 +pytest-asyncio==0.21.0 +pytest-cov==4.0.0 pytest-custom-exit-code==0.3.0 -pytest-xdist==1.32.0 -# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed -coverage==4.5.4 +pytest-xdist==3.2.1 +coverage==7.2.5 bandit==1.6.2 protobuf==3.17.3; python_version == '2.7' chardet>=2.0,<5.0 @@ -23,7 +20,7 @@ json-delta==2.0 ConfigArgParse==1.2.3 six==1.14.0 pyyaml==5.3.1 -packaging==20.4 +packaging==23.1 Jinja2==2.11.2 markupsafe==2.0.1; python_version > '2.7' markupsafe==1.1.1; python_version == '2.7' diff --git a/eng/tox/tox.ini b/eng/tox/tox.ini index e72a2d8a880b..aa1b26718f2b 100644 --- a/eng/tox/tox.ini +++ b/eng/tox/tox.ini @@ -6,13 +6,18 @@ # In all cases, whenever we install an azure-* package from a requirement (read: not a specific file), the only source will be from the dev feed. # once we've downloaded these dependencies, only then do we install other packages from pypi. [tox] +requires= + # Ensure that we're running a version of tox compatible with this config + # 4.4.10 is the lower bound because it incorporates a fix for a bug with `--root`: + # https://github.com/tox-dev/tox/pull/2962 + tox>=4.4.10 # note that this envlist is the default set of environments that will run if a target environment is not selected. envlist = whl,sdist [tools] deps = - -r ../../../eng/test_tools.txt + -r {repository_root}/eng/test_tools.txt [coverage:paths] @@ -31,27 +36,25 @@ deps = [dependencytools] deps = - -r ../../../eng/dependency_tools.txt + -r {repository_root}/eng/dependency_tools.txt [packaging] pkgs = - wheel==0.37.0 - packaging==20.4 - urllib3==1.26.12 + wheel==0.40.0 + packaging==23.1 + urllib3==1.26.15 tomli==2.0.1 +[pytest] +ignore_args=--ignore=.tox --ignore=build --ignore=.eggs +default_args = --junitxml={tox_root}/test-junit-{envname}.xml --verbose --durations=10 --ignore=azure {[pytest]ignore_args} [testenv] -ignore_args=--ignore=.tox --ignore=build --ignore=.eggs -default_pytest_params = --junitxml={toxinidir}/test-junit-{envname}.xml --verbose --durations=10 --ignore=azure {[testenv]ignore_args} parallel_show_output =True skip_install = true skipsdist = true usedevelop = false -platform = linux: linux - macos: darwin - windows: win32 passenv = * download=true requires= @@ -64,85 +67,83 @@ setenv = VIRTUALENV_PIP=20.3.3 VIRTUALENV_SETUPTOOLS=67.6.0 deps = {[base]deps} -changedir = {toxinidir} -install_command = python -m pip install {opts} {packages} --cache-dir {toxinidir}/../.tox_pip_cache_{envname} --ignore-installed +install_command = python -m pip install {opts} {packages} --cache-dir {tox_root}/../.tox_pip_cache_{envname} commands = - {envbindir}/python -m pip --version - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py -d {envtmpdir} -p {toxinidir} -w {envtmpdir} - {envbindir}/python -m pip freeze - pytest \ - {[testenv]default_pytest_params} \ - {posargs} \ - {toxinidir} + python -m pip --version + python {repository_root}/eng/tox/create_package_and_install.py -d {envtmpdir} -p {tox_root} -w {envtmpdir} + python -m pip freeze + pytest {[pytest]default_args} {posargs} {tox_root} [testenv:pylint] +description=Lints a package with a pinned version of pylint skipsdist = true skip_install = true usedevelop = false -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5002 deps = {[base]deps} commands = - {envbindir}/python -m pip install azure-pylint-guidelines-checker==0.0.5 --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python -m pip install azure-pylint-guidelines-checker==0.0.5 --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/run_pylint.py -t {toxinidir} + python {repository_root}/eng/tox/run_pylint.py -t {tox_root} [testenv:next-pylint] +description=Lints a package with pylint (version {[testenv:next-pylint]pylint_version}) +pylint_version=2.15.8 skipsdist = true skip_install = true usedevelop = false -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5002 deps = {[base]deps} commands = - {envbindir}/python -m pip install pylint==2.15.8 - {envbindir}/python -m pip install azure-pylint-guidelines-checker==0.0.8 --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python -m pip install {[testenv:next-pylint]pylint_version} + python -m pip install azure-pylint-guidelines-checker==0.0.8 --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/run_pylint.py -t {toxinidir} --next=True + python {repository_root}/eng/tox/run_pylint.py -t {tox_root} --next=True [testenv:mypy] +description=Typechecks a package with mypy (version {[testenv:mypy]mypy_version}) +mypy_version=1.0.0 skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5003 deps = {[base]deps} - mypy==1.0.0 + mypy=={[testenv:mypy]mypy_version} types-chardet==4.0.3 types-requests==2.27.9 types-six==1.16.10 types-redis==4.3.21.2 commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/run_mypy.py -t {toxinidir} + python {repository_root}/eng/tox/run_mypy.py -t {tox_root} [testenv:next-mypy] +description=Typechecks a package with the latest version of mypy skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5020 @@ -154,41 +155,41 @@ deps = types-six types-redis commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/run_mypy.py -t {toxinidir} --next=True + python {repository_root}/eng/tox/run_mypy.py -t {tox_root} --next=True [testenv:pyright] +description=Typechecks a package with pyright (version {[testenv:pyright]pyright_version}) +pyright_version=1.1.287 skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5018 deps = {[base]deps} - pyright==1.1.287 + pyright=={[testenv:pyright]pyright_version} commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/create_dependencies_and_install.py \ - -p {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/run_pyright.py -t {toxinidir} + python {repository_root}/eng/tox/create_dependencies_and_install.py -p {tox_root} + python {repository_root}/eng/tox/run_pyright.py -t {tox_root} [testenv:next-pyright] +description=Typechecks a package with the latest version of static type-checker pyright skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5021 @@ -196,21 +197,20 @@ deps = {[base]deps} pyright commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/create_dependencies_and_install.py \ - -p {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/run_pyright.py -t {toxinidir} --next=True + python {repository_root}/eng/tox/create_dependencies_and_install.py -p {tox_root} + python {repository_root}/eng/tox/run_pyright.py -t {tox_root} --next=True [testenv:verifytypes] +description=Verifies the "type completeness" of a package with pyright skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5019 @@ -218,81 +218,72 @@ deps = {[base]deps} pyright==1.1.287 commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/create_dependencies_and_install.py \ - -p {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/run_verifytypes.py -t {toxinidir} + python {repository_root}/eng/tox/create_dependencies_and_install.py -p {tox_root} + python {repository_root}/eng/tox/run_verifytypes.py -t {tox_root} [testenv:whl_no_aio] +description=Builds a wheel without aio and runs tests skipsdist = true skip_install = true -changedir = {toxinidir} deps = {[base]deps} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5004 commands = - - {envbindir}/python -m pip uninstall aiohttp --yes - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py -d {envtmpdir} -p {toxinidir} -w {envtmpdir} - {envbindir}/python {toxinidir}/../../../eng/tox/try_import.py aiohttp -p {toxinidir} - {envbindir}/python -m pip freeze - pytest \ - {[testenv]default_pytest_params} \ - --ignore-glob='*async*.py' \ - {posargs} \ - {toxinidir} + - python -m pip uninstall aiohttp --yes + python {repository_root}/eng/tox/create_package_and_install.py -d {envtmpdir} -p {tox_root} -w {envtmpdir} + python {repository_root}/eng/tox/try_import.py aiohttp -p {tox_root} + python -m pip freeze + pytest {[pytest]default_args} --ignore-glob='*async*.py' {posargs} --no-cov {tox_root} + +[testenv:whl] +description=Builds a wheel and runs tests [testenv:sdist] +description=Builds a source distribution and runs tests skipsdist = true skip_install = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5005 deps = {[base]deps} commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ + python {repository_root}/eng/tox/create_package_and_install.py \ -d {envtmpdir} \ - -p {toxinidir} \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python -m pip freeze - pytest \ - {posargs} \ - --no-cov \ - {[testenv]ignore_args} \ - {toxinidir} + python -m pip freeze + pytest {posargs} --no-cov {[pytest]ignore_args} {tox_root} [testenv:develop] +description=Tests a package skipsdist = false skip_install = false usedevelop = true -changedir = {toxinidir} deps = {[base]deps} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5006 commands = - pytest \ - {posargs} \ - --ignore=.tox \ - {toxinidir} + pytest {posargs} --ignore=.tox {tox_root} [testenv:sphinx] +description=Builds a package's documentation with sphinx skipsdist = true skip_install = true -changedir = {toxinidir} passenv = * setenv = {[testenv]setenv} @@ -305,27 +296,21 @@ deps = mistune<2.0.0 m2r==0.2.1 commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../eng/tox/prep_sphinx_env.py -d {distdir} -t {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/run_sphinx_apidoc.py \ - -w {distdir} \ - -r {toxinidir} - - {envbindir}/python {toxinidir}/../../../eng/tox/run_sphinx_build.py \ - -w {distdir}/unzipped/docgen \ - -o {distdir}/site \ - -r {toxinidir} + python {repository_root}/eng/tox/prep_sphinx_env.py -d {envtmpdir}/dist -t {tox_root} + python {repository_root}/eng/tox/run_sphinx_apidoc.py -w {envtmpdir}/dist -r {tox_root} + python {repository_root}/eng/tox/run_sphinx_build.py \ + -w {envtmpdir}/dist/unzipped/docgen \ + -o {envtmpdir}/dist/site \ + -r {tox_root} [testenv:depends] -platform = linux: linux - macos: darwin - windows: win32 -changedir = {toxinidir} +description = Ensures all modules in a target package can be successfully imported setenv = {[testenv]setenv} PROXY_URL=http://localhost:5008 @@ -333,78 +318,73 @@ deps = {[packaging]pkgs} cryptography<4 commands = - {envbindir}/python -m pip install {toxinidir}/../../../tools/azure-sdk-tools --no-deps - {envbindir}/python {toxinidir}/../../../eng/tox/sanitize_setup.py -t {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ + python -m pip install {repository_root}/tools/azure-sdk-tools --no-deps + python {repository_root}/eng/tox/sanitize_setup.py -t {tox_root} + python {repository_root}/eng/tox/create_package_and_install.py \ -d {envtmpdir} \ - -p {toxinidir} \ + -p {tox_root} \ -w {envtmpdir} \ --force-create true - {envbindir}/python -m pip freeze - {envbindir}/python {toxinidir}/../../../eng/tox/import_all.py -t {toxinidir} + python -m pip freeze + python {repository_root}/eng/tox/import_all.py -t {tox_root} [testenv:verifywhl] +description=Verify directories included in whl and contents in manifest file skipsdist = true skip_install = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5009 deps = {[packaging]pkgs} commands = - {envbindir}/python -m pip install {toxinidir}/../../../tools/azure-sdk-tools --no-deps - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py -d {envtmpdir} -p {toxinidir} --skip-install True - {envbindir}/python {toxinidir}/../../../eng/tox/verify_whl.py -d {envtmpdir} -t {toxinidir} + python -m pip install {repository_root}/tools/azure-sdk-tools --no-deps + python {repository_root}/eng/tox/create_package_and_install.py -d {envtmpdir} -p {tox_root} --skip-install True + python {repository_root}/eng/tox/verify_whl.py -d {envtmpdir} -t {tox_root} [testenv:verifysdist] +description=Verify directories included in sdist and contents in manifest file. Also ensures that py.typed configuration is correct within the setup.py skipsdist = true skip_install = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5010 deps = {[packaging]pkgs} commands = - {envbindir}/python -m pip install {toxinidir}/../../../tools/azure-sdk-tools --no-deps - {envbindir}/python {toxinidir}/setup.py --q sdist --format zip -d {envtmpdir} - {envbindir}/python {toxinidir}/../../../eng/tox/verify_sdist.py -d {envtmpdir} -t {toxinidir} + python -m pip install {repository_root}/tools/azure-sdk-tools --no-deps + python {tox_root}/setup.py --q sdist --format zip -d {envtmpdir} + python {repository_root}/eng/tox/verify_sdist.py -d {envtmpdir} -t {tox_root} [testenv:devtest] +description=Tests a package against dependencies installed from a dev index deps = {[base]deps} -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5011 commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py -d {envtmpdir} -p {toxinidir} - {envbindir}/python {toxinidir}/../../../eng/tox/install_dev_build_dependency.py -t {toxinidir} - pytest \ - {[testenv]default_pytest_params} \ - --ignore=.tox \ - {posargs} \ - {toxinidir} + python {repository_root}/eng/tox/create_package_and_install.py -d {envtmpdir} -p {tox_root} + python {repository_root}/eng/tox/install_dev_build_dependency.py -t {tox_root} + pytest {[pytest]default_args} --ignore=.tox {posargs} {tox_root} [deptestcommands] commands = - {envbindir}/python {toxinidir}/../../../eng/tox/install_depend_packages.py -t {toxinidir} -d {env:DEPENDENCY_TYPE:} -w {envtmpdir} - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py -d {envtmpdir} -p {toxinidir} -w {envtmpdir} --pre-download-disabled - {envbindir}/python -m pip freeze - {envbindir}/python {toxinidir}/../../../eng/tox/verify_installed_packages.py --packages-file {envtmpdir}/packages.txt - pytest {[testenv]default_pytest_params} {posargs} --no-cov {toxinidir} + python {repository_root}/eng/tox/install_depend_packages.py -t {tox_root} -d {env:DEPENDENCY_TYPE:} -w {envtmpdir} + python {repository_root}/eng/tox/create_package_and_install.py -d {envtmpdir} -p {tox_root} -w {envtmpdir} --pre-download-disabled + python -m pip freeze + python {repository_root}/eng/tox/verify_installed_packages.py --packages-file {envtmpdir}/packages.txt + pytest {[pytest]default_args} {posargs} --no-cov {tox_root} [testenv:latestdependency] +description=Tests a package against the released, upper-bound versions of its azure dependencies deps = {[dependencytools]deps} {[tools]deps} -changedir = - {toxinidir} passenv = * setenv = {[testenv]setenv} @@ -415,13 +395,13 @@ commands = [testenv:mindependency] +description=Tests a package against the released, lower-bound versions of its azure dependencies deps = azure-mgmt-keyvault<7.0.0 azure-mgmt-resource<15.0.0 azure-mgmt-storage<15.0.0 {[dependencytools]deps} {[tools]deps} -changedir = {toxinidir} passenv = * setenv = {[testenv]setenv} @@ -432,6 +412,7 @@ commands = [testenv:apistub] +description=Generate an api stub of a package ( for https://apiview.dev ) skipsdist = true skip_install = true usedevelop = false @@ -443,12 +424,13 @@ deps = {[base]deps} commands = # install API stub generator - {envbindir}/python -m pip install -r {toxinidir}/../../../eng/apiview_reqs.txt --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" - {envbindir}/python -m pip freeze - {envbindir}/python {toxinidir}/../../../eng/tox/run_apistubgen.py -t {toxinidir} -w {envtmpdir} {posargs} + python -m pip install -r {repository_root}/eng/apiview_reqs.txt --index-url="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" + python -m pip freeze + python {repository_root}/eng/tox/run_apistubgen.py -t {tox_root} -w {envtmpdir} {posargs} [testenv:bandit] +description=Runs bandit, a tool to find common security issues, against a package skipsdist = true skip_install = true usedevelop = false @@ -460,16 +442,17 @@ deps = {[base]deps} importlib-metadata<5.0 commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ + python {repository_root}/eng/tox/create_package_and_install.py \ -d {envtmpdir} \ - -p {toxinidir} \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python -m pip freeze - {envbindir}/python {toxinidir}/../../../eng/tox/run_bandit.py -t {toxinidir} + python -m pip freeze + python {repository_root}/eng/tox/run_bandit.py -t {tox_root} [testenv:samples] +description=Runs a package's samples skipsdist = false skip_install = false usedevelop = false @@ -481,26 +464,26 @@ deps = {[base]deps} subprocess32; python_version < '3.5' commands = - {envbindir}/python -m pip freeze - {envbindir}/python {toxinidir}/../../../scripts/devops_tasks/test_run_samples.py -t {toxinidir} + python -m pip freeze + python {repository_root}/scripts/devops_tasks/test_run_samples.py -t {tox_root} [testenv:breaking] +description=Runs the breaking changes checker against a package skipsdist = true skip_install = true usedevelop = true -changedir = {toxinidir} setenv = {[testenv]setenv} PROXY_URL=http://localhost:5017 deps = {[base]deps} jsondiff==1.2.0 - -e {toxinidir}/../../scripts/breaking_changes_checker + -e {repository_root}/scripts/breaking_changes_checker commands = - {envbindir}/python {toxinidir}/../../../eng/tox/create_package_and_install.py \ - -d {distdir} \ - -p {toxinidir} \ + python {repository_root}/eng/tox/create_package_and_install.py \ + -d {envtmpdir}/dist \ + -p {tox_root} \ -w {envtmpdir} \ --package-type sdist - {envbindir}/python {toxinidir}/../../../scripts/breaking_changes_checker/detect_breaking_changes.py -t {toxinidir} + python {repository_root}/scripts/breaking_changes_checker/detect_breaking_changes.py -t {tox_root} diff --git a/eng/tox/toxfile.py b/eng/tox/toxfile.py new file mode 100644 index 000000000000..6fbe64a8d1cf --- /dev/null +++ b/eng/tox/toxfile.py @@ -0,0 +1,19 @@ +from logging import getLogger +from pathlib import Path + +from tox import plugin +from tox.config.sets import CoreConfigSet +from tox.session.state import State + +logger = getLogger(__file__) + +NAME = "azure-sdk-for-python tox plugin" + +@plugin.impl +def tox_add_core_config(core_conf: CoreConfigSet, state: State): + core_conf.add_constant( + "repository_root", + "The root of this git repository", + next(p for p in Path(core_conf["config_file_path"]).resolve().parents if (p / ".git").exists()), + ) + diff --git a/scripts/auto_release/requirement.txt b/scripts/auto_release/requirement.txt index 698652f950c2..9827e5f623c5 100644 --- a/scripts/auto_release/requirement.txt +++ b/scripts/auto_release/requirement.txt @@ -1,7 +1,7 @@ azure-identity python-dotenv==0.15.0 ghapi==0.1.19 -packaging==21.3 +packaging==23.1 pytest==6.2.5 azure-storage-blob==12.9.0 fastcore==1.3.25 \ No newline at end of file diff --git a/scripts/breaking_changes_checker/README.md b/scripts/breaking_changes_checker/README.md index 6846916da9de..b3b69acb06aa 100644 --- a/scripts/breaking_changes_checker/README.md +++ b/scripts/breaking_changes_checker/README.md @@ -9,15 +9,15 @@ Add your package name to the `RUN_BREAKING_CHANGES_PACKAGES` found [here](https: ## Run locally with tox -**1) Install tox and tox-monorepo:** +**1) Install tox:** -`pip install tox tox-monorepo` +`pip install tox<5` **2) Run the `breaking` environment.** Here we run the breaking changes tool against azure-storage-blob, for example: -`C:\azure-sdk-for-python\sdk\storage\azure-storage-blob>tox -c ../../../eng/tox/tox.ini -e breaking` +`C:\azure-sdk-for-python\sdk\storage\azure-storage-blob>tox run -c ../../../eng/tox/tox.ini --root . -e breaking` ## Ignore a reported breaking change diff --git a/scripts/devops_tasks/common_tasks.py b/scripts/devops_tasks/common_tasks.py index 805e26e4ea38..59cfd531a992 100644 --- a/scripts/devops_tasks/common_tasks.py +++ b/scripts/devops_tasks/common_tasks.py @@ -9,19 +9,14 @@ # package targeting during release. import glob -from subprocess import check_call, CalledProcessError, Popen import os import errno -import shutil import sys import logging -import ast -import textwrap -import io -import re -import fnmatch -import platform -from typing import Tuple, Iterable + +from subprocess import check_call, CalledProcessError, Popen +from argparse import Namespace +from typing import Iterable # Assumes the presence of setuptools from pkg_resources import parse_version, parse_requirements, Requirement, WorkingSet, working_set @@ -37,7 +32,6 @@ DEV_REQ_FILE = "dev_requirements.txt" NEW_DEV_REQ_FILE = "new_dev_requirements.txt" - logging.getLogger().setLevel(logging.INFO) @@ -105,18 +99,18 @@ def run_check_call( # This function generates code coverage parameters -def create_code_coverage_params(parsed_args, package_name): +def create_code_coverage_params(parsed_args: Namespace, package_path: str): coverage_args = [] if parsed_args.disablecov: logging.info("Code coverage disabled as per the flag(--disablecov)") coverage_args.append("--no-cov") else: - current_package_name = package_name.replace("-", ".") - coverage_args.append("--cov={}".format(current_package_name)) + namespace = ParsedSetup.from_path(package_path).namespace + coverage_args.append("--cov={}".format(namespace)) coverage_args.append("--cov-append") logging.info( "Code coverage is enabled for package {0}, pytest arguements: {1}".format( - current_package_name, coverage_args + namespace, coverage_args ) ) return coverage_args diff --git a/scripts/devops_tasks/create_coverage.py b/scripts/devops_tasks/create_coverage.py index 79231f232d51..1cf679cee522 100644 --- a/scripts/devops_tasks/create_coverage.py +++ b/scripts/devops_tasks/create_coverage.py @@ -32,7 +32,6 @@ def collect_tox_coverage_files(): for f in files: if re.match(".coverage_*", f): coverage_files.append(os.path.join(root, f)) - fix_dot_coverage_file(os.path.join(root, f)) logging.info(".coverage files: {}".format(coverage_files)) @@ -59,20 +58,25 @@ def generate_coverage_xml(): logging.error("Coverage file is not available in {} to generate coverage XML".format(coverage_dir)) -def fix_dot_coverage_file(coverage_file): +def fix_coverage_xml(coverage_file): print("running 'fix_dot_coverage_file' on {}".format(coverage_file)) out = None - with open(coverage_file) as cov_file: + with open(coverage_file, encoding="utf-8") as cov_file: line = cov_file.read() - out = re.sub("\/\.tox\/[\s\S]*?\/site-packages", "/", line) + + # replace relative paths in folder structure pattern + out = re.sub("\/\.tox\/[\s\S_]*?\/site-packages", "", line) + + # replace relative paths in python import pattern + out = re.sub("\.?\.tox[\s\S\.\d]*?\.site-packages", "", out) if out: with open(coverage_file, 'w') as cov_file: cov_file.write(out) - if __name__ == "__main__": collect_tox_coverage_files() generate_coverage_xml() - create_coverage_report() \ No newline at end of file + create_coverage_report() + fix_coverage_xml(os.path.join(root_dir, 'coverage.xml')) \ No newline at end of file diff --git a/scripts/devops_tasks/setup_execute_tests.py b/scripts/devops_tasks/setup_execute_tests.py index de4fbc02f4b9..916f8c59412e 100644 --- a/scripts/devops_tasks/setup_execute_tests.py +++ b/scripts/devops_tasks/setup_execute_tests.py @@ -121,7 +121,7 @@ def run_tests(targeted_packages, test_output_location, test_res, parsed_args): local_command_array = command_array[:] # Get code coverage params for current package - coverage_commands = create_code_coverage_params(parsed_args, package_name) + coverage_commands = create_code_coverage_params(parsed_args, target_package) # Create local copy of params to pass to execute local_command_array.extend(coverage_commands) diff --git a/scripts/devops_tasks/tox_harness.py b/scripts/devops_tasks/tox_harness.py index 44239fb331e7..39b37275a507 100644 --- a/scripts/devops_tasks/tox_harness.py +++ b/scripts/devops_tasks/tox_harness.py @@ -198,6 +198,25 @@ def collect_log_files(working_dir): for f in glob.glob(os.path.join(root_dir, "_tox_logs", "*")): logging.info("Log file: {}".format(f)) +def cleanup_tox_environments(tox_dir: str, command_array: str) -> None: + """The new .coverage formats are no longer readily amended in place. Because we can't amend them in place, + we can't amend the source location to remove the path ".tox//site-packages/". Because of this, we will + need the source where it was generated to stick around. We can do that by being a bit more circumspect about which + files we actually delete/clean up! + """ + if "--cov-append" in command_array: + folders = [folder for folder in os.listdir(tox_dir) if "whl" != folder] + for folder in folders: + try: + shutil.rmtree(folder) + except Exception as e: + # git has a permissions problem. one of the files it drops + # cannot be removed as no one has the permission to do so. + # lets log just in case, but this should really only affect windows machines. + logging.info(e) + pass + else: + shutil.rmtree(tox_dir) def execute_tox_serial(tox_command_tuples): return_code = 0 @@ -218,7 +237,8 @@ def execute_tox_serial(tox_command_tuples): if in_ci(): collect_log_files(cmd_tuple[1]) - shutil.rmtree(tox_dir) + + cleanup_tox_environments(tox_dir, cmd_tuple[0]) if os.path.exists(clone_dir): try: @@ -241,7 +261,7 @@ def prep_and_run_tox(targeted_packages: List[str], parsed_args: Namespace, optio :param parsed_args: An argparse namespace object from setup_execute_tests.py. Not including it will effectively disable "customizations" of the tox invocation. :param options_array: When invoking tox, these additional options will be passed to the underlying tox invocations as arguments. - When invoking of "tox -e whl -c ../../../eng/tox/tox.ini -- --suppress-no-test-exit-code", "--suppress-no-test-exit-code" the "--" will be + When invoking of "tox run -e whl -c ../../../eng/tox/tox.ini -- --suppress-no-test-exit-code", "--suppress-no-test-exit-code" the "--" will be passed directly to the pytest invocation. """ if parsed_args.wheel_dir: @@ -260,11 +280,18 @@ def prep_and_run_tox(targeted_packages: List[str], parsed_args: Namespace, optio tox_execution_array = [sys.executable, "-m", "tox"] + if parsed_args.tenvparallel: + tox_execution_array.extend(["run-parallel", "-p", "all"]) + else: + tox_execution_array.append("run") + + # Tox command is run in package root, make tox set package root as {toxinidir} + tox_execution_array += ["--root", "."] local_options_array = options_array[:] # Get code coverage params for current package package_name = os.path.basename(package_dir) - coverage_commands = create_code_coverage_params(parsed_args, package_name) + coverage_commands = create_code_coverage_params(parsed_args, package_dir) local_options_array.extend(coverage_commands) pkg_egg_info_name = "{}.egg-info".format(package_name.replace("-", "_")) @@ -324,8 +351,6 @@ def prep_and_run_tox(targeted_packages: List[str], parsed_args: Namespace, optio tox_execution_array.extend(["-e", filtered_tox_environment_set]) - if parsed_args.tenvparallel: - tox_execution_array.extend(["-p", "all"]) if parsed_args.tox_env == "apistub": local_options_array = [] diff --git a/scripts/release_helper/requirement.txt b/scripts/release_helper/requirement.txt index 3f1b2e7723b9..fbb1ddc9882f 100644 --- a/scripts/release_helper/requirement.txt +++ b/scripts/release_helper/requirement.txt @@ -2,5 +2,5 @@ PyGithub==1.55 requests==2.28.0 azure-devops==6.0.0b4 msrest>=0.6.21 -packaging==23.0 +packaging==23.1 urllib3==1.26.15 \ No newline at end of file diff --git a/scripts/release_sdk_status/requirement.txt b/scripts/release_sdk_status/requirement.txt index d97bf73cb661..528e24a7962e 100644 --- a/scripts/release_sdk_status/requirement.txt +++ b/scripts/release_sdk_status/requirement.txt @@ -7,4 +7,4 @@ urllib3==1.26.6 pytest==6.2.5 pytest-cov==3.0.0 PyGithub==1.55 -packaging==21.3 \ No newline at end of file +packaging==23.1 \ No newline at end of file diff --git a/sdk/core/azure-core/tests/async_tests/test_rest_asyncio_transport.py b/sdk/core/azure-core/tests/async_tests/test_rest_asyncio_transport.py index 194efb4a8dde..d8b6319d8396 100644 --- a/sdk/core/azure-core/tests/async_tests/test_rest_asyncio_transport.py +++ b/sdk/core/azure-core/tests/async_tests/test_rest_asyncio_transport.py @@ -9,10 +9,11 @@ from rest_client_async import AsyncTestRestClient import pytest +import pytest_asyncio from utils import readonly_checks -@pytest.fixture +@pytest_asyncio.fixture async def client(port): async with AsyncioRequestsTransport() as transport: async with AsyncTestRestClient(port, transport=transport) as client: diff --git a/sdk/core/azure-core/tests/async_tests/test_rest_response_backcompat_async.py b/sdk/core/azure-core/tests/async_tests/test_rest_response_backcompat_async.py index 20248b9848f2..564c698aabe7 100644 --- a/sdk/core/azure-core/tests/async_tests/test_rest_response_backcompat_async.py +++ b/sdk/core/azure-core/tests/async_tests/test_rest_response_backcompat_async.py @@ -5,6 +5,7 @@ # license information. # ------------------------------------------------------------------------- import pytest +import pytest_asyncio from azure.core.pipeline.transport import HttpRequest as PipelineTransportHttpRequest from azure.core.rest import HttpRequest as RestHttpRequest from azure.core.pipeline import Pipeline @@ -19,8 +20,7 @@ def old_request(port): return PipelineTransportHttpRequest("GET", "http://localhost:{}/streams/basic".format(port)) -@pytest.fixture -@pytest.mark.asyncio +@pytest_asyncio.fixture async def get_old_response(old_request): async def _callback(transport, **kwargs): async with transport() as sender: @@ -44,8 +44,7 @@ def new_request(port): return RestHttpRequest("GET", "http://localhost:{}/streams/basic".format(port)) -@pytest.fixture -@pytest.mark.asyncio +@pytest_asyncio.fixture async def get_new_response(new_request): async def _callback(transport, **kwargs): async with transport() as sender: diff --git a/sdk/media/azure-media-analytics-edge/dev_requirements.txt b/sdk/media/azure-media-analytics-edge/dev_requirements.txt index 08b52149d5f2..031c5086f3f6 100644 --- a/sdk/media/azure-media-analytics-edge/dev_requirements.txt +++ b/sdk/media/azure-media-analytics-edge/dev_requirements.txt @@ -4,5 +4,3 @@ -e ../../identity/azure-identity aiohttp>=3.0; python_version >= '3.5' aiodns>=2.0; python_version >= '3.5' -tox>=3.20.0 -tox-monorepo>=0.1.2 \ No newline at end of file diff --git a/sdk/media/azure-media-analytics-edge/docs/DevTips.md b/sdk/media/azure-media-analytics-edge/docs/DevTips.md index 0b9d965181eb..e4d02545fd8f 100644 --- a/sdk/media/azure-media-analytics-edge/docs/DevTips.md +++ b/sdk/media/azure-media-analytics-edge/docs/DevTips.md @@ -11,26 +11,8 @@ Tox is the testing and virtual environment management tool that is used to verify our sdk will be installed correctly with different Python versions and interpreters. To run tox follow these instructions -``` -pip install tox tox-monorepo -cd path/to/target/folder -tox -c eng/tox/tox.ini -``` -To run a specific tox command from your directory use the following commands: -```bash -> tox -c ../../../eng/tox/tox.ini -e sphinx -> tox -c ../../../eng/tox/tox.ini -e pylint -> tox -c ../../../eng/tox/tox.ini -e mypy -> tox -c ../../../eng/tox/tox.ini -e whl -> tox -c ../../../eng/tox/tox.ini -e sdist -``` -A quick description of the five commands above: -* sphinx: documentation generation using the inline comments written in our code -* lint: runs pylint to make sure our code adheres to the style guidance -* mypy: runs the mypy static type checker for Python to make sure that our types are valid -* whl: creates a whl package for installing our package -* sdist: creates a zipped distribution of our files that the end user could install with pip - +Please see the repo's [contributing guide](https://github.com/Azure/azure-sdk-for-python/blob/main/CONTRIBUTING.md#building-and-testing) for instructions +on how to install and run `tox`. ### Troubleshooting tox errors diff --git a/sdk/videoanalyzer/azure-media-videoanalyzer-edge/dev_requirements.txt b/sdk/videoanalyzer/azure-media-videoanalyzer-edge/dev_requirements.txt index 6d14633f7d01..198cc317f2f3 100644 --- a/sdk/videoanalyzer/azure-media-videoanalyzer-edge/dev_requirements.txt +++ b/sdk/videoanalyzer/azure-media-videoanalyzer-edge/dev_requirements.txt @@ -4,5 +4,3 @@ -e ../../identity/azure-identity aiohttp>=3.0 aiodns>=2.0 -tox>=3.20.0 -tox-monorepo>=0.1.2 \ No newline at end of file diff --git a/sdk/videoanalyzer/azure-media-videoanalyzer-edge/docs/DevTips.md b/sdk/videoanalyzer/azure-media-videoanalyzer-edge/docs/DevTips.md index 265409765a60..3d451b42e908 100644 --- a/sdk/videoanalyzer/azure-media-videoanalyzer-edge/docs/DevTips.md +++ b/sdk/videoanalyzer/azure-media-videoanalyzer-edge/docs/DevTips.md @@ -11,26 +11,8 @@ Tox is the testing and virtual environment management tool that is used to verify our sdk will be installed correctly with different Python versions and interpreters. To run tox follow these instructions -``` -pip install tox tox-monorepo -cd path/to/target/folder -tox -c eng/tox/tox.ini -``` -To run a specific tox command from your directory use the following commands: -```bash -> tox -c ../../../eng/tox/tox.ini -e sphinx -> tox -c ../../../eng/tox/tox.ini -e pylint -> tox -c ../../../eng/tox/tox.ini -e mypy -> tox -c ../../../eng/tox/tox.ini -e whl -> tox -c ../../../eng/tox/tox.ini -e sdist -``` -A quick description of the five commands above: -* sphinx: documentation generation using the inline comments written in our code -* lint: runs pylint to make sure our code adheres to the style guidance -* mypy: runs the mypy static type checker for Python to make sure that our types are valid -* whl: creates a whl package for installing our package -* sdist: creates a zipped distribution of our files that the end user could install with pip - +Please see the repo's [contributing guide](https://github.com/Azure/azure-sdk-for-python/blob/main/CONTRIBUTING.md#building-and-testing) for instructions +on how to install and run `tox`. ### Troubleshooting tox errors diff --git a/tools/azure-sdk-tools/ci_tools/versioning/requirements.txt b/tools/azure-sdk-tools/ci_tools/versioning/requirements.txt index a5b557a29095..15a167270360 100644 --- a/tools/azure-sdk-tools/ci_tools/versioning/requirements.txt +++ b/tools/azure-sdk-tools/ci_tools/versioning/requirements.txt @@ -1,4 +1,4 @@ -packaging==19.2 +packaging==23.1 pyparsing==2.4.5 six==1.13.0 setuptools==67.6.0 \ No newline at end of file diff --git a/tools/azure-sdk-tools/setup.py b/tools/azure-sdk-tools/setup.py index c77015e24aa9..3a161b94c8ef 100644 --- a/tools/azure-sdk-tools/setup.py +++ b/tools/azure-sdk-tools/setup.py @@ -20,7 +20,7 @@ "pyopenssl", "python-dotenv", "PyYAML", - "urllib3", + "urllib3<2", "tomli" ]