GH-38342: [Python] Update to_pandas to use non-deprecated DataFrame constructor

[jorisvandenbossche]

Rationale for this change

Avoiding a deprecation warning from pandas

• Closes: [Python] pandas internals: Passing a manager object to DataFrame() constructor is deprecated #38342

[jorisvandenbossche]

@github-actions crossbow submit -g python

[github-actions]

Revision: cccae65

Submitted crossbow builds: ursacomputing/crossbow @ actions-0102d30519

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[jorisvandenbossche]

@github-actions crossbow submit -g python

[github-actions]

Revision: 82db749

Submitted crossbow builds: ursacomputing/crossbow @ actions-306d85618d

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-38-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 2fadab2.

There were 7 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-12-01 18:35:12Z
  • dataset-selectivity (Python) with dataset=nyctaxi_multi_ipc_s3, selectivity=10%
  • dataset-selectivity (Python) with dataset=nyctaxi_multi_ipc_s3, selectivity=100%
• and 5 more (see the report linked below)

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.