ARROW-4629: [Python] Pandas arrow conversion slowed down by imports

[fjetter]

The local imports slow down the conversion from pandas to arrow significantly (see here)

[kszucs]

Pandas is an optional dependency of pyarrow, that's why pandas is not imported here.

[fjetter]

I changed it to an import iff pandas is available

[xhochy]

@wesm I'm just realising that here is one of the potential problems users may have reported about pdcompat import problems. This path should be supported without pandas but currently isn't.

[wesm]

yes, I would agree with you. can you create a follow up issue about this? We should set up a docker-compose "no pandas" build to make sure that the project is usable without pandas

[wesm]

I went ahead and opened https://issues.apache.org/jira/browse/ARROW-4640

[xhochy]

@fjetter Can you expand this to also only trigger the line once a user has Pandas? I added the necessary changes as Suggested Edits.

[codecov-io]

Codecov Report

Merging #3706 into master will decrease coverage by 21.31%.
The diff coverage is 57.14%.

@@             Coverage Diff             @@
##           master    #3706       +/-   ##
===========================================
- Coverage   87.79%   66.48%   -21.32%     
===========================================
  Files         688      323      -365     
  Lines       84280    48123    -36157     
  Branches     1081        0     -1081     
===========================================
- Hits        73994    31993    -42001     
- Misses      10175    16130     +5955     
+ Partials      111        0      -111

Impacted Files	Coverage Δ
python/pyarrow/pandas_compat.py	96.3% <100%> (-0.04%)	⬇️
python/pyarrow/array.pxi	68.75% <50%> (-0.45%)	⬇️
cpp/src/arrow/pretty_print.h	0% <0%> (-100%)	⬇️
cpp/src/parquet/murmur3.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/testing/util.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/io/hdfs-internal.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/util/sse-util.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/testing/random.h	0% <0%> (-100%)	⬇️
cpp/src/parquet/hasher.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/util/int-util.cc	0.39% <0%> (-99.21%)	⬇️
... and 512 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e61bcf...eb5c8ba. Read the comment docs.

[pitrou]

I'm surprised we're importing Pandas inconditionally. We probably shouldn't do that, as Pandas is quite slow to import:

$ time python -c "import numpy"

real	0m0,097s
user	0m0,077s
sys	0m0,020s

$ time python -c "import pandas"

real	0m0,300s
user	0m0,272s
sys	0m0,028s

Here is a comparison of PyArrow import time with and without Pandas:

$ time python -c "import pyarrow"

real	0m0,360s
user	0m0,305s
sys	0m0,037s

$ time python -c "import sys; sys.modules['pandas'] = None; import pyarrow"

real	0m0,144s
user	0m0,124s
sys	0m0,020s

=> more than twice faster without.

[xhochy]

@pitrou Any ideas on how to avoid these local imports and also have the benefit of only loading pandas when needed?

[pitrou]

Imports are reasonably cheap once the module is already loaded, but it's probably better to avoid doing them in a tight loop. So hoisting the import outside of critical loops should be sufficient.

>>> %timeit import pandas                                                                                                                                         
102 ns ± 0.497 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> %timeit import pyarrow.pandas_compat as pdcompat                                                                                                              
253 ns ± 7.27 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

This is in pure Python, though. Cython does not seem to implement the same optimizations as CPython does:

>>> %timeit lib._noop_bench()                                                                                                                                     
66.3 ns ± 0.379 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>>> %timeit lib._import_bench()                                                                                                                                   
928 ns ± 19.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

(edit: issue opened for Cython at cython/cython#2854)

[xhochy]

@pitrou Are you happy with the change here and we will deal with the Pandas import issue separately or should this patch be adapted before merging?

[pitrou]

Yes, I've created ARROW-4637 for the pandas import.

[wesm]

@xhochy @pitrou we could create a singleton object that exposes the pandas APIs that we need. The imports will be performed on creation of the singleton