IO Converters

[jreback]

discussed in #4698

convert_data would return an appropriate Converter object to do various types of conversions. Mainly useful when you have to chunk both sides of these.

csv -> hdf
sql -> hdf
hdf -> csv

c = pd.convert_data('csv','hdf', chunksize=....)
c.read_csv(input_path. ......)
c.to_hdf(output_path, key.....)
c.execute()

pretty simple under the hood...
biggest issue is MAY need to read the input file twice (for read_csv) (if ints are detected in the first input chunk), because the dtype MAY change (e.g. you get a NaN later in later chunks)

useful?

[jreback]

blaze to the rescue!

@cpcloud