SUB-character in a csv causes read_csv() with C-Engine to detect EOF

[Khris777]

Problem description

If there is a SUB-character in a string in a csv, read_csv() with the standard C-engine returns

ParserError: Error tokenizing data. C error: EOF inside string starting at line 0

The Python-engine can read the file fine.

It seems I can't put example data with a SUB-character here, so I pasted an example line here instead:
https://pastebin.com/x6QPY4Hf
Just paste the line into a csv and try to read it with read_csv().

I don't know if this behaviour is expected or not since this character is indeed used as EOF in certain cases, however I see little sense in having a SUB character interpreted as EOF in the middle of a csv file.

commit: None

python: 3.6.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64

pandas: 0.20.2

[TomAugspurger]

Seems to work for me, could you try updating your version of pandas?

In [8]: pd.read_csv("foo.csv").columns
Out[8]: Index(['12456', 'XY', 'Pedfg', '234'], dtype='object')

In [9]: text = open('foo.csv').read()

In [10]: text
Out[10]: '12456,XY,"\x1aPedfg",234\n'

[Khris777]

Looks like 0.20.3 isn't on conda yet.

EDIT:
My output when reading using Python engine "shows" the SUB-character:

pd.read_csv(ETKADataDir+file,engine='python').columns
Out[7]: Index(['12456', 'XY', '�Pedfg', '234'], dtype='object')

[TomAugspurger]

Ok, I'll check on that. It is available on conda-forge with `conda install -c conda-forge pandas`

…

On Thu, Jul 13, 2017 at 12:34 AM, Khris777 ***@***.***> wrote: Looks like 0.20.3 isn't on conda yet. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16893 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHItUaLykoXhuS_vd3AlS5qSYAymy0ks5sNayCgaJpZM4OVo_Q> .

[gfyoung]

@Khris777 : Let us know if updating works for you. In any case, a test in the code-base will be needed to confirm this.

[Khris777]

Can confirm that updating solves the problem.

[gfyoung]

@Khris777 : Slow down there 😄 ! Even though the issue is resolved, we should add a test to ensure that this issue doesn't pop up in the future. Afterwards, we'll close this.

Care to add your example code in a PR?

[Khris777]

PR = Pull Request?
I have never done something like this before here, honestly I don't know exactly what you are asking me to do here. 😕

[gfyoung]

@Khris777 : yes, PR = pull request

Essentially, I am asking you to add your raw pastebin string as a test to the read_csv function, something like this format would work:

def test_...(self):
    data = "12456,XY,"�Pedfg",234"
    result = self.read_csv(StringIO(data))
    expected = <construct-expected-DataFrame-here>
    tm.assert_frame_equal(df, result)

The following directory has a lot of good examples that you can follow (you would likely add your test under common.py). Here is also the link containing general info about contributing to pandas.

If you would like to attempt contributing the code yourself, we'll be happy to help you out. If you are unable to do so at this time, we can take care of adding the test ourselves, but more contributors are always preferred 😄