Fix csv diff compared to Spark CPU

[res-life]

Add test case

Description

Fix csv diff compared to Spark CPU
closes #20812

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[res-life]

/ok to test

[copy-pr-bot]

/ok to test

@res-life, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[res-life]

/ok to test b76068c

[res-life]

It's draft, not sure if this PR will fails other cases.

[res-life]

pre-commit.ci autofix

[GaryShen2008]

Hi @vuule , we tried to fix the csv diff issue by Cursor. We tried on the cases, it seems working. But we don't have confidence about other cases. Can you review this PR to feedback if it makes sense or not? Thanks.

[res-life]

/ok to test dc4c828

[vuule]

I'll take a look, thanks for the PR!

[vuule]

One of the new tests fails with the changes in this PR. So the fix here is about as far as I got on my end. Looking into fixing the second test case.

[res-life]

@vuule I double checked, this PR fixes the second test case of #20812. But this PR has regression.

[vuule]

The new test is failing for me

27: [ RUN      ] CsvReaderTest.DoubleQuotesContinuous
27: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:557: Failure
27: Failed
27: first difference: lhs[0] = "packageName":"test","type":"test","url_scheme":false,"referer":",test, rhs[0] = "packageName":"test","type":"test","url_scheme":false,"referer":"",test
27: Google Test trace:
27: /home/coder/cudf/cpp/tests/io/csv_test.cpp:2687:  <--  line of failure
27: 
27: 
27: [  FAILED  ] CsvReaderTest.DoubleQuotesContinuous (0 ms)

[vuule]

/ok to test 0098d49

[vuule]

The core issue seems to be that cuDF always applies some string processing steps that should only be done when the string is actually quoted.

I think I have a fix (all tests pass), but it seems to lower the performance quite a bit in some cases.
Working on getting the changes into a mergeable state.

[res-life]

@vuule Thanks very much for helping this issue which is from a customer.
I'll close this draft PR and feel free to put up your own PR.
After your PR is ready, I'll validate the customer query later.

[vuule]

I'm fine with pushing to this PR, no need to close