feat: support Nested Structs

[gord02]

This PR is to change the substrait-java virtualTableScan to use nested structs in place of the now deprecated struct literals.

The issue

Key changes:

• Add the visitation pattern logic
• Proto to Rel conversion
• Rel to proto conversion
• Spark to Substrait
• Substrait to Spark
• SubstraitRelNodeConverter: substrait virtualTableScan to calcite logicalValues, required creating a project for non-literal expressions
• SubstraitRelVisitor: Calcite to substrait conversion that removes the extra project added to account for non-literal expressions

Testing:

• LogicalValuesTest tested the conversion from pojo to calcite node back to pojo
• VirtualTableScanTest tests the creation of a VirtualTableScan using nested struct, and the equivalence of the pojo objects created from the proto VirtualTableScan set using the deprecated values, to the new fields attribute
• VirtualTableScanTest also validates that VirtualTableScan proto rows cannot be set using both values and fields

Testing:
./gradlew test --tests io.substrait.isthmus.LogicalValuesTest --debug-jvm

[bestbeforetoday]

@gord02 you will need to make sure that the subject line and text of all commits in your pull request conform to the structure dictated by conventional commits. You can look at the output of the Lint commits for semantic-release build check to see problems flagged, and also the CONTRIBUTING guide for tips.

When you have lots of commits that need fixing, it can sometimes be easier to squash them into fewer commits to reduce the number of changes you need to make.

[nielspardon]

Does this PR replace #417 which seems to have tried to do a similar change?

[gord02]

Hi @nielspardon! This PR completes and builds off of that PR yes. So it can be deleted.

[benbellick]

I created an issue to track this particular problem @ #587. @gord02 would you mind including it somewhere in your PR description so that it gets linked?

[benbellick]

I have left a few comments on here so far. I think some of them might cause some reshuffling around so I'll save some of the more in-depth suggestions for the future. One other thing, it would be great if you could introduce some test which specifically demonstrates the logic you introduced where deserialized protos can handle either the presence of fields or values, but not both. I can imagine a test in which you constuct two protos, one via values and one via fields, and just demonstrate that once you render them both as POJOs, they are equal. That's just one idea though.

Anyways, thanks for all of your work! Let me know if anything I wrote isn't clear :)

[benbellick]

Thanks for your patience, both in waiting for my review and for dealing with my comments! I appreciate the work. I didn't get a chance to review all of the PR, but wanted to flush the comments I had so far before running off to a meeting. 🙇

[benbellick]

Okay got the chance to do another pass. I think I'm just going to review the core section for now until I think its in a good place, and then I'll continue onwards with the review in isthmus / spark if that works for you! Thanks again for your work 🚀

[benbellick]

very minor changes to suggest within core, but after those, everything there looks good! I will come back and review the isthmus stuff in a bit. As mentioned before, I am not as familiar there so it could be a good idea to rope in @vbarua or someone.

Nice work!

[vbarua]

Left some comments. The biggest thing I noticed is the logic for converting to Substrait to Calcite doesn't quite work. Your LogicalValuesTest doesn't check for right invariant to detect it.

[vbarua]

If you inspect the rowType of the LogicalProject you've created, you'll notice that it only outputs a single column. We should have two columns, because the VirtualTable we're converting has two columns. The issues is that Calcite, unlike Substrait, only outputs the values of project expressions.

[gord02]

The way I wrote the code was just to use the logicalProject to store expressions that could not fit in the logicalValues table, which are all non-literal expressions. Should this be changed such that all the expressions are reflected in the logicalProject?

[vbarua]

This doesn't actually output the correct number of columns when you use it. I left a comment in LogicalValueTest about it.