Revision 4f69c98ae95681cf972fa6701c94dbbb28e40d80 authored by sychen on 09 September 2022, 21:36:39 UTC, committed by Dongjoon Hyun on 09 September 2022, 21:36:39 UTC
### What changes were proposed in this pull request? Increase ORC test coverage. [ORC-1205](https://issues.apache.org/jira/browse/ORC-1205) Size of batches in some ConvertTreeReaders should be ensured before using ### Why are the changes needed? When spark reads an orc with type promotion, an `ArrayIndexOutOfBoundsException` may be thrown, which has been fixed in version 1.7.6 and 1.8.0. ```java java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:387) at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:740) at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1069) at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:65) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? add UT Closes #37808 from cxzl25/SPARK-39830-3.3. Authored-by: sychen <sychen@ctrip.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent aaa8292
File | Mode | Size |
---|---|---|
fairscheduler.xml.template | -rw-r--r-- | 1.1 KB |
log4j2.properties.template | -rw-r--r-- | 3.3 KB |
metrics.properties.template | -rw-r--r-- | 8.9 KB |
spark-defaults.conf.template | -rw-r--r-- | 1.3 KB |
spark-env.sh.template | -rwxr-xr-x | 4.4 KB |
workers.template | -rw-r--r-- | 865 bytes |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...