DX-24413, DX-24414 : Gandiva cache peformance improvements #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

projjal wants to merge 1 commit into dremio:rel-460 from projjal:gandivacache

projjal commented Aug 5, 2020 •

edited

Loading

Prewarm the gandiva cache on gandiva startup time
Make the cache size configurable

projjal requested review from pravindra and vvellanki

August 5, 2020 09:22

projjal changed the title ~~Prewarm the gandiva cache during loading time. Make the cache size co…~~ DX-24413, DX-24414 : Gandiva cache peformance improvements

projjal force-pushed the gandivacache branch from c93f763 to 7cbaf22 Compare

August 5, 2020 09:33

projjal requested a review from praveenbingo

August 5, 2020 09:34


          Prewarm gandiva cache on startup. Make cache size configurable

c4c56e2

projjal force-pushed the gandivacache branch from 7cbaf22 to c4c56e2 Compare

August 6, 2020 07:35

praveenbingo suggested changes

View reviewed changes

cpp/src/gandiva/cache.h Outdated

praveenbingo Aug 5, 2020

cant we instead pass this as a configuration (that is in turn passed from a support option) ? that way we can support other customers too..

https://github.com/apache/arrow/blob/master/cpp/src/gandiva/configuration.h

cpp/src/gandiva/jni/jni_common.cc

    
                        std::ifstream fin;

                        fin.open(dir_iter->path().string(), std::ios::binary);

                        if (!fin.is_open()) {

                          std::cerr << "[Gandiva Cache Prewarm] Failure opening warmup cache file"

praveenbingo Aug 6, 2020

maybe also mention which file..

cpp/src/gandiva/jni/jni_common.cc

    
                        int32_t schema_len;

                        if (remaining < sizeof(schema_len)) {

                          std::cerr << "[Gandiva Cache Prewarm] Invalid file." << std::endl;

praveenbingo Aug 6, 2020

across the board - file name along with errors.

cpp/src/gandiva/jni/jni_common.cc

    
                vector_expander_ret_capacity_ =

                    env->GetFieldID(vector_expander_ret_class_, "capacity", "J");

                PrewarmCache();

praveenbingo Aug 6, 2020

is this going to block all queries until the cache is warmed? should we do this in the background instead for e.g. if we have a 100 expression list this is going to take upto a minute to prepare the cache during which no query can progress

praveenbingo reviewed

View reviewed changes

cpp/src/gandiva/jni/jni_common.cc

    
                  }

                  fs::path path(env_path);

                  if (!fs::is_directory(path) && !fs::create_directories(path)) {

praveenbingo Aug 6, 2020

is this thread safe? two threads trying to create the directory at the same time

cpp/src/gandiva/jni/jni_common.cc

    
                    new ProjectorHolder(schema_ptr, ret_types, std::move(projector)));

                module_id = projector_modules_.Insert(holder);

                if (!cache_hit) {

praveenbingo Aug 6, 2020

so is the expectation, apple will get files from all executors? wont there be duplicates in that case..

also is the expectation that cache size be at-least as big to accommodate all expressions needed?

are new queries expected in this cluster - if yes and new a query comes in now we might evict one of these expressions. is that ok

stevelorddremio pushed a commit to stevelorddremio/arrow that referenced this pull request


          apacheGH-41145: [R][CI] test-r-dev-duckdb fails installing duckdb (ap…

…ache#41152)

### Rationale for this change

An error is received installing R duckdb:

```
dremio#15 18.13 > remotes::install_github('duckdb/duckdb-r', build = FALSE)
dremio#15 18.27 Error: Failed to install 'unknown package' from **GitHub:**
dremio#15 18.27   Line starting 'Roxyg ...' is malformed!
```

Some searching seems to suggest that this is because R cannot process UTF-8 characters in DESCRIPTION files if the `LANG` is set to `C`.

### What changes are included in this PR?

The `LANG` is set to `C.UTF-8` in the dockerfile for this CI job

### Are these changes tested?

The change only affects a test

### Are there any user-facing changes?

No
* GitHub Issue: apache#41145

Authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

lriggs pushed a commit to lriggs/arrow that referenced this pull request


          apacheGH-41145: [R][CI] test-r-dev-duckdb fails installing duckdb (ap…

a2c5a9a

…ache#41152)

### Rationale for this change

An error is received installing R duckdb:

```
dremio#15 18.13 > remotes::install_github('duckdb/duckdb-r', build = FALSE)
dremio#15 18.27 Error: Failed to install 'unknown package' from **GitHub:**
dremio#15 18.27   Line starting 'Roxyg ...' is malformed!
```

Some searching seems to suggest that this is because R cannot process UTF-8 characters in DESCRIPTION files if the `LANG` is set to `C`.

### What changes are included in this PR?

The `LANG` is set to `C.UTF-8` in the dockerfile for this CI job

### Are these changes tested?

The change only affects a test

### Are there any user-facing changes?

No
* GitHub Issue: apache#41145

Authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

lriggs pushed a commit to lriggs/arrow that referenced this pull request


          apacheGH-40633: [C++][Python] Fix ORC crash when file contains unknow…

1e5d6e5

…n timezone (apache#45051)

### Rationale for this change

If the timezone database is present on the system, but does not contain a timezone referenced in a ORC file, the ORC reader will crash with an uncaught C++ exception.

This can happen for example on Ubuntu 24.04 where some timezone aliases have been removed from the main `tzdata` package to a `tzdata-legacy` package. If `tzdata-legacy` is not installed, trying to read a ORC file that references e.g. the "US/Pacific" timezone would crash.

Here is a backtrace excerpt:
```
dremio#12 0x00007f1a3ce23a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
dremio#13 0x00007f1a3ce39391 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
dremio#14 0x00007f1a3f4accc4 in orc::loadTZDB(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#15 0x00007f1a3f4ad392 in std::call_once<orc::LazyTimezone::getImpl() const::{lambda()dremio#1}>(std::once_flag&, orc::LazyTimezone::getImpl() const::{lambda()dremio#1}&&)::{lambda()dremio#2}::_FUN() () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#16 0x00007f1a4298bec3 in __pthread_once_slow (once_control=0xa5ca7c8, init_routine=0x7f1a3ce69420 <__once_proxy>) at ./nptl/pthread_once.c:116
dremio#17 0x00007f1a3f4a9ad0 in orc::LazyTimezone::getEpoch() const ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#18 0x00007f1a3f4e76b1 in orc::TimestampColumnReader::TimestampColumnReader(orc::Type const&, orc::StripeStreams&, bool) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#19 0x00007f1a3f4e84ad in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#20 0x00007f1a3f4e8dd7 in orc::StructColumnReader::StructColumnReader(orc::Type const&, orc::StripeStreams&, bool, bool) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#21 0x00007f1a3f4e8532 in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#22 0x00007f1a3f4925e9 in orc::RowReaderImpl::startNextStripe() ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#23 0x00007f1a3f492c9d in orc::RowReaderImpl::next(orc::ColumnVectorBatch&) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
dremio#24 0x00007f1a3e6b251f in arrow::adapters::orc::ORCFileReader::Impl::ReadBatch(orc::RowReaderOptions const&, std::shared_ptr<arrow::Schema> const&, long) ()
   from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900
```

### What changes are included in this PR?

Catch C++ exceptions when iterating ORC batches instead of letting them slip through.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#40633

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>

akravchukdremio pushed a commit to akravchukdremio/arrow that referenced this pull request


          apacheGH-48894: [Python][C++] Use base Azure::Core::RequestFailedExce…

c183167

…ption instead of final Azure::Storage::StorageException and set minimum nodejs on conda env to 16 for Azurite to work (apache#48895)

### Rationale for this change

nodejs 12 is currently being installed on conda. CI jobs are failing and or segfaulting due to azurite failing with old versions.

```
2026-01-13T18:32:39.6961900Z dremio#15 [ 9/11] RUN /arrow/ci/scripts/install_azurite.sh
2026-01-13T18:32:39.9624124Z dremio#15 0.417 Node.js version = v12.4.0
2026-01-13T18:32:42.2087322Z dremio#15 2.663 npm WARN deprecated rimraf@ 3.0.2: Rimraf versions prior to v4 are no longer supported
2026-01-13T18:32:42.3917601Z dremio#15 2.846 npm WARN deprecated uuid@ 3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
2026-01-13T18:32:51.4870197Z dremio#15 11.94 npm WARN deprecated glob@ 7.2.3: Glob versions prior to v9 are no longer supported
2026-01-13T18:32:51.7035681Z dremio#15 12.01 npm WARN deprecated inflight@ 1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
2026-01-13T18:33:02.1406491Z dremio#15 22.59 /opt/conda/envs/arrow/bin/azurite -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/azurite.js
2026-01-13T18:33:02.3841290Z dremio#15 22.60 /opt/conda/envs/arrow/bin/azurite-queue -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/queue/main.js
2026-01-13T18:33:02.3842792Z dremio#15 22.60 /opt/conda/envs/arrow/bin/azurite-blob -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/blob/main.js
2026-01-13T18:33:02.3844216Z dremio#15 22.60 /opt/conda/envs/arrow/bin/azurite-table -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/table/main.js
2026-01-13T18:33:02.3846002Z dremio#15 22.66 npm WARN applicationinsights@ 2.9.8 requires a peer of applicationinsights-native-metrics@* but none is installed. You must install peer dependencies yourself.
2026-01-13T18:33:02.3847278Z dremio#15 22.66 
2026-01-13T18:33:02.3847564Z dremio#15 22.66 + azurite@ 3.35.0
2026-01-13T18:33:02.3848038Z dremio#15 22.66 added 376 packages from 296 contributors in 20.644s
2026-01-13T18:33:02.3848830Z dremio#15 22.69 /opt/conda/envs/arrow/bin/azurite
2026-01-13T18:33:02.8929329Z dremio#15 23.35 /opt/conda/envs/arrow/lib/node_modules/azurite/node_modules/fs-extra/lib/util/async.js:14
2026-01-13T18:33:02.8930231Z dremio#15 23.35         (err) => err ?? new Error('unknown error')
2026-01-13T18:33:02.8930740Z dremio#15 23.35                       ^
```

The job on PyArrow was segfaulting due to an Exception being thrown but not catch. In general we were using `Azure::Storage::StorageException` but `Azure::Core::Http::TransportException` could also be thrown on some cases.
Bot are final but inherit from `Azure::Core::RequestFailedException`.

### What changes are included in this PR?

- Pin minimum nodejs version to 16 so the failure doesn't happen again.
- Update catching `Azure::Storage::StorageException` to `Azure::Core::RequestFailedException` so `Azure::Core::Http::TransportException` is also catch.

### Are these changes tested?

Yes on CI.

### Are there any user-facing changes?

No

* GitHub Issue: apache#48894

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet