Update DataFusion instructions / Enable swap on small machines#804
Update DataFusion instructions / Enable swap on small machines#804rschu1ze merged 2 commits intoClickHouse:mainfrom
Conversation
alamb
force-pushed
the
alamb/fix_low_mem_machines
branch
from
e6e271a to
9d5f358
Compare
alamb
force-pushed
the
alamb/fix_low_mem_machines
branch
from
bbefbb9 to
3e4ced3
Compare
|
FYI @pmcgleenon and @waynexia I wonder what you think of this approach (just turning on swap)? |
|
FYI @rschu1ze -- I believe this PR is now ready for review |
| source ~/.cargo/env | ||
|
|
||
| if [ $(free -g | awk '/^Mem:/{print $2}') -lt 12 ]; then | ||
| echo "LOW MEMORY MODE" |
There was a problem hiding this comment.
This (and the corresponding change in datafusion are the only functional changes). This implements the request by @rschu1ze in #749 (comment))
| @@ -0,0 +1,37 @@ | |||
| #!/bin/bash | |||
|
|
|||
| # This script converts the raw `result.csv` data from `benchmark.sh` into the | |||
There was a problem hiding this comment.
This is the helper script from #749 (with a different name) for our convenience when producing new run results
| ## Cookbook: Generate benchmark results | ||
|
|
||
| ## Generate benchmark results | ||
| Follow instructions in the [datafusion](../datafusion/README.md) directory. |
There was a problem hiding this comment.
I reduced the duplication of the documentation and updated the Known Issues
|
@alamb I verified the changes locally - they work well, thanks. Let me know if you like me to update the results as well (this makes sense only if swapping is expected to change the measurements). |
2 participants
Rationale
I would like to
Since we last successfully ran the benchmarks ourselves (47.0.0), new smaller machines (c6a.xlarge 4core, 8G RAM) where the existig scripts have struggled a lot. Specifically:
rustcis OOM killed (due to the link flags we pass)datafusion-clifails to run ClickBench queries with 8GB of RAM apache/datafusion#18473)Also, the existing instructions are somewhat outdated
Changes
make-json.shscript to help us evaluate the current status locallyNon Changes
Note this PR does NOT update any results (I will make a follow on PR with actual numbers, I want to get the scripts into shape first)
Testing
I tested on these machines following the instructions