benchmark/nixlbench: --config_file param supported #989

anton-nayshtut · 2025-11-04T08:52:28Z

This PR introduces config file support to nixlbench.

The name of a config file can be specified using the --config_file command-line parameter. The config file is in INI format.

Each existing command-line parameter can also be placed in the [global] section of the configuration file, so the following invocations:

nixlbench --etcd_endpoints http://localhost:2379 --backend POSIX --filepath /mnt/test --posix_api_type AIO

and

nixlbench --config_file /tmp/nixlbench.config

where /tmp/nixlbench.config contains:

[global]
etcd_endpoints=http://localhost:2379
backend=POSIX
filepath=/mnt/test
posix_api_type=AIO

are identical.

If a parameter exists in the config file and is also explicitly specified on the command line, the latter takes precedence.

copy-pr-bot · 2025-11-04T08:52:32Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2025-11-04T08:52:37Z

👋 Hi anton-nayshtut! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

aranadive · 2025-11-11T08:46:27Z

@anton-nayshtut Thanks so much adding this. I had two general comments.
Did you consider using JSON format for config file? From a usability, portability aspect its well-known, with number of tools already around it. Plus, we can reuse it for kvbench since json is easily parsed in python.
Another option, would be to use the flagfile option of Gflags - to pass all the values/parameters from command line into the file. This would be easier than json or ini since we wouldnt have to go through licensing again.

For CXXOPTS, Im unclear why this was required? If you use the gflags flagfile now, any other options after the file should overwrite the ones in the file.

WDYT?

vvenkates27

Some minor comments, and why not JSon?

vvenkates27 · 2025-11-04T21:58:34Z

benchmark/nixlbench/src/utils/external/ini.h

@@ -0,0 +1,624 @@
+/**


License needs to be fixed?

It's not our code. It's an external MIT-licensed library. We can't change the license, AFAIK. But I exempted the external files from the relevant CI checks.

vvenkates27 · 2025-11-04T21:59:22Z

benchmark/nixlbench/src/utils/external/cxxopts.hpp

@@ -0,0 +1,2925 @@
+/*


Same here .. fix license

benchmark/nixlbench/src/utils/utils.cpp

vvenkates27 · 2025-11-07T15:54:53Z

benchmark/nixlbench/src/utils/external/cxxopts.hpp

+#  include <regex>
+#endif  // CXXOPTS_NO_REGEX
+
+// Nonstandard before C++17, which is coincidentally what we also need for <optional>


What is the benefit of CXX_OPTS compared to gflags? Why can't we parse using glags even for a config file?

See the commit message for benchmark/nixlbench: switched to cxxopts - I discuss it at length there. Hope it makes sense.

anton-nayshtut · 2025-11-11T10:56:41Z

@anton-nayshtut Thanks so much adding this. I had two general comments. Did you consider using JSON format for config file? From a usability, portability aspect its well-known, with number of tools already around it. Plus, we can reuse it for kvbench since json is easily parsed in python. Another option, would be to use the flagfile option of Gflags - to pass all the values/parameters from command line into the file. This would be easier than json or ini since we wouldnt have to go through licensing again.

For CXXOPTS, Im unclear why this was required? If you use the gflags flagfile now, any other options after the file should overwrite the ones in the file.

WDYT?
Hi Adit. Thanks for your review!

As for the questions:

INI vs JSON

Sure, I considered using JSON for config files and I agree that, as you mentioned, it has some advantages. However, JSON also has some downsides that led me to prefer INI format.

My thoughts were along these lines:

INI format is often a better choice than JSON for simple configuration files because it’s more approachable, easier to manually edit, and visually cleaner for basic use cases.

INI files are highly readable and easy to work with - especially for non-programmers or for quick manual edits. They support organizing settings into basic sections and key-value pairs, so simple configs stay tidy and quick to review. Comments are supported in INI (unlike JSON by default), letting you explain settings directly in the file, which is a huge advantage for documentation and collaboration. INI files are much simpler than JSON, and avoid extra punctuation, strict syntax, and nested requirements - reducing the chance of errors when hand-editing.

In summary, if your configs are flat, small, and likely to be edited by hand, INI is much easier for users to read and update than JSON. JSON is better when you need complex or hierarchical data, but INI shines for basic, human-friendly settings.

Some additional considerations:

Many popular test and benchmark tools use INI format for config files, like fio, tox, bdperf, etc.
INI files are easily generated and parsed in Python with the ConfigParser module.

Of course, I'm certainly open to revisiting this decision and would be happy to go with JSON if you or others in the community feel it's a better fit.

CXXOPTS vs gflags

See the commit message for benchmark/nixlbench: switched to cxxopts commit - I discuss it at length there. Hope the explanation makes sense.

aranadive · 2025-11-13T09:55:48Z

@anton-nayshtut Thanks so much adding this. I had two general comments. Did you consider using JSON format for config file? From a usability, portability aspect its well-known, with number of tools already around it. Plus, we can reuse it for kvbench since json is easily parsed in python. Another option, would be to use the flagfile option of Gflags - to pass all the values/parameters from command line into the file. This would be easier than json or ini since we wouldnt have to go through licensing again.
For CXXOPTS, Im unclear why this was required? If you use the gflags flagfile now, any other options after the file should overwrite the ones in the file.
WDYT?
Hi Adit. Thanks for your review!

As for the questions:

INI vs JSON

Sure, I considered using JSON for config files and I agree that, as you mentioned, it has some advantages. However, JSON also has some downsides that led me to prefer INI format.

My thoughts were along these lines:

INI format is often a better choice than JSON for simple configuration files because it’s more approachable, easier to manually edit, and visually cleaner for basic use cases.

INI files are highly readable and easy to work with - especially for non-programmers or for quick manual edits. They support organizing settings into basic sections and key-value pairs, so simple configs stay tidy and quick to review. Comments are supported in INI (unlike JSON by default), letting you explain settings directly in the file, which is a huge advantage for documentation and collaboration. INI files are much simpler than JSON, and avoid extra punctuation, strict syntax, and nested requirements - reducing the chance of errors when hand-editing.

In summary, if your configs are flat, small, and likely to be edited by hand, INI is much easier for users to read and update than JSON. JSON is better when you need complex or hierarchical data, but INI shines for basic, human-friendly settings.

Some additional considerations:

Many popular test and benchmark tools use INI format for config files, like fio, tox, bdperf, etc.

INI files are easily generated and parsed in Python with the ConfigParser module.

Of course, I'm certainly open to revisiting this decision and would be happy to go with JSON if you or others in the community feel it's a better fit.

CXXOPTS vs gflags

See the commit message for benchmark/nixlbench: switched to cxxopts commit - I discuss it at length there. Hope the explanation makes sense.

To me it looks like INI are an upgraded version from just using a flagfile with gflags (https://gflags.github.io/gflags/#special). That also allows us to specify comments, and just copy gflags into it as they are today (no need for extra deps). We are not changing any parameters anyways so this would be the easiest option.

CXXOPTS vs GFlags
In terms of priority GFLAGS can already give this if the flagfile is specified last. Anything in the flagfile will overrule the commandline one and if no value is specified the default is taken. Based on the code, you need to explicitly parse ini file then import to cxxopts. If we stick with gflags flagfile this should be handled automatically.

I didnt understand the advantage of dynamic arguments, since the file parsing and command line parsing are handled separately. Wouldn't you still need logic to match the ini parsed value to cxxopts? Wouldn't these be interpreted separately, i.e. task0.local and task0.remote are considered two different parameters without any hierarchy?

There is also TOML (its made for config files) which is a cross between INI and JSON files. That might be a nicer option over INI to use when multi-tests can be run to give mroe flexibility in specifying config.

anton-nayshtut · 2025-11-20T11:56:52Z

To me it looks like INI are an upgraded version from just using a flagfile with gflags (https://gflags.github.io/gflags/#special). That also allows us to specify comments, and just copy gflags into it as they are today (no need for extra deps). We are not changing any parameters anyways so this would be the easiest option.

Oh, perfect! I wasn’t aware of the --flagfile support - thanks for pointing that out.

Just a small note: gflags is already a dependency, so with my patch it’s actually being replaced, not simply added as an extra dependency.

CXXOPTS vs GFlags In terms of priority GFLAGS can already give this if the flagfile is specified last. Anything in the flagfile will overrule the commandline one and if no value is specified the default is taken. Based on the code, you need to explicitly parse ini file then import to cxxopts. If we stick with gflags flagfile this should be handled automatically.

Yeah, this is definitely an advantage of gflags.

However, my original idea was that if a user defines an option in a config file but also specifies it on the command line, the command line value would override the config file - whereas gflags lets the value in the flagfile override the command line one (depending on argument order). I realize now that gflags logic could also be fine; it’s just a matter of what we decide.

I didnt understand the advantage of dynamic arguments, since the file parsing and command line parsing are handled separately. Wouldn't you still need logic to match the ini parsed value to cxxopts? Wouldn't these be interpreted separately, i.e. task0.local and task0.remote are considered two different parameters without any hierarchy?

My intent was that we could support both dynamically defined config sections (like in fio config) and command-line overrides for the same parameters. For example, you could define a job in a config file (my.conf) like:

[example_job1]
backend=POSIX
posix_api_type=AIO
filepath=/mnt/test1
max_batch_size=128
start_batch_size=128
op_type=READ

Then, parameters could be overridden from the command line like so:

./nixlbench --config_file my.conf --example_job1.op_type=WRITE

We don't need this now, so I probably overstepped. I mostly wanted an infrastructure flexible enough to handle any scenario we might want in the future. :-)

There is also TOML (its made for config files) which is a cross between INI and JSON files. That might be a nicer option over INI to use when multi-tests can be run to give mroe flexibility in specifying config.

I'm perfectly fine with TOML as well, and I’d be happy to rewrite everything in that format if you think it’s better.

Let’s decide whether we want to keep using gflags for now (not prioritizing command-line options), move to INI, or switch to another format like TOML.

@aranadive , @benlwalker

Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

gflags-specific parameter definitions have been replaced with the NB_ARG_x macros. This will help minimize changes when we move to a different parameter parsing infrastructure in upcoming patches. This patch doesn't change any logic. Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

NB_ARG parameter getter macro introduced. This will help us to minimize changes when we'll move to a different parameter parsing infrastructure in upcoming patches. This patch doesn't change any logic. Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

We'll use cxxopts for command line parsing in upcoming patches. Copied from https://github.com/jarro2783/cxxopts SHA: 8df9a4d2716ebf1c68eced0b6d9de185c6cfd50a Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

This patch replaces gflags-based parameter parsing with cxxopts-based parsing, and removes the gflags dependency. This change will enable us to support config file-based parameters in upcoming patches. Unfortunately, gflags is not flexible enough for our needs. For example, it does not indicate whether a parameter was explicitly provided by the user or if it is using a default. This functionality is required for correct parameter specification priority: first, a value provided by the user on the command line; then a value from the config file; and finally, the default value. Additionally, gflags does not support dynamically formed arguments, which is needed to handle a list of tasks as discussed. For this use case, the config file would look like: [general] ... num_tasks=2 [task0] local=... remote=... type=write ... [task1] local=... remote=... type=read ... This translates to command line parameters like: --num_tasks=2 --task0.local=... ... --task1.local=... Since gflags is a compile-time mechanism, it cannot support this level of dynamic argument handling. Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

We'll use ini-cpp for config file parsing in upcoming patches. Copied from https://github.com/SSARCandy/ini-cpp/ SHA: 513c0947477855d3925446084779697145246714 Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

iyastreb · 2025-11-22T12:05:11Z

benchmark/nixlbench/src/utils/utils.h

+    loadParams(cxxopts::ParseResult &result);
+    template<class T>
+    static T
+    getParamValue(std::unique_ptr<inih::INIReader> &ini,


You rarely need to pass unique_ptr by non-const pointer, unless you modify it inside.
If object must exist - pass it by reference
If object might be nullptr - pass it by pointer

name can be std::string_view

const - Thanks! Done!
name - I'm not sure I get it. Is there something wrong with const char \"?

iyastreb · 2025-11-22T12:06:17Z

benchmark/nixlbench/src/utils/utils.cpp

+        try {
+            return ini->Get<T>("global", name);
+        }
+        catch (std::runtime_error &) {


better catch exceptions by const ref
Why not printing an error to std::cerr?

const - Thanks a lot! Fixed!
error - it's not a real error. It just means that a specific value is not present in the config file.

iyastreb · 2025-11-22T12:08:01Z

benchmark/nixlbench/src/utils/utils.cpp

+        try {
+            ini = std::make_unique<inih::INIReader>(config_file);
+        }
+        catch (std::runtime_error &e) {


Thanks! Fixed!

iyastreb · 2025-11-22T12:09:24Z

benchmark/nixlbench/src/utils/utils.cpp

+xferBenchConfig::loadParams(cxxopts::ParseResult &result) {
+    std::unique_ptr<inih::INIReader> ini;
+
+    if (result.count("config_file")) {


maybe define constants for these string literals?

config_file is the only special-case string literal, and it's only being treated specially in these two consecutive lines of code. So, I thought defining a constant in this case would not be necessary. Please comment.

iyastreb · 2025-11-22T12:12:09Z

benchmark/nixlbench/src/utils/utils.cpp

+    return result[name].as<T>();
+}
+
+int


if function returns only 0/-1 - maybe use bool as a return type?

Good question. In the current code, the xferBenchConfig::loadFromFlags function that has this prototype, and I just kept it that way to minimize the diff.

iyastreb · 2025-11-22T12:32:46Z

benchmark/nixlbench/src/utils/external/ini.h

+ * @throws std::runtime_error if there is an error parsing the INI file
+ */
+inline INIReader::INIReader(FILE* file) {
+    _error = ini_parse_file(file, ValueHandler, this);


it's a bit weird design to store an error in tmp variable
why not just throw an exception from ini_parse_file?

iyastreb · 2025-11-22T12:34:34Z

benchmark/nixlbench/src/utils/external/ini.h

+                     const T& v);
+
+    template <typename T = std::string>
+    void UpdateEntry(const std::string& section, const std::string& name,


according to our policy methods should be named with camelCase.
You can check the entire change with clang-tidy:

git diff -U0 HEAD~1 | clang-tidy-diff-19.py -clang-tidy-binary=clang-tidy-19 -p1 -path=build-x86_64

iyastreb · 2025-11-22T12:37:44Z

benchmark/nixlbench/src/utils/external/ini.h

+ * @return The map representing the values in the given section
+ * @throws std::runtime_error if the section is not found
+ */
+inline const std::unordered_map<std::string, std::string> INIReader::Get(


no need to make a copy of the map I guess

iyastreb · 2025-11-22T12:39:16Z

benchmark/nixlbench/src/utils/external/ini.h

+                                   const std::string& name, const T& v) {
+    if (!_values[section][name].size()) {
+        throw std::runtime_error("key '" + std::string(name) +
+                                 "' not exist in section '" + section + "'.");


Suggested change

"' not exist in section '" + section + "'.");

"' does not exist in section '" + section + "'.");

iyastreb · 2025-11-22T12:41:54Z

benchmark/nixlbench/src/utils/external/ini.h

+inline void INIReader::UpdateEntry(const std::string& section,
+                                   const std::string& name,
+                                   const std::vector<T>& vs) {
+    if (!_values[section][name].size()) {


since it's used in several places - maybe create a function checkDuplicate

This patch introduces config file support to nixlbench. The name of a config file can be specified using the --config_file command-line parameter. The config file is in INI format. Each existing command-line parameter can also be placed in the "[global]" section of the configuration file, so the following invocations: nixlbench --etcd_endpoints http://localhost:2379 --backend POSIX --filepath /mnt/test --posix_api_type AIO and nixlbench --config_file /tmp/nixlbench.config where /tmp/nixlbench.config contains: [global] etcd_endpoints=http://localhost:2379 backend=POSIX filepath=/mnt/test posix_api_type=AIO are identical. If a parameter exists in the config file and is also explicitly specified on the command line, the latter takes precedence. Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

anton-nayshtut requested review from a team, aranadive, brminich and ovidiusm as code owners November 4, 2025 08:52

pull-request-size bot added the size/XXL label Nov 4, 2025

github-actions bot added the external-contribution label Nov 4, 2025

aranadive requested a review from vvenkates27 November 4, 2025 09:39

anton-nayshtut force-pushed the antonn/nixlbench_ini branch from 26925b6 to 86ac305 Compare November 6, 2025 11:33

anton-nayshtut requested review from a team as code owners November 6, 2025 11:33

anton-nayshtut force-pushed the antonn/nixlbench_ini branch from 80c13c4 to 8a829d4 Compare November 6, 2025 11:51

vvenkates27 reviewed Nov 11, 2025

View reviewed changes

anton-nayshtut force-pushed the antonn/nixlbench_ini branch from 8a829d4 to d83a3df Compare November 20, 2025 15:31

anton-nayshtut added 6 commits November 20, 2025 17:39

nixlbench: utils autoformatted by clang

c51a6e7

Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

benchmark/nixlbench: cxxopts added

84d5ec6

We'll use cxxopts for command line parsing in upcoming patches. Copied from https://github.com/jarro2783/cxxopts SHA: 8df9a4d2716ebf1c68eced0b6d9de185c6cfd50a Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

benchmark/nixlbench: ini-cpp added

391d897

We'll use ini-cpp for config file parsing in upcoming patches. Copied from https://github.com/SSARCandy/ini-cpp/ SHA: 513c0947477855d3925446084779697145246714 Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

anton-nayshtut force-pushed the antonn/nixlbench_ini branch from d83a3df to fbfda41 Compare November 20, 2025 16:32

iyastreb reviewed Nov 22, 2025

View reviewed changes

anton-nayshtut added 3 commits November 23, 2025 10:18

copyright-check.sh: nixlbench external dir excluded

37af844

Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

clang-format.yml: nixlbench external dir excluded

e3d710c

Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>

anton-nayshtut force-pushed the antonn/nixlbench_ini branch from fbfda41 to e3d710c Compare November 23, 2025 08:19

	"' not exist in section '" + section + "'.");
	"' does not exist in section '" + section + "'.");

benchmark/nixlbench: --config_file param supported #989

Are you sure you want to change the base?

benchmark/nixlbench: --config_file param supported #989

Uh oh!

Conversation

anton-nayshtut commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Nov 4, 2025

Uh oh!

github-actions bot commented Nov 4, 2025

Uh oh!

aranadive commented Nov 11, 2025

Uh oh!

vvenkates27 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anton-nayshtut commented Nov 11, 2025

Uh oh!

aranadive commented Nov 13, 2025

Uh oh!

anton-nayshtut commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anton-nayshtut Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anton-nayshtut commented Nov 4, 2025 •

edited

Loading

anton-nayshtut Nov 23, 2025 •

edited

Loading