2d vs. 1d array mismatch when running compare-results.js

[drewrip]

I was trying to compare 3 iterations I ran with crucible using compare-results.js. I was running into the error below:

]# crucible wrapper "node /opt/crucible/subprojects/core/CommonDataModel/queries/cdmq/compare-results.js --add-runs 93577fef-4cc1-4709-b24a-c8e2bac10c20"
*** NOTICE: Crucible is at least 47 commits behind.  See 'crucible repo info' for details. ***

Get all iterations from 0 tag filters
Get all iterations from 0 param filters
Total iterations: 3
Finding all tag names
allTagNames:
[
  "topology"
]
Finding all param args
allParamArgs:
[
  "repo_tag",
  "config_dataset_max_queries",
  "config_dataset_max_input_tokens",
  "config_load_options_concurrency",
  "config_output_dir",
  "config_storage_type",
  "config_dataset_file",
  "config_load_options_type",
  "config_dataset_min_input_tokens",
  "config_dataset_max_sequence_tokens",
  "config_plugin",
  "config_plugin_options_streaming",
  "config_plugin_options_endpoint",
  "config_dataset_min_output_tokens",
  "config_plugin_options_use_tls",
  "config_plugin_options_model_name",
  "config_output_format",
  "config_output_file",
  "config_dataset_max_output_tokens",
  "config_plugin_options_host",
  "config_load_options_duration",
  "config_extra_metadata_replicas"
]
Building param and tag look-up tables
Finding only the tag names which are present in all iterations
Finding all different values for each tag and param
allIterIds.length: 3
mgetIterMetrics
[ 'llm::throughput', 'llm::throughput', 'llm::throughput' ]
/opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/cdm.js:34
    b.forEach((e) => {
      ^

TypeError: b.forEach is not a function
    at /opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/cdm.js:34:7
    at Array.forEach (<anonymous>)
    at consolidateAllArrays (/opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/cdm.js:33:5)
    at mgetIterMetrics (/opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/cdm.js:564:24)
    at getIters (/opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/cdm.js:1431:17)
    at Object.<anonymous> (/opt/crucible/repos/https:github.com:perftool-incubator/CommonDataModel/queries/cdmq/compare-results.js:86:16)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1427:10)
    at Module.load (node:internal/modules/cjs/loader:1206:32)
    at Module._load (node:internal/modules/cjs/loader:1022:12)

Node.js v20.12.2

This version of CommonDataModel is 3d50b8d (2025.1).

What seems to be happening, is that consolidateAllArrays requires a 2d array, however mgetPrimaryMetric is only providing it with a 1d array. There seems in general to be a discrepancy with which mget... functions return a 2d or 1d array. Some of them explicitly flatten their output from mSearch and others don't. I'm not sure of what the design is for this code, maybe they should all return the same type. But for now, my simple solution is to make consolidateAllArrays generic for an N-dimensional array:

function consolidateAllArrays(a) {
  const c = [];
  a.forEach((b) => {
    if(Array.isArray(b)){
      consolidateAllArrays(b).forEach((d) => {
        if (!c.includes(d)) c.push(d);
      });
    } else {
      if (!c.includes(b)) c.push(b);
    }
  });
  return c;
}

[atheurer]

Well, I have to admit that I did not expect anyone to use this script :) It was very much a work in progress from maybe 3 or 4 years ago. But I am glad you are looking at it! I can't recall why I would have some mget* returning 1D vs 2D array, unless a query for a single iteration returned an array, and therefore a query with multiple iterations would yield a 2D array. Making consolidateAllArrays generic seems fine.

[drewrip]

I was hoping to give compare-results to folks in PSAP as a simple way of viewing the results of all of the iterations in their run. I think this seems like it would be the easiest way to do so. It'd be great to take advantage of the whole ecosystem that's already in place. I kept digging into some related issues that popped up after the problem I described above, but I'll have to put it on the back burner as I still can't get it to work correctly. I keep running into random bugs (most of them seemingly related to providing 2d vs 1d arrays in different places). Hopefully I can find a minute to get this working though, because I think it's a nice alternative to having to manually extract the metrics you're interested in with PPL or SQL.

[atheurer]

Yes, I completely agree. Maybe I can look at it later this week.