diff --git a/docs/groovy/reference/table-operations/create/rollup.md b/docs/groovy/reference/table-operations/create/rollup.md
index 87966bbbb89..56779ad74f6 100644
--- a/docs/groovy/reference/table-operations/create/rollup.md
+++ b/docs/groovy/reference/table-operations/create/rollup.md
@@ -28,6 +28,8 @@ The following aggregations are supported:
- [`AggCountDistinct`](../group-and-aggregate/AggCountDistinct.md)
- [`AggCountWhere`](../group-and-aggregate/AggCountWhere.md)
- [`AggFirst`](../group-and-aggregate/AggFirst.md)
+- [`AggFormula`](../group-and-aggregate/AggFormula.md)
+- [`AggGroup`](../group-and-aggregate/AggGroup.md)
- [`AggLast`](../group-and-aggregate/AggLast.md)
- [`AggMax`](../group-and-aggregate/AggMax.md)
- [`AggMin`](../group-and-aggregate/AggMin.md)
@@ -124,6 +126,78 @@ result = source.rollup(aggList, false, "N", "M")
+## Formula Aggregations in Rollups
+
+When a rollups includes a formula aggregation, care should be taken with the function being applied. On each tick, the formula is evaluated for each changed row in the output table. Because the aggregated rows include many source rows, the input vectors to a formula aggregation can be very large (at the root level, they are the entire source table). If the formula is not efficient with large input vectors, the performance of the rollup can be poor.
+
+By default, the formula aggregation operates on a group of all the values as they appeared in the source table. In this example, the `Value` column contains the same vector that is used as input to the formula:
+
+```groovy
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 10, 10, 20, 20, 30, 30))
+simpleSum = source.rollup(List.of(AggGroup("Value"), AggFormula("Sum = sum(Value)")), "Key")
+```
+
+To calculate the sum for the root row, every row in the source table is read. The Deephaven engine provides no mechanism to provide detailed update information for a vector. Thus, every time the table ticks, the formula is completely re-evaluated.
+
+### Formula Reaggregation
+
+Formula reaggregation can be used to limit the size of input vectors while evaluating changes to a rollup. Each level of the rollup must have the same constituent types and names, which can make formulating your query more complicated.
+
+```groovy
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 10, 10, 20, 20, 30, 30))
+reaggregatedSum = source.updateView("Sum=(long)Value").rollup(List.of(AggFormula("Sum = sum(Sum)").asReaggregating()), "Key")
+```
+
+The results of `reaggregatedSum` are identical to `simpleSum`; but they are evaluated differently. In simpleSum, the source table is read twice: once to calculate the lowest-level sums, and a second time to calculate the top-level sum. In the reaggregated example, the source table is read once to calculate the lowest-level sums; and then the intermediate sums for each `Key` are read to calculate the top-level sum. If a row was added to `source` with the key `Delta`; then `simpleSum` would read that row, calculate a new sum for `Delta` and the top-level would read all eight rows of the table. The `reaggregatedSum` would similarly calculate the new sum for `Delta`, but the top-level would only read the intermediate sums for `Alpha`, `Bravo`, `Charlie`, and `Delta` instead of all eight source rows. As the number of states and the size of the input tables increase, the performance impact of evaluating a formula over all rows the table increases.
+
+In the previous example, the `Sum` column evaluated the [`sum(IntVector)`](https://docs.deephaven.io/core/javadoc/io/deephaven/function/Numeric.html#sum(io.deephaven.vector.IntVector)) function at every level of the rollup and produced a `long`. If the original table with an `int` column was used, then the lowest-level rollup would provide an `IntVector` as input to the `sum` and the next-level would provide `LongVector`. Similarly, the source table had a column named `Value`; whereas the aggregation produces a result named `Sum`. To address both these issues, before passing `source` to rollup, we called `updateView` to cast the `Value` column to `long` as `Sum`. If we ran the same example without the cast:
+
+```groovy syntax
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 10, 10, 20, 20, 30, 30))
+reaggregatedSum = source.rollup(List.of(AggFormula("Value = sum(Value)").asReaggregating()), "Key")
+```
+
+We instead get an Exception message indicating that the formula cannot be applied properly:
+
+```text
+java.lang.ClassCastException: class io.deephaven.engine.table.vectors.LongVectorColumnWrapper cannot be cast to class io.deephaven.vector.IntVector (io.deephaven.engine.table.vectors.LongVectorColumnWrapper and io.deephaven.vector.IntVector are in unnamed module of loader 'app')
+```
+
+### Formula Depth
+
+Formula aggregations may include the constant `__FORMULA_DEPTH__` column, which is the depth of the formula aggregation in the rollup tree. The root node of the rollup has a depth of 0, the next level is 1, and so on. This can be used to implement distinct aggregations at each level of the rollup. For example:
+
+```groovy
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 10, 10, 20, 20, 30, 30))
+firstThenSum = source.rollup(List.of(AggFormula("Value = __FORMULA_DEPTH__ == 0 ? sum(Value) : first(Value)")), "Key")
+```
+
+In this case, for each value of `Key`, the aggregation returns the first value. For the root level, the aggregation returns the sum of all values. When combined with a reaggregating formula, even more interesting semantics are possible. For example, rather than summing all of the values; we can sum the values from the prior level:
+
+```groovy
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 10, 10, 20, 20, 30, 30))
+firstThenSum = source.updateView("Value=(long)Value").rollup(List.of(AggFormula("Value = __FORMULA_DEPTH__ == 0 ? sum(Value) : first(Value)").asReaggregating()), "Key")
+```
+
+Another simple example of reaggration is a capped sum. In this example, the sums below the root level are capped at 40:
+
+```groovy
+source = newTable(
+ stringCol("Key", "Alpha", "Alpha", "Alpha", "Bravo", "Bravo", "Charlie", "Charlie"),
+ intCol("Value", 10, 20, 15, 20, 15, 25, 35))
+cappedSum = source.updateView("Value=(long)Value").rollup(List.of(AggFormula("Value = __FORMULA_DEPTH__ == 0 ? sum(Value) : min(sum(Value), 40)").asReaggregating()), "Key")
+```
+
## Related documentation
- [`AggAvg`](../group-and-aggregate/AggAvg.md)
diff --git a/docs/groovy/reference/table-operations/group-and-aggregate/AggFormula.md b/docs/groovy/reference/table-operations/group-and-aggregate/AggFormula.md
index d82a335f7bf..bec64775953 100644
--- a/docs/groovy/reference/table-operations/group-and-aggregate/AggFormula.md
+++ b/docs/groovy/reference/table-operations/group-and-aggregate/AggFormula.md
@@ -7,6 +7,7 @@ title: AggFormula
## Syntax
```
+AggFormula(formula)
AggFormula(formula, paramToken, columnNames...)
```
@@ -21,19 +22,21 @@ The user-defined formula to apply to each group. This formula can contain:
- Mathematical operations such as `*`, `+`, `/`, etc.
- [User-defined closures](../../../how-to-guides/groovy-closures.md)
-If `paramToken` is not `null`, the formula can only be applied to one column at a time, and it is applied to the specified `paramToken`. If `paramToken` is `null`, the formula is applied to any column or literal value present in the formula. The use of `paramToken` is deprecated.
+The formula is typically specified as `OutputColumn = Expression` (when `paramToken` is `null`). The formula is applied to any column or literal value present in the formula. For example, `Out = KeyColumn * max(ValueColumn)`, produces an output column with the name `Out` and uses `KeyColumn` and `ValueColumn` as inputs.
+
+If `paramToken` is not `null`, the formula can only be applied to one column at a time, and it is applied to the specified `paramToken`. In this case, the formula does not supply an output column, but rather it is derived from the `columnNames` parameter. The use of `paramToken` is deprecated.
Key column(s) can be used as input to the formula. When this happens, key values are treated as scalars.
-The parameter name within the formula. If `paramToken` is `each`, then `formula` must contain `each`. For example, `max(each)`, `min(each)`, etc. Use of this parameter is deprecated.
+The parameter name within the formula. If `paramToken` is `each`, then `formula` must contain `each`. For example, `max(each)`, `min(each)`, etc. Use of this parameter is deprecated. A non-null value is not permitted in [rollups](../create/rollup.md).
-The source column(s) for the calculations.
+The source column(s) for the calculations. The source column names are only used when `paramToken` is not `null`, and are thus similarly deprecated.
- `"X"` applies the formula to each value in the `X` column for each group.
- `"Y = X"` applies the formula to each value in the `X` column for each group and renames it to `Y`.
diff --git a/docs/groovy/snapshots/147d1e4ef58979040021c4e15859a139.json b/docs/groovy/snapshots/147d1e4ef58979040021c4e15859a139.json
new file mode 100644
index 00000000000..7a63af13972
--- /dev/null
+++ b/docs/groovy/snapshots/147d1e4ef58979040021c4e15859a139.json
@@ -0,0 +1 @@
+{"file":"reference/table-operations/create/rollup.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"int"}],"rows":[[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}],[{"value":"Charlie"},{"value":"30"}]]}},"reaggregatedSum":{"type":"HierarchicalTable","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Sum","type":"long"}],"rows":[[{"value":""},{"value":"130"}],[{"value":"Alpha"},{"value":"30"}],[{"value":"Bravo"},{"value":"40"}],[{"value":"Charlie"},{"value":"60"}]],"rowDepths":[1,2,2,2]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/74a7db0b5abe5682f828a576c69e2222.json b/docs/groovy/snapshots/74a7db0b5abe5682f828a576c69e2222.json
deleted file mode 100644
index 25793bcdd4b..00000000000
--- a/docs/groovy/snapshots/74a7db0b5abe5682f828a576c69e2222.json
+++ /dev/null
@@ -1 +0,0 @@
-{"file":"how-to-guides/excel/excel-client.md","objects":{"static_table":{"type":"Table","data":{"columns":[{"name":"X","type":"int"}],"rows":[[{"value":"5"}],[{"value":"2"}],[{"value":"9"}],[{"value":"8"}],[{"value":"1"}],[{"value":"7"}],[{"value":"4"}],[{"value":"8"}],[{"value":"6"}],[{"value":"9"}]]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/a79ff17b43af37762a7fc93060ed494e.json b/docs/groovy/snapshots/a79ff17b43af37762a7fc93060ed494e.json
new file mode 100644
index 00000000000..4abb4083908
--- /dev/null
+++ b/docs/groovy/snapshots/a79ff17b43af37762a7fc93060ed494e.json
@@ -0,0 +1 @@
+{"file":"reference/table-operations/create/rollup.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"int"}],"rows":[[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}],[{"value":"Charlie"},{"value":"30"}]]}},"simpleSum":{"type":"HierarchicalTable","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"io.deephaven.vector.IntVector"},{"name":"Sum","type":"long"}],"rows":[[{"value":""},{"value":"10,10,10,20,20,30,30"},{"value":"130"}],[{"value":"Alpha"},{"value":"10,10,10"},{"value":"30"}],[{"value":"Bravo"},{"value":"20,20"},{"value":"40"}],[{"value":"Charlie"},{"value":"30,30"},{"value":"60"}]],"rowDepths":[1,2,2,2]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/db3562b0dc23b199a91d0762dad1ac03.json b/docs/groovy/snapshots/db3562b0dc23b199a91d0762dad1ac03.json
deleted file mode 100644
index d22df14b1c1..00000000000
--- a/docs/groovy/snapshots/db3562b0dc23b199a91d0762dad1ac03.json
+++ /dev/null
@@ -1 +0,0 @@
-{"file":"how-to-guides/excel/excel-client.md","objects":{"crypto_table":{"type":"Table","data":{"columns":[{"name":"Timestamp","type":"java.time.Instant"},{"name":"Exchange","type":"java.lang.String"},{"name":"Price","type":"double"},{"name":"Size","type":"double"}],"rows":[]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/df5a083869378962413e5e1ec88361d6.json b/docs/groovy/snapshots/df5a083869378962413e5e1ec88361d6.json
new file mode 100644
index 00000000000..3ddb3f43ca3
--- /dev/null
+++ b/docs/groovy/snapshots/df5a083869378962413e5e1ec88361d6.json
@@ -0,0 +1 @@
+{"file":"reference/table-operations/create/rollup.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"int"}],"rows":[[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"20"}],[{"value":"Alpha"},{"value":"15"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Bravo"},{"value":"15"}],[{"value":"Charlie"},{"value":"25"}],[{"value":"Charlie"},{"value":"35"}]]}},"cappedSum":{"type":"HierarchicalTable","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"long"}],"rows":[[{"value":""},{"value":"115"}],[{"value":"Alpha"},{"value":"40"}],[{"value":"Bravo"},{"value":"35"}],[{"value":"Charlie"},{"value":"40"}]],"rowDepths":[1,2,2,2]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/f1ee5c94013279dcc75428cd1e365ea4.json b/docs/groovy/snapshots/f1ee5c94013279dcc75428cd1e365ea4.json
new file mode 100644
index 00000000000..a532e456acd
--- /dev/null
+++ b/docs/groovy/snapshots/f1ee5c94013279dcc75428cd1e365ea4.json
@@ -0,0 +1 @@
+{"file":"reference/table-operations/create/rollup.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"int"}],"rows":[[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}],[{"value":"Charlie"},{"value":"30"}]]}},"firstThenSum":{"type":"HierarchicalTable","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"long"}],"rows":[[{"value":""},{"value":"130"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}]],"rowDepths":[1,2,2,2]}}}}
\ No newline at end of file
diff --git a/docs/groovy/snapshots/f2208a297db36f58594c7ecc25d3713f.json b/docs/groovy/snapshots/f2208a297db36f58594c7ecc25d3713f.json
new file mode 100644
index 00000000000..f081a76cd49
--- /dev/null
+++ b/docs/groovy/snapshots/f2208a297db36f58594c7ecc25d3713f.json
@@ -0,0 +1 @@
+{"file":"reference/table-operations/create/rollup.md","objects":{"source":{"type":"Table","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"int"}],"rows":[[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}],[{"value":"Charlie"},{"value":"30"}]]}},"firstThenSum":{"type":"HierarchicalTable","data":{"columns":[{"name":"Key","type":"java.lang.String"},{"name":"Value","type":"long"}],"rows":[[{"value":""},{"value":"60"}],[{"value":"Alpha"},{"value":"10"}],[{"value":"Bravo"},{"value":"20"}],[{"value":"Charlie"},{"value":"30"}]],"rowDepths":[1,2,2,2]}}}}
\ No newline at end of file
diff --git a/docs/python/snapshots/74a7db0b5abe5682f828a576c69e2222.json b/docs/python/snapshots/74a7db0b5abe5682f828a576c69e2222.json
deleted file mode 100644
index 33cb2460f88..00000000000
--- a/docs/python/snapshots/74a7db0b5abe5682f828a576c69e2222.json
+++ /dev/null
@@ -1 +0,0 @@
-{"file":"core/docs/how-to-guides/excel/excel-add-in.md","objects":{"static_table":{"type":"Table","data":{"columns":[{"name":"X","type":"int"}],"rows":[[{"value":"4"}],[{"value":"7"}],[{"value":"7"}],[{"value":"6"}],[{"value":"7"}],[{"value":"9"}],[{"value":"1"}],[{"value":"2"}],[{"value":"8"}],[{"value":"4"}]]}}}}
\ No newline at end of file
diff --git a/docs/python/snapshots/db3562b0dc23b199a91d0762dad1ac03.json b/docs/python/snapshots/db3562b0dc23b199a91d0762dad1ac03.json
deleted file mode 100644
index f0726a5616d..00000000000
--- a/docs/python/snapshots/db3562b0dc23b199a91d0762dad1ac03.json
+++ /dev/null
@@ -1 +0,0 @@
-{"file":"core/docs/how-to-guides/excel/excel-client.md","objects":{"crypto_table":{"type":"Table","data":{"columns":[{"name":"Timestamp","type":"java.time.Instant"},{"name":"Exchange","type":"java.lang.String"},{"name":"Price","type":"double"},{"name":"Size","type":"double"}],"rows":[]}}}}
\ No newline at end of file
diff --git a/engine/api/src/main/java/io/deephaven/engine/table/Table.java b/engine/api/src/main/java/io/deephaven/engine/table/Table.java
index f297cda94b6..6703718ace8 100644
--- a/engine/api/src/main/java/io/deephaven/engine/table/Table.java
+++ b/engine/api/src/main/java/io/deephaven/engine/table/Table.java
@@ -567,6 +567,11 @@ CloseableIterator objectColumnIterator(@NotNull String co
*
*
*
+ * A blink table can be converted to an append-only table with
+ * {@link io.deephaven.engine.table.impl.BlinkTableTools#blinkToAppendOnly(io.deephaven.engine.table.Table)}.
+ *
+ *
+ *
* Some aggregations (in particular {@link #groupBy} and {@link #partitionBy} cannot provide the desired blink table
* aggregation semantics because doing so would require storing the entire stream of blink updates in memory. If
* that behavior is desired, use {@code blinkToAppendOnly}. If on the other hand, you would like to group or
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationContext.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationContext.java
index fd9ee394974..02f5958f67a 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationContext.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationContext.java
@@ -270,10 +270,12 @@ UnaryOperator[] initializeRefreshing(@NotNull final QueryTabl
*
* @param upstream The upstream {@link TableUpdateImpl}
* @param startingDestinationsCount The number of used destinations at the beginning of this step
+ * @param modifiedOperators an array of booleans, parallel to operators, indicating which operators were modified
*/
- void resetOperatorsForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
- for (final IterativeChunkedAggregationOperator operator : operators) {
- operator.resetForStep(upstream, startingDestinationsCount);
+ void resetOperatorsForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount,
+ final boolean[] modifiedOperators) {
+ for (int ii = 0; ii < operators.length; ii++) {
+ modifiedOperators[ii] = operators[ii].resetForStep(upstream, startingDestinationsCount);
}
}
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
index 28d9d96f8cd..9812aaf7ae6 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
@@ -33,11 +33,8 @@
import io.deephaven.base.verify.Assert;
import io.deephaven.chunk.ChunkType;
import io.deephaven.chunk.attributes.Values;
-import io.deephaven.engine.table.ChunkSource;
-import io.deephaven.engine.table.ColumnDefinition;
-import io.deephaven.engine.table.ColumnSource;
+import io.deephaven.engine.table.*;
import io.deephaven.engine.table.impl.*;
-import io.deephaven.engine.table.Table;
import io.deephaven.engine.table.impl.by.rollup.NullColumns;
import io.deephaven.engine.table.impl.by.rollup.RollupAggregation;
import io.deephaven.engine.table.impl.by.rollup.RollupAggregationOutputs;
@@ -93,10 +90,12 @@
import io.deephaven.engine.table.impl.by.ssmpercentile.SsmChunkedPercentileOperator;
import io.deephaven.engine.table.impl.select.SelectColumn;
import io.deephaven.engine.table.impl.select.WhereFilter;
+import io.deephaven.engine.table.impl.sources.IntegerSingleValueSource;
import io.deephaven.engine.table.impl.sources.ReinterpretUtils;
import io.deephaven.engine.table.impl.ssms.SegmentedSortedMultiSet;
import io.deephaven.engine.table.impl.util.freezeby.FreezeByCountOperator;
import io.deephaven.engine.table.impl.util.freezeby.FreezeByOperator;
+import io.deephaven.qst.type.IntType;
import io.deephaven.time.DateTimeUtils;
import io.deephaven.util.annotations.FinalDefault;
import io.deephaven.util.type.ArrayTypeUtils;
@@ -201,12 +200,14 @@ public static AggregationContextFactory forRollupBase(
* mutated by {@link AggregationProcessor}.
* @param nullColumns Map of group-by column names and data types to aggregate with a null-column aggregation
* @param rollupColumn the name of the rollup column in the result, used to traverse to the next lower level nodes
+ * @param source the original source table of the rollup (not the table we are reaggregating)
* @return The {@link AggregationContextFactory}
*/
public static AggregationContextFactory forRollupReaggregated(
@NotNull final Collection aggregations,
@NotNull final Collection> nullColumns,
- @NotNull final ColumnName rollupColumn) {
+ @NotNull final ColumnName rollupColumn,
+ @NotNull final Table source) {
if (aggregations.stream().anyMatch(agg -> agg instanceof Partition)) {
rollupUnsupported("Partition");
}
@@ -214,7 +215,7 @@ public static AggregationContextFactory forRollupReaggregated(
reaggregations.add(RollupAggregation.nullColumns(nullColumns));
reaggregations.addAll(aggregations);
reaggregations.add(Partition.of(rollupColumn));
- return new AggregationProcessor(reaggregations, Type.ROLLUP_REAGGREGATED);
+ return new WithSource(reaggregations, Type.ROLLUP_REAGGREGATED, source);
}
/**
@@ -263,6 +264,8 @@ public static AggregationContextFactory forSelectDistinct() {
public static final ColumnName EXPOSED_GROUP_ROW_SETS = ColumnName.of("__EXPOSED_GROUP_ROW_SETS__");
+ public static final ColumnName ROLLUP_FORMULA_DEPTH = ColumnName.of("__FORMULA_DEPTH__");
+
/**
* Create a trivial {@link AggregationContextFactory} to {@link Aggregation#AggGroup(String...) group} the input
* table and expose the group {@link io.deephaven.engine.rowset.RowSet row sets} as {@link #EXPOSED_GROUP_ROW_SETS}.
@@ -732,6 +735,154 @@ final void addCountWhereOperator(@NotNull CountWhere countWhere) {
addOperator(new CountWhereOperator(countWhere.column().name(), whereFilters, recorders, filterRecorders),
null, inputColumnNames);
}
+
+ /**
+ * @return the index of an existing group by operator, or -1 if no operator was found
+ */
+ int existingGroupByOperatorIndex() {
+ for (int ii = 0; ii < operators.size(); ++ii) {
+ if (operators.get(ii) instanceof GroupByChunkedOperator) {
+ return ii;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * @return the index of an existing group by reaggregation operator, or -1 if no operator was found
+ */
+ int existingGroupByReaggregateIndex() {
+ for (int ii = 0; ii < operators.size(); ++ii) {
+ if (operators.get(ii) instanceof GroupByReaggregateOperator) {
+ return ii;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Ensures that the existing GroupByChunkedOperator has the required input/output columns
+ *
+ * @param createExtraPairs When true, create all of the pairs for the group by operator. When false, if there
+ * are any inputs that match the pairs we'll pass the operator through. The AggGroup aggregation can't
+ * just tack pairs onto an existing operator, because the order would be incorrect. Formulas in a rollup
+ * don't expose the results of the shared grouping, so just tacking them on is fine.
+ * @param hideExtras true if the extra pairs should be hidden from results, false otherwise
+ */
+ GroupByChunkedOperator ensureGroupByOperator(final QueryTable table,
+ final int existingOperatorIndex,
+ final String exposeRowSetAs,
+ final MatchPair[] matchPairs,
+ final boolean createExtraPairs,
+ final boolean hideExtras) {
+ boolean recreate = false;
+ final GroupByChunkedOperator existing = (GroupByChunkedOperator) operators.get(existingOperatorIndex);
+ if (exposeRowSetAs != null) {
+ if (existing.getExposedRowSetsAs() == null) {
+ recreate = true;
+ } else {
+ if (!existing.getExposedRowSetsAs().equals(exposeRowSetAs)) {
+ throw new UnsupportedOperationException(
+ "AggGroupBy cannot have inconsistent exposed row redirections names: " +
+ existing.getExposedRowSetsAs() + " != " + exposeRowSetAs);
+ }
+ }
+ }
+ final List newPairs = new ArrayList<>(Arrays.asList(existing.getAggregatedColumnPairs()));
+ List existingHidden = existing.getHiddenResults();
+ final List hiddenResults = new ArrayList<>(existingHidden == null ? List.of() : existingHidden);
+ for (MatchPair matchPair : matchPairs) {
+ final String input = matchPair.input().name();
+ if (Arrays.stream(existing.getAggregatedColumnPairs()).noneMatch(p -> p.input().name().equals(input))) {
+ // we didn't have this in the input at all
+ newPairs.add(matchPair);
+ hiddenResults.add(matchPair.output().name());
+ recreate = true;
+ } else if (createExtraPairs
+ && Arrays.stream(existing.getAggregatedColumnPairs()).noneMatch(p -> p.equals(matchPair))) {
+ newPairs.add(matchPair);
+ if (hideExtras) {
+ hiddenResults.add(matchPair.output().name());
+ }
+ recreate = true;
+ }
+ }
+ if (!recreate) {
+ // we're totally satisfied with the existing operator for use with a secondary operator that pulls an
+ // output from it to the desired name
+ return existing;
+ }
+
+ final String newExposeRowsetName = exposeRowSetAs == null ? existing.getExposedRowSetsAs() : exposeRowSetAs;
+ final MatchPair[] newMatchPairArray = newPairs.toArray(MatchPair[]::new);
+ final GroupByChunkedOperator newOperator =
+ new GroupByChunkedOperator(table, true, newExposeRowsetName, hiddenResults, newMatchPairArray);
+
+ // any formula operators that used the old group by operator must be updated
+ for (IterativeChunkedAggregationOperator operator : operators) {
+ if (operator instanceof FormulaMultiColumnChunkedOperator) {
+ ((FormulaMultiColumnChunkedOperator) operator).updateGroupBy(newOperator, false);
+ } else if (operator instanceof FormulaChunkedOperator) {
+ ((FormulaChunkedOperator) operator).updateGroupBy(newOperator, false);
+ }
+ }
+
+ operators.set(existingOperatorIndex, newOperator);
+ return newOperator;
+ }
+
+ GroupByReaggregateOperator ensureGroupByReaggregateOperator(final QueryTable table,
+ final int existingOperatorIndex,
+ final String exposeRowSetAs,
+ final MatchPair[] matchPairs) {
+ boolean recreate = false;
+ final GroupByReaggregateOperator existing =
+ (GroupByReaggregateOperator) operators.get(existingOperatorIndex);
+ if (exposeRowSetAs != null) {
+ if (existing.getExposedRowSetsAs() == null) {
+ recreate = true;
+ } else {
+ if (!existing.getExposedRowSetsAs().equals(exposeRowSetAs)) {
+ throw new UnsupportedOperationException(
+ "AggGroupBy cannot have inconsistent exposed row redirections names: " +
+ existing.getExposedRowSetsAs() + " != " + exposeRowSetAs);
+ }
+ }
+ }
+ final List newPairs = new ArrayList<>(Arrays.asList(existing.getAggregatedColumnPairs()));
+ List existingHidden = existing.getHiddenResults();
+ final List hiddenResults = new ArrayList<>(existingHidden == null ? List.of() : existingHidden);
+ for (MatchPair matchPair : matchPairs) {
+ final String input = matchPair.input().name();
+ if (Arrays.stream(existing.getAggregatedColumnPairs()).noneMatch(p -> p.input().name().equals(input))) {
+ newPairs.add(matchPair);
+ hiddenResults.add(matchPair.output().name());
+ recreate = true;
+ }
+ }
+ if (!recreate) {
+ // we're totally satisfied with the existing operator for use with a secondary operator that pulls an
+ // output from it to the desired name
+ return existing;
+ }
+
+ final String newExposeRowsetName = exposeRowSetAs == null ? existing.getExposedRowSetsAs() : exposeRowSetAs;
+ final MatchPair[] newMatchPairArray = newPairs.toArray(MatchPair[]::new);
+ final GroupByReaggregateOperator newOperator =
+ new GroupByReaggregateOperator(table, true, newExposeRowsetName, hiddenResults, newMatchPairArray);
+
+ // any formula operators that used the old group by operator must be updated
+ for (IterativeChunkedAggregationOperator operator : operators) {
+ // Only FormulaMultiColumn operators need to be adjusted, a FormulaChunkedOperator cannot participate
+ // in a rollup.
+ if (operator instanceof FormulaMultiColumnChunkedOperator) {
+ ((FormulaMultiColumnChunkedOperator) operator).updateGroupBy(newOperator, false);
+ }
+ }
+
+ operators.set(existingOperatorIndex, newOperator);
+ return newOperator;
+ }
}
// -----------------------------------------------------------------------------------------------------------------
@@ -798,21 +949,13 @@ public void visit(@NotNull final Partition partition) {
@Override
public void visit(@NotNull final Formula formula) {
+ validateFormulaIsNotReaggregating(formula);
final SelectColumn selectColumn = SelectColumn.of(formula.selectable());
// Get or create a column definition map composed of vectors of the original column types (or scalars when
// part of the key columns).
final Set groupByColumnSet = Set.of(groupByColumnNames);
- if (vectorColumnDefinitions == null) {
- vectorColumnDefinitions = table.getDefinition().getColumnStream().collect(Collectors.toMap(
- ColumnDefinition::getName,
- (final ColumnDefinition cd) -> groupByColumnSet.contains(cd.getName())
- ? cd
- : ColumnDefinition.fromGenericType(
- cd.getName(),
- VectorFactory.forElementType(cd.getDataType()).vectorType(),
- cd.getDataType())));
- }
+ maybeInitializeVectorColumns(groupByColumnSet, table.getDefinition(), Map.of());
// Get the input column names from the formula and provide them to the groupBy operator
final String[] allInputColumns =
@@ -823,20 +966,21 @@ public void visit(@NotNull final Formula formula) {
final String[] inputKeyColumns = partitioned.get(true).toArray(String[]::new);
final String[] inputNonKeyColumns = partitioned.get(false).toArray(String[]::new);
- if (!selectColumn.getColumnArrays().isEmpty()) {
- throw new IllegalArgumentException("AggFormula does not support column arrays ("
- + selectColumn.getColumnArrays() + ")");
- }
- if (selectColumn.hasVirtualRowVariables()) {
- throw new IllegalArgumentException("AggFormula does not support virtual row variables");
+ validateSelectColumnForFormula(selectColumn);
+ final GroupByChunkedOperator groupByChunkedOperator;
+ final int existingGroupByOperatorIndex = existingGroupByOperatorIndex();
+ if (existingGroupByOperatorIndex >= 0) {
+ // if we have an existing group by operator, then use it (or update it to reflect our input columns)
+ groupByChunkedOperator = ensureGroupByOperator(table, existingGroupByOperatorIndex, null,
+ makeSymmetricMatchPairs(inputNonKeyColumns), false, false);
+ } else {
+ groupByChunkedOperator =
+ makeGroupByOperatorForFormula(makeSymmetricMatchPairs(inputNonKeyColumns), table, null);
}
- // TODO: re-use shared groupBy operators (https://github.com/deephaven/deephaven-core/issues/6363)
- final GroupByChunkedOperator groupByChunkedOperator = new GroupByChunkedOperator(table, false, null,
- Arrays.stream(inputNonKeyColumns).map(col -> MatchPair.of(Pair.parse(col)))
- .toArray(MatchPair[]::new));
final FormulaMultiColumnChunkedOperator op = new FormulaMultiColumnChunkedOperator(table,
- groupByChunkedOperator, true, selectColumn, inputKeyColumns);
+ groupByChunkedOperator, existingGroupByOperatorIndex < 0, selectColumn, inputKeyColumns, null,
+ null);
addNoInputOperator(op);
}
@@ -878,8 +1022,9 @@ public void visit(@NotNull final AggSpecFirst first) {
@Override
public void visit(@NotNull final AggSpecFormula formula) {
unsupportedForBlinkTables("Formula");
- // TODO: re-use shared groupBy operators (https://github.com/deephaven/deephaven-core/issues/6363)
- final GroupByChunkedOperator groupByChunkedOperator = new GroupByChunkedOperator(table, false, null,
+ // Note: we do not attempt to reuse the groupBy operator for the deprecated "each" formula, we only reuse
+ // them for the new-style multi-column formula operators
+ final GroupByChunkedOperator groupByChunkedOperator = new GroupByChunkedOperator(table, false, null, null,
resultPairs.stream().map(pair -> MatchPair.of((Pair) pair.input())).toArray(MatchPair[]::new));
final FormulaChunkedOperator formulaChunkedOperator = new FormulaChunkedOperator(groupByChunkedOperator,
true, formula.formula(), formula.paramToken(), compilationProcessor,
@@ -895,7 +1040,18 @@ public void visit(AggSpecFreeze freeze) {
@Override
public void visit(@NotNull final AggSpecGroup group) {
unsupportedForBlinkTables("Group");
- addNoInputOperator(new GroupByChunkedOperator(table, true, null, MatchPair.fromPairs(resultPairs)));
+
+ final int existingOperator = existingGroupByOperatorIndex();
+ if (existingOperator >= 0) {
+ // Reuse the operator, adding a result extractor for the new result pairs
+ GroupByChunkedOperator existing =
+ ensureGroupByOperator(table, existingOperator, null, MatchPair.fromPairs(resultPairs), false,
+ false);
+ addNoInputOperator(existing.resultExtractor(resultPairs));
+ } else {
+ addNoInputOperator(
+ new GroupByChunkedOperator(table, true, null, null, MatchPair.fromPairs(resultPairs)));
+ }
}
@Override
@@ -972,6 +1128,72 @@ public void visit(@NotNull final AggSpecVar var) {
}
}
+ private static void validateSelectColumnForFormula(SelectColumn selectColumn) {
+ if (!selectColumn.getColumnArrays().isEmpty()) {
+ throw new IllegalArgumentException("AggFormula does not support column arrays ("
+ + selectColumn.getColumnArrays() + ")");
+ }
+ if (selectColumn.hasVirtualRowVariables()) {
+ throw new IllegalArgumentException("AggFormula does not support virtual row variables");
+ }
+ }
+
+ private static void validateFormulaIsNotReaggregating(Formula formula) {
+ if (formula.reaggregateAggregatedValues()) {
+ throw new IllegalArgumentException("AggFormula does not support reaggregating except in a rollup.");
+ }
+ }
+
+ private void maybeInitializeVectorColumns(Set groupByColumnSet, final TableDefinition definition,
+ Map> extraColumns) {
+ if (vectorColumnDefinitions != null) {
+ return;
+ }
+ vectorColumnDefinitions = new LinkedHashMap<>();
+ definition.getColumnStream().forEach(cd -> {
+ ColumnDefinition resultDefinition;
+ if (groupByColumnSet.contains(cd.getName())) {
+ resultDefinition = cd;
+ } else {
+ resultDefinition = ColumnDefinition.fromGenericType(
+ cd.getName(),
+ VectorFactory.forElementType(cd.getDataType()).vectorType(),
+ cd.getDataType());
+ }
+ vectorColumnDefinitions.put(cd.getName(), resultDefinition);
+ });
+ vectorColumnDefinitions.putAll(extraColumns);
+ }
+
+
+ private @NotNull GroupByChunkedOperator makeGroupByOperatorForFormula(final MatchPair[] pairs,
+ final QueryTable table, final String exposedRowsets) {
+ final boolean register = exposedRowsets != null;
+ return new GroupByChunkedOperator(table, register, exposedRowsets, null, pairs);
+ }
+
+ /**
+ * Convert the array of column names to MatchPairs of the form {@code Col_GRP__ROLLUP__}
+ *
+ * @param cols the columns to convert
+ * @return the mangled name matchpairs
+ */
+ private static MatchPair @NotNull [] makeMangledMatchPairs(String[] cols) {
+ return Arrays
+ .stream(cols).map(col -> new MatchPair(col + ROLLUP_GRP_COLUMN_ID + ROLLUP_COLUMN_SUFFIX, col))
+ .toArray(MatchPair[]::new);
+ }
+
+ /**
+ * Convert the array of strings to MatchPairs of the form Col=Col
+ *
+ * @param columns the columns to convert to MatchPairs
+ * @return an array of MatchPairs
+ */
+ private static MatchPair @NotNull [] makeSymmetricMatchPairs(String[] columns) {
+ return Arrays.stream(columns).map(col -> new MatchPair(col, col)).toArray(MatchPair[]::new);
+ }
+
// -----------------------------------------------------------------------------------------------------------------
// Rollup Unsupported Operations
// -----------------------------------------------------------------------------------------------------------------
@@ -994,12 +1216,6 @@ default void visit(@NotNull final LastRowKey lastRowKey) {
rollupUnsupported("LastRowKey");
}
- @Override
- @FinalDefault
- default void visit(@NotNull final Formula formula) {
- rollupUnsupported("Formula");
- }
-
// -------------------------------------------------------------------------------------------------------------
// AggSpec.Visitor for unsupported column aggregation specs
// -------------------------------------------------------------------------------------------------------------
@@ -1015,12 +1231,6 @@ default void visit(AggSpecFreeze freeze) {
rollupUnsupported("Freeze");
}
- @Override
- @FinalDefault
- default void visit(@NotNull final AggSpecGroup group) {
- rollupUnsupported("Group");
- }
-
@Override
@FinalDefault
default void visit(@NotNull final AggSpecFormula formula) {
@@ -1066,6 +1276,7 @@ private static void rollupUnsupported(@NotNull final String operationName, final
*/
private final class RollupBaseConverter extends Converter
implements RollupAggregation.Visitor, UnsupportedRollupAggregations {
+ private final QueryCompilerRequestProcessor.BatchProcessor compilationProcessor;
private int nextColumnIdentifier = 0;
@@ -1074,6 +1285,14 @@ private RollupBaseConverter(
final boolean requireStateChangeRecorder,
@NotNull final String... groupByColumnNames) {
super(table, requireStateChangeRecorder, groupByColumnNames);
+ this.compilationProcessor = QueryCompilerRequestProcessor.batch();
+ }
+
+ @Override
+ AggregationContext build() {
+ final AggregationContext resultContext = super.build();
+ compilationProcessor.compile();
+ return resultContext;
}
// -------------------------------------------------------------------------------------------------------------
@@ -1108,6 +1327,90 @@ public void visit(@NotNull final Partition partition) {
addNoInputOperator(partitionOperator);
}
+ @Override
+ public void visit(AggSpecGroup group) {
+ unsupportedForBlinkTables("Group for rollup");
+
+ final int indexOfExistingOperator = existingGroupByOperatorIndex();
+ if (indexOfExistingOperator >= 0) {
+ // share the existing operator for groupBy in a rollup base
+ final GroupByChunkedOperator existing = ensureGroupByOperator(table, indexOfExistingOperator,
+ EXPOSED_GROUP_ROW_SETS.name(), MatchPair.fromPairs(resultPairs), false, false);
+ addNoInputOperator(existing.resultExtractor(resultPairs));
+ } else {
+ addNoInputOperator(new GroupByChunkedOperator(table, true, EXPOSED_GROUP_ROW_SETS.name(),
+ null,
+ MatchPair.fromPairs(resultPairs)));
+ }
+ }
+
+ @Override
+ public void visit(Formula formula) {
+ unsupportedForBlinkTables("Formula for rollup");
+
+ final SelectColumn selectColumn = SelectColumn.of(formula.selectable());
+
+ // Get or create a column definition map composed of vectors of the original column types (or scalars when
+ // part of the key columns).
+ final Set groupByColumnSet = Set.of(groupByColumnNames);
+ // For the base of a rollup, we use the original table definition, but tack on a rollup depth column
+ maybeInitializeVectorColumns(groupByColumnSet, table.getDefinition(), Map.of(ROLLUP_FORMULA_DEPTH.name(),
+ ColumnDefinition.of(ROLLUP_FORMULA_DEPTH.name(), IntType.of())));
+
+ // Get the input column names from the formula and provide them to the groupBy operator
+ final String[] allInputColumns =
+ selectColumn.initDef(vectorColumnDefinitions, compilationProcessor).toArray(String[]::new);
+
+ final Map> partitioned = Arrays.stream(allInputColumns)
+ .collect(Collectors.partitioningBy(
+ o -> groupByColumnSet.contains(o) || ROLLUP_FORMULA_DEPTH.name().equals(o)));
+ final String[] inputKeyColumns = partitioned.get(true).toArray(String[]::new);
+ final String[] inputNonKeyColumns = partitioned.get(false).toArray(String[]::new);
+
+ validateSelectColumnForFormula(selectColumn);
+
+ final GroupByChunkedOperator groupByChunkedOperator;
+ final boolean delegate;
+
+ final int existingGroupByOperatorIndex = existingGroupByOperatorIndex();
+ final MatchPair[] mangledMatchPairs = makeMangledMatchPairs(inputNonKeyColumns);
+
+ if (formula.reaggregateAggregatedValues()) {
+ if (existingGroupByOperatorIndex >= 0) {
+ groupByChunkedOperator =
+ ensureGroupByOperator(table, existingGroupByOperatorIndex, null, mangledMatchPairs, true,
+ true);
+ delegate = false;
+ } else {
+ // When we are reaggregating, we do not expose the rowsets, because the next level creates a
+ // completely fresh operator
+ groupByChunkedOperator = makeGroupByOperatorForFormula(mangledMatchPairs, table, null);
+ // the operator is not added, so there is delegation
+ delegate = true;
+ }
+ } else {
+ if (existingGroupByOperatorIndex >= 0) {
+ groupByChunkedOperator = ensureGroupByOperator(table, existingGroupByOperatorIndex,
+ EXPOSED_GROUP_ROW_SETS.name(), mangledMatchPairs, true, false);
+ delegate = false;
+ } else {
+ // When we do not reaggregate, the next level needs access to our exposed group row sets
+ groupByChunkedOperator =
+ makeGroupByOperatorForFormula(mangledMatchPairs, table, EXPOSED_GROUP_ROW_SETS.name());
+ addNoInputOperator(groupByChunkedOperator);
+ // we added the operator, so we cannot delegate
+ delegate = false;
+ }
+ }
+
+ final IntegerSingleValueSource depthSource = new IntegerSingleValueSource();
+ depthSource.set(groupByColumnNames.length);
+
+ final FormulaMultiColumnChunkedOperator op = new FormulaMultiColumnChunkedOperator(table,
+ groupByChunkedOperator, delegate, selectColumn, inputKeyColumns, null, depthSource);
+ addNoInputOperator(op);
+ }
+
// -------------------------------------------------------------------------------------------------------------
// AggSpec.Visitor
// -------------------------------------------------------------------------------------------------------------
@@ -1215,6 +1518,7 @@ IterativeChunkedAggregationOperator apply(
private final class RollupReaggregatedConverter extends Converter
implements RollupAggregation.Visitor, UnsupportedRollupAggregations {
+ private final QueryCompilerRequestProcessor.BatchProcessor compilationProcessor;
private int nextColumnIdentifier = 0;
private RollupReaggregatedConverter(
@@ -1222,6 +1526,14 @@ private RollupReaggregatedConverter(
final boolean requireStateChangeRecorder,
@NotNull final String... groupByColumnNames) {
super(table, requireStateChangeRecorder, groupByColumnNames);
+ this.compilationProcessor = QueryCompilerRequestProcessor.batch();
+ }
+
+ @Override
+ AggregationContext build() {
+ final AggregationContext resultContext = super.build();
+ compilationProcessor.compile();
+ return resultContext;
}
// -------------------------------------------------------------------------------------------------------------
@@ -1265,6 +1577,111 @@ public void visit(@NotNull final Partition partition) {
addNoInputOperator(partitionOperator);
}
+ @Override
+ public void visit(AggSpecGroup group) {
+ final ColumnSource groupRowSet = table.getColumnSource(EXPOSED_GROUP_ROW_SETS.name());
+ final MatchPair[] pairs = new MatchPair[resultPairs.size()];
+ for (int ii = 0; ii < resultPairs.size(); ++ii) {
+ pairs[ii] = new MatchPair(resultPairs.get(ii).output().name(), resultPairs.get(ii).output().name());
+ }
+ final int existingGroupByOperatorIndex = existingGroupByReaggregateIndex();
+ if (existingGroupByOperatorIndex >= 0) {
+ final GroupByReaggregateOperator existing = ensureGroupByReaggregateOperator(table,
+ existingGroupByOperatorIndex, EXPOSED_GROUP_ROW_SETS.name(), pairs);
+ addNoInputOperator(existing.resultExtractor(resultPairs));
+ } else {
+ addOperator(new GroupByReaggregateOperator(table, true, EXPOSED_GROUP_ROW_SETS.name(), null, pairs),
+ groupRowSet,
+ EXPOSED_GROUP_ROW_SETS.name());
+ }
+ }
+
+ @Override
+ public void visit(Formula formula) {
+ final SelectColumn selectColumn = SelectColumn.of(formula.selectable());
+
+ // Get or create a column definition map composed of vectors of the original column types (or scalars when
+ // part of the key columns).
+ final Set groupByColumnSet = Set.of(groupByColumnNames);
+
+ // for a reaggregated formula, we can't use the input definition as is; we want to use the definition from
+ // the source table; but tack on our rollup depth column
+ AggregationProcessor thisProcessor = AggregationProcessor.this;
+ final TableDefinition sourceDefinition = ((WithSource) thisProcessor).source.getDefinition();
+ maybeInitializeVectorColumns(groupByColumnSet, sourceDefinition, Map.of(ROLLUP_FORMULA_DEPTH.name(),
+ ColumnDefinition.of(ROLLUP_FORMULA_DEPTH.name(), IntType.of())));
+
+ // Get the input column names from the formula and provide them to the groupBy operator
+ final String[] allInputColumns =
+ selectColumn.initDef(vectorColumnDefinitions, compilationProcessor).toArray(String[]::new);
+
+ final Map> partitioned = Arrays.stream(allInputColumns)
+ .collect(Collectors.partitioningBy(
+ o -> groupByColumnSet.contains(o) || ROLLUP_FORMULA_DEPTH.name().equals(o)));
+ final String[] inputKeyColumns = partitioned.get(true).toArray(String[]::new);
+ final String[] inputNonKeyColumns = partitioned.get(false).toArray(String[]::new);
+
+ validateSelectColumnForFormula(selectColumn);
+
+ final Map renames = new HashMap<>();
+ final MatchPair[] groupPairs = new MatchPair[inputNonKeyColumns.length];
+
+ for (int ii = 0; ii < inputNonKeyColumns.length; ++ii) {
+ final String mangledColumn = inputNonKeyColumns[ii] + ROLLUP_GRP_COLUMN_ID + ROLLUP_COLUMN_SUFFIX;
+ if (table.hasColumns(mangledColumn)) {
+ groupPairs[ii] = new MatchPair(mangledColumn, mangledColumn);
+ renames.put(mangledColumn, inputNonKeyColumns[ii]);
+ } else {
+ // reagg uses the output name
+ groupPairs[ii] = new MatchPair(mangledColumn, inputNonKeyColumns[ii]);
+ // we are not changing the input column name, so don't need the rename
+ renames.put(inputNonKeyColumns[ii], inputNonKeyColumns[ii]);
+ }
+ }
+
+ final IntegerSingleValueSource depthSource = new IntegerSingleValueSource();
+ depthSource.set(groupByColumnNames.length);
+
+ if (formula.reaggregateAggregatedValues()) {
+ GroupByChunkedOperator groupByOperator;
+
+ final int existingIndex = existingGroupByOperatorIndex();
+ if (existingIndex >= 0) {
+ groupByOperator = ensureGroupByOperator(table, existingIndex, null, groupPairs, true, true);
+ } else {
+ final List hiddenPairs =
+ Arrays.stream(groupPairs).map(mp -> mp.left().name()).collect(Collectors.toList());
+ groupByOperator = new GroupByChunkedOperator(table, false, null, hiddenPairs, groupPairs);
+ }
+
+ // everything gets hidden
+ final FormulaMultiColumnChunkedOperator op =
+ new FormulaMultiColumnChunkedOperator(table, groupByOperator,
+ true, selectColumn, inputKeyColumns, renames, depthSource);
+
+ addOperator(op, null, inputNonKeyColumns);
+ } else {
+ final ColumnSource groupRowSet = table.getColumnSource(EXPOSED_GROUP_ROW_SETS.name());
+ GroupByReaggregateOperator groupByOperator;
+
+ final int existingIndex = existingGroupByReaggregateIndex();
+ if (existingIndex >= 0) {
+ groupByOperator = ensureGroupByReaggregateOperator(table, existingIndex,
+ EXPOSED_GROUP_ROW_SETS.name(), groupPairs);
+ } else {
+ groupByOperator =
+ new GroupByReaggregateOperator(table, true, EXPOSED_GROUP_ROW_SETS.name(), null,
+ groupPairs);
+ addOperator(groupByOperator, groupRowSet, EXPOSED_GROUP_ROW_SETS.name());
+ }
+
+ final FormulaMultiColumnChunkedOperator op =
+ new FormulaMultiColumnChunkedOperator(table, groupByOperator,
+ false, selectColumn, inputKeyColumns, renames, depthSource);
+ addOperator(op, groupRowSet, EXPOSED_GROUP_ROW_SETS.name());
+ }
+ }
+
// -------------------------------------------------------------------------------------------------------------
// AggSpec.Visitor
// -------------------------------------------------------------------------------------------------------------
@@ -1583,7 +2000,7 @@ private static AggregationContext makeExposedGroupRowSetAggregationContext(
// noinspection unchecked
return new AggregationContext(
new IterativeChunkedAggregationOperator[] {
- new GroupByChunkedOperator(inputQueryTable, true, EXPOSED_GROUP_ROW_SETS.name()),
+ new GroupByChunkedOperator(inputQueryTable, true, EXPOSED_GROUP_ROW_SETS.name(), null),
new CountAggregationOperator(null)
},
new String[][] {ArrayTypeUtils.EMPTY_STRING_ARRAY, ArrayTypeUtils.EMPTY_STRING_ARRAY},
@@ -1593,7 +2010,7 @@ private static AggregationContext makeExposedGroupRowSetAggregationContext(
// noinspection unchecked
return new AggregationContext(
new IterativeChunkedAggregationOperator[] {
- new GroupByChunkedOperator(inputQueryTable, true, EXPOSED_GROUP_ROW_SETS.name())
+ new GroupByChunkedOperator(inputQueryTable, true, EXPOSED_GROUP_ROW_SETS.name(), null)
},
new String[][] {ArrayTypeUtils.EMPTY_STRING_ARRAY},
new ChunkSource.WithPrev[] {null},
@@ -2159,4 +2576,14 @@ public static AggregationRowLookup getRowLookup(@NotNull final Table aggregation
Assert.neqNull(value, "aggregation result row lookup");
return (AggregationRowLookup) value;
}
+
+ private static class WithSource extends AggregationProcessor {
+ private final @NotNull Table source;
+
+ private WithSource(@NotNull Collection aggregations, @NotNull Type type,
+ @NotNull Table source) {
+ super(aggregations, type);
+ this.source = source;
+ }
+ }
}
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/BaseBlinkFirstOrLastChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/BaseBlinkFirstOrLastChunkedOperator.java
index 9b5cee1f077..0e505e3c901 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/BaseBlinkFirstOrLastChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/BaseBlinkFirstOrLastChunkedOperator.java
@@ -98,11 +98,12 @@ public final boolean requiresRowKeys() {
@Override
@OverridingMethodsMustInvokeSuper
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
if ((redirections = cachedRedirections.get()) == null) {
cachedRedirections = new SoftReference<>(redirections = new LongArraySource());
ensureCapacity(startingDestinationsCount);
}
+ return false;
}
// -----------------------------------------------------------------------------------------------------------------
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ByteBlinkSortedFirstOrLastChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ByteBlinkSortedFirstOrLastChunkedOperator.java
index d4520da86f6..1bdd13a7bde 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ByteBlinkSortedFirstOrLastChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ByteBlinkSortedFirstOrLastChunkedOperator.java
@@ -57,11 +57,12 @@ public void ensureCapacity(final long tableSize) {
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
- super.resetForStep(upstream, startingDestinationsCount);
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ final boolean modified = super.resetForStep(upstream, startingDestinationsCount);
if (isCombo) {
changedDestinationsBuilder = RowSetFactory.builderRandom();
}
+ return modified;
}
@Override
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/CharBlinkSortedFirstOrLastChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/CharBlinkSortedFirstOrLastChunkedOperator.java
index 092f753192a..13268ffa445 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/CharBlinkSortedFirstOrLastChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/CharBlinkSortedFirstOrLastChunkedOperator.java
@@ -53,11 +53,12 @@ public void ensureCapacity(final long tableSize) {
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
- super.resetForStep(upstream, startingDestinationsCount);
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ final boolean modified = super.resetForStep(upstream, startingDestinationsCount);
if (isCombo) {
changedDestinationsBuilder = RowSetFactory.builderRandom();
}
+ return modified;
}
@Override
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ChunkedOperatorAggregationHelper.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ChunkedOperatorAggregationHelper.java
index 234b3be7ef9..b876e1347ea 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ChunkedOperatorAggregationHelper.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/ChunkedOperatorAggregationHelper.java
@@ -537,7 +537,7 @@ private TableUpdate computeDownstreamIndicesAndCopyKeys(
@NotNull final ModifiedColumnSet resultModifiedColumnSet,
@NotNull final UnaryOperator[] resultModifiedColumnSetFactories) {
final int firstStateToAdd = outputPosition.get();
- ac.resetOperatorsForStep(upstream, firstStateToAdd);
+ ac.resetOperatorsForStep(upstream, firstStateToAdd, modifiedOperators);
if (upstream.removed().isNonempty()) {
doRemoves(upstream.removed());
@@ -2085,7 +2085,9 @@ public void onUpdate(@NotNull final TableUpdate upstream) {
}
private void processNoKeyUpdate(@NotNull final TableUpdate upstream) {
- ac.resetOperatorsForStep(upstream, 1);
+ final boolean[] modifiedOperators = new boolean[ac.size()];
+
+ ac.resetOperatorsForStep(upstream, 1, modifiedOperators);
final ModifiedColumnSet upstreamModifiedColumnSet =
upstream.modified().isEmpty() ? ModifiedColumnSet.EMPTY
@@ -2099,7 +2101,6 @@ private void processNoKeyUpdate(@NotNull final TableUpdate upstream) {
ac.initializeSingletonContexts(opContexts, upstream,
od.operatorsWithModifiedInputColumns);
- final boolean[] modifiedOperators = new boolean[ac.size()];
// remove all the removals
if (upstream.removed().isNonempty()) {
doNoKeyRemoval(upstream.removed(), ac, opContexts, allColumns, modifiedOperators);
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/DoubleBlinkSortedFirstOrLastChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/DoubleBlinkSortedFirstOrLastChunkedOperator.java
index 14822045a1b..bff986e9440 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/DoubleBlinkSortedFirstOrLastChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/DoubleBlinkSortedFirstOrLastChunkedOperator.java
@@ -57,11 +57,12 @@ public void ensureCapacity(final long tableSize) {
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
- super.resetForStep(upstream, startingDestinationsCount);
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ final boolean modified = super.resetForStep(upstream, startingDestinationsCount);
if (isCombo) {
changedDestinationsBuilder = RowSetFactory.builderRandom();
}
+ return modified;
}
@Override
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FloatBlinkSortedFirstOrLastChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FloatBlinkSortedFirstOrLastChunkedOperator.java
index e5a8bc53e42..7bb38240ca2 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FloatBlinkSortedFirstOrLastChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FloatBlinkSortedFirstOrLastChunkedOperator.java
@@ -57,11 +57,12 @@ public void ensureCapacity(final long tableSize) {
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
- super.resetForStep(upstream, startingDestinationsCount);
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ final boolean modified = super.resetForStep(upstream, startingDestinationsCount);
if (isCombo) {
changedDestinationsBuilder = RowSetFactory.builderRandom();
}
+ return modified;
}
@Override
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java
index 176a14a28f6..fe9bb782d03 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java
@@ -39,8 +39,8 @@
*/
class FormulaChunkedOperator implements IterativeChunkedAggregationOperator {
- private final GroupByChunkedOperator groupBy;
- private final boolean delegateToBy;
+ private GroupByChunkedOperator groupBy;
+ private boolean delegateToBy;
private final String[] inputColumnNames;
private final String[] resultColumnNames;
@@ -301,12 +301,13 @@ public UnaryOperator initializeRefreshing(@NotNull final Quer
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
if (delegateToBy) {
groupBy.resetForStep(upstream, startingDestinationsCount);
}
updateUpstreamModifiedColumnSet =
upstream.modified().isEmpty() ? ModifiedColumnSet.EMPTY : upstream.modifiedColumnSet();
+ return false;
}
@Override
@@ -494,4 +495,9 @@ private boolean[] makeObjectOrModifiedColumnsMask(@NotNull final ModifiedColumnS
}
return columnsMask;
}
+
+ public void updateGroupBy(GroupByChunkedOperator groupBy, boolean delegateToBy) {
+ this.groupBy = groupBy;
+ this.delegateToBy = delegateToBy;
+ }
}
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
index 2bf75160fdf..7cdffaba562 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
@@ -3,6 +3,7 @@
//
package io.deephaven.engine.table.impl.by;
+import io.deephaven.base.verify.Assert;
import io.deephaven.chunk.*;
import io.deephaven.chunk.attributes.ChunkLengths;
import io.deephaven.chunk.attributes.ChunkPositions;
@@ -19,11 +20,14 @@
import io.deephaven.engine.table.impl.sources.ArrayBackedColumnSource;
import io.deephaven.util.SafeCloseable;
import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
import java.util.Arrays;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.function.UnaryOperator;
+import java.util.stream.Collectors;
import static io.deephaven.engine.table.impl.sources.ArrayBackedColumnSource.BLOCK_SIZE;
@@ -34,11 +38,15 @@ class FormulaMultiColumnChunkedOperator implements IterativeChunkedAggregationOp
private final QueryTable inputTable;
- private final GroupByChunkedOperator groupBy;
- private final boolean delegateToBy;
+ private GroupByOperator groupBy;
+ private boolean delegateToBy;
private final SelectColumn selectColumn;
private final WritableColumnSource resultColumn;
private final String[] inputKeyColumns;
+ @Nullable
+ private final ColumnSource formulaDepthSource;
+ @Nullable
+ private final Map renames;
private ChunkSource formulaDataSource;
@@ -60,18 +68,24 @@ class FormulaMultiColumnChunkedOperator implements IterativeChunkedAggregationOp
* be false if {@code groupBy} is updated by the helper, or if this is not the first operator sharing
* {@code groupBy}.
* @param selectColumn The formula column that will produce the results
+ * @param renames a map from input names in the groupBy operator (i.e. mangled names) to input column names in the
+ * formula
*/
FormulaMultiColumnChunkedOperator(
@NotNull final QueryTable inputTable,
- @NotNull final GroupByChunkedOperator groupBy,
+ @NotNull final GroupByOperator groupBy,
final boolean delegateToBy,
@NotNull final SelectColumn selectColumn,
- @NotNull final String[] inputKeyColumns) {
+ @NotNull final String[] inputKeyColumns,
+ @Nullable Map renames,
+ @Nullable final ColumnSource formulaDepthSource) {
this.inputTable = inputTable;
this.groupBy = groupBy;
this.delegateToBy = delegateToBy;
this.selectColumn = selectColumn;
this.inputKeyColumns = inputKeyColumns;
+ this.renames = renames;
+ this.formulaDepthSource = formulaDepthSource;
resultColumn = ArrayBackedColumnSource.getMemoryColumnSource(
0, selectColumn.getReturnedType(), selectColumn.getReturnedComponentType());
@@ -199,7 +213,7 @@ public boolean modifyRowKeys(final SingletonContext context,
@Override
public boolean requiresRowKeys() {
- return delegateToBy;
+ return delegateToBy && groupBy.requiresRowKeys();
}
@Override
@@ -222,14 +236,31 @@ public void propagateInitialState(@NotNull final QueryTable resultTable, int sta
}
final Map> sourceColumns;
- if (inputKeyColumns.length == 0) {
+ if (inputKeyColumns.length == 0 && formulaDepthSource == null && renames == null) {
// noinspection unchecked
sourceColumns = (Map>) groupBy.getInputResultColumns();
} else {
final Map> columnSourceMap = resultTable.getColumnSourceMap();
- sourceColumns = new HashMap<>(groupBy.getInputResultColumns());
+ sourceColumns = new HashMap<>(groupBy.getInputResultColumns().size() + 1);
+ for (Map.Entry> entry : groupBy.getInputResultColumns().entrySet()) {
+ final String columnName = entry.getKey();
+ final String renamed;
+ if (renames != null && (renamed = renames.get(columnName)) != null) {
+ sourceColumns.put(renamed, entry.getValue());
+ } else {
+ sourceColumns.put(columnName, entry.getValue());
+ }
+ }
Arrays.stream(inputKeyColumns).forEach(col -> sourceColumns.put(col, columnSourceMap.get(col)));
+ sourceColumns.put(AggregationProcessor.ROLLUP_FORMULA_DEPTH.name(), formulaDepthSource);
}
+ final List missingColumns = selectColumn.getColumns().stream()
+ .filter(column -> !sourceColumns.containsKey(column)).collect(Collectors.toList());
+ if (!missingColumns.isEmpty()) {
+ throw new IllegalStateException(
+ "Columns " + missingColumns + " not found, available columns are: " + sourceColumns.keySet());
+ }
+
selectColumn.initInputs(resultTable.getRowSet(), sourceColumns);
formulaDataSource = selectColumn.getDataView();
@@ -263,8 +294,7 @@ public UnaryOperator initializeRefreshing(@NotNull final Quer
final String[] inputColumnNames = selectColumn.getColumns().toArray(String[]::new);
final ModifiedColumnSet inputMCS = inputTable.newModifiedColumnSet(inputColumnNames);
return inputToResultModifiedColumnSetFactory = input -> {
- if (groupBy.getSomeKeyHasAddsOrRemoves() ||
- (groupBy.getSomeKeyHasModifies() && input.containsAny(inputMCS))) {
+ if (groupBy.hasModifications(input.containsAny(inputMCS))) {
return resultMCS;
}
return ModifiedColumnSet.EMPTY;
@@ -272,12 +302,13 @@ public UnaryOperator initializeRefreshing(@NotNull final Quer
}
@Override
- public void resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
+ public boolean resetForStep(@NotNull final TableUpdate upstream, final int startingDestinationsCount) {
if (delegateToBy) {
groupBy.resetForStep(upstream, startingDestinationsCount);
}
updateUpstreamModifiedColumnSet =
upstream.modified().isEmpty() ? ModifiedColumnSet.EMPTY : upstream.modifiedColumnSet();
+ return false;
}
@Override
@@ -408,4 +439,9 @@ public void close() {
private static long calculateContainingBlockLastKey(final long firstKey) {
return (firstKey / BLOCK_SIZE) * BLOCK_SIZE + BLOCK_SIZE - 1;
}
+
+ public void updateGroupBy(GroupByOperator groupBy, boolean delegateToBy) {
+ this.groupBy = groupBy;
+ this.delegateToBy = delegateToBy;
+ }
}
diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/GroupByChunkedOperator.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/GroupByChunkedOperator.java
index f5d28a99731..a15610c4fe6 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/by/GroupByChunkedOperator.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/by/GroupByChunkedOperator.java
@@ -3,6 +3,7 @@
//
package io.deephaven.engine.table.impl.by;
+import io.deephaven.api.Pair;
import io.deephaven.base.verify.Assert;
import io.deephaven.chunk.attributes.ChunkLengths;
import io.deephaven.chunk.attributes.ChunkPositions;
@@ -22,9 +23,7 @@
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
-import java.util.Arrays;
-import java.util.LinkedHashMap;
-import java.util.Map;
+import java.util.*;
import java.util.function.UnaryOperator;
import static io.deephaven.engine.table.impl.sources.ArrayBackedColumnSource.BLOCK_SIZE;
@@ -33,7 +32,7 @@
* An {@link IterativeChunkedAggregationOperator} used in the implementation of {@link Table#groupBy},
* {@link io.deephaven.api.agg.spec.AggSpecGroup}, and {@link io.deephaven.api.agg.Aggregation#AggGroup(String...)}.
*/
-public final class GroupByChunkedOperator implements IterativeChunkedAggregationOperator {
+public final class GroupByChunkedOperator implements GroupByOperator {
private final QueryTable inputTable;
private final boolean registeredWithHelper;
@@ -44,8 +43,8 @@ public final class GroupByChunkedOperator implements IterativeChunkedAggregation
private final ObjectArraySource