Timeseries library initial version

[Erikvv]

Related/parent issue: Zenmo/zero_engine#77

Create an initial version of a library to read and manipulate timeseries in Lux Engine.

[GillisHommen]

Some references: ISO 8601 standard for time formats, and Java LocalDate:
https://www.baeldung.com/java-format-localdate-iso-8601-t-z
Let's try to comply with this standard.

Other things to make clear:
-What is the from- and to-time for each sample? (Is a kWh datapoint at 14:00, the consumption between 13:45 and 14:00, or between 14:00 and 14:15?)

This class will be used by several modules; zorm, vallum, zero_engine, zero_interface-loader. Should become a 'stand alone' .jar file?

[Erikvv]

Serialization

ISO 8601

It's always good to stick to ISO 8601 / RFC 3339. Also the duration format like PT15M for quarter-hours. It is however a serialization format which is not the core of this library. All we have to do is verify that Jackson does the right thing.

LocalDate

LocalDate

I have my doubts on LocalDate. It has no timezone information. I also can't imagine it's any faster than Instant with all those fields.

class Instant {
    private final long seconds;
    private final int nanos;
}

class LocalDateTime {
    /** LocalDate members */
    private final int year;
    private final short month;
    private final short day;

    /** LocalTime members */
    private final byte hour;
    private final byte minute;
    private final byte second;
    private final int nano;
}

LocalDateTime can be used for presentation but even for presentation I would convert Instant to ZonedDateTime and then pass that to DateTimeFormatter.

Temporal all the things

All of these classes implement Temporal. We could program this library against Temporal. Later we could implement an optimized version of Temporal. For example with an internal state of minutes since unix epoch.

class OptimizedInstant implements Temporal {
    private final long minutes;
}

Datapoint definition

-What is the from- and to-time for each sample? (Is a kWh datapoint at 14:00, the consumption between 13:45 and 14:00, or between 14:00 and 14:15?)

In zummon I've defined the timestamp as being the start of the interval. It makes more sense to me. For example when plotting daily metrics in a graph, you want to show the day of the start of the interval.

I suspect many measurement devices do it differently because they submit results at the end of the interval. I'm not aware of a consistent standard.

Distribution

This class will be used by several modules; zorm, vallum, zero_engine, zero_interface-loader. Should become a 'stand alone' .jar file?

If we don't pull in any dependencies I suspect the standalone jar (uberjar) will have the same size as a normal jar file.

We could remove technically remove the library from the Vallum ubjerjar if it's already in the model. But I think the savings in size would be small. There are probably larger gains to be found when de-duplicating Uplux and Vallum.

Accessor interface

Logically deducing I end up with an interface like so:

interface TimeSeriesAccessor {
    fun getValueAt(
        intervalStart: Temporal,
        step: TemporalAmount
    ): Double
}

Rate vs periodic

I noticed this phenomenon in case of a mismatch between the interval step and the requested step: an implementation of the above interface will be different depending on whether the time series represents a rate (like kW), a cumulative amount, or a periodic amount.

• rate: it should average across the interval
• cumulative: it should pick the last value in the interval
• periodic: it should add the values in the interval

(this is all assuming the requested step is longer than the stored step)

It seems to me that this behavior should be internal to the class and not exposed in getValueAt

I don't think we ever use cumulative time series so we'd have two implementations of TimeSeriesAccessor, or two strategies inside one implementation.

Performance

This interface is obviously slower than a direct index into array-based implementation. The implementation requires a division to determine the array offset. But the current implementation of ProfilePointer using double t_h also does such a division.

We can have an optimized forward iterator that tracks the index to avoids the division. Like so:

class ForwardIterationOptimizedArrayBackedTimeSeries(
    private val start: Temporal,
    private val step: TemporalAmount,
    private val values: DoubleArray,
): TimeSeriesAccessor {
    private var currentOffset: Temporal = start
    private var currentIndex: Int = 0

    override fun getValueAt(intervalStart: Temporal, step: TemporalAmount): Double {
        if (intervalStart > currentOffset) {
            reset()
        }

        while (intervalStart < currentOffset) {
            currentOffset += step
            currentIndex++
        }

        return values[currentIndex]
    }

    private fun reset() {
        currentOffset = start
        currentIndex = 0
    }
}

(this ignores the situation where extra logic is needed because the steps don't match. Which would again require multiple divisions..... Maybe matching the steps is a prerequisite for this optimization, or this is not a fruitful direction)

Standard Iterator

We could use a standard iterator as an optimization:

class TimeSeries {
    fun createIterator(
        intervalStart: Temporal,
        step: TemporalAmount
    ): Iterator<Double>
}

But that would require the library user to track that Iterator::next() is only called once per time step. I feel that this is too onerous of a demand.

[Erikvv]

Writing

When progressively writing a time series, we need to manage the size.

class ArrayBackedTimeSeries(
    private val start: Temporal,
    private var step: TemporalAmount,
    private var values: DoubleArray = DoubleArray(0),
) {
    fun put(
        intervalStart: Temporal,
        step: TemporalAmount,
        value: Double,
    ): Unit {
        if (values.isEmpty()) {
            this.step = step
        }

        if (step != this.step) {
            throw Exception("When writing to a timeseries, all values must have the same step")
        }

        val offset = Duration.between(start, intervalStart)
            .dividedBy(Duration.from(step))
            .toInt()

        while (offset >= values.size) {
            resize()
        }

        values[offset] = value
    }

    fun resize() {
        values = values.copyOf(values.size * 2 + 1)
    }
}

But this would lead to a lot of zeroes at the end, so we would need to track the actual size in another variable.

Probably there is some data structure library out there which can fix this for us. For example https://javadoc.io/doc/it.unimi.dsi/fastutil/latest/it/unimi/dsi/fastutil/doubles/DoubleArrayList.html

[Erikvv]

Time series in Pandas: https://pandas.pydata.org/docs/user_guide/timeseries.html

[GillisHommen]

Writing

When progressively writing a time series, we need to manage the size.

class ArrayBackedTimeSeries(
private val start: Temporal,
private var step: TemporalAmount,
private var values: DoubleArray = DoubleArray(0),
) {
fun put(
intervalStart: Temporal,
step: TemporalAmount,
value: Double,
): Unit {
if (values.isEmpty()) {
this.step = step
}

    if (step != this.step) {
        throw Exception("When writing to a timeseries, all values must have the same step")
    }

    val offset = Duration.between(start, intervalStart)
        .dividedBy(Duration.from(step))
        .toInt()

    while (offset >= values.size) {
        resize()
    }

    values[offset] = value
}

fun resize() {
    values = values.copyOf(values.size * 2 + 1)
}

}
But this would lead to a lot of zeroes at the end, so we would need to track the actual size in another variable.

Probably there is some data structure library out there which can fix this for us. For example https://javadoc.io/doc/it.unimi.dsi/fastutil/latest/it/unimi/dsi/fastutil/doubles/DoubleArrayList.html

We could decide that size should be fixed after an instance is created?

[GillisHommen]

Wrt. sampling performance; currently in the ProfilePointer, there are often multiple users of the same profile. Therefore, each timestep, the method ProfilePointer.updateValue(t_h) is called once, and all users take ProfilePointer.getCurrentValue();