Skip to content

Recording rule generates incorrect values? #19580

@ejortegau

Description

@ejortegau

Describe the bug
rate_counter() function seems to give incorrect values?

To Reproduce
Steps to reproduce the behavior:
Steps to reproduce the behavior:

  1. Started Loki (SHA or version): 3.5.1

  2. Query:

rate_counter(
 {service_name="my_app", host_name="host-01.example.com"}
 | json
 | metric_name=~"udp_messages_total"
 | status="received"
 | drop  time,  observed_timestamp
 | unwrap value [2m]
)

Expected behavior
My logs look as follows:

2025-10-19 20:30:27.596 info{
  "level": "info",
  "metric_name": "udp_messages_total",
  "msg": "Counter value",
  "status": "received",
  "time": "2025-10-19T20:30:27+02:00",
  "value": 106200
}
2025-10-19 20:29:27.596 info{
  "level": "info",
  "metric_name": "udp_messages_total",
  "msg": "Counter value",
  "status": "received",
  "time": "2025-10-19T20:29:27+02:00",
  "value": 103990
}
2025-10-19 20:28:27.595 info{
  "level": "info",
  "metric_name": "udp_messages_total",
  "msg": "Counter value",
  "status": "received",
  "time": "2025-10-19T20:28:27+02:00",
  "value": 101824
}
2025-10-19 20:27:27.595 info{
  "level": "info",
  "metric_name": "udp_messages_total",
  "msg": "Counter value",
  "status": "received",
  "time": "2025-10-19T20:27:27+02:00",
  "value": 99608
}

As shown above, there in an increase of the value attribute between each log line and the next of ~2200. Given that logs are emitted every minute, I would expect that if I use rate_counter on value for these log entries, I would get something close to 2200 udp_messages_total / (60 seconds) = 36 udp_messages / second. However, I am instead getting a value close to half of that.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugSomehing is not working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions