Skip to content

fix: correct data format parsing and make start time parameter required #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rubenszinho merged 2 commits into from
Apr 17, 2025
Merged

fix: correct data format parsing and make start time parameter required #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d06519f
fix: correct data format parsing and make start time parameter required
rubenszinho Apr 17, 2025
d6d3131
fix: treat error in the main file by properly checking the return val…
rubenszinho Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 26 additions & 101 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,42 +42,31 @@ cd enemeter-data-processing
make build
```

## Data Format

ENEMETER data is provided in CSV format with four columns:

1. **Time Delta (milliseconds)** - Time elapsed since the last measurement
2. **Temperature (millicelsius)** - System temperature
3. **Voltage (microvolts)** - Battery voltage
4. **Current (nanoamperes)** - System current (positive when charging, negative when discharging)

Example:
```
0000052051,0004815430,-1275169536,0000023192
0000000047,0004815430,-1275169536,0000023192
0000000047,0004815790,-1276443264,0000023192
```

## Basic Usage

Process a CSV file with default settings:
Process a CSV file with required parameters:

```bash
./enemeter-data-processing --input=data/data.csv
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00"
```

The `--start` parameter specifies the starting time of the measurements and is required.
**Important:** You must include both the date AND time of day in the format "YYYY-MM-DD HH:MM:SS".

Save results to a text file:

```bash
./enemeter-data-processing --input=data/data.csv --output=report.txt
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00" --output=esp32_report.txt
```

## Command-line Options

### Input/Output Options
### Required Parameters
- `--input=<path>`: Path to the input CSV file
- `--start=<time>`: Start time for measurements (format: YYYY-MM-DD HH:MM:SS) - must include time of day

- `--input=<path>`: Path to the input CSV file (required)
- `--output=<path>`: Path to save the output report (optional)
### Optional Parameters
- `--output=<path>`: Path to save the output report
- `--format=<text|json|csv>`: Output format (default: text)

### Processing Options
Expand All @@ -88,8 +77,8 @@ Save results to a text file:

### Time Filtering Options

- `--start=<time>`: Start time for filtering (format: YYYY-MM-DD[THH:MM:SS])
- `--end=<time>`: End time for filtering (format: YYYY-MM-DD[THH:MM:SS])
- `--start=<time>`: (Required) Start time for measurements (format: YYYY-MM-DD HH:MM:SS) - must include time of day
- `--end=<time>`: End time for filtering (format: YYYY-MM-DD HH:MM:SS)
- `--window=<duration>`: Time window to process (e.g., 1h, 30m, 24h)

### Data Filtering Options
Expand Down Expand Up @@ -123,123 +112,59 @@ Save results to a text file:
Process a CSV file and display all metrics:

```bash
./enemeter-data-processing --input=data/data.csv
./enemeter-data-processing process --input=data/esp32.csv --start="2023-04-01 08:00:00"
```

### Memory-Efficient Processing for Large Files

Use streaming mode with sampling for very large files:

```bash
./enemeter-data-processing --input=big-data.csv --stream --sample=10
./enemeter-data-processing process --input=big-data.csv --start="2023-04-01 10:15:30" --stream --sample=10
```

This processes only every 10th record, reducing memory requirements.

### Filtering by Time
### Time-Based Analysis

Process data from a specific time range:
To analyze data from a specific time period:

```bash
./enemeter-data-processing --input=data.csv --start="2025-04-01" --end="2025-04-02"
```

Process the last 24 hours of data:
# Analyze data between specific dates and times
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --end="2023-04-02 17:30:00"

```bash
./enemeter-data-processing --input=data.csv --window=24h
# Analyze data for a specific duration
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 12:45:00" --window=24h
```

### Extracting Specific Metrics

Get only temperature statistics in JSON format:

```bash
./enemeter-data-processing --input=data.csv --metric=temperature --format=json
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --metric=temperature --format=json
```

Extract hourly energy consumption in CSV format:

```bash
./enemeter-data-processing --input=data.csv --metric=energy_by_hour --format=csv
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --metric=energy_by_hour --format=csv
```

### Filtering by Data Values

Process only records with voltage between specified values:

```bash
./enemeter-data-processing --input=data.csv --volt-min=3500000 --volt-max=4200000
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --volt-min=3500000 --volt-max=4200000
```

Process only records where temperature is above a certain threshold:

```bash
./enemeter-data-processing --input=data.csv --min-temp=25000
./enemeter-data-processing process --input=esp32.csv --start="2023-04-01 08:00:00" --min-temp=25000
```

## Output Examples

### Text Output (Default)

```
========== ENEMETER DATA PROCESSING REPORT ==========
Input File: data.csv
Date: 2025-04-09 10:15:32
Data Points: 1253
Time Range: 2025-04-08 08:30:15 to 2025-04-09 08:30:10

ENERGY METRICS
-------------
Total Energy Consumed: 156.4578 joules
Average Power: 0.0548 watts
Peak Power: 0.1245 watts
Estimated Energy per Day: 157.9825 joules
Measurement Duration: 86395.00 seconds

TEMPERATURE STATISTICS
---------------------
Minimum Temperature: 22.45 °C
Maximum Temperature: 28.95 °C
Average Temperature: 24.75 °C

... additional sections ...
```

### JSON Output

```json
{
"TotalJoules": 156.4578,
"AveragePowerWatts": 0.0548,
"PeakPowerWatts": 0.1245,
"JoulesPerDay": 157.9825,
"DurationSeconds": 86395.0,
"TemperatureStats": {
"MinTempCelsius": 22.45,
"MaxTempCelsius": 28.95,
"AvgTempCelsius": 24.75
},
...
}
```

### CSV Output

```
Metric,Value
TotalJoules,156.457800
AveragePowerWatts,0.054800
PeakPowerWatts,0.124500
JoulesPerDay,157.982500
DurationSeconds,86395.00
...
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the GPL License - see the LICENSE file for details.
### Text Output (Default)
163 changes: 163 additions & 0 deletions cmd/analyze-csv/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
package main

import (
"encoding/csv"
"flag"
"fmt"
"math"
"os"
"strconv"
)

func main() {
inputFile := flag.String("input", "", "Path to the CSV file to analyze")
sampleSize := flag.Int("samples", 10, "Number of sample rows to display")
flag.Parse()

if *inputFile == "" {
fmt.Println("Error: Please specify an input file with --input")
os.Exit(1)
}

file, err := os.Open(*inputFile)
if err != nil {
fmt.Printf("Error opening file: %v\n", err)
os.Exit(1)
}
defer func() {
if closeErr := file.Close(); closeErr != nil {
fmt.Printf("Warning: Error closing file: %v\n", closeErr)
}
}()

reader := csv.NewReader(file)

fmt.Printf("Analyzing file: %s\n", *inputFile)
fmt.Printf("Showing %d sample rows:\n\n", *sampleSize)

var timeDeltas []int64
var voltages []int64
var currents []int64
var temps []int64

fmt.Println("RAW DATA SAMPLES:")
fmt.Println("---------------------------------------------------------------")
fmt.Printf("%-15s %-15s %-15s %-15s\n", "TIME_DELTA", "VOLTAGE", "CURRENT", "TEMP")
fmt.Println("---------------------------------------------------------------")

for i := 0; i < *sampleSize; i++ {
row, err := reader.Read()
if err != nil {
break
}

if len(row) != 4 {
fmt.Printf("Row %d has %d columns, expected 4\n", i+1, len(row))
continue
}

timeDelta, _ := strconv.ParseInt(row[0], 10, 64)
voltage, _ := strconv.ParseInt(row[1], 10, 64)
current, _ := strconv.ParseInt(row[2], 10, 64)
temp, _ := strconv.ParseInt(row[3], 10, 64)

timeDeltas = append(timeDeltas, timeDelta)
voltages = append(voltages, voltage)
currents = append(currents, current)
temps = append(temps, temp)

fmt.Printf("%-15d %-15d %-15d %-15d\n", timeDelta, voltage, current, temp)
}

fmt.Println("\nVALUE RANGES:")
fmt.Println("---------------------------------------------------------------")

minTimeDelta, maxTimeDelta := findMinMax(timeDeltas)
minTemp, maxTemp := findMinMax(temps)
minVoltage, maxVoltage := findMinMax(voltages)
minCurrent, maxCurrent := findMinMax(currents)

fmt.Printf("Time Delta: Min=%d ms, Max=%d ms\n", minTimeDelta, maxTimeDelta)
fmt.Printf("Temperature: Min=%d, Max=%d (Raw value)\n", minTemp, maxTemp)
fmt.Printf(" Min=%.2f °C, Max=%.2f °C (Millicelsius)\n", float64(minTemp)/1000.0, float64(maxTemp)/1000.0)
fmt.Printf(" Min=%.2f °C, Max=%.2f °C (Raw/100)\n", float64(minTemp)/100.0, float64(maxTemp)/100.0)

fmt.Printf("Voltage: Min=%d, Max=%d (Raw value)\n", minVoltage, maxVoltage)
fmt.Printf(" Min=%.6f V, Max=%.6f V (Microvolts)\n", float64(minVoltage)/1000000.0, float64(maxVoltage)/1000000.0)
fmt.Printf(" Min=%.6f V, Max=%.6f V (Millivolts)\n", float64(minVoltage)/1000.0, float64(maxVoltage)/1000.0)

fmt.Printf("Current: Min=%d, Max=%d (Raw value)\n", minCurrent, maxCurrent)
fmt.Printf(" Min=%.6f A, Max=%.6f A (Nanoamperes)\n", float64(minCurrent)/1000000000.0, float64(maxCurrent)/1000000000.0)
fmt.Printf(" Min=%.6f A, Max=%.6f A (Microamperes)\n", float64(minCurrent)/1000000.0, float64(maxCurrent)/1000000.0)

fmt.Printf("\nSUGGESTED UNITS FOR YOUR ESP32 DATA:\n")
fmt.Printf("---------------------------------------------------------------\n")

suggestUnits(0, maxTemp, minVoltage, maxVoltage, minCurrent, maxCurrent)

fmt.Printf("ENEMETER DATA FORMAT INFORMATION:\n")
fmt.Printf("---------------------------------------------------------------\n")
fmt.Printf("Column order: TIME_DELTA, VOLTAGE, CURRENT, TEMP\n")
fmt.Printf("Temperature: Values are in millicelsius (°C = value / 1000)\n")
fmt.Printf("Voltage: Values are in microvolts (V = value / 1000000)\n")
fmt.Printf("Current: Values are in nanoamperes (A = value / 1000000000)\n")
}

func findMinMax(values []int64) (int64, int64) {
if len(values) == 0 {
return 0, 0
}

min := values[0]
max := values[0]

for _, val := range values {
if val < min {
min = val
}
if val > max {
max = val
}
}

return min, max
}

func suggestUnits(_, maxTemp, minVoltage, maxVoltage, minCurrent, maxCurrent int64) {
// Suggest temperature units
if maxTemp > 100000 {
fmt.Println("Temperature: Values appear to be in raw ADC format")
fmt.Println(" Consider using raw_value / 10 as °C")
} else if maxTemp > 10000 {
fmt.Println("Temperature: Values appear to be in millicelsius")
fmt.Println(" Consider using raw_value / 1000 as °C")
} else if maxTemp > 1000 {
fmt.Println("Temperature: Values appear to be in centicelsius")
fmt.Println(" Consider using raw_value / 100 as °C")
} else {
fmt.Println("Temperature: Values appear to be direct celsius readings")
}

// Suggest voltage units
if math.Abs(float64(minVoltage)) > 1000000 || math.Abs(float64(maxVoltage)) > 1000000 {
fmt.Println("Voltage: Values appear to be in microvolts")
fmt.Println(" Consider using raw_value / 1000000 as V")
} else if math.Abs(float64(minVoltage)) > 1000 || math.Abs(float64(maxVoltage)) > 1000 {
fmt.Println("Voltage: Values appear to be in millivolts")
fmt.Println(" Consider using raw_value / 1000 as V")
} else {
fmt.Println("Voltage: Values appear to be direct voltage readings (V)")
}

// Suggest current units
if maxCurrent > 1000000 || minCurrent < -1000000 {
fmt.Println("Current: Values appear to be in nanoamperes")
fmt.Println(" Consider using raw_value / 1000000000 as A")
} else if maxCurrent > 1000 || minCurrent < -1000 {
fmt.Println("Current: Values appear to be in microamperes")
fmt.Println(" Consider using raw_value / 1000000 as A")
} else {
fmt.Println("Current: Values appear to be in milliamperes")
fmt.Println(" Consider using raw_value / 1000 as A")
}
}
Loading
Loading