Using Home Assistant to monitor energy usage and compare tariffs

noitsnotme

I asked about this in the Octopus Tracker thread and someone suggested I start a new thread to avoid clogging the other.

I’m just about to set up Home Assistant, just waiting for the thin PC I’ve ordered to arrive.

Something I would like to do is compare my energy consumption between this year and last.  I have half hourly data from the beginning of 2022 through Bright (Glowmarkt) and can also access it through Octopus since July this year.

I would like to be able to pick a recent day and compare it to the same day last year.  We have just had a new boiler fitted and I’d like to compare roughly how much gas it’s using compared to the old boiler.

it would also be good to be able to compare different Octopus tariffs based on my actual usage.

If anyone is using HA and is doing anything along these lines I’d be interested to see your methods.

ArbitraryRandom

To start, the CSV will contain a lot of information I don't think you need for what you're asking (MPAN/MPRN, postcodes,  and other random fields that are of use if you're doing a regional analysis, but not for a single household). 

As a basic usage analysis, when you're working with half-hourly consumption, your dataset only needs to consist of a series of 48 daily readings for each meter - with each half hourly data point reporting the total amount of energy consumed in the previous 30 minutes. 

Format wise, that means the easiest way to display (IMO) is: 

• The date that the readings was taken
• The 48 data points making up a full days readings (and one sheet for each meter)

Pick your format (the two sources will almost certainly use different column headers, so make sure they're actually recording what you think they are - and check with the time if the 'first' reading is midnight or 00:30). One row per day; each row having 48 half-hourly readings as separate columns - once you have your dataset (sanitised), you can do your analysis using different rates/unit values. 

mmmmikey

To add to @ArbitraryRandom post above, I'd also give some thought about how you're going to handle daylight saving time and time formats. Even the Octopus API is a bit schizophrenic about this, returning data in one of three different formats. The situation can quickly get even more confused if you're using data from multiple sources.

As far as HA is concerned, I'm mainly using it for controlling various devices and monitoring energy usage. I get energy usage data from different device types (e.g. Tapo plugs, Sonoff switches) and it's good to be able to see it in a consistent format through a single app. 

I heat my water and charge the batteries at the cheapest (Agile) time, and I've been doing that by using a IFTTT integrations that set switches in HA. Although this has worked well most of the time it's become unreliable of late so I've just gone live with a new configuration that uses the Octopus API directly, cutting out IFTTT. I have a Python program running on a Raspberry Pi that basically replicates what IFTTT was doing - so far so good.

It's playing with the API that has lead me to monitoring tariff rates automatically. Another Python program downloads my energy data each day, calculates what it has cost using the corresponding Agile rates and also what it would have cost if I had been on Tracker or SVR. The same program that downloads my own energy data also downloads the Agile, Tracker and SVR. The output at the moment is a simple display that shows the total cost, average unit rate and standing charge for each of the 3 tariffs (Agile, Tracker & SVR). The display shows ths for the most recent day, and averages over the last 7 and 28 days. At the moment Agile is a worthwhile amount better for me than Tracker and there's clear, blue water between Tracker and SVR. This won't tell me if I should change tariff or not - it will just flag up a warning that it's time to review.

noitsnotme

@ArbitraryRandom the CSVs from both Bright and Octopus are very simple.  They both have only 3 columns albeit slightly different - Bright has ‘epochTime’, kWh and date/time and Octopus has consumption in m3 and the start & finish date/time for each 30 min slot.

I haven’t compared them in detail yet to see if Bright is as accurate as Octopus and obviously I would need to convert the m3 to kWh before comparing.  But if Bright is accurate that has the longest dataset and would save me having to compare 2 slightly different datasets.  Both have APIs.

@mmmmikey good point about the daylight savings, I’ll watch out for that.  I like the sound of what you’ve done with the tariff comparisons.  

I see there is a Home Energy Management feature in HA but a few posts say it’s somewhat limited and you can’t compare a day/week/month with one from a previous year.

I’m having a Zappi EV charger fitted next week and then solar panels and a house battery in a few weeks time once the G99 application (hopefully!) goes through.  So that will be lots more data to integrate.

I can see I’m just about to go down a potentially very deep rabbit hole with HA 😄

ArbitraryRandom

Re the conversation, I know bright use a fixed calorific value - but I don't know what that is and a quick google isn't kicking it up. As your dataset overlaps you could try and work it out algebraically, or drop Hildebrand an email? 

Neither will be supremely accurate unless you pull in the average calorific value for your region... but I'd say close enough is good enough (at least to start). 

The_Green_Hornet

If you want an off the shelf Home Assistant solution you could always look at this custom component:

BottlecapDave/HomeAssistant-OctopusEnergy: Unofficial Home Assistant integration for interacting with Octopus Energy (github.com)

FreeBear

Another HA user here... I use a couple of Peacefair PZEM-016 & PZEM-014 to monitor electricity consumption. Only recently had my gas & electricity meters replaced with SMETS2 models, so getting a Hildebrand Glow IHD is on the cards. That way, I can get realtime data directly from the meter(s) using the MQTT API. In the meantime, I'm logging data from the boiler using an ESP32 & OpenTherm interface - Recording burner on/off time & modulation gives me a reasonably good indicator of gas consumption.

All data is being stored in an Influx database and graphs generated by Grafana - A word of warning. Influx databases can get very large which could cause a problem if you do a reboot of the data store (had to massively increase the systemd timeout for the influx daemon after saving >12m of data). Also, set the include & exclude domains/entities in the HA influx component so that you are recording just the data you want to generate graphs for. If you don't, you'll end up with masses of irrelevant data clogging up the database. Oh, and don't use SD cards for data storage - You will kill them in a matter of weeks of months.

noitsnotme

@ArbitraryRandom I've calculated it to a calorific value of roughly 40.  That's close enough anyway and after conversion of one dataset, brings it within 2 decimal places of the other with the sample set I used.

@The_Green_Hornet Thanks, I'll take a look at that.

@FreeBear I've just had a Viessmann boiler fitted and have discovered there is a Viessman ViCare integration for Home Assistant so that's more data to add.  I was going to get an RPi4 for HA but after a lot of reading it seemed better to go for a thin client PC, so I ordered a refurbished Lenovo Thinkcentre M700 with a 126GB SSD.  It's not much more energy hungry than the Pi and wasn't much more expensive at £70.  Should be arriving next week.

Bendo

I'm not in the least bit interested in using HA to monitor tariffs etc, but using it with the Octopus integration is great for a nice simple numeric state automation against current unit price for agile to turn on all manner of things.

I Just need to get a zigbee or z wave switch to replace the switch on my boilermate that makes it run on electric rather than using the gas boiler and then I'll have it automatically turn the heating on using the 6KW electric heater when plunge pricing is in effect.

FreeBear

noitsnotme said: I've just had a Viessmann boiler fitted and have discovered there is a Viessman ViCare integration for Home Assistant so that's more data to add.  I was going to get an RPi4 for HA but after a lot of reading it seemed better to go for a thin client PC

Viessmann uses OpenTherm, although it doesn't implement the full range of commands - I had the Vitodens 050 installed earlier in the year and have been busy with the ESP32 software - I really should upload my code so that you can see what data is/isn't available via OpenTherm.

Unless you register & pay for ViCare, you are limited to the number of times you can download data from their cloud server. And from what I can gather, the service isn't that reliable - Can suffer from outages that last for days rather than hours. Personally, I'd rather cut out remote servers if I can get the data locally.

As for hardware, I run Home Assistant of a NanoPi Neo with a custom heatsink, and use an Advantech ARK-6320 as the data store. Eventually, I intend to remove the NanoPi from the system once I've migrated the nexus433 functionality to an ESP32 (or got rid of it entirely). Whilst the Raspberry Pi is a compact little board, there are plenty of equally capable alternatives & clones. But you can't beat a proper hard drive for data storage, and quite a few (refurbished) industrial computers will do the job.

noitsnotme

FreeBear said:

noitsnotme said: I've just had a Viessmann boiler fitted and have discovered there is a Viessman ViCare integration for Home Assistant so that's more data to add.  I was going to get an RPi4 for HA but after a lot of reading it seemed better to go for a thin client PC

Viessmann uses OpenTherm, although it doesn't implement the full range of commands - I had the Vitodens 050 installed earlier in the year and have been busy with the ESP32 software - I really should upload my code so that you can see what data is/isn't available via OpenTherm.

Unless you register & pay for ViCare, you are limited to the number of times you can download data from their cloud server. And from what I can gather, the service isn't that reliable - Can suffer from outages that last for days rather than hours. Personally, I'd rather cut out remote servers if I can get the data locally.

I also had a Vitodens 050 installed.  I went with the external weather compensator so not sure if I can use OpenTherm at the same time?  I’m not entirely sure the WC was the right choice.  The boiler was only installed a couple of weeks ago and we’ve not yet needed the heating on and I need to balance the radiators first anyway.

Are you using any other hardware to control the boiler through OpenTherm (eg Nest, Tado, etc)?  Or is that what you use ESP32 for?

I may need to consider ditching the WC if it limits what I can do with other devices/software.