Data Revisions
When looking at yesterday’s data yesterday vs. today in our dashboard, you might have noticed some changes. For example, yesterday, the index might have been 13.5%, but today’s data for yesterday is 13.8%. How is it possible? Is it a bug?
No.
The data is correct, despite obvious changes.
The historical changes happen for two* main reasons.
- 1.The data providers change their calculation model, categories, or sometimes raw data due to corrections or taxes. The next time our algorithm queries their APIs, it sees new data and rewrites the historical data in our database.
- 2.New data streams into our database from the providers who can only offer monthly or weekly data updates.
The most prominent provider updating only once a month is the Bureau of Labor statistics releasing their CPI numbers with a month and a 10-day delay.
We still take 40% of the CPI data from the BLS, and that data updates between the 10th and the 12th each month and is streamed into our index a day later, changing historical data. For example, the recent data for March released in mid-April changed all data for March and April. The BLS’ ‘March’ data then repeats in our database for the entire April and 10 days of May until the official CPI index updates next month, changing the historical data again.
Why do we put the BLS’ March data in quotation marks? It’s because it’s not, in fact, data for March but a mixture of current surveys, old surveys, and forecasts or approximations for March that will then settle over time into the ‘actual’ March data.
*Changes may also happen due to errors in our system or that of our providers. Such changes are not desirable and are usually flagged by our QC and corrected as soon as possible.
The BLS data are not final at the time of reporting. Instead, they revise all their data for 5 years from the moment they are reported through yearly seasonal adjustments and revisions of the adjusted numbers.
We call the time it takes between current proxy data and the final data a data settlement time or period. The BLS has a settlement period of 5 years (!) for the real data from surveys, census, consumer trends, and taxes to stream in and historical data to get adjusted.
The BLS doesn’t adjust their reported numbers (the inflation rates based on the data on the day of the reporting), but they do report the seasonally adjusted and revised raw CPI numbers, which one can check via interactive charts through FRED. The raw CPI number is the price index for a specific month instead of the familiar % inflation rate, which is the 12-month change between raw CPIs today and a year ago.
The 40% of our index is still delayed due to the BLS component and is likely to be revised over a period of 5 years by the BLS.
Our other provider’s maximum delay is up to 2 months for the Penn State data. Other providers offer their data with a 1 day, 1-2 week or 1 month delay depending on their data collection model and constraints of their APIs.
The Truflation index is the most current public information about inflation you can get anywhere at the moment. Can it be better and more current?
Most likely.
Is it crucial that it’s updated daily without delays? Probably not.
Actually, you tell us if you need the inflation data to be daily, weekly, or monthly. Your voice will help us design the data visualization that helps you use this data.
Our data is streamed through the Chainlink oracle and available to various blockchains. Daily data, which we call 'reported data,' is shared on our social media and can be recorded on the blockchain and immutable forever.
The daily reported data is the most current and recent data only on the day of its reporting. Once that day passes, new incoming data will be more correct or recent. This is why we agreed to continue streaming the historical data from our providers as they update it, including their past records.
Minimal interference is also part of our path to decentralization to limit our interaction with the APIs data.
In the crypto world, we are all very used to real-time data streams. Data updated once a day seems old. However, in the ‘data world’, the aggregators don’t stream information hourly or even daily but rather monthly, quarterly, and yearly.
When you think about it, the consumer trends and prices don’t update hourly or daily either.
So is there a point in providing a daily inflation index?
We thought there was. It lets us see the early trends in the data. It also feels more real, more crypto, and more fun to have the numbers changing daily rather than monthly like the old boring CPI.
To make that happen, we are sourcing from as many daily price providers as we can while others updates once a week or once a month. In time, we hope to develop forecast algorithms to fill the gaps and even predict future trends.
But the biggest question remains, do you want your inflation data to stream daily.