FERC Form 1 – Annual Report of Major Electric Utilities#
Source URL |
|
|---|---|
Source Description |
The Federal Energy Regulatory Commission (FERC) Form 1 is a comprehensive financial and operating report submitted annually for electric rate regulation, market oversight analysis, and financial audits by Major electric utilities, licensees and others. |
Respondents |
Major electric utilities and licenses. |
Records Liberated |
~13.2 million (116 raw tables), ~4.7 million (23 clean tables) |
Source Format |
XBRL (.XBRL) and Visual FoxPro Database (.DBC/.DBF) |
Download Size |
2051 MB |
Temporal Coverage |
1994-2024 |
PUDL Code |
|
Unprocessed Source Data Archive |
|
Issues |
Open FERC Form 1 – Annual Report of Major Electric Utilities issues |
PUDL Database Tables#
We’ve segmented the processed data into the following normalized data tables. Clicking on the links will show you a description of the table as well as the names and descriptions of each of its fields.
We’ve also created the following tables mapping manually assigned PUDL IDs to FERC respondent IDs, enabling a connection between the FERC and EIA data sets.
Background#
The FERC Form 1, otherwise known as the Electric Utility Annual Report, contains financial and operating data for major utilities and licensees. Much of it is not publicly available anywhere else.
Download additional documentation#
Data available through PUDL#
PUDL integrates data from all years in which digital filings are available, spanning both the DBF (1994-2020) and XBRL (2021-present) reporting eras. There are more than 100 tables in the raw databases, and PUDL has so far only integrated a subset of them (see list above). The integrated tables mostly pertain to plant-level operations, utility assets & liabilities, and utility income & expenses. Integrated tables combine data from the old DBF and new XBRL reporting formats into a single coherent structure covering all years from 1994 to the present.
Who submits this data?#
As outlined in the Commission’s Uniform System of Accounts Prescribed for Public Utilities and Licensees Subject To the Provisions of The Federal Power Act (18 C.F.R. Part 101), to qualify as a respondent, entities must exceed at least one of the following criteria for three consecutive years prior to reporting:
1 million MWh of total sales
100MWh of annual sales for resale
500MWh of annual power exchanges delivered
500MWh of annual wheeling for others (deliveries plus losses)
Note
Publicly owned utilities and member-owned electric cooperatives are not required to file the FERC Form 1. However, many of them do file the analogous USDA Rural Utilities Service (RUS) Forms 7 and 12.
Annual responses are due in April of the following year, with new XBRL filings added to FERC’s RSS feed on a rolling basis. Revisions and late filings typically continue to trickle in for months after the filing deadline.
What does the original data look like?#
General Comments#
The FERC 1 database is poorly normalized and the data itself does not appear to be subject to much quality control. For this reason, we strongly encourage the use of the processed FERC data in the PUDL database when it is available.
To identify the source data for each record, many of the FERC Form 1 tables in PUDL
contain a record_id field that indicates what table and record originates from (in
either the XBRL or DBF-derived SQLite databases). The record_id is not in general a
unique identifier within a given PUDL database table, because in many cases the original
data has been reshaped and normalized such that several records in the PUDL database can
be traced back to a single record in the original data.
DBF Reporting Era (1994-2020)#
The data is published as a collection of Visual FoxPro databases: one per year beginning in 1994. The databases all share a very similar structure and contain a total of 116 data tables and ~8GB of raw data (though 90% of that data is in 3 tables containing binary data). The final release of Visual FoxPro was v9.0 in 2007. Its extended support period ended in 2015. The bridge application which allowed this database to be used in Microsoft Access has been discontinued. FERC’s use of this database format created a significant barrier to data access.
The DBF files are no longer being updated by FERC. When revisions to older filings are submitted, they appear in the raw XBRL data, but those revisions are not currently being integrated into the PUDL outputs. The DBF files covering 1994-2020 reporting are archived on Zenodo under concept DOI 10.5281/zenodo.4127043
XBRL Reporting Era (2021-present)#
New XBRL filings are published on a rolling basis to an RSS feed as they are received by FERC. We periodically capture a snapshot of all available filings from that feed and archive it on Zenodo under concept DOI 10.5281/zenodo.4127043 alongside the old DBF database files.
The structure of the XBRL filings is defined by an XBRL taxonomy. The taxonomies defining the structure of all the FERC XBRL filings can be browsed on the FERC website. Like the old DBF data base schemas, the XBRL taxonomies are updated annually, but typically change very slowly. We archive the taxonomies alongside the filing data and use them to define the structure of the database of raw filings that we build out of the XBRL.
Notable Irregularities#
We occasionally see revisions to older filings from the DBF reporting era appear in the XBRL data. Ideally these revised data would supersede the older data from the DBF files. We have not yet dedicated resources to implementing this refinement since it’s a significant amount of work, and there are very few of these filings.
The FERC Form 1 DBF databases are not particularly relational. The only foreign key
relationships that exist map respondent_id fields in the individual data tables back
to f1_respondent_id. In theory, most of the data tables use report_year,
respondent_id, row_number, spplmnt_num and report_prd as a composite
primary key.
In practice, there are several thousand records (out of ~12 million), including some in almost every table, that violate the uniqueness constraint on those primary keys. Since there aren’t many meaningful foreign key relationships anyway, rather than dropping the records with non-unique natural composite keys, we chose to preserve all of the records and use surrogate auto-incrementing primary keys in the cloned SQLite database.
Lots of the data included in the FERC tables is unstructured and difficult to parse. None of the tables have record identification and they sometimes contain multiple rows pertaining to the same plant or portion of a plant. For example, a utility might report values for individual plants as well as the sum total, rendering any aggregations performed on the column inaccurate. Sometimes there are values reported for the total rows and not the individual plants making them difficult to simply remove. Moreover, these duplicate rows are incredibly difficult to identify.
To improve the usability of this data we have done a lot of work to link FERC plants and utilities to their EIA counterparts, and to associate related FERC plant records across different reporting years.
We have manually compiled a mapping between the FERC utility IDs for the DBF and XBRL reporting eras.
We have manually compiled a mapping between those unified FERC utilities and the corresponding EIA utilities.
We have compiled a manual mapping between the FERC plant names and corresponding EIA plant IDs.
We have created a programmatic, probabilistic linkage between sub-plant parts reporting in FERC Form 1 and the associated generators, prime movers, ownership, or other slices of EIA plants.
We have created a programmatic, probabilistic linkage between related FERC plant records, since FERC does not use plant IDs.
We have only integrated a small subset of the available Form 1 tables. If there’s data you need from Form 1 in bulk, you can hire us to liberate it, or we’re happy to accept open source contributions. Some tables only require a couple of hours of work, while others can be quite involved.
PUDL Data Transformations#
To see the transformations applied to the data in each table, you can read the
docstrings for pudl.transform.ferc1 created for each table’s
respective transform function.