Developer Guide

• Development Setup
  • Fork and Clone the PUDL Repository
  • Create the PUDL Dev Environment
  • Updating the PUDL Development Environment
  • Set Up Code Linting
  • Creating a Workspace
• Running the ETL Pipeline
  • Database initialization
  • Database schema migration
  • Dagster
  • Running the ETL via the Dagster UI
  • Running the ETL with CLI Commands
  • Settings Files
  • The Fast ETL
  • The Full ETL
  • Custom ETL
  • Additional Notes
  • Foreign Keys
• Developing with Dagster
  • Reloading code locations
  • Viewing Logs
  • Keeping local Dagster instance up-to-date
  • Assets getting out of sync
  • Configuring resources
• Project Management
  • Issues and Project Tracking
  • Our GitHub Workflow
  • Pull Requests
  • Releases
  • User Support
• Testing PUDL
  • Software Tests
  • Running the tests and other tasks with pixi
  • Selecting Input Data for Integration Tests
  • Running pytest Directly
• Data validation quickstart
  • Setup
  • Running the data validation tests
  • Updating table schemas
  • Updating row counts
• Data validation reference
  • Overview
  • Common workflows
  • PUDL Specific Design Choices
  • Branch builds
  • Running dbt directly
  • Data validation in our integration tests
  • Debugging data validation failures
  • Applying pre-defined validations to existing data
  • Adding new tables
  • Defining new data validation tests
• Metadata editing guide
  • Dataset-level metadata
  • Table-level metadata
  • Field metadata
• Building the Documentation
• Working with the Datastore
  • Adding a new Dataset to the Datastore
• Converting raw FERC data to SQLite
• Existing Data Updates
  • 1. Obtain Fresh Data
  • 2. Map the Structure of the New Data
  • 3. Test Data Extraction
  • 4. Update Table & Column Transformations
  • 5. Update the PUDL DB Schema
  • 6. Connect Datasets
  • 7. Update the Output Routines
  • 8. Run the ETL
  • 9. Run and Update Data Validations
  • 10. Update the Documentation
• Run a Versioned Release
  • Here’s how to do it!
• PUDL ID Mapping
  • The Status of FERC 1 and EIA IDs
  • PUDL IDs
  • FERC1 Utility IDs
  • Checking for Unmapped Records
  • Assigning PUDL IDs to Unmapped Records
  • Testing Newly Mapped Records
  • Integrating Newly Mapped Records into PUDL
• Naming Conventions
  • Asset Naming Conventions
  • Columns and Field Names
  • Naming Conventions in Code
• Data and ETL Design Guidelines
  • Input vs. Output Data
  • Minimize Data Alteration
  • Make Tidy Data
  • Use Consistent Units
  • Partition Big Data
  • Complete, Continuous Time Series
• Nightly Data Builds
  • Breaking the Builds
  • The GitHub Action
  • Deployment Action
  • Google Compute Engine
  • Docker
  • How to access the nightly build outputs from AWS
  • How to access the nightly build outputs and logs (for the Catalyst team only)
• FERC EQR Data Builds
  • Notable differences from nightly builds
• Infrastructure as Code
  • Overview
  • Setup
  • Development
• Managing External Contributions
  • Overview
  • Guidelines for contributors
  • Github incantations