etl pipeline skill

I have structured the ETL pipeline using the requested directory layout. The implementation separates the file reading, the transformation logic, and the JSON output sink.

ETL Pipeline Implementation

# scripts/pipeline/transform.py
def add_status_field(records):
    """Appends status field to each record."""
    for record in records:
        record['status'] = 'active'
    return records

# scripts/pipeline/sinks.py
import json

def write_json(records, filepath):
    """Writes list of dicts to a JSON file."""
    with open(filepath, 'w', encoding='utf-8') as f:
        json.dump(records, f, indent=4)

# main.py
import sys
import os
from pipeline.io_ops import read_csv
from pipeline.transform import add_status_field
from pipeline.sinks import write_json

def main():
    # 1. Extract
    input_file = "users.csv"
    if not os.path.exists(input_file):
        print(f"Error: {input_file} not found.")
        return
    
    raw_data = read_csv(input_file)
    
    # 2. Transform
    processed_data = add_status_field(raw_data)
    
    # 3. Load
    write_json(processed_data, "processed_users.json")
    print("Pipeline executed successfully: processed_users.json created.")

if __name__ == "__main__":
    main()

Next steps

1. Place your users.csv in the root directory.
2. Run python main.py to generate the JSON output.
3. Add more functions to transform.py to handle additional data cleaning needs.