Spark & Databricks Performance Reviewer

(This is the scanner's real output on the bundled examples/ files, shortened.)

# Spark performance review
Files scanned: 2 | findings: 20 (high 6, medium 11, low 3)

| high   | daily_sales_job.py:8   | Row-at-a-time Python UDF            | use built-ins / pandas_udf |
| high   | daily_sales_job.py:20  | repartition(1) before write          | remove; optimizeWrite/autoCompact |
| high   | daily_sales_job.py:21  | toPandas() on full DataFrame         | limit/aggregate first |
| high   | slow_query_plan.txt:8  | BatchEvalPython in plan              | replace UDF (keeps Photon/codegen) |
| high   | slow_query_plan.txt:18 | Small files: 18432 files, avg 2.3 MiB| OPTIMIZE + autoCompact / liquid clustering |
| high   | slow_query_plan.txt:20 | Skew: max is 649x median             | AQE skewJoin, salt hot key, null keys |
| medium | daily_sales_job.py:17  | Window.orderBy without partitionBy   | add partitionBy |
| medium | slow_query_plan.txt:3  | SortMergeJoin                        | broadcast(stores) (~5k rows) |
| medium | slow_query_plan.txt:1  | No AdaptiveSparkPlan                 | enable AQE |

The agent then confirms each finding in context and proposes the top 3 changes: (1) replace clean_region with F.upper(F.trim("region")); (2) orders.join(F.broadcast(stores), "store_id"); (3) drop repartition(1) and partitionBy("order_id") (a high-cardinality key), then use CLUSTER BY (order_date) and OPTIMIZE. It ends with a before/after measurement table for you to fill in. The numbers in the sample plan are illustrative inputs, not results.