Snowflake MigrationClickHouse Workshops

Superset di Snowflake

Membangun tiga dashboard operasional di sisi sumber: koneksi, dataset, chart, dan perakitan.

Panduan ini menuntun Anda membuat ketiga dashboard di Apache Superset pada http://localhost:8088 (admin / admin).

Urutan pengerjaan:

  1. Jalankan Superset dan daftarkan koneksi Snowflake
  2. Buat semua dataset (query SQL yang disimpan sebagai dataset bernama)
  3. Bangun chart dan rakit dashboard

Langkah 1: Jalankan Superset dan Hubungkan ke Snowflake

Jalankan Superset

Image Superset dikustomisasi agar menyertakan driver Snowflake dan ClickHouse. Gunakan --build pada run pertama supaya Docker membangunnya:

cd "$(git rev-parse --show-toplevel)/workshop_public/snowflake_migration_lab/01-setup-snowflake/superset"
source ../.env
docker compose up -d --build

Daftarkan koneksi database (otomatis)

Dari direktori superset/, jalankan:

source ../.env && ./init_superset.sh

Skrip menunggu Superset siap, lalu mendaftarkan NYC Taxi — Snowflake (Source) secara otomatis. Output yang diharapkan:

>>> Superset is up.
>>> Authenticated.
>>> CSRF token obtained.
>>> Registering: NYC Taxi — Snowflake (Source)
    Registered successfully.

Daftarkan koneksi database (alternatif manual)

Jika Anda lebih suka mendaftarkan melalui UI:

  1. Buka Settings → Database Connections → + Database
  2. Pilih Snowflake
  3. Isi SQLAlchemy URI — URL-encode setiap karakter khusus dalam password Anda (# → %23, ! → %21, @ → %40, dan seterusnya):
snowflake://<USER>:<URL_ENCODED_PASSWORD>@<SNOWFLAKE_ORG>-<SNOWFLAKE_ACCOUNT>/NYC_TAXI_DB/ANALYTICS?warehouse=ANALYTICS_WH&role=ANALYST_ROLE
  1. Setel Display Name: NYC Taxi — Snowflake (Source)
  2. Di bawah Advanced → SQL Lab: aktifkan Allow this database to be explored dan Allow DML
  3. Klik Test Connection → seharusnya menampilkan "Connection looks good!"
  4. Klik Connect

Langkah 2: Buat Semua Dataset

Semua chart memakai dataset virtual — query SQL yang disimpan sebagai dataset bernama.

Cara membuat setiap dataset:

  1. Buka Datasets → + Dataset
  2. Pilih database: NYC Taxi — Snowflake (Source)
  3. Klik Create dataset from SQL query dan tempelkan SQL di bawah ini
  4. Simpan dengan nama yang ditunjukkan
  5. Setelah menyimpan: buka Datasets → ikon pensil → tab Columns → "Sync columns from source" → Save. Tanpa langkah ini, chart builder akan menampilkan 0 kolom.

Dataset Dashboard 1

ops_hourly_revenue — Revenue per jam menurut borough

Schema: ANALYTICS

SELECT
    DATE_TRUNC('hour', pickup_at)                      AS hour_bucket,
    pickup_borough,
    COUNT(*)                                           AS trip_count,
    SUM(total_amount_usd)                              AS total_revenue,
    AVG(tip_amount_usd / NULLIF(fare_amount_usd, 0))  AS avg_tip_rate,
    AVG(trip_distance_miles)                           AS avg_distance_miles
FROM ANALYTICS.FACT_TRIPS
WHERE pickup_at >= DATEADD('day', -7, CURRENT_TIMESTAMP())
  AND pickup_borough IS NOT NULL
GROUP BY 1, 2
ORDER BY 1 DESC, total_revenue DESC

ops_zone_agg — Agregat zone

Schema: ANALYTICS

SELECT
    hour_bucket,
    zone_id,
    trips,
    revenue,
    avg_distance
FROM ANALYTICS.AGG_HOURLY_ZONE_TRIPS
WHERE hour_bucket >= DATEADD('day', -7, CURRENT_TIMESTAMP())

ops_payment_split — Pembagian payment type

Schema: ANALYTICS

SELECT
    payment_type,
    COUNT(*)              AS trip_count,
    SUM(total_amount_usd) AS total_revenue
FROM ANALYTICS.FACT_TRIPS
WHERE pickup_at >= DATEADD('day', -7, CURRENT_TIMESTAMP())
GROUP BY 1

Dataset Dashboard 2

exec_rolling_avg — Rata-rata bergulir 7 hari

Schema: ANALYTICS

SELECT
    pickup_at::DATE                                          AS trip_date,
    COUNT(*)                                                 AS daily_trip_count,
    AVG(trip_distance_miles)                                 AS daily_avg_distance,
    AVG(AVG(trip_distance_miles)) OVER (
        ORDER BY pickup_at::DATE
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
    )                                                        AS rolling_7d_avg_distance,
    SUM(total_amount_usd)                                    AS daily_revenue,
    SUM(SUM(total_amount_usd)) OVER (
        ORDER BY pickup_at::DATE
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
    )                                                        AS rolling_7d_revenue
FROM ANALYTICS.FACT_TRIPS
GROUP BY 1
ORDER BY 1 DESC
LIMIT 365

exec_top_trips — 10 trip bernilai tertinggi per borough

Schema: ANALYTICS

Memakai QUALIFY milik Snowflake — salah satu tantangan migrasi utama. Penulisan ulang untuk ClickHouse memerlukan subquery.

SELECT
    trip_id,
    pickup_at,
    pickup_borough,
    total_amount_usd,
    tip_amount_usd,
    trip_distance_miles,
    ROW_NUMBER() OVER (
        PARTITION BY pickup_borough
        ORDER BY total_amount_usd DESC
    ) AS rank_in_borough
FROM ANALYTICS.FACT_TRIPS
WHERE pickup_at::DATE = CURRENT_DATE() - 1
QUALIFY rank_in_borough <= 10
ORDER BY pickup_borough, rank_in_borough

exec_surge — Dampak surge pricing

Schema: ANALYTICS

SELECT
    CASE
        WHEN surge_multiplier >= 2.0 THEN 'High Surge (2x+)'
        WHEN surge_multiplier >= 1.5 THEN 'Medium Surge (1.5–2x)'
        WHEN surge_multiplier > 1.0  THEN 'Low Surge (1–1.5x)'
        ELSE 'No Surge (1x)'
    END                                    AS surge_category,
    COUNT(*)                               AS trip_count,
    ROUND(AVG(total_amount_usd), 2)        AS avg_total_fare,
    ROUND(AVG(fare_amount_usd), 2)         AS avg_base_fare,
    ROUND(AVG(surge_multiplier), 2)        AS avg_surge
FROM ANALYTICS.FACT_TRIPS
WHERE surge_multiplier IS NOT NULL
GROUP BY 1
ORDER BY avg_surge DESC

Dataset Dashboard 3

dqa_rating_dist — Distribusi rating driver

Schema: RAW ← ubah ini saat membuat dataset

Mengquery RAW.TRIPS_RAW secara langsung melalui sintaks colon-path VARIANT. Ini adalah query yang sengaja dibuat lambat — target benchmark ClickHouse.

SELECT
    ROUND(TRIP_METADATA:driver.rating::FLOAT, 1)                           AS rating_bucket,
    COUNT(*)                                                               AS trip_count,
    ROUND(AVG(TOTAL_AMOUNT), 2)                                            AS avg_fare,
    ROUND(AVG(DATEDIFF('minute', PICKUP_DATETIME, DROPOFF_DATETIME)), 1)   AS avg_duration_minutes
FROM RAW.TRIPS_RAW
WHERE TRIP_METADATA:driver IS NOT NULL
  AND TRIP_METADATA:driver.rating IS NOT NULL
GROUP BY 1
ORDER BY 1

dqa_vehicle — Revenue menurut vehicle type

Schema: ANALYTICS

SELECT
    vehicle_type,
    COUNT(*)                 AS trip_count,
    SUM(total_amount_usd)    AS total_revenue,
    AVG(total_amount_usd)    AS avg_fare,
    AVG(trip_distance_miles) AS avg_distance
FROM ANALYTICS.FACT_TRIPS
WHERE vehicle_type IS NOT NULL
GROUP BY 1
ORDER BY total_revenue DESC

dqa_traffic — Traffic level vs durasi trip

Schema: ANALYTICS

SELECT
    traffic_level,
    COUNT(*)                   AS trip_count,
    AVG(duration_minutes)      AS avg_duration_minutes,
    AVG(trip_distance_miles)   AS avg_distance_miles,
    AVG(total_amount_usd)      AS avg_fare
FROM ANALYTICS.FACT_TRIPS
WHERE traffic_level IS NOT NULL
GROUP BY 1
ORDER BY avg_duration_minutes DESC

dqa_platform — Tren app platform

Schema: ANALYTICS

SELECT
    pickup_at::DATE       AS trip_date,
    app_platform,
    COUNT(*)              AS trip_count,
    AVG(surge_multiplier) AS avg_surge
FROM ANALYTICS.FACT_TRIPS
WHERE app_platform IS NOT NULL
  AND pickup_at >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY 1, 2
ORDER BY 1 DESC

Langkah 3: Bangun Dashboard

Kesepuluh dataset sekarang sudah siap. Buat chart dan tambahkan ke dashboard.


Dashboard 1: Operations Command Center

Tujuan: Tampilan operasional real-time yang menunjukkan 7 hari terakhir. Ini dashboard pertama yang dialihkan partner ke ClickHouse di Bagian 2.

Buat dashboard:

  1. Dashboards → + Dashboard
  2. Judul: Operations Command Center
  3. Auto-refresh: setiap 15 menit (··· → Edit dashboard → Auto-refresh)

Chart 1: Trips per Hour (Line chart)

  • Chart type: Line Chart
  • Dataset: ops_hourly_revenue
  • X-axis: hour_bucket
  • Metrics: SUM(trip_count)
  • Series: pickup_borough
  • Title: Trips per Hour — Last 7 Days

Chart 2: Revenue by Borough (Bar chart)

  • Chart type: Bar Chart
  • Dataset: ops_hourly_revenue
  • X-axis: pickup_borough
  • Metrics: SUM(total_revenue)
  • Sort: menurun berdasarkan metrik
  • Title: Total Revenue by Borough — Last 7 Days

Chart 3: Payment Type Split (Pie chart)

  • Chart type: Pie Chart
  • Dataset: ops_payment_split
  • Dimension: payment_type
  • Metric: SUM(trip_count)
  • Show labels: aktif
  • Title: Trip Count by Payment Type

Chart 4: Total Trips (Big Number)

  • Chart type: Big Number with Trendline
  • Dataset: ops_hourly_revenue
  • Metric: SUM(trip_count)
  • Title: Total Trips (Last 7 Days)

Chart 5: Borough Performance Summary (Table)

  • Chart type: Table
  • Dataset: ops_hourly_revenue
  • Columns: pickup_borough, SUM(trip_count), SUM(total_revenue), AVG(avg_tip_rate)
  • Row limit: 10
  • Sort: SUM(total_revenue) menurun
  • Title: Borough Performance Summary

Tata letak:

[ Total Trips — Big Number ]  [ Total Revenue — Big Number (add 2nd)  ]
[ Trips per Hour — Line chart (full width)                            ]
[ Revenue by Borough — Bar ]  [ Payment Type Split — Pie             ]
[ Borough Performance — Table (full width)                            ]

Dashboard 2: Executive Weekly Report

Tujuan: Tampilan strategis untuk tinjauan bisnis mingguan. Menampilkan window function dan sintaks khusus Snowflake (QUALIFY) yang memerlukan penulisan ulang di ClickHouse.

Buat dashboard:

  1. Dashboards → + Dashboard
  2. Judul: Executive Weekly Report
  3. Auto-refresh: 1 jam

Chart 6: Rolling 7-Day Revenue Trend (Line chart)

  • Chart type: Line Chart
  • Dataset: exec_rolling_avg
  • X-axis: trip_date
  • Metrics: MAX(daily_revenue), MAX(rolling_7d_revenue)
  • Title: Daily Revenue with 7-Day Rolling Average

Chart 7a: Daily Trip Volume (Big Number)

  • Chart type: Big Number with Trendline
  • Dataset: exec_rolling_avg
  • Metric: MAX(daily_trip_count)
  • Title: Daily Trip Volume (Last Year)

Chart 7b: Rolling 7-Day Average Distance (Line chart)

  • Chart type: Line Chart
  • Dataset: exec_rolling_avg
  • X-axis: trip_date
  • Metrics: MAX(rolling_7d_avg_distance)
  • Title: Rolling 7-Day Average Distance (miles)

Chart 8: Top 10 Trips per Borough (Table)

  • Chart type: Table
  • Dataset: exec_top_trips
  • Query Mode: RAW RECORDS ← penting: dataset ini memakai QUALIFY sehingga Superset tidak boleh mengagregasi ulang
  • Columns: pickup_borough, rank_in_borough, total_amount_usd, tip_amount_usd, trip_distance_miles, pickup_at
  • Sort By: total_amount_usd menurun
  • Row limit: 60
  • Title: Top 10 Trips per Borough — Yesterday
  • Catatan: Memakai QUALIFY — khusus Snowflake, harus ditulis ulang sebagai subquery untuk ClickHouse

Chart 9: Surge Pricing Breakdown (Mixed chart)

  • Chart type: Mixed Chart ← gunakan ini, bukan Bar Chart; Bar Chart tidak mendukung sumbu sekunder
  • Dataset: exec_surge
  • X-axis: surge_category
  • Query A — Bar: metrik SUM(trip_count), label Trip Count
  • Query B — Line: metrik MAX(avg_total_fare), label Avg Total Fare, Y-axis: Right
  • Sort: SUM(trip_count) menurun
  • Title: Trip Volume and Average Fare by Surge Category

Chart 10: Surge Distribution (Pie chart)

  • Chart type: Pie Chart
  • Dataset: exec_surge
  • Dimension: surge_category
  • Metric: SUM(trip_count)
  • Title: Surge Pricing Distribution

Tata letak:

[ Rolling Revenue — Line chart (full width)                              ]
[ Daily Trip Volume — Big Number (50%) ]  [ Avg Distance — Line (50%)   ]
[ Top 10 Trips — Table (60%) ]  [ Surge Distribution — Pie (40%)        ]
[ Surge Breakdown — Bar chart (full width)                               ]

Dashboard 3: Driver & Quality Analytics

Tujuan: Penyelaman mendalam pada performa driver dan kualitas trip. Sengaja menjadi dashboard paling lambat — mengquery RAW.TRIPS_RAW secara langsung melalui akses VARIANT. Catat waktu query di sini sebagai baseline untuk benchmark performa ClickHouse di Bagian 2.

Buat dashboard:

  1. Dashboards → + Dashboard
  2. Judul: Driver & Quality Analytics
  3. Auto-refresh: 1 jam

Chart 11: Trip Count by Driver Rating (Bar chart)

  • Chart type: Bar Chart
  • Dataset: dqa_rating_dist
  • X-axis: rating_bucket
  • Metrics: SUM(trip_count)
  • Title: Trip Count by Driver Rating
  • Catatan: Memindai RAW.TRIPS_RAW dengan akses VARIANT — perhatikan waktu query dibandingkan ClickHouse

Chart 12: Average Fare by Rating (Line chart)

  • Chart type: Line Chart
  • Dataset: dqa_rating_dist
  • X-axis: rating_bucket
  • Metrics: MAX(avg_fare)
  • Title: Average Fare by Driver Rating

Chart 13: Revenue by Vehicle Type (Horizontal bar)

  • Chart type: Bar Chart (horizontal)
  • Dataset: dqa_vehicle
  • X-axis: vehicle_type
  • Metrics: SUM(total_revenue), SUM(trip_count) (sumbu sekunder)
  • Title: Revenue and Trip Count by Vehicle Type

Chart 14: Traffic Level Impact (Bar chart)

  • Chart type: Bar Chart
  • Dataset: dqa_traffic
  • X-axis: traffic_level
  • Metrics: MAX(avg_duration_minutes), MAX(avg_distance_miles) (sumbu sekunder)
  • Title: Average Trip Duration and Distance by Traffic Level

Chart 15: Daily Trips by App Platform (Line chart)

  • Chart type: Line Chart
  • Dataset: dqa_platform
  • X-axis: trip_date
  • Metrics: SUM(trip_count)
  • Series: app_platform
  • Title: Daily Trips by App Platform — Last 30 Days

Chart 16: Surge by Platform (Table)

  • Chart type: Table
  • Dataset: dqa_platform
  • Columns: app_platform, SUM(trip_count), AVG(avg_surge)
  • Row limit: 10
  • Title: Surge by Platform

Tata letak:

[ Trip Count by Rating — Bar ]  [ Avg Fare by Rating — Line            ]
[ Revenue by Vehicle Type — Horizontal bar (full width)                ]
[ Traffic Level Impact — Bar (50%) ]  [ Surge by Platform — Table (50%)]
[ Daily Trips by Platform — Line chart (full width)                    ]

Langkah 4: Verifikasi

  1. Buka setiap dashboard dan pastikan semua chart memuat tanpa error
  2. Untuk Dashboard 3, catat waktu eksekusi query dqa_rating_dist di Snowflake UI → Activity → Query History — simpan ini sebagai benchmark migrasi Anda

Chart Superset yang menampilkan distribusi rating driver pada dataset Snowflake, terkonsentrasi antara 4.0 dan 5.0

Langkah 5: Ekspor untuk Digunakan Ulang

Setelah dashboard selesai, ekspor agar run berikutnya dapat mengimpornya secara otomatis:

  1. Buka setiap dashboard → ··· → Export (tersimpan sebagai .zip)
  2. Letakkan file-nya di superset/dashboards/:
    • 01_operations_command_center.zip
    • 02_executive_weekly_report.zip
    • 03_driver_quality_analytics.zip
  3. Jalankan ulang ./init_superset.sh — skrip akan mengimpornya secara otomatis pada setup berikutnya

Penting — kredensial placeholder di ZIP yang di-commit.

Setiap *.zip yang di-commit memiliki koneksi database yang disunting di databases/*.yaml:

sqlalchemy_uri: snowflake://LAB_USER:XXXXXXXXXX@MYORG-MYACCOUNT/NYC_TAXI_DB/ANALYTICS?role=ANALYST_ROLE&warehouse=ANALYTICS_WH
  • Impor otomatis (./init_superset.sh) — bekerja apa adanya. Skrip mendaftarkan koneksi Snowflake yang sebenarnya dari .env sebelum mengimpor, lalu menerapkan ulang URI yang benar setelah setiap impor (lihat _update_db di init_superset.sh), sehingga nilai placeholder ditimpa dengan kredensial Anda yang sebenarnya.
  • Impor manual melalui UI Superset — database yang diimpor akan dibuat dengan URI placeholder dan tidak akan terhubung. Setelah impor, buka Settings → Database Connections → Edit pada entri tersebut dan ganti sqlalchemy_uri dengan URI Snowflake Anda yang sebenarnya (mis. snowflake://<USER>:<PASSWORD>@<ORG>-<ACCOUNT>/NYC_TAXI_DB/ANALYTICS?role=ANALYST_ROLE&warehouse=ANALYTICS_WH).
  • Mengekspor ulang dashboard Anda sendiri — Superset menanamkan account locator dan username Anda ke dalam databases/*.yaml saat ekspor. Sebelum meng-commit ZIP hasil ekspor ulang Anda, sunting nilai-nilai tersebut kembali menjadi MYORG-MYACCOUNT / LAB_USER agar identifier akun Anda tidak bocor ke riwayat git.

Di halaman ini

Langkah 1: Jalankan Superset dan Hubungkan ke SnowflakeJalankan SupersetDaftarkan koneksi database (otomatis)Daftarkan koneksi database (alternatif manual)Langkah 2: Buat Semua DatasetDataset Dashboard 1ops_hourly_revenue — Revenue per jam menurut boroughops_zone_agg — Agregat zoneops_payment_split — Pembagian payment typeDataset Dashboard 2exec_rolling_avg — Rata-rata bergulir 7 hariexec_top_trips — 10 trip bernilai tertinggi per boroughexec_surge — Dampak surge pricingDataset Dashboard 3dqa_rating_dist — Distribusi rating driverdqa_vehicle — Revenue menurut vehicle typedqa_traffic — Traffic level vs durasi tripdqa_platform — Tren app platformLangkah 3: Bangun DashboardDashboard 1: Operations Command CenterChart 1: Trips per Hour (Line chart)Chart 2: Revenue by Borough (Bar chart)Chart 3: Payment Type Split (Pie chart)Chart 4: Total Trips (Big Number)Chart 5: Borough Performance Summary (Table)Dashboard 2: Executive Weekly ReportChart 6: Rolling 7-Day Revenue Trend (Line chart)Chart 7a: Daily Trip Volume (Big Number)Chart 7b: Rolling 7-Day Average Distance (Line chart)Chart 8: Top 10 Trips per Borough (Table)Chart 9: Surge Pricing Breakdown (Mixed chart)Chart 10: Surge Distribution (Pie chart)Dashboard 3: Driver & Quality AnalyticsChart 11: Trip Count by Driver Rating (Bar chart)Chart 12: Average Fare by Rating (Line chart)Chart 13: Revenue by Vehicle Type (Horizontal bar)Chart 14: Traffic Level Impact (Bar chart)Chart 15: Daily Trips by App Platform (Line chart)Chart 16: Surge by Platform (Table)Langkah 4: VerifikasiLangkah 5: Ekspor untuk Digunakan Ulang
ID