Snowflake ETL Example With Pipe, Stream & Task Objects Part-07
Watch E2E Snowflake ETL Demo
You can watch the complete hands on video tutorial
Data Validation Post Data Loading
select count(*) from ch19.landing_zone.landing_order; --10003 (10001) new records + one update
select count(*) from ch19.landing_zone.landing_item; --2793 (2791)new records + one update
select count(*) from ch19.landing_zone.landing_customer; -- 8889 (8887) new records + one update
select count(*) from ch19.landing_zone.landing_order_stm;
select count(*) from ch19.landing_zone.landing_item_stm;
select count(*) from ch19.landing_zone.landing_customer_stm;
select count(*) from ch19.curated_zone.curated_order; --10001(10002) one updated and one inserted
select count(*) from ch19.curated_zone.curated_item; --2791 (2792) one updated and one inserted
select count(*) from ch19.curated_zone.curated_customer; -- 8887 (8888) one updated and one inserted
select count(*) from ch19.consumption_zone.order_fact; --5740 (5741)
select count(*) from ch19.consumption_zone.item_dim; --2791 (2791)
select count(*) from ch19.consumption_zone.customer_dim; -- 8887 (8888)
use schema ch19.consumption_zone;
select * from table(information_schema.task_history())
where name in ('ORDER_FACT_TSK')
order by scheduled_time;
-- lets validate the change using time travel feature
select * from ch19.consumption_zone.customer_dim where customer_id = 'AAAAAAAAPOJJJDAA'
union all
select * from ch19.consumption_zone.customer_dim at(offset => -60*10) where customer_id = 'AAAAAAAAPOJJJDAA';
select * from ch19.consumption_zone.item_dim where item_id = 'AAAAAAAACDLBXPPP'
union all
select * from ch19.consumption_zone.item_dim at(offset => -60*10) where item_id = 'AAAAAAAACDLBXPPP';
SQL Scripts - Part 01 to Part 07
- Part-01 covers curated Zone SQL Script & Data Loading
- Part-02 covers curated Zone SQL Script & Data Loading
- Part-03 covers consumption Zone SQL Script & Data Loading
- Part-04 covers stage and pipe creation script for landing layer
- Part-05 stream and task creation for curated layers
- Part-06 stream and task cration for consumption layer
- Part-07 Lets validate the merge statements (Insert/Update)
Watch E2E Snowflake ETL Demo
You can watch the complete hands on video tutorial