Snowflake ETL Example With Pipe, Stream & Task Objects Part-04
Watch E2E Snowflake ETL Demo
You can watch the complete hands on video tutorial
Step - Create Object and list them
-- order stage
create stage delta_orders_s3
url = 's3://toppertips/delta/orders'
comment = 'feed delta order files';
-- item stage
create stage delta_items_s3
url = 's3://toppertips/delta/items'
comment = 'feed delta item files';
-- customer stage
create stage delta_customer_s3
url = 's3://toppertips/delta/customers'
comment = 'feed delta customer files';
show stages;
Create Pipe Objects for each of the table
create or replace pipe order_pipe
auto_ingest = true
as
copy into landing_order from @delta_orders_s3
file_format = (type=csv COMPRESSION=none)
pattern='.*order.*[.]csv'
ON_ERROR = 'CONTINUE';
create or replace pipe item_pipe
auto_ingest = true
as
copy into landing_item from @delta_items_s3
file_format = (type=csv COMPRESSION=none)
pattern='.*item.*[.]csv'
ON_ERROR = 'CONTINUE';
create or replace pipe customer_pipe
auto_ingest = true
as
copy into landing_customer from @delta_customer_s3
file_format = (type=csv COMPRESSION=none)
pattern='.*customer.*[.]csv'
ON_ERROR = 'CONTINUE';
Step - Review Pipe Status
show pipes;
select system$pipe_status('order_pipe');
select system$pipe_status('item_pipe');
select system$pipe_status('customer_pipe');
SQL Scripts - Part 01 to Part 07
- Part-01 covers curated Zone SQL Script & Data Loading
- Part-02 covers curated Zone SQL Script & Data Loading
- Part-03 covers consumption Zone SQL Script & Data Loading
- Part-04 covers stage and pipe creation script for landing layer
- Part-05 stream and task creation for curated layers
- Part-06 stream and task cration for consumption layer
- Part-07 Lets validate the merge statements (Insert/Update)
Watch E2E Snowflake ETL Demo
You can watch the complete hands on video tutorial