All Stories

What is new in Spark 3.0

This write up talks about Apache Spark 3.0 features and improvements. Apache Spark 3.0 is a major release and currently available in preview mode. Release 3.0.0 is a major release...

Snowflake Architecture Cheat Sheet

Architecture No software, No Hardware, No maintenance. Snowflake is provided as Software-as-a-Service (SaaS) that runs completely on cloud infrastructure Snowflake uses a central data repository for persisted data that is...

Snowflake Data Warehouse Best Practices

Snowflake is a cloud-native and easy to use virtual data warehouse system. Since it is built on the top of the cloud-native platform, traditional best practices do not applicable anymore....

Snowflake Data Warehouse Glossary

Snowflake Data Warehouse Glossary talks about all the important keywords which a developer must know. Snowflake cloud data platform is cloud-native, faster, easier to use, and far more flexible than...

PySpark Tutorial

In this PySpark Tutorial, we will understand why PySpark is becoming popular among data engineers and data scientist. This PySpark Tutorial will also highlight the key limilation of PySpark over...

What Is Data Lineage

What is data lineage and why it is important. Data lineage is nothing but its origins and transformation that data goes through with time. Data lineage can also be expressed...

Data Lineage Vs Data Provenance

Data Lineage and Data Provenance are not the same thing. Many data engineer and architect use them interchangible but they are two different concept and has its separate meaning.

Spark Dataframe Minus Minutes Operation In Scala

How to perform minus operation on a date type or timestamp time.

What Is Apache Nifi

Apache NiFi is a software project from the Apache Software Foundation designed to automate the flow of data between software systems (file system, RDBMS, APIs etc in and out) ....

Check Unix OS Version Using Putty

Many times you get access to a Unix or Linux box via terminal. Before start using the terminal, you may want to know the Unix flavor. This article will help...