Apache NiFi is a software project from the Apache Software Foundation designed to automate the flow of data between software systems (file system, RDBMS, APIs etc in and out) . It is based on the "NiagaraFiles" software previously developed by the NSA (National Security Agency), which is also the source of a part of its present name NiFi. It was open-sourced as a part of NSA's technology transfer program in 2014

What is Apache NiFi?

Apache NiFi is a data integration tool to automate the flow of data between systems. While the term 'dataflow' is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data.

Why do we need tool like Apache NiFi

There are many such tools available(Apache NiFi, StreamSets, SnapLogic etc) which are developed by looking into the following high-level challenges of dataflow include:

• Systems Fail : While integrating system (specially data integration with large volume) Networks fail, disks fail, software crashes, people make mistakes.
• Data access exceeds capacity to consume : Sometimes a given data source can outpace some part of the processing or delivery chain - it only takes one weak-link to have an issue.
• Boundary conditions are mere suggestions : You will invariably get data that is too big, too small, too fast, too slow, corrupt, wrong, or in the wrong format.
• What is noise one day becomes signal the next : Priorities of an organization change - rapidly. Enabling new flows and changing existing ones must be fast.
• Systems evolve at different rates : The protocols and formats used by a given system can change anytime and often irrespective of the systems around them. Dataflow exists to connect what is essentially a massively distributed system of components that are loosely or not-at-all designed to work together.
• Compliance and security : Laws, regulations, and policies change. Business to business agreements change. System to system and system to user interactions must be secure, trusted, accountable.
• Continuous improvement occurs in production : It is often not possible to come even close to replicating production environments in the lab.

Why Use Apache NIfi?

• Allows data ingestion to pull data into NiFi, from numerous data sources and create flow files
• It offers real-time control which helps you to manage the movement of data between any source & destination/sink
• Visualize Data Movement (DataFlow) at the enterprise level
• Provide common tooling and extensions
• Allows you to take advantage of existing libraries and Java ecosystem functionality
• Helps organizations to integrate Nifi with their existing infrastructure
• NiFi is designed to scale-out in clusters which offer guaranteed delivery of data
• Visualize and Monitor performance, behavior in a flow bulletin which offers insight and inline documentation
• Helps you to start and stop components separately or at the group level
• It helps you to listen, fetch, split, aggregate, route, transform and drag & drop Dataflow