Category: apache Nifi

Apache NiFi – Queues ”; Previous Next The Apache NiFi data flow connection has a queuing system to handle the large amount of data inflow. These queues can handle very large amount of FlowFiles to let the processor process them serially. The queue in the above image has 1 flowfile transferred through success relationship. A user can check the flowfile by selecting the List queue option in the drop down list. In case of any overload or error, a user can also clear the queue by selecting the empty queue option and then the user can restart the flow to get those files again in the data flow. The list of flowfiles in a queue, consist of position, UUID, Filename, File size, Queue Duration, and Lineage Duration. A user can see all the attributes and content of a flowfile by clicking the info icon present at the first column of the flowfile list. Print Page Previous Next Advertisements ”;

Apache NiFi – Creating Flows ”; Previous Next Apache NiFi offers a large number of components to help developers to create data flows for any type of protocols or data sources. To create a flow, a developer drags the components from menu bar to canvas and connects them by clicking and dragging the mouse from one component to other. Generally, a NiFi has a listener component at the starting of the flow like getfile, which gets the data from source system. On the other end of there is a transmitter component like putfile and there are components in between, which process the data. For example, let create a flow, which takes an empty file from one directory and add some text in that file and put it in another directory. To begin with, drag the processor icon to the NiFi canvas and select GetFile processor from the list. Create an input directory like c:inputdir. Right-click on the processor and select configure and in properties tab add Input Directory (c:inputdir) and click apply and go back to canvas. Drag the processor icon to the canvas and select the ReplaceText processor from the list. Right-click on the processor and select configure. In the properties tab, add some text like “Hello tutorialspoint.com” in the textbox of Replacement Value and click apply. Go to settings tab, check the failure checkbox at right hand side, and then go back to the canvas. Connect GetFIle processor to ReplaceText on success relationship. Drag the processor icon to the canvas and select the PutFile processor from the list. Create an output directory like c:outputdir. Right-click on the processor and select configure. In the properties tab, add Directory (c:outputdir) and click apply and go back to canvas. Go to settings tab and check the failure and success checkbox at right hand side and then go back to the canvas. Connect the ReplaceText processor to PutFile on success relationship. Now start the flow and add an empty file in input directory and you will see that, it will move to output directory and the text will be added to the file. By following the above steps, developers can choose any processor and other NiFi component to create suitable flow for their organisation or client. Print Page Previous Next Advertisements ”;

Apache NiFi – Useful Resources ”; Previous Next The following resources contain additional information on Apache NiFi. Please use them to get more in-depth knowledge on this. Useful Links on Apache NiFi Apache NiFi − Official site of Apache NiFi. Apache NiFi @ Wikipedia − Apache NiFi, its history and various other terms has been explained in simple language. Useful Books on Apache NiFi To enlist your site on this page, please drop an email to [email protected] Print Page Previous Next Advertisements ”;

Apache NiFi – Processors Relationship ”; Previous Next In an Apache NiFi data flow, flowfiles move from one to another processor through connection that gets validated using a relationship between processors. Whenever a connection is created, a developer selects one or more relationships between those processors. As you can see in the above image, the check boxes in black rectangle are relationships. If a developer selects these check boxes then, the flowfile will terminate in that particular processor, when the relationship is success or failure or both. Success When a processor successfully processes a flowfile like store or fetch data from any datasource without getting any connection, authentication or any other error, then the flowfile goes to success relationship. Failure When a processor is not able to process a flowfile without errors like authentication error or connection problem, etc. then the flowfile goes to a failure relationship. A developer can also transfer the flowfiles to other processors using connections. The developer can select and also load balance it, but load balancing is just released in version 1.8, which will not be covered in this tutorial. As you can see in the above image the connection marked in red have failure relationship, which means all flowfiles with errors will go to the processor in left and respectively all the flowfiles without errors will be transferred to the connection marked in green. Let us now proceed with the other relationships. comms.failure This relationship is met, when a Flowfile could not be fetched from the remote server due to a communications failure. not.found Any Flowfile for which we receive a ‘Not Found’ message from the remote server will move to not.found relationship. permission.denied When NiFi unable to fetch a flowfile from the remote server due to insufficient permission, it will move through this relationship. Print Page Previous Next Advertisements ”;