README.md

Everything Apache NiFi

Timothy Spann @PaaSDev

• https://datainmotion.dev/
• http://dev.to/tspannhw

Updates

• Apache NiFi 1.14.0 is now out. Check it out. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.14.0
• https://www.youtube.com/watch?app=desktop&v=h503MibiENY&feature=youtu.be
• https://bryanbende.com/development/2021/07/19/apache-nifi-1-14-0-secure-by-default
• Latest Version - Apache NiFi 1.13.2! Don't use Apache NiFi 1.13.1
• Apache NiFi 1.13.2.2 - https://www.datainmotion.dev/2021/04/nifi-on-cloudera-data-platform-upgrade.html
• Apache NiFi 1.13.0 - https://www.datainmotion.dev/2021/02/new-features-of-apache-nifi-1130.html
• Apache NiFi 1.12 - https://twitter.com/pvillard31/status/1296469452180119553
• Apache NiFi on K8 - https://twitter.com/AppInt4All/status/1306643414725750784?s=20
• Questions on NiFi Jam - https://blog.cloudera.com/top-5-questions-about-apache-nifi/

Must Watch Video

• Apache NiFi Part 5: Primary Node / Complex Expression Language: https://www.youtube.com/watch?v=JbUjYr6Kd3I
• Apache NiFi Part 4: Scheduling: https://www.youtube.com/watch?v=pZq0EbfDBy4
• Apache NiFi Part 3: Load Balancing: https://www.youtube.com/watch?v=by9P0Zi8Dk8
• Apache NiFi Anti-Patterns Part 2: Flow Layout by Mark: https://www.youtube.com/watch?v=v1CoQk730qs&feature=youtu.be
• Apache NiFi Anti-Patterns, Part 1: by Mark: https://www.youtube.com/watch?v=RjWstt7nRVY

Install

• https://www.cloudera.com/downloads/cdf/cfm-trial.html
• http://nifi.apache.org/download.html
• https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html

Don't Run NiFi Batch Jobs... But you can. Try NiFi Stateless, but you don't want to use Airflow try this

• https://benyaakobi.medium.com/nifi-scheduling-a522a1c9e740
• http://funnifi.blogspot.com/2018/08/one-and-done-flows-aka-job-processing.html
• https://getindata.com/blog/nifi-ingestion/fast-and-easy-what-could-possibly-go-wrong-one-year-history-certain-nifi-flow/
• https://boristyukin.com/how-to-run-sqoop-from-nifi/
• https://www.hadoopinrealworld.com/building-a-data-pipeline-with-apache-nifi/

Support

• https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/release-notes/content/supported-nifi-processors.html
• https://docs.cloudera.com/management-console/cloud/connection-to-private-subnets/topics/mc-endpoint_access_gateway.html

Cloudera Data Flow - Kubernetes Deployment

• https://docs.cloudera.com/dataflow/cloud/bp-flow-definition/topics/cdf-flow-isolation.html

DevOps / Tools / Scripts

• https://github.com/asdaraujo/data-in-motion
• https://github.com/asdaraujo/flink-kafka-replicator
• https://github.com/asdaraujo/kafka-client-examples

Load Balancing

• https://medium.com/@nlabadie/apache-nifi-syslog-load-balancing-c43d87480deb
• https://t.co/4ANiUPh706?amp=1
• https://community.cloudera.com/t5/Community-Articles/NiFi-Cluster-and-Load-Balancer/ta-p/248227
• https://community.cloudera.com/t5/Support-Questions/NIFI-S2S-communication-issue-between-two-secured-NIFI/td-p/193227

Syslog

• https://medium.com/@nlabadie/apache-nifi-netflow-to-syslog-117d46867ae1
• https://medium.com/@nlabadie/apache-nifi-sftp-csv-to-syslog-json-d9da6938defa
• https://medium.com/@nlabadie/apache-nifi-pulling-from-mysql-and-sending-to-syslog-181dd4ae969c
• https://nathanlabadie.com/apache-nifi-netflow-to-syslog/
• https://blog.davidvassallo.me/2019/04/09/consuming-netflow-using-nifi/

Grok

• http://grokconstructor.appspot.com/do/match#result
• https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/additionalDetails.html
• https://stackoverflow.com/questions/59291548/how-to-use-nifi-extractgrok-properly

Documentation

• https://nifi.apache.org/docs.html
• https://nifi.apache.org/docs/nifi-docs/html/overview.html
• https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
• https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
• https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html
• https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html
• https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
• https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
• https://nifi.apache.org/docs/nifi-docs/html/user-guide.html
• https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html
• https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

Demos / Examples

• https://www.cloudera.com/content/dam/www/marketing/resources/demos/demo-real-time-data-streaming-using-apache-kafka.landing.html
• https://developer.ibm.com/solutions/security/articles/distributed-real-time-threat-detection-engine/
• https://michalklempa.com/2016/10/using-nifi-to-ingest-snmp-into-avro/
• https://github.com/geoHeil/streaming-reference/tree/master
• https://blog.min.io/minio-events-with-apache-nifi/
• https://gist.github.com/neallee2012/3eb625094a46f8e3f0cc9028e04a5093
• https://medium.com/digital-transformation-and-platform-engineering/streaming-data-lake-and-analytics-with-apache-nifi-c651700b56db

Cybersecurity

• https://www.pwc.co.uk/issues/cyber-security-services/research/signal-att-and-ck-part-2.html

Reprise Demos

• https://www.getreprise.com/blog/using-apache-nifi-to-capture-covid-19-data/

Blogs

• http://lonnifi.blogspot.com/
• https://www.datainmotion.dev/
• https://blogs.apache.org/nifi/
• https://bryanbende.com/
• http://funnifi.blogspot.com/
• http://ijokarumawak.github.io/blog/
• https://pierrevillard.com/
• https://www.nifi.rocks/
• https://blog.cloudera.com/?s=nifi
• https://dzone.com/users/297029/bunkertor.html
• https://medium.com/@abdelkrim.hadjidj
• https://www.calculatedsystems.com/content
• https://mikethomsen.github.io/
• https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=nifi*
• https://mvnrepository.com/artifact/org.apache.nifi
• https://github.com/apache/nifi
• https://apachenifi.slack.com/join/shared_invite
• http://flink.apache.org/
• https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
• https://www.youtube.com/watch?v=pZq0EbfDBy4
• https://medium.com/apache-nifi-security
• https://lonnifi.blogspot.com/
• https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
• https://www.youtube.com/channel/UCBHGHVGIihzmuOM86D8IfHQ/featured
• http://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
• https://www.datacrafts.fr/NiFi-Democratizing-Record-Processors/
• https://www.datacrafts.fr/Error-Handling/
• https://github.com/hau-mal/articles/blob/master/nifi-ingest-weather-data-en.md
• https://medium.com/@ben2460/using-nifi-to-react-to-dns-changes-f1d3a348a595
• https://docs.cloudera.com/cdf-datahub/7.2.7/nifi-snowflake/topics/cfm-snowflake-egress.html
• https://medium.com/@kar9475/parameter-context-retry-flowfile-back-pressure-prediction-in-nifi-1-10-release-6c4aa7def082
• https://bryanbende.com/development/2021/02/17/apache-nifi-saml-keycloak
• https://community.cloudera.com/t5/Community-Articles/How-to-import-a-flow-to-NiFi-registry-in-CDP-Cloud/ta-p/308335
• http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/
• https://pierrevillard.com/2017/02/07/using-counters-in-apache-nifi/
• https://pierrevillard.com/2018/08/29/monitoring-driven-development-with-nifi-1-7/
• https://www.baeldung.com/iot-data-pipeline-mqtt-nifi
• https://www.clearpeaks.com/streaming-data-to-hive-using-nifi-and-kafka-in-a-kerberized-cdh-cluster/?utm_source=social&utm_medium=linkedin&utm_campaign=cdh-nifi-kafka-kerberos-blog-post
• https://github.com/medined/nifi-prototype
• https://boristyukin.com/building-near-real-time-big-data-lake-part-2/
• https://gpdb.docs.pivotal.io/connectors/apache-nifi/1-0/overview.html
• https://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
• https://community.cloudera.com/t5/Community-Articles/Using-NiFi-for-CDE-Jobs-Pipeline/ta-p/310122
• https://www.clearpeaks.com/connecting-nifi-with-activemq/
• https://medium.com/@Sushil_Kumar/data-quality-with-apache-nifi-9fa5223bbf54

FHIR / HL7 / Health / Medical

• https://bastide.org/2020/05/15/apache-nifi-and-ibm-fhir-server-invokehttp-and-ssl/

Containers / Docker / K8

• https://hub.docker.com/r/apache/nifi-registry
• https://hub.docker.com/r/apache/nifi/
• https://hub.docker.com/r/apache/nifi-toolkit
• https://hub.docker.com/r/apache/nifi-stateless

Examples

• https://lilneutron.medium.com/realtime-stock-trade-analysis-with-apache-nifi-kafka-and-flink-and-python-92b1c1bb9a54

NiFi Frameworks

• https://github.com/nagarajjayakumar/NFaaS

Sequencing and Ordering

Expression Language:

• random Description: Returns a random whole number ( 0 to 2^63 - 1) using an insecure random number generator.

• nextInt Description: Returns a one-up value (starting at 0) and increasing over the lifetime of the running instance of NiFi

• ${now():toNumber()}

• SELECT RPATH(message, '/c_ip') as xxx from flowfile

get recordpath for things like nested structured in json or xml

Other Options

• https://gist.github.com/ijokarumawak/7e6158460cfcb0b5911acefbb455edf0
• https://medium.com/google-cloud/gcp-pub-sub-ordering-and-apache-nifi-5e1ac9a14752
• http://funnifi.blogspot.com/2018/08/database-sequence-lookup-with.html
• https://community.cloudera.com/t5/Support-Questions/Generate-Sequence-number-in-Apache-NIFI/td-p/220571
• http://www.randomnumberapi.com/api/v1.0/random

Ingest and Process Logs

• https://bryanbende.com/development/2017/10/04/apache-nifi-processing-multiline-logs
• https://www.tigeranalytics.com/blog/real-time-nifi-alerts-automated-log-capturing-spark/
• https://nathanlabadie.com/nifi-pulling-data-from-splunk/
• https://www.youtube.com/watch?v=4yBc7hHvaQU
• https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
• https://dzone.com/articles/reading-sudo-logs-with-apache-nifi
• https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html
• https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/
• https://www.keedio.org/real-time-data-ingestion-processing-and-storage-of-syslog-using-apache-nifi-flink-kafka-and-cassandra/
• https://www.hadoopinrealworld.com/building-a-data-pipeline-with-apache-nifi/
• https://www.futurespace.es/en/apache-nifi-flujo-de-extraccion-validacion-transformacion-y-carga-de-ficheros-caso-de-uso-real/
• https://nxlog.co/documentation/nxlog-user-guide/apache-nifi.html
• http://bigapps.fr/apache-nifi-for-dataflow-and-real-time-streaming-with-apache-kafka/

Community Q&A

• https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=nifi
• https://apachenifi.slack.com/join/shared_invite
• https://community.cloudera.com/t5/Support-Questions/Nifi-CICD-with-mutiple-flows-having-controller-services/m-p/299771#M219835
• https://community.cloudera.com/t5/Support-Questions/How-to-add-custom-attributes-to-AttributesToJSON/td-p/308326
• https://community.cloudera.com/t5/Community-Articles/Using-NiFi-for-CDE-Jobs-Pipeline/ta-p/310122
• https://community.cloudera.com/t5/Support-Questions/NiFi-Load-balancing-internal-or-external/td-p/310131
• https://community.cloudera.com/t5/Support-Questions/Automatic-deployment-of-NiFi-Flow/m-p/311319#M224586
• https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631
• https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211
• https://community.cloudera.com/t5/Community-Articles/Simple-NiFi-template-for-flow-management-datahub/ta-p/311733
• https://community.cloudera.com/t5/Community-Articles/Connecting-Nifi-to-Impala-using-Simba-JDBC-driver/tac-p/312714#M6179
• https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418
• https://stackoverflow.com/questions/66259273/nifi-kerberos-ticket-login-not-supported-by-this-nifi
• https://community.cloudera.com/t5/Support-Questions/PrometheusReportingTask-with-same-port-and-multiple-nifi/m-p/316092#M226712
• https://github.com/chadgeary/nifi
• https://nathanlabadie.com/recombining-multiline-logs-with/
• https://nathanlabadie.com/video-tutorial-nifi-netflow-and-csv-ingest/
• https://nathanlabadie.com/nifi-pulling-data-from-splunk/
• https://nathanlabadie.com/running-nifi-in-a-docker-container/
• https://nathanlabadie.com/adding-a-timestamp-to-elasticsearch-from-apache-nifi/
• https://nathanlabadie.com/video-tutorial/
• https://nathanlabadie.com/the-basics-how/
• https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/
• https://blog.octo.com/en/accelerating-nifi-flows-delivery-part-1/
• https://www.fullstackanalyst.io/blog/open-source/processing-adobe-analytics-data-feeds-with-apache-nifi-for-adobe-experience-platform/

SAAS Interfaces

• https://github.com/jdye64/nifi-addons/tree/master/Processors/nifi-salesforce

Software Artefacts

• https://mvnrepository.com/artifact/org.apache.nifi
• https://github.com/apache/nifi
• https://hub.docker.com/r/apache/nifi/

Must Read Articles

• https://pierrevillard.com/best-of-nifi/
• http://lonnifi.blogspot.com/2020/01/rules-integration-with-apache-nifi.html
• https://www.calculatedsystems.com/post/what-is-nifi-the-core-concepts
• https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet

Conference Talks

• https://github.com/alopresto/slides
• http://slideshare.com/bunkertor
• https://github.com/tspannhw/dws2017sydney
• https://community.cloudera.com/t5/Community-Articles/Real-Time-Ingesting-and-Transforming-Sensor-and-Social-Data/ta-p/247202
• http://iotfusion.net/session/enterprise-iiot-edge-processing-with-apache-nifi-minifi-and-deep-learning/
• https://www.slideshare.net/bunkertor/deep-learning-on-hdp-2018-prague
• https://www.slideshare.net/bunkertor/minifi-and-apache-nifi-iot-in-berlin-germany-2018
• https://dataworkssummit.com/session/apache-deep-learning-101/
• https://github.com/tspannhw/StrataNYC2018
• https://feathercast.apache.org/2018/10/02/apache-deep-learning-101-timothy-spann
• https://www.slideshare.net/bunkertor/cdf-edge-to-ai-with-cem-april-9-2019-princeton-fod-meetup
• https://www.slideshare.net/bunkertor/the-edge-to-ai-deep-dive-barcelona-meetup-march-2019
• https://www.datainmotion.dev/2019/03/barcelona-dataworks-summit-march-2019.html
• https://dzone.com/articles/dataworks-summit-and-nosql-day-review-2019-washing
• https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html
• https://github.com/tspannhw/StrataNYC2019
• https://www.slideshare.net/Hadoop_Summit/eventdriven-messaging-and-actions-using-apache-flink-and-apache-nifi
• https://www.slideshare.net/BryanBende/apache-nifi-sdlc-improvements

Quick Tips

• https://community.cloudera.com/t5/Community-Articles/Connect-Nifi-to-Cloudera-Hive-with-Kerberos-and-TLS/ta-p/249476
• https://community.cloudera.com/t5/Community-Articles/Part-2-Nifi-flow-creation-to-parse-new-images-and-run-the/ta-p/244380
• https://docs.cloudera.com/cem/1.2.2/collecting-audit-events/topics/cem-setting-up-ConsumeWindowsEventLog.html
• https://community.cloudera.com/t5/Support-Questions/Nifi-Multiple-predicate-in-recordpath-filter/m-p/306604
• https://community.cloudera.com/t5/Support-Questions/Can-executesql-processor-store-state-for-incremental-loading/td-p/224975
• https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267
• https://www.nifi.rocks/record-path-cheat-sheet/
• https://www.nifi.rocks/expression-language-cheat-sheet/
• https://dev.to/tspannhw/one-minute-nifi-tip-calcite-sql-notes-561
• https://dev.to/tspannhw/new-and-improved-it-s-nifi-1k6o
• https://dev.to/tspannhw/quick-tip-nifi-json-cleanup-3o22
• https://dev.to/tspannhw/openssl-ssl-hosting-in-nifi-2k5f
• http://ijokarumawak.github.io/nifi/2017/01/27/nifi-s2s-local-to-aws/
• https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/building-a-dataflow/content/site-to-site.html
• https://www.calculatedsystems.com/post/the-nifi-expression-language
• https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet
• http://lonnifi.blogspot.com/2020/01/rules-integration-with-apache-nifi.html
• https://www.calculatedsystems.com/post/getting-started-with-apache-nifi-5-common-questions
• https://levelup.gitconnected.com/an-overview-of-apache-nifi-and-toolkit-cli-deployments-785978dbce3b
• Set to Integer SQL Type https://community.cloudera.com/t5/Support-Questions/Nifi-ExectueSQL-how-to-force-a-parameter-to-be-a-string/td-p/240117
• Make sure to use Cache Server https://community.cloudera.com/t5/Support-Questions/Could-not-connect-to-Distributed-Map-Cache-server-to-get/td-p/269340
• Static End-Point Security https://community.cloudera.com/t5/Support-Questions/Advanced-Update-Attribute-won-t-display-correctly-in-NiFi/td-p/306729
• Use -Record https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.QueryDatabaseTableRecord/index.html
• Merge Attributes https://community.cloudera.com/t5/Support-Questions/Nifi-Processors-status-merge-attributes-into-single-flowfile/td-p/308470
• https://community.cloudera.com/t5/Support-Questions/NIFI-quot-No-available-buckets-quot-for-saving-flow-version/td-p/308171
• https://medium.com/analytics-vidhya/managing-your-data-flows-with-apache-nifi-ef559be2fcf4
• https://springml.com/blog/apache-nifi-integration-on-google-cloud/
• https://ryan-templeton.medium.com/snowflake-resource-monitor-reports-delivered-to-slack-using-apache-nifi-8dfd4fc4d579
• https://pierrevillard.com/2016/03/13/get-data-from-dropbox-using-apache-nifi/

Understanding NiFi Internals / Architecture

• https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418
• https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-max-thread-pools-and-processor-concurrent/ta-p/248920
• https://community.cloudera.com/t5/Community-Articles/Leveraging-group-based-Ranger-authorization-in-NiFi/ta-p/248000
• https://community.cloudera.com/t5/Community-Articles/NiFi-Understanding-how-to-use-Process-Groups-and-Remote/ta-p/245486
• https://community.cloudera.com/t5/Community-Articles/How-to-address-JVM-OutOfMemory-errors-in-NiFi/ta-p/244431
• https://community.cloudera.com/t5/Community-Articles/Understanding-NiFi-processor-s-quot-Run-Duration-quot/ta-p/248921
• https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166
• https://community.cloudera.com/t5/Community-Articles/How-to-expand-existing-NiFi-cluster-fault-tolerance-using/ta-p/245082
• https://community.cloudera.com/t5/Community-Articles/How-to-determine-which-FlowFiles-are-associated-to-the-same/ta-p/249185
• https://community.cloudera.com/t5/Community-Articles/How-to-achieve-better-load-balancing-using-NiFi-s-Site-To/ta-p/246279

Cloud

• https://community.cloudera.com/t5/Community-Articles/Spark-Structured-Streaming-example-with-CDE/ta-p/302050
• https://community.cloudera.com/t5/Community-Articles/How-to-configure-NiFi-S2S-from-local-machine-to-CDP-Public/ta-p/301235
• https://community.cloudera.com/t5/Support-Questions/CDP-Public-Cloud-Data-Replication/m-p/309362
• https://community.cloudera.com/t5/Community-Articles/How-to-connect-Nifi-to-Kafka-in-Data-Hub-on-CDP/ta-p/294441
• https://community.cloudera.com/t5/Support-Questions/Can-GetHDFS-processor-catch-changes/td-p/201212
• https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-Selects-via-Apache/ta-p/308376
• https://community.cloudera.com/t5/Community-Articles/How-to-use-Sqoop-to-Hydrate-Cloudera-Data-Warehouse/ta-p/303037
• https://community.cloudera.com/t5/Community-Articles/Harnessing-the-Data-Lifecycle-for-Customer-Experience/ta-p/294910
• https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
• https://community.cloudera.com/t5/Community-Articles/Harnessing-the-hybrid-Cloud-use-CDP-Cloud-to-evaluate-COVID/ta-p/305220
• https://community.cloudera.com/t5/Community-Articles/How-to-write-data-from-NiFi-to-Kudu-in-CDP-Public-Cloud/ta-p/300402
• https://cloudacademy.com/blog/moving-data-to-s3-with-apache-nifi/
• https://community.cloudera.com/t5/Community-Articles/Simple-NiFi-template-for-flow-management-datahub/ta-p/311733
• https://www.cloudera.com/tutorials/cdp-importing-rdbms-data-into-hive.html
• https://community.cloudera.com/t5/Community-Articles/Working-with-S3-Compatible-Data-Stores-via-Apache-NiFi/ta-p/244584
• https://research.nccgroup.com/2021/01/04/using-aws-and-azure-for-cost-effective-log-ingestion-with-data-processing-pipelines-for-siems/

ETL / ELT / CDC / Load / Ingest

• https://datahovel.com/2019/11/14/big-data-and-stream-processing-101-part-4-how-to-do-a-simple-key-value-enrichment-in-apache-nifi/
• https://thirdeyedata.io/etl-pipeline-demonstration-using-apache-nifi/

Dockerization / K8

• https://www.nifi.rocks/official-nifi-docker-image/

Reporting Tasks

• https://www.nifi.rocks/developing-a-custom-apache-nifi-reporting-task/
• https://dev.to/tspannhw/sql-reporting-task-for-cloudera-flow-management-hdf-apache-nifi-260f

Error Handling

• https://community.cloudera.com/t5/Community-Articles/How-to-address-JVM-OutOfMemory-errors-in-NiFi/ta-p/244431

Templates and Examples

• https://github.com/xmlking/nifi-examples

Sizing and Performance

• https://www.datainmotion.dev/2020/07/sizing-your-apache-nifi-cluster-for.html
• https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/

Reporting and Monitoring

• https://www.datainmotion.dev/2020/07/report-on-this-apache-nifi-1114-monitor.html

Use Cases and Examples

• Retail Price Comparisons https://github.com/tspannhw/retail-dynamic-shelf-pricing/tree/main
• Sensors https://github.com/tspannhw/meetup-sensors
• Twitter to Cloud https://github.com/tspannhw/ClouderaNow2020
• Airline Sentiment https://github.com/tspannhw/airline-sentiment-streaming
• NiFi + Kafka Connect Pipeline HDFS https://github.com/tspannhw/NiFItoKafkaConnect
• Send Messages to Discord Channels https://github.com/tspannhw/NiFiToDiscord
• Capture Data to SOLR Search Indexes https://github.com/tspannhw/nifi-solr-example
• Post to Slack https://github.com/tspannhw/nifi-1.10-templates
• Hosting a Website in NiFi https://github.com/tspannhw/webdataingest
• Ingest/Export to Cloud Data Warehouse https://github.com/tspannhw/clouddatawarehouse
• Several use cases https://github.com/tspannhw/ApacheNiFiByExamples
• Hortonworks Gallery https://github.com/hortonworks-gallery/nifi-templates
• Apache Examples https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
• Cryptocurrency https://github.com/tspannhw/nifi-flows
• Energy Monitoring https://github.com/tspannhw/nifi-energy-monitoring
• Leprechaun Detector https://github.com/tspannhw/leprechaun-detector
• More Examples https://www.batchiq.com/nifi-flow-examples.html
• Email Extraction https://github.com/MikeThomsen/nifi-email-extraction-bundle
• Data Generation https://github.com/MikeThomsen/nifi-datageneration-bundle
• SQL Server and Kerberos https://www.exabeam.com/information-security/apache-nifi-sql-server-and-kerberos-authentication/
• Files and Netflow https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
• MySQL to SYSLOG https://www.exabeam.com/siem/apache-nifi-send-from-mysql-to-syslog/
• https://yauaa.basjes.nl/UDF-ApacheNifi.html

Sources & Sinks

• HBase/Phoenix https://www.datainmotion.dev/2020/07/phoenix-hbase-nifi-resources.html
• Operational Database https://www.cloudera.com/tutorials/cdp-cod-phoenix-sql-using-hbase-nifi-kafka.html
• REST https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html
• REST https://dev.to/tspannhw/cloudera-flow-management-101-let-s-build-a-simple-rest-ingest-to-cloud-datawarehouse-with-lowcode-powered-by-apache-nifi-2275
• Search https://dev.to/tspannhw/building-search-indexes-with-apache-nifi-streams-2ea7
• Twitter https://dev.to/tspannhw/harnessing-the-data-lifecycle-for-customer-experience-optimization-streaming-classifications-on-twitter-streams-2p0f
• Impala, Kudu, Kafka https://dev.to/tspannhw/read-apache-impala-apache-kudu-tables-and-send-to-apache-kafka-in-bulk-easily-with-apache-nifi-4n3g
• Google Analytics API https://dev.to/tspannhw/quicktip-ingesting-google-analytics-api-with-apache-nifi-mg1
• Image Upload / Slack https://dev.to/tspannhw/nifi-1-10-postslack-easy-image-upload-22mh
• Hive / ORC https://dev.to/tspannhw/using-grovepi-with-raspberry-pi-and-minifi-agents-for-data-ingest-to-parquet-kudu-orc-kafka-hive-and-impala-2h9k
• REST / Monitoring https://dev.to/tspannhw/monitoring-cloudera-edge-flow-manager-and-cloudera-streams-messaging-manager-with-apache-nifi-387k
• Generate JSON https://dev.to/tspannhw/generating-json-data-in-apache-nifi-4apo
• Kafka, REST for Stocks https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221
• MongoDB https://community.cloudera.com/t5/Community-Articles/Exporting-and-Importing-Data-from-MongoDB-in-the-Cloud-with/ta-p/249470
• PowerPoint https://community.cloudera.com/t5/Community-Articles/Converting-PowerPoint-Presentations-into-French-from-English/ta-p/248974
• Documents to HDFS https://community.cloudera.com/t5/Community-Articles/Scanning-Documents-into-Data-Lakes-via-Tesseract-MQTT-Python/ta-p/248492
• Flight Data https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940
• Street Cameras https://community.cloudera.com/t5/Community-Articles/Ingesting-and-Analyzing-Street-Camera-Data-from-Major-US/ta-p/249194
• Logs https://github.com/tspannhw/nifi-logs
• Beacons, BLE https://github.com/tspannhw/minifi-estimote
• Deep Speech https://github.com/tspannhw/nifi-deepspeech
• Syslog https://www.exabeam.com/siem/apache-nifi-ingest-syslog-data/
• IBM Event Streams https://www.ibm.com/cloud/blog/connecting-ibm-event-streams-to-apache-nifi
• Mark Logic JDBC https://www.marklogic.com/blog/apache-nifi-quickstart/
• Microsoft Teams https://aussie.nifi.ninja/blog/
• Office 365 https://github.com/pontus-vision/pontus-nifi-processors-office365

NiFi Stateless

• https://dev.to/tspannhw/exploring-apache-nifi-1-10-parameters-and-stateless-engine-44n4
• Example with Kafka https://github.com/tspannhw/stateless-examples
• https://blog.cloudera.com/paving-the-pathway-to-cloudera-data-platform-with-cloudera-dataflow/

Machine Learning, Deep Learning, AI

• https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with-djl-2d79
• https://dev.to/tspannhw/tracking-air-quality-with-apache-nifi-cloudera-data-science-workbench-pyspark-and-parquet-28c
• https://dev.to/tspannhw/updating-machine-learning-models-at-the-edge-with-apache-nifi-and-minifi-3ama
• https://community.cloudera.com/t5/Community-Articles/Using-Apache-NiFi-with-Apache-MXNet-GluonCV-for-YOLO-3-Deep/ta-p/248979?es_id=10e1da1507
• https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-Processor-for-Apache-MXNet-SSD-Single-Shot/ta-p/249240
• https://community.cloudera.com/t5/Community-Articles/Apache-Deep-Learning-101-Processing-Apache-MXNet-Model/ta-p/247944
• https://community.cloudera.com/t5/Community-Articles/Detecting-Language-with-Apache-NiFi/ta-p/248424
• https://community.cloudera.com/t5/Community-Articles/Steganography-with-Apache-NiFi/ta-p/244728
• Model Servers https://community.cloudera.com/t5/Community-Articles/Using-the-new-MXNet-Model-Server-with-Apache-NiFi/ta-p/247555
• https://github.com/tspannhw/OpenSourceComputerVision
• https://github.com/tspannhw/ApacheDeepLearning202
• https://github.com/tspannhw/DWS2019Barcelona
• https://github.com/tspannhw/ApacheDeepLearning101
• https://community.cloudera.com/t5/Community-Articles/Open-NLP-Example-Apache-NiFi-Processor/ta-p/249293

Best Practices

• https://dev.to/tspannhw/no-more-spaghetti-flows-2emd
• https://dev.to/tspannhw/apache-nifi-load-balancing-via-load-balanced-connections-593m
• https://www.datainmotion.dev/2019/06/performance-testing-apache-nifi-part-1.html
• http://lonnifi.blogspot.com/2019/11/back-pressure-prediction-deep-dive.html
• https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999

Tutorials

• https://www.cloudera.com/tutorials/analyze-transit-patterns-with-apache-nifi.html

How To

• Connect to Atlas https://dev.to/tspannhw/connecting-apache-nifi-to-apache-atlas-for-data-governance-at-scale-in-streaming-f5n
• sFTP https://community.cloudera.com/t5/Community-Articles/How-to-Retrieve-files-from-a-SFTP-server-using-NiFi-GetSFTP/ta-p/245430?search-action-id=37601510770&search-result-uid=245430

Integration Testing

• https://www.datainmotion.dev/2019/04/integration-testing-for-apache-nifi.html

IoT, Devices, MQTT, Sensors

• https://dev.to/tspannhw/predicting-sensor-readings-with-time-series-machine-learning-3ngc
• https://dev.to/tspannhw/edgeai-google-coral-with-coral-environmental-sensors-and-tpu-with-nifi-and-minifi-updated-efm-oh9
• https://dev.to/tspannhw/analyzing-wood-burning-stoves-with-flank-stack-minifi-flink-nifi-kafka-kudu-36on
• https://github.com/tspannhw/IoTFusion2018Talk

Administration,CLI, REST, Management, DevOps, CI/CD, Security

• https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
• https://dev.to/tspannhw/apache-nifi-and-nifi-registry-administration-3c92
• https://dev.to/tspannhw/using-nifi-cli-to-restore-nifi-flows-from-backups-18p9
• https://dev.to/tspannhw/nifi-toolkit-cli-for-nifi-1-10-213h
• https://dev.to/tspannhw/backup-and-restore-nifi-registry-templates-14m
• https://nipyapi.readthedocs.io/en/latest/
• https://community.cloudera.com/t5/Community-Articles/How-to-create-user-generated-keys-for-securing-NiFi/ta-p/245551
• https://github.com/tspannhw/BackupRegistry
• https://community.cloudera.com/t5/Community-Articles/DevOps-Tips-Using-the-Apache-NiFi-Toolkit-with-Apache-NiFI-1/ta-p/248376
• https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-Flow-Versioning-and-Automation/ta-p/247976
• https://dzone.com/articles/finding-nifi-errors
• https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060

Quick Starts

• https://github.com/apache/kudu/blob/master/examples/quickstart/nifi/README.adoc
• https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
• https://community.cloudera.com/t5/Community-Articles/Launching-a-NiFi-Docker-Instance/ta-p/248197
• https://hub.docker.com/r/apache/nifi/
• https://docs.cloudera.com/cdf-datahub/7.1.0/quick-start-fm/topics/cdf-datahub-fm-cluster-setup-quick-start-overview.html

Workshops & Crash Courses

• https://github.com/tspannhw/ClouderaFlowManagementWorkshop
• https://github.com/aperepel/nifi-workshop-2019
• https://www.youtube.com/watch?v=fblkgr1PJ0o
• https://github.com/abajwa-hw/masterclass
• https://github.com/simonellistonball/masterclass-hdf

FLaN(ifi)K

• https://dev.to/tspannhw/introducing-mm-flank-an-apache-flink-stack-for-rapid-streaming-development-from-edge-2-ai-5c12

Meetups

• https://www.meetup.com/futureofdata-princeton/
• https://www.meetup.com/futureofdata-philadelphia/

Videos

• https://www.youtube.com/watch?v=IUjz-rhA3xs
• https://www.youtube.com/user/HadoopSummit/videos
• https://www.youtube.com/channel/UCDIDMDfje6jAvNE8DGkJ3_w?view_as=subscriber
• https://www.youtube.com/user/clouderahadoop
• https://www.youtube.com/watch?v=XsL63ZQYmLE
• https://www.youtube.com/watch?v=xsC4cjSBv8k
• https://www.youtube.com/watch?v=TmdROpjVtM4
• https://www.youtube.com/watch?v=IUjz-rhA3xs
• https://www.youtube.com/watch?v=Tt8TSlHu7PE
• https://www.youtube.com/watch?v=QQks0KAvT58
• https://www.youtube.com/watch?v=QWXXaRE1VUI

Cloud

• https://dev.to/tspannhw/streaming-data-with-cloudera-data-flow-cdf-into-public-cloud-cdp-188b
• https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html
• https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-service-account.html
• https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-idbroker-mapping.html
• https://docs.cloudera.com/cdf-datahub/7.2.0/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-data-target.html
• https://www.calculatedsystems.com/post/moving-data-to-the-cloud
• https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c

Building Custom Processors

• https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Processor-Using-IntelliJ/ta-p/244343
• https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html
• https://github.com/alopresto/nifi-byop-dws
• https://github.com/DevWorxCo/nifi-starter
• https://github.com/influxdata/nifi-influxdb-bundle

Custom Processors

• https://github.com/All-of-Us-Research-Program/nifi-fhir-processor
• https://github.com/1904labs/nifi-standardize-date-bundle
• https://github.com/bbende/nifi-dependency-example
• https://github.com/tspannhw/ExecuteClouderaML
• https://github.com/minyk/nifi-headlessbrowser-processor
• https://github.com/tspannhw/nifi-djl-processor
• https://github.com/tspannhw/nifi-tensorflow-processor
• https://github.com/tspannhw/nifi-mxnetinference-processor
• https://github.com/tspannhw/nifi-extracttext-processor
• https://github.com/tspannhw/nifi-langdetect-processor
• https://github.com/tspannhw/nifi-attributecleaner-processor
• https://github.com/tspannhw/nifi-convertjsontoddl-processor
• https://github.com/tspannhw/nifi-postimage-processor
• https://github.com/tspannhw/GetWebCamera
• https://github.com/tspannhw/nifi-imageextractor-processor
• https://github.com/tspannhw/nifi-puttwitter-processor
• https://github.com/tspannhw/nifi-tensorflow-processor
• https://github.com/simonellistonball/nifi-OpenCV
• https://github.com/jdye64/nifi-openalpr
• https://github.com/AnkurJais/nifi-bigquery-custom-processor
• https://github.com/dDMarley/nifi-custom-processors
• https://github.com/myagmurs/bitmapprocessor
• https://github.com/linksmart/nifi-customlistentcp-processor
• https://github.com/andrewpalumbo/nifi-processors
• https://github.com/ligaroba/apache-nifi-smpp-processor
• https://github.com/shekhar-kotekar/nifi-word-count-custom-processor
• https://github.com/philenius/sum-json-array-of-numbers-custom-nifi-processor
• https://github.com/bongye/nifi-bitcoin
• https://github.com/ahmednaggar19/nifi-custom-processor-count-record-attribute
• https://github.com/PPcraft/iec104-bundle
• https://github.com/A-Hussien2011/ConvertJsonToHQL
• https://github.com/gudiseva/nifi-kafka-timestamp
• https://github.com/davidglevy/example-nifi-csv-cleaner
• https://github.com/kai5263499/nifi-binarygrpc
• https://github.com/adammichalsky/Hana-Custom-Processor
• https://github.com/cukuzz/nifi-custom-hdfs-processors-example
• https://github.com/jdye64/nifi-addons
• https://github.com/apsaltis/nifi-soap
• https://github.com/jdye64/nifi-gpiorest
• https://github.com/everwatchsolutions/aces-nifi-processors-bundle
• https://github.com/swiftops/nifi-infinispan-processor
• https://github.com/franziska-w/nifi-custom-processor
• https://github.com/wahid18benz/Apache-NiFi-Custom-Processor
• https://github.com/laurencedaluz/nifi-satori-bundle
• https://github.com/mataram/Nifi-NSQ-Consumer
• https://github.com/ORION6194/ApacheNifi-CustomProcessors
• https://github.com/bpenelli/nifi-bpenelli-bundle-1.8.0
• https://github.com/simonellistonball/PiWiNiFi
• https://github.com/tspannhw/nifi-corenlp-processor
• https://github.com/nagarajjayakumar/nifi-hana-custom-processor
• https://github.com/tspannhw/nifi-extracttext-processor
• https://github.com/tspannhw/nifi-nlp-processor
• https://github.com/dormstd/nifi-CKANProcessorBundle
• https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-Processor-Building-a-SQL-DDL-Schema-From-A-JSON/ta-p/247989
• https://github.com/iss-lab/nifi-stanfordcorenlp-processor
• https://github.com/tspannhw/nifi-puttwitter-processor
• https://github.com/compose-ex/nifi-compose-bundle
• https://github.com/BRGM/Nifi-WKB-to-WKT-processor
• https://github.com/nagarajjayakumar/nifi-hana-custom-processor
• https://github.com/dvas0004/nifi-loadbalancer
• https://github.com/dvas0004/nifi-webscraper

Migrate Flume Flows to NiFi

JMS, RDBMS, Kafka, Syslog, Twitter, Parquet, Kudu, Hive, HDFS, REST

• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-jms-to-x-and-x-to-jms-1g02
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-any-relational-database-to-from-anywhere-45d1
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-syslog-to-kafka-3a1k
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-31d1
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-1npb
• https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-55gj
• Templates https://github.com/tspannhw/flume-to-nifi

FDLC, SDLC, DevOps, Backup, Restore, Migration, CI/CD

• https://www.datainmotion.dev/2019/03/apache-nifi-operations-and-monitoring.html
• https://community.cloudera.com/t5/Community-Articles/How-to-import-a-flow-to-Nifi-registry-in-CDP-Cloud/ta-p/308335
• https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html
• https://www.datainmotion.dev/2020/10/automating-building-migration-backup.html
• https://www.datainmotion.dev/2020/08/deleting-schemas-from-cloudera-schema.html
• https://www.datainmotion.dev/2019/09/backup-and-restore-nifi-registry.html
• https://www.datainmotion.dev/2019/04/simple-apache-nifi-operations-dashboard.html
• https://www.youtube.com/watch?v=yKmVBTeZS4c
• https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-82e1ee866fab
• https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/
• https://community.cloudera.com/t5/Community-Articles/NiFi-SDLC-Automation-in-Python-with-NiPyApi-Part-1/ta-p/247693
• https://github.com/Chaffelson/nipyapi
• https://docs.cloudera.com/cfm/2.0.4/cfm-getting-started/topics/cfm-gs-overview.html
• https://leanjavaengineering.wordpress.com/2018/07/26/quick-tip-using-git-with-nifi-registry-in-docker/
• https://github.com/michalklempa/docker-nifi-registry
• https://github.com/steven-matison/NiFi-Templates
• https://github.com/tspannhw/BackupRegistry
• https://github.com/itarano/nifi-pre-pro-registry
• https://github.com/erdrix/nigoapi
• https://blog.cloudera.com/cloudera-flow-management-continuous-delivery-while-minimizing-downtime/
• https://github.com/alopresto/slides
• https://bryanbende.com/development/2017/05/10/apache-nifi-1-2-0-component-versioning
• https://www.infoq.com/presentations/secure-iot-apache-minifi/
• https://biconsulting.hu/letoltes/2020/2020budapestdata/D1_Daniel_Chaffelson_Cloudera.pdf

NiFi Notes

• https://www.javahotchocolate.com/notes/nifi.html
• https://martin.atlassian.net/wiki/spaces/lestermartin/pages/740196355/NiFi+Cheat+Sheet

Webinars

• https://www.brighttalk.com/webcast/18452/479025/demo-jam-live-create-data-flows-with-apache-nifi
• https://www.brighttalk.com/webcast/15417/354772

Integrations

• https://www.stardog.com/labs/blog/stardog-data-flow-automation-with-nifi/
• https://hackernoon.com/data-ingestion-using-apache-nifi-for-building-data-lake-using-twitter-data-1f3d3da6fe1d

CDP Public Cloud, AWS, Azure, GCP

• https://community.cloudera.com/t5/Community-Articles/How-to-use-NiFi-to-write-API-data-to-CDP-CDW/ta-p/306770
• https://www.datainmotion.dev/2020/04/streaming-data-with-cloudera-data-flow.html
• https://www.datainmotion.dev/2020/07/flank-in-cloud-huge-cloudera-data.html
• https://www.datainmotion.dev/2020/12/new-release-for-hdf-352-and-cloudera.html
• https://www.datainmotion.dev/2020/04/streaming-data-with-cloudera-data-flow.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kafka-ingest/topics/cdf-datahub-fm-kafka-ingest-overview.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hive-ingest/topics/cdf-datahub-nifi-hive-ingest.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-hbase-ingest/topics/cdf-datahub-nifi-hbase-ingest.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-kudu-ingest/topics/cdf-datahub-nifi-kudu-ingest.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html
• https://docs.cloudera.com/cfm/2.0.4/site-to-site/topics/cdf-datahub-site-to-site.html
• https://docs.cloudera.com/cdf-datahub/7.2.2/nifi-aws-ingest/topics/cdf-datahub-fm-s3-ingest-overview.html
• https://docs.cloudera.com/cdf-datahub/7.2.6/planning-your-flow-management-deployment/topics/cdf-datahub-fm-cluster-layout.html
• https://medium.datadriveninvestor.com/how-to-build-a-simple-data-flow-with-apache-nifi-1ef3b25fe2c4

MemSQL

• https://www.youtube.com/watch?v=Jnz-a_wCYp4
• https://medium.com/@moha.ajori/using-apache-nifi-to-load-tweets-from-twitter-api-to-memsql-19e19a3be20e

GreenPlum DB

• https://www.cdata.com/kb/tech/greenplum-jdbc-apache-nifi.rst
• https://github.com/DanielePalaia/gpss-nifi

Teradata

• https://www.cdata.com/kb/tech/teradata-jdbc-apache-nifi.rst
• https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783

Microsoft SQL Server (MS SQL)

• https://community.cloudera.com/t5/Support-Questions/Nifi-Connection-to-MSSQL-server-DB/td-p/95642
• https://www.cdata.com/kb/tech/sql-jdbc-apache-nifi.rst
• https://nathanlabadie.com/apache-nifi-ms-sql-and-kerberos-authentication/
• https://gist.github.com/ijokarumawak/42c257afb5e80361e502564085d7999e
• https://medium.com/@danielyahn/nifi-ms-sql-integrated-security-9efc184872db
• https://rapidminernotes.blogspot.com/2018/01/populating-sql-server-from-apache-nifi.html

Oracle

• https://www.cdata.com/kb/tech/oracledb-jdbc-apache-nifi.rst
• https://www.progress.com/blogs/tutorial-access-data-via-jdbc-with-apache-nifi
• https://www.youtube.com/watch?v=XsL63ZQYmLE
• https://www.youtube.com/watch?v=qsIJFcQYp2E
• John Kuchmek at ApacheCon 2020 Incrementally Streaming from Oracle -> https://www.youtube.com/watch?v=nPewLy7Zjkg&list=PLlmpIaH7URGXYTdiHhlYbSkAO_B_RDDn0&index=28

S3 Data Indexing

• https://blog.cloudera.com/how-to-index-data-from-s3-via-nifi-using-cdp-data-hubs/

Cloud

• https://www.batchiq.com/nifi-process-s3-event-notifications.html
• https://gist.github.com/jvwing
• https://www.batchiq.com/process-cloudtrail-events-with-nifi.html
• https://www.batchiq.com/s3-ingest-with-nifi.html
• https://www.batchiq.com/database-extract-with-nifi.html
• https://www.batchiq.com/database-ingest-with-nifi.html

CDC / Database

• https://community.cloudera.com/t5/Community-Articles/Incremental-Fetch-in-NiFi-with-QueryDatabaseTable/ta-p/247073
• https://community.cloudera.com/t5/Community-Articles/Change-Data-Capture-CDC-with-Apache-NiFi-Part-1-of-3/ta-p/246623
• https://github.com/edufer/nifi-cdc-sqlserver
• https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4
• https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.13.2/org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL/
• https://www.linkedin.com/pulse/achieving-incremental-fetch-change-data-capture-via-apache-rajpal/
• https://community.cloudera.com/t5/Community-Articles/PutHive3Streaming-to-do-Basic-CDC-with-NiFi/ta-p/249187
• https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-Selects-via-Apache/ta-p/308376
• https://community.cloudera.com/t5/Community-Articles/Basic-CDC-on-SAP-HANA-using-Apache-NIFI/ta-p/249323
• https://www.qlik.com/us/data-management/nifi
• https://www.marklogic.com/blog/apache-nifi-quickstart/
• https://www.vertica.com/kb/NifiCG/Content/Partner/NifiCG.htm
• https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Your-Hadoop-DataLake/ta-p/247927
• https://www.batchiq.com/nifi-flow-examples.html
• https://blog.pythian.com/database-migration-using-apache-nifi/
• https://www.datainmotion.dev/2019/04/oracle-golden-gate-to-apache-kafka-to.html
• https://github.com/anair-it/nifi-docker
• https://medium.com/trendyol-tech/using-apache-nifi-for-migrating-data-fe1c224c7b16

Provenance Checking

Quick Tips: check nifi security policy on provenance, check configuration for repo-type, check for available disk space, unsynced time, restart all servers. Configuration: nifi.provenance.repository.max.storage.time

• https://javasupportzone.blogspot.com/2019/07/possible-reasons-for-processors-data.html
• https://community.cloudera.com/t5/Support-Questions/NiFi-Provenance-Not-Available-in-UI/td-p/179999#:~:text=Re%3A%20NiFi%20Provenance%20Not%20Available%20in%20UI&text=Try%20stopping%20Nifi%20and%20purging,read%2Fwrite%20in%20set%20directory.
• https://stackoverflow.com/questions/41487509/data-provenance-disabled-nifi

Videos

• https://www.youtube.com/watch?v=RHFkCtbe3kc

Misc

• https://www.cloudera.com/content/dam/www/marketing/resources/solution-briefs/data-in-motion-philosophy.pdf.landing.html

Other Languages

• https://apuntandoanulo.com/category/apache-nifi/

Legacy / Older Editions

• https://github.com/jfrazee/awesome-nifi
• https://dzone.com/articles/apache-nifi-10-cheatsheet
• https://dzone.com/articles/monitoring-for-apache-nifi-17
• https://www.cdata.com/kb/tech/salesforce-jdbc-apache-nifi.rst
• https://github.com/UKHomeOffice/cdp-nifi-processors
• https://github.com/dstreev/nifi-processor-examples