Recent Posts

Wednesday, 21 November 2018

Apache Kafka - Use cases and Applications

Apache Kafka Use cases
     Apache Kafka can be used for various purposes in an organization, such as
1. Messaging service
     Millions of messages can be sent and received in real-time. In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
 
2. Real-time stream processing
     Kafka can be used to process a continuous stream of information in real-time and pass data to stream processing systems such as Storm.

3. Log aggregation
     Kafka can be used to collect physical log files from multiple systems and store it in a central location such as HDFS.

4. Commit log service
     Kafka can be used as an external commit log for distributed systems. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka, helps in supporting this use-case.
 
5. Event sourcing
     A time ordered sequence of events can be maintained through Kafka.

6. Kafka Metrics
     For operational monitoring data, Kafka is often used.
This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.

7. Website Activity Tracking
     Website activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting.
Apache Kafka Applications
     Kafka supports many of today’s best industrial applications.
1. Twitter 
     Twitter  is one of the best Kafka Applications. A famous online social networking service or a platform Twitter uses Kafka. Basically, it provides a way to send and receive users tweets. Through this platform, registered users can read and post tweets, but unregistered users can only read tweets. However, it uses Storm-Kafka as a part of their stream processing infrastructure. 
2. LinkedIn
     Another Kafka Application is LinkedIn. For activity stream data and operational metrics, LinkedIn uses Apache Kafka. There are several products like LinkedIn Newsfeed, LinkedIn Today, for online message consumption and in addition to offline analytics systems like Hadoop, Kafka messaging system helps LinkedIn. Moreover, we can say the strong durability of Kafka is also one of the key factors in connection with LinkedIn. 
3. Netflix
     An American multinational provider of on-demand internet streaming media, Netflix, also uses Kafka. Basically, for the purpose of real-time monitoring and event processing, it uses Kafka. 
4. Mozilla
     In 1998, members of Netscape created a free-software community, Mozilla. In order to collect performance and usage data from the end-users browser for projects like Telemetry, Test Pilot, etc. Kafka will soon be replacing a part of Mozilla current production system. 
5. Oracle
     Oracle provides native connectivity to Kafka from its Enterprise Service Bus product called OSB (Oracle Service Bus) which allows developers to leverage OSB built-in mediation capabilities to implement staged data pipelines. 

Next Tutorial : Apache Kafka APIs 

Previous Tutorial : Apache Kafka - Pros and Cons 
 

2 comments: