Dynamic rule processing for data streams using Flink & Serverless on AWS

The write up below requires some working knowledge of Apache Flink along with basic understanding of AWS. Apache Flink is a Complex Event Processing (CEP) framework primarily used for working with continuous data streams at hyperscale offering tools to work with a wide variety of use cases.

If you’re new to Flink and do not have the vaguest idea, I’d recommend reading to the Flink application including the docker-compose file. To make it easy, the docker file already includes the Kafka topics creation utilized within the Flink job.

These are steps you would need to follow,

• Build the JAR file.


• Install docker.


• Run the docker command — docker-compose up -d
  


• Go to http://localhost:8081 and submit a job with the JAR file created in step 1.


• Connect to the Kafka container — docker container exec -it <cluster-name> /bin/bash
  


• Push messages to the appropriate Kafka topic to monitor the flow — kafka-console-producer — broker-list localhost:9092 — topic <name>
  

Conclusion

The above solution works great when there are a finite number of rules, although there is no direct restriction on the amount of data Flink can hold in the internal state. It’s the performance and code management that needs to be taken into consideration.

There could be an argument about using a Lambda function instead of Flink to achieve the same result. While that makes complete sense in terms of simplicity, when real-time processing at a heavy scale is paramount, Flink is the better choice.

It is also possible to use two different sources and then subject the combined stream to the Broadcasted rules. Most complex scenarios are likely possible with Flink when compared to other forms of compute for data streams. It all boils down to what works best in the long run and where application management is simpler.

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen