Reactive programming is often explained in overly theoretical terms. According to the Reactive Manifesto, all Responsive, Resilient, Elastic and Message Driven systems are reactive. Responsiveness has to do with response time and throughput. Resiliency has to do with fault tolerance and availability. Elasticity implies the ability to support different loads at different times, which may impact performance, availability and operational expenses. And being Message-Driven allows the system to behave according to messages passing from one component to another, setting clear boundaries between components while keeping interactions among them transparent. 

A note on streams: streams are an abstraction over unbounded data. They are often compared to batches. Imagine we have two integers. Applying a common function to such integers, such as the sum operator (+) sounds feasible. Now, what happens if the operands wont stop coming because, for example, they come from some user input, which is using some application in real-time. Then, the sum becomes a challenge. Using streams leverages a different approach to this problem. We will dive deeper on composing streams later on.

Event streams are particularly powerful in programming languages that support functional programming to a certain degree. Multi-paradigm languages, like C# or Scala have some built-in abstractions to represent data streams. In this article, we will explore one particular implementation of reactive streams using modern Java and Project Reactor. This will serve us as a service layer in a web application. In future articles we will discuss how this service layer could interoperate with a WebFlux web access layer. Follow along!

What you will need

In order to follow this guide make sure you have:
Basic knowledge of Java applications and functional programming
Experience developing some kind of asynchronous algorithms
Some development environment, such as Visual Studio Code or IntelliJ IDEA
Java 17 (which can be installed directly on IntelliJ)

What you can expect

After completing this guide you should be able to:
Identify basic use cases for reactive programming
Explain how to set up a basic Reactor application
Feel comfortable reading basic algorithms written in a reactive style
Recommend articles and blog posts written by experts to sustain decisions on architecture and design similar to the ones used in the example.
In this guide we will use Java, Gradle, Spring-Boot and IntelliJ CE.

The Application 
We will create a user-interaction processing microservice PoC. We will focus on the service layer and stub the web access layer (which we will implement on WebFlux in another article) and the data access layer.  

Creating the application

In order to get started create a Spring basic application with Gradle. You can do it using Spring Initializr, manually setting it up or cloning our starter project on GitHub. Maven works as well, but we prefer Gradle’s succinct syntax instead of XML. 

Let us fetch our dependencies from Mvn Repository. Let us briefly present the principal dependencies used in our project:

• Spring-boot-starter is a dependency that groups up many other dependencies to get started with Spring-boot based applications. It’s just a convenient bundle of features to speed up our project set up. We won’t make use of spring-boot features just yet, but we will when we integrate our service layer with WebFlux.
• Lombok is one of our favorite Java projects. It allows us to write concise and elegant Java code. It is powerful enough to help us get rid of most Java boilerplate code.
• Project Reactor is a reactive programming library. It excels on non-blocking asynchronous data processing in a high-level syntax while interacting with Java low-level asynchronous APIs.

Our build.gradle dependencies block looks like this:
dependencies { implementation ‘org.springframework.boot:spring-boot-starter’ testImplementation ‘org.springframework.boot:spring-boot-starter-test’ compileOnly ‘org.projectlombok:lombok:1.18.28’ annotationProcessor ‘org.projectlombok:lombok:1.18.28’ testCompileOnly ‘org.projectlombok:lombok:1.18.28’ testAnnotationProcessor ‘org.projectlombok:lombok:1.18.28’ implementation ‘io.projectreactor:reactor-core:3.5.8’}
First, let’s grasp some of Lombok’s features. One of our favorite features is the built-in builder pattern and the Data classes. With such, we can write a simple model class in a file called DomElement.java inside the models package with this code:@Data@Builder(setterPrefix = “with”)public class DomElement { private String name; private String type;}
The @Data lombok annotation allows us to spare constructors, accessors, modifiers, equality comparator and toString methods. Lombok’s data feature behaves similarly to Java Records, although they are not exactly the same. @Data classes can be fine tuned, and customization can be found in Lombok’s documentation. For example, fields marked as transient won’t be taken into consideration for equality comparators or hash codes.

The @Builder annotation enables us to avoid creating numerous overloaded constructors or cluttering our classes with excessive setters. Instead, it allows us to use the Builder pattern and chain-call our setters with some syntactic sugar on top. It ends up building an elegant Fluent Interface. In Baeldung there is an article worthy taking a look on combining the Builder pattern with Fluent Interfaces.

Because of the scope we have to Lombok in our Gradle dependencies, we can access the methods provided by the Builder during development time. Take a look at the setterPrefix option we added to our builder.

Instead of calling traditional void setters, we can use its corresponding with[attribute] method instead, and chain them. Builder setters are not void so they can be composable and be chain called. To obtain the object being built, the method chain call must end with .build()

Also add a UserInteraction model. The purpose of this class is to represent some user interaction with a DOM element. We will model this as a wrapper class for a DomElement field and a userId field, which is a string. Create a UserInteraction.java file in the models package and make it look like this:
@Data@Builder(setterPrefix = “with”)public class UserInteraction { private DomElement domElement; private UUID userId; private String interaction; private long unixTimestamp;}
Now we can get started with the service itself. We will stub out the source of user interactions (in this context, the controller that feeds them to the service layer), hard-coding some Java Beans. We will hold the user interactions in some convenient data structure. Later on, we will wire the structure into our application so we could easily replace it with an API controller that may be listening to some front end component.  

It would be interesting to filter user interactions. For example, listing only user interactions that occurred between two timestamps, or filtering out according to DomElement type. Let’s design this behavior by contract in our service.
public interface UserInteractionService {List<UserInteraction> filterByDomElementType(String domElementType);List<UserInteraction> getUserMostRecentInteraction(UUID userUUID);}
This is almost the way to go. We are missing the reactive aspect of things. In order to make this contract reactive, we are going to make use of one of the key concepts in Java Reactor: the Flux. 

In the Project Reactor Flux documentation we can find the following illustration, clearly explaining how Flux works.

Flux and Mono are the main Reactive Streams abstractions used by Reactor. Mono is similar to Flux, but it accepts at most one element. Most Flux and Mono operators accept one or more of its kind, and also return one such. Look at the following example:
Flux<String> uppercaseCities = Flux.fromArray(new String[]{“boston”, “new york”, “seattle”}).map(city -> city.toUpperCase());
This code snippet presents a way to create a Flux from an array and then transforms all the elements initially present in the string array to uppercase, producing a new Flux. Fluxes can be created from Reactor native sources and from most Java iterables. Note that most Flux operations can be chained together in a fluent fashion, just as we defined our Builder earlier on. 

The reactive contract for our UserInteractionService class will look like this:
public interface UserInteractionService {Flux<UserInteraction> filterByDomElementType(String domElementType);Mono<UserInteraction> getUserMostRecentInteraction(UUID userUUID);}
Let’s build an implementation for this interface. We will build a simple implementation with stub data, and we will refactor it as we go to keep it tidy up, leverage Spring capabilities and foster maintainability. Start by adding some test data. We will help you with that!
private List<UserInteraction> supplyData(){DomElement button = DomElement.builder() .withName(“register-button”) .withType(“button”) .build();DomElement contactUsLink = DomElement.builder() .withName(“contact-us-link”) .withType(“href”) .build();UserInteraction contactUsLinkHover = UserInteraction.builder() .withDomElement(contactUsLink) .withUserId(UUID.randomUUID()) .withInteraction(“hover”) .withUnixTimestamp(System.currentTimeMillis() / 1000L) .build();UserInteraction contactUsLinkClick = UserInteraction.builder() .withDomElement(contactUsLink) .withUserId(UUID.randomUUID()) .withInteraction(“click”) .withUnixTimestamp((System.currentTimeMillis() + 3000) / 1000L) .build();UserInteraction registerButtonClick = UserInteraction.builder() .withDomElement(button) .withUserId(UUID.randomUUID()) .withInteraction(“click”) .withUnixTimestamp(System.currentTimeMillis() – 50000 / 1000L) .build();return List.of(contactUsLinkHover, contactUsLinkClick, registerButtonClick);}
These are the stub user interactions our system will work with in this PoC. Refactor that by wrapping all that hard-coded data in a private method, so our public methods aren’t flooded with stub data initialization.
@Overridepublic Mono<UserInteraction> getUserMostRecentInteraction(UUID userUUID) { Flux<UserInteraction> userInteractions = Flux.fromIterable(this.supplyData()); userInteractions.subscribe(System.out::println); return userInteractions .filter(ui -> ui.getUserId().equals(userUUID)) .reduce((ui1, ui2) -> ui1.getUnixTimestamp() > ui2.getUnixTimestamp() ? ui1 : ui2);}
There is a lot going on there. Let’s break it down a bit.

1. First, we are creating a Flux out of our user interactions out of the List of stubs we hard-coded. We can create Fluxes out of any Iterable or array in Java.
2. In the first method, we are just applying a filter operation according to the dom element type of the interaction. We will only keep those that match with the method parameter.
3. In the second method we are first calling a subscribe method. The purpose of this subscribe method is to print the flux elements one by one. The subscribe method allows to apply side effects and also triggers the flux operations. Fluxes are lazy: nothing happens until we subscribe. After that we are invoking another filter operator followed by a reduce operator, which will end up returning a single element: the most recent user interaction for the parameter user id.
4. The :: is called method reference operator. It allows passing a function as an argument to other functions that require such an argument.

The Flux operations pipeline is called to be lazy. Fluxes and Monos are lazy. No computation will trigger until we subscribe. The client of the pipeline is responsible for triggering the computation, and is free to choose when to do it by subscribing. It’s really interesting how we can think of two stages: assembly and execution. We can chain together many complex (and remember: asynchronous and nonblocking!) operations without worrying about stages: the entire pipeline will trigger at once.

There are many pros when choosing lazy structures. For example, if our filter operations were conditional, that is to say there are certain situations where they won’t be executed, then our code won’t waste any resources by defining the flux assembly. Also, we can materialize our query whenever we want. 

Before refactoring our application, let’s test it out. Add the following code to your main method. Don’t worry about the SpringApplication line for now.
@SpringBootApplicationpublic class ContentRecommenderApplication {
public static void main(String[] args) { UserInteractionService service = new UserInteractionServiceImpl(); service.filterByDomElementType(“button”) .subscribe(ui -> System.out.println(ui.toString())); SpringApplication.run(UserInteractionApplication.class, args); }}
And run it.

Refactoring

Our PoC works, but it’s far from being acceptable. Bearing in mind that we plan to scale this service PoC up to an actual reactive microservice on top of WebFlux we can tidy up things a bit. Also, we can leverage Spring Framework’s capabilities such as automated dependency injection. 

We want to autowire dependencies so that when we add some WebFlux controllers they can easily locate the service they will communicate with. Also, we can take advantage of the interface-implementation split to enforce compliance with the Dependency Inversion Principle. Note that, for the sake of simplicity our interfaces and implementations will reside within the same physical component. They could be split up into two JARs to foster maintainability and modularity. 

Furthermore, we want to clear up our hard-coded data initialization from our service. We will autowire data initialization in a data access package, so that it mocks database access behavior. 

There is one detail we often overlook when designing software and that is good package/module design. We often design in the direction of classes and objects, but we tend to forget about modules. We will break packages up so that we comply with the Stable Dependency Principle (depending on the direction of stability) and the Stable Abstraction Principle (a package should be abstract as it is stable). In our case, we have completely abstract and completely volatile packages, and our volatile packages depend on the abstract ones. Also, we have defined interfaces for business rules, so it makes sense to assume those are our stable packages. 

Our packages will look like the following:

And we will define three beans in a @Configuration class named ServiceConfiguration:
@Beanpublic List<UserInteraction> userInteractions(){ DomElement button = DomElement.builder() .withName(“register-button”) .withType(“button”) .build(); DomElement contactUsLink = DomElement.builder() .withName(“contact-us-link”) .withType(“href”) .build(); UserInteraction contactUsLinkHover = UserInteraction.builder() .withDomElement(contactUsLink) .withUserId(UUID.randomUUID()) .withInteraction(“hover”) .withUnixTimestamp(System.currentTimeMillis() / 1000L) .build(); UserInteraction contactUsLinkClick = UserInteraction.builder() .withDomElement(contactUsLink) .withUserId(UUID.randomUUID()) .withInteraction(“click”) .withUnixTimestamp((System.currentTimeMillis() + 3000) / 1000L) .build(); UserInteraction registerButtonClick = UserInteraction.builder() .withDomElement(button) .withUserId(UUID.randomUUID()) .withInteraction(“click”) .withUnixTimestamp(System.currentTimeMillis() – 50000 / 1000L) .build(); return List.of(contactUsLinkHover, contactUsLinkClick, registerButtonClick);}

@Beanpublic UserInteractionRepository userInteractionRepository(){ return new UserInteractionRepositoryImpl();}
@Beanpublic UserInteractionService userInteractionService(){ return new UserInteractionServiceImpl();}

That’s our supplyData method turned into a Bean plus the instantiation of our Service and Repository interfaces. Now our Service class looks clean and is not responsible for creating stub data anymore. Our last step is to set up our main method. This should not be necessary if we already had our WebFlux controllers, but for the time being, make it look like the following:

public static void main(String[] args) {SpringApplication.run(UserInteractionApplication.class, args);ApplicationContext ctx = new AnnotationConfigApplicationContext(ServiceConfiguration.class);UserInteractionService userInteractionService = ctx.getBean(UserInteractionService.class);userInteractionService.filterByDomElementType(“button”) .map(UserInteraction::toString) .subscribe(System.out::println);
}

And that is how we can make our service get all the user interactions for buttons.

Check our full example here.

Next steps

We have developed a service layer for a reactive microservice. The most interesting future step is adding WebFlux for our application to be exposed to the web. Also, some high-throughput non-blocking persistence mechanism would be welcome too!

Stay ahead of the curve on the latest trends and insights in big data, machine learning and artificial intelligence. Don’t miss out and subscribe to our newsletter!

In the previous article we focused on getting started with Apache Flink. In order to run our app we do also need some Cassandra instance running. We will set up a Cassandra local cluster with Docker and learn how to query the messages Flink sent. We will also show our pipeline in action.

Remember the pipeline looks like the following:

Setting up a local Cassandra server with Docker

Before getting our Flink app running, we should have a Cassandra table to store our values. Flink offers many connectors! You can check them out here. In this example we chose to show the Apache Cassandra in action.

There are many ways to host Cassandra: a local server, a local containerized server, managed on AWS Keyspaces and so on. One of the simplest ways is to get it running on a Docker container. If you do not have Docker, find out how to get it in the official documentation.  

Open a new terminal tab and run:

docker run -p 9042:9042 –rm –name cassandra -d cassandra:3.11

Your Cassandra instance is now running on port 9042.

The easiest way to communicate with your Cassandra server is through cqlsh. CQL stands for Cassandra Query Language (remember Cassandra is not SQL). SH Stands for shell. You can download this tool here.

Once it is installed, the following commands will connect to your Cassandra server, create a Keyspace and a table with a single column: payload.

USE cassandraSink;

You can get all the records living in the table by querying:

Building and submitting the jar

Now that our code is set up, it is time for the sbt assembly plugin to shine. We will now run the Scala project with the Apache Flink dependency we built in the previous article. Open up a console in your Scala project root folder (where we placed our build.sbt file). Run the command sbt assembly. 

Congrats! Your jar file should be on its way. You should be able to find it soon on the /target/scala-2.12 folder. 

Open up a new terminal tab on your Flink source directory. To get our Flink app running:

And that is it. Your Flink app should be now running.

Demo

We will now post a value to our SpringBoot API. Assuming the API is now running locally:

Let us query our Kafka topic. Open up a terminal tab in your Kafka source directory.

We can see:

Now, let us query our Cassandra table:

And that is it. Our real time big data processing pipeline is working as expected. Ideally, this pipeline should be hosted in the cloud. This task will be covered in a future article.

Wrapping up

The Apache Software Foundation is working really hard on providing us with amazing open-source big data tools. Apache Kafka is really easy to get started with and has an interesting learning curve. One of the most outstanding things about it is that it provides many connectors for both pulling and pushing records.

Apache Flink provides a customizable environment to develop big data processing applications with many connectors as well. Connectors for Java and Scala using the latest DataSource API are widely available. Also, Flink becomes a really powerful alternative for unbounded stream processing. 

This pipeline could be used to feed data to many other applications. Once the data is stored in Apache Cassandra, it could be used by any other application to make decisions (for example a machine learning model).

It is important to consider how robust this pipeline is. Apache Kafka, Apache Flink and Apache Cassandra are built upon distributed architectures. They are resilient and, thanks to Flink, offer an exactly once processing policy, which guarantees both efficiency and consistency. 

All these three technologies can easily migrate from on premise setups to fully cloud managed setups. We can say that scalability is another interesting aspect of this pipeline. 

Following this last idea, we consider that showing a cloud setup for this pipeline could be an interesting challenge, and could be a great idea for upcoming articles. Stay tuned to get to know more about top-notch practices and technology to make the most out of your data!

Missed any of our previous articles on this pipeline? Check them here:

• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: the project
• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: Apache Kafka topic
• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: Apache Flink app with Scala

In the previous article we focused on getting started with Apache Kafka. We will now focus on Apache Flink: the core technology in our pipeline. We will set up a Flink cluster and learn how to build and submit an app. We will also quickly see some Scala and Scala tools needed to build this app appropriately.

Remember the pipeline looks like the following:

A little Flink architecture

Similarly to what happened with Kafka, Flink architecture should be explained in detail. Given that the purpose of this guide is to show an easy and quick setup for getting started with Flink, we will only mention the Job Manager and the Task Managers. 

The Job Manager is the process responsible for coordinating the execution of applications. In particular, distributed applications. It schedules jobs, mediates resource access, reacts to critical events (such as node failure) and so on.

The Task Managers are responsible for actual operations on data. They have slots, which are hosts for actual processing. 

There is a lot more to Flink. We do not expect more than an intuitive understanding of its architecture. Flink has some great documentation on its architecture, and it is an advised reading! Check it out here.

Setting up a local Flink Cluster

It is worth mentioning that Flink 1.15.0 runs on Scala 2.12. We can configure the version our application uses with SBT, in case we have installed a different Scala version. 

Before we get hands on our Flink job we have to get a cluster up and running. The very first task is installing Flink. Similar to Kafka, Flink can be downloaded from the Apache Flink downloads site. 

The Flink source folder has two subfolders worth pointing out: bin and conf. Both contain the same information they had for Kafka. There is one more folder worth mentioning: log. When the Flink Cluster launches, this folder populates. Logs are written there when the cluster is launched or terminated, as well as when jobs are executed.

In order to launch the Flink Cluster, open a new terminal tab at the Flink source folder and run:

The cluster should be up and running and ready to receive jobs to run. We can check it is running by accessing the Flink web UI running on localhost:8081.

It is time to create a simple Flink application to run as a job. The idea is to consume the Kafka records we fed our Kafka topic. Finally, we will write the value of such records in a Cassandra table.

Building a Scala application with Flink

In order to easily get started with Scala, we recommend installing SBT. SBT is the preferred package manager in the Scala community. SBT can be easily installed through Homebrew on Mac, Chocolatey on Windows or rpm on Linux. Find out more details here. 

 SBT offers some template projects for getting started with Scala. In order to create the most simple, open a terminal tab on the directory you would like to create the project. Then run:

You will be prompted to name your project. After that, you are ready to go! Go to your project root directory (the one that has a build.sbt file) and try running:

sbt run 

Now the project is being compiled and executed. You should soon see:

Who said getting started with Scala was hard?

SBT Configuration

Before moving to sbt specific syntax, there is one detail we have to take into consideration. Flink allows us to submit .JAR files. Most of the time, we need to add dependencies that are not native to Flink (for instance, connectors). This information must live within the Jar. A Jar that carries all of its dependencies is known as a fat jar or uber jar. In order to build a fat jar, we need a specific plugin: sbt-assembly plugin. Sbt-assembly works like Maven dependency shading.

According to the sbt plugin documentation a valid approach to install plugins within a project is creating a plugins.sbt file. This file should be located in a /project directory in the project root directory. The only plugin we need for now is the assembly plugin. Therefore, our plugins.sbt should look like this:

addSbtPlugin(“com.eed3si9n” % “sbt-assembly” % “1.2.0”);

As mentioned before, Flink requires Scala 2.12.x to run. All project-level configuration for Scala will be made on the build.sbt file.

Each entry in this file is called a setting expression. Each setting expression consists of a key, an operator and a value. The most common setting expression is the dependency expression. Let us see how they look:

This line adds the Flink base library to our project. Verify it does match the definition we just built: libraryDependencies is a key. += is an operator and “org.apache.flink” % “flink-core” % “1.15.0” is a value. These last three strings are Maven artifacts groupId, artifactId and version.

With this brief introduction, let us see the code we need in our build.sbt file in order to get started:

This code is pretty straightforward. First, we specify the Scala version. Remember we need Scala 2.12.x in order to run Flink. The second chunk is jar file metadata. The third chunk are the dependencies we need. The fourth chunk resolves conflicts introduced by the assembly plugin. 

More information on sbt can be found in the official documentation.

The Scala app

The following code snippet is the most basic code (with virtually no actual data processing) to sink events from a Kafka topic to a Cassandra table. In this context, sinking data is another (rather fancy) word for pushing data from one system to another.

This code should be in the /src/main/scala directory. We called the file Main.scala, but you could choose any different name as long as the file and the name of the Scala object are the same. Make sure your Scala object extends the App trait. 

There is a lot going on here. Let us break it a little down.

• Main is a Scala object. Objects are similar to Java classes, but they do only have one instance. This is the Scala native mechanism to implement the Singleton pattern. 
• For this endeavor a Scala trait works just like a Java interface.
• In Scala 2, extending the App trait means the same as implementing the main method in Java.
• This example assumes that there is a local Cassandra server running on localhost port 9042. We will cover a really simple way to achieve this later on.
• The Kafka source topic is called flink-input.
• The Kafka events value is a simple string object. It is possible to deserialize different input formats, but for the sake of simplicity we chose to send a string.

Troubleshooting

Debugging Flink applications can be tricky. The following is a personal tip for unix-like OS users. To quickly check the most recent Flink job logs, change directory to the Flink source directory and run:

This will output the most recent logs on your TaskExecutor.

Despite the Flink app being complete, the app itself is not enough to be tested. We still need some kind of Cassandra instance running to sink values to. The following entry in this series of articles will tackle this. 

Before moving on, can you think of a similar implementation of this pipeline using some other technology for data streams processing? What advantages and disadvantages would it offer? We will leave one idea for you to get started:

• Apache Spark Streaming 

Source code: The Flink app can be found within the source code repository.

Missed any of our previous articles on this pipeline? Check them here:

• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: the project
• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: Apache Kafka topic

In the previous article we presented the project we are learning to build. We will now focus on Apache Kafka. We will set up a Kafka cluster and learn how to create a topic. We will discuss a couple ways to populate this Kafka topic.

Remember the pipeline looks like the following:

A little Kafka architecture

Explaining the Kafka architecture can become complex and overly theoretical. In this section we will briefly discuss as little Kafka as we need to build this pipeline.

Kafka is an event streaming framework. Events and messages are two words for the same thing in this context. Events are sent to topics. Topics are nothing but a simple ordered sequence of events. Topics can be partitioned and replicated.

Kafka servers, or brokers, are responsible for topics log. Kafka brokers are coordinated thanks to the Zookeeper.  The Zookeeper basically manages servers and topics. We say we have a Kafka cluster when we launch brokers coordinated by the Zookeeper.

Producers are actors that write on topics. Consumers usually read from them. This (simplified) setup would look something like this:

And that is it. That will be just enough Kafka to get started. Bear in mind that this explanation simplified the Kafka architecture a lot. There are lots of resources on this interesting architecture. One of our favorite blog entries to learn about Kafka is this one.

Setting up a local Kafka Cluster

The very first thing that has to be done is installing Scala and Kafka. In order to install Scala, check the official Scala download site. We want to install Scala 2.13. Installing Kafka is as simple as downloading the compressed file available in the Apache Kafka download site. For this endeavor we want the version supported by Scala 2.13. Also, Scala needs Java 1.8 or higher to run. Make sure to get an appropriate JDK.

Once we have extracted the compressed Kafka files, there are two folders that are worth inspecting. The first one is bin. In this folder there are a bunch of .sh (shell script) files. These files will be used to run our Zookeeper and server, as well as managing and querying topics. There are many scripts we will not use in this project.

The second folder we should pay attention to is config. In this folder we can find many files with the .properties extension. These files will be fed as arguments to our shell scripts execution. For example, we can set our Zookeeper (zookeeper.properties) and Kafka server (server.properties) host and port there.

The first thing to get Kafka running is to launch the Zookeeper. Head to your favorite terminal and change directory to your Kafka source directory (the one we downloaded moments ago). Get the Zookeeper up and running executing the following command:

We should see a lot of information on screen. This is how it should behave if the Zookeeper could successfully launch!

The following step to take is getting up a broker node, this is to say, a server. To do this, execute the following command in a new terminal tab or window:

We are now ready to create a Kafka topic and bind it to the server we have just created. Since our previous tab is running our Kafka broker, we have to get up a new one. 

We have called the topic ‘flink-input’, replicated it only one time and bound it to the server we created. Note that the replication factor can not exceed the number of brokers (in this case we only have one broker). 

Populating and debugging the topic

In our local set up we experimented with a Spring Boot based REST API. Building a RESTful API escapes from this article’s scope. In order to get started with Kafka we recommend getting used to the scripts Kafka offers. Let us now see how to populate and query a Kafka topic using a terminal.

If you cd to your Apache Kafka root folder, you can run 

./bin/kafka-topics.sh –bootstrap-server=localhost:9092 –list

In order to see your Kafka topics listed. It would be interesting to write some messages to our topic, right? Try running:

The console will now enter in interactive mode and every line you type will be stored as an event in your Kafka topic.

Try querying you topic running the following command:

We can see:

Our Kafka cluster is now up and running! We already know how to use Kafka’s basic features. Easy enough, right?

In the next entry of this series of articles we will move on with Flink.  Before moving on, can you think of a similar implementation using any other technology that supports data streaming? What advantages and disadvantages would it have compared to the Apache Kafka implementation? We will leave some ideas for you to get started:

• Apache Pulsar
• RabbitMQ
• AWS Kinesis

Missed any of our previous articles on this pipeline? Check them here:

• First steps towards building a real-time big data processing pipeline with Apache Kafka and Apache Flink with Scala: the project

Every business can take advantage of data. An important challenge for organizations is to design powerful systems that can transform data into strategic decisions as fast as possible. The truth is, large amounts of data can become overwhelming.

In this article we will introduce two modern technologies that may serve as allies when it comes to processing large volumes of data in real time: Apache Flink and Apache Kafka.

Through a series of articles we will build a pipeline that work as the foundation for many interesting applications, for example:

• Social media analytics system that reacts to user input, such as clicks or keyboard events.
• IoT applications that read metrics from sensors and feed them to actuators.
• Bank transactions in order to prevent potentially fraudulent actions.

And many more!

Why Apache Flink?

Apache Spark is a tried and true open source framework that allows near real-time data processing. Despite Spark being a popular choice among big data developers, Apache Flink is becoming one of the top technologies for a particular niche: real-time processing. 

Unlike Flink, Apache Spark does not offer native stream processing. Spark uses micro-batch processing, which can simulate real-time, but its latency is slightly higher than Flink’s. Also, Flink’s job optimization is done automatically. There is no need to keep an eye on shuffles, broadcasts, and so on.

Why Apache Kafka?

Many software implementations serve as message brokers. Each one has its own perks and use cases. Some offer easy integration while others may offer, for example, robustness.

One of the most popular message brokers is RabbitMQ, which may serve as a reliable source of message streaming in many use cases. When it comes to big data applications, some properties must be guaranteed, and that is not exactly the case of RabbitMQ. Some of these properties are partition tolerance, high throughput, horizontal scalability and strong redundancy. These make Kafka an ideal candidate for big data real-time processing.  

Why Scala?

Scala is a very fast growing language which mixes the best of the object oriented programming paradigm with the best of the functional programming languages. It is incredibly versatile, succinct and considerably performant. Scala compiles into Java bytecode, which is finally executed by the JVM.

Spark made Scala one of the top programming languages for big data applications. Take into consideration that Scala is the most performing language for Spark. This fact made the big data ecosystem pay attention to Scala not only as an elegant language, but also as a performant one. As a plus, both Flink and Kafka were written in Scala (as well as in Java).  

The project

We will briefly discuss how to build a short real-time processing pipeline with 2 main modules. The following diagram depicts the main components and how they should interact.

The first component is a Spring Boot REST API. This API exposes an endpoint which accepts HTTP Post requests. The body of these requests can be any object serialization (JSON, ProtoBuf, XML and so on). In this example, the endpoint will expect a JSON object which contains the Kafka topic to publish the message to, the Kafka message key and the Kafka message value.

The second element involved in the pipeline is a Kafka topic. This topic will act as a message exchange. The Kafka setup for this example will be on local standalone mode to keep it simple. 

The third element is a Flink job, which will be run in a local cluster on standalone mode. The Flink job is responsible for subscribing to the Kafka topic as a consumer. It will process the messages and push it to a Cassandra table running on a Docker container.

Finally, as mentioned above, we have a Cassandra table within a keyspace. Cassandra is a NoSQL distributed database. It is an ideal candidate for applications that need a manageable replication factor for the data.

We will cover the setup for Kafka, Flink and Cassandra. The Kafka topic producer can be any application compatible with a Kafka source, or the Kafka console producer script itself.

This setup can be fairly easily hosted in the cloud but in this article we chose to run everything locally so as to better portray what is going on behind the scenes. We built a similar pipeline with a managed Cassandra keyspace hosted on AWS Keyspaces. Since Flink does not allow consistency level customization for Scala tuples, the cloud version of this pipeline used the Cassandra Flink connector for Java to sink POJOs. 

In upcoming articles we will build the components involved in the pipeline. For the sake of simplicity, this example will cover an on-premise setup, starting with Kafka. 

Stay tuned for more articles about top-notch technology and practices related to making the most out of your data!