This guide demonstrates how your Quarkus application can utilize SmallRye Reactive Messaging to interact with Apache Kafka.
Prerequisites
To complete this guide, you need:
-
less than 15 minutes
-
an IDE
-
JDK 11+ installed with
JAVA_HOMEconfigured appropriately -
Apache Maven 3.8.1
-
Docker and docker-compose to run the application in containers
Architecture
In this guide, we are going to develop two applications communicating with Kafka. The first application sends a quote request to Kafka and consumes Kafka messages from the quote topic. The second application receives the quote request and sends a quote back.
The first application, the producer, will let the user request some quotes over a HTTP endpoint.
For each quote request a random identifier is generated and returned to the user, to put the quote request on pending.
At the same time the generated request id is sent over a Kafka topic quote-requests.
The second application, the processor, in turn, will read from the quote-requests topic put a random price to the quote, and send it to a Kafka topic named quotes.
Lastly, the producer will read the quotes and send them to the browser using server-sent events. The user will therefore see the quote price updated from pending to the received price in real-time.
Solution
We recommend that you follow the instructions in the next sections and create applications step by step. However, you can go right to the completed example.
Clone the Git repository: git clone https://github.com/quarkusio/quarkus-quickstarts.git, or download an archive.
The solution is located in the kafka-quickstart directory.
Creating the Maven Project
First, we need to create two projects.
To create the producer project, open code.quarkus.io. The link already configures the application to select the two extensions we need to the producer:
-
The Kafka connector for reactive messaging
-
RESTEasy Reactive and its Jackson support (to handle JSON) to serve the HTTP endpoint.
Click on the "Generate your application" button and download the zip. Once downloaded, unzip the file on your file system.
Now, let’s create the processor project. Open code.quarkus.io. This link only select a single extension, the Kafka connector. Click on the "Generate your application" button and download the zip. Once downloaded, unzip the file on your file system alongside the producer project.
At that point, you should have the following structure:
.
├── processor
│ ├── README.md
│ │ ├── pom.xml
│ │ ├── src
│ │ │ ├── main
│ │ │ │ ├── docker
│ │ │ │ │ └── ...
│ │ │ │ ├── java
│ │ │ │ └── resources
│ │ │ │ └── application.properties
│ │ │ └── test
│ ├── mvnw
│ ├── mvnw.cmd
│ ├── pom.xml
└── producer
├── README.md
├── mvnw
├── mvnw.cmd
├── pom.xml
└── src
└── main
├── docker
│ └── ...
├── java
└── resources
└── application.properties
|
Dev Services
No need to start a Kafka broker when using the dev mode or for tests. Quarkus starts a broker for you automatically. See Dev Services for Kafka for details. |
The Quote object
The Quote class will be used in both producer and processor projects.
For the sake of simplicity we will duplicate the class.
In both projects, create the src/main/java/org/acme/kafka/model/Quote.java file, with the following content:
package org.acme.kafka.model;
public class Quote {
public String id;
public int price;
/**
* Default constructor required for Jackson serializer
*/
public Quote() { }
public Quote(String id, int price) {
this.id = id;
this.price = price;
}
@Override
public String toString() {
return "Quote{" +
"id='" + id + '\'' +
", price=" + price +
'}';
}
}
JSON representation of Quote objects will be used in messages sent to the Kafka topic and also in the server-sent events sent to web browsers.
Quarkus has built-in capabilities to deal with JSON Kafka messages. In a following section we will create serializer/deserializer classes for Jackson.
Sending quote request
Inside the producer project create the src/main/java/org/acme/kafka/producer/QuotesResource.java file, and add the following content:
package org.acme.kafka.producer;
import java.util.UUID;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.acme.kafka.model.Quote;
import org.eclipse.microprofile.reactive.messaging.Channel;
import org.eclipse.microprofile.reactive.messaging.Emitter;
import io.smallrye.mutiny.Multi;
@Path("/quotes")
public class QuotesResource {
@Channel("quote-requests") Emitter<String> quoteRequestEmitter; (1)
/**
* Endpoint to generate a new quote request id and send it to "quote-requests" Kafka topic using the emitter.
*/
@POST
@Path("/request")
@Produces(MediaType.TEXT_PLAIN)
public String createRequest() {
UUID uuid = UUID.randomUUID();
quoteRequestEmitter.send(uuid.toString()); (2)
return uuid.toString();
}
}
| 1 | Inject a Reactive Messaging Emitter to send messages to the quote-requests channel. |
| 2 | On a post request, generate a random UUID and send it to the Kafka topic using the emitter. |
We need to instruct Quarkus to map the quote-requests channel to a Kafka topic.
In the src/main/resources/application.properties of the producer project, add:
# Configure the outgoing Kafka topic quote-requests
mp.messaging.outgoing.quote-requests.connector=smallrye-kafka
All we need to specify is the smallrye-kafka connector.
By default, the Kafka connector uses the channel name (quote-requests) as the Kafka topic name.
It configures the serializer automatically.
Processing quote requests
Now let’s consume the quote request and give out a price.
Inside the processor project, create the src/main/java/org/acme/kafka/processor/QuoteProcessor.java file and add the following content:
package org.acme.kafka.processor;
import java.util.Random;
import javax.enterprise.context.ApplicationScoped;
import org.acme.kafka.model.Quote;
import org.eclipse.microprofile.reactive.messaging.Incoming;
import org.eclipse.microprofile.reactive.messaging.Outgoing;
import io.smallrye.reactive.messaging.annotations.Blocking;
/**
* A bean consuming data from the "request" Kafka topic and giving out a random quote.
* The result is pushed to the "quotes" Kafka topic.
*/
@ApplicationScoped
public class QuotesProcessor {
private Random random = new Random();
@Incoming("requests") (1)
@Outgoing("quotes") (2)
@Blocking (3)
public Quote process(String quoteRequest) throws InterruptedException {
// simulate some hard working task
Thread.sleep(200);
return new Quote(quoteRequest, random.nextInt(100));
}
}
| 1 | Indicates that the method consumes the items from the requests channel |
| 2 | Indicates that the objects returned by the method are sent to the quotes channel |
| 3 | Indicates that the processing is blocking and cannot be run on the caller thread. |
For every Kafka record from the quote-requests topic, it calls the process method, and sends the returned Quote object to the quotes channel.
As with the previous example we need to configure the connectors in the application.properties file, to map the requests and quotes channels to Kafka topics:
%dev.quarkus.http.port=8081
# Configure the incoming Kafka topic `quote-requests`
mp.messaging.incoming.requests.connector=smallrye-kafka
mp.messaging.incoming.requests.topic=quote-requests
mp.messaging.incoming.requests.auto.offset.reset=earliest
# Configure the outgoing Kafka topic `quotes`
mp.messaging.outgoing.quotes.connector=smallrye-kafka
mp.messaging.outgoing.quotes.value.serializer=io.quarkus.kafka.client.serialization.ObjectMapperSerializer
Note that in this case we have one incoming and one outgoing connector configuration, each one distinctly named. The configuration keys are structured as follows:
mp.messaging.[outgoing|incoming].{channel-name}.property=value
The channel-name segment must match the value set in the @Incoming and @Outgoing annotation:
-
quote-requests→ Kafka topic from which we read the quote requests -
quotes→ Kafka topic in which we write the quotes
|
More details about this configuration is available on the Producer configuration and Consumer configuration section from the Kafka documentation. These properties are configured with the prefix |
Also, for the outgoing configuration we specified the serializer because we are sending a Quote object as the message payload.
Quarkus provides default implementations for Kafka serializer/deserializer pairs using Jackson ObjectMapper.
ObjectMapperSerializer can be used to serialize all objects via Jackson.
Receiving quotes
Back to our producer project.
Let’s modify the QuotesResource to consume quotes, bind it to an HTTP endpoint to send events to clients:
import io.smallrye.mutiny.Multi;
[...]
@Channel("quotes") Multi<Quote> quotes; (1)
/**
* Endpoint retrieving the "quotes" Kafka topic and sending the items to a server sent event.
*/
@GET
@Produces(MediaType.SERVER_SENT_EVENTS) (2)
public Multi<Quote> stream() {
return quotes; (3)
}
| 1 | Injects the quotes channel using the @Channel qualifier |
| 2 | Indicates that the content is sent using Server Sent Events |
| 3 | Returns the stream (Reactive Stream) |
Again we need to configure the incoming quotes channel inside producer project.
Add the following inside application.properties file:
# Configure the outgoing Kafka topic quote-requests
mp.messaging.outgoing.quote-requests.connector=smallrye-kafka
# Configure the incoming Kafka topic quotes
mp.messaging.incoming.quotes.connector=smallrye-kafka
|
In this guide we explore Smallrye Reactive Messaging framework to interact with Apache Kafka. Quarkus extension for Kafka also allows using Kafka clients directly. |
JSON serialization via Jackson
Finally, we will configure JSON serialization for messages using Jackson.
Previously we’ve seen the usage of ObjectMapperSerializer to serialize objects via Jackson.
For the corresponding deserializer class, we need to create a subclass of ObjectMapperDeserializer.
So, let’s create it inside producer project on src/main/java/org/acme/kafka/model/QuoteDeserializer.java
package org.acme.kafka.model;
import io.quarkus.kafka.client.serialization.ObjectMapperDeserializer;
public class QuoteDeserializer extends ObjectMapperDeserializer<Quote> {
public QuoteDeserializer() {
// pass the class to the parent.
super(Quote.class);
}
}
No need to add any configuration for this inside application.properties file.
Quarkus will automatically detect this serializer.
|
Message serialization in Kafka
In this example we used Jackson to serialize/deserialize Kafka messages. For more options on message serialization see Using Apache Kafka with Reactive Messaging - Serialization. We strongly suggest adopting a contract-first approach using a schema registry. To learn more about how to use Apache Kafka with the schema registry and Avro follow the Using Apache Kafka with Schema Registry and Avro guide. |
The HTML page
Final touch, the HTML page reading the converted prices using SSE.
Create inside the producer project src/main/resources/META-INF/resources/quotes.html file, with the following content:
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Quotes</title>
<link rel="stylesheet" type="text/css"
href="https://cdnjs.cloudflare.com/ajax/libs/patternfly/3.24.0/css/patternfly.min.css">
<link rel="stylesheet" type="text/css"
href="https://cdnjs.cloudflare.com/ajax/libs/patternfly/3.24.0/css/patternfly-additions.min.css">
</head>
<body>
<div class="container">
<div class="card">
<div class="card-body">
<h2 class="card-title">Quotes</h2>
<button class="btn btn-info" id="request-quote">Request Quote</button>
<div class="quotes"></div>
</div>
</div>
</div>
</body>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<script>
$("#request-quote").click((event) => {
fetch("/quotes/request", {method: "POST"})
.then(res => res.text())
.then(qid => {
var row = $(`<h4 class='col-md-12' id='${qid}'>Quote # <i>${qid}</i> | <strong>Pending</strong></h4>`);
$(".quotes").append(row);
});
});
var source = new EventSource("/quotes");
source.onmessage = (event) => {
var json = JSON.parse(event.data);
$(`#${json.id}`).html(function(index, html) {
return html.replace("Pending", `\$\xA0${json.price}`);
});
};
</script>
</html>
Nothing spectacular here. On each received quote, it updates the page.
Get it running
You just need to run both applications. In a first terminal, run:
> mvn -f producer quarkus:dev
In a separate terminal, run:
> mvn -f processor quarkus:dev
Quarkus starts a Kafka broker automatically, configures the application and shares the Kafka broker instance between different applications. See Dev Services for Kafka for more details.
Open http://localhost:8080/quotes.html in your browser and request some quotes by clicking the button.
Running in JVM or Native mode
When not running in dev or test mode, you will need to start your Kafka broker.
You can follow the instructions from the Apache Kafka website or create a docker-compose.yaml file with the following content:
version: '2'
services:
zookeeper:
image: quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
command: [
"sh", "-c",
"bin/zookeeper-server-start.sh config/zookeeper.properties"
]
ports:
- "2181:2181"
environment:
LOG_DIR: /tmp/logs
networks:
- kafkaquickstart-network
kafka:
image: quay.io/strimzi/kafka:0.23.0-kafka-2.8.0
command: [
"sh", "-c",
"bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
]
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
LOG_DIR: "/tmp/logs"
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
networks:
- kafkaquickstart-network
producer:
image: quarkus-quickstarts/kafka-quickstart-producer:1.0-${QUARKUS_MODE:-jvm}
build:
context: producer
dockerfile: src/main/docker/Dockerfile.${QUARKUS_MODE:-jvm}
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
ports:
- "8080:8080"
networks:
- kafkaquickstart-network
processor:
image: quarkus-quickstarts/kafka-quickstart-processor:1.0-${QUARKUS_MODE:-jvm}
build:
context: processor
dockerfile: src/main/docker/Dockerfile.${QUARKUS_MODE:-jvm}
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
networks:
- kafkaquickstart-network
networks:
kafkaquickstart-network:
name: kafkaquickstart
Make sure you first build both applications in JVM mode with:
> mvn -f producer package
> mvn -f processor package
Once packaged, run docker compose up.
| This is a development cluster, do not use in production. |
You can also build and run our applications as native executables. First, compile both applications as native:
> mvn -f producer package -Pnative -Dquarkus.native.container-build=true
> mvn -f processor package -Pnative -Dquarkus.native.container-build=true
Run the system with:
> export QUARKUS_MODE=native
> docker compose up
Going further
This guide has shown how you can interact with Kafka using Quarkus. It utilizes SmallRye Reactive Messaging to build data streaming applications.
For the exhaustive list of features and configuration options check the Reference guide for Apache Kafka Extension.