Overview
In this Spotfire® Streaming HowTo, we will use the Spotfire® Streaming Input Adapter for Apache Kafka Consumer to consume message from a Microsoft Azure EventHub. This is a step-by-step guide and assumes no prior knowledge of Azure EventHubs or Apache Kafka.
Prerequisites
- Spotfire® Streaming 10.4.x or later installed
- A Microsoft online account (usually created via live.com)
- A Microsoft Azure subscription (the free account is sufficient to perform the examples in this document)
- Internet access to Microsoft Azure, especially to port 9093 on your chosen subdomain of servicebus.windows.net
Directions
Complete the instructions that follow.
Part 1: Create Azure Event Hubs Namespace and Event Hubs
1. Create a Kafka-enabled EventHubs Namespace
Tip: Microsoft has published documentation for this process here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create-kafka-enabled. You may wish to check this page in case Microsoft has updated this page or the related functionality since this HOWTO document was created in September 2019.
- Use a web browser to navigate to the Azure Portal (portal.azure.com)
- Sign in to your Azure account (details out of scope of this HowTo
- In the upper-left hand corner of the Portal screen, click on Create a resource
- Enter Event Hubs in the Search marketplace... text box and press the Enter key
- On the resulting Event Hubs screen, click the Create button
-
On the Create Namespace screen, enter the following values:
- Name: <a unique name just for you>
- Pricing tier: Standard
- Enable Kafka: check the checkbox
- Subscription: Free trial
- Resource group: <a unique resources group name, just for you>
- Location: <pick something close to you>
- Enable Auto-Inflate: leave this option unchecked
- Press the Create button (which is below the option entry form, in case you have to scroll):
- It may take a while for the name space to be created and deployed. You can monitor the progress of this operations in the notifications area in the upper right-hand corner of the Azure Portal screen:
- When the namespace is available, a Notification to that effect will appear:
-
For the purposes of this HowTo, secure your namespace to only accept connections from your client IP address:
-
On your Event Hub's Namespace page, click on Settings > Firewalls and virtual networks
- Allow access: Click the Selected networks radio button
- Check the Firewall > Add your client IP address checkbox
- Click the Save button up near the top of the screen
-
On your Event Hub's Namespace page, click on Settings > Firewalls and virtual networks
Tip: If you keep this namespace for any future uses and use it from other IP addresses, you will have to manage the allowed addresses from this screen.
2. Create an Event Hub within the Namespace
- In the left-hand column of the Azure Portal screen, click on Dashboard
- Under My Dashboard > All resources, locate the newly created Event Hubs Namespace
- Click on the name of the Namespace to bring up the Namespaces's detail screen
- Click on + Event Hub to create an Event Hub in the Namespace
-
On the Create Event Hub screen:
- Name: topic1
- Partition Count: 1 (the default)
- Message Retention: 1 (the default)
- Capture: Off (the default)
- Press the Create button
- Verify that topic1 is created and has the options you selected by clicking on Entities > Event Hubs on the Event Hubs Namespace screen
Part 2: Configure the EventFlow modules for Azure Event Hubs
1. Start StreamBase Studio and navigate to the SB Authoring perspective
-
Open StreamBase Studio
- Double-click on the StreamBase Studio 10.4 icon on the Windows Desktop
- Accept the default value for your StreamBase Studio workspace location and name:
- Click Launch
- You are presented with the Welcome screen for StreamBase Studio
- In the upper right corner of the Welcome screen, click Workbench.
- This action takes you to the StreamBase Studio Demos (SB Demo) perspective.
- In the upper right corner, click Go To Authoring:
- This action results in a display of the SB Authoring perspective:
Note: In StreamBase Studio, a perspective is a collection of views that support a set of related tasks.
2. Import the Azure Events Hubs Using Kafka example project into your workspace.
-
From the top menu, choose File > Import...
- Choose General > Existing Projects into Workspace, and click Next
- In the next screen, choose Select archive file
-
Click Browse... and choose .../StreamingExampleKafkaAzureEventHubs1044.zip (attached to this page)
- - Click Finish
- Click Cancel to skip the installation of an SVN Connector, if asked
3. Copy the Connection String to the clipboard
- In the Azure Portal, from the Event Hubs Namespace screen for your namespace, click on Settings > Shared Access Policies
- Click on RootManagedSharedAccessKey
- Copy the Connection String Primary Key
4. Configure the Broker and Connection String into the EventFlow modules
- In the StreamBase Studio Project Explorer view, open example_kafka_azureeventhubs > src/main/configurations > sbengine.conf
- Paste your connection string from the clipboard to replace YOUR_CONNECTION_STRING_GOES_HERE, as the value of the EventHubsConnectionString operator parameter.
- Copy the fully qualified domain name of your Event Hubs Namespace to replace YOUR_BROKER_FQDN_GOES_HERE, as the partial value of the KafkaBrokers operator parameter
- The resulting sbengine.conf file will look something like this, using your namespace's name and shared access key:
- Save your sbengine.conf file (Studio Menu Bar > File > Save All)
Part 3: Run the Kafka Consumer and Producer Programs
1. Run the Consumer program
- In the StreamBase Studio Project Explorer view, navigate to example_kafka_azureeventhubs > src/main/eventflow > com.tibco.streaming.example.kafka_azureeventhubs
- Select consumer.sbapp and right-click on it to bring up the context menu
- Select Run As > EventFlow Fragment
- The fragment is running and ready to receive client input when the message This application is currently running at sb://yourhost... appears on the application editor canvas:
2. Subscribe to the topic1 topic
-
In the Manual Input view:
- Fragment: ensure that it is consumer
-
Stream: ensure that it is ConsumerCommand
- command: subscribe
- topic: topic1
- leave the rest of the input fields set to null (the default)
- Click Send Data
-
In the Console view, a successful subscribe command results in a number of INFO log messages. In particular, look for these:
- Successfully logged in
- Subscription to topic 'topic1' successful
Tip: It's possible to see success messages here, but due to network connectivity configurations still not be able to connect to the Kafka broker for the Event Hubs Namespace. At this point the subscribe command appears successful. If there are connectivity issues connecting to the broker, connection error messages will appear after a minute or so as the Consumer adapter retries the connection periodically.
3. Run the Producer program
- In the upper right corner of the Studio window, click on the SB Authoring perspective button
- In the StreamBase Studio Project Explorer view, select producer.sbapp and right-click on it to bring up the context menu
- Select Run As > EventFlow Fragment from the context menu
4. Select the Producer fragment for input
- In the Manual Input view, choose producer from the Fragment drop-down:
5. Publish a message to the Kafka topic
-
For the Stream KafkaDataIn, fill in the following values:
- topic: topic1
- message: hello
- key: 1
- partition: null
- Click Send Data
6. Observe that the message is consumed
- In the Output Streams view, note the first line of the grid is for message=hello, key=1
7. Stop the producer and consumer fragments
- On the Studio tool bar, press the double red square ( ) icon.
-
In the Selection Needed dialog
- Check the checkbox to the left of consumer
- Check the checkbox to the left of producer
- Click OK
8. Clean up Azure resources
- If you are not going to be using the Event Hubs Namespaces and Event Hub created during this lab again, delete them to avoid incurring any further Azure charges.
-
To delete your Event Hub Namespace
- o In the Azure Portal, go to your Dashboard screen
- o Under My Resources > All Resources, click on the name of your Namespace
- o On the Event Hubs Namespace screen, click the Delete icon
-
In the Delete Namespace dialog
- Type in the name of your namespace to confirm the deletion
- Click the Delete button
Recommended Comments
There are no comments to display.