Jump to content

  • HowTo Connect to Microsoft Azure Event Hubs using the Spotfire® Streaming Input Adapter for Apache Kafka Consumer

     Share

    Overview

    In this Spotfire® Streaming HowTo, we will use the Spotfire® Streaming Input Adapter for Apache Kafka Consumer to consume message from a Microsoft Azure EventHub. This is a step-by-step guide and assumes no prior knowledge of Azure EventHubs or Apache Kafka.

    Prerequisites

    • Spotfire® Streaming 10.4.x or later installed
    • A Microsoft online account (usually created via live.com)
    • A Microsoft Azure subscription (the free account is sufficient to perform the examples in this document)
    • Internet access to Microsoft Azure, especially to port 9093 on your chosen subdomain of servicebus.windows.net

    Directions

    Complete the instructions that follow.

    Part 1: Create Azure Event Hubs Namespace and Event Hubs

    1. Create a Kafka-enabled EventHubs Namespace

    Tip: Microsoft has published documentation for this process here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create-kafka-enabled. You may wish to check this page in case Microsoft has updated this page or the related functionality since this HOWTO document was created in September 2019.

    • Use a web browser to navigate to the Azure Portal (portal.azure.com)
    • Sign in to your Azure account (details out of scope of this HowTo
    • In the upper-left hand corner of the Portal screen, click on Create a resource

    image1_14.png.6de95f05510690993b3ce2e2331874c8.png

    • Enter Event Hubs in the Search marketplace... text box and press the Enter key

    image2_15.png.5b6003454fe5215a7bc2c4f65a4a3413.png

    • On the resulting Event Hubs screen, click the Create button

    image3_11.png.deceffc5f6b6940d9cbb16077938d338.png

    • On the Create Namespace screen, enter the following values:
      • Name: <a unique name just for you>
      • Pricing tier: Standard
      • Enable Kafka: check the checkbox
      • Subscription: Free trial
      • Resource group: <a unique resources group name, just for you>
      • Location: <pick something close to you>
      • Enable Auto-Inflate: leave this option unchecked

    image4_8.png.57ce3195733afe5aeeba9f567b94f2af.png

    • Press the Create button (which is below the option entry form, in case you have to scroll):

    image5_9.png.6822ff1d26bda1111503070de25b8083.png

    • It may take a while for the name space to be created and deployed. You can monitor the progress of this operations in the notifications area in the upper right-hand corner of the Azure Portal screen:

    image6_8.png.420a270698f05a1535880ba5cdfbaea3.png

    • When the namespace is available, a Notification to that effect will appear:

    image7_6.png.18dd0065c7726d2e459602f199b87e1a.png

    • For the purposes of this HowTo, secure your namespace to only accept connections from your client IP address:
      • On your Event Hub's Namespace page, click on Settings > Firewalls and virtual networks
        • Allow access: Click the Selected networks radio button
        • Check the Firewall > Add your client IP address checkbox
        • Click the Save button up near the top of the screen

    Tip: If you keep this namespace for any future uses and use it from other IP addresses, you will have to manage the allowed addresses from this screen.

     

    image8_5.png.c2510328c13bfbe7f4b3a4865100ce37.png

     

     

    2. Create an Event Hub within the Namespace

    • In the left-hand column of the Azure Portal screen, click on Dashboard

    image9_5.png.f1bf2680cc75dc28d011de5597640409.png

    • Under My Dashboard > All resources, locate the newly created Event Hubs Namespace

    image10_9.png.56a8f0be8052a112b43ae872a62786ea.png

    • Click on the name of the Namespace to bring up the Namespaces's detail screen
    • Click on + Event Hub to create an Event Hub in the Namespace

    image11_12.png.971891639fab7ed9267ad81934aa58f5.png

    • On the Create Event Hub screen:
      • Name: topic1
      • Partition Count: 1 (the default)
      • Message Retention: 1 (the default)
      • Capture: Off (the default)
    • Press the Create button

    image12_7.png.394c8878755e3465b2f6a4540ab8e23e.png

    • Verify that topic1 is created and has the options you selected by clicking on Entities > Event Hubs on the Event Hubs Namespace screen

    image13_7.png.d6cce20532880c35e995e8e611aa0aae.png

     

    Part 2: Configure the EventFlow modules for Azure Event Hubs

    1. Start StreamBase Studio and navigate to the SB Authoring perspective

    • Open StreamBase Studio
      • Double-click on the StreamBase Studio 10.4 icon on the Windows Desktop

    image14_8.png.5048911488930670914b2f6a69ef965e.png

    • Accept the default value for your StreamBase Studio workspace location and name:

    image15_8.png.a38d5202684db71890c6ad221478890c.png

    • Click Launch
    • You are presented with the Welcome screen for StreamBase Studio

    image16_8.png.42d11077f3f3be935895af649dfa73dc.png

    • In the upper right corner of the Welcome screen, click Workbench.
    • This action takes you to the StreamBase Studio Demos (SB Demo) perspective.
    • In the upper right corner, click Go To Authoring:

    image17_9.thumb.png.e8812ff9a5f734b6aa0fda47e73510d6.png

    • This action results in a display of the SB Authoring perspective:

    image18_7.thumb.png.29b03dee21d3261a7da34b4c7e996750.png

    Note: In StreamBase Studio, a perspective is a collection of views that support a set of related tasks.

    2. Import the Azure Events Hubs Using Kafka example project into your workspace.

    • From the top menu, choose File > Import...
      • Choose General > Existing Projects into Workspace, and click Next
      • In the next screen, choose Select archive file
      • Click Browse... and choose .../StreamingExampleKafkaAzureEventHubs1044.zip (attached to this page)
        • - Click Finish
      • Click Cancel to skip the installation of an SVN Connector, if asked

    3. Copy the Connection String to the clipboard

    • In the Azure Portal, from the Event Hubs Namespace screen for your namespace, click on Settings > Shared Access Policies

    image19_6.png.2d4b6e2c3bc9116ef8cd2b791b2ccd56.png

    • Click on RootManagedSharedAccessKey

    image20_7.png.78bc6914656a4283f154a58449ca319b.png

     

    • Copy the Connection String Primary Key

    image29_2.png.bf73117e4e1a47e43215946c1d033c67.png

    4. Configure the Broker and Connection String into the EventFlow modules

    • In the StreamBase Studio Project Explorer view, open example_kafka_azureeventhubs > src/main/configurations > sbengine.conf

    image22_7.png.304074f0a905700ed7ac7ac82b29f7e7.png

    • Paste your connection string from the clipboard to replace YOUR_CONNECTION_STRING_GOES_HERE, as the value of the EventHubsConnectionString operator parameter.

    image23_6.png.3e3eabc7fed7b13f6b9f68c6ec95516d.png

    • Copy the fully qualified domain name of your Event Hubs Namespace to replace YOUR_BROKER_FQDN_GOES_HERE, as the partial value of the KafkaBrokers operator parameter
    • The resulting sbengine.conf file will look something like this, using your namespace's name and shared access key:

    image24_7.thumb.png.b047eb4aaedc54a5125187f55f62a013.png

    • Save your sbengine.conf file (Studio Menu Bar > File > Save All)

    Part 3: Run the Kafka Consumer and Producer Programs

    1. Run the Consumer program

    • In the StreamBase Studio Project Explorer view, navigate to example_kafka_azureeventhubs > src/main/eventflow > com.tibco.streaming.example.kafka_azureeventhubs
    • Select consumer.sbapp and right-click on it to bring up the context menu
    • Select Run As > EventFlow Fragment

    image25_2.png.c685c2c8934d8215710726a1cd8a882f.png

    • The fragment is running and ready to receive client input when the message This application is currently running at sb://yourhost... appears on the application editor canvas:

    image26_2.png.1c5024f47f9f7ab50c288c23f38c028f.png

    2. Subscribe to the topic1 topic

    • In the Manual Input view:
      • Fragment: ensure that it is consumer
      • Stream: ensure that it is ConsumerCommand
        • command: subscribe
        • topic: topic1
        • leave the rest of the input fields set to null (the default)
      • Click Send Data
    • In the Console view, a successful subscribe command results in a number of INFO log messages. In particular, look for these:
      • Successfully logged in
      • Subscription to topic 'topic1' successful

    Tip: It's possible to see success messages here, but due to network connectivity configurations still not be able to connect to the Kafka broker for the Event Hubs Namespace. At this point the subscribe command appears successful. If there are connectivity issues connecting to the broker, connection error messages will appear after a minute or so as the Consumer adapter retries the connection periodically.

    3. Run the Producer program

    • In the upper right corner of the Studio window, click on the SB Authoring perspective button

    image27_1.png.3651c3202943cdb1a97d5d2f64902263.png

    • In the StreamBase Studio Project Explorer view, select producer.sbapp and right-click on it to bring up the context menu
    • Select Run As > EventFlow Fragment from the context menu

    4. Select the Producer fragment for input

    • In the Manual Input view, choose producer from the Fragment drop-down:

    image28_3.png.4c9f709028fac3866a483b394538b3c1.png

     

    5. Publish a message to the Kafka topic

    • For the Stream KafkaDataIn, fill in the following values:
      • topic: topic1
      • message: hello
      • key: 1
      • partition: null
    • Click Send Data

    6. Observe that the message is consumed

    • In the Output Streams view, note the first line of the grid is for message=hello, key=1

    image29_2.png.b79d91007b16a7b237147f9b496e3f5b.png

    7. Stop the producer and consumer fragments

    • On the Studio tool bar, press the double red square ( ) icon.
    • In the Selection Needed dialog
      • Check the checkbox to the left of consumer
      • Check the checkbox to the left of producer
      • Click OK

    image31_2.png.20fc843a445814f5687294a15373a54a.png

    8. Clean up Azure resources

    image32_1.png.f450f2f6f39e516483dee04cf240b8d1.png

    • If you are not going to be using the Event Hubs Namespaces and Event Hub created during this lab again, delete them to avoid incurring any further Azure charges.
    • To delete your Event Hub Namespace
      • o In the Azure Portal, go to your Dashboard screen
      • o Under My Resources > All Resources, click on the name of your Namespace
      • o On the Event Hubs Namespace screen, click the Delete icon
    • In the Delete Namespace dialog
      • Type in the name of your namespace to confirm the deletion
      • Click the Delete button

    image33_2.png.c985711d6b312fce3d2bbdd30a453e16.png

     

    streambaseexamplekafkaeventhubs1044.zip

    example_kafka_simple_1061.zip

     Share
    Go to articles

    User Feedback

    Recommended Comments

    There are no comments to display.


×
×
  • Create New...