5

My org is looking into the possibility of using Kafka data streaming as part of our applications.

In order to understand this a little bit more I have been doing some research online and think I've grasped a high level understanding of Kafka, it's purpose and it's advantages etc. But I'm wanting to understand more about how to system test an application that utilises Kafka, however I have been struggling to find much information out there that provides some pointers in this area so was hoping there was someone out there who has experienced this first hand and could help point me in the right direction.

At the moment I just have more questions then answers in areas such as:

  • Logging, how can you trace messages through Kafka to help determine if / where an issue has occurred.

  • Replaying message, if a message fails how can this be replayed.

  • How can you query Kafka, is it possible to query Kafka to verify that messages have been stored correctly or to run your own queries on the messages to confirm that further analysis done on the streams is correct.

  • Automated testing with Kafka, our automation suite currently utilises SpecFlow and Selenuim, is it possible to use the same tools to automate tests where kafka is a system component?

Any help, links to blogs / videos etc. would be very welcome.

Improve this question

1 Answer 1

Reset to default
1

So Kafka is a message broker basically. I haven't utilized it personally but here is the detailed documentation including the APIs that are available to utilize: https://kafka.apache.org/documentation/

The biggest question when looking at a message brokering system is how it fits into an application. Usually a system like this is specifically designed to handle the messages that then are fed into an application. The specific calls to the messages are programed into some sort of user interface. Unless you work for Apache on the Kafka product I think your main focus should be on how your project/projects are specifically targeting utilizing Kafka. If your trying to "test Kafka" then I'd assume you are working for apache as it's a product.

I would consider targeting the following questions and getting those answers:

  1. What is the implementation strategy of my org for utilizing Kafka?
  2. Within that implementation strategy what are the interfaces that will be developed for end users to access message content managed via Kafka?
  3. Once you know those two answers you can then ask what technology will best automate testing of these developed interfaces which will render the managed message content?
  4. After you have that then you should know where the originating source data is coming from in which case then it would be how best to check the source with the final product?
  5. Then you can get into tracking failures and logs to ensure you find where gaps may exist...

Unless your unit testing, you need to start from the final product and go backwards. Integration testing start at the APIs specifically, not the product under the API. So your final picture will look more like a GUI/RSS feed/something else that gives me x from interacting with y functionality. Integration of y with Kafka and Kafka with the original source(s). Check the original source(s) to see if x = original source(s) x.

It's all data being sent and managed, it's just how it's being constructed is the key. Think of it this way, if you had 15 different databases and pulled info from all into a mix you would want to ensure the constructed mix of data was the same as the original parts it was constructed from. Same principal here with a message broker like Kafka.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.