My org is looking into the possibility of using Kafka data streaming as part of our applications.
In order to understand this a little bit more I have been doing some research online and think I've grasped a high level understanding of Kafka, it's purpose and it's advantages etc. But I'm wanting to understand more about how to system test an application that utilises Kafka, however I have been struggling to find much information out there that provides some pointers in this area so was hoping there was someone out there who has experienced this first hand and could help point me in the right direction.
At the moment I just have more questions then answers in areas such as:
Logging, how can you trace messages through Kafka to help determine if / where an issue has occurred.
Replaying message, if a message fails how can this be replayed.
How can you query Kafka, is it possible to query Kafka to verify that messages have been stored correctly or to run your own queries on the messages to confirm that further analysis done on the streams is correct.
Automated testing with Kafka, our automation suite currently utilises SpecFlow and Selenuim, is it possible to use the same tools to automate tests where kafka is a system component?
Any help, links to blogs / videos etc. would be very welcome.