Many clusters are set up, each with a tiny handful of topics and partitions driving huge cloud provider costs and inefficient use of resources. GDPR, PII, PCI, Regulations: how can you even consider applying them at scale?.Security, encryption of data: a black hole.People turnover will create knowledge-loss.Data may be duplicated as teams might create their own clusters to hold the same information.Reduce cost or complexity by clearing up topics.The organization has no way to enforce various strategies or is going to be in pain: No one else except the Product team owning one Kafka knows what the topics are for. Topic and cluster configuration varies hugely and doesn't adhere to best practices - infinite expiry can result in substantial storage costs, expiry is not correctly considered, data can be lost, etc.Security is inconsistent, with different (or no) policies across the teams.They can rely on Kafka Expert in the company, but the goal is similar to the DevOps culture:Īll the Kafka clusters and operations are going to be different as there are no gatekeepers to enforce organizational rules, eg: The organization encourages teams to self-serve their own Kafka as they were building their own Spring Boot application: they become responsible for Kafka resource management, topic creation, ACLS, Kafka Connect, and general Kafka admin. They will move fast and deal with their problems their way. On the other side of the spectrum, we have the full decentralized model: let each Product team do their thing. Let's decentralize Kafka ownership, Free The Data! # One typical solution of organizations is to grow this Central team, adding more people to absorb the throughput of requests instead of fixing the real issue: don't create a bottleneck. This Central team would LOVE to focus on Kafka and unleashing value from its data, spreading it even more in the organization and accelerating how developers work with it… but they are spending time answering all the support questions and trying not to be an administration bottleneck!
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |