Read brokers and Partition size infomation from input json file instead of Zookeeper #257
Conversation
1) stream all partitions metrics to DD is too expensive, read from file 2) zookeeper only allows 1MB size for ZNODE, too small to store all our partition infos
|
Generally the idea would be to write other metricsfetcher-like tools that would feed the data into ZooKeeper; this ensures that topicmappr is always reading the source data in exactly one way and minimizes config flags and codepaths to maintain. |
|
Yes I am agree with it, but our partition size map is just too huge(too long topic names and partitions), so the map exceed the Max datalengh of zookeeper znode(1MB) and can't be inserted, that's why we want to coopte with the file inputs option. |
|
This is a problem I wanted to solve in general by either supporting compression or breaking the data structure up a bit (e.g. separate entries for each topic), adding versioning to the data structure as well to cope with the various formats. I'll open an issue for this. |
|
For reference: #258 |
| @@ -1,6 +1,6 @@ | |||
| package main | |||
|
|
|||
| import "github.com/DataDog/kafka-kit/cmd/topicmappr/commands" | |||
| @@ -29,6 +29,8 @@ func init() { | |||
| rebuildCmd.Flags().Bool("use-meta", true, "Use broker metadata in placement constraints") | |||
| rebuildCmd.Flags().String("out-path", "", "Path to write output map files to") | |||
| rebuildCmd.Flags().String("out-file", "", "If defined, write a combined map of all topics to a file") | |||
| rebuildCmd.Flags().String("partitions-size-in-file", "", "Read Topics partitions sizes from a file") | |||
There was a problem hiding this comment.
Please add the changes in this file to the rebalance.go file too. That will allow file input for the rebalance option too.
We want have option to read Partition size information from an input file
Currently it relies on metricfetcher to grab metrics from datadog.