Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

get offsets count is off by 1 for each partition #144

Closed
joshua-courtney-okcupid opened this issue Jun 13, 2023 · 2 comments
Closed

get offsets count is off by 1 for each partition #144

joshua-courtney-okcupid opened this issue Jun 13, 2023 · 2 comments

Comments

@joshua-courtney-okcupid
Copy link

when I use topicctl get offsets the total count is off by the number of partitions. I suspect that that the number of messages in each partition is being calculated as last_offset - first_offset when it should be last_offset - first_offset + 1

Here's an example output. I know that I just put 100,000 test messages into this topic, but the total only shows 99900

[0]:~/kafka $ topicctl get offsets queue.purge.delete  --cluster-config=cluster-config/dev/cluster.yaml
[2023-06-13 07:59:59]  INFO Partition bounds for topic queue.purge.delete:
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
  PARTITION | FIRST OFFSET |      FIRST TIME      | LAST OFFSET |      LAST TIME       | MESSAGES | DURATION | AVG RATE
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
  0         | 2955         | 2023-06-13T11:55:20Z | 3939        | 2023-06-13T11:55:21Z | 984      | 373ms    | 2638/sec
  1         | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  2         | 2997         | 2023-06-13T11:55:20Z | 3995        | 2023-06-13T11:55:21Z | 998      | 373ms    | 2675/sec
  3         | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  4         | 2943         | 2023-06-13T11:55:20Z | 3923        | 2023-06-13T11:55:21Z | 980      | 373ms    | 2627/sec
  5         | 3090         | 2023-06-13T11:55:20Z | 4119        | 2023-06-13T11:55:21Z | 1029     | 373ms    | 2758/sec
  6         | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  7         | 2940         | 2023-06-13T11:55:20Z | 3919        | 2023-06-13T11:55:21Z | 979      | 373ms    | 2624/sec
  8         | 3189         | 2023-06-13T11:55:20Z | 4251        | 2023-06-13T11:55:21Z | 1062     | 373ms    | 2847/sec
  9         | 2742         | 2023-06-13T11:55:20Z | 3655        | 2023-06-13T11:55:21Z | 913      | 373ms    | 2447/sec
  10        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  11        | 2958         | 2023-06-13T11:55:20Z | 3943        | 2023-06-13T11:55:21Z | 985      | 373ms    | 2640/sec
  12        | 2892         | 2023-06-13T11:55:20Z | 3855        | 2023-06-13T11:55:21Z | 963      | 373ms    | 2581/sec
  13        | 3081         | 2023-06-13T11:55:20Z | 4107        | 2023-06-13T11:55:21Z | 1026     | 373ms    | 2750/sec
  14        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  15        | 3129         | 2023-06-13T11:55:20Z | 4171        | 2023-06-13T11:55:21Z | 1042     | 373ms    | 2793/sec
  16        | 3048         | 2023-06-13T11:55:20Z | 4063        | 2023-06-13T11:55:21Z | 1015     | 373ms    | 2721/sec
  17        | 3093         | 2023-06-13T11:55:20Z | 4123        | 2023-06-13T11:55:21Z | 1030     | 373ms    | 2761/sec
  18        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  19        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  20        | 3036         | 2023-06-13T11:55:20Z | 4047        | 2023-06-13T11:55:21Z | 1011     | 373ms    | 2710/sec
  21        | 3147         | 2023-06-13T11:55:20Z | 4195        | 2023-06-13T11:55:21Z | 1048     | 373ms    | 2809/sec
  22        | 3006         | 2023-06-13T11:55:20Z | 4007        | 2023-06-13T11:55:21Z | 1001     | 373ms    | 2683/sec
  23        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  24        | 3171         | 2023-06-13T11:55:20Z | 4227        | 2023-06-13T11:55:21Z | 1056     | 373ms    | 2831/sec
  25        | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  26        | 3171         | 2023-06-13T11:55:20Z | 4227        | 2023-06-13T11:55:21Z | 1056     | 373ms    | 2831/sec
  27        | 2697         | 2023-06-13T11:55:20Z | 3595        | 2023-06-13T11:55:21Z | 898      | 373ms    | 2407/sec
  28        | 2958         | 2023-06-13T11:55:20Z | 3943        | 2023-06-13T11:55:21Z | 985      | 373ms    | 2640/sec
  29        | 2994         | 2023-06-13T11:55:20Z | 3991        | 2023-06-13T11:55:21Z | 997      | 373ms    | 2672/sec
  30        | 2982         | 2023-06-13T11:55:20Z | 3975        | 2023-06-13T11:55:21Z | 993      | 373ms    | 2662/sec
  31        | 3012         | 2023-06-13T11:55:20Z | 4015        | 2023-06-13T11:55:21Z | 1003     | 373ms    | 2689/sec
  32        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  33        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  34        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  35        | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  36        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  37        | 3060         | 2023-06-13T11:55:20Z | 4079        | 2023-06-13T11:55:21Z | 1019     | 373ms    | 2731/sec
  38        | 2955         | 2023-06-13T11:55:20Z | 3939        | 2023-06-13T11:55:21Z | 984      | 373ms    | 2638/sec
  39        | 3081         | 2023-06-13T11:55:20Z | 4107        | 2023-06-13T11:55:21Z | 1026     | 373ms    | 2750/sec
  40        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  41        | 2952         | 2023-06-13T11:55:20Z | 3935        | 2023-06-13T11:55:21Z | 983      | 373ms    | 2635/sec
  42        | 2970         | 2023-06-13T11:55:20Z | 3959        | 2023-06-13T11:55:21Z | 989      | 373ms    | 2651/sec
  43        | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  44        | 2967         | 2023-06-13T11:55:20Z | 3955        | 2023-06-13T11:55:21Z | 988      | 373ms    | 2648/sec
  45        | 2868         | 2023-06-13T11:55:20Z | 3823        | 2023-06-13T11:55:21Z | 955      | 373ms    | 2560/sec
  46        | 2928         | 2023-06-13T11:55:20Z | 3903        | 2023-06-13T11:55:21Z | 975      | 373ms    | 2613/sec
  47        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  48        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  49        | 3054         | 2023-06-13T11:55:20Z | 4071        | 2023-06-13T11:55:21Z | 1017     | 373ms    | 2726/sec
  50        | 3051         | 2023-06-13T11:55:20Z | 4067        | 2023-06-13T11:55:21Z | 1016     | 373ms    | 2723/sec
  51        | 3045         | 2023-06-13T11:55:20Z | 4059        | 2023-06-13T11:55:21Z | 1014     | 373ms    | 2718/sec
  52        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  53        | 3099         | 2023-06-13T11:55:20Z | 4131        | 2023-06-13T11:55:21Z | 1032     | 373ms    | 2766/sec
  54        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  55        | 3141         | 2023-06-13T11:55:20Z | 4187        | 2023-06-13T11:55:21Z | 1046     | 373ms    | 2804/sec
  56        | 2973         | 2023-06-13T11:55:20Z | 3963        | 2023-06-13T11:55:21Z | 990      | 373ms    | 2654/sec
  57        | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  58        | 2895         | 2023-06-13T11:55:20Z | 3859        | 2023-06-13T11:55:21Z | 964      | 373ms    | 2584/sec
  59        | 2967         | 2023-06-13T11:55:20Z | 3955        | 2023-06-13T11:55:21Z | 988      | 373ms    | 2648/sec
  60        | 3012         | 2023-06-13T11:55:20Z | 4015        | 2023-06-13T11:55:21Z | 1003     | 373ms    | 2689/sec
  61        | 3045         | 2023-06-13T11:55:20Z | 4059        | 2023-06-13T11:55:21Z | 1014     | 373ms    | 2718/sec
  62        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  63        | 3039         | 2023-06-13T11:55:20Z | 4051        | 2023-06-13T11:55:21Z | 1012     | 373ms    | 2713/sec
  64        | 3033         | 2023-06-13T11:55:20Z | 4043        | 2023-06-13T11:55:21Z | 1010     | 373ms    | 2707/sec
  65        | 3063         | 2023-06-13T11:55:20Z | 4083        | 2023-06-13T11:55:21Z | 1020     | 373ms    | 2734/sec
  66        | 2997         | 2023-06-13T11:55:20Z | 3995        | 2023-06-13T11:55:21Z | 998      | 373ms    | 2675/sec
  67        | 3129         | 2023-06-13T11:55:20Z | 4171        | 2023-06-13T11:55:21Z | 1042     | 373ms    | 2793/sec
  68        | 2913         | 2023-06-13T11:55:20Z | 3883        | 2023-06-13T11:55:21Z | 970      | 373ms    | 2600/sec
  69        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  70        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  71        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  72        | 2865         | 2023-06-13T11:55:20Z | 3819        | 2023-06-13T11:55:21Z | 954      | 373ms    | 2557/sec
  73        | 3042         | 2023-06-13T11:55:20Z | 4055        | 2023-06-13T11:55:21Z | 1013     | 373ms    | 2715/sec
  74        | 3021         | 2023-06-13T11:55:20Z | 4027        | 2023-06-13T11:55:21Z | 1006     | 373ms    | 2697/sec
  75        | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  76        | 2841         | 2023-06-13T11:55:20Z | 3787        | 2023-06-13T11:55:21Z | 946      | 373ms    | 2536/sec
  77        | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  78        | 2973         | 2023-06-13T11:55:20Z | 3963        | 2023-06-13T11:55:21Z | 990      | 373ms    | 2654/sec
  79        | 3060         | 2023-06-13T11:55:20Z | 4079        | 2023-06-13T11:55:21Z | 1019     | 373ms    | 2731/sec
  80        | 2991         | 2023-06-13T11:55:20Z | 3987        | 2023-06-13T11:55:21Z | 996      | 373ms    | 2670/sec
  81        | 2844         | 2023-06-13T11:55:20Z | 3791        | 2023-06-13T11:55:21Z | 947      | 373ms    | 2538/sec
  82        | 3072         | 2023-06-13T11:55:20Z | 4095        | 2023-06-13T11:55:21Z | 1023     | 373ms    | 2742/sec
  83        | 3069         | 2023-06-13T11:55:20Z | 4091        | 2023-06-13T11:55:21Z | 1022     | 373ms    | 2739/sec
  84        | 3117         | 2023-06-13T11:55:20Z | 4155        | 2023-06-13T11:55:21Z | 1038     | 373ms    | 2782/sec
  85        | 2976         | 2023-06-13T11:55:20Z | 3967        | 2023-06-13T11:55:21Z | 991      | 373ms    | 2656/sec
  86        | 2871         | 2023-06-13T11:55:20Z | 3827        | 2023-06-13T11:55:21Z | 956      | 373ms    | 2563/sec
  87        | 3201         | 2023-06-13T11:55:20Z | 4267        | 2023-06-13T11:55:21Z | 1066     | 373ms    | 2857/sec
  88        | 2904         | 2023-06-13T11:55:20Z | 3871        | 2023-06-13T11:55:21Z | 967      | 373ms    | 2592/sec
  89        | 3069         | 2023-06-13T11:55:20Z | 4091        | 2023-06-13T11:55:21Z | 1022     | 373ms    | 2739/sec
  90        | 3108         | 2023-06-13T11:55:20Z | 4143        | 2023-06-13T11:55:21Z | 1035     | 373ms    | 2774/sec
  91        | 2823         | 2023-06-13T11:55:20Z | 3763        | 2023-06-13T11:55:21Z | 940      | 373ms    | 2520/sec
  92        | 3030         | 2023-06-13T11:55:20Z | 4039        | 2023-06-13T11:55:21Z | 1009     | 373ms    | 2705/sec
  93        | 3015         | 2023-06-13T11:55:20Z | 4019        | 2023-06-13T11:55:21Z | 1004     | 373ms    | 2691/sec
  94        | 2928         | 2023-06-13T11:55:20Z | 3903        | 2023-06-13T11:55:21Z | 975      | 373ms    | 2613/sec
  95        | 3015         | 2023-06-13T11:55:20Z | 4019        | 2023-06-13T11:55:21Z | 1004     | 373ms    | 2691/sec
  96        | 2880         | 2023-06-13T11:55:20Z | 3839        | 2023-06-13T11:55:21Z | 959      | 373ms    | 2571/sec
  97        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  98        | 3156         | 2023-06-13T11:55:20Z | 4207        | 2023-06-13T11:55:21Z | 1051     | 373ms    | 2817/sec
  99        | 3072         | 2023-06-13T11:55:20Z | 4095        | 2023-06-13T11:55:21Z | 1023     | 373ms    | 2742/sec
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
[2023-06-13 07:59:59]  INFO Total bounds for topic queue.purge.delete across all partitions:
-----------------------+----------------------+----------------+----------+-------------
     EARLIEST TIME     |     LATEST TIME      | TOTAL MESSAGES | DURATION |  AVG RATE
-----------------------+----------------------+----------------+----------+-------------
  2023-06-13T11:55:20Z | 2023-06-13T11:55:21Z | 99900          | 373ms    | 267828/sec
-----------------------+----------------------+----------------+----------+-------------
@petedannemann
Copy link
Contributor

Hi @joshua-courtney-okcupid could you open a PR with this fix and write a unit test to reproduce what you are seeing?

@hhahn-tw hhahn-tw mentioned this issue Oct 6, 2023
Closed
@petedannemann
Copy link
Contributor

Fixed by #171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@petedannemann @joshua-courtney-okcupid