Client failed to connect to right redis master

[mahanam]

ISSUE: The client is not maintaining a proper connection with the right master.
COnfiguration and setup details:
Bitnami charts
NAME: redis-ha
REVISION: 1
RELEASED: Wed Aug 19 19:04:31 2020
CHART: redis-10.7.16
USER-SUPPLIED VALUES:
{}
Setup configuration.log

Actually i triggered my redis-go client towards my setup => [single master {redis+sentinel} and 3 slaves{redis+sentinel}]

1. The first time it's working fine it's connected to the right Redis master.
2. Now after some time I have deleted the current master pod {redis+sentinel}
3. After failover, the client able to pick the right master at this time.
4. Now after some time I have deleted the master pod {redis+sentinel}
5. But here, the client stuck into the loop with the old IP address which is a stale entry.
   =========================================================
6. Magic comes here, It will connect to the right Redis master when you restart your client.
   While in this process, it reconnects the appropriate master after a client restart
   ======================================================
7. After (6) point have killed the running and working client just to check whether it will work for next time.
   =========================================================
8. When i start the client again it stucks with stale ip address as mentioned in (5) point

ISSUE: The client is not maintaining a proper connection with the right master.
Please guide me with proper behavior and the right configuration.

LOGS:
2020/08/19 19:13:07 Initializing the redis client...
2020/08/19 19:13:07 Getting the sentinel redis master...
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="7a732b2572c10a61e22e6e79885d1286b943669d" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="852f11a9d3634574e8786838ce5fb26c43f6bec1" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:470: sentinel: discovered new sentinel="3301ae7a855f25d2a6b0085a879c3609be823e72" for master="mymaster"
redis: 2020/08/19 19:13:07 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.7:6379"
2020/08/19 19:13:07 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:13:07 Retrieved key 0:0
2020/08/19 19:13:08 Retrieved key 1:1
2020/08/19 19:13:09 Retrieved key 2:2
2020/08/19 19:13:10 Retrieved key 3:3
2020/08/19 19:13:11 Retrieved key 4:4
2020/08/19 19:13:12 Retrieved key 5:5
2020/08/19 19:13:13 Retrieved key 6:6
2020/08/19 19:13:14 Retrieved key 7:7
2020/08/19 19:13:15 Retrieved key 8:8
2020/08/19 19:13:16 Retrieved key 9:9
2020/08/19 19:13:17 Retrieved key 10:10
2020/08/19 19:13:18 Retrieved key 11:11
2020/08/19 19:13:19 Retrieved key 12:12
2020/08/19 19:13:20 Retrieved key 13:13
2020/08/19 19:13:21 Retrieved key 14:14
2020/08/19 19:13:22 Retrieved key 15:15
2020/08/19 19:13:25 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:27 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:27 Retrieved key 16:
2020/08/19 19:13:29 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:30 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:30 Retrieved key 17:
2020/08/19 19:13:33 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:35 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:35 Retrieved key 18:
2020/08/19 19:13:38 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:40 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:40 Retrieved key 19:
2020/08/19 19:13:43 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:44 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:44 Retrieved key 20:
2020/08/19 19:13:47 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:49 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:49 Retrieved key 21:
2020/08/19 19:13:51 Write Errordial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:52 dial tcp 10.233.77.7:6379: connect: invalid argument
2020/08/19 19:13:52 Retrieved key 22:
redis: 2020/08/19 19:13:53 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:13:53 Retrieved key 23:23
2020/08/19 19:13:54 Retrieved key 24:24
2020/08/19 19:13:55 Retrieved key 25:25
2020/08/19 19:13:56 Retrieved key 26:26
2020/08/19 19:13:57 Retrieved key 27:27
2020/08/19 19:13:58 Retrieved key 28:28
...
2020/08/19 19:15:11 Retrieved key 100:100
2020/08/19 19:15:12 Retrieved key 101:101
redis: 2020/08/19 19:15:13 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.12:6379"
2020/08/19 19:15:13 Retrieved key 102:102
2020/08/19 19:15:14 Retrieved key 103:103
2020/08/19 19:15:15 Retrieved key 104:104
2020/08/19 19:15:16 Retrieved key 105:105
...
2020/08/19 19:16:47 Retrieved key 195:195
redis: 2020/08/19 19:16:48 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:16:50 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:52 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:52 Retrieved key 196:
2020/08/19 19:16:54 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:56 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:16:56 Retrieved key 197:
2020/08/19 19:16:58 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:00 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:00 Retrieved key 198:
2020/08/19 19:17:02 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:03 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:03 Retrieved key 199:

2020/08/19 19:17:05 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:07 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:07 Retrieved key 200:
2020/08/19 19:17:09 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:10 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:10 Retrieved key 201:
2020/08/19 19:17:13 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:14 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:14 Retrieved key 202:
2020/08/19 19:17:17 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:18 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:18 Retrieved key 203:
2020/08/19 19:17:21 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:23 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:23 Retrieved key 204:
2020/08/19 19:17:26 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:27 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:27 Retrieved key 205:
2020/08/19 19:17:30 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:31 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:31 Retrieved key 206:
2020/08/19 19:17:34 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:35 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:35 Retrieved key 207:
2020/08/19 19:17:37 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:38 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:38 Retrieved key 208:
2020/08/19 19:17:41 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:43 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:43 Retrieved key 209:
2020/08/19 19:17:46 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:48 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:48 Retrieved key 210:
2020/08/19 19:17:50 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:51 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:51 Retrieved key 211:
2020/08/19 19:17:54 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:56 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:56 Retrieved key 212:
2020/08/19 19:17:58 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:59 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:17:59 Retrieved key 213:
2020/08/19 19:18:02 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:04 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:04 Retrieved key 214:
2020/08/19 19:18:06 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:08 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:08 Retrieved key 215:
2020/08/19 19:18:10 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:12 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:12 Retrieved key 216:
2020/08/19 19:18:14 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:17 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:17 Retrieved key 217:
2020/08/19 19:18:19 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:20 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:20 Retrieved key 218:
2020/08/19 19:18:22 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:24 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:18:24 Retrieved key 219:

===============================================================================
2020/08/19 19:18:58 Initializing the redis client...
2020/08/19 19:18:58 Getting the sentinel redis master...
redis: 2020/08/19 19:18:58 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.13:6379"
2020/08/19 19:18:58 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:18:58 Retrieved key 0:0
2020/08/19 19:18:59 Retrieved key 1:1
2020/08/19 19:19:00 Retrieved key 2:2
2020/08/19 19:19:01 Retrieved key 3:3
2020/08/19 19:19:02 Retrieved key 4:4
2020/08/19 19:19:03 Retrieved key 5:5
2020/08/19 19:19:04 Retrieved key 6:6
2020/08/19 19:19:05 Retrieved key 7:7
2020/08/19 19:19:06 Retrieved key 8:8
2020/08/19 19:19:07 Retrieved key 9:9

=============================================================================================
2020/08/19 19:20:20 Initializing the redis client...
2020/08/19 19:20:20 Getting the sentinel redis master...
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="852f11a9d3634574e8786838ce5fb26c43f6bec1" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="3301ae7a855f25d2a6b0085a879c3609be823e72" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:470: sentinel: discovered new sentinel="57bff9e9e2cca87c135ec4eefc146a08f4654b9d" for master="mymaster"
redis: 2020/08/19 19:20:20 sentinel.go:438: sentinel: new master="mymaster" addr="10.233.77.11:6379"
2020/08/19 19:20:22 Connection failed with redis master serverdial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:22 Successfully connected to the redis master server:[10.233.13.207:26379]
2020/08/19 19:20:24 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:25 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:25 Retrieved key 0:
2020/08/19 19:20:27 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:29 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:29 Retrieved key 1:
2020/08/19 19:20:32 Write Errordial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:34 dial tcp 10.233.77.11:6379: connect: invalid argument
2020/08/19 19:20:34 Retrieved key 2:

Snippet of my code:

if config.RedisSentinel {
    log.Print("Getting the sentinel redis master...")
    RClient = redis.NewFailoverClient(&redis.FailoverOptions{
        MasterName:    "mymaster",
        SentinelAddrs: []string{"10.233.13.207:26379"},
        SentinelPassword: "iFFRAhsXq0",
        Password: "iFFRAhsXq0",
        MaxRetries:       10,
        DB: 0,
    })
} else {
    log.Print("Getting the redis client...")
    RClient = redis.NewClient(&redis.Options{
        Addr:     config.RedisAddr,
        Password: config.RedisPassword,
        DB:       config.RedisDB,
    })
}

err := RClient.Ping(ctx).Err()
if err != nil {
    log.Print("Connection failed with redis master server", err)
}

log.Print("Successfully connected to the redis master server:", config.RedisSentinelAddrs)

}

Expected behavior
If you delete the current master pod the client should be able to connect the right redis master after failover window

[Queetinliu]

you should write all your sentinels,other than only one sentinels.