v1.22.2

Dragonfly v1.22.2

This is a patch release.

fix(docker/healthcheck): run netstat port retreival command as dfly (#3647)
fix: data inconsistency on replica during replication (#3710)

v1.22.1

Dragonfly v1.22.1

This is a patch release. Urgency to update: low

fix: infinite recursive calls in the allocation tracker #3665

Full Changelog: v1.22.0...v1.22.1

v1.22.0

Dragonfly v1.22.0

Some prominent changes include:

• Lots of stability fixes around JSON and Search APIS:
  #3459, #3284, #3537, #3544,#3580, #3582, #3578, #3591, #3597, #3539
• Support HELLO AUTH handshake: #3486
• Implement FT.TAGVALS: #3493
• Introduce Dragonfly specific lua pragmas: #3517
• Memcache now replicates flags correctly: #3538
• Memcache API now supports absolute timestamps as well: #3607
• PEXPIRETIME and EXPIRETIME are implemented by @Eunoia1729 - thank you!
• Last but not least, a significant contribution by @cunla: a massive test suite, likely the largest in Dragonfly's history, with 12K lines of code. This contribution has sparked further efforts to enhance Dragonfly's compatibility and stability. The initial results are included in this release.

What's Changed

• fix: missing logs on ci timeout by @kostasrim in #3452
• fix: JSON.MSET command by @BorysTheDev in #3459
• chore: integrate quicklist changes from valkey by @romange in #3440
• feat(json_family): Add support of the JSON legacy mode by @BagritsevichStepan in #3284
• fix: pytest teardown exception in os.remove(LAST_LOGS) by @kostasrim in #3463
• chore: change how we track memory_budget during evictions by @romange in #3457
• chore: add timeout to replication sockets by @romange in #3434
• chore(server): Remove old blocking debug by @dranikpg in #3460
• chore: SinkReplyBuilder2 with vec batching by @dranikpg in #3454
• feat: Allow pre-declaring Lua SHAs to run with undeclared keys by @chakaz in #3465
• chore(server): Use parser more in list_family by @dranikpg in #3461
• fix: Make replica's REPLCONF IP-ADDRESS optional by @chakaz in #3473
• chore: make tiered_test more reliable by @romange in #3477
• chore(facade): Update SinkReplyBuilder2 by @dranikpg in #3474
• refactor: set info_replication_valkey_compatible=true by @BorysTheDev in #3467
• chore: relax the version check in test_replicate_old_master by @romange in #3478
• chore: Download Dragonfly versions only when needed by @chakaz in #3479
• fix: division by zero bug by @romange in #3482
• test: fix test_scripts by @BorysTheDev in #3485
• chore(facade): RedisReplyBuilder2 base by @dranikpg in #3475
• chore(facade): MCReplyBuilder2 by @dranikpg in #3480
• chore(facade): RedisReplyBuilder2 (extensions) by @dranikpg in #3481
• fix: big value serialization corner cases by @kostasrim in #3430
• fix: HELLO AUTH with non default user by @kostasrim in #3486
• fix: flake in teardown of test_unknown_dfly_env by @kostasrim in #3487
• fix: test_replicate_old_master missing serialization_max_chunk_size by @kostasrim in #3498
• chore: add logs to debug tiered memory failures by @romange in #3499
• feat: implement FT.TAGVALS by @romange in #3493
• test: fix test_acl_deluser by @BorysTheDev in #3488
• fix: sanitizers post failure on google chat by @kostasrim in #3496
• chore(generic_family): Fix bad data format error in the RESTORE command by @BagritsevichStepan in #3501
• fix: sanitizers false positives by @kostasrim in #3495
• feat: Expose replica_reconnect_count for Prometheus metrics by @BagritsevichStepan in #3370
• fix: pytest warnings by @kostasrim in #3497
• fix: missing virtual destructors in ReplyBuilder2 by @kostasrim in #3506
• chore: split engine_shard file from engine_shard_set by @romange in #3507
• chore: introduce a secondary TaskQueue for shards by @romange in #3508
• fix: clang warnings by @romange in #3509
• chore: reduce pipelining latency by reusing existing shard fibers by @romange in #3494
• fix: bugs around the growth of a tiered file by @romange in #3516
• chore: Change Lua embedded flags syntax by @chakaz in #3517
• feat(cluster): Allow appending RDB to existing store by @chakaz in #3505
• fix: disable ThreadLocalMutex when big value ser is off by @kostasrim in #3521
• chore(io): Introduce (carefully) new io with use_new_io flag by @dranikpg in #3513
• chore(io): Optimize repeated ReservePiece calls by @dranikpg in #3525
• fix: disable sanitizers false positive by @kostasrim in #3522
• chore(config): make pipeline_squash configurable by @dranikpg in #3529
• chore(traffic loger): use pipelining and print/analyze commands by @dranikpg in #3527
• fix: skip empty objects on load and replication by @kostasrim in #3514
• fix: deadlock in Heartbeat() by @kostasrim in #3530
• fix(search_family): Add error on creating indexes from non-zero databases by @BagritsevichStepan in #3537
• fix: macos build by @kostasrim in #3536
• fix: MC flags size and serialization #3134 by @BorysTheDev in #3538
• fix(transaction): Don't set continuation for blocking by @dranikpg in #3540
• fix: named volume permissions in docker by @romange in #3518
• fix(jsonpath): Add JsonPath grammar for the child identifier in brackets by @BagritsevichStepan in #3533
• fix: build errors in sanitizers daily workflow by @kostasrim in #3542
• test: tests using fakeredis to check compatibility by @cunla in #3534
• chore: fix search replication by @dranikpg in #3547
• chore: add deallocation logs in the allocation tracker by @romange in #3549
• fix: fakeredis tests support python 3.10 by @cunla in #3550
• feat(server_family): Add backup/restore Prometheus metrics by @BagritsevichStepan in #3520
• docs: Fakeredis tests README by @cunla in #3552
• fix: zinterstore correctly finds weights by @romange in #3554
• fix(search_family): Fix query parsing for the integer tags in OR expression by @BagritsevichStepan in #3544
• chore(server): Fix watch by @dranikpg in #3557
• chore(search): fix numeric index query in rev order by @dranikpg in #3555
• chore: allow limiting pipelining queue by length by @romange in #3551
• fix: add unsupported_server_types to fakeredis tests by @cunla in #3560
• fix: string compatibility issues by @romange in #3564
• fix(server): Fix ZRANGEBYLEX limit params by @dranikpg in #3562
• fix: compatibility around list and string commands by @romange in #3565
• chore(server): Unify zset arg parsing by @dranikpg in #3563
• fix: compatibility around list,string and sort commands by @romange in #3568
• chore(server): Sort correctly in ZINTER by @dranikpg in #3566
• chore(server): Unify zset boolean operations into single function by @dranikpg in #3567
• chore(transaction): More blocking tests by @dranikpg in #3546
• fix: delete empty dense sets in HGetGeneric by @kostasrim in #3543
• chore(search): Silence query parser error by @dranikpg in https:...

v1.21.4

Dragonfly v1.21.4

This is a patch release.

chore: fix search replication (#3547)

v1.21.3

Dragonfly v1.21.3

This is a patch release.

fix: display real replica remote address in INFO logs
fix: crash with concurrent multi-shard BLMOVE operations #3540

This follows up on Dragonfly v1.21.2 release.

v1.21.2

Dragonfly v1.21.2

This is a patch release.

fix: disable code that can cause potential deadlocks during the replication (#3521)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy #3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

Full Changelog: v1.21.1...v1.21.2

v1.21.1

Dragonfly v1.21.1

This is a patch release.

fix: the replication from older masters to newer versions (#3473)

This follows up on Dragonfly v1.21.0 release, that includes the following prominent features:

• Alpha release of SSD Data tiering - enabled with flag --tiered_prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

v1.21.0

Dragonfly v1.21.0

Some prominent changes include:

• Alpha release of SSD Data tiering - enabled with flag --prefix some/path/basename
• Very basic support of multi-tenancy @3260
• HSETEX now supports NX option, see our docs.
• Added support for JSON.MERGE and for JSON.MSET.
• valkey's replica-announce-ip and --cluster--announc--ip are consolidated via --announce-ip flag.

What's Changed

• feat(server): master stop sending exec opcode to replica by @adiholden in #3289
• chore: optimize zpopminmax operations by @romange in #3291
• fix(json_family): Fix error in JsonFamilyTest.MGet by @BagritsevichStepan in #3285
• fix: define macro WITH_AWS in cmake when flag is ON by @kostasrim in #3296
• chore: Add 'memory arena show' command by @romange in #3298
• fix: missing logs on pytest failures #3255 by @BorysTheDev in #3272
• chore: refactor compact_object and introduce materialize method by @romange in #3300
• fix(acl): loading interleaved plain and hashed passwords by @kostasrim in #3297
• chore: Add CompactObj Raw methods by @romange in #3303
• chore: On invalid TTL, print the TTL first by @chakaz in #3306
• chore: moving functions + renaming in tiered storage by @romange in #3304
• fix(tests): Add missing awaits by @chakaz in #3273
• chore: refactoring around tiered storage by @romange in #3299
• feat: yield when serialization is in progress by @kostasrim in #3220
• refactor: Use DbContext, OpArgs and Transaction to access DbSlice by @chakaz in #3311
• chore: Separate tiered serialization format from object values by @romange in #3310
• chore: add more community links to README by @Niennienzz in #3308
• fix(cluster): Join on specified attempt id by @chakaz in #3305
• chore(acl): add test with requirepass and aclfile by @kostasrim in #3312
• chore: skip test_cluster_flushall_during_migration by @kostasrim in #3316
• fix (pytest): generate unique random dbfilename for tests by @adiholden in #3317
• chore: bypass decoding/encoding of the data when performing offloading by @romange in #3315
• refactor: acl helpers and global tables by @kostasrim in #3309
• chore(tiering): Fixes by @dranikpg in #3225
• fix(migration): Use transactions! by @dranikpg in #3266
• fix: forbid DFLYCLUSTER commads set for emulated cluster mode by @BorysTheDev in #3307
• feat(namespaces): Initial support for multi-tenant by @chakaz in #3260
• chore: improve dfly_bench stats by @romange in #3320
• chore: remove replace_deleted flag from hnswlib by @dranikpg in #3323
• feat(test): Improve benchmark workflow by @adiholden in #3330
• fix: Proper shutdown sequence with Namespaces by @chakaz in #3333
• fix(test): copy logs for failed test during TEARDOWN phase by @BorysTheDev in #3331
• fix(json_family): fix JSON.STRAPPEND command for JSON legacy mode by @BagritsevichStepan in #3264
• chore(tiering): add protection against overruning memory budget by @romange in #3327
• chore: Add coordinated omission mode by @romange in #3332
• chore: implement sequential pass without the overlapping traffic by @romange in #3335
• feat: add an option to flush serialized entries on threshold limit by @kostasrim in #3241
• feat(hset_family): Add NX option to HSETEX by @BagritsevichStepan in #3295
• Fix blocking commands moved error by @BorysTheDev in #3334
• fix: ub in RegisterOnChange and regression tests for big values by @kostasrim in #3336
• fix: Cancel outgoing migration when retrying / closing by @chakaz in #3339
• chore: Make KeyIndex iterable by @dranikpg in #3326
• fix: AllocationTracker::Remove return value was reversed by @romange in #3341
• chore: remove redundant metrics from memory stats by @romange in #3345
• fix: corruption in replication stream by @romange in #3344
• chore: clean up TaskQueue since we do not need multiple fibers for it by @romange in #3348
• chore: pull helio by @romange in #3350
• chore: Log connection context when issuing dangerous cmds by @chakaz in #3352
• chore: small rename and add dcheck on LocalBlockingCounter by @kostasrim in #3356
• refactor: reduce number of logs for cluster by @BorysTheDev in #3357
• chore: fixes to dfly_bench by @romange in #3353
• chore: fix test_parser_memory_stats flakiness by @romange in #3354
• chore(server): Introduce StringSetWrapper by @dranikpg in #3347
• fix: do not upload offload values on a first hit by @romange in #3360
• chore(tiering): Range functions + small refactoring by @dranikpg in #3207
• fix: failure in test_cluster_fuzzymigration by @romange in #3363
• fix(server): Require >=1 args to GETEX by @chakaz in #3366
• fix(transaction): Fix namespace access by @dranikpg in #3364
• chore: disable compression on big values by @kostasrim in #3358
• fix: protect OnJournalEntry with ConditionGuard by @kostasrim in #3367
• chore: Introduce CoolQueue by @romange in #3365
• chore: small fixes around tiering by @romange in #3368
• chore: add a test for HeapSize() function by @romange in #3349
• chore: simplify computation of used_mem_current by @romange in #3372
• chore: disable cluster_fuzzymigration by @kostasrim in #3373
• chore(replica): remove unused methods in the Replica class by @BagritsevichStepan in #3374
• fix(transaction): Properly store block cancel status by @dranikpg in #3371
• chore: Track db_slice table memory instantly by @romange in #3375
• chore: add mem test for big values and default the flag by @kostasrim in #3369
• update: replication_acks_interval flag to 1000 by @kostasrim in #3378
• chore: dfly_bench - print ongoing error counts by @romange in #3382
• chore: introduce a cool queue that gradually retires cool items by @romange in #3377
• chore: update cached stats inside PollExecution by @romange in #3376
• fix: Fix test_take_over_seeder by @chakaz in #3385
• chore: reenable evictions upon insertion to avoid OOM rejections by @romange in #3387
• fix: remove fiber guard from non atomic section by @kostasrim in #3381
• chore: do not preempt on db_slice::RegisterOnChange by @kostasrim in #3388
• fix: disable inline transactions when db_slice has registered callbacks by @kostasrim in #3391
• fix: test_big_value_serialization_memory_limit shutdown timeout by @kostasrim in #3390
• chore: set serialization_max_chunk_size to 1 byte by @kostasrim in #3379
• chore: tiere...

v1.20.1

Dragonfly v1.20.1

This is a patch release.

fix: S3 access for loading/saving snapshots (#3296)

Signed-off-by: kostas kostas@dragonflydb.io

v1.20.0

Dragonfly v1.20.0

Some prominent changes include:

• Improvements around client side caching (#3136 #3158)
  Specifically, rueidis is now supported (#2454)
• Improvements around search and vector search APIs, FT.ALTER is now supported (#3144, #3148, #3178, #3186)
• JSON.MSET is added (#3167)
• Expiry notifications are now supported (#3154)

What's Changed

• test(cluster_mgr): Add tests for cluster_mgr.py by @chakaz in #3129
• chore: get rid of kv_args and replace it with slices to full_args by @romange in #3101
• fix: Fix live-lock in connection test by @chakaz in #3135
• chore: call breaker_cb_ on shutdown by @romange in #3128
• chore: Add option to disable aws dependency by @dranikpg in #3077
• feat(cluster): Add important logs by @chakaz in #3138
• chore: a small cleanup by @romange in #3140
• test: skip test_cluster_migration_cancel, it is broken by @BorysTheDev in #3146
• chore: update helio by @kostasrim in #3150
• fix: fix bug in cluster/slot_set by @BorysTheDev in #3143
• chore: recommit client tracking by @kostasrim in #3136
• feat(search): Tag field options (separator, case sensitivity) by @dranikpg in #3144
• feat(search): basic FT.ALTER by @dranikpg in #3148
• chore: Introduce pipeline back-pressure by @romange in #3152
• fix: lua and client tracking by @kostasrim in #3163
• feat: client tracking optout by @kostasrim in #3158
• feat: add noloop subcommand in client tracking by @kostasrim in #3164
• fix(ci): docker not using iouring by @kostasrim in #3169
• fix(ci): add missing docker option on reg tests workflow by @kostasrim in #3172
• fix: acl compatibility by @kostasrim in #3147
• chore: Streamer is rewritten with async interface by @romange in #3108
• feat(json): MSET by @dranikpg in #3167
• fix(cluster): Wait for flow creation before cancelling it by @chakaz in #3159
• chore: improve ft.create for vector search by @romange in #3178
• feat: add incoming migration error processing by @BorysTheDev in #3175
• chore: add parsing support for the rest of VECTOR parameters by @romange in #3179
• fix(cluster): Wait before all access to slot migrations by @chakaz in #3180
• fix(server): fix bug in replication on cached mode by @adiholden in #3156
• chore: small acl compat changes by @kostasrim in #3177
• chore: allow calling Context::ReportError from I/O dispatch fiber by @romange in #3182
• chore: add a lexer test covering KNN query by @romange in #3185
• chore(search): Add __vector_score alias by @dranikpg in #3186
• chore: Refactor string span management by @dranikpg in #3165
• chore(ci): run replication tests on arm by @kostasrim in #3168
• chore(search): improve parser error reporting by @romange in #3184
• fix(reply_builder): remove virtual modifier in SendError(ErrorReply) method by @BagritsevichStepan in #3191
• fix(cluster): Support FLUSHALL while slot migration is in progress by @chakaz in #3173
• fix: fix RegisterOnChange methods for journal and db_slice by @BorysTheDev in #3171
• fix(generic_family): fix RenameGeneric command for non-string data types by @BagritsevichStepan in #3181
• fix(server): Rename confusing flag replica_reconnect_on_master_restart by @chakaz in #3193
• fix(unit tests): fix generic family info test by @adiholden in #3187
• test: improve cluster_fuzzy_migration test by @BorysTheDev in #3197
• chore(core): Remove DfImpl inside ScoredMap by @dranikpg in #3199
• chore(tiering): Remove IoMgr by @dranikpg in #3198
• test(cluster): Make sure migration maintains TTL by @chakaz in #3188
• feat(cluster): Support STICK bit in slot migration by @chakaz in #3200
• chore: Re-enable previously flaky test by @chakaz in #3196
• feat(acl): add support of multiple passwords by @kostasrim in #3189
• chore(tiering): More advanced tiering tests by @dranikpg in #3201
• chore: add replica-priority flag by @kostasrim in #3204
• chore: fix tiering regtest test by @dranikpg in #3208
• feat(generic_family): Assemble RESTORE and STICK commands into one commmand during replication of the RENAME command by @BagritsevichStepan in #3209
• core(search): Add EF_RUNTIME parameter by @dranikpg in #3206
• fix(sanitizers): failing json test by @kostasrim in #3176
• feat(server): expiry notifications by @dranikpg in #3154
• fix: replicaof_reject_on_load flake by @kostasrim in #3203
• fix(cluster): Don't miss keys when migrating slots by @chakaz in #3218
• fix(tiering): Fix pending leak during immediate stash error by @dranikpg in #3219
• chore: provide basic logging to catch possible command errors by @romange in #3213
• chore: disable tiering test by @dranikpg in #3236
• fix(transaction): Don't transactionalize empty EVAL inside EXEC by @dranikpg in #3231
• fix: fix deadlock in DFLYCLUSTER CONFIG command and outgoing migration finaliztion by @BorysTheDev in #3239
• chore(tiered): minor fixes + expose buffer allocation type stats by @romange in #3234
• fix(pytest): timed ticker for simpler conditions by @dranikpg in #3242
• fix(server): Fix SCAN deadlock by @dranikpg in #3235
• chore: introduce back-pressure to tiered storage by @romange in #3243
• chore: improve dfly_bench by @romange in #3244
• chore(tiering): External alloc free page chaining by @dranikpg in #3237
• fix: total_stash_overflows statistic by @romange in #3248
• chore: update action versions by @romange in #3250
• chore(lua): Return which undeclared key was accessed by @chakaz in #3245
• chore: replace session wide fixtures with scope by @kostasrim in #3251
• chore: pull helio and adjust mimalloc for 32MiB segments by @kostasrim in #3174
• chore: add more logs around the duplicate value case in small_bins by @romange in #3254
• fix: fix move error during migration finalization by @BorysTheDev in #3253
• chore: upload all the logs from /tmp by @romange in #3263
• fix(tiering): Throttle snapshot load by @dranikpg in #3249
• chore: initiate grow preemptively by @romange in #3267
• chore(monitoring): add more dashboards + memcached by @romange in #3268
• fix(json_family): fix JSON.GET commmand for JSON legacy mode by @BagritsevichStepan in #3261
• chore: dfly_bench can send traffic to memcached by @romange in #3271
• chore: print effective QPS of the server. by @romange in #3274
• chore: debugging fixes by @romange in https...