ReOpenLDAP-1.0.0

Briefly:

1. The first stable release ‎ReOpenLDAP‬ on Great Victory Day!
2. Fixed huge number of bugs. Made large number of improvements.
3. On-line replication works robustly in the mode multi-master.

Currently ReOpenLDAP operates in telco industry throughout Russia:

• few 2x2 multi-master clusters with online replication.
• up to ~100 millions records, up to ~100 Gb data.
• up to ~10K updates per second, up to ~25K searches.

Seems no anyone other LDAP-server that could provide this (replication fails,
not reaches required performance, or just crashes).

New:

• slapd: 'keepalive' config option.
• slapd: adds biglock's latency tracer (-DSLAPD_BIGLOCK_TRACELATENCY=deep).
• mdbx: lifo-reclaimig for weak-to-steady conversion.
• contrib: backport - ITS#6826 conversion scripts.
• mdbx: simple ioarena-based benchmark.
• syncrepl: 'require-present' config option.
• syncprov: 'syncprov-showstatus' config option.

Documentation:

• man-ru: 'syncprov-showstatus none/running/all' feature.
• man: libreldap ITS#7506 Properly support DHParamFile (backport).

Major bugs:

• syncrepl: fix RETARD_ALTER when no-cookie but incomming entryCSN is newer.
• mdbx: backport - ITS#8412 fix NEXT_DUP after cursor_del.
• mdbx: backport - ITS#8406 fix xcursors after cursor_del.
• backend-mdb: fix 'forgotten txn' bug.
• syncprov: fix error handling when syncprov_findcsn() fails.
• syncprov: fix rare segfault in search_cleanup().
• backend-bdb/hdb: fix cache segfault.
• syncprov: fix possibility of loss changes.
• syncprov: fix error handling in find-max/csn/present.
• syncprov: fix 'missing present-list' bug.
• syncprov: avoid lock-order-reversal/deadlock (search under si-ops mutex).
• slapd: fix segfault in connection_write().
• mdbx: backport - ITS#8363 Fix off-by-one in mdb_midl_shrink().
• mdbx: backport - ITS#8355 fix subcursors.
• syncprov: avoid deadlock with biglock and/or threadpool pausing.

Minor bugs:

• syncrepl: refine status-nofify for dead/dirty cases.
• syncrepl: more o_dont_replicate for syncprov's mock status.
• syncrepl: don't notify QS_DIRTY before obtain connection.
• syncprov: refine matchops() for search-cleanup case.
• slapd: fix valgrind-checks for sl-malloc.
• liblber: fix hipagut support for realloc.
• backend-ldap: fix/remove gentle-kick.
• mdbx: workaround for pthread_setspecific's memleak.
• mdbx: clarify mdbx_oomkick() for LMDB-mode.
• syncrepl: backport - ITS#8413 don't use str2filter on precomputable filters.
• mdbx: always copy the rest of page (MDB_RESERVE case).
• mdbx: fix nasty/stupid mistake in cmp-functions.
• mdbx: backport - ITS#8393 fix MDB_GET_BOTH on non-dup record.
• slapd: request thread-pool pause only for SLAP_SERVER_MODE.
• slapd: fix backover bug (since 532929a).
• syncrepl: enforce csn/cookie while recovering lost-delete(s).
• syncrepl: fix 'quorum' for mad configurations.
• backend-mdb: fix mdb_opinfo_get() error handling.
• syncrepl: fix 'limit-concurrent-refresh' feature.
• slapd: ignore EBADF in epoll_ctl(DEL).
• syncprov: fix rare assert-failure on race with abandon.
• mdbx: fix mdb_kill_page() for MDB_PAGEPERTURB.
• libldap: backport - ITS#8385 Fix use-after-free with GnuTLS.
• syncprov: fix minor op-memleak.
• syncrepl: don't skipping retarded DELETE-notification with UUID.
• syncrepl: don't replicate glue-ancestors, but not an entry.
• syncrepl: del_nonpresent() - filtering glue entries as usual.
• syncrepl: consider notifications with the same CSN as an 'echo'.
• syncrepl: checking for present-list before delete-nonpresent.
• syncrepl: accepts extra refresh-present in multi-master.
• syncprov: send cookie even if entryCSN available.
• syncprov: oversight refresh-present for multi-master.
• syncprov: fix persistent-search - rework cleanup and release.
• syncprov: don't skips sending DELETE-notify to the originator.
• syncprov: don't filter refresh-resp just by originator.
• slapd: stop scan on self-committed in slap_get_commit_csn().
• syncrepl: enumerates operations to distinguish from each other.
• backend-mdb: fix mistake backport ITS#8226.
• syncrepl: fix & rework compare_cookies().
• syncrepl: fix pre-condition for delete-nonpresent.
• mdbx: mdbx_chk - empty freedb record isn't an error.
• backend-mdb: fix infinite loop in callback removal.
• mdbx: fix percent in mdbx_txn_straggler().
• slapd: fix and make optional ordering of pending-csn queue.
• mdbx: fix madvise() flags, it is not a bitmask.
• mdbx: cleanup inherited errno's bug.
• syncrepl: accept empty incoming cookies if iddqd/idclip is off.
• syncprov: don't skip 1900-sub csn in SS_CHANGED case.
• syncprov: backport - ITS#8365 partially revert ITS#8281.
• backend-mdb: more to avoid races on mi_numads.
• backend-mdb: backport - ITS#8360 fix ad info after failed txn.
• backend-mdb: backport - ITS#8226 limit size of read txns in searches.
• slapd: cleanup bullshit around op->o_csn.
• syncprov: wake waiting mod-ops when handle loop-pause.
• syncprov: don't block mod-ops by waiting fetch-ops when pool-pause pending.
• accesslog: backport - ITS#8351 fix callback init.
• syncprov: mutual fetch/modify - wakes opposite if waiting was broken.
• syncrepl: fix race on cookieState->cs_ref.

Performance:

• mdbx: more likely/unlikely for mdb_node_add.
• slapd: remove crutch-locks from config-backend.
• mdbx: don't memcpy when src eq dest.
• backend-mdb: logs begin/end of OOM, but not an iteration.
• mdbx: refine find_oldest() and oom_kick().
• mdbx: refine/speedup mdb_cmp_memn().
• mdbx: MADV_REMOVE for unallocated space.
• mdbx: extra backlog's page for MDB_LIFORECLAIM.
• mdbx: rework backlog for freeDB deletion.
• mdbx: refine mdb_env_sync0().
• mdbx: refine mdbx_cmp2int().
• mdbx: backport - mdb_drop optimization.
• slapd: remove unused scoped-locks.
• syncrepl: non-modal del_nonpresent().
• syncprov: less locking for mock ContextCSN.
• syncrepl: less debug/logging.
• slapd: less debug/logging for biglock.
• mdbx: more __inline/__hot.
• backend-mdb: less debug/logging for dreamcatcher.
• mdbx: refine mdb_meta_head_r().
• mdbx: refine mdb_env_sync().
• syncrepl: simplify biglock usage in msg-loop.
• slapd: kill bconfig's crutch mutex.
• syncrepl: minor speedup check_for_retard().
• mdbx: msync only used part instead of entire db.

Build:

• backend-mdb: ability to use old/origin liblmdb.
• libldap: ITS#8353 more for OpenSSL 1.1.x compat.
• libldap: ITS#8353 partial fixes (openssl 1.1.x).

Cosmetics:

• mdbx: reporting 'Unallocated' instead of 'Free pages'.
• mdbx: reporting 'detaited' instead of 'reading'.

Other:

• mdbx: be a bit more precise that mdb_get retrieves data (ITS#8386).
• mdbx: adds MDBX_ALLOC_KICK for freelist backlog.
• check MDB_RESERVE against MDB_DUPSORT.
• syncrepl: dead/dirty/refresh/process/ready for quorum-status.
• syncrepl: idclip-resync when present-list leftover after del_nonpresent().
• syncrepl: fix missing strict-refresh in config-unparse.
• syncprov: tracking for refresh-stage of sync-psearches.
• syncprov: jammed & robust sync.
• mdbx: adds MDB_PAGEPERTURB.

For news and changes early than 2016 please refer to ChangeLog

Stable internal version (2016-03-31)

Corresponds to OpenLDAP 2.4.44, but less buggy.

A major huge change is that replication is works properly here (in contrast from OpenLDAP).

Stable internal version (2015-06-23)

Corresponds to OpenLDAP 2.4.41, but less buggy.

In general is a one problem - replication has a enough of bugs inherited from OpenLDAP.
Changes may get lost by replication, but more over - a lot of records could be incorrectly deleted inside the replication scope.

Semistable internal version (2015-05-15)

Corresponds to OpenLDAP 2.4.40++, but less buggy.

Major changes since PS/003.00:

• mdb_chk tool;
• quorum and limit-concurrent-refresh features;
• some fixes and crutches for replication;

Semistable internal version (2015-04-07)

Corresponds to OpenLDAP 2.4.40+, but less buggy.

Major changes:

• fixed ~5K warnings from gcc;
• fixed ~25 critical (segfault) bugs;
• fixed ~25 memory leaks;
• Russian man pages;
• internal memory checker;
• oom-handler for LMDB;
  ...

Semistable internal version (2014-10-29)

Based on OpenLDAP 2.4.40 and includes second (working) version of LIFO-reclaiming feature.
But has a most of bugs that were just inherited from an original OpenLDAP.

Broken internal version just for history (very old).

Based on OpenLDAP 2.4.38 and includes first (wrong) version of LIFO-reclaiming feature.

forking point from original OpenLDAP (2015-01-07).

The squashed snapshot of OpenLDAP's branch OPENLDAP_REL_ENG_2_4 at the 2015-01-07.