Release v0.30.0

ADDITIONS

Watchman now filters out indexed records based on the first character's phonetic match. This is helpful to eliminate most
low scoring results and reduces CPU usage.

You can force scoring search terms against every indexed record by setting DISABLE_PHONETIC_FILTERING=yes.

Release v0.29.2

IMPROVEMENTS

• cmd/server: only wrap pipeline steps with debugStep during debugging
• cmd/server: pool x/text/transform.Transformer instances

BUILD

• build: update golang.org/x/text
• cmd/server: reduce memory usage in benchmark setup

Release v0.29.1

IMPROVEMENTS

• feat: read WITH_EU_SCREENING_LIST as option to skip EU CSL list
• feat: usaddress: add Similarity function
• remove MySQL and sqlite dependencies, resolves #468
• remove bloat not needed any more when we build with CGO_ENABLED=0
• remove database from dependencies and build system

BUILD

• build(deps): bump body-parser and express in /webui
• build(deps): bump braces from 3.0.2 to 3.0.3 in /webui
• build(deps): bump ejs from 3.1.9 to 3.1.10 in /webui
• build(deps): bump express from 4.18.2 to 4.19.2 in /webui
• build(deps): bump follow-redirects from 1.15.4 to 1.15.6 in /webui
• build(deps): bump google.golang.org/protobuf from 1.31.0 to 1.33.0
• build(deps): bump micromatch from 4.0.5 to 4.0.8 in /webui
• build(deps): bump rexml from 3.2.6 to 3.3.6 in /docs
• build(deps): bump rollup from 2.79.1 to 2.79.2 in /webui
• build(deps): bump webpack from 5.89.0 to 5.95.0 in /webui
• build(deps): bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /webui
• build(deps): bump webrick from 1.8.1 to 1.8.2 in /docs
• build(deps-dev): bump nokogiri from 1.16.2 to 1.16.5 in /docs
• build: convert docker-compose to docker compose
• build: switch from docker-compose
• chore(deps): update dependency github-pages to v232
• fix(deps): update dependency express to v4.19.2 [security]
• fix(deps): update react monorepo to v18.3.1
• fix(deps): updates body-parser from 1.20.2 to 1.20.3
• fix(deps): updates express from 4.19.2 to 4.21.0

Release v0.28.2

IMPROVEMENTS

• fix: download all files if the initial dir does not exist

Release v0.28.1

IMPROVEMENTS

• fix: close and cleanup downloaded files after successful parsing

BUILD

• build(deps-dev): bump nokogiri from 1.13.10 to 1.16.2 in /docs
• chore(deps): update dependency github-pages to v231

Release v0.28.0

ADDITIONS

• feat: add /crypto endpoint and extract digital currency addresses from OFAC
• feat: add SEARCH_MAX_WORKERS

IMPROVEMENTS

• build: fix npm build for Openshift
• build: force latest stable Go when building docker images
• fix: accumulate SDN comments for merging overflow after all files are read
• fix: allocate known-capacity arrays
• fix: copy extended remarks from sdn_comments.csv
• largest: improve performance, simplify insert step

BUILD

• build(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0

Release v0.27.0

This release of Watchman includes additional improvements to the search match scores to reduce false positives and increase true positive matches. A few of the specific improvements are:

1. Compare tokens in the search to the index tokens
   • i.e. "find matches for every search token" rather than "find match for every indexed token"
   • Improves scores of searches that don't include "middle" names
   • Prevents sanctioned names that are 1 word (HADI, EMMA, KAMILA) matching long searches
   • Has a side-effect that short search terms will have more false positives. I think this is a good trade off as the sanction lists will always contain the full name, but the search might not
2. Once a token has matched something, it can't match a different token
   • This prevents names with repeated words having artificially high scores
   • e.g. prevents any search containing "Vladimir" matching "VLADIMIROV, Vladimir Vladimirovich"
3. Weights each word-score by the length of the word, relative to the search and indexed name
   • This corrects for error that is introduced by splitting names into tokens and doing piecewise Jaro-Winkler scoring
   • Combing word-scores using a simple average gives short words (like Li, Al) equal weight to much longer words
   • The length-weighted scores are comparable to what you get by doing whole-name to whole-name Jaro-Winkler comparison
4. Punishes word-scores when the matching tokens have significantly different length
5. Punishes word-scores when the matching tokens start with different letters

Release v0.26.1

This release of Watchman has removed Company/Customer models and Watches. They've been deprecated for a while and do not perform as users expect. Stay tuned for a future Moov OSS project integrating with Watchman for sanctions screening.

IMPROVEMENTS

• feat: return matchedName in non-OFAC results
• search: apply more edge case logic to decrease bad scoring
• search: return matchedName for OFAC SDNs, Alts, and DPL records
• test: remove duplicate (and skipped) UK/EU CSL tests

Release v0.25.0

This release of Watchman lowers most match percentages by comparing names better.

IMPROVEMENTS

• fix: close xml encoder
• fix: panic cleanup from newer linter rules
• cmd/server: only check adjacent terms for local jaro max score
• cmd/server: read ADJACENT_SIMILARITY_POSITIONS env var
• cmd/server: weight term score by length similarity

BUILD

• build: bump numerous javascript dependencies
• build: update to Go 1.21
• build: update to node 20
• build: update Debian, Fedora, node, and Go base images

Release v0.24.2

IMPROVEMENTS

• fix: keep numbers during stopwords step
• fix: stop setting level=error for info logs

BUILD

• bump golang.org/x/crypto to v0.6.0
• bump golang.org/x/net from 0.6.0 to 0.7.0
• build(deps): bump activesupport from 6.1.7.2 to 7.0.4.3 in /docs
• build(deps): bump webpack from 5.75.0 to 5.76.1 in /webui