Add Prefix, Suffix and Ngram UDFs #12392

deemoliu · 2024-02-09T21:54:16Z

feature: Adding ngram, prefix, postfix UDFs

Context:

We are onboarding a use case and trying the inrease query throughput. We tested the QPS cannot further improved with the existing REGEXP_LIKE queries or text_match queries. The queries as follows

select col1, col2 from table where REPEXP_LIKE(col3, '^data*')
select col1, col2 from table where REGEXP_LIKE(col3, 'data$')
select col1, col2 from table where REGEXP_LIKE(col3, '*data*')
select col1, col2 from table where TEXT_MATCH(col3, '/data*/')
...

The plan is to generated the derived columns that persisted prefix, postfix, and ngram to use inverted indexes to filter the result fast, and add the text match indexes to do validation after filtering to avoid false positive result.

This patch is created to generate prefix, postfix, and ngrams for a field.
it can be used by the following transformation config

       {
          "columnName": "col_prefix",
          "transformFunction": "prefixes(col, 3, null)"
        },
       {
          "columnName": "col_suffix",
          "transformFunction": "suffixes(col, 3, null)"
        },
       {
          "columnName": "col_3gram",
          "transformFunction": "ngrams(col, 3)"
        },
       {
          "columnName": "col_1gram_2gram_3gram",
          "transformFunction": "ngrams(col, 1, 3)"
        },

codecov-commenter · 2024-02-09T22:30:19Z

Codecov Report

Attention: Patch coverage is 89.18919% with 4 lines in your changes missing coverage. Please review.

Project coverage is 62.19%. Comparing base (59551e4) to head (179d70a).
Report is 892 commits behind head on master.

Files	Patch %	Lines
.../pinot/common/function/scalar/StringFunctions.java	89.18%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12392      +/-   ##
============================================
+ Coverage     61.75%   62.19%   +0.43%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2502      +66     
  Lines        133233   136584    +3351     
  Branches      20636    21145     +509     
============================================
+ Hits          82274    84942    +2668     
- Misses        44911    45364     +453     
- Partials       6048     6278     +230

Flag	Coverage Δ
custom-integration1	`<0.01% <0.00%> (-0.01%)`	⬇️
integration	`<0.01% <0.00%> (-0.01%)`	⬇️
integration1	`<0.01% <0.00%> (-0.01%)`	⬇️
integration2	`0.00% <0.00%> (ø)`
java-11	`62.15% <89.18%> (+0.44%)`	⬆️
java-21	`62.05% <89.18%> (+0.43%)`	⬆️
skip-bytebuffers-false	`62.17% <89.18%> (+0.42%)`	⬆️
skip-bytebuffers-true	`35.02% <89.18%> (+7.30%)`	⬆️
temurin	`62.19% <89.18%> (+0.43%)`	⬆️
unittests	`62.18% <89.18%> (+0.43%)`	⬆️
unittests1	`46.73% <89.18%> (-0.16%)`	⬇️
unittests2	`27.95% <0.00%> (+0.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

Jackie-Jiang

Are there equivalent/similar functions in other commonly used DBs (e.g. PostgreSQL)? We should try to match the behavior

deemoliu · 2024-03-06T18:51:15Z

Are there equivalent/similar functions in other commonly used DBs (e.g. PostgreSQL)? We should try to match the behavior

i think postgresql provide a 3-gram module called pg_trgm where pg means PostgreSQL.
it including the following functions
show_trgm (text)

Do you think i should rename the function similarly?

Jackie-Jiang · 2024-03-06T22:23:34Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   * @param maxlength the max length of the prefix strings for the string.
+   * @return generate an array of prefix strings of the string that are shorter than the specified length.
+   */
+  @ScalarFunction


You want to add alias unique_prefixes, same for other functions

Do we need unique though? The prefixes will always be unique because they all have different length

sg. i think the reason of unique_prefixes is to reserve prefixes for other purpose or implementations. if no objection, let me use prefixes() then.

Jackie-Jiang · 2024-03-06T22:25:01Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   */
+  @ScalarFunction
+  public static String[] uniquePrefixes(String input, int maxlength) {
+    ObjectSet<String> prefixSet = new ObjectLinkedOpenHashSet<>();


We don't need a set since all prefixes have different length. Same for other functions

updated, thanks

Jackie-Jiang · 2024-03-06T22:26:07Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   */
+  @ScalarFunction
+  public static String[] uniquePrefixesWithPrefix(String input, int maxlength, String prefix) {
+    if (prefix == null) {


In order to accept null, you want to annotate it as nullableParameters. Please also annotate the parameter to be @Nullable

updated, thanks

This is not addressed ^^
Take a look at ScalarFunction.class. You need to annotate it as nullableParameters

thanks @Jackie-Jiang for pointer. updated

Jackie-Jiang · 2024-04-16T00:14:17Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   */
+  @ScalarFunction
+  public static String[] prefixes(String input, int maxlength) {
+    ObjectList<String> prefixList = new ObjectArrayList<>();


ObjectArrayList is not really buying us anything here.
Given we know the number of prefixes upfront, we can directly allocate the array

replaced the objectArrayList with a fixSizeArr. thanks

Jackie-Jiang · 2024-04-16T00:15:11Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   */
+  @ScalarFunction
+  public static String[] uniquePrefixesWithPrefix(String input, int maxlength, String prefix) {
+    if (prefix == null) {


This is not addressed ^^
Take a look at ScalarFunction.class. You need to annotate it as nullableParameters

Jackie-Jiang · 2024-04-16T00:15:46Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

@@ -18,6 +18,10 @@
 */
 package org.apache.pinot.common.function.scalar;

+import it.unimi.dsi.fastutil.objects.ObjectArrayList;


No need to use fastutil here. We can use the java default ones

Jackie-Jiang

Mostly good

Jackie-Jiang · 2024-04-19T18:27:41Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   * @param prefix the prefix to be prepended to prefix strings generated. e.g. '^' for regex matching
+   * @return generate an array of prefix matchers of the string that are shorter than the specified length.
+   */
+  @ScalarFunction(nullableParameters = true, names = {"prefix"})


Do you want to alias it to prefix? I don't think this is really prefix

Jackie-Jiang · 2024-04-19T18:27:49Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   * @param suffix the suffix string to be appended for suffix strings generated. e.g. '$' for regex matching.
+   * @return generate an array of suffix matchers of the string that are shorter than the specified length.
+   */
+  @ScalarFunction(nullableParameters = true, names = {"suffix"})


updated, thanks @Jackie-Jiang

Jackie-Jiang · 2024-04-19T18:29:22Z

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java

+   */
+  @ScalarFunction
+  public static String[] uniqueNgrams(String input, int minGram, int maxGram) {
+    ObjectSet<String> ngramSet = new ObjectLinkedOpenHashSet<>();


hi @Jackie-Jiang ngrams doesn't guarantee to be unique, right? so the usage of Set is to dedup and avoid duplicates.

npawar · 2024-05-22T20:14:44Z

Has this been added to the docs @deemoliu ?

ankitsultana reviewed Feb 12, 2024

View reviewed changes

ankitsultana reviewed Feb 13, 2024

View reviewed changes

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java Outdated Show resolved Hide resolved

xiangfu0 added the feature label Feb 19, 2024

ankitsultana reviewed Feb 23, 2024

View reviewed changes

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java Outdated Show resolved Hide resolved

ankitsultana reviewed Feb 23, 2024

View reviewed changes

pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java Outdated Show resolved Hide resolved

Jackie-Jiang reviewed Feb 24, 2024

View reviewed changes

Jackie-Jiang reviewed Mar 6, 2024

View reviewed changes

deemoliu force-pushed the qiaochu/ngram-udf branch from 9fa29e7 to 9859704 Compare April 15, 2024 22:01

Jackie-Jiang reviewed Apr 16, 2024

View reviewed changes

Jackie-Jiang reviewed Apr 19, 2024

View reviewed changes

deemoliu added 6 commits April 23, 2024 10:29

Add Prefix, Suffix and Ngram UDFs

172a015

address code comment

d5d76b5

address code comment

449031c

address code comments

5431393

address code comments

3056269

address code comments

179d70a

deemoliu force-pushed the qiaochu/ngram-udf branch from 0af362a to 179d70a Compare April 23, 2024 17:35

Jackie-Jiang approved these changes Apr 23, 2024

View reviewed changes

Jackie-Jiang merged commit 36c4b9a into apache:master Apr 23, 2024
19 of 20 checks passed

Jackie-Jiang added the documentation label Apr 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Prefix, Suffix and Ngram UDFs #12392

Add Prefix, Suffix and Ngram UDFs #12392

deemoliu commented Feb 9, 2024 •

edited

Loading

codecov-commenter commented Feb 9, 2024 •

edited

Loading

Jackie-Jiang left a comment

deemoliu commented Mar 6, 2024

Jackie-Jiang Mar 6, 2024

Jackie-Jiang Mar 6, 2024

deemoliu Apr 15, 2024

Jackie-Jiang Mar 6, 2024

deemoliu Apr 15, 2024

Jackie-Jiang Mar 6, 2024

deemoliu Apr 15, 2024

Jackie-Jiang Apr 16, 2024

deemoliu Apr 16, 2024

Jackie-Jiang Apr 16, 2024

deemoliu Apr 16, 2024

Jackie-Jiang Apr 16, 2024

Jackie-Jiang Apr 16, 2024

deemoliu Apr 16, 2024

Jackie-Jiang left a comment

Jackie-Jiang Apr 19, 2024

Jackie-Jiang Apr 19, 2024

deemoliu Apr 23, 2024

Jackie-Jiang Apr 19, 2024

deemoliu Apr 23, 2024

npawar commented May 22, 2024

Add Prefix, Suffix and Ngram UDFs #12392

Add Prefix, Suffix and Ngram UDFs #12392

Conversation

deemoliu commented Feb 9, 2024 • edited Loading

codecov-commenter commented Feb 9, 2024 • edited Loading

Codecov Report

Jackie-Jiang left a comment

Choose a reason for hiding this comment

deemoliu commented Mar 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

npawar commented May 22, 2024

deemoliu commented Feb 9, 2024 •

edited

Loading

codecov-commenter commented Feb 9, 2024 •

edited

Loading