Search Against Custom Fields

[NEMA-IT]

Is it possible to enter an SGQL query as a string? For example, I want to find a contact that has a custom field with my CRM ID. Example query:
custom_field_crm_id = '48f82f2b-ae27-48d7-8a7d-7ce3074cf11e'

[Jericho]

The current implementation of the the contacts search API in StrongGrid was designed according to the published documentation. In particular, the list of field that can be used in filter criteria is defined by the list SendGrid has made public here: https://docs.sendgrid.com/for-developers/sending-email/segmentation-query-language#fields

All this to say that I did not design the search functionality to allow searching on custom fields. To be honest, I didn't even know you could use custom fields. Did you find documentation that indicates we can use custom fields as a search criteria?

[NEMA-IT]

The documentation states, "A number of fields are available on every contact. These include the strings..."
https://docs.sendgrid.com/for-developers/sending-email/segmentation-query-language

I take this to mean that the list of fields is not all the fields, just the most common ones. When I search against a custom field (like my previous example), it works (returns the correct contact).

[Jericho]

The statement you highlighted could indeed be interpreted to mean that the list is not exhaustive and you could conclude that the API allows other fields (such as custom fields for example) for searching/segmenting purposes. But on the other hand, I contacted SendGrid more than two years ago and asked them to revise this list of fields and make sure all searchable fields were listed. They pushed an update a few months later to document all the missing fields. At no point did they mention that custom fields were searchable or that there were any other searchable fields.

What's frustrating is that SendGrid archived their documentation GitHub repository in August 2021 which means that we no longer have a way to ask for clarification. On top of that, I noticed that they (recently?) removed the ability to raise issues in their public repos (such as the "sendgrid-csharp" pictured below) which means we can't ask questions on GitHub either. As far as I can tell, the only way to ask question is to contact their support team.

Can you contact their support and try to get an authoritative answer as to whether custom fields are searchable or not? (I know you tested it and it sounds like custom fields are indeed searchable but it's worth getting an "official" answer before investing too much time and effort in this feature). While you do that, I'll start investigating what it would take to allow custom fields when searching/segmenting in StrongGrid.

[Jericho]

To make matters more complicated: it appears that SendGrid is working on a new query language (seems to be SQL based) which you can use when defining segments. but as far as I can tell, you can't use this new segmentation language for searching purposes (not yet anyway).

[NEMA-IT]

I've submitted a support ticket to SendGrid regarding querying against custom fields. Separately, couldn't there be an overload for SearchAsync() that supports a string query? (If the concern is casting the response correctly, then presumably TResult type could be defined when called)

[Jericho]

Thanks for contacting support. While we wait for their response I have started working on overhauling searching/segmenting. My goal is to allow you to continue defining criteria like this (where you pick the field from an enum and therefore you get intellisense, no risk of misspelling the field name, etc.):

var firstNameCriteria = new SearchCriteriaEqual(ContactsFilterField.FirstName, "John");

and also have the flexibility of specifying your criteria like so (you provide the field name as a string) :

var customFieldCriteria = new SearchCriteriaEqual("my_custom_field", "some value");

by the way, a side benefit of allowing you to specify field name as a string is that you will be able to use the "well defined" functions (see #317) that SendGrid supports such as "LENGTH" for example:

var nameLengthCriteria = new SearchCriteriaGreaterThan("LENGTH(last_name)", 10);

I should have a beta available before the end of the week in case you are interested in testing it and providing feedback.

[NEMA-IT]

SendGrid says custom fields supported in search. Their response below:

Jesse (SendGrid)
May 18, 2022, 12:47 PM PDT
Hello,

My name is Jesse and I'll be happy to help today!

You are correct, custom fields are supported as once configured they are seen the same as any other contact field. These are supported in this manner and you should feel confident in proceeding to develop your application!

Please let me know if you have any additional questions or need assistance with anything else.

Thank you,
Jesse || Support Enablement || Twilio SendGrid

[Jericho]

Excellent, thanks for relaying this info. It confirms that their documentation needs to be clarified.

I have published a beta on my MyGet feed. Let me know if you have a chance to test.

[Jericho]

I have published another beta to my MyGet feed which allows you to specify the query as a string when searching for contacts and when creating or updating a segment. Additionally, a "Version 2" query can be specified when creating/updating a segment.

[Jericho]

@NEMA-IT have you had a chance to test the betas I published? Do you have any feedback?

[NEMA-IT]

Sorry, I’m on vacation

[Jericho]

Let me know when you are back and you had a chance to test the new functionality I added for you.

[NEMA-IT]

So sorry, I’m back on vacation! I’ll take a look when I return. Lucius

[Jericho]

@NEMA-IT I would really like some feedback on the improvements.

[NEMA-IT]

Thanks @Jericho! Sorry for the delayed response. I just tried the beta and I was able to search by custom field:

var contact = client.Contacts.SearchAsync($"custom_field = '100000003'").Result?.FirstOrDefault();

[Jericho]

Thanks for confirming.

[Jericho]

FYI, this has been released.

[NEMA-IT]

Awesome! Can’t wait to give it a try.

[Jericho]

Please note: in this latest release, StrongGrid uses SendGrid's new v2 query language by default when creating/updating segments because it was announced that v1 will be made obsolete in December. However, documentation does not indicate that v2 queries can be used when searching for contacts or searching for email activities. Therefore v1 is still being used when searching.