This is looking great.

IIUC:

• ISearchProvider would be implemented by different search backends (Solr, ES, Postgres, etc).
• ISearchFeature could be implemented by:
  • a scheming plugin that would translate entries in the schema file to changes in the schema (e.g. to index a particular custom field)
  • site plugins that wanted to customize the search in a way not covered by the scheming search functionality
  • feature plugins that add extra functionality with custom field types, params etc (e.g. ckanext-spatial)

It might be useful to test these proposed interfaces against real use cases to see if they cover them:

1. spatial search in ckanext-spatial

This currently works leveraging custom Solr field types. There are actually two types of search with different Solr field types, but the mechanics would be the same so let's focus on one. To provide spatial search:

1. A new location_rpt field type is added to the Solr schema field definitions
2. A new spatial_geom field of type location_rpt is added to the Solr schema
3. At index time (before_dataset_index()), spatial information is read (by default from a custom field/extra named spatial, but it could come from elsewhere), validated and stored in the spatial_geom field in the expected format
4. At search time (before_dataset_search()), user input is captured via custom parameters (ext_bbox={minx},{miny},{maxx},{maxy}), translated to something Solr understands ({{!field f=spatial_geom}}Intersects(ENVELOPE({minx}, {maxx}, {maxy}, {miny}))) and added to the fq_list param

An hypothetical Elastic Search powered spatial search would follow the same mechanism, but in that case the spatial field type is already supported out of the box (so no point 1).

So to replicate the current Solr based query:

• Point 1 sounds more like an ISearchFeature concern. Would that be a separate setup command in the feature plugin?
• Once that is in place, point 2 would be done via ISearchFeature.search_schema. Would the extra search params (bbox) be defined here as well or in a separate mehtod?
• ISearchFeature.entity_type would return package (or maybe dataset?? 🥲 )
• Point 3 would be replaced by ISearchFeature.format_search_data() (assuming this gets the full validated dict)
• Point 4 would remain in their current hook as hooks like before_dataset_search would still be supported right? Or do we want to move them from IPackageController to ISearchFeature? I suppose we need to do if we want to support customizing the search across different entities. Of course the way extensions custom parameters are managed will change (no more ext_ nonsense) but the principle would be the same
• ISearchFeature.existing_record_ids() and ISearchFeature.fetch_records() would not be implemented as we are handling just datasets so core will take care of it right?

2. Indexing and querying pages

Ckanext-sitesearch supports indexing page contents if ckanext-pages is installed, and performing searches. There is no additional setup other than installing and configuring ckanext-pages, but the indexing is done via a separate command (ckan sitesearch rebuild pages). When rebuilding the index, all page models are retrieved from the database

So in this case:

• ISearchFeature.entity_type would return a custom pages
• ISearchFeature.search_schema would return a whole new schema for the entity type pages (title, description, modified date etc, and a catch all field)
• ISearchFeature.existing_record_ids() and ISearchFeature.fetch_records() need to be implemented using the Page models and dictization to return whatever ids or objects needed to index them.

There are still some blind spots for me but overall sounds like a good direction to follow!

Trying to visualize a bit more how the proposed interfaces would play together. In this case testing how the spatial search might look like:

Querying

# ckan/logic/action/get.py (or an extension for now)
def search(context, data_dict):

    # Check auth


    backend = get_search_backend()


    # Validate data_dict. Any key not in the standard schema is moved to
    # additional_params
    # Q: Do we validate just the common interface params here or get additional schema
    #    entries from the search feature plugin to validate additional params like `bbox`?
    schema = default_search_query_schema()
    for plugin in PluginImplementations(ISearchFeature):
        plugin.search_schema(schema)

    # Call query method of the relevant backend

    query_dict = { 
        ... 
    }
    result = backend.search_query(**query_dict)

    return result


# ckanext-search-solr
class SolrSearchProvider(ISearchProvider):

    def search_query(self, query, filters,...):

        for plugin in PluginImplementations(ISearchFeature):

            # Search feature plugins will probably want to modify the query, filters etc parameters
            # Q: what's the best way of passing them to the search feature plugins? 
            #    wrapped in a query_dict? modify in place?
            plugin.before_search(query, filters, sort, lang, additional_paramas, ...)

        # Construct actual Solr query

        # Call Solr Client

        # Parse results to adapt them to the common interface format

        return results

    

# ckanext-spatial
class SolrSpatialSearch(ISearchFeature):

    def before_search(self, query, filters, etc):

        # Check additional_params for a `bbox` param

        # Add a new filter based on that
        # Note: the filters format suggestion was a field_name: field_value type, but this doesn't fit 
        # this convention as it defines the whole value for fq, so I'm suggesting a convention to add `fq` filters
        # directly
        filters["fq"] = "{{!field f=spatial_geom}}Intersects(ENVELOPE({minx}, {maxx}, {maxy}, {miny}))"

Indexing

# This could be a CLI command or an action
def index(entity_type, entity_ids)

    # Check auth

    # Get whatever data needs to be indexed, e.g. a validated data_dict for datasets or ISearchEntity.fetch_records()
    # for custom entities
    data_dict = get_search_data_for_entity()

    # Call the index method of the relevant backend
    backend = get_search_backend()

    backend.index_search_record(entity_type, id_, data_dict)


# ckanext-search-solr
class SolrSearchProvider(ISearchProvider):

    def index_search_record(entity_type, id_, data_dict):
        for plugin in PluginImplementations(ISearchFeature):
            plugin.before_index(entity_type, id_, data_dict)


# ckanext-spatial
class SolrSpatialSearch(ISearchFeature):
    def before_index(entity_type, id_, data_dict):

        # Check if entity is dataset, and if it has a `spatial` geojson field

        # if so, add the relevant field to the data to index
        data_dict["spatial_field"] = wkt_version_of_geometry

Instead of a before_search that mutates the parameters can we do something like:

    processed = {}
    for plugin in PluginImplementations(ISearchFeature):
        processed.update(plugin.process_additional_params(additional_params))

then the processed dict could be passed through to search containing trusted values (not arbitrary user-provided ones) and we don't have to mutate the parameters passed.

Same sort of thing for indexing. A before_index that mutates parameters feels like a poor design because it mixes inputs and outputs and makes it harder for plugins to work with one another.

Let's keep the validated version of the entity in one dict and the record to be indexed in another. Something kind of like:

    def index_search_record(entity_type, id_, data_dict):
        solr_record = {}
        for plugin in PluginImplementations(ISearchFeature):
            solr_record.update(plugin.augment_search_record(entity_type, id_, data_dict))
        # the rest of solr_record populated here

@sagargg @gavram do you feel like you have enough to create a small Proof of Concept?

I would focus on small, focused POCs to not complicate things and then tie it all together progressively:

1. A simple search action with a basic Solr provider implementing ISearchProvider, that supports indexing datasets and another core entity, plus returns query results. We can use this to polish the external API interface and the return format
2. A basic search feature plugin, can be the spatial search described above. This would test how to index custom fields and how to provide additional parameters to modify the standard search
3. Same as the two above but with Elastic search
4. Indexing and querying of custom entities (like pages)

How does this sound?

BTW I've put together the various snippets discussed in the discussion in a gist so it's easy to follow:

https://gist.github.com/amercader/2f0f54e1fcf33bad5b7d8b10aa2a10f8

Going forward maybe we need to re-think what we send to index and what actually gets indexed.

Right now we pass the following to the function that indexes datasets in Solr:

• The un-validated data_dict (default schema). These are the fields that get indexed based on the Solr schema, i.e. they get actually passed to pysolr for indexing
• This same un-validated data_dict gets indexed as a data_dict field
• The validated data_dict (custom schema) gets indexed as a validated_data_dict field

Setting aside the issue of space and whether the indexed data_dict is needed at all, it seems wrong that what we are indexing (and what gets passed to IPackageController.before_dataset_index() is the un-validated data_dict. Users might have validators in place that affect the field values, and when thinking about your custom schema and what fields get indexed and how it makes more sense to think about it in terms of how you define your custom fields in scheming than how CKAN internally stores them (e.g. as extras, dumped json lists, etc)

What if the new search:

• Received the validated_data_dict, and used that to index the relevant fields based on the search schema (which we want to integrate in the schema definition anyway)
• Only stored the validated_data_dict in the search index, to keep the current caching mechanism in place

If a site really needs the un-validated data_dict stored the can hook into before_dataset_index and store them themselves but for the majority of sites it seems like the validated version would suffice in most cases.

@wardi does this make sense?

@amercader I like this suggested syntax, it's composable, extendable and succinct.

We can compose or clauses and operations, e.g. an "or" clause combined with ranges:

"filters": {
    "field3": [
        {"gte": 9000, "lt": 10000},
        {"gte": 15000, "lt": 19000}
    ]
}

We can extend it with new operations using plugins:

"filters": {
    "published_date": {"days_old": 30}
}

Strings / numbers / bools / null are short-form for {"eq": the_value}. We must use an "eq" operation when the value for comparison may be a list or a dict, e.g.

"filters": {
    "json_field": {"eq"': {"a": "b"}},         # "a" is not an operation
    "text_array_field": {"eq": ["a", "b"]}    # "[]" is not an or clause
}

tangent 1: filter_operations

Operations appearing only after column names is a limitation, though. There isn't a way to express operations that cover multiple columns like "admin = true or organization = public"

We also don't have a way to express pagination based on the sort order like "records where (year, month) > (2024, 5)" where year and month are separate columns.

What could these examples look like? We could have a separate argument for filters where the column name must be a parameter, e.g. "admin = true or organization = public" could be:

"filter_operations": [
    {"eq": ["admin", true]},
    {"eq": ["organization", "public"]}
]

"records where (year, month) > (2024, 5)" could be:

"filter_operations": {
    "gt": [["year", "month"], [2024, 5]]
}

• "filters" is column — operation — value
• "filter_operations" is operation — [column(s), value(s)]

tangent 2: advanced sorting

That last year, month example almost works for pagination, but if the sort order combines ascending and descending columns we might be better off defining "after" and "before" operations that automatically apply to the sort order e.g.

"sort": ["row asc", "seat desc"],
"filter_operations": {
    "after": [53, "R"]
}

where "after" would "know" that it's looking for records [53, "Q"], [53, "P"], ..., [53, "A"], [54, "Z"], ...

Or do we drop support for sorting multiple columns in different orders? I'm not sure how to represent that last example in the general case with a postgresql backend, it's likely not possible in general for all search backends.

If we require that sorting/pagination only be done on one unique indexed column at a time then we avoid this problem. That shouldn't be a problem for dataset search except that we need to ensure things that might not be unique (e.g. date_updated) are combined with something that is unique like the package id behind the scenes.

tangent 3: language support

We also need to think about language support.

It's reasonable for a user to have multiple language versions of dataset metadata, then e.g. ask for the datasets to be returned sorted by "Spanish title with es_ES collation". If the backend remembers the collation type of each field then this shouldn't affect the search interface, but it will be important to be able to declare fields to be indexed with specific collation types so that users don't get the wrong results.

For general search (users passing a value to the q parameter) of multilingual metadata we don't know the language the user would like to search against. We could decide to use a trigram index for any q search or we have to generate multiple full text indexes, one for each language stemming rule supported by the site and users will need to pass the language they're using for their q parameter.

I think my tangent 1 and tangent 2 are solved by having this $or operator.

We can handle the case of a field starting with $ by saying that users must double the $ if it appears as the first character in a field name. This way sites can add their own custom top level operators too. e.g. if someone needs to be able to compare fields in the same record:

"filters": {
    "$eq_fields": ["created_date", "modified_date"]
}

This also opens up the possibility for the datastore to allow custom advanced queries on large datasets (e.g. to support a visualization or summary) without needing to open up access to the generic datastore_search_sql.

So our filters dict keys would take the form:

"field_name": { "operator": <parameters>, ... }

"$$field_with_a_$_at_the_start": { "operator": <parameters>, ... }

or

"$top_level_operator": <parameters>

With these short forms:

"field_name": <not list or dict>  =>  "field_name": { "eq": <not list or dict> }

"field_name": <list of dicts> => "$or": [ { "field_name": <dict 1>, ..., "field_name": <dict n> } ]

"field_name": <list of non-dicts>  =>  "field_name": { "in": <list of non-dicts> }

and if the filters value itself is a list instead of a dict:

[ <filter1>, <filter2> ]  =>  { "$or": [ <filter1>, <filter2> ] }

Filters that are not a dict or list are invalid, and the validity of parameters depend on the operator.

@wardi I'm working on an initial implementation of the filters validation. I think it would be great if search providers got the "expanded" form of the provided filters, i.e. with all the different shorthands expanded to the standard filter form so they only have to worry about that particular part of the syntax.

So at the CKAN core level (e.g. in the datastore_search or the new search actions) filters will be validated and expanded and then passed to the backends query function.

Do you think that the expansion should take place in the validator function itself or should be done in two steps (first validation, then expansion)?

Also I'm wondering if it's worth that we expand all filters to $or and $and constructs so we reduce ambiguity for providers and they only have to worry about one way of encoding the filters, even if it's the more verbose one.

So even if users pass {"field1": "value1", "field2": "value2"} to the API, providers will always receive something like

{
    "$and": [
        {"field1": {"eq": "value1"}},
        {"field2": {"eq": "value2"}},
    ]
}
   

@wardi (and anyone interested!) can you check this and see if I missed something?

https://hackmd.io/@amercader/Hk2PqOPxgx

We would be forced to namespace the different entity type results if not all the entities will be part of the same index, right? When they're in separate indexes there's no way to order the results against each other.

But then what does it mean to paginate the results? We would be getting the next page of "limit" entities for each entity type that still have results?

That doesn't sound very intuitive or user-friendly.

Instead we could choose limit this API to only entity types that are part of the same index. It would be up to the site to configure the entity types covered, potentially even including datastore rows or unstructured documents.

The entity type counts should be available as one of the facets and filters available, even though they aren't "real" fields in the entities themselves.

For results I can think of three ways to indicate the entity types but I don't love any of them:

1. a separate list of result entity types:

{
  "results": [
    <validated_dataset_dict 1>,
    <validated_org_dict 1>,
    <validated_dataset_dict 2>,
    <validated_group_dict 1>,
    <validated_org_dict 2>
  ],
  "result_entity_types": [
    "package", "organization", "package", "group", "organization"
  ]
}

2. embedded in the entity dicts returned:

{
  "results": [
    {
      "entity_type": "package",
      <contents of validated_dataset_dict 1>
    },
    {
      "entity_type": "organization",
      <contents of validated_org_dict 1>
    },
    …
  ],
}

3. in envelopes around each result:

{
  "results": [
    {
      "entity_type": "package",
      "result": <validated_dataset_dict 1>
    },
    {
      "entity_type": "organization",
      "result": <validated_org_dict 1>
    },
    …
  ],
}

(1) means not modifying the results list which is nice, but requires iterating over parallel lists when we care about the types
(2) is the cleanest but means returning entities that are modified from what was stored, this might cause a problem with round-tripping unless we ignore entity_type in our validation schemas. We could use a prefix like "@entity_type" to hint that these values aren't part of the entity data itself.
(3) is ugly and extra work to get the actual results, but it does give us a place for other pseudo-fields like "score" when returning results based on how close a text match was

For encouraging index-based pagination the "start" parameter could also accept a list instead of a typical integer offset. Each item in the list will correspond to one of the sort field values for the item at the end of the previous page.

e.g. if the request included

"sort": ["published_date", "author"]

Then a valid value for "start" could be:

"start": ["2021-01-20", "Smith, J"]

The search API would convert this to filters before sending it to the back end to execute, in this case the "start" parameter would be converted to filters something like: published_date > 2021-01-20 OR (published_date = 2021-01-20 AND author > "Smith, J")

As part of the search response we can include a "next_page_start" value that is a pre-populated index-based pagination "start" value for the next page of results.

It's weird to be returning overlapping values in facets and search_facets, that might have been about maintaining backwards compatibility at some point but we could drop one of them for this new API

Can the "title" and "display name" values for facets reliably be stored by all search back ends? There's seemingly no allowance for multiple language versions of each. Maybe we're better off keeping the simpler facets response instead of the search_facets one?