Searching Multiple Indices and Types in Elasticsearch

This article shows how to do searches across multiple indices and types in Elasticsearch using ElasticsearchCRUD. Elasticsearch provides an Indices Filter, a Type Filter, and an Indices Query which can be used when working with multiple indices and types. Aggregations searches and Count requests can also be executed using multiple indices, types.

Code: https://github.com/damienbod/ElasticsearchParentChildGrandChild

Other Tutorials:

Part 1: ElasticsearchCRUD introduction
Part 2: MVC application search with simple documents using autocomplete, jQuery and jTable
Part 3: MVC Elasticsearch CRUD with nested documents
Part 4: Data Transfer from MS SQL Server using Entity Framework to Elasticsearch
Part 5: MVC Elasticsearch with child, parent documents
Part 6: MVC application with Entity Framework and Elasticsearch
Part 7: Live Reindex in Elasticsearch
Part 8: CSV export using Elasticsearch and Web API
Part 9: Elasticsearch Parent, Child, Grandchild Documents and Routing
Part 10: Elasticsearch Type mappings with ElasticsearchCRUD
Part 11: Elasticsearch Synonym Analyzer using ElasticsearchCRUD
Part 12: Using Elasticsearch German Analyzer
Part 13: MVC google maps search using Elasticsearch
Part 14: Search Queries and Filters with ElasticsearchCRUD
Part 15: Elasticsearch Bulk Insert
Part 16: Elasticsearch Aggregations With ElasticsearchCRUD
Part 17: Searching Multiple Indices and Types in Elasticsearch
Part 18: MVC searching with Elasticsearch Highlighting
Part 19: Index Warmers with ElasticsearchCRUD

The example uses parent, child, grandchild mappings in a single index using the parent _Id for the routing. This ensures that the grandchild document and the parent document are saved to the same shard. Once the mapping has been created, 3 documents are added in a bulk request. The resulting index has 3 type mappings in the league index.

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 528

{
	"leagues": {
		"mappings": {
			"team": {
				"_parent": {
					"type": "leaguecup"
				},
				"_routing": {
					"required": true
				},
				"properties": {
					"id": {
						"type": "long"
					},
					"name": {
						"type": "string"
					},
					"stadium": {
						"type": "string"
					}
				}
			},
			"leaguecup": {
				"properties": {
					"description": {
						"type": "string"
					},
					"id": {
						"type": "long"
					},
					"name": {
						"type": "string"
					}
				}
			},
			"player": {
				"_parent": {
					"type": "team"
				},
				"_routing": {
					"required": true
				},
				"properties": {
					"age": {
						"type": "integer"
					},
					"assists": {
						"type": "integer"
					},
					"goals": {
						"type": "integer"
					},
					"id": {
						"type": "long"
					},
					"name": {
						"type": "string"
					},
					"position": {
						"type": "string"
					}
				}
			}
		}
	}
}

Searching across n-types in a single index

To seach a single index which contains multiple types, a new ElasticsearchMapping needs to be created. This maps an object type to the required index.

public class GlobalLeaguesElasticsearchMapping : ElasticsearchMapping
{
	public override string GetIndexForType(Type type)
	{
		return "leagues";
	}

	public override string GetDocumentType(Type type)
	{
		return "";
	}
}

This is then added to the ElasticsearchContext which is used for the search.

  ElasticsearchMappingResolver.AddElasticSearchMappingForEntityType(typeof (object),
				new GlobalLeaguesElasticsearchMapping());

Now a search using the object type will search for any document type inside the leagues index. The Routing is used so that we search on the correct shard.

using (var context = new ElasticsearchContext(ConnectionString, Config))
{
	context.TraceProvider = new ConsoleTraceProvider();
	var result = context.Search<object>(new Search(), 
		new SearchUrlParameters
		{
			Routing = leagueId.ToString(CultureInfo.InvariantCulture)
		});

	Console.WriteLine("Found {0}, Expected 3", result.PayloadResult.Hits.Total);
}

Searching across all indices

It is also possible to search all indices and all types in Elasticsearch. For this, the GlobalElasticsearchMapping utility class can be used. This is added to a new context for the object type.

_elasticsearchMappingResolver.AddElasticSearchMappingForEntityType(typeof(object), new GlobalElasticsearchMapping());

This can then be used to count all documents on the server.

using (var context = new ElasticsearchContext(_connectionString, _elasticsearchMappingResolver))
{
	long countValue = context.Count<object>();
	Console.WriteLine("Global Count for all indices: {0}, Expected at least 3", countValue);
}

Or it can be used to search using all indices and types.

using (var context = new ElasticsearchContext(_connectionString, _elasticsearchMappingResolver))
{
	var result = context.Search<object>(new Search()
	{
		Query = new Query(new MatchAllQuery())
	});
}

Using the Data from a global search

The results of a multiple type search can be evaluated using the GetSourceFromJToken method.
Each object in the hit is returned as a JToken because the returned type was defined as an object. The hit.TypeInIndex can be used to find out what kind of document it is. This can then be parsed back to a class.

foreach (var hit in result.PayloadResult.Hits.HitsResult)
{
	string type = hit.TypeInIndex;
	if (type == "player")
	{
		var player = hit.GetSourceFromJToken<Player>();
		Console.WriteLine("Found a player: {0}, {1}", player.Name, player.Id);
	}
	else if (type == "team")
	{
		var team = hit.GetSourceFromJToken<Team>();
		Console.WriteLine("Found a team: {0}, {1}", team.Name, team.Id);
	}
}

Searching across n-types with an indices filter and a type filter

The search could also filter in or filter out the different types using the IndicesFilter and the TypeFilter. The following code snippet filters for the team type and the player type using the OrFilter and removes the leaguecup type using the NoMatchFilter property. If the NoMatchFilterNone is set, all non-defined types are not included in the results.

var search = new Search
{
	Filter = new Filter(
		new IndicesFilter(
			new List<string> {"leagues"},
			new OrFilter(
				new List<IFilter>
				{
					new TypeFilter("team"),
					new TypeFilter("player")
				}
			)
		)
		{
			NoMatchFilter = new TypeFilter("leaguecup")
		}
	)
};

The search can be sent to Elasticsearch as follows:

using (var context = new ElasticsearchContext(_connectionString, _elasticsearchMappingResolver))
{
	context.TraceProvider = new ConsoleTraceProvider();
	var result = context.Search<object>(search,
		new SearchUrlParameters
		{
			Routing = leagueId.ToString(CultureInfo.InvariantCulture)
		});

	Console.WriteLine("Found {0}, Expected 2", result.PayloadResult.Hits.Total);
}

This is sent to Elasticsearch as a HTTP Post request with the correct Json body.

POST http://localhost:9200/_search?&routing=1 HTTP/1.1
Content-Type: application/json
Host: localhost:9200
Content-Length: 179
Expect: 100-continue

{
	"filter": {
		"indices": {
			"indices": ["leagues"],
			"filter": {
				"or": {
					"filters": [{
						"type": {
							"value": "team"
						}
					},
					{
						"type": {
							"value": "player"
						}
					}]
				}
			},
			"no_match_filter": {
				"type": {
					"value": "leaguecup"
				}
			}
		}
	}
}

At present it is not possible to search n-indices in a single search request using ElasticsearchCRUD. This may be added in a later version. The Indices filter can be used instead.

Links:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/multi-index.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/multi-index-multi-type.html

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-search.html

Tags: child parent search, Elasticsearch, ElasticsearchCRUD, Indices Filter, Lucene, multiple indices, Type Filter