Elasticsearch CRUD .NET provider

The article explains how to use the ElasticsearchCRUD NuGet package. ElasticsearchCRUD is designed so that you can do CRUD operations for any entity and insert, delete, update or select single documents from Elasticsearch. The package only includes basic search or query possibilities.

Code: https://github.com/damienbod/ElasticsearchCRUD
NuGet Package: https://www.nuget.org/packages/ElasticsearchCRUD/
Issues: https://github.com/damienbod/ElasticsearchCRUD/issues

Tutorials:

Part 1: ElasticsearchCRUD introduction
Part 2: MVC application search with simple documents using autocomplete, jQuery and jTable
Part 3: MVC Elasticsearch CRUD with nested documents
Part 4: Data Transfer from MS SQL Server using Entity Framework to Elasticsearch
Part 5: MVC Elasticsearch with child, parent documents
Part 6: MVC application with Entity Framework and Elasticsearch
Part 7: Live Reindex in Elasticsearch
Part 8: CSV export using Elasticsearch and Web API
Part 9: Elasticsearch Parent, Child, Grandchild Documents and Routing
Part 10: Elasticsearch Type mappings with ElasticsearchCRUD
Part 11: Elasticsearch Synonym Analyzer using ElasticsearchCRUD
Part 12: Using Elasticsearch German Analyzer
Part 13: MVC google maps search using Elasticsearch
Part 14: Search Queries and Filters with ElasticsearchCRUD
Part 15: Elasticsearch Bulk Insert
Part 16: Elasticsearch Aggregations With ElasticsearchCRUD
Part 17: Searching Multiple Indices and Types in Elasticsearch
Part 18: MVC searching with Elasticsearch Highlighting
Part 19: Index Warmers with ElasticsearchCRUD

Examples:

Simple autocomplete search
This examples shows how to do a simple search using an MVC application with jQuery autocomplete and Elasticsearch simple documents .

Using ElasticsearchCRUD with NESTED documents (NEST for search)
This example uses Elasticsearch nested documents. The documents can be created, deleted, updated or searched for. The autocomplete search searches the documents as well as the nested objects.

Elasticsearch child, parent documents in a MVC application
This example uses Elasticsearch child/parent documents. All documents are saved inside the same index each with a different type. The child and parent documents are saved on the same shard. It is possible to do CRUD operations with all child documents or search for child/parent documents.

Data Transfer MS SQLServer 2014 With EntityFramework To Elasticsearch
This examples show how to transfer entities to documents in Elasticsearch. The entities are saved as nested documents.

MVC application with Entity Framework and Elasticsearch
This example demonstrates how to use Entity Framework as you primary database and Elasticsearch for the search in an MVC application. The application needs to create, update, delete documents in the search engine when ever Entity Framework changes, deletes or updates an entity.

Live Reindexing in Elasticsearch
This example shows how to do a live reindex in Elasticsearch. There is no downtime. The old index is accessed using an alias. The new index is created from the old index using scan and scroll and a document mapper. Then the alias to switched to access the new index. Then if required, the old index could be removed.

Web API CSV Export using Elasticsearch (scan and scroll)
This example shows how to export data from Elasticsearch ( _search scan and scroll) to Web API as a CSV file (using WebApiContrib.Formatting.Xlsx). The export is displayed in real time using SignalR. The example also provides a SignalR TraceProvider for ElasticsearchCRUD.

ConsoleElasticsearchCrudExample
A basic CRUD example.

ElasticsearchCRUD.Integration.Test
The integration tests shows lots of examples for ElasticsearchCRUD.

Damienbod.AnimalProvider
Example showing mapping configuration.

Why ElasticsearchCRUD?

I wanted a simple way to add, update or delete my application entities into Elasticsearch documents. A single context can work with many entity types and can be adjusted very easily as required, for example to lowercase for all properties, or any specific mapping can be defined. ElasticLINQ does not provide CRUD operations at present, and although you can fulfill almost any requirement with NEST( I also think its a great API), I found it a little bit too complicated to use.

History

Version 2.4.1.1
– support for ASP.NET Core RTM
– fixed, create index with mapping
– updating type for edge_ngram

Version 2.3.1.1-RC2
– support for ASP.NET Core RC2
– updated to elasticsearch 2.3.1
– fixed some bugs from previous versions
– updated Json.NET

Version 2.0.2.1-rc1 2015.11.20
– Support for ASP.NET 5 rc1, dnxcore50, and net451
– Elasticsearch client bug fixes

Version 2.0.0-beta8 2015.11.17
– Support for ASP.NET 5 beta8 dnxcore50 and dnx451

Version 2.0.0 2015.11.16
– Support for Elasticsearch 2.0.0

Version 1.0.29.2 2015.03.16
– bug fix, Id in Hit result should not be an int

Version 1.0.29 2015.02.13
– support search highlighting requests in search model
– support for search rescore
– support for the _warmer API, PUT and DELETE
– support warmers in Create Index
– regexp filter, query max_determinized_states support version 1.4.3
– bug fix, GeoDistanceSort GeoPoint field or array of geo_points is required and not optional

Version 1.0.28 2015.02.10
– Indices Filter, Type Filter, Indices Query
– bug fix Terms Aggregation Result properties not set
– support search for multiple indices and types
– Use type safe distance unit in geo Precision property

Version 1.0.27 2015.02.07
– support for Aggregations
Min Aggregation, Max Aggregation, Sum Aggregation, Avg Aggregation, Stats Aggregation, Extended Stats Aggregation, Value Count Aggregation, Percentiles Aggregation, Percentile Ranks Aggregation, Cardinality Aggregation, Geo Bounds Aggregation, Top hits Aggregation, Scripted Metric Aggregation, Global Aggregation, Filter Aggregation, Filters Aggregation, Filters Named Aggregation, Missing Aggregation, Nested Aggregation, Reverse nested Aggregation, Children Aggregation, Terms Aggregation, Significant Terms Aggregation, Range Aggregation, Date Range Aggregation, Histogram Aggregation, Date Histogram Aggregation, Geo Distance Aggregation, GeoHash grid Aggregation
– Remove string conversion for _id field in bulk insert

Version 1.0.26 2015.01.24
– Support for core type geometrycollection
– Support for nested filter and query
– Support mapping for nested objects
– Added more search queries:
Common Terms Query, Function Score Query, GeoShape Query, Has Child Query, Has Parent Query, Ids Query, More Like This Query, Nested Query, Prefix Query, Query String Query, Simple Query String Query, Regexp Query, Span First Query, Span Multi Term Query, Span Near Query, Span Not Query, Span Or Query, Span Term Query, Top Children Query, Wildcard Query

Version 1.0.25 2015.01.18
– Support for search filters:
And Filter, Bool Filter, Exists Filter, Geo Bounding Box Filter, Geo Distance Filter, Geo Distance Range Filter, Geo Polygon Filter, GeoShape Filter, GeoShape Indexed Filter, Geohash Cell Filter, Has Child Filter, Has Parent Filter, Ids Filter, Limit Filter, Match All Filter, Missing Filter, Not Filter, Or Filter, Prefix Filter, Query Filter, Range Filter, Regexp Filter, Script Filter, Term Filter, Terms Filter
– support for sort
– support for Filter in Alias
– Support for Queries in Scan and Scroll,
– support search objects
– support for basic queries:
Match Query, Multi Match Query, Bool Query, Boosting Query, Constant Score Query, Dis Max Query, Filtered Query, Fuzzy Like This Query, Fuzzy Like This Field Query, Fuzzy Query, Match All Query, Range Query, Term Query, Terms Query

Version 1.0.24 2015.01.05
– Support for geo_point index and mapping
– Support for geo_shape index and mapping
– Supporting the following Geo Shape Types:
point, linestring, polygon, multipoint, multilinestring, multipolygon, envelope, circle

Version 1.0.23 2015.01.02
– Added search highlighting and refactored the hits results
– Support for alias in create index
– Added a missing const icu_tokenizer
– Support for alias routing and filter parameters
– Added _id attribute which can be used instead of the Key Data Annotations attribute

Version 1.0.22 2014.12.15
– support for custom char_filters
– support for custom similarity
– added all DateTime format options

Version 1.0.21 2014.12.09
– support for index Token filters, custom filters
– support for index Tokenizers, custom tokenizers
– support for index Analyzers, custom analyzers
– support for _all and _source mappings
– support for support analysis mappings, settings
– support update index analysis settings
– support for mapping Analyzer
– refactored the search results to conform with the search API hits/hit etc

Version 1.0.20 2014.11.28
– support optimize index
– support close index
– support open index
– support update index settings
– support CreateMapping for existing index
– support specific routing in mappings

Version 1.0.19 2014.11.22
– support for Delete mapping (Index Type)
– support for sync delete index
– bug fix: bool types should not require an [ElasticsearchBoolean] attribute to create mapping
– DeleteDocument, add support/tests for explicit routed documents
– IndexType Mapping fix for grandchild documents serialization

Version 1.0.18 2014.11.21
– support for Elasticsearch Core Types mappings as Attribute definitions
(string, float, double, byte, short, integer,long, date, boolean, binary)
– support for similarity mappings
– support copy_to mappings
– support fields mappings (multi-fields)
– support CreateIndex simple, nested or child/parent document with or without routing
– support for auto init mapping, simple, nested or child/parent document with or without routing

Version 1.0.17 2014.11.14
– Added support for search exists
– Added support for document routing child/parent/grandchild or whatever
– Used routing pro code configuration
– Bug Fix Grandchildren documents are not always saved to the proper shard

Version 1.0.16 2014.11.09
– Added live Reindex support for child parent indexes
– Added support for index exists
– Added support for Alias exists
– Added support for IndexType exists
– Add support for Exists with any URI
– Fixed Console TraceProvider Bug

Version 1.0.14 2014.11.06
– Added live Reindex support
– Added index and indexType mapping utilities
– Added console min trace level to ConsoleTraceLogger
– Fix for scan and scroll implementation

Version 1.0.13 2014.11.05
– Support for alias
– Support for _search scan and scroll

Version 1.0.12 2014.10.31
– Bug fix SearchById

Version 1.0.11 2014.10.29
– Added ClearCache API support
– Added HTTP Head request to test if a document exists DocumentExists
– Added DeleteByQuery API support

Version 1.0.10 2014.10.25
– Support for Elasticsearch Count API
– Return hits/total in search results
– Added code documentation and included in NuGet deployment
– Removed search for child documents per parent

Version 1.0.9 2014.10.22
– Added Get child document for parent Id method to context
– Add SearchById method to context
– Create a Child document possible
– It is possible now to update, or index child documents which belong to a parent document
– Added non-functional tests for child documents, parent documents
– Added Search for child documents of a document
– Added Search with Json String for type T
– Initial Mappings for child parent type relationships
– Using key attribute to identity ids

Version 1.0.8 2014.10.17
– Support Collection of Objects Property in entity as child documents
– Support Single Object Property in entity as child document
– Support Array of Objects Property in entity as child documents
– Add JsonIgnore Property Attribute for Elasticsearch
– Add Configuration/Mapping for child Object definition NESTED or document
– Bug Fix Only the first child object is processed if it is defined in a parent List

Version 1.0.7 2014.10.12
– Bug in 1-n-m-1 mapping for EF entities
– Added diagnostics for HttpClient Request and response
– Added diagnostics for JsonWriter
– Added Entity Framework Data Transfer Tests
– Added Configuration to turn on/off Nested objects IncludeChildObjectsInDocument

Version 1.0.6 2014.10.12
– Add support for Entity Framework dynamic proxy entities
– Add Error Handling when child Entity has a reference to its parent Entity
– support for HashSet properties
– Exception Handling for M-N relationships, and circular relationships detection
– Support for 1 to N Entity Framework entities

Version 1.0.5 2014.10.05
– Support Array of Objects Property in entity as NESTED document
– Support Collection of Objects Property in entity as NESTED documents
– Support Single Object Property in entity as NESTED document
– Support for simple type List properties as NESTED document
– Support for simple type Array properties as NESTED document

Version 1.0.4 2014.10.02
– sync / async methods added for CRUD
– Better Error handling
– Improvement Tracing, added System.Diagnostics Tracing support
– More Tests
– URL bug fix for GetEntity

Version 1.0.3 2014.09.30
– sync save changes added
– default mapping settings changed, lowercase for everything and index = plural

Version 1.0.2 2014.09.27
– support for multiple Entity/Document Types
– Error handling improvements
– Delete index

Version 1.0.1
Support for single entity, initial version.

Example: Setting up a basic CRUD Entity to documents.

Add the ElasticsearchCRUD NuGet package to your application:

Create the entity class which will be saved as a document in Elasticsearch:

public class Skill
{
  public long Id { get; set; }
  public string Name { get; set; }
  public string Description { get; set; }
  public DateTimeOffset Created { get; set; }
  public DateTimeOffset Updated { get; set; }
}

Create your ElasticsearchContext and IElasticsearchMappingResolver. The context requires the Elasticsearch URL. The index is defined in the mapping file along with the type and the property mapping.

IElasticsearchMappingResolver elasticsearchMappingResolver = new ElasticsearchMappingResolver();
	

Now we can add some documents to Elasticsearch

IElasticsearchMappingResolver elasticsearchMappingResolver = new ElasticsearchMappingResolver();
using (var elasticsearchContext = new ElasticsearchContext("http://localhost:9200/", elasticsearchMappingResolver))
{
     // Add some tracing for the console
    elasticsearchContext.TraceProvider = new ConsoleTraceProvider();

    // create some documents
    elasticsearchContext.AddUpdateDocument(TestData.SkillEf, TestData.SkillEf.Id);
    elasticsearchContext.AddUpdateDocument(TestData.SkillOrm, TestData.SkillOrm.Id);
    elasticsearchContext.AddUpdateDocument(TestData.SkillSQLServer, TestData.SkillSQLServer.Id);
    elasticsearchContext.AddUpdateDocument(TestData.SkillGermanWithFunnyLetters, TestData.SkillGermanWithFunnyLetters.Id);
    elasticsearchContext.AddUpdateDocument(TestData.SkillLevel, TestData.SkillLevel.Id);

    var addEntitiesResult = elasticsearchContext.SaveChanges();

    Console.WriteLine(addEntitiesResult.PayloadResult);
    Console.WriteLine(addEntitiesResult.Status);
    Console.WriteLine(addEntitiesResult.Description);
}

Here’s how you can select, update or delete a document

using (var elasticsearchContext = new ElasticsearchContext("http://localhost:9200/", elasticsearchMappingResolver))
{
        // Add some tracing for the console
	elasticsearchContext.TraceProvider = new ConsoleTraceProvider();

	// get a entity and update it, then delete an entity
	Skill singleEntityWithId = elasticsearchContext.GetEntity<Skill>("14");

        // Update a document
	singleEntityWithId.Updated = DateTime.UtcNow;
	elasticsearchContext.AddUpdateDocument(TestData.SkillOrm, TestData.SkillOrm.Id);

        // Delete a document
	elasticsearchContext.DeleteDocument<Skill>("11");
	

	elasticsearchContext.AddUpdateDocument(TestData.SkillEf, TestData.SkillEf.Id);
	var nextResult = elasticsearchContext.SaveChanges();

	Console.WriteLine(nextResult.PayloadResult);
	Console.WriteLine(nextResult.Status);
	Console.WriteLine(nextResult.Description);
}

Here’s what the result looks like:

Delete an Elasticsearch index

You can also delete any index from Elasticsearch. To delete an index, you must set the AllowDeleteForIndex property to true. I added this, so that indexes are not deleted per default settings. Usually, you do not need to delete indexes.

IElasticsearchMappingResolver _elasticsearchMappingResolver = new ElasticsearchMappingResolver();

using (var context = new ElasticsearchContext("http://localhost:9200/", _elasticsearchMappingResolver))
{
  context.AllowDeleteForIndex = true;
  var entityResult = context.DeleteIndex<SkillTestEntityNoIndex>();
  entityResult.Wait();			
}

What if you require special entity/document mappings?

You can implement the ElasticsearchSerializerMapping and define the mappings as required. The following example maps the documents using lower case characters only.

public class SkillElasticsearchMapping : ElasticsearchMapping
{
		/// <summary>
		/// Only required if you have some special mapping or want to remove some properties or use attributes..
		/// </summary>
		/// <param name="entity"></param>
                public override void MapEntityValues(object entity, ElasticsearchCrudJsonWriter elasticsearchCrudJsonWriter, bool beginMappingTree = false)
		{
			Skill skillEntity = entity as Skill;
			MapValue("id", skillEntity.Id, elasticsearchCrudJsonWriter.JsonWriter);
			MapValue("name", skillEntity.Name, elasticsearchCrudJsonWriter.JsonWriter);
			MapValue("description", skillEntity.Description, elasticsearchCrudJsonWriter.JsonWriter);
			MapValue("created", skillEntity.Created.UtcDateTime, elasticsearchCrudJsonWriter.JsonWriter);
			MapValue("updated", skillEntity.Updated.UtcDateTime, elasticsearchCrudJsonWriter.JsonWriter);
		}

		/// <summary>
		/// Use this if you require special mapping for the Elasticsearch document type. For example you could pluralize your Type or set everything to lowercase
		/// </summary>
		/// <param name="type"></param>
		/// <returns></returns>
		public override string GetDocumentType(Type type)
		{
			return "cooltype";
		}

		/// <summary>
		/// Use this if the index is named differently to the default type.Name.ToLower
		/// </summary>
		/// <param name="type"></param>
		/// <returns></returns>
		public override string GetIndexForType(Type type)
		{
			return "skills";
		}
}

And you can add this to the IElasticsearchMappingResolver implementation:

IElasticsearchMappingResolver elasticsearchMappingResolver = new ElasticsearchMappingResolver();

// You only require a mapping if the default settings are not good enough
elasticsearchMappingResolver.AddElasticSearchMappingForEntityType(typeof(Skill), new SkillElasticsearchMapping());
var elasticsearchContext = new ElasticsearchContext("http://localhost:9200/", elasticsearchMappingResolver);

You could also map the properties like this:

/// <summary>
/// Here you can do any type of entity mapping
/// </summary>
/// 
public override void MapEntityValues(object entity, ElasticsearchCrudJsonWriter elasticsearchCrudJsonWriter, bool beginMappingTree = false)
{
  var propertyInfo = entity.GetType().GetProperties();
  foreach (var prop in propertyInfo)
  {
     MapValue(prop.Name, prop.GetValue(entity),  elasticsearchCrudJsonWriter.JsonWriter);
  }
}

Tracing:

You can also trace the requests, responses using your favorite logger. All you need to do is to implement the ITraceProvider and set the TraceProvider property in the context.

elasticsearchContext.TraceProvider = new ConsoleTraceProvider();

Or you can trace using System.Diagnostics:

elasticsearchContext.TraceProvider = new TraceProvider("tracingExample");

And here’s an example of the System.Diagnostics configuration:

<system.diagnostics>
    <trace autoflush="true"/>
    <sources>
      <source name="tracingExample"
              switchName="tracingSwitch"
              switchType="System.Diagnostics.SourceSwitch" >
        <listeners>
          <clear/>        
          <add name="textwriterListener"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="application.log"
            traceOutputOptions="ProcessId, DateTime, Callstack" />
        </listeners>
      </source>
    </sources>
    <switches>
      <add name="tracingSwitch" value="Verbose" />
    </switches>
</system.diagnostics>

Implementation example of the ITraceProvider:

public class ConsoleTraceProvider : ITraceProvider
{
	public void Trace(TraceEventType level, string message, params object[] args)
	{
		SetForegroundColor(level);
		Console.WriteLine(level + ": " + message, args);
	}

	public void Trace(TraceEventType level, Exception ex, string message, params object[] args)
	{
		SetForegroundColor(level);
		Console.WriteLine(level + ": " + ex.Message + ex.InnerException + message, args);
	}

	private void SetForegroundColor(TraceEventType level)
	{
		switch (level)
		{
			case TraceEventType.Critical:
			{
				Console.ForegroundColor = ConsoleColor.Red;
				break;
			}
			case TraceEventType.Error:
			{
				Console.ForegroundColor = ConsoleColor.Magenta;
				break;
			}
			case TraceEventType.Warning:
			{
				Console.ForegroundColor = ConsoleColor.Yellow;
				break;
			}
			case TraceEventType.Verbose:
			{
				Console.ForegroundColor = ConsoleColor.Gray;
				break;
			}
			default:
			{
				Console.ForegroundColor = ConsoleColor.White;
				break;
			}
		}
	}
}

What’s missing, could be added:

I’m still deciding on what features I should add in future releases.

Here’s my list of issues. If you would like to add some requests, features, please add them here:
https://github.com/damienbod/ElasticsearchCRUD/issues

Queries and search functions
This is one of the main reasons for using Elasticsearch. Other packages have implemented these features already.

Links:

http://www.elasticsearch.org/blog/introducing-elasticsearch-net-nest-1-0-0-beta1/

https://github.com/CenturyLinkCloud/ElasticLINQ

http://www.elasticsearch.org/

https://github.com/elasticsearch/elasticsearch-net

http://nest.azurewebsites.net/

Tags: .NET, CRUD, Elasticsearch, ElasticsearchCRUD, HttpClient, Lucene, NoSQL