MVC CRUD with Elasticsearch NESTED documents

This article demonstrates how to create Elasticsearch documents with NESTED objects using jQuery, jTable and moment.js in the MVC Razor View, and using MVC, ElasticsearchCRUD and Elasticsearch as the backend. The SearchController provides all CRUD Actions so you can experiment with the 1 to n entities, or the Elasticsearch nested documents. Not all Elasticsearch clients provide a way to view documents with Nested objects.

Code: https://github.com/damienbod/WebSearchWithElasticsearchNestedDocuments

Other tutorials:

Part 1: ElasticsearchCRUD introduction
Part 2: MVC application search with simple documents using autocomplete, jQuery and jTable
Part 3: MVC Elasticsearch CRUD with nested documents
Part 4: Data Transfer from MS SQL Server using Entity Framework to Elasticsearch
Part 5: MVC Elasticsearch with child, parent documents
Part 6: MVC application with Entity Framework and Elasticsearch
Part 7: Live Reindex in Elasticsearch
Part 8: CSV export using Elasticsearch and Web API
Part 9: Elasticsearch Parent, Child, Grandchild Documents and Routing
Part 10: Elasticsearch Type mappings with ElasticsearchCRUD
Part 11: Elasticsearch Synonym Analyzer using ElasticsearchCRUD
Part 12: Using Elasticsearch German Analyzer
Part 13: MVC google maps search using Elasticsearch
Part 14: Search Queries and Filters with ElasticsearchCRUD
Part 15: Elasticsearch Bulk Insert
Part 16: Elasticsearch Aggregations With ElasticsearchCRUD
Part 17: Searching Multiple Indices and Types in Elasticsearch
Part 18: MVC searching with Elasticsearch Highlighting
Part 19: Index Warmers with ElasticsearchCRUD

Model with 1 to n Entities

The model used to interact with Elasticsearch has a 1 to n relationship. The class SkillWithListOfDetails has a list of SkillDetail objects. These classes will be saved to Elasticsearch with the SkillDetail list as a nested object. This child object can be search just like any other property in the parent object, SkillWithListOfDetails.

public class SkillWithListOfDetails
{
  [Required]
  [Range(1, long.MaxValue)]
  public long Id { get; set; }
		
  [Required]
  public string Name { get; set; }
	
  [Required]
  public string Description { get; set; }
  public DateTimeOffset Created { get; set; }
  public DateTimeOffset Updated { get; set; }

  public List<SkillDetail> SkillDetails { get; set; }
}

The SkillDetail is used as the child class. No Id is required for the parent foreign key, because the child is NESTED when stored to Elasticsearch.

public class SkillDetail
{
  [Required]
  [Range(1, long.MaxValue)]
  public long Id { get; set; }
		
  [Required]
  public string SkillLevel { get; set; }
		
  [Required]
  public string Details { get; set; }
  public DateTimeOffset Created { get; set; }
  public DateTimeOffset Updated { get; set; }
}

Controller Create, Create with ElasticsearchCRUD

The Create Elasticsearch functionality is implemented using EleashseachCRUD. To use download ElasticsearchCRUD using NuGet:

This uses the default IElasticSearchMappingResolver which saves the index pluralized, sets the type to the class name without the namespace and also saves all properties to lower case.

An id is required and is not auto generated. Auto-Generated Ids is not supported by ElasticseachCRUD. Usually Elasticsearch is not the primary persistence and the entities which will be saved to the search engine as documents already have an id. If an Id needs to be created, you could Auto-Generate a new random Guid.

private const string ConnectionString = "http://localhost:9200/";
private readonly IElasticsearchMappingResolver _elasticsearchMappingResolver = new ElasticsearchMappingResolver();

public void AddUpdateEntity(SkillWithListOfDetails skillWithListOfDetails)
{
	using (var context = new ElasticsearchContext(ConnectionString, _elasticsearchMappingResolver))
	{
		context.AddUpdateDocument(skillWithListOfDetails, skillWithListOfDetails.Id);
		context.SaveChanges();
	}
}

The provider can then be used in the SearchController. The action method accepts the model and the string containing the child list of SkillDetail entities. This createSkillDetailsList string property is a json string created from the view using javascript and jTable.

[HttpPost]
[Route("Index")]
public ActionResult Index(SkillWithListOfDetails model, string createSkillDetailsList)
{
	if (ModelState.IsValid)
	{
		model.Created = DateTime.UtcNow;
		model.Updated = DateTime.UtcNow;

		model.SkillDetails =
			JsonConvert.DeserializeObject(createSkillDetailsList, typeof(List<SkillDetail>)) as List<SkillDetail>;

		_searchProvider.AddUpdateDocument(model);
		return Redirect("Search/Index");
	}

	return View("Index", model);
}

The create view is a simple MVC razor partial view. This view uses the SkillWithListOfDetails model and sends a simple form to the MVC controller action. The input button calls a javascript function which gets all the SkillDetail rows from the jTable create table and adds the to the input hidden item. Then it executes the submti()

@model WebSearchWithElasticsearchNestedDocuments.Search.SkillWithListOfDetails
<div id="createForm">
    @using (Html.BeginForm("Index", "Search"))
    {
        @Html.ValidationSummary(true)

        <fieldset class="form">
            <legend>CREATE a new document in the search engine</legend>
            <table width="800">
                <tr>
                    <th></th>
                    <th></th>
                </tr>
                <tr>
                    <td>
                        @Html.Label("Id:")
                    </td>
                    <td>
                        @Html.EditorFor(model => model.Id)
                        @Html.ValidationMessageFor(model => model.Id)
                    </td>
                </tr>
                <tr>
                    <td>
                        @Html.Label("Name:")
                    </td>
                    <td>
                        @Html.EditorFor(model => model.Name)
                        @Html.ValidationMessageFor(model => model.Name)
                    </td>
                </tr>
                <tr>
                    <td>
                        @Html.Label("Description:")
                    </td>
                    <td>
                        @Html.EditorFor(model => model.Description)
                        @Html.ValidationMessageFor(model => model.Description)
                    </td>
                </tr>
                <tr>
                    <td colspan="2">
                        <div id="createtableskilldetails" />
                        <input id="createSkillDetailsList" name="createSkillDetailsList" type="hidden" />
                    </td>
                </tr>
                <tr>
                    <td>
                        <br />
                        <input type="button" onclick="SumbitCreateForm()" value="Add Skill" style="width:200px" />
                    </td>
                    <td></td>
                </tr>
            </table>
        </fieldset>
    }

</div>

The SearchCreate is a MVC PartialView. This is used in the Index View. The index view contains all the javascript implementation. This should be implements in a separate js file and bundled. The javascript code uses 3 js libraries, moment.js and jTable and jQuery (with UI).

@model WebSearchWithElasticsearchNestedDocuments.Search.SkillWithListOfDetails

<fieldset class="form">
    <legend>SEARCH for a document in the search engine</legend>
    <table width="500">
        <tr>
            <th></th>
        </tr>
        <tr>
            <td>
                <label for="autocomplete">Search: </label>
            </td>
        </tr>
        <tr>
            <td>
                <input id="autocomplete" type="text" style="width:500px" />
            </td>
        </tr>
    </table>
</fieldset>

@section scripts
{
    <link href="~/Scripts/jtable/themes/jqueryui/jtable_jqueryui.min.css" rel="stylesheet" />
    <script src="~/Scripts/jtable/jquery.jtable.min.js"></script>
    <script src="~/Scripts/moment.min.js"></script>
    <script type="text/javascript">

        $(document).ready(function () {

            var updateResults = [];
            $("input#autocomplete").autocomplete({
                source: function(request, response) {
                    $.ajax({
                        url: "http://localhost:50227/Search/search",
                        dataType: "json",
                        data: {
                            term: request.term,
                        },
                        success: function(data) {
                            var itemArray = new Array();
                            for (i = 0; i < data.length; i++) {
                                itemArray[i] = { label: data[i].Name, value: data[i].Name, data: data[i] }
                            }

                            console.log(itemArray);
                            response(itemArray);
                        },
                        error: function(data, type) {
                            console.log(type);
                        }
                    });
                },
                select: function(event, ui) {
                    $("#spanupdateId").text(ui.item.data.Id);
                    $("#spanupdateCreated").text(moment(ui.item.data.Created).format('DD/MM/YYYY HH:mm:ss'));
                    $("#spanupdateUpdated").text(moment(ui.item.data.Updated).format('DD/MM/YYYY HH:mm:ss'));
                    $("#updateName").text(ui.item.data.Name);
                    $("#updateDescription").text(ui.item.data.Description);
                    $("#updateName").val(ui.item.data.Name);
                    $("#updateDescription").val(ui.item.data.Description);

                    if (ui.item.data.SkillDetails) {
                        updateResults = ui.item.data.SkillDetails;
                    }
                    
                    $('#updatetableskilldetails').jtable('load');

                    $("#updateId").val(ui.item.data.Id);
                    $("#updateCreated").val(ui.item.data.Created);
                    $("#updateUpdated").val(ui.item.data.Updated);

                    $("#spandeleteId").text(ui.item.data.Id);
                    $("#deleteId").val(ui.item.data.Id);
                    $("#deleteName").text(ui.item.data.Name);

                    console.log(ui.item);
                }
            });
       
            $('#updatetableskilldetails').jtable({
                title: 'Skill Details',
                paging: false,
                pageSize: 5,
                sorting: true,
                multiSorting: true,
                defaultSorting: 'Name asc',
                actions: {
                    listAction: function (postData) {
                        console.log("Loading from custom function...");
                        return {
                            "Result": "OK",
                            "Records": updateResults 
                        };
                    },
                    deleteAction: function (postData) {
                        console.log("delete action called for:" + JSON.stringify(postData));
                        return {
                            "Result": "OK"
                        };
                    },
                    createAction: function (postData) {
                        var data = getQueryParams(postData);
                        return {
                            "Result": "OK",
                            "Record": { "Id": data["Id"], "SkillLevel": data["SkillLevel"], "Details": data["Details"], "Created": data["Created"], "Updated": moment() }
                        }
                    },
                    updateAction: function (postData) {
                        return {
                            "Result": "OK",
                        };
                    }
                },
                fields: {
                    Id: {
                        key: true,
                        create: true,
                        edit: true,
                        list: true
                    },
                    SkillLevel: {
                        title: 'SkillLevel',
                        width: '20%'
                    },
                    Details: {
                        title: 'Details',
                        width: '30%'
                    },
                    Created: {
                        title: 'Created',
                        edit: false,
                        create: false,
                        width: '20%',
                        display: function (data) { return moment(data.record.Created).format('DD/MM/YYYY HH:mm:ss'); }
                    },
                    Updated: {
                        title: 'Updated',
                        edit: false,
                        create: false,
                        width: '20%',
                        display: function (data) { return moment(data.record.Updated).format('DD/MM/YYYY HH:mm:ss'); }
                    }
                }
            });

            $('#createtableskilldetails').jtable({
                title: 'Skill Details',
                paging: false,
                pageSize: 5,
                sorting: true,
                multiSorting: true,
                defaultSorting: 'Name asc',
                actions: {
                    deleteAction: function (postData) {
                        console.log("delete action called for:" + JSON.stringify(postData));
                        return {
                            "Result": "OK"
                        };
                    },
                    createAction: function(postData) {
                        var data = getQueryParams(postData);
                        return {
                            "Result": "OK",
                            "Record": { "Id": data["Id"], "SkillLevel": data["SkillLevel"], "Details": data["Details"], "Created": moment(), "Updated": moment() }
                        }
                    },
                    updateAction: function(postData) {
                        return {
                            "Result": "OK",
                        };
                    }
                },
                fields: {
                    Id: {
                        key: true,
                        create: true,
                        edit: true,
                        list: true
                    },
                    SkillLevel: {
                        title: 'SkillLevel',
                        width: '20%'
                    },
                    Details: {
                        title: 'Details',
                        width: '30%'
                    },
                    Created: {
                        title: 'Created',
                        edit: false,
                        create: false,
                        width: '20%',
                        display: function(data) { return moment(data.record.Created).format('DD/MM/YYYY HH:mm:ss'); }
                    },
                    Updated: {
                        title: 'Updated',
                        edit: false,
                        create: false,
                        width: '20%',
                        display: function(data) { return moment(data.record.Updated).format('DD/MM/YYYY HH:mm:ss'); }
                    }
                }
            });      
        }); // End of document ready

        function getQueryParams(qs) {
            qs = qs.split("+").join(" ");

            var params = {},
                tokens,
                re = /[?&]?([^=]+)=([^&]*)/g;

            while (tokens = re.exec(qs)) {
                params[decodeURIComponent(tokens[1])] = decodeURIComponent(tokens[2]);
            }

            return params;
        }

        function getAllRowsOfjTableUpdateSkillDetailsList() {
            var $rows = $('#updatetableskilldetails').find('.jtable-data-row');
            var headers = ["Id", "SkillLevel", "Details", "Created", "Updated"];

            var data = [];
            $.each($rows, function() {
                var rowData = {};
                for (var j = 0; j < 5; j++) {
                    console.log(headers[j] + ":" +  this.cells[j].innerHTML);
 
                    rowData[headers[j]] = this.cells[j].innerHTML;
                }
                data.push(rowData);
            });

            $("#updateSkillDetailsList").val(JSON.stringify(data));
        }

        function getAllRowsOfjTableCreateSkillDetailsList() {
            var $rows = $('#createtableskilldetails').find('.jtable-data-row');
            var headers = ["Id", "SkillLevel", "Details", "Created", "Updated"];

            var data = [];
            $.each($rows, function () {
                var rowData = {};
                for (var j = 0; j < 5; j++) {
                    console.log(headers[j] + ":" + this.cells[j].innerHTML);

                    rowData[headers[j]] = this.cells[j].innerHTML;
                }
                data.push(rowData);
            });

            $("#createSkillDetailsList").val(JSON.stringify(data));
        }

        function SumbitUpdateForm() {
            getAllRowsOfjTableUpdateSkillDetailsList();
            $("#updateForm form").submit();
        }

        function SumbitCreateForm() {
            getAllRowsOfjTableCreateSkillDetailsList();
            $("#createForm form").submit();
        }

    </script>
}

@Html.Partial("SearchUpdate")

@Html.Partial("SearchDelete")

@Html.Partial("SearchCreate")

Now new Elastissearch documents with Nested object arrays can be created. The View looks like this:

Elasticsearch index and mapping

When you check the mapping in the Elasticsearch search engine, you will find the new document with the nested array of child items.
http://localhost:9200//_mapping

{
  "skillwithlistofdetailss": {
    "mappings": {
      "skillwithlistofdetails": {
        "properties": {
          "created": {
            "type": "date",
            "format": "dateOptionalTime"
          },
          "description": {
            "type": "string"
          },
          "id": {
            "type": "long"
          },
          "name": {
            "type": "string"
          },
          "skilldetails": {
            "properties": {
              "created": {
                "type": "date",
                "format": "dateOptionalTime"
              },
              "details": {
                "type": "string"
              },
              "id": {
                "type": "long"
              },
              "skilllevel": {
                "type": "string"
              },
              "updated": {
                "type": "date",
                "format": "dateOptionalTime"
              }
            }
          },
          "updated": {
            "type": "date",
            "format": "dateOptionalTime"
          }
        }
      }
    }
  }
}

Search with a Query string search

The search is built using the search class which contains a query string search. This query takes wildcards which can be used for the autocomplete. This could be optimized by using a different query type.

The term is split into different search terms with a * wildcard at the end of each. The search also searches the nested arrays.

private static readonly Uri Node = new Uri(ConnectionString);

public IEnumerable<SkillWithListOfDetails> QueryString(string term)
{
	var names = "";
	if (term != null)
	{
		names = term.Replace("+", " OR *");
	}

	var search = new ElasticsearchCRUD.Model.SearchModel.Search
	{
		From= 0,
		Size = 10,
		Query = new Query(new QueryStringQuery(names + "*"))
	};
	IEnumerable<SkillWithListOfDetails> results;
	using (var context = new ElasticsearchContext(ConnectionString, _elasticSearchMappingResolver))
	{
		results = context.Search<SkillWithListOfDetails>(search).PayloadResult.Hits.HitsResult.Select(t => t.Source);
	}
	return results;
}

The search is then used in the SearchController. This returns the collection as a Json array with is used directly from the autocomplete control.

$[Route("Search")]
public JsonResult Search(string term)
{
	return Json(_searchProvider.QueryString(term), "SkillWithListOfDetails", JsonRequestBehavior.AllowGet);
}

View autocomplete with jTable

The autocomplete control uses this Json result and allows the user to select a single document. When a document is selected, it is displayed in the update control.

<input id="autocomplete" type="text" style="width:500px" />

$("input#autocomplete").autocomplete({
source: function(request, response) {
	$.ajax({
		url: "http://localhost:50227/Search/search",
		dataType: "json",
		data: {
			term: request.term,
		},
		success: function(data) {
			var itemArray = new Array();
			for (i = 0; i < data.length; i++) {
				itemArray[i] = { label: data[i].Name, value: data[i].Name, data: data[i] }
			}

			console.log(itemArray);
			response(itemArray);
		},
		error: function(data, type) {
			console.log(type);
		}
	});
},
select: function(event, ui) {
	$("#spanupdateId").text(ui.item.data.Id);
	$("#spanupdateCreated").text(moment(ui.item.data.Created).format('DD/MM/YYYY HH:mm:ss'));
	$("#spanupdateUpdated").text(moment(ui.item.data.Updated).format('DD/MM/YYYY HH:mm:ss'));
	$("#updateName").text(ui.item.data.Name);
	$("#updateDescription").text(ui.item.data.Description);
	$("#updateName").val(ui.item.data.Name);
	$("#updateDescription").val(ui.item.data.Description);

	if (ui.item.data.SkillDetails) {
		updateResults = ui.item.data.SkillDetails;
	}
	
	$('#updatetableskilldetails').jtable('load');

	$("#updateId").val(ui.item.data.Id);
	$("#updateCreated").val(ui.item.data.Created);
	$("#updateUpdated").val(ui.item.data.Updated);

	$("#spandeleteId").text(ui.item.data.Id);
	$("#deleteId").val(ui.item.data.Id);
	$("#deleteName").text(ui.item.data.Name);

	console.log(ui.item);
}
});

The update control displays the parent and the child objects. The list of child Skilldetails is displayed in the jTable javascript component.

DateTime with moment.js

moment.js library is used to display the Json DateTime items in a readable format. The items are then used in the jTable and the input forms.

This package can be download using NuGet (Moment.js). It is used as follows:

moment(ui.item.data.Created).format('DD/MM/YYYY HH:mm:ss')

Update with ElasticsearchCRUD

The update method takes the received data from the view and updates all the Updated timestamps. The child SkillDetail list is added to the entity and this is then updated in Elasticsearch.

public void UpdateSkill(long updateId, string updateName, string updateDescription, List<SkillDetail> updateSkillDetailsList)
{
	using (var context = new ElasticsearchContext(ConnectionString, _elasticsearchMappingResolver))
	{
		var skill = context.GetDocument<SkillWithListOfDetails>(updateId);
		skill.Updated = DateTime.UtcNow;
		skill.Name = updateName;
		skill.Description = updateDescription;
		skill.SkillDetails = updateSkillDetailsList;

		foreach (var item in skill.SkillDetails)
		{
			item.Updated = DateTime.UtcNow;
		}

		context.AddUpdateDocument(skill, skill.Id);
		context.SaveChanges();
	}
}

The delete method uses the _id filed to delete the document.

Delete with ElasticsearchCRUD

public void DeleteSkill(long deleteId)
{
	using (var context = new ElasticsearchContext(ConnectionString, _elasticsearchMappingResolver))
	{
		context.DeleteDocument<SkillWithListOfDetails>(deleteId);
		context.SaveChanges();
	}
}

Conclusion

Using ElasticsearchCRUD, it is super easy to add, update, remove documents with 1 to n relationships. The child elements are nested in the parent document. Collections or arrays of objects as well as simple type collections/arrays are supported. Using Elasticsearch with ElasticsearchCRUD, you can create complex search queries.

Links:

https://www.nuget.org/packages/ElasticsearchCRUD/

http://www.elasticsearch.org/blog/introducing-elasticsearch-net-nest-1-0-0-beta1/

http://www.elasticsearch.org/

https://github.com/elasticsearch/elasticsearch-net

http://nest.azurewebsites.net/