Full-text search for your back-end using Azure Search and Azure Mobile Services

 

In a previous blog post, I described the implementation of full-text search capabilities for your Azure Mobile Services back-end. With the recent availability of Azure Search —search as a service in Azure— I thought it would be useful to revisit the idea and modify it using this cool new Azure capability.

 

 

Defining the Index’s Schema

A document is the basic unit of data in Azure Search. Each document contains a set of fields, and each field has a set of attributes that define how the index handles the data associated which each field. When we create an index in Azure Search, we must create a schema that outlines attributes for each field. In the schema, we outline the types of fields available, which fields are searchable, and other interesting capabilities. For example, we determine whether a field can be used as a search suggestion, for ordering or filtering the results, and as a facet, such as showing a count of results by brand if you search for “sports cars.”

 

Note: For more information about the schema definition, click here.

Following the spirit of a “code first” approach, let’s create a custom attribute that models the attributes of the index. We can then use it in the properties of an entity to create the metadata that will define the schema of the index.

[AttributeUsage(AttributeTargets.Property)]
public class IndexableAttribute : Attribute
{

private string _name;

public string Name
{
get { return _name; }
set { _name = value; }
}

public readonly string Type;
public readonly bool Searchable;

private bool _filterable = true;

public bool Filterable
{
get { return _filterable; }
set { _filterable = value; }
}

private bool _sortable = true;

public bool Sortable
{
get { return _sortable; }
set { _sortable = value; }
}


private bool _facetable = true;

public bool Facetable
{
get { return _facetable; }
set { _facetable = value; }
}


private bool _suggestions = false;

public bool Suggestions
{
get { return _suggestions; }
set { _suggestions = value; }
}

private bool _key = false;

public bool Key
{
get { return _key; }
set { _key = value; }
}


private bool _retrievable = false;

public bool Retrievable
{
get { return _retrievable; }
set { _retrievable = value; }
}

public IndexableAttribute(string type, bool searchable )
{
this.Type = type;
this.Searchable = searchable;
}
}

Capturing Changes

Whenever an entity is modified or created, we need to capture and queue these changes. The benefit of queuing these changes is that we can decouple the process of maintaining the index from the request that updates the database.

The process of sending the entity to the queue consists of getting the information for each property’s custom attribute and value. The following code shows the implementation of this process. Full code here. 

public class IndexQueueManager
{
...

public TData EnqueueEntity<TData>(Task<TData> tEntity, string action) where TData : class, ITableData
{

if (tEntity.IsFaulted)
{
throw new InvalidOperationException("Task in faulted state", tEntity.Exception);
}

var ixEntity = GetEntityIndexInfo<TData>(tEntity.Result, action);

var msg = new BrokeredMessage(JsonConvert.SerializeObject(ixEntity));
msg.TimeToLive = new TimeSpan(7, 0, 0, 0);
_indexerQ.Send(msg);

return tEntity.Result;
}
...
}

Note: The object that goes into the queue is an instance of EntityIndexInfo. You can find the source code here.

For the queue’s implementation, we will use the queuing capabilities of the Azure Service Bus. Service Bus provides orderly delivery of the messages, which is an important consideration because we need to guarantee updates to the index in the same order that changes to an entity occur. The following code shows the constructor of the IndexQueueManager class where we initialize the queue client.

public IndexQueueManager(string connString)
{
if (connString == null)
{
throw new ArgumentNullException("connString");
}

var mgr = NamespaceManager.CreateFromConnectionString(connString);

if (!mgr.QueueExists("amsindexerqueue"))
{
var q = mgr.CreateQueue("amsindexerqueue");
}

_indexerQ = QueueClient.CreateFromConnectionString(connString,"amsindexerqueue");

}

Finally, one elegant way to capture when these changes occurred is by overriding the insert, replace, update, and delete methods of the EntityDomainManager. The following code shows the implementation of a derived domain manager that, after performing a database operation, sends the entity to the queue.

 

public class IndexableEntityDomainManager<TData> : EntityDomainManager<TData> where TData : class, ITableData, new()
{
private IndexQueueManager IndexManager;

public IndexableEntityDomainManager(DbContext context, HttpRequestMessage request, ApiServices services, string ServiceBusConnString)
: base(context, request, services)
{
if (ServiceBusConnString == null)
{
throw new ArgumentNullException("ServiceBusConnString");
}

this.IndexManager = new IndexQueueManager(ServiceBusConnString);
}
public override Task<TData> ReplaceAsync(string id, TData data)
{
return base.ReplaceAsync(id, data)
.ContinueWith<TData>((t) => IndexManager.EnqueueEntity<TData>(t, "mergeOrUpload"));
}

public override Task<TData> UpdateAsync(string id, Delta<TData> patch)
{
return base.UpdateAsync(id, patch)
.ContinueWith<TData>((t) => IndexManager.EnqueueEntity<TData>(t, "merge")); ;
}

public override Task<TData> InsertAsync(TData data)
{
return base.InsertAsync(data)
.ContinueWith<TData>((t) =>IndexManager.EnqueueEntity<TData>(t, "upload"));
}

public override Task<bool> DeleteAsync(string id)
{
return base.DeleteAsync(id)
.ContinueWith<bool>((t) =>
{
IndexManager.EnqueueEntity<TData>(Task<TData>.Run(() => new TData() { Id = id }), "delete");
return t.Result;
});

}
}

Maintaining the Index

Now that we have the changes in the queue, we need to read the messages from the queue and update the index. The following code first reads a message and checks whether an index exists. If an index does not exist, the code creates one according to the definition included in the message. Finally, the code creates a document instance containing the changes to the entity and sends a request to update the index with this information. Check out the full implementation here.

public class IndexQueueManager
{
...

public async Task<Boolean> ReadFromQueueAndIndexAsync()
{
try
{
var msg = _indexerQ.Receive();

if (msg == null)
{
return false;
}

var indexInfo = JsonConvert.DeserializeObject<EntityIndexInfo>(msg.GetBody<string>());

await CreateIndexIfNotExistsAsync(indexInfo);

var indexDoc = GetIndexDocument(indexInfo);

await IndexDocumentAsync(indexDoc);

_indexerQ.Complete(msg.LockToken);

return true;
}
catch(Exception ex)
{
_indexerQ.Abort();

throw ex;
}

}
...
}

After we wrap up this code as a scheduled job, we are ready to enable this functionality in our back-end!

public class IndexerJob : ScheduledJob
{
public async override Task ExecuteAsync()
{
var mgr = new IndexQueueManager(CloudConfigurationManager.GetSetting("Search_ServiceBusConnString"));
var maxNumberOfMgs = 1000;
var processMsgs=0;

while (await mgr.ReadFromQueueAndIndexAsync() && processMsgs < maxNumberOfMgs)
{
processMsgs++;
Services.Log.Info(string.Format("Processed message. Message count {0}",processMsgs.ToString()));

}

}
}

Using What We Built

First things first: let’s create an Azure Search instance using the preview portal.

 CreateSearch

 

We are ready to use our custom attribute and domain manager. Let’s create a data model and decorate the properties using the custom attribute.


public class RockBand:EntityData
{
[Indexable("Edm.String", true, Suggestions = true, Retrievable = true, Facetable = false)]
public string Name { get; set; }

[Indexable("Edm.String", true,Facetable=false)]
public string Description { get; set; }

[Indexable("Edm.String", true)]
public string Genre { get; set; }
}

Next, let’s create a table controller for the model that uses our domain manager.

public class RockBandController : TableController<RockBand>
{
protected override void Initialize(HttpControllerContext controllerContext)
{
base.Initialize(controllerContext);
MobileServiceContext context = new MobileServiceContext();
DomainManager = new IndexableEntityDomainManager<RockBand>(context, Request, Services, CloudConfigurationManager.GetSetting("Search_ServiceBusConnString"));
}

// GET tables/RockBand
public IQueryable<RockBand> GetAllRockBand()
{
return Query();
}

// GET tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
public SingleResult<RockBand> GetRockBand(string id)
{
return Lookup(id);
}

// PATCH tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task<RockBand> PatchRockBand(string id, Delta<RockBand> patch)
{
return UpdateAsync(id, patch);
}

// POST tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
public async Task<IHttpActionResult> PostRockBand(RockBand item)
{
RockBand current = await InsertAsync(item);
return CreatedAtRoute("Tables", new { id = current.Id }, current);
}

// DELETE tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
public Task DeleteRockBand(string id)
{
return DeleteAsync(id);
}

}

Now we can insert a record, run the indexer job and check the index’s schema via the portal.

image image

image

 

Finally, we can perform a search using a client app and view our results!

image

You can find the source code for the whole solution here.


2 Comments

  • Hi Jesus,

    thanks a lot for a such a good example for Azure Search! It's really helpful! Just one question about it: do I need to publish the Giventocode.MobileServices.AzureSearch project or it enough to refer to it from the Giventocode.AzureSearch one and publish it only?

    Thanks in advance for the answer

Post Reply