"I Like Big Data for $400" - Adding Azure Search to the DocumentDB stack - Part 3


Integrate Azure Search into our application

Over the past two days, we've built a DocumentDB collection and consumed that collection using Angular and Electron. Today, we will wrap up our I like Big Data for $400 series by integrating the full text search capabilities of Azure Search. Azure Search allows developers to quickly integrate powerful search and faceting capabilities in their applications using an easy to consume REST API. In addition to the easy-to-use API, Azure Search is integrated into a variety of data sources such as DocumentDB, Azure SQL and Azure Storage. This integration means you don't have to worry about writing complex data indexers and you can, instead, focus on improving your users search experience.

You can learn more about Azure Search by viewing the Microsoft Azure Search Documentation (https://docs.microsoft.com/en-us/azure/search/) or any number of wonderful resources such as Chad Campbell's "Adding Search Abilities to Your Apps with Azure Search" Pluralsight course (https://www.pluralsight.com/courses/azure-adding-search-abilities-apps).

In this walk through, we will see how quick it is to index our Jeopardy question repository and integrate full text search capabilities into our application.

Create the Azure Search index

Much like DocumentDB, Azure Search services are grouped into a parent container. In DocumentDB, that container is called an account, and in Azure Search it is called a "Search Service." You can create a new Search Service by logging into your Azure portal, creating a new resource, and locating the Azure Search service under "Web + Mobile" and Azure Search.
Create an Azure Search Service

Just like DocumentDB, the URL you select for your search service should be globally unique. We will use cs-sqlsaturday-search. We can add it to the resource group of our choosing (cs-sqlsaturday in our case), and select a location nearest to us. Once we enter that information, we need to select the pricing tier.

Azure Search has a free tier we can use, however, the free tier is limited to 10,000 documents (our repository has 216,000). In order to handle the full repository size, we will need to select the basic plan, which is $75.14/month). The basic plan supports up to 1M documents with 5 indexes. We can scale much higher by selecting higher pricing plans. For more information on scaling and pricing plans, check the Azure Search Documentation.

Once we've selected a pricing tier, we can click create and Azure will provision our Azure Search service.
Azure Search Service Portal Blade

Now that our search service has been provisioned, we need to create an index. An index in Azure Search represents a specific type of document that we would like to search against. In our case, we need to create an index for our Jeopardy Questions. To do THAT, we will make use of the tight integration with DocumentDB.

In the Azure Portal, select navigate to the DocumentDB Account you created in Day 1 of our series.

In the blade menu, locate the "Add Azure Search" option.

Add Azure Search From DocumentDB

From the blade that displays, we can easily configure and integrate our DoucmentDB collection into Azure Search.

We begin by selecting the Azure Search service we just created.

When we do that, we are asked to configure a New Data source

Create Data Connection
We will name our datasource jeopardyquestions,

We will select the database and collection we created in day 1

And finally we will tell it OK.

If we had a more complex data structure, we could write SQL that would flatten and structure our underlying data in a flat table that can be consumed by Azure Search.

When we tell it OK, Azure Search will examine the data in the collection and will create the basic settings needed to create an index.

Create an Azure Search Index

For each field in our data, we will need to determine if the field can be retrievable, filterable, sortable, facetable or searchable.
  • Data that is retrievable can be returned in the search results. This is much like giving the field the ability to be returned in a select statement.
  • Data that is filterable can be filtered in your results. This is useful for categorical data or data that can easily be split into categories or ranges. For instance, you wouldn't want to enable filterable on a full text field like Question, but it is helpful for fields such as Round and Category.
  • Data that is sortable can be used to sort results.
  • Data that is facetable can be split into facets and reported on. Facets are a useful way of exploring data by giving your user a preview of how adding that filter will impact the search results. If you've used e-commerce sites such as Amazon you've seen facets in action.
    Example of Facets
  • Data that is searchable is indexed for full-text searching. This is part of the real power of Azure Search. It allows you to use a powerful natural language search interface with very little technical overhead.

In our case, we will configure the index as follows:
Search Index Settings

When we click OK, we will be given the option to create an indexer. An indexer allows us to schedule the routine synching of data between or data collection and Azure Search. All we need to do is give it a name and schedule how often it should run. We will also need to set one advanced option, click Advanced options and check "Base-64 Encode Keys" since we are using the rid as our key, we need to encode those values since they have non-alphanumeric data.

Create an Indexer

When you set those values, click OK and OK again and your indexer will run with the set schedule.

You can edit that schedule in the future by going to the Azure Search Service blade, clicking the Indexers widget and edit the indexer and set the schedule to where you need.

Indexers Widget
Edit Indexer

Search Explorer

Now that we have an index and have indexed our DocumentDB collection, let's perform some basic searches of the data. To do that, we will use the Search Explorer inside the Azure Portal.
First, navigate to your search service and click the "Search explorer" button in the toolbar.

Access Search Explorer

From there we will be able to create a search. If you leave the default values and click search, Azure Search will return the top results with no search criteria applied.

First Search in Search Explorer

We can do a more advanced search by typing any search string in the "Query string" box...ie "Eagle"

Eagle Search using Azure Search Explorer

You can create more advanced search criteria using OData. You can learn more about how to do that here (https://docs.microsoft.com/en-us/rest/api/searchservice/simple-query-syntax-in-azure-search).

Integrate the Question Search tool in Angular

When you enter information in the Search Explorer, it generates a Request URL. We can use that URL to create a REST request and consume the results using your HTTP tool of choice. In our case, we will use Angular's HTTP service much like we did in Day 2 of this series to return data from Azure Search.

Before we get started, we need a query key from our search service. To get one, Go to Keys in the Azure Search blade and click "Manage Query Keys" and finally, copy the Key value provided.
Access Query Keys in Azure Search
Manage Query Key Dialog
We will use that key in the search service we are about to build.

Back in VSCode...

To continue our app development, we will need to add a component and service back in our angular application. Once again, we will use angular-cli to perform these actions.

Return to your command prompt and make sure you are in the client application folder that you created in Day 2.

Run the following commands:

ng g service services/search
ng g component Search

Open the src/app/services/search.service.ts file and set it to:

import { Injectable } from '@angular/core';
import { Headers, Http } from '@angular/http';

import 'rxjs/add/operator/toPromise';

@Injectable()
export class SearchService {
private searchKey: string = "5D157834E534D08347138B30A06E0557";
private searchBaseUrl: string = "https://cs-sqlsaturday-search.search.windows.net/indexes/questions/docs?api-version=2016-09-01";

constructor(private http: Http) { }

getSearchResults(searchString: string, airDateStart: Date, airDateEnd: Date, showJeopardy: boolean, showDoubleJeopardy: boolean, showFinalJeopardy: boolean): any {
var searchUrl = this.buildSearchUrl(searchString, airDateStart, airDateEnd, showJeopardy, showDoubleJeopardy, showFinalJeopardy);
let headers = new Headers();
this.addAuthorizationHeader(headers);
return this.http.get(searchUrl, {
headers: headers
})
.toPromise()
.then(response => response.json())
.catch(this.handleError);
}

private buildSearchUrl(searchString: string, airDateStart: Date, airDateEnd: Date, showJeopardy: boolean, showDoubleJeopardy: boolean, showFinalJeopardy: boolean): string {
console.log(this.formatDate(airDateStart));

var searchFilter = "&search=" + searchString + "&facet=Round&facet=Category";
var airDateFilter = "AirDate ge " + this.formatDate(airDateStart) + " and AirDate le " + this.formatDate(airDateEnd);
//add airDateFilter
searchFilter = searchFilter + "&$filter=" + airDateFilter + " and ("+ this.formatRoundFilter(showJeopardy, showDoubleJeopardy, showFinalJeopardy) +")";
console.log(this.searchBaseUrl + searchFilter);
return this.searchBaseUrl + encodeURI(searchFilter);
}

private handleError(error: any): Promise<any> {
console.error('An error occurred', error); // for demo purposes only
return Promise.reject(error.message || error);
}
private addAuthorizationHeader(headers: Headers) {
headers.append('api-key', this.searchKey);
}
private formatDate(date: Date): string {
var month = date.getMonth() + 1;
var day = date.getDate();
var monthStr: string;
var dayStr: string;

if (month < 10)
monthStr = "0" + month.toString();
else
monthStr = month.toString();

if (day < 10)
dayStr = "0" + day.toString();
else
dayStr = day.toString();

return date.getFullYear().toString() + "-" + monthStr + "-" + dayStr + "T00:00:00-08:00";
}
private formatRoundFilter(showJeopardy: boolean, showDoubleJeopardy: boolean, showFinalJeopardy: boolean) {
var roundFilter = "";
if (showJeopardy)
roundFilter = "Round eq 'Jeopardy!'";

if (showDoubleJeopardy)
if (showJeopardy)
roundFilter = roundFilter + " or Round eq 'Double Jeopardy!'";
else
roundFilter = "Round eq 'Double Jeopardy!'";

if (showFinalJeopardy)
if(showJeopardy || showDoubleJeopardy)
roundFilter = roundFilter + " or Round eq 'Final Jeopardy!'";
else
roundFilter = "Round eq 'Final Jeopardy!'";

return roundFilter;
}
}

Make sure you change your search key to match the query key we created above and change your searchBaseUrl to match your Request URL from your search explorer minus "&search=*".

This code generates a search query URL that applies filters for round and show date range as well as a text search query. Once a URL is generated, it uses HTTP and rxjs to return a promise and json results from our search call.

To complete the process, we need to update our search component to consume the search service.

Open src/app/search/search.component.ts and set it to:

import { Component, OnInit } from '@angular/core';
import { SearchService } from '../services/search.service';

@Component({
selector: 'app-search',
templateUrl: './search.component.html',
styleUrls: ['./search.component.css']
})
export class SearchComponent implements OnInit {
searchResults = { value: [] };
searchString = "";
airDateStart = new Date(1990, 0, 1);
airDateEnd = new Date();
jeopardy = true;
doubleJeopardy = true;
finalJeopardy = true;
columns = [
{ name: "Show Number", prop: "ShowNumber" },
{ name: "Air Date", prop: "AirDate" },
{ name: "Round", prop: "Round" },
{ name: "Category", prop: "Category" },
{ name: "Value", prop: "Value" },
{ name: "Question", prop: "Question" },
{ name: "Answer", prop: "Answer" }

]

constructor(private searchService: SearchService) {
}

ngOnInit() {
this.getSearchResults();
}

getSearchResults(): void {
this.searchService.getSearchResults(this.searchString, this.airDateStart, this.airDateEnd, this.jeopardy, this.doubleJeopardy, this.finalJeopardy).then(results => {
this.searchResults = results;

});
}

}

We have a single call getSearchResults() that passes a search string, an air date range and round flags to our search service, and saves the results in our results property.

Let's set our view (src/app/search/search.component.html) to this:

<div class="row">
<div class="col-md-8" style="height: 90vh">
<ngx-datatable
class="material"
[rows]="searchResults.value"
[columns]="columns"
[headerHeight]="50"
[footerHeight]="50"
[rowHeight]="50"
[limit]="15">
</ngx-datatable>
</div>
<div class="col-md-4">

<div class="form-group">
<label for="searchString">Search</label>
<input type="text" class="form-control" id="searchString" placeholder="Search" [(ngModel)]="searchString">
</div>
<div class="form-group">
<label>Air Date</label><br>
<label for="fromDate">From</label>
<datepicker #dp [(ngModel)]="airDateStart" (navigate)="date = $event.next"></datepicker>
<label for="thruDate" style="margin-top:5px;">Thru</label>
<datepicker #dp [(ngModel)]="airDateEnd" (navigate)="date = $event.next"></datepicker>
</div>
<div class="form-group">
<label>Round </label><br/>
<label>
<input type="checkbox" value="Jeopardy!" [(ngModel)]="jeopardy">Jeopardy!
</label><br/>
<label>
<input type="checkbox" value="Double Jeopardy!" [(ngModel)]="doubleJeopardy">Double Jeopardy!
</label><br/>
<label>
<input type="checkbox" value="Final Jeopardy!" [(ngModel)]="finalJeopardy">Final Jeopardy!
</label>
</div>
<div class="form-group">
<span class="btn btn-large btn-info" (click)="getSearchResults()">Search!</span>
</div>

</div>
</div>

We are using ngx-datatable to display the results in a formatted datagrid. Let's go ahead and install that:

  1. run
    npm install @swimlane/ngx-datatable@6.3.0 -save
  2. Add the ngx-datatable stylesheets to angular-cli.json
    "styles": [
    "../node_modules/bootstrap/dist/css/bootstrap.min.css",
    "../node_modules/@swimlane/ngx-datatable/release/themes/material.css",
    "../node_modules/@swimlane/ngx-datatable/release/assets/icons.css",
    "styles.css"
    ],

Next, let's set some styling for the search component src/app/search/search.component.css

label {
color: white;
}

.form-group {
border-top: 1px solid lightgrey;
margin-top: 10px;
padding-top: 10px;
}

Finally we need to register our search service as a provider and register the search route with our router module.

Open src/app/app.module.ts
Update your RouterModule code to look like this:

RouterModule.forRoot([
{
path: '',
redirectTo: '/game',
pathMatch: 'full'
},
{
path: 'game',
component: GameComponent
},
{
path: 'search',
component: SearchComponent
}
]),
],

Add the following imports to the header:

import { SearchService } from './services/search.service';
import { DatepickerModule } from 'ng2-bootstrap';
import { NgxDatatableModule } from '@swimlane/ngx-datatable';

Update the imports array to add:

    NgxDatatableModule,
DatepickerModule.forRoot(),

Add SearchService to the providers array.

Now that all that is complete, we should be able to run our application, click on the search link and search our repository of Jeopardy! questions.

Final Search Screen

Conclusion

This concludes our series on integrating Angular, Electron, Azure DocumentDB and Azure Search. DocumentDB and Azure Search are components in Microsoft's Data Platform that allow developers to rapidly create powerful applications using modern Web development tools. When combined with the power of Angular and Electron, you can quickly create a powerful app that can scale globally and create game-changing functionality for your users.

Are there areas you would like to know more about? We are looking for additional blog topics and would to hear from our readers in what they would want to see? Reach out to us on twitter @SigaoStudios or leave a comment here and let us know what we should explore next.

"I Like Big Data for $400" - Implementing a Jeopardy! Question Explorer using DocumentDB, AzureSearch, Angular and Electron - Part 2 If the plan doesn’t work, change the plan but never the goal.