Technical introduction to Coveo for Sitecore

Sitecore Developer

Valtech

June 13, 2019

Coveo is a fantastic search API to use with Sitecore. It provides great features such as indexing external resources and Sitecore content, and indexing or excluding the HTML part of the website. It has several placeholders and components that help developers to build search pages with helpful functionalities, such as facets and groups to filter search, etc. It also provides the ability to customize some of the components to match your requirements. By having rules and categories in different components, one can take advantage of filtered indexes, etc.

• Download the Coveo for Sitecore ZIP file for your Sitecore version. E.g. Coveo for Sitecore 9.0.
• Install the ZIP file as a package in Sitecore.
• During the installation click on overwrite, whenever seen.
• One step of the installation is to “Sign-up or Login to Coveo Cloud”. Click on the button.

• Using your credentials, login to Coveo Cloud.

• Authorize the group.

• Select your organization or create a new one.

7.a. If you select “Create new organization”, you will be given a text box to enter the organization name. You can also select the license plan that you need.

• In the next step, you should enter the name of the Sitecore user that you’d like to use, as well as the password.

• After clicking on the “Next” button, in the next step you will have two options for Indexing. By selecting “Index rendered HTML”, Coveo will index all HTML pages on the website. And if you select “Only index Sitecore item data”, Coveo will index only Sitecore items, not the HTML part of it. Please notice that by selecting “Index rendered HTML” Coveo indexes only the loaded HTML content, not JavaScript results.

• In the next step you will be asked if you’d like to “Disable Security Permissions on Documents” or not. By selecting that, Coveo will index all items as Anonymous, which will result in a performance increase.

• In the next step, Coveo asks if it can automatically rename some configuration files. It is recommended to leave it as checked and click on next button. Otherwise the system administrator has to remove “.example” extension from the configuration files, that exist in /App_Config/Include/Coveo and /App_Config/Modules/Coveo, manually.

• By clicking on Next in the last step, you will be redirected to the final page of the installation. Click on close to close the window.

• After the installation is completed, you will be able to see the “Coveo Search” section in the Sitecore control panel.

• During this stage, if you click on “Coveo Cloud administration console”, there is no resource in your cloud console. That is because you need to sync your Coveo index in Sitecore.

So you have to sync Coveo. To do that, you must click on “Action” in the Coveo Search section of Sitecore control panel, then click on the “Sync” button.

• During this stage, you need to build Coveo Indexes. So you should click on Index Manager in Sitecore Control Panel, and check the Coveo Master and Web indexes to be built.

If you take a look at Coveo Cloud Console, and navigate to “Sources” page, you will see that Coveo Master and Web indexes are being built. Please note that you can access the cloud console using “Coveo Cloud administration console” in the Sitecore control panel, or by navigating to the following URL:
https://platform.cloud.coveo.com/admin

As you can see there are no items in the indexes content. But once the content is built, the number of items will be shown.

• To make sure that all the indexes are built, you can navigate to the Sitecore control panel, and click on “Diagnostic page” in the Coveo Search section. Then you will be redirected to the diagnostic page. If all the Coveo Components are up and running, then Coveo is ready to use.

• During this stage, you need to check Coveo Cloud Console Status to see if your organization status is healthy. You can find the Status at the bottom left-hand side of the console navigation.

• Now you can navigate to the following folder in your Sitecore instance to see the Coveo configurations:
C:\inetpub\wwwroot\{your sitecore instance}\App_Config\Modules\Coveo
If you open up Coveo.SearchProvider.config file, you will see some configurations that will be explained as follows:
As explained in the config file, since Sitecore 9 already has Solr as its default search provider, Coveo will add indexes to the same configuration element as other indexes.

• As you can see in the following image, there is an attribute called “patch:before” that “*[1]” or “*[2]” has been assigned to it. Those show the orders in which Indexes are being displayed in the index manager. If you would like to patch any extra index, make sure that you patch:before it after the original indexes. E.g. patch:before=”*[3]”.

• Coveo has this ability to cache the elements before the next query on the index refreshes the cache. The maximumAge defines the delay the query results will be cached.

• The next thing to notice is the crawlers, as you can see two crawlers have been added by Coveo on the master database, one for the content items, and another one for media library assets. The reason that there are two crawlers is that web crawler must be able to respect the security permissions of items.

• Please note that Coveo creates one index per database (master and web). If you have distributed databases, Coveo creates new indexes for each database.

• Indexing is sequential in Sitecore, and after for pushing to Coveo Cloud there are threads. Up to 10 threads will be built to push the indexes to Coveo Cloud.

• It is a known issue that if Coveo indexing takes a long time in Sitecore, and Sitecore gets timed out in between, validation of all the items will crash. And you have to rebuild again.

• In Coveo Cloud Console, you are given this ability to add more sources to be indexed. For example, suppose that you would like to add Sitecore Helix to your sources. In order to do so you must navigate to the “Sources” page in the console, and click on “ADD SOURCE” button.

Then you should scroll down to see Web, and click on that.

Then select Web Cloud and in the newly opened window enter “Helix” as your Source name. You should also add its URL (https://helix.sitecore.net/) to the Site URL.

On the right-hand side, you will see some settings sections. If you expand “CRAWLING SETTINGS” you will see that in addition to the crawling options, there is a section called “WEB SCRAPING”.

In that section you can configure a JSON web scraper to ignore parts of the website that you do not wish to be indexed. In order to do that you must do the following:
25.a. Install Coveo Extension in your Chrome browser.

25.b. Go to the Sitecore Helix website with Chrome browser. Open up Development Tools of the browser, and select the part of the website that you would like to exclude from indexing. E.g. select the left navigation part. Get the class name of the division, and add it to your browser web scraper.

25.c. Then click on“Copy to clipboard” button, to copy the JSON object of the scraper.

25.d. In this step, you should add the copied JSON object to the “JSON configuration” part of the source, and click on “ADD AND BUILD SOURCE”.

25.e. Now you have to give it some time to build the index.

25.f. Now if you navigate to Content Browser in the cloud console, and look for the items in the navigation part of Helix, you cannot see any result.

• The next feature that you can see is Field Map in the Coveo Cloud Console. You can see that the fields are Facetable, Sortable, etc. You can also edit them. Please note that the fields that start with a prefix “f” belong to Coveo index.

Creating Search Result Page:
• First you have to create a page with some placeholder in it. Please note that Sitecore sample page is not MVC, thus, it doesn’t work with Coveo Hive components.
• Add Coveo Search Interface as a controller to the placeholder (Renderings/Coveo Hive/Search Interfaces/Coveo Search Interface).
• Add the search interface to the placeholder on the page.

• Create a new data source as a content for the interface.

• By adding the Coveo search interface, you will see that a component will be added, which has a placeholder called UI Content.

• You can add a modular frame to that placeholder.

• By adding the modular frame, some new placeholders will be added to the page (e.g. Header Section, UI Header, Main Section, UI Footer, Footer Section).
• In the Main Section, click on “Add here” and add a component called “Results Section” and click on the “Save” button of the page, and let the page be reloaded.

• By adding the Results Section, two new placeholders will be added to the page (Results Header, and Results List).
• By clicking on “Add here” button of the Results List, add a “Coveo Results List” component to the placeholder, and add a datasource to that.

• Choose a layout for the results list.

• After adding all the components, you’ll see that no result appeared on the page. That is because you have to load the rendering on the page. To do so, you have to add the following line of code to the page:

Then you’ll see a result page like the following:

• In order to be able to search keywords, you need to add a global search box to the page, and then connect it as an external component to the search interface. To do so you need to add an external component section to the page (Renderings/Coveo Hive/Sections/External Components Section).

• Then in the “External Components” placeholder add “Global Search Section”.

• Then add Coveo Global Searchbox to the newly created placeholder (Global Searchbox).

• By editing the datasource of the component, specify the Search page URI.

• Then you will see the searchbox appear on the page, and works.

• In the next step you can add more components to the page. E.g. Add a Result Summary Section to the “Results Header” placeholder.

By adding the result summary section, a new placeholder called “Results Summary Section” will be added to the page, using which you can add “Coveo Query Summary” or “Coveo Query Duration” to the page, and create new data sources for them.

• Then you can add pagination to the result list. To do so you need to add “Coveo Pager” component to the “UI Results Footer” placeholder.

• You can also add a sorting component to the result list. To do so, you need to locate Results Header placeholder of the Results Summary Section. Then by clicking on the “Add here” button, add a Results Header Section.

• Then click on “Results Header Section” placeholder, click on “Add here” button, and select “Results Sorts Section” from the opened dialog.

• In the newly added placeholders, select “Sorts” placeholder, and click on the “Add here” button. Then select a sort component. E.g. Coveo Relevancy Sort.

• In the same way, you can add other sort components as well.

• Additionally, in the same way you can add Tabs. Please note that in the tabs you can add external sources, so it will list the results of the external source in the results list as well.

• As you may have noticed, after adding the results section, an error appeared on the page:

In order to resolve the error, you need to add the Coveo Analytics Component to the page. To do so add “Coveo for Sitecore Analytics” to the UI Header placeholder. Usage analytics is one of the most important features of a Coveo solution.
Without them, it is impossible to continuously and automatically improve the search results relevance and provide intelligent query suggestions through machine learning.
It is very important to insert a Coveo Analytics or Coveo for Sitecore Analytics component to your search interfaces.

• In the next step you can add a result template in order to change the template of the results being displayed. In order to do so, you need to click on the “Add here” button of the “Result Templates” placeholder and select a template, “Coveo File Result Template” in this case.

Then create a datasource for that and change the “File name template” field to “Default”, and save the page. The results you see will resemble the following:

• You can customize the template by adding views to the following path, and referencing the name of the view in “File template name” section of the template.
/Coveo/Hive/templates/{view}.cshtml

• In order to add new fields to the template you need to use a syntax like the following:
<tr data-field='<%= coveoFieldName("@@{field name}") %>' data-caption='<%= translateLabel("{Label}") %>' data-split-values='true' />

• In this step you can add a facet to the page. To do so, you need to select the “Main Section” placeholder of the Search Interface. By clicking on the “Add here” button, you can select a “Facet Section”.

• Then click on the Facets placeholder and add the “Coveo Facet” to that.

• Then you will see that it says “The field name must be set.”

• In order to resolve the error, you should click on the Coveo edit button, and select a field name in the datasource of the facet. You should also enter a name and title for the facet as well.

• Then you will see that the facet will appear on the page.

Computed Index Fields:
• It is possible in Coveo to index fields that do not exist in the templates. The values of these fields are computed by code. The example of these fields are in App_Config/Include/Coveo/Coveo.SearchProvider.Custom.config file. You can uncomment them or change them. 

The example of such fields in the config file are as follows:

<!- <field fieldName="imageUrl" sourceField="Image">Coveo.SearchProvider.ComputedFields.ImageUrlComputedField, Coveo.SearchProviderBase</field> <field fieldName="imageAlt" sourceField="Image">Coveo.SearchProvider.ComputedFields.ImageAltComputedField, Coveo.SearchProviderBase</field> -->  <!- <field fieldName="theNewFieldToCreate" sourceField="theReferencedFieldId" referencedFieldName="referencedField">Coveo.SearchProvider.ComputedFields.ReferencedFieldComputedField, Coveo.SearchProviderBase</field> -->

ConfigurableComputedField based computed index fields can use the attributes of the <field> element in the Coveo.SearchProvider.config file.

<field fieldName="imageUrl" sourceField="Image">Coveo.SearchProvider.ComputedFields.ImageUrlComputedField, Coveo.SearchProviderBase</field> For example, the ImageUrlComputedField reads its sourceField attribute value by calling

string sourceField = GetAttributeValue("sourceField");

If you create a computed index field, you must implement the getter of the "ReturnType" property with one of the valid values:

Integer

Number

date

datetime

string

public string ReturnType { get { return "datetime"; } set { } }

Computed index fields code is executed at indexing time. If they do too much work or their code is not optimized, they can really slow down the whole indexing process. Be really cautious with your code!
Avoid running HTTP requests
Avoid querying a search index
A warning is written in the Sitecore logs when computed index fields execute search queries

Filtering Search Results:

Filtering can be done at 2 moments:
1. Indexing time
Called indexing scope ○ Restrict the number of indexed Sitecore items to avoid indexing items you know you don't want the users to search for
Examples: Index crawler root, Inbound filter, Excluded template… ○ Done by developers.
1. Query time
Restrict the search results returned for a search query
Done by developers ○ Can also be done by content authors, administrators and marketers with the Enterprise editions of Coveo for Sitecore

It is recommended to reduce the number of indexed items over filtering the search interfaces results to improve indexing time and query performance.
You can do this with any combination of:
1. Using multiple crawlers and setting their root item
2. Using Coveo inbound filter pipeline processors
3. Including/Excluding templates

Inbound Filters:

Coveo Specific Pipeline:

This pipeline is run only for Coveo indexes and allow you to choose for which indexes to run each processor. To use it you can navigate to your website\App_Config\Include and create a folder called “Training”. Then you can add a config file called “IndexingScope.config”. You can then add the following code to the file:

<processor

type="Coveo.SearchProvider.CoveoInboundFilters.HasLayoutInboundFilter, Coveo.SearchProviderBase">

<ApplyToPaths>/sitecore/content</ApplyToPaths>

</processor>

</coveoInboundFilterPipeline>

</pipelines>

</sitecore>

</configuration>

Exclude Templates for Changing Indexing Scope:

To exclude a specific template from indexing, add the following code to the config file (IndexingScope.config), after pipelines.

<coveo>

</exclude>

</documentOptions>

</defaultIndexConfiguration>

</coveo>

Then you need to rebuild the indexes to change the indexing.

At query time, you can filter search results using the Properties Editor. In order to do so, you need to click on the Coveo Experience Button. In the window that has popped up, click on “Edit rule” link of “Filter expression rules”.

You can choose a rule, in this case “where specific field compares to specific value”.

Then click on the links and give them values. E.g. for “specific” put “_templatename” and for “compares to” select “is equal to”, and for “specific value” enter “Page”. Then you will see that the number of the results will reduce, and the result shows only the items with page template.

Filtering Search Results in the JS UI:
• Another way of filtering search results is via JS UI. In order to do that, you first need to create a view rendering in Sitecore:
a. right-click>Insert>Rendering Folder. Call that Training.
b. On Training folder, right-click>Insert> View Rendering
c. Change the value of path to “/Views/Training/LaunchJsExpressions.cshtml”.
d. Save the item.
e. From the file system, navigate to Views folder of the site.
f. Create and open “Training” folder.
g. Create and open “LaunchJsExpressions.cshtml”.
h. Add the following code to the file:
@using Coveo.UI.Components
@using Coveo.UI.Components.Extensions

@Html.Partial(Partials.EDIT_TITLE, Html.Coveo().GetViewName())

i. Save the file.
j. From the Sitecore Content Editor, select the Settings content item ”/sitecore/Content/Global/Settings”
k. Insert a new Folder (right-click > Insert > Folder)
Name: Coveo Hive Placeholder Extenders
l. In this new folder, insert a new Placeholder Extender item (right-click > Insert > Insert from template)
m. In the Insert from Template dialog
Template: /Templates/Coveo Hive/Structure/Placeholder Extender
Item Name: Header Section Placeholder Extender
n. Choose Insert.

o. In the Placeholder Key field, select the Header Section placeholder
/Placeholder Settings/Coveo Hive/Sections/Header Section

p. In the Allowed Controls field, click the Edit link
q. In the Select Items dialog, select the rendering that you created /Layout/Renderings/Training/JSExpressions

r. Choose OK
s. Save the item
t. From the Sitecore Content Editor, select your "CoveoSearch" item (/sitecore/Content/Home/CoveoSearch)
u. Access the Experience Editor (Publish > Experience Editor). Select the Header Section placeholder.
v. Click the "Add here" button. In the Select a Rendering dialog, select Launch JavaScript Expressions
w. Choose Select.

x. Save the item.
y. Now you will see that the search result will be changed corresponding to the filtering.

Boosting Search Results:

Modify the order of the search results by adding or removing points to their ranking score. Points are added to a subset of the search results matching a query expression.

Example: $qre(expression:'@TemplateName == "Product"', modifier: '10')

To do so, from the Sitecore Experience Editor select the Coveo Search Interface component.
Click the Coveo experience button in the floating toolbar.
In the Control Properties dialog, edit the boost expression rules (Query Ranking > Boost expression rules > Edit rule link)
In the Create rule dialog, click the "where the item is the specific item" condition under Coveo Items.
”remove number from document score" action
In the Edit rule bottom section, click the links:
specific: Select "Content/Home" and choose OK.
number: Enter "100" and choose OK.
In the Control Properties dialog, choose OK.

Boosting Search Results in the JS UI:
• From the file system, browse to the Training views folder, your Sitecore instance folder>\Website\Views\Training
• Open your "LaunchJsExpressions.cshtml" file
• Add these lines at the end to add a Query Ranking Expression (QRE)
<div id="articlesRankingExpression" class="CoveoForSitecoreRankingExpression"></div>
<script> document.addEventListener('DOMContentLoaded', function() {
var expressionBuilder = new Coveo.ExpressionBuilder();
var fieldName = CoveoForSitecore.Context.fields.toCoveo('@@_template');
var articleTemplateShortId = 'd9019e30f95446ccaa703e928c40b5d0'; expressionBuilder.addFieldExpression(fieldName, '==', [articleTemplateShortId]);
var qre = "$qre(expression: '" + expressionBuilder.build() + "', modifier: '10')"; document.getElementById('articlesRankingExpression').dataset.scRankingExpression = qre; }); </script>
• Save the file
• Then you can verify the changes
Result Ranking:
• The main difference between a search engine and a standard sorted list is the ability to sort documents based on their relevance. This is called result ranking.
• There are several tools to affect result ranking, such as the Coveo for Sitecore Ranking Editor. But before we can fully use them, we need to understand how default ranking is calculated.
• Term weighting is the first phase. It will add a score to a document based on its proximity with the terms in the user’s query. The proximity will be higher if the terms are found:
a. In the title
b. In the summary
c. In the concepts
d. In the address (URI)
e. With formatting (bold, italic, title...)
f. In upper case
g. Close to their stemming classes (e.g.: universe/university)
• Document weighting will judge the quality of the top 50,000 documents of the previous phase. The most important qualities are:
a. Last modified date
b. Quality evaluation at indexing time
c. Matches the user language
d. Query matches the document title
e. Collaborative Rating: User-Based Rating
f. Term Frequency: Frequency of the terms in the free-text searchable body
g. Term Proximity: Proximity between the searched terms in the documents
• The ranking is only done by the index when sorting results by relevance.
• Once all the phases are applied, each document will have a score. The search results are sorted by this score.
Debug Panel:
• In the Experience Editor, hold the ALT key and double-click beside one of the results of the result list. A modal box will appear on the page.

• Using that, for example, you can check “Enable query debug” on the window that appears, to be able to see ranking info of your query, where the total weight defines its position on the result list.

Coveo also offers a Search API. The Search API is in charge of authenticating users. This is normally done using either:
1. An API key
2. A search token (method used by Coveo for Sitecore)
3. SAML
4. Claims

On each request sent by the browser, the Search UI will add the configured token in the request’s header. In Coveo for Sitecore, an API Key is automatically created in the Coveo Cloud organization when it is linked at installation time. It is encrypted and stored in the Coveo.CloudPlatformClient.Custom.config file. It is used to:

Remotely configure the Coveo Cloud organization
Request search tokens for individual visitors to the website

You don't have to manually create any additional API Key.

Allows you to perform authenticated and secure search. JWT tokens (JSON web token) are:

Unique for each user.
Short lived (15 minutes).
Cryptographically signed so that they cannot be tampered on the client side.
Act as credentials for the user
Are available in the page served by the web server:
- Using Coveo, you can also personalize the search layout. E.g. you can hide some components for a target audience. To do so, you should navigate to the experience editor in Sitecore, and select your component. Then click on the personalize component button. After that a dialog will be open in which you can create a new condition, give it a name and edit its rule.
- In the Create rule dialog
1. Search for "pattern"
2. Click the "where the current visit matches the specific pattern card in the specific profile" condition under the Visit category
3. In the Edit rule part at the bottom, click these links: i. Last "specific" link: Select the "Audience Segment" ii. First "specific" link: Select the "Digital Marketer"
4. Choose OK.

Analytics:

You can enable analytics fields for indexed items. To do so, in the file system, navigate to \Website\App_Config\Include\Training, and create a file called “Analytics.config”. Then add the following code to the file:

<coveo>

</coveo>

</sitecore>

</configuration>

Then you need to rebuild the indexes.
We previously activated Usage Analytics by inserting the “Coveo For Sitecore Analytics” component to the page. Please note that logging occurs only for visits on the published site. Visits from the experience editor are not logged.
The search page will send events in 4 categories:

Search
Click
Page View
Custom

Each event has a large amount of context metadata

You can see the reports in Coveo Cloud Console.

Europe

North America

Latin America

Asia Pacific

Middle East and North Africa

A technical introduction to Coveo for Sitecore

Recent cases and insights

Case

USDA

Case

Findbolig

Case

Copenhagen Card

Insight

Skift Global Forum recap: The trends redefining travel

Contact us

A technical introduction to Coveo for Sitecore

Share

Recent cases and insights

Case

USDA

Case

Findbolig

Case

Copenhagen Card

Insight

Skift Global Forum recap: The trends redefining travel

Contact us