Category Archives: Development

SharePoint Search {User.Property} Query Variables and Scalability

This was something I stumbled into when working on a large global Intranet (>100k user platform) being built on SharePoint 2013. This is a WCM publishing site using “Search Driven” content leveraging Managed Metadata tagging combined with {User.Property} tokens to deliver “personalised” content. Now .. if there were 2 “buzz words” to market SharePoint 2013 they would be “search driven content” and “personalised results”, so I was surprised at what I found.

The Problem

So we basically found that page load times were >20 seconds and our SharePoint Web Servers were maxed out at 100% CPU usage. The load test showed that performance was very good with low load, but once we started ramping the load up CPU usage went up extremely quickly and rapidly ended up being almost un-usable.

It is worth bearing in mind that this is a completely “cloud friendly” solution, so zero server-side components, using almost exclusively “out of the box” web parts (mostly “Search Result Web Parts”, they would have been “Content by Search” but this was a Standard SKU install). We also use Output caching and blob caching, as well as minified and cached assets to slim down the site as much as possible,

Also worth noting that we have 10 (ten) WFE servers, each with 4 CPU cores and 32GB RAM (not including a whole battery of search query servers, index servers, and other general “back-end” servers). So we weren’t exactly light on oomph in the hardware department.

We eventually found it was the search result web parts (we have several on the home page) which were flattening the web servers. This could be easily proved by removing those web parts from the page and re-running our Load Tests (at which point CPU load dropped to ~60% and page load times dropped to 0.2 sec per page even above our “maximum capacity” tests).

What was particularly weird is that the web servers were the ones maxing out their CPU. The Search Query Component servers (dedicated hardware) were not too heavily stressed at all!

Query Variables anyone?

So the next thing we considered is that we make quite liberal use of “Query Variables” and in particular the {User.Property} ones. This allows you to use a “variable” in your Search Query which is swapped out “on the fly” for the values in that user’s SharePoint User Profile.

In our example we had “Location” and “Function” in both content and the User Profile Database, all mapped to the same MMS term sets. The crux of if is that it allows you to “tag” a news article with a specific location (region, country, city, building) and a specific function (e.g. a business unit, department or team) and when users hit the home page they only see content “targeted” at them.

To me this is what defines a “personalised” intranet .. and is the holy grail of most comms teams

However, when we took these personalisation values out (i.e. replacing {User.Location} with some actual Term ID GUID values) performance got remarkedly better! We also saw a significant uplift in the CPU usage on our Query Servers (so they were approaching 100% too).

So it would appear that SOMETHING in the use of Query Variables was causing a lot of additional CPU load on the Web Servers!

It does what??

So, now we get technical. I used JetBrains “DotPeek” tool to disassemble some of the SharePoint Server DLLs to find out what on earth happens when a Query Variable is passed in.

I was surprised at what I found!

I ended up delving down into the Microsoft.Office.Server.Search.Query.SearchExecutor class as this was where most of the “search” based activity went on, in particular in the PreExecuteQuery() method. This in turn referred to the Microsoft.SharePoint.Publishing.SearchTokenExpansion class and its GetTokenValue() method.

It then hits a fairly large switch statement with any {User.Property} tokens being passed over to a static GetUserProperty() method, which in turn calls GetUserPropertyInner(). This is where the fun begins!

The first thing it does is call UserProfileManager.GetUserProfile() to load up the current users SharePoint profile. There doesn’t appear to be any caching here (so this is PER TOKEN instance. If you have 5 {user.property} declarations in a single query, this happens 5 times!).

The next thing that happens is that it uses profile.GetProfileValueCollection() to load the property values from the UPA database, and (if it has the IsTaxonomic flag set) calls GetTaxonomyTerms() to retrieve the term values. These are full-blown “Term” objects which get created from calls to either TaxonomySession.GetTerms() or TermStore.GetTerms(). Either way, this results in a service/database roundtrip to the Managed Metadata Service.

Finally it ends up at GetTermProperty() which is just a simple bit of logic to build out the Keyword Query Syntax for Taxonomy fields (the “#0” thing) for each Term in your value collection.

So the call stack goes something like this:

SearchExecutor::PreExecuteQuery()
=> SearchTokenExpansion::GetTokenValue()
=> GetUserProperty()
=> GetUserPropertyInner()
=> UserProfileManager::GetUserProfile()
=> UserProfile::Properties.GetPropertyByName().CoreProperty.IsTaxonomic
If it is (which ours always are) then …
=> UserProfile::GetProfileValueCollection()::GetTaxonomyTerms()
=> TermStore::GetTerms()
Then for each term in the collection
=> SearchTokenExpansion::GetTermProperty()
This just builds out the “#0” + term.Id.ToString() query value

So what does this really mean?

Well lets put a simple example here.

Lets say you want to include a simple “personalised” search query to bring back targeted News content.

{|NewsFunction:{User.Function}} AND {|NewsLocation:{User.Location}}

This looks for two Search Managed Properties (NewsFunction and NewsLocation) and queries those two fields using the User Profile properties “Function” and “Location” respectively. Note – This supports multiple values (and will concatenate the query with “NewsFunction: OR NewsFunction:” as required)

On the Web Server this results in:

  • 2x “GetUserProfile” calls to retrieve the user’s profile
  • 2x “GetPropertyByName” calls to retrieve the attributes of the UPA property
  • 2x “GetTerms” queries to retrieve the term values bound to that profile

And this is happening PER PAGE REFRESH, PER USER.

So … now it suddenly became clear.

With 100k users hitting the home page it was bottlenecking the Web Servers because every home page hit resulted in double the amount of server-side lookups to the User Profile Service and Managed Metadata Service (on top of all of the other standard processing).

So how to get round this?

The solution we are gunning for is to throw away the Search Web Parts and build our own using REST calls to the Search API and KnockoutJS for the data binding.

This allows us to use client-side caching of the query (including any “expanded” query variables, and caching of their profile data) and we can even cache the entire search query result if needed so “repeat visits” to the page don’t result in additional server load.

Finally…
This was a fairly high profile investigation, including Microsoft coming in for a bit of a chat about some of the problems we’re facing. After some investigation they did confirm another option (which didn’t work for us, but useful to know) which is this:

  • Query Variables in the Search Web Part are processed by the Web Server before being passed to the Query Component
  • The same query variables in a Result Source or Query Rule will be processed on the Query Server directly!

So if you have a requirement which you can compartmentalise into a Query Rule or Result Source, you might want to look at that approach instead to reduce the WFE processing load.

Cheers! And good luck!

Customising the Content Search Web Part – Part 4 – Packaging & Deployment in Visual Studio

This is the final post in a series I have been writing on the Content by Search Web Part (aka CSWP).

  1. What you get in the box
  2. Custom Display Templates with JavaScript
  3. Going Old Skool with XSLT
  4. Packaging & Deployment in Visual Studio (this post)

So in this final post we will be looking at how your Content Search Web Parts can be deployed as WSP packages in Visual Studio.

The first thing you will need to decide is:

Do you deploy the HTML Designer File?

 

Or just the final JS file?

This was really triggered with a discussion I had with Chris O’Brien (@ChrisO_Brien) when we got talking about Content Search Web Parts and what the best approach was for deploying them.

In my opinion it really comes down to what environment you are deploying to, and whether the admin / power users will need to have access to a (much easier to modify) designer file.

Deploying the JS File

This is definitely the easiest approach as it doesn’t really involve anything too complicated. You would still start off with your HTML Designer file and deploy it to your DEV box, but you would then extract the compiled JS file and drop THAT file into Visual Studio.

You can then deploy it using a standard Module element.

<?xml version="1.0" encoding="utf-8"?>

<Elements xmlns="https://schemas.microsoft.com/sharepoint/">

  <Module Name="MJHDisplayTemplates" Url="_catalogs/masterpage/Display Templates/Content Web Parts" Path="MJH Display Templates" RootWebOnly="TRUE">

    <File Url="Item_MJHSummary.js" Type="GhostableInLibrary" />

  </Module>

</Elements>

Deploying the HTML Designer File

This is a little bit more tricky. The main problem is that the JS file is compiled “on the fly” by an SPItemEventReceiver in the Master Page and Page Layouts Gallery. Of course, event receivers do not get fired when a file is dropped in from a module from a feature, so you basically need to go and give SharePoint a prod to make it “do its thing”.

My approach is to use a Feature Receiver to “touch” the file (which prompts the event receiver to fire) so that your JS file is then compiled.

In order to make this more dynamic we will inject the Feature ID as a property of the SPFile which is actually provisioned by the module. Thankfully this is a relatively easy thing to achieve.

<?xml version="1.0" encoding="utf-8"?>

<Elements xmlns="https://schemas.microsoft.com/sharepoint/">

  <Module Name="MJHDisplayTemplates" Url="_catalogs/masterpage/Display Templates/Content Web Parts" Path="MJH Display Templates" RootWebOnly="TRUE">

    <File Url="Item_MJHSummary.html" Type="GhostableInLibrary">

      <Property Name="FeatureId" Value="$SharePoint.Feature.Id$" Type="string"/>

    </File>

  </Module>

</Elements>

The trick then is to have a Feature Receiver which looks for all of the files which have that property and modify the file in some way (I just pull the file bytes and push it back again, basically uploading a duplicate copy of the file, just calling SPListItem.Update() or SPFile.Update() didn’t seem to work!).

string featureId = properties.Feature.Definition.Id.ToString();

SPSite site = properties.Feature.Parent as SPSite;

SPFolder displayTemplateFolder = rootWeb.GetFolder("_catalogs/masterpage/Display Templates/Content Web Parts");

if(displayTemplateFolder.Exists)

{

    SPList parentList = folder.ParentWeb.Lists[folder.ParentListId];

 

    SPFileCollection files = folder.Files;

    var templateFiles = from SPFile f

                          in files

                          where String.Equals(f.Properties["FeatureId"] as string, featureId, StringComparison.InvariantCultureIgnoreCase)

                          select f;

 

    List<Guid> guidFilesToModify = new List<Guid>();

    foreach (SPFile file in templateFiles)

    {

        guidFilesToModify.Add(file.UniqueId);

    }

 

    foreach (Guid fileId in guidFilesToModify)

    {

        // instantiate new object to avoid modifying the collection during enumeration

        SPFile file = parentList.ParentWeb.GetFile(fileId);

 

        // get the file contents

        byte[] fileBytes = file.OpenBinary();

 

        // re-add the same file again, forcing the event receiver to fire

        folder.Files.Add(file.Name, fileBytes, true);

    }

}

So in the above code sample (which is inside a “Feature Activated” method) we are retrieving the Feature ID for the feature which is activating. We then proceed to the folder where we provisioned our files and did a simple query to pull out those files which have the feature ID in their properties (which we set in our module above).

We then pull the binary data of the file as a Byte Array, and then push exactly the same file back into the folder (which triggers the event receiver to fire).

And that should be all you need!

Customising the Content Search Web Part – Part 3 – Going old Skool with XSLT

This is the third post in a series I will be writing on the Content by Search Web Part (aka CSWP).

  1. What you get in the box
  2. Custom Display Templates with JavaScript
  3. Going Old Skool with XSLT (this post)
  4. Packaging & Deployment in Visual Studio

Now I am admittedly going to cop out here. I was originally intending to write this up in detail but to be honest it has already been done (very well) before.

So .. I would invite you to read the most excellent blog post from Waldek Mastykarz (@waldekm).

Using server-side rendering with Content Search Web Part in SharePoint 2013

He not only shows you how to hook up your JS Display Template with a server-side XSL file, but also shows you how to deploy the files and it’s primary usage (as a Search Crawler output).

If you want to use this method for all of your web requests then you can set the AlwaysRenderOnServer property of the CSWP to “true” and it will always use your XSL template file.

 

Embedding Sandbox Web Parts into Page Layouts and Master Pages

Note – this is applicable to both SharePoint 2010 and SharePoint 2013 .. and despite rumours of “The Sandbox is deprecated” many features in SharePoint 2013 rely on the Sandbox so I don’t see it going anywhere for the time being

This is something that I have been told many times, it is not possible to embed a server control developed in the Sandbox onto a Page Layout or Master Page because you can’t add the appropriate Tag Prefix to the header.

If you consider the standard way of doing this for a Farm / Full Trust solution:

<% @Register TagPrefix="MJH" Namespace="MJH.Examples.WebControls" Assembly="MJH.Examples, Version=1.0.0.0, Culture=neutral, PublicKeyToken=16e05f568fffc0d1" %>

<!-- Custom Web Control-->
<MJH:SampleCustomControl runat="server" />

In the sandbox we obviously can’t reference the Assembly in the same way because (a) it isn’t deployed to the GAC / BIN folders and (b) it runs in a completely different process so won’t be JIT compiled by the .Net Framework… but there is a solution!

Microsoft.SharePoint.WebPartPages.SPUserCodeWebPart allows you to embed a Web Part from an assembly running in the Sandbox.

<!-- custom sandbox web part -->
<WebPartPages:SPUserCodeWebPart 
      runat="server" 
      AssemblyFullName="MJH.Examples.Sandbox, Version=1.0.0.0, Culture=neutral, PublicKeyToken=60d6fc7a560ae45c" 
     SolutionId="4faf8ff6-36ea-406f-af25-4d89472a41e1" 
     TypeFullName="MJH.Examples.Sandbox.WebParts.SampleCustomControl" >
</WebPartPages:SPUserCodeWebPart> 

You still need the “AssemblyFullName” attribute. I’ve found the easy way to get this is to add a feature receiver to a Feature in Visual Studio then copy-paste it from the Manifest in Visual Studio (which contains the full text that you need).

So thats it! Happy coding

Provisioning List Views in the onet.xml with custom Web Part properties

I’ve seen this come up numerous times (with a few people telling me “its not possible without writing code”) so I thought I’d chuck up a simple code sample showing you how this is done in SharePoint 2013.

Lets say you want to add some List View Web Parts to a custom Team Site for a Tasks list:

  • My Tasks
  • Tasks In Progress
  • Tasks Overdue

You would initially do this kind of thing in your onet.xml:

<View List="Lists/Tasks" BaseViewID="1" WebPartZoneID="Header" />
<View List="Lists/Tasks" BaseViewID="2" WebPartZoneID="Header" />
<View List="Lists/Tasks" BaseViewID="3" WebPartZoneID="Header" />

Of course the main problem here is that you have zero control over any of the Web Part properties (such as the Title, border, or anything else). The default title will use the same title as the list, which of course (when you have the same list more than once) offers the stunningly un-useful:

  • Tasks (1)
  • Tasks (2)
  • Tasks (3)

The solution is to include some additional Web Part properties in embedded CDATA tags:

<View List="Lists/Tasks" BaseViewID="1" WebPartZoneID="Header"> <![CDATA[

<webParts>
      <webPart xmlns="https://schemas.microsoft.com/WebPart/v3">
          <metaData>
              <type name="Microsoft.SharePoint.WebPartPages.XsltListViewWebPart, Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
              <importErrorMessage>Cannot import this Web Part.</importErrorMessage>
          </metaData>
          <data>
              <properties>
                  <property name="Title">My Tasks</property>

              </properties>
          </data>
      </webPart>
</webParts>
]]> </View>

This gives you effectively complete control over any of the Web Part Properties (see XsltListViewWebPart Properties).

Happy coding!