Tag Archives: SharePoint 2010

Search Core Results Web Part with Dynamic Date and User Profile Tokens

If you just want the goodies then you can get them here:

The Big Fat Disclaimer – This has not been thoroughly tested for a production environment. I have also removed references in my snippets below to caching and error handling to try and keep it brief. The downloadable version uses both caching and error handling, but it is still really just a proof of concept and you should TEST it before you deploy it! I take no responsibility if your production servers blow up!

I must have seen this requirement dozen of times on different projects, having search results which either:

  • Filter using a User Profile Property of the current user
  • Filter using a dynamic date range (e.g. using the “TODAY” token)
  • Specify the Sort-By (which is normally restricted to either “Relevance” or “Modified Date”

The requirement for functionality of this nature come up extremely frequently on Intranet projects. For example:

“Show News Articles from the past 7 days which filter based on the user’s location”
“Show events coming up in the next 3 months”
“Show discussions / wiki entries / blog posts which include the current user’s Ask Me About values”

FixedQuery used in the Web Part:
Author:[UPP-PreferredName] AND Write:[TODAY-180]..[TODAY] AND ContentType:Event

Well .. on my current client project these very requirements came up .. so this time I decided to knock together the basics of the web part in my spare time and then “donate” it to the project… and this post describes how I built it, what goes on under the hood, and also includes both the source code as well as a downloadable WSP package with the working Web Part in it.

Step 1 – Extending the Search Core Results Web Part
So .. to get us started, lets kick off by creating our actual Web Part. I am going to be extending the Search CoreResultsWebPart (link to MSDN).
This is easy enough to achieve by simply creating a new Web Part in Visual Studio and inheriting from the CoreResultsWebPart class. This will make sure our web part gets all of the functionality and properties that the normal Search Results web part does without any additional effort.

   1:  [ToolboxItemAttribute(false)]
   2:  public class ExtendedSearchWebPart : CoreResultsWebPart
   3:  {
   5:  }

That is the easy bit …

Step 2 – Overriding the Query and SortOrder
Now the next bit to tackle is how to override the actual query that gets executed. Well the best place to do this is to override the ConfigureDataSourceProperties method. This method gets called before the query is actually executed against the Search engine itself.

You can then leverage the CoreResultsWebPart.DataSource property (which is of type CoreResultsDataSource). This is what allows all of the magic to happen.

   1:  protected override void ConfigureDataSourceProperties()
   2:  {
   3:      // only perform actions when we are trying to show search results
   4:      // i.e. not when you're in Design Mode
   5:      if (this.ShowSearchResults)
   6:      {
   7:          // call the base web part method
   8:          base.ConfigureDataSourceProperties();
  10:          // get the data source object
  11:          CoreResultsDatasource dataSource = this.DataSource as CoreResultsDatasource;
  13:          // override the query being executed
  14:          dataSource.Query = "Author:\"Martin Hatch\"";
  16:          // remove the original sort order
  17:          dataSource.SortOrder.Clear();
  18:          dataSource.SortOrder.Add("Title", Microsoft.Office.Server.Search.Query.SortDirection.Ascending);
  19:      }
  20:  }

So lets talk through the code above.

First off we want to make sure we are only executing our custom code when we are actually trying to retrieve search results. This is a fail-safe block as some instances this will be false (such as when you are editing the web part or if you are viewing the “Design” view in SharePoint Designer). We also need to call the base method (as you typically would when overriding a method call!).

Then things get interesting. Line 11 has us create our “CoreResultsDataSource” object from the local “DataSource” property. This has two properties which we are modifying:

Query (line 14) – This allows us to change or completely override the query which is being executed. This will be the entire query including the Fixed Query, Appended Query and whatever the user typed into their search box (if you are using this on a Search Results page). In my example above I am simply overriding the result so that it just searches for items created by “Martin Hatch” (me!)

SortOrder (lines 17 and 18) – This allows us to override the sort order, allowing us to select ANY indexed Search Property you want (I expect excluding rich text fields of course!). In my example, I am sorting by Title in Ascending order.
This can then be easily extended to provide custom Web Part properties to allow the Sort functionality to be specified by the page editor.

Step 3 – Making it re-usable Part 1 – Dynamic Date ranges
 So now that we can override the query easily we can move on to adding some of the good stuff. I decided to go with a relatively simple Token Replacement function using a simple [TODAY] token to represent the current date:

  • [TODAY] (todays date)
  • [TODAY+7] (today plus 7 days)
  • [TODAY-7] (today minus 7 days)

So .. how do we code this in? Well .. I am quite lazy and don’t really get on with regular expressions (if you are reading this and you are a RegEx guru.. by all means download the source code, refactor it and send it back, cheers!).

So I started off by creating a bunch of class level constants which I would use to recognise the tokens that we are looking for above:

private const string TODAY_PLACEHOLDER = "[TODAY]";
private const string TODAY_ADD_STARTSTRING = "[TODAY+";
private const string TODAY_SUBTRACT_STARTSTRING = "[TODAY-";
private const string TOKEN_ENDSTRING = "]";

The following code can then be swapped out for our ConfigureDataSourceProperties method.

   1:  protected override void ConfigureDataSourceProperties()
   2:  {
   3:      // only perform actions when we are trying to show search results
   4:      // i.e. not when you're in Design Mode
   5:      if (this.ShowSearchResults)
   6:      {
   7:          // call the base web part method
   8:          base.ConfigureDataSourceProperties();
  10:          // get the data source object
  11:          CoreResultsDatasource dataSource = this.DataSource as CoreResultsDatasource;
  13:          // get the current Fixed Query value from the web part
  14:          string strQuery = this.FixedQuery;
  16:          // swap out the exact "today" date
  17:          if (strQuery.IndexOf(TODAY_PLACEHOLDER) != -1)
  18:          {
  19:              strQuery = strQuery.Replace(TODAY_PLACEHOLDER, DateTime.UtcNow.ToShortDateString());
  20:          }
  22:          // perform all of the "Add Days" calculations
  23:          while (strQuery.IndexOf(TODAY_ADD_STARTSTRING) != -1)
  24:          {
  25:              strQuery = CalculateQueryDates(strQuery, TODAY_ADD_STARTSTRING, true);
  26:          }
  28:          // perform all of the "Remove Days" calculations
  29:          while (strQuery.IndexOf(TODAY_SUBTRACT_STARTSTRING) != -1)
  30:          {
  31:              strQuery = CalculateQueryDates(strQuery, TODAY_SUBTRACT_STARTSTRING, false);
  32:          }
  34:          // swap out the Fixed Query for our Calculated Query
  35:          dataSource.Query = dataSource.Query.Replace(this.FixedQuery, strQuery);
  36:      }
  37:  }

This then calls the CalculateQueryDates support method which I put together:

   1:  private static string CalculateQueryDates(string strQuery, string startStringToLookFor, bool AddDays)
   2:  {
   3:      try
   4:      {
   5:          // get the index of the first time this string appears
   6:          int firstIndex = strQuery.IndexOf(startStringToLookFor);
   8:          // get the text which appears BEFORE this bit
   9:          string startString = strQuery.Substring(0, firstIndex);
  11:          // get the text which appears AFTER this bit
  12:          string trailingString = strQuery.Substring(firstIndex);
  13:          int endIndex = trailingString.IndexOf(TOKEN_ENDSTRING);
  14:          if (endIndex + 1 == trailingString.Length)
  15:          {
  16:              // there is nothing else after this
  17:              trailingString = "";
  18:          }
  19:          else
  20:          {
  21:              trailingString = trailingString.Substring(endIndex +1);
  22:          }
  24:          // find the number of days
  25:          string strDays = strQuery.Substring(firstIndex + startStringToLookFor.Length);
  26:          strDays = strDays.Substring(0, strDays.IndexOf(TOKEN_ENDSTRING));
  27:          int days = int.Parse(strDays);
  29:          // re-construct the query afterwards
  30:          if (AddDays)
  31:          {
  32:              strQuery = startString + DateTime.UtcNow.AddDays(days).ToShortDateString() + trailingString;
  33:          }
  34:          else
  35:          {
  36:              // subtract days
  37:              strQuery = startString + DateTime.UtcNow.AddDays(0 - days).ToShortDateString() + trailingString;
  38:          }
  40:          return strQuery;
  41:      }
  42:      catch (FormatException ex)
  43:      {
  44:          throw new FormatException("The format of the [TODAY] string is invalid", ex);
  45:      }
  46:      catch (ArgumentNullException ex)
  47:      {
  48:          throw new FormatException("The format of the [TODAY] string is invalid. Could not convert the days value to an integer.", ex);
  49:      }
  50:  }

So you should be able to see we are using simple String.IndexOf() method calls to find out if our Tokens are present.

If they are then we simply calculate the DateTime value based on the static DateTime.UtcNow property and use String.Replace() methods to swap out these into our query text.

When we are using [TODAY+X] or [TODAY-X] we simply use DateTime.UtcNow.AddDays(X) or DateTime.UtcNow.AddDays(0-X) and use the same String.Replace() method.

The search syntax is exactly the same as it was previously, and the Keyword Syntax is very powerful.

Example: Using [TODAY] Token query syntax

Write:[TODAY] – this will return all items that were modified today
Write>[TODAY-7] – this will return all items that were modified in the past week
Write:[TODAY-14]..[TODAY-7] – this will return all items that were modified between 2 weeks ago and 1 week ago

So we already have a powerful and reusable search component .. but there is more!

Step 4 – Making it re-usable Part 2 – Dynamic User Profile Properties
The next one is to allow us to pull in User Profile Properties so that we can start doing searches based on the current user’s profile values.
For this we needed to create new replacable Tokens, for which I decided to use:

  • [UPP-{User Profile Property Internal Name}]
  • [UPP-PreferredName] (swaps out for the users name)
  • [UPP-SPS-Responsibility] (swaps out for their “Ask Me About” values)
  • etc ..

So .. we add another class level constant (same as we did for our DateTime tokens)

private const string USER_PROFILE_PROP_STARTSTRING = "[UPP-";

We can then use this in our code, in exactly the way we did before (using String.IndexOf(), String.SubString() and String.Replace() methods).

So we add the following additional code to our ConfigureDataSourceProperties method;

   1:  if (dataSource.Query.IndexOf(USER_PROFILE_PROP_STARTSTRING) != -1 &&
   2:      UserProfileManager.IsAvailable(SPServiceContext.Current))
   3:  {
   4:      string strQuery = dataSource.Query;
   6:      while (strQuery.IndexOf(USER_PROFILE_PROP_STARTSTRING) != -1)
   7:      {
   8:          strQuery = ReplaceUserProfilePropertyTokens(strQuery);
   9:      }
  11:      if (strQuery != dataSource.Query)
  12:      {
  13:          dataSource.Query = strQuery;
  14:      }
  15:  }

This uses the additional method call ReplaceUserProfilePropertyTokens which is shown below:

   1:  private static string ReplaceUserProfilePropertyTokens(string strQuery)
   2:  {
   3:      // retrieve the current user's Profile
   4:      UserProfileManager upm = new UserProfileManager(SPServiceContext.Current);
   5:      UserProfile profile = upm.GetUserProfile(false);
   7:      if (profile == null)
   8:      {
   9:          throw new ApplicationException("The current user does not have a User Profile");
  10:      }
  12:      // extract the user profile property name from the token
  13:      int startIndex = strQuery.IndexOf(USER_PROFILE_PROP_STARTSTRING);
  14:      string strPropertyName = strQuery.Substring(startIndex + USER_PROFILE_PROP_STARTSTRING.Length);
  15:      strPropertyName = strPropertyName.Substring(0, strPropertyName.IndexOf(TOKEN_ENDSTRING));
  17:      string strToReplace = strQuery.Substring(startIndex);
  18:      strToReplace = strToReplace.Substring(0, strToReplace.IndexOf(TOKEN_ENDSTRING) + 1);
  20:      try
  21:      {
  22:          // get the value
  23:          UserProfileValueCollection propertyValue = profile[strPropertyName];
  24:          string strValues = String.Empty;
  26:          foreach (object propValue in propertyValue)
  27:          {
  28:              if (propValue.ToString().IndexOf(" ") == -1)
  29:              {
  30:                  strValues += propValue.ToString() + " OR ";
  31:              }
  32:              else
  33:              {
  34:                  strValues += "\"" + propValue.ToString() + "\" OR ";
  35:              }
  36:          }
  38:          if (strValues.Length > 0)
  39:          {
  40:              strValues = strValues.Substring(0, strValues.Length - 4);
  41:          }
  43:          // swap the value out in the query
  44:          strQuery = strQuery.Replace(strToReplace, strValues);
  46:      }
  47:      catch (ArgumentException ex)
  48:      {
  49:          throw new FormatException("The User Profile Property specified in your UPP token does not exist", ex);
  50:      }
  51:      return strQuery;
  52:  }

So there are a few things to point out here which might trip you up:

  • We are using the UserProfileManager.IsAvailable() method to find out if we have a user profile service application provisioned and assigned to the current Web Application.
  • At the moment this code throws an error if the current user doesn’t have a User Profile. You may want to handle this differently for your environment?
  • Handling of multi-value fields. At the moment all we do is take the string values and concatenate them with “OR” in the middle. So if you had “Value1; Value2” as your property value the Token replacement would put “Value1 OR Value2” as the search query.

As long as the content editors are aware of the behaviour this allows us to create quite complex queries.

Example if we now used the Fixed Query:

([UPP-SP-Responsibility]) AND Write:[TODAY-14]..[TODAY]

Then for a user who’s “Ask Me About” properties were “SharePoint” and “IT Administration” then the resulting Search Query would be:

(SharePoint OR “IT Administration”) AND Write:13/05/2012..17/05/2012

If another user comes along whose “Ask Me About” property was just set to “Marketing” then the resulting Search Query would be:

(Marketing) AND Write:13/05/2012..17/05/2012

This is without changing any of the web part properties, and allows us to drive dynamic content from a single web part to our entire user base.

Hopefully you can see that this is incredibly powerful and flexible.

Step 5 – Making it re-usable Part 3 – Controllable Sort By
The final step is to allow our content editors to control the “Sort By” functionality. The default OOTB webpart only allows us to sort by “Relevance” or “Last Modified”, which is fine when you are looking at general search results, but when you are building custom components (such as news, links or event feeds) you typically want to control the order by date or title or something a little more usable for the specific component.

So this bolt-in allows you to control the Sort By. First off we need to add some Web Part Properties so that the user can modify their values:

   1:  [Personalizable(PersonalizationScope.Shared)]
   2:  [WebBrowsable(true)]
   3:  [WebDescription("Sort by this managed property")]
   4:  [WebDisplayName("Managed Property")]
   5:  [Category("Sort Override")]
   6:  public string OrderByProperty { get; set; }
   8:  [Personalizable(PersonalizationScope.Shared)]
   9:  [WebBrowsable(true)]
  10:  [WebDescription("Sort direction")]
  11:  [Category("Sort Override")]
  12:  public Microsoft.Office.Server.Search.Query.SortDirection SortDirection { get; set; }

This will provide the Web Part property editing functionality:

Once we have done that, we can add the following code to our ConfigureDataSourceProperties method (yes .. this method really is where all of the grunt work goes on in this web part!)

   1:  // if OrderByProperty is not set, use default behavior
   2:  if (!string.IsNullOrEmpty(OrderByProperty))
   3:  {
   4:      // change the sortorder
   5:      dataSource.SortOrder.Clear();
   6:      dataSource.SortOrder.Add(OrderByProperty, SortDirection);
   7:  }

And that is all there is to it.

Step 6 – Enjoy!
So congratulations if you made it this far. I know this was a long blog post but thought it was worth walking through it properly.

If you have any questions, feedback or questions then please get in touch using the comments, and here are links to the downloads (which are also referenced at the top of this blog post)

Some notes about the “final” version:

  • The code structure is slightly different because the DateTime [TODAY] queries are cached using Web Part Properties for better performance
  • the [TODAY] token is case sensitive!
  • There is an extra “Debug Mode” checkbox in the Web Part Properties which when enabled spits out the entire query being executed at the bottom of the search results.
  • Code contains an “Editor Part” .. this just clears out the Cache value when the web part properties are modified

Usage Summary:

Tokens you can use are:

  • [TODAY]
  • [TODAY+X] (add X days)
  • [TODAY-X] (remove X days)
  • [UPP-{Internal Name of User Profile Property}]

Example Usage

Sample user has:
Name: Martin Hatch
Ask Me About: SharePoint; Solution Architecture; Code

ContentType:Event AND ([UPP-SPS-Responsibility]) AND Write:[TODAY-7]..[TODAY]
ContentType:Event AND (SharePoint OR “Solution Architecture” OR Code) AND Write:12/05/2012..17/05/2012
Returns all events which were updated within the past week, and contain the current user’s “Ask Me About” values.

Author:[UPP-PreferredName] IsDocument:1
Author:”Martin Hatch” IsDocument:1
Returns all documents written by the current user

Author:[UPP-PreferredName] Write>=[TODAY-14]
Author:”Martin Hatch” Write>=02/05/2012
Returns all content created by the current user and updated within the past 2 weeks

My top missing features in Windows Phone 7 – SharePoint Contacts, Tasks and Calendars

I have been an advocate of Windows Phone 7 (WP7) for quite some time now, especially having owned and used a WP7 device for over 9 months now. However, there are some things that I really do find lacking.

Microsoft have been pushing the SharePoint integration with WP7 quite hard, especially with the recent announcements for SharePoint Online to support BCS (which basically opens up the door to have WCF calls made from your Office 365 SharePoint Online site, so you could do two-way WP7 applications which integrate tightly with your SharePoint applications!).

However, there are some features which I think would really concrete this device for small businesses as THE device to have.

Let me give you two example quotes from small businesses who were looking to implement Office 365 / SharePoint Online with a Windows Phone 7;

Plumbing Business – We would like to use SharePoint task lists to document jobs and push those tasks to our plumbers using their phone.
Our problem is that when the plumber updates the task, we want to it write back to SharePoint..” 

Pub Landlords – We want to use SharePoint Online to track and store all of our suppliers and contacts that we use, but how do I get those contacts onto my phone? If one of my suppliers calls, how do I know who they are?”

These are just examples of some of the reasons why I think WP7 is missing a trick.

Missing Feature 1 – Use SharePoint Contacts as Phone Book
At the moment there is no way for my WP7 device to use a SharePoint contact list as a phone book. This is a MAJOR piece of functionality I have been asked about by the last 3 clients who were implementing SharePoint Online.

They have all of their contacts centralised and through the web. They can bring them into the desktop Outlook application (with even two-way synchronisation!) but they can’t even get them read-only on their phone.

You really should be allowed to “link” one or more SP contact lists with your phone book, so that when someone in your sales team get a phone call the contact details pop-up.

Missing Feature 2 – WP7 overlay with SharePoint Calendar
Pretty similar story here for the calendar. Person is out on site and wants to engage with their {Supplier | Customer | Partner}. They get asked simple requests like “can you book a meeting room for us next week?”

How awesome would it be to be able to bring up a SharePoint Calendar being used for resource bookings, central meetings or events, and overlay it with your own personal calendars (so that you get notifications and can view everyhing in one calendar).

They already do this with Hotmail / Live accounts, so why not SharePoint? This really is a must .. the ability to invite / write to that Calendar from the standard Calendar interface would be a bonus!

Missing Feature 3 – Working with SharePoint Tasks
This is a standard one here .. and really works on so many different levels;

  • I want to see my tasks in a SharePoint List in my WP7 calendar
  • I want to get notifications on my WP7 when a SharePoint task is overdue
  • I want to be able to update the details of a SharePoint task while I am roaming, and people in the office to see those details straight away

If you can wire in the SharePoint workflow / events to the Task list this suddenly becomes very very powerful! You could build entire business centric applications using nothing other than some centrally controlled Task Lists, some Workflow (which you knocked up in SharePoint Designer in a couple of days) and an off-the-shelf WP7 device!


This really is the tip of the iceberg, I could go on and on, but I really think that until this gets fixed the WP7 will continue to be nothing more than a decent consumer device, with little to offer to businesses beyond what other handsets are doing.

Any Android / iPhone / Blackberry (or lets face it .. 10 year old Nokia) can synchronise your Exchange Mailbox … it is the SharePoint (and other LOB) integration that will make WP7 an “Enterprise” device!!

Summary of SharePoint Saturday UK 2011

Well, this was actually my first SharePoint Saturday experience and I have to say I was massively impressed! The whole day was very well organised and felt like other SharePoint conferences I have been to in the past (with a great variety of the sessions available and excellent quality and depth of the content being presented).

I actually brought a friend with me to SPSUK and he is mostly looking at Office 365 and Windows Phone technologies so that ended up being one of my main focuses as well. I also spent some time prepping (and packing away) from my session, as well as some time in the “Ask the Experts” room (where I met @MossLover and @SharePointBuzz for the first time :)) so I didn’t get round to as many sessions as I would have otherwise liked.

Configuring Kerberos in a SharePoint 2010 Farm (#SPSUK06)
I was up first presenting this session and I was very pleased with how it went. We had a great turnout, some really good questions and (to my relief) all of the demos worked really well first time! 🙂 This was a re-run of my SUGUK session in August on the same subject and I’m quite pleased with how the session shaped up.

It was very nice getting people coming up to me during the breaks, in the Ask the Experts session or even on twitter and email afterwards (asking questions, or just telling me how much they enjoyed the session) .. these kind of touch points really make the whole thing worth while 🙂

If you are looking for my slide decks then you can find them here:
They are branded for SUGUK but the content is the same so you should find everything you need 🙂

  • Download PowerPoint Slide Deck (PPTX) (zip)
  • View online using PowerPoint Web App
  • (PS – The PowerPoint Web App is powered by Office 365, so hope it works well for you. Feedback welcome!)

    Extending SharePoint 2010 LOB Apps to Windows Phone 7 (#SPSUK23)
    This was a good presentation by Chris Forbes (@chris_e_forbes) on Windows Phone 7 development and SharePoint 2010 integration. This is an area I am getting very interested in for two major reasons:

    1. Office 365 now supports BCS in SharePoint Online, so you can write “no-code” methods of calling WCF web services (which potentially allows the Windows Phone 7 “push notification” services which Microsoft host)
    2. The new “Mango” (Windows Phone 7.5) release includes back-ground tasks, which may allow a background application to respond to push notifications and execute custom code.

    This really opens up the doors in terms of having a very powerful zero-infrastructure solution leveraging both Office 365 and Windows Phone 7!

    Sort your processes with easy, effective InfoPath Forms and SharePoint Workflows (#SPSUK15)
    My final session of the day was with Ian Woodgate (@ianwoodgate) and ran through some cool InfoPath techniques (easy cascading drop-downs) and especially the InfoPath “Approval” mechanism which is being championed by Laura Rogers (@WonderLaura).

    Everytime I look at InfoPath I get more and more impressed, and with Office 365 it really does open up a lot of doors in terms of process automation, workflow and external communications without having to write any custom code (which is ideal when, in SharePoint Online, your development is limited to the SharePoint 2010 Sandbox which restricts a lot of methods).

    Steve Fox – “SharePoint and the Cloud: Crash or Convergence?”
    The end of the day was spent with Steve Fox (@redmondhockey) from Microsoft giving us some live demos of the new Windows Azure platform and some SharePoint 2010 integration (both on-premise and using SharePoint Online) as well as Windows Phone 7.

    Wrap Up
    This was a really good day. I was quite surprised at the number of people there (for a free event, all day on a Saturday) and everyone had a very relaxed non-commercial attitude to the day which was refreshing for a “conference” type event.

    I will definately be going to the next one .. and I seriously recommend that you do too!

    To sum up the day I’ll quote from my friend (@Denyerec)

    Back from , or by its other name . Very worthwhile day out.

    Scaling to 10,000 unique permissions – Part 2 – The Solution

    This follows on from my previous post; Part 1 – The Problem

    The main requirement was:

    • One SharePoint 2010 site
    • 10,000+ uniquely permissioned objects each with a different user account

    In this post we will be discussing the solution which involves programmatically creating unique permissions in a way which will scale for (what should be) well over 10,000 uniquely permissioned items…

    Introducing yet another little known SharePoint API call …

    This is only possible because of one of the new SharePoint 2010 API calls;



    This basically adds the specified SPRoleAssignment but does not create any of the Limited Access scopes on the parent objects.

    This is pretty straight forward and works in exactly the same way to normal Role Assignments in SharePoint 2010, we simply use the AddToCurrentScopeOnly() method instead of using the Add() method, for example:


       1: // fetch the Principal object which we are granting access to

       2: SPUser user = web.EnsureUser("Domain\\UserAccount");


       4: // create a Role Assignment binding

       5: SPRoleAssignment roleAssignment = new SPRoleAssignment(user);


       7: // apply contribute permissions

       8: roleAssignment.RoleDefinitionBindings.Add(

       9:     web.RoleDefinitions["Contribute"]);


      11: // grant permissions to the list item using the CURRENT SCOPE ONLY

      12: // this ensures that Limited Access scopes are NOT created

      13: // for parent objects (we're going to have to do that bit ourselves!)

      14: item.RoleAssignments.AddToCurrentScopeOnly(roleAssignment)


    It is very important to understand that you still need to grant “Limited Access” (it wasn’t put in just for laughs, it does have a purpose). Granting “Limited Access” means that the object has access to core information on parent objects to enable construction of things like the breadcrumb, and retrieval of core files needed to render the interface.

    This then means it is up to us (the developers) to go back and create each of those in a more efficient way. The problem is .. you can’t assign “Limited Access” programmatically…

    What do you mean .. I can’t assign Limited Access??

    Well, I don’t really know why they did this, but if you try and assign it programmatically (Limited Access is actually a “Permission Level” in SharePoint) you will get errors (admittedly you can’t do this through the user interface either!).

    So, the workaround (again) is to create your own permission level which includes exactly the same permissions that “Limited Access” would have granted. This is:

    • View Application Pages
    • Browse User Information
    • Use Remote Interfaces
    • Use Client Integration Features
    • Open

    You can call this anything you like (I called mine “SP Limited Access”) as long as you know what it means.

    The code to do this is as follows:


       1: internal SPRoleDefinition GetLimitedAccessRole(SPWeb web)

       2:         {

       3:             string strRoleDefinition = "SP Limited Access";


       5:             // only exists in webs with unique role definitions

       6:             if (web.HasUniqueRoleDefinitions)

       7:             {

       8:                 try

       9:                 {

      10:                     // try to retrieve the role definition

      11:                     return web.RoleDefinitions[strRoleDefinition];

      12:                 }

      13:                 catch (SPException)

      14:                 {

      15:                     // SPException means it does not exist


      17:                     // create our custom limited access role

      18:                     SPRoleDefinition roleDef = new SPRoleDefinition();


      20:                     // give it a name and description

      21:                     roleDef.Name = "SP Limited Access";

      22:                     roleDef.Description = "Identical to standard " + 

      23:                         "Limited Access rights. " + 

      24:                         "Used to provide access to parent objects of " + 

      25:                         "uniquely permissioned content";


      27:                     // apply the base permissions required

      28:                     roleDef.BasePermissions = SPBasePermissions.ViewFormPages 

      29:                         | SPBasePermissions.Open 

      30:                         | SPBasePermissions.BrowseUserInfo 

      31:                         | SPBasePermissions.UseClientIntegration 

      32:                         | SPBasePermissions.UseRemoteAPIs;


      34:                     // add it to the web

      35:                     web.RoleDefinitions.Add(roleDef);

      36:                 }


      38:                 return web.RoleDefinitions[strRoleDefinition];

      39:             }

      40:             else

      41:             {

      42:                 // try the parent web

      43:                 return GetLimitedAccessRole(web.ParentWeb);

      44:             }

      45:         }

    I’ve created my new Limited Access Permission Level .. now what?

    One thing does need to be made clear, there is absolutely no point you just creating all of the Security Scopes that SharePoint would have created (you’ll end up with the same mess we were trying to avoid in the first place).

    The solution is to create a group for all of the “Limited Access” users for that List or Web. It really is up to you whether you use Active Directory Security Groups or SharePoint Groups. I decided to use AD security groups; mainly because I didn’t want to clog up the Site Collection “groups” functionality, and didn’t want idiot Site Collection admins from removing the group members (or worse .. the groups themselves!) and breaking the site collection.

    Note – I haven’t included the code to create and modify Active Directory Security Groups here, if nothing else because there are thousands of resources out there showing you how to modify AD groups programmatically, and Code Project has a particularly good reference: Howto: (Almost) Everything In Active Directory via C#

    You will need to create a group for each parent object which has unique permissions although in my example it is only really the SPWeb (web site) that we are worried about as the libraries and folders are well within the security scope threshold.

    So we have our 20 libraries and our root web site. So in our example we would have to create 21 different AD security groups:

    • One group to store all Limited Access users for the root web site
    • 20 groups to store all Limited Access for the libraries (one for each library)

    Then, following this example you can then use the following code to grant “Limited Access” to one of the libraries (and just rinse and repeat for the other libraries and the root web site);


       1: // fetch the "SP Limited Access" role definition

       2: SPRoleDefinition limitedAccessRole = GetLimitedAccessRole(web);


       4: // get SPPrincipal object for the AD Group we created

       5: SPUser adGroup = web.EnsureUser("My Custom AD Group Name");


       7: // set the role assignments for this group

       8: SPRoleAssignment roleAssignment = new SPRoleAssignment(adGroup);

       9: roleAssignment.RoleDefinitionBindings.Add(limitedAccessRole);


      11: // grant "Limited Access" to the AD Group for this list

      12: // we only have to do this once! After this we simply 

      13: // need to add members to this AD Group every time we 

      14: // add users to one of the parent objects!

      15: list.RoleAssignments.AddToCurrentScopeOnly(roleAssignment)

    So having done this for all of the parent objects we now have our 21 custom Active Directory groups, each one of which has been granted “Limited Access” to one of the required “parent” objects for our folders.

    From here on in it should be smooth sailing. You simply need to make sure that every time you programmatically add a new user to one of the folders you also make sure they get added to the relevant AD Groups (so that the “Limited Access” chain is not broken).

    The following diagram really explains what we have done:


    I have tested this model for over 16,000 unique AD accounts across hundreds of folders in hundreds of document libraries and I cannot notice any discernable drop off in performance (nothing that can’t be explained by simply having a really large number of libraries and folders anyway!) so initial tests show that this is working very well indeed 🙂

    What I also ended up doing (to make this slightly more robust) is to build my own application page which users can use to Grant Permissions through the UI (so we don’t need to write custom code every time a new “Limited Access” scope is needed).

    I then wrote an HttpModule to auto-redirect any requests to the out-of-the-box page (_layouts/AclInv.aspx) to the custom page so that if anyone tried to use the native user interface it would ALWAYS be executing my own custom code (which creates all of the AD Groups and SP Limited Access scopes programmatically, without the user having to worry about it!)

    The great thing about this solution is that it doesn’t matter how many users or groups you are adding to your SharePoint site .. you only ever have 1 Limited Access security scope for each List / Web!

    Thanks for sticking with me through these two posts .. if you made it this far then thanks for reading and I would love to hear your comments! 🙂

    Scaling to 10,000 unique permissions – Part 1 – The Problem

    This post was borne out of a client requirement which popped up on my radar. I’m currently working for a leading global Business Intelligence provider in London, and they were looking to implement a particular third party piece of software. This software relies on SharePoint for file storage and my client wanted to roll this out to their customers “extranet” style with each customer having uniquely secured content (files and folders).

    Now .. first off their customers include in excess of 10,000 different companies (i.e. > 10,000 users) so early warning bells immediately started ringing in terms of scalability.

    Secondly, to make this worse, the software required all content to be stored in a single SharePoint site .. so now my early warning system had gone into full meltdown and a state of high alert was reached.
    So to boil this down …

    • One SharePoint 2010 site
    • 10,000+ uniquely permissioned objects each with a different user account

    A Library with 10,000 uniquely permissioned folders?? Possible? My first instincts said no… so it was time to get my problem solving hat on and do some digging ..

    Investigating the Limits of SharePoint 2010

    I would like to think that any SharePoint {Consultant | Developer | Architect | <insert profession>} worth their salt would have read the Software and Capacity Planning guidelines (or at least be aware of it!) .. so that was my first pit-stop.

    Note – I also stumbled across a great blog post by SharePoint infrastructure veteran Joel Oleson and his Best Practices for Enterprise User Scalability in SharePoint. This goes into detail about the specific size of an ACL (and the reason why this is limited, specifically in Windows) which although a good read wasn’t really relevant to my problem.

    The Microsoft TechNet article SharePoint Server 2010 capacity management: Software boundaries and limits (https://technet.microsoft.com/en-us/library/cc262787.aspx) is a great resource and contains one absolutely key entry:

    Security Scope – 1,000 per list (threshold)
    The maximum number of unique security scopes set for a list should not exceed 1,000. 

    A scope is the security boundary for a securable object and any of its children that do not have a separate security boundary defined.  

    A scope contains an Access Control List (ACL), but unlike NTFS ACLs, a scope can include security principals that are specific to SharePoint Server. The members of an ACL for a scope can include Windows users, user accounts other than Windows users (such as forms-based accounts), Active Directory groups, or SharePoint groups.

    So what is a Security Scope then? Ok I admit it does tend to get a bit bogged down in terminology.
    To put it simply … each time you grant access to a new principal (user account or group) then you are creating a new Security Scope.

    The other thing to consider pickup is that this is not just limited to lists! Any list that inherits permissions will pick up their permissions from the parent web (site) so you also need to adhere to this at the web level too!

    This means that you should not have more than 1000 security scopes at EITHER the Site or List level.

    Ignoring this limit can do real damage to your farm …

    There is even a Microsoft Knowledgebase article explaining why; SharePoint performance degradation with a large number of unique security scopes in lists (https://support.microsoft.com/kb/2420771)

    This is really explained in far more detail in two most excellent blog posts:

    The first post describes the problem of trying to create more than 1000 security scopes, and what happens when you do this: https://wbblog.datapolis.com/2011/03/setting-item-permissions-with-workflow.html

    The second post is by James Love (a.k.a. @jimmywim) and goes into real “deep dive” detail looking into the root cause of the problem (SQL Server and ACL GUIDs) and how this problem can actually bring down your ENTIRE FARM and not just the list / site you are working on!

    A quote from the second post is as follows:

    “When you load up a huge list with lots of item level permissions, a single operation gets every single GUID associated with the ACL for that item and passes that back to the data access layer of SharePoint. When the database retrieves the actual list item data, it will pass in all of the ACL Guids back in as one long string, all concatenated together. The query to get the data creates a table variable re-assembles the the item level ACL Guid associated with each item. How the rest of the query deals with this is anyone’s guess at the moment – this table variable might just be passed back to the calling COM object (though I thought they couldn’t be used this way….) for the COM object to then sort out which item should be visible to which “scope” (or ACL).

    So, what can we take away form this? Passing 640k of data about the place, for a SQL Query to do some substring math and converting to Guids will soon bring your database server to its knees. This is one request and it takes 2000ms to work. Imagine if you have 5 requests per second or more hitting this list!”

    Both are excellent appendums to this post and well worth looking at for another angle and a bit more detail!

    Why does this become my problem?

    Now .. looking back to my original problem some of you may be thinking, OK no problem; you can just create yourself 20 different lists / libraries .. and have 500 unique permissions in each list??

    Well .. so you might think .. and here I introduce the juggernaut that is Limited Access Scopes!

    Anyone who has spent any time around SharePoint security will have noticed the odd “Limited Access” permission popping up in their site from time to time. “Limited Access” is automatically allocated to a parent Folder, List or Web whenever a child object has a unique permission allocated to it.

    You can easily see these being created if you break permission inheritence to a list and just add a few accounts to that list. The parent Web will not have a “Limited Access” scope created for each user account you have added.

    Now hopefully the bright will already have spotted the problem .. it doesn’t matter how many lists or libraries you create .. every single user or group that you add will end up in the parent Web site with “Limited Access” (and every single Parent Web heading upwards).

    The following diagram explains why.

    You simply cannot get away from this fact. If are adding 10,000 unique permissions with different user accounts then you will end up with 10,000 security scopes at the root web!

    Note – It should be noted that the number of “Limited Access” scopes created is limited to the number of Security Principals you are adding.

    If you are adding from a pool of 50 users then you will only ever be adding a maximum of 50 new “Limited Access” scopes (one for each user account).

    For this reason it is a good idea to use Groups when adding permissions as this limits the number of “Limited Access” scopes which are created .. but this won’t solve your problem if you have over 1000 different security principals!

    So that was the crux of my problem .. on investigation this does look to be a major major problem (and an “impossible fix”) but it would seem not! There IS a workaround (one which I have tested to over 15,000 unique user accounts and works very very well indeed)…

    The solution, workaround, and code samples are all included in Part 2

    I’m speaking at SUGUK London

    Well I am very pleased to announce that I will be speaking at the SharePoint User Group UK (London) on Thursday 25th August.

    I am basically doing a “practice run” of my Configuring Kerberos in a SharePoint 2010 Farm which I am doing at SharePoint Saturday UK later this year.

    This will involve;

    • Configuring Kerberos Live on a SharePoint 2010 farm, taking it from NTLM to Kerberos/Negotiate authentication
    • Configuring SQL and Analysis Services to use Constrained Delegation
    • Configuring SP2010 Excel Services to pass through the authentication credentials using the Claims to Windows Token Services
    • How to prove it is all working using “out of the box” tools
    • A few other resources, caveats and tricks

    This is a FREE event at LBi offices in central London, full details, signup and map details can be found on the SUGUK forum: https://suguk.org/forums/thread/27083.aspx

    Should be a great event, hope to see you there and have a SharePint afterwards.

    I’m speaking at SharePoint Saturday UK (SPSUK)

    I am proud to announce that I have been selected as one of the speakers at the SPSUK event this year in Nottingham on 12th November. The full session list can be found on their website and includes loads of great sessions throughout the day!

    My session is going to be about configuring Kerberos in a SharePoint 2010 Farm and will include a live walkthrough of setting up Kerberos from scratch for a SharePoint 2010 environment, and how to use standard “out of the box” tools to prove that it is working.

    This will include Excel Services and Analysis Services, constrained delegation and configuring the Claims to Windows Token Service.

    This is the first IT Pro Session (9:15am) so hope you get there early! 🙂

    For those of you who don’t know about this fantastic event it is FREE to attend and consists of a whole day of SharePoint goodness. Here is a short quote from their website:

    SharePoint Saturday UK 2010 was a great success with around 200 attendees!!

    Brett Lonsdale (Lightning Tools), Mark Macrae (ID Live), and Tony Pounder (ID Live) are repeating last years efforts to bring you SharePoint Saturday UK 2011.

    This year the event will be held in Nottingham at the East Midlands Conference Centre. The EMCC is close to East Midlands airport, Nottingham train station, and the M1 motorway with easy links to the M6 motorway.

    If SharePoint Saturday is new to you, expect a day of great international speakers including Microsoft employees, SharePoint MVPs and SharePoint community experts, great content, and of course SharePint!

    Hope to see you there! If you haven’t already then sign up now!

    How to programmatically add the Related List Webpart to a list form

    This is one thing that I have used a few times through the browser and always thought it was a feature missing from WSS 3.0 / MOSS 2007. But it was only when I tried to do this (relatively simple) thing as part of a list definition that I realised how oblique it was, and how little information there seems to be explaining how to do it… This blog post will try to address some of that.

    So first off .. what is a Related List Webpart? This wasn’t really a surprise but if you look under the hood you will notice that when you use the “Related List” ribbon button then it actually does 2 things:

    • Adds a new List View Webpart (in this case an XsltListViewWebPart) which is bound to the related list
    • Creates a web part connection from the (already present) ListFormWebPart which provides the ID which the related list will filter on.

    In order to set this up programmatically we basically have to replicate this. In my example I am running this from a Feature Receiver which will setup the web parts. I have 2 lists which I will be referring to:

    • Main List (the main list which I will be modifying)
    • Related List (a list which contains a lookup field containing values from List A)

    STEP 1 – Add a new List View Web Part to the List Form

    In this step we will be modifying the DISPLAY form (DispForm.aspx). This is probably the most common approach although you can follow very similar steps for the Edit Form if you so wish)

    // these are the 2 lists we will work with
    SPList mainList = web.Lists[“Main List”];
    SPList relatedList = web.Lists[“Related List”];
    // this is the Display Form web part page
    SPFile dispForm = web.GetFile(mainList.DefaultDisplayFormUrl);
    // get the web part manage which we will use to interact
    SPLimitedWebPartManager wpm =
    dispForm.GetLimitedWebPartManager System.Web.UI.WebControls.WebParts.PersonalizationScope.Shared);
    // only execute this code if there is a single web part (the default for this list)
    // we don’t want to add the web parts over and over again ..
    if (wpm.WebParts.Count == 1)
       // create our List View web part
       XsltListViewWebPart wp = new XsltListViewWebPart();
       // Hook up the list and the view
       wp.ListId = relatedList.ID;
       wp.ListName = relatedList.ID.ToString();
       wp.ViewGuid = relatedList.DefaultView.ID.ToString();
       wp.XmlDefinition = relatedList.DefaultView.GetViewXml();
       // set basic properties along with the title
       wp.AllowConnect = true;
       wp.Title = “Related items from “ + relatedList.Title;
       wp.ChromeType = PartChromeType.TitleAndBorder;
       // add the web part to the Main zone in position 2
       wpm.AddWebPart(wp, “Main”, 2);
       // save the page

    So you can see from the example code above we are adding a simple List View webpart for our related list items. We are binding it to the “Related List” using the ID, and also choosing the View we want to display (here we have just used the Default View for simplicity).

    We need to make sure we set “AllowConnect” (as we will need to use Connections in Step 2) and also set the ChromeType to “Title and Border” (as the default for Web Parts in List Forms is “None”).

    Finally we add the web part and save the changes to the Web Part Page …

    now for the slightly less well documented bit ..

    STEP 2  – Create Web Part Connections between ListForm and ListView webparts
    In this step we are going to take the Web Part which we added in STEP 1 and use standard Web Part Connections to bind them together … don’t be scared, it isn’t as hard as yoy might think 🙂

    We are going to be doing several things here:

    1. We will get instances of both web parts which need to be connected
    2. We will create “Connection Point” objects for each web part
    3. Specify which fields we want to filter on
    4. We will create a “Transformer” object which is able to pass values between each web part
    5. We will use the Web Part Manager to create the connection, using all of that information above.

    // get instances of our two web parts
    System.Web.UI.WebControls.WebParts.WebPart consumer = wpm.WebParts[1];
    WebPart provider = wpm.WebParts[0];

    // Create our two Connection Point objects
    // These are specific to our two types of Web Part

    ProviderConnectionPoint providerPoint = wpm.GetProviderConnectionPoints(provider)[“ListFormRowProvider_WPQ_”];

    ConsumerConnectionPoint consumerPoint = wpm.GetConsumerConnectionPoints(consumer)[“DFWP Filter Consumer ID”];

    // create our “Transformer”
    // we also specify which Field names we want to “connect” together
    // here I am connecting my Related List’s “MyLookupField” and filtering it
    // using the “ID” of my List Form’s item.

    SPRowToParametersTransformer optimus = new SPRowToParametersTransformer();
    optimus.ProviderFieldNames = new string[] { “ID” };
    optimus.ConsumerFieldNames = new string[] { “MyLookupField” };

    // Connect the two web parts together, using our Connection Point objects
    // along with our Transformer

    wpm.SPConnectWebParts(provider, providerPoint, consumer, consumerPoint, optimus);

    And that is it! You don’t need to do any more “Update” or “Save” methods .. your Related List web part should now be finished 🙂

    So just to be clear, we are using the standard SPLimitedWebPartManager object and calling the SPConnectWebParts method. The only slightly odd call is the reference to the “SPRowToParametersTransformer” which is specific to the type of Connection we are making (the List Form representing a “Row”).

    Where did this information come from?
    This is the easy bit 🙂 I added a Related Lists Webpart using the browser, then opened up DispForm.aspx using SharePoint Designer .. and found this: 

    <WebPartPages:SPWebPartConnection ConsumerConnectionPointID=”DFWP Filter Consumer ID” ConsumerID=”g_dc64a1e0_c2f4_4302_86df_e4d184203bbd” ID=”c225174896″ ProviderConnectionPointID=”ListFormRowProvider_WPQ_” ProviderID=”g_ffb9e36b_bc6d_489e_b7fc_e93048c32f5c”><WebPartPages:SPRowToParametersTransformer ConsumerFieldNames=”IMSDataSource” ProviderFieldNames=”ID”></WebPartPages:SPRowToParametersTransformer>

    I have highlighted the references to the 3 things which made all of the dots join up for me: )
    so if you are ever stuck, make sure you look in SharePoint Designer for :

    • Consumer Connection Point ID (“DFWP Filter Consumer ID”)
    • Provider Connection Point ID (“ListFormRowProvider_WPQ_”)
    • Transformer Type (SPRowToParametersTransformer)

    So thats it 🙂 Hope this was useful, comments always welcome

    How to add the “Contact Us” gadget to your Office 365 Publishing Site

    This follows on from my previous post which describes how to enable the Publishing Features for your Office 365 website and resolving issues with Custom Master Pages in Office 365.

    Having done that on my own site (www.hatchsolutions.co.uk) I found that one of the key “Gadgets” was missing .. the Contact Us form.

    This was an immediate conundrum.. Sandbox Solutions do not allow email access so I knew I couldn’t use that as an out, so I thought I would hunt down that Gadget.

    STEP 1 – Add Office 365 namespaces to your Custom Master Page
    This is a pre-requisite, as you will be using some of the “Office 365” web controls. I found these in the header of the “oob” root.master which you will find in the All Files section of your public website in SharePoint Designer.

    Add these to the top of your Custom Master Page.

    <%@ Import namespace=”Microsoft.SharePoint.Spx.WebSite.Controls” %>
    <%@ Import namespace=”Microsoft.SharePoint.Spx.WebsiteCore” %>
    <%@ Import namespace=”Microsoft.SharePoint.Spx.WebsiteCore.Controls” %>

    (note the special Microsoft.SharePoint.Spx namespace!)

    STEP 2 – Create Custom Page Layout with Contact Us Control Embedded
    This was the way I chose to do it .. you don’t have to do the same (although I couldn’t work out an easier way of doing this myself).

    I created a new custom Page Layout (based from one of the others). I deliberately removed any references to any custom code-behind or code-blocks.

    This means that I will be running OOB code, and won’t be consuming any Sandbox Resource points!

    Then add the following ASP.Net snippet into your page:

    <iws:ContactUsControl runat=”server” EmailTo=”info@hatchsolutions.co.uk” CultureName=”en-US” SendToBCM=”on” xmlns:iws=”iws” />

    This is the same ASP.Net tags that get used when you drop the gadget onto your page.
    You can see in my example I have added my own EmailTo and CultureName attributes .. you can set these to whatever is appropriate.

    The other thing to note is that you must place this control in an EditModePanel with PageDisplayMode=”Display”. The Contact Us form otherwise tries to validate the page when you try and Save Changes or Publish (not good!)

    And that is it .. simply save your changes, publish your page layout and create your own page! 🙂
    I have created my own custom page layout here;

    Please let me know how you get on with yours!

    Office 365: Custom Master Page – Resolving “Code blocks are not allowed in this file”

    This was an annoying error for quite some time. Office 365 (SharePoint Online) gives you full control to create, deploy and select custom master pages, but if you try using them you often get the error:

    Code blocks are not allowed in this file

    The reason for this is because of the locked down security permissions in the Master Page Gallery. This is fair enough, just means you can’t have custom code running on your Master Page (but seriously .. how often do you actually need to do that?)

    So .. I created my Master Page using the awesome CodePlex Starter Master Pages. And this was fine until I deployed it (using Sandbox Solutions) to my Office 365. Bam .. error message (Code blocks are not .. blah blah).

    Well .. I went through my Master Page line by line .. I couldn’t actually FIND any code blocks. I could find a bunch of expressions (but these don’t count!) such as

    • <%@ Import %>
    • <%@ Register %>
    • <%$Resource ..>

    The Resource tags reference resource strings. This is used ALL over the OOB controls and is generally very good practice!

    I finally worked out what was causing it .. the actual Master page tag itself!
        <%@ Master language=”C#” %>

    The “language” attribute was causing all of the problems! (even though we didn’t have any code!)

    So I removed the language attribute (leaving me with):

        <%@ Master  %>

    And hurrah! I all started working. Hope this helps! Happy branding!

    « Older Entries