Scaling to 10,000 unique permissions – Part 1 – The Problem

Posted on by 26 comments

This post was borne out of a client requirement which popped up on my radar. I’m currently working for a leading global Business Intelligence provider in London, and they were looking to implement a particular third party piece of software. This software relies on SharePoint for file storage and my client wanted to roll this out to their customers “extranet” style with each customer having uniquely secured content (files and folders).

Now .. first off their customers include in excess of 10,000 different companies (i.e. > 10,000 users) so early warning bells immediately started ringing in terms of scalability.

Secondly, to make this worse, the software required all content to be stored in a single SharePoint site .. so now my early warning system had gone into full meltdown and a state of high alert was reached.
So to boil this down …

  • One SharePoint 2010 site

  • 10,000+ uniquely permissioned objects each with a different user account

A Library with 10,000 uniquely permissioned folders?? Possible? My first instincts said no… so it was time to get my problem solving hat on and do some digging ..

Investigating the Limits of SharePoint 2010

I would like to think that any SharePoint {Consultant | Developer | Architect | <insert profession>} worth their salt would have read the Software and Capacity Planning guidelines (or at least be aware of it!) .. so that was my first pit-stop.

Note – I also stumbled across a great blog post by SharePoint infrastructure veteran Joel Oleson and his Best Practices for Enterprise User Scalability in SharePoint. This goes into detail about the specific size of an ACL (and the reason why this is limited, specifically in Windows) which although a good read wasn’t really relevant to my problem.

The Microsoft TechNet article SharePoint Server 2010 capacity management: Software boundaries and limits (https://technet.microsoft.com/en-us/library/cc262787.aspx) is a great resource and contains one absolutely key entry:

Security Scope – 1,000 per list (threshold)
The maximum number of unique security scopes set for a list should not exceed 1,000. 

A scope is the security boundary for a securable object and any of its children that do not have a separate security boundary defined.  

A scope contains an Access Control List (ACL), but unlike NTFS ACLs, a scope can include security principals that are specific to SharePoint Server. The members of an ACL for a scope can include Windows users, user accounts other than Windows users (such as forms-based accounts), Active Directory groups, or SharePoint groups.

So what is a Security Scope then? Ok I admit it does tend to get a bit bogged down in terminology.
To put it simply … each time you grant access to a new principal (user account or group) then you are creating a new Security Scope.

The other thing to consider pickup is that this is not just limited to lists! Any list that inherits permissions will pick up their permissions from the parent web (site) so you also need to adhere to this at the web level too!

This means that you should not have more than 1000 security scopes at EITHER the Site or List level.

Ignoring this limit can do real damage to your farm …

There is even a Microsoft Knowledgebase article explaining why; SharePoint performance degradation with a large number of unique security scopes in lists (https://support.microsoft.com/kb/2420771)

This is really explained in far more detail in two most excellent blog posts:

The first post describes the problem of trying to create more than 1000 security scopes, and what happens when you do this: https://wbblog.datapolis.com/2011/03/setting-item-permissions-with-workflow.html

The second post is by James Love (a.k.a. @jimmywim) and goes into real “deep dive” detail looking into the root cause of the problem (SQL Server and ACL GUIDs) and how this problem can actually bring down your ENTIRE FARM and not just the list / site you are working on!
https://e-junkie-chronicles.blogspot.com/2011/03/sharepoint-2010-performance-with-item_23.html

A quote from the second post is as follows:

“When you load up a huge list with lots of item level permissions, a single operation gets every single GUID associated with the ACL for that item and passes that back to the data access layer of SharePoint. When the database retrieves the actual list item data, it will pass in all of the ACL Guids back in as one long string, all concatenated together. The query to get the data creates a table variable re-assembles the the item level ACL Guid associated with each item. How the rest of the query deals with this is anyone’s guess at the moment – this table variable might just be passed back to the calling COM object (though I thought they couldn’t be used this way….) for the COM object to then sort out which item should be visible to which “scope” (or ACL).

So, what can we take away form this? Passing 640k of data about the place, for a SQL Query to do some substring math and converting to Guids will soon bring your database server to its knees. This is one request and it takes 2000ms to work. Imagine if you have 5 requests per second or more hitting this list!”

Both are excellent appendums to this post and well worth looking at for another angle and a bit more detail!

Why does this become my problem?

Now .. looking back to my original problem some of you may be thinking, OK no problem; you can just create yourself 20 different lists / libraries .. and have 500 unique permissions in each list??

Well .. so you might think .. and here I introduce the juggernaut that is Limited Access Scopes!

Anyone who has spent any time around SharePoint security will have noticed the odd “Limited Access” permission popping up in their site from time to time. “Limited Access” is automatically allocated to a parent Folder, List or Web whenever a child object has a unique permission allocated to it.

You can easily see these being created if you break permission inheritence to a list and just add a few accounts to that list. The parent Web will not have a “Limited Access” scope created for each user account you have added.

Now hopefully the bright will already have spotted the problem .. it doesn’t matter how many lists or libraries you create .. every single user or group that you add will end up in the parent Web site with “Limited Access” (and every single Parent Web heading upwards).

The following diagram explains why.

You simply cannot get away from this fact. If are adding 10,000 unique permissions with different user accounts then you will end up with 10,000 security scopes at the root web!

Note – It should be noted that the number of “Limited Access” scopes created is limited to the number of Security Principals you are adding.

If you are adding from a pool of 50 users then you will only ever be adding a maximum of 50 new “Limited Access” scopes (one for each user account).

For this reason it is a good idea to use Groups when adding permissions as this limits the number of “Limited Access” scopes which are created .. but this won’t solve your problem if you have over 1000 different security principals!

So that was the crux of my problem .. on investigation this does look to be a major major problem (and an “impossible fix”) but it would seem not! There IS a workaround (one which I have tested to over 15,000 unique user accounts and works very very well indeed)…

The solution, workaround, and code samples are all included in Part 2 …

Posted in Uncategorized tagged with , , , ,
  • Martin Hatch

    This blog post was written for SharePoint 2010 .. the limits in SharePoint 2013 are different:

    2013
    http://technet.microsoft.com/en-us/library/cc262787.aspx

    2010
    http://technet.microsoft.com/en-us/library/cc262787(v=office.14).aspx

  • Arup Biswas

    Great post indeed, thanks a lot Martin for your valubale response. just to further clarify about unique permission and security scope; security scope is permssion level and unique permission will be counted as an when I break the inheritance even if I grant same user with same permission or different. is this right?

    Security scope limitation is 1000 per list NOT 5000, correct?
    Unique permission per list/library is 50000 but recommended only 5000, correct?

    And if the user count is greater than 5000 then what would be your suggestion to implement scenario mentioned above.

  • Martin Hatch

    Ok, it is 5000 security scopes per item / list / web

    If you only have 200 users and you are adding users individually to each document then you won't have more than 200 "Limited Access" scopes being created, so that won't cause a problem.

    If you are using SharePoint Groups then you are probably going to run into those limitations (max. number of groups per user and max. number of groups per site collection).

  • Arup Biswas

    Unable to edit my post so posting another!!

    group (group1,2,3) will be replaced with role (role1,2,3)
    I am not clear about unique permission and security scope? If we made 50000 folders with unique permission, the number of security scope is egals to 50000! Is recommended to not exceed 1000 by list & unqiue permission to 5000.

  • Arup Biswas

    I am planning a request management portal on SharePoint 2010. The user baes is around 200 but 5 differnt role will be there. Set of files associated with each request will have unique permission.

    Example:
    Every request will have some set of documents. So per month the request will be around 3000 but the associated document count will be 20,000. The below logic will be impose documents associated with each request(list item)..
    1. User group1 can have read/write permission to his/her uploaded files but they can have read only permission to other files uploaded by others.
    2. User group1 can have read/write permission to his/her uploaded files but they can have read only permission to other files uploaded by User group1.
    3. User group3 can have read/write permission to his/her uploaded files but they can have read only permission to other files uploaded by user group 1 & 2.I am planning to create a SP list to manage request details and document library to manage files associated with request. May be folder for each request and break the permission of that folder/sub-folder. But the unique permission may be reached quickly. So could you please suggest some approach ??

  • Anonymous

    I have two questions.

    1) How can I tell how many securable objects I have on my site collection? Is it as easy as going to each of my subsites and totaling the site permissions (including limited access)?

    2) I am confused as to whether inheriting limited access up to the root level counts against me multiple times or just one time. For example, I have a sub site that inherits from a parent. In the sub site I give unique permissions which throws up a limited access in the sub site and the parent. Does that count once or twice?

    Thank you!
    -Cynthia

  • Salman Aziz

    use SPRoleAssigmentCollection.AddToCurrentScopeOnly.

  • Kyle

    Ok, that's what I figured regarding security scopes.

    Both lists should have 4 unique permissions however (3 items + default permissions for the list).

    SP has a Unique permissions limit per list of 50,000 by default:
    http://office.microsoft.com/en-us/sharepoint-foundation-help/manage-lists-and-libraries-with-many-items-HA010377496.aspx

  • Martin Hatch

    Hi Kyle

    Well, in your example (assuming this is the first time Group1, Group2 and Group3 have been allocated permissions within that web) then:

    Library1 would result in 3 new security scopes being created
    Library2 would result in 2 new security scopes being created

    But .. if you created BOTH Library1 and Library2 then you would still only have 3 security scopes (Limited Access for each of the groups)

    Does this make sense?

  • Kyle

    Excellent article Martin.

    Question:
    Library1 has 3 items. Permissions on item1 to group1, item2 to group2, and item3 to group3.
    Library2 has 3 items. Permissions on item1 to group1, item2 to group2, and item3 to group1.
    Does Library1 have more security scopes/unique permissions?

  • Martin Hatch

    yes .. sorry .. Limited Access is the permission level but a scope is created if you are granting a new user account specific access to a uniquely secured object.

    Hence why if you add 10,000 list items, break inheritance on every single one, and then grant a different user permissions to each item, then you end up with 10,000 new security scopes at the parent web (each of which grants the "Limited Access" permission)

  • Anonymous

    Martin,

    "Limited Access" is not a scope, it is a permission level. When unique permissions are assigned to a principal, "Limited Access" gets added to an existing scope (not a new scope) up the hierarchy.

    John

  • Martin Hatch

    Anon ..

    It was intended in the context of the problem. Granting UNIQUE PERMISSIONS to a new principal will create a new Security Scope… specifically a "Limited Access" scope on the parent objects

  • Anonymous

    "each time you grant access to a new principal (user account or group) then you are creating a new Security Scope."

    I have my doubts about the above statement. Each time you grant access to a new principal you are NOT creating a new scope. A new scope is created when a securable object is made to have unique permissions, i.e broke inheritance from parent.

  • Martin Hatch

    Hey Mark,

    To clarify, each assignment of a Security Principal (AD user, AD group or SP Group) will result in a "Security Scope".

    So .. if you have 2000 users who are members of a single SharePoint Group .. and that group is granted permissions to something then you have created 1 Security Scope. If you add all 2000 users to the item individually THEN you will have created 2000 new security scopes.

    The important thing to realise is that “Limited Access” scopes are also Security Scopes .. and these get created on the first uniquely permissioned parent object when you have broken inheritance.

    So if you have 1000 libraries or sub-sites with broken inheritence and 1 group each .. then the parent web could end up with at least 1000 new "Limited Access" scopes being added

  • Mark Stokes

    Hi Martin, Great Article.

    This has helped me connect a few dots. I just want to check the following line: "each time you grant access to a new principal (user account or group) then you are creating a new Security Scope." This makes sense and matches the scenario I am seeing on a client project.

    I just want to clarify though:
    I have a list which breaks inheritance from it's parent web, which inherits from the site collection root web.. All the items (about 2000) inherit permissions from the list. There are a large number of SPUsers and SP Groups (with many users) having been given permission on the list level. Lets say the total number of users either directly added or within a SharePoint group is 1500, does this mean 1. there will be 1500 Security Scopes on that list? 2. There will be 1500 security scopes on the site colleciton root web level as well with limited access (even if they have access to the site)?

    Thanks mate.

    Mark Stokes

  • Martin Hatch

    Richard,

    If the site doesn't inherit permissions then THAT site ends up with thousands of security scopes.

    So any other lists in that site (which inherit permissions by default) with also have thousands of security scopes .. and will potentially crash your server.

    Not ideal by any stretch

  • Richard

    Hi Martin,

    Yes, in that example it's bad. But if the site which contains the list doesn't inherit permissions and none of the lists inherit permissions then it doesn't matter how many limited access there are on the site as it won't propagate to the parent site or any lists in that site.

    This would work for the original problem as long as new lists were only added in a controlled manner and role inheritance broken immedidately and tidied up.

    It all depends on where your security boundaries are.

    Richard

  • Martin Hatch

    Richard,

    My experience is not quite the same:

    * sub-site which inherits permissions
    * Library with unique permissions
    * folder with unique permissions

    Add user to folder:

    Limited Access granted to:
    * parent list
    * parent web

    So if I had loads of unique lists .. or loads of unique folders .. each of those would result in a new security scope at the web level.

    Because most of the other lists in my site inherit permissions .. they will all have thousands of security scopes too!

    This is bad m'kay?

  • Richard

    Hi Martin,

    I have rechecked and the limited access permission is only set on the controlling web permission scope. So if the web containing the list has unique permissions, there is no limited access in any of the parent webs.

    So if all lists have unique permissions within a web with more than 1000 different permissions, then there is no problem.

    So you can split the items across multiple lists so each list has less than 1000 scopes, within a single web without any messing around with limited access permissions. As long as every list in that site has unique permissions (and appropriate of course).

    This turns it into a governance issue rather than a technical solution once the lists are set up.

  • Martin Hatch

    Romain, thanks very much! 🙂

    Richard, I believe the "Limited Access" scope is set at each of the parent objects, regardless (unless they are inheriting .. in which case they are skipped as they will "inherit" the Limited Access from the parent).

    I may be wrong, feel free to double check and let me know.

  • Romain KEIRSEBILCK

    Amazing Article!
    I post it as my "Amazing SharePoint article of the week"

    PingBack from http://www.kerseub.net/Lists/Posts/Post.aspx?ID=5

  • Richard

    Been thinking about this overnight. The limitation on the web/site itself is only a problem if you have a list which inherits permissions. So if you don't have any lists inheriting permissions it won't matter about the limited access permission numbers on the web.

    So if you place the document libraries in a site which doesn't inherit permissions, either one site or one site per document library, which are then only used for holding these lists, you should be OK.

    This would seem to be a simpler method to implement and maintain and would work with SharePoint 2007.
    Not knowing all the details of your requirements I can't tell if it would work in your case.

  • Martin Hatch

    Hi Ian,

    At this point not entirely sure, but well within the document library / view limits (I think we're talking dozens, not hundreds).

  • Ian Chivers

    Hi Martin, great article. How many documents are they planning on storing in the folders ?

  • Sudip Misra

    great article! thanks