Windows Server 2019, SharePoint Server 2019 and a broken Blob Cache

I am one of those SharePoint early adopters. My first MOSS 2007, SP2010 and SP2013 commercials projects all went live before the main product hit General Availability, so I’m very much used to those niggling issues you get before a product hits the shelves.

However, in this case I believe I have found a configuration bug relating to commercial GA products.

The builds where this error is known to occur (and I have replicated this on a brand new VM build just to double check) are:

  • Windows Server 2019 Standard (Version: 1809 | Build: 17763.168)
  • SharePoint Server 2019 (Version: 16.0.10337.12109 – RTM Build)

Update : I have also today replicated the same issue on a fully-patch server setup (updated on 22nd January 2019):

  • Windows Server 2019 Standard (Version: 1809 | Build: 17763.253)
  • SharePoint Server 2019 (Version: 16.0.10339.12102 – December 2018 CU)

Disclaimer : My findings below are purely what I have observed with my own installs. Your own mileage may vary, but if you DO see this happening, then I have a fix described which should get you rolling again!

For those who aren’t sure if their BlobCache settings are actually correct – you should definitely make sure your web.config values are correct by checking out the Microsoft Docs article: Configure cache settings for a web application in SharePoint Server

So what is the problem with BlobCache, exactly?

In a nutshell what I have observed is that the BlobCache will not initialise. The designated blobcache folder is never populated and features which rely on it do not work (i.e. SharePoint Publishing Cache and Image Renditions).

I should note that this is not a permission problem with the actual blobcache folder itself (the default location being C:\BlobCache\14). I triple checked the permissions and they were all correct. In fact if you delete that folder and perform an IISRESET you’ll even find it gets re-created .. just it won’t have any contents (and “blob caching” as a function will not work).

I could not find any related issues in the browser console, ULS logs or IIS logs. The only error I found was in the Windows Event Viewer (under “Web Content Management”, Event ID 5538).

An error occured in the blob cache.  The exception message was ‘Retrieving the COM class factory for remote component with CLSID {2B72133B-3F5B-4602-8952-803546CE3344} from machine <machinename> failed due to the following error: 80070005 <machinename>.’.

Anyone with a history of COM errors will probably get an instant shiver and feelings of dread about going through DCOM configuration! And you would be exactly right .. this is a DCOM error. The error code (80070005) also tells us that this is an “Access Denied” error .. so we know that this relates to permissions.

In order to test this premise I did something very simple:

  • Add the application pool accounts to Local Administrators

I performed the usual IISRESET /noforce .. and voila! BlobCache started working .. hurrah!

But before you cheer, this really is NOT an ideal scenario to be in (app pool accounts should NOT be in the local admin group). So I removed those accounts from the administrators group .. and kept on digging ..

DCOM and Registry Permissions .. what decade is it again?

So yes .. this feels like the “good old days” when you wasted days tracking down erroneous GUIDs and DCOM config weirdisms .. and it seems in the year of 2019 this is going to pester us all once again.

However, as is always the case with COM components, things are not as straight forward as they might at first seem.

Having done some searching around for that CLSID (2B72133B-3F5B-4602-8952-803546CE3344) it appears to be a reference to a remote configuration component for IIS. This is NOT a DCOM component. However, much google-fu later (after reading a whole bunch of posts about automating Azure deployments .. yeh it seems the error above is fairly common when dealing with IIS config) I found that this is actually a COM class which  tries to call a DCOM components. Specifically the Application Host Admin API for IIS 7.0 (also known as “ahadmin”).

The error code we are seeing (80070005 = access denied) means the application pool account does not have permissions to launch or execute this DCOM component.

Now .. this is the point where, if you haven’t had to deal with DCOM permissions before, you might get a little stuck. You see, if you open up Component Services, find the “ahadmin” DCOM config entry .. you will notice that the “Security” options are all greyed out. You can’t modify them.

So now .. we need to ninja up our Registry skills.

We can see from the Component Services window the “Application ID” of “ahadmin” is {9fa5c497-f46d-447f-8011-05d03d7d7ddc}. So this allows us to build the Registry Key where this particular application is defined:

So we can launch RegEdit and look for the following path:

HKLM\SOFTWARE\Classes\AppID\{9fa5c497-f46d-447f-8011-05d03d7d7ddc}

(you will know it is the right one because the default key will read “ahadmin”). Right click on the node and select “Permissions”, and go to “Advanced” settings (you might need to wait a while for it to resolve all of the security principals before “Advanced” becomes available).

What you will need to do is two things:

  • Change the “Owner” (by default this will be “TrustedInstaller” – I changed this to the local “Administrators”)
  • Update permissions for “Administrators” so that they have “Full Control” (in addition to “Read”).

Having done this, reload “Component Services” and you should find that you can now edit the “ahadmin” DCOM permissions.

Change the following permissions for “Launch and Activation” as well as “Access Permissions”

  • Grant the local WSS_WPG group “Local” and “Remote” permissions (i.e. all of them)

Note – The WSS_WPG (Windows SharePoint Services – Worker Process Group) is a special group which SharePoint maintains to include all of the designated “application pool accounts” for the SharePoint Web Applications. So it is much better to use this group rather than permission each account individually. 

Now you just need a trusty IISRESET and we will have taken care of that nasty COM permissions error …

What do you mean .. more?

Yeh .. sorry .. you will probably find having done all that your BlobCache STILL doesn’t work.

This time however, the error in Windows Event Viewer will have changed (same “Web Content Management” category, and Event ID 5538)

An error occured in the blob cache.  The exception message was ‘Filename: redirection.config’

 

Error: Cannot read configuration file due to insufficient permissions

This one is a little bit easier to resolve, and should be a fairly quick solve.

It is referring to the “redirection.config” in the IIS config folder.

C:\Windows\System32\inetsrv\Config

Change the permissions on that folder and grant the same “WSS_WPG” local group “read” access to that folder (it doesn’t need more than “read” as it won’t be making changes to the IIS config files).

One more IISRESET and you should be done!

Voila … BlobCache joy once more!

BlobCache should now be correctly populating, and your web applications in SharePoint should be creating Blob Cache entries specific to their web.config entries.

The Windows Event Viewer should now show that it is creating the Blob Cache folders correctly (this time a slightly different Event ID : 7358)

Creating new cache folder ‘C:\BlobCache\14\1381895183\iXPJhtlC+ECvfzPRK3EHMA\’.

And you can see this folder created happily in the file system

And Image Rendition editing should also be working…

(and serving the correct images)

Summary

So for a quick summary .. we had to do some DCOM Config changes, Registry Key permissions and IIS config permissions.

Ironically this error was nothing to do with file system permissions to the Blob Cache folder itself (which is typically the only thing which has needed troubleshooting).

So .. quick crib-sheet:

  • Take ownership of the “ahadmin” Registry Key, and grant admins Full Control
  • Grant WSS_WPG {Launch | Activation | Access} permissions for the “ahadmin” DCOM component, accessed from “Component Services”
  • Grant WSS_WPG “read” permissions to the IIS “Config” folder in the file system

Hopefully this article was helpful! Chime in the comments if you have any questions (or if you’ve seen this issue before yourself!)

Update – 23/01/19

One of my colleagues at Content and Code reached out to Microsoft to report this issue and find out if they were aware of it.

Suffice to say Microsoft are indeed aware of this issue and are currently working on a fix.