Monthly Archives: November 2009

How to add a Lookup Field to your List dynamically and programmatically

This is an old old problem, and for Content Types it has been easily realised that this can be solved by using a Feature Receiver and creating the lookup through code (to set the lookup list GUID .. something that you cannot do through CAML).

The biggest problem with Lookup fields is with List Definitions. You can still add the Lookup Field to your schema.xml, but it won’t do anything without knowing which list it is supposed to lookup to so you typically end up with a field that doesn’t work.

Now you can get around this IF you know the URL or the ID of the field you want to lookup to, but most list definitions that you release can be created anywhere so this is very rarely possible. Now .. in my particular example, we wanted the Lookup to lookup to itself! In this case, there really isn’t any URL or method through the schema.xml that we can use for this, and managed code is the only route … the main problem there is that there is no “ListAdded” event that you can trap when your list gets created … but then I had a spark of inspiration!

The solution was surprisingly simple and came in a bit of a eureka moment… SPListEventReceiver “OnFieldAdding” event.

You can bind in event receivers using a variety of methods (not covered here) but for my example I bound it into the schema.xml as part of my list definition (so this code only ever executes when the list is created).

The “OnFieldAdding” event then executes every time a new field is added to the list, and this includes fields provisioned from the schema.xml! All I then needed to do was identify the field (which I could do easily because I knew the Field ID) and then I could use managed code to manipulate the SPField object and fill in all the blanks that I couldn’t do from the schema.xml!

I really like this method, as it has opened the door to a way of executing code on a list when it is created. Now I admit it’s not exactly bullet proof because it potentially executes the code a LOT of times (although ideally you would remove the event handler once you’ve finished executing what you needed to), but it was a really nice “other option” .. and certainly one I hadn’t considered before!

NewSID is dead.. duplicate machine SID no longer a problem?

Well, this one was gob-smacking! A colleague of mine (Tristan Watkins) pointed me at this article from Mark Russinovich, the developer of the “NewSID” tool that so many people use for creating a new machine SID for a machine (typically after a re-image or copying a virtual machine).
Well, it seems we don’t need to bother anymore and never really did, at least in the vast majority of cases!
It’s a little surprising that the SID duplication issue has gone unquestioned for so long, but everyone has assumed that someone else knew exactly why it was a problem. To my chagrin, NewSID has never really done anything useful and there’s no reason to miss it now that it’s retired.
To read more, go check out Mark’s blog post here: https://blogs.technet.com/markrussinovich/archive/2009/11/03/3291024.aspx
Update – A good follow up summary has also been written by one of my colleagues, Tristan Watkins.

Presenting at SUGUK London 25th November

Yep, I’m taking the plunge and finally presenting publicly 🙂
 
I’m going to take the stand to talk about Developing an Accessible SharePoint System based on our experiences in designing and building the Intranet and Website for the Royal National Institute for Blind People (RNIB).
 
I’ll be going into technical detail about development techniques for customising the front-end and back-end interface of SharePoint, as well as some of the more rounded issues around accessibility (such as the age old "accessibility versus compliance" discussion).
 
I’ll also hopefully get the opportunity to show off our new SAS (SharePoint Accessibility Solution) framework and show the audience an example of a WCAG 2.0 AAA system running on MOSS 2007.
 
Please feel free to come along (assuming you are in the London area on November 25th, arrive 6:00pm for a 6:30pm start!). It’s free to attend, and Microsoft usually do a good show by providing free pizza, tea and coffee! 🙂
 
You can sign up at the SUGUK forum thread here:
 
Look forward to seeing you there!

Load Testing SharePoint 2010 with Visual Studio Team Test

 

So exactly what do we mean by "load testing" when it comes to SharePoint 2010? There are lots of methods that people tend to point towards, and I’ve heard "hits/visits per day" and "throughput" bandied about, but at the end of the day it comes down to 2 things:

 

  1. Requests Per Second

The requests per second literally means how many requests for information each server is capable of responding to per second. Each page may consist of dozens of artifacts, and for each artifact the browser needs to make a "request", therefore the more of these  "requests" it can serve the better.

 

  1. Server Response Time.

The response time represents any processing on the server side (or TTLB – Time to Last Byte). This doesn’t factor in network latency or bandwidth though!

 

So the first thing you should think about is what can influence those metrics? And you end up with 5 different elements of your SharePoint 2010 farm:

  • WFE
  • Storage
  • Network
  • App Servers
  • SQL

 

This, as I’m sure you can imagine, can involve a LOT of testing. Simply testing the WFE on their own is going to be struggle for your average developer, and if you don’t have any industry testing experience you are going to have a hard time, but this is where the new SharePoint 2010 wave continues to make it’s presence felt. ..

 

SharePoint 2010 Load Testing Toolkit

This is a new set of tools being released with the SharePoint 2010 Administration Toolkit and represents the easiest possible way of load testing your SharePoint environment. The main objective here is to:

 

  • Standardise and simplify the cost of load testing.
  • Simulate common SharePoint operations
  • Be used as reference to create other custom tests (for custom code, for example!)

 

The whole thing relies on the IIS analysis logs. These logs give pointers on where users are going, what kinds of requests they are doing (GET / PUT) as well as the types of files they are typically accessing (ASPX / CSS / JS / JPEG / DOCX / etc…)

 

The Load Testing Toolkit will analyse your IIS logs and automatically generate a set of loads tests to appropriately match your environment, producing automated scripts that can be run in Visual Studio (either Team System or Team Test Edition).

 

How hard can it be?

It is really quite simple (well, according to the ridiculously simple explanation at the SharePoint 2009 conference!). You literally point the tool at your IIS logs, and it spits out an entire suite of tests, for WFE, SQL, Storage, etc .. Including all the metrics you would want (from CPU, RAM, Network, Disk I/O and even SQL , ASP.Net and .Net Framework specific performance counters).

 

Then you just run it and analyse the results!

 

Analyse That!

The analysis couldn’t be simpler. With "Requests Per Second" and "Response Times" two of the metrics generated by the Visual Studio test reports, you really can’t go far wrong.

 

If you do find a problem, then you can delve into the new SharePoint 2010 "Usage Database" (which now runs on SQL Server) in order to identify exactly what was causing your dip in performance (say when someone deletes a large list?).

 

Tips and Tricks

There are a few gotchas, one thing is to be careful of "Validation Rules" in Visual Studio. Typically it will be happy with pages that return "200" codes. This of course includes Error and Access Denied pages (which SharePoint will handle, and returns a perfectly valid page (hence the 200 code!)).

 

It is also recommended that you let your test "Warm up" for around an hour before you start taking the results seriously.  This allows all of the operations, timers and back-end mechanics of SharePoint to properly settle down, and means you are getting a realistic experience of what the environment will react like once it is bedded into it’s production environment.

 

Finally, the SharePoint Usage Logging Database is a great location to grab information out of, so why not leverage other great aspects of the Office 2010 family. You could pull through the Usage DB information into Excel 2010 (perhaps using PowerPivot?) so that you can spin out charts and pivot tables to easily drill down into your data.

 

Typically load testing tells you WHEN bottlenecks are occurring, but the Usage Database can tell you WHAT is causing the bottlenecks!

SharePoint 2010: Architecture Guidance – things everyone should know!

Well, the final day of the conference came and with it some of the most useful sessions (from my perspective). One of which was the "Architecture Guidance for SharePoint 2010". This hopefully distils some of that information. It’s not a be all and end all, but hopefully points you in the right direction so that you can focus your research a little better!

 

[UPDATED: 27/10/2009 16:09]

 

UI Design

  • Entire interface in SharePoint 2010 to be W3C XHTML compliant
  • SharePoint 2010 "more accessible mode" to be WCAG 2.0 AA compliant
  • New ribbon interface replaces toolbars and menus (and considerations for old "CustomAction" commands which may no longer work!)
  • Wiki content allows web parts to be dropped in (removing over-reliance on web part zones)

 

Lists

There are a whole load of new List capabilities (in addition to the "External List" that BSC brings to the plate!).

  • Lookup to Multiple

This means that when you create a new lookup column, you can now pull down additional fields from the lookup list item and use them for filtering.

  • CAML support for Joins!

You can now perform "JOIN" operations in your CAML queries for linking lists together.

  • Enforced List Relationships

You can now enforce specific relationships for lookup columns with two options:

  • Restrict Delete – cannot delete parent if child items exist.
  • Cascade Delete – If you delete the parent, all child items are automatically deleted (recycle bin aware with "restore" options!)
  • Store-level enforcement

This is code level "required fields", so now you can enforce the requirements even through code !

  • Unique Fields

Specify a unique field, so that no two values can match (e.g. Email addresses in contacts list)

  • Compound Indices

If you want to query by 2 fields, you can now index both at once as a compound index.

  • <In> clause for reverse lookups

This allows a CAML query to do a reverse lookup to get all child items that are associated with the parent!

  • Formula based validation

e.g. Don’t allow Field2 to be lower than Field1.

 

Workflows

  • Out of the box SharePoint 2010 workflows can now be extended in SharePoint Designer 2010.
  • SharePoint Designer 2010 can be used to create "re-usable" workflows
  • Site Workflows – to manage processes across an entire site.
  • You can now import a SharePoint Designer 2010 workflow into Visual Studio 2010!
  • Import/Export workflow using Visio 2010 for visual workflow modelling.

 

Content & Document Management

  • "Document Sets" allow you to treat a group of documents as a single item (with 1 version history, group executed workflow and policy, and a "download as zip" option).
  • Managed Metadata Service  allows cross-farm Content Type management and a pre-defined enterprise taxonomy structure! This is a killer-app, bringing true enterprise content management to SharePoint 2010.
  • Enterprise Wiki’s allow more rapid "in edit" content, as well as Web Parts deployed directly into the rich text editor (no more web part zones?).
  • Spelling check and broken link check when you "check-in" WCM pages.

 

Event Handlers

Three new event handlers added (at last!!)

  • WebAdded – Fired every time a child site is created in the web.
  • ListAdded – Fired every time a list is created in the web.
  • Feature Upgrading  – Fired when a feature has it’s "upgrade" method called (more on this in a future blog post).

 

Security

  • Editing of ASPX pages now required "Designer" permissions (instead of contribute).
  • XSS (Cross Site Scripting) protection for pages and web parts.
  • HTML pages will now "force download" by default. This stops people from uploading HTML files with malicious scripts, so if you click on an HTML file in a document library you will get a download dialog instead of the file opening in the browser!
  • There are still no field level permissions (it was estimated that this would add a 30% overhead to performance! Maybe in a future release)

 

BI and Connectivity

  • New Business Connectivity Services (BCS) allows no-code connections of databases and LOB systems to content types and lists with two-way synchronisation of data  and full CRUD support.
  • BCS interactivity from within Office clients, allowing LOB system data to be edited directly from desktop applications (such as Outlook and Word).
  • PowerPivot for Excel allows upwards of 100 million rows into an excel workbook with phenominal performance.

 

Office Application Support

  • New web level services for applications (Excel / Visio with JavaScript events!)
  • SharePoint Workspace to replace "Groove" for offline file support and editing.
  • Office Web Applications to allow for direct opening and editing of documents from within the browser!
  • InfoPath 2010 can now be used to edit the List forms out of the box!

 

Databases

  • Still a 100GB "limit" for content databases.
  • Still cannot have site collections spanning multiple databases.
  • New support for "Failover" databases, SharePoint 2010 is now SQL mirror aware!
  • All "Service Applications" have their own SQL database, along with many other new databases (e.g. Feed Activity, Social Data, Usage Logs).
  • New "read only content databases" open the door for simple content deployment (utilising SQL log shipping or database replication).

 

Content Deployment

  • All execution now in Timer Jobs.
  • Performance (and memory usage) improved.
  • Export routine now creates database snapshot to improve data integrity!

 

Sandboxed Solutions

  • Ability to upload WSPs directly into the content database to execute in minimal permissions using "virtual files" (no impact on the file system!)
  • Resource throttling, code performance checking and "bad routine" blocking
  • Provides new best practice for code development and deployment!

 

Search

  • New FAST search with thumbnail views (and navigation!) for office documents
  • Improved relevancy and non-query searching
  • 2 new search products (FAST based)
  • New refinement panel for advanced sorting and filtering "on the fly"
  • Multi-lingual support with over 80 languages built-in.

 

Social Networking

  • New My Sites structure
  • Activity Feeds to provide updates on user activity with an extensible architecture!
  • "Social Feedback" functions akin to Delicious and Digg allowing tagging of any URL based content, and subsequent discussions around items that have been "tagged".
  • Ratings mechanism distributed throughout the product.

 

I’m sure there are many other things, so please let me know if there’s anything else you think should "make the grade" and I’ll see if I can add it in 🙂

100 million rows in Excel? PowerPivot.. a first look from the SharePoint Conference 2009

 

"Project Gemini" has been batted around for a while now but it was unveiled at the conference that it is now known as SQL PowerPivot for Excel 2010 and SQL PowerPivot for SharePoint 2010.

 

What does it do?

In short, PowerPivot allows you to pull data into an Excel workbook from almost any data source. This can be SQL databases, Analysis Services Cubes, or any ODBC data source.

 

This is all handled via the import wizard, which contains a nice interface to setup which tables and filters you want to apply (the wizard then generates the necessary query).

 

You then have access to a whole raft of Excel Formulas (and a bunch of new aggregation and time intelligence formulas) that you can use to add new columns to the data. You can even bring in your own Excel worksheets as tables of data that can be linked up to the other data sources (say to provide foreign key tables where the lookups are stored and managed in Excel!)

 

Ok … So what’s so special about this?

Well, the main thing that is impressive is that they demonstrated an example system running with over 100,000,000 rows of data! Now remember that this is running from Microsoft Excel!

 

You could then add your own extension columns (using simple Excel style formulas) and the whole  data set refreshes in seconds.

 

So the performance is good huh?

The performance is quite simply jaw-dropping.

 

One of the demo sessions the presenter imported over 3.5 million rows of data from a SQL Analysis Services cube and it imported in just under 2 minutes.

 

He then created a pivot table of the total sales data, split into rows by country.

He then added "slices" so that you can flick between sales figures for different years or product categories.

 

With all of these calculations the pivot table was refreshing it’s data in under 2 seconds!

 

Not even SQL Reporting Services can execute that fast, and this is in EXCEL so the user has full control over the pivots and can filter / query / change the results as much as they like.

 

How does it actually work then?

The main thing that PowerPivot does is that the database columns are separated out and compressed individually. Foreign key values can then be separately indexed and this makes the compression levels fantastic.

 

Take an example of a foreign currency field for Europe. Regardless of how many rows of data you have that column is only ever going to contain a small number of different values (£, €, etc). You could have one thousand rows or one billion rows and it would still have the same variation in the values. This makes it extremely compressible so you can get extremely large data sets down to a very small footprint.

 

When you then query the data set it loads those columns into memory for execution, so you end up with a column based querying model running directly from memory (which is the reason it is so incredibly extremely fast).

 

Now before you start wondering if this will only work on beefy 64-bit workstations with RAM in double figures I have been assured by the presenter that this works fine on a 2GB netbook! Although he was running the demo on a quad core laptop (presumably with about 8GB of RAM).

 

What about SharePoint 2010 then?

Well, SharePoint 2010 has support for Excel Services, and with SQL PowerPivot for SharePoint 2010 you can publish Excel Workbooks containing PowerPivot data sets directly to SharePoint!

 

This allows you the flexibility to share and present your workbooks with colleagues and other users of the SharePoint platform directly from the browser!

 

Even better than this, if you save an Excel Workbook containing PowerPivot data to a document library, then you can import that into another PowerPivot workbook!

This means that your PowerPivot workbook has actually become a data source in it’s own right, paving the way for true BI applications being built with this technology!

PerformancePoint Services 2010 new features

Some very nice new features for PerformancePoint Services 2010 for creating SharePoint 2010 dashboards.

 

The KPI web parts and filters now execute Asynchronously, so you can expect your web parts to refresh and update without page refreshes (hurrah!)

 

There was also some very nice cool stuff around Time Intelligent Filtering. If you are using SQL Analysis Services then you can use small formula functions like "month" or "year" and it will automatically calculate the query that needs to be called.

 

So for an example, you can create queries to pull through data for:

  • Sales this month ("month")
  • Sales last month ("month-1")
  • Sales this month last year ("(year-1).month")
  • Sales last month, last year ("(year-1).month-1")

 

All without any code and without going into SQL, very impressive.

 

There is also improved SharePoint connection settings so that you can associate SharePoint list data with your OLAP based KPIs. This allows you to use SharePoint lists to configure your scorecard information. But better than that, you can also configure your web parts to allow in-place editing of that scorecard information, so now the editing of the scorecard data can take place for within the dashboard itself!

 

Probably the best feature (and certainly got the most applause from the audience at SharePoint Conference 2009) is single-click deployment to SharePoint from the Dashboard Designer application.

 

You can now setup SharePoint connections to configure your dashboards, and from a single click of the button it will compile and deploy all of your dashboards into your SharePoint environment!

Social Feedback and Activity in SharePoint 2010 – Ratings, Tags and Notes

The social functionality in SharePoint 2010 has been massively improved from the previous versions of SharePoint, and one of the areas is around the concept of Social Feedback.
 
Question: How many times have you found a useful link somewhere on the internet, but had no way to usefull record that and get feedback from your colleagues?
 
Well, SharePoint 2010 social feedback can help with this, you can now "tag" any source on the internet (or intranet) which has a URL. This is stored in your "tags" section on your My Site, and also appears in your "Activity Feed" (which is one of the new areas in the SharePoint 2010 My Site).
 
Other users can also post "notes" relating to your tag, which effectively creates a discussion board around the "tagging" activity, allowing conversations around something that has been tagged.
 
Now, one of the key points is Security Trimming. Lets take this example: what happens if you Tag a document that someone else doesn’t have access to?
 
The good news is that social tagging uses the Search Index to provide security trimming on content that is stored in SharePoint.
 
This provides the capability for senior managers to tag confidential documents (and hold conversations about that using notes) but those tags (and notes) are not visible to anyone who doesn’t have read-access to the document!
 
On top of this is included a Ratings feature, where you can rate content within SharePoint lists (finally, the death of third party "rate my content" web parts).
 
This means that SharePoint 2010 now has similar social feedback functionality as other products like Digg or Delicious, in that you can tag and rate content, and other people can interact with that "tag" creating a discussion.
 
Architecture
All of the Social Feedback information in SharePoint 2010 is stored in a separate "Social Database". This sits alongside the Profile Database.
 
There are then "Gatherers" (Timer Jobs) which will collect all of the changes to both the Social Database and the Profile Database and this is stored in another database for Activity Feeds (the Activity Feed Database) with foreign key pointers back to the Profile Database (so you know who’s activity it is).
 
The performance is impressive, aiming for 2000 requests per second, and in terms of storage they are looking to support over 600,000,000 rows of data! They claim that this is sufficient for activity (including social feedback) for 400,000 users over 5 years!
 
Extensibility
You can also hook into this process yourself. You can build your own "Gatherer" jobs to collect information from any data source that you like.
 
A good example is a CRM database, so that you can show activity in CRM in the My Site Activity Feed, showing when people schedule meetings or achieve sales activites.
 
 
All in all the Social Feedback and Activity in SharePoint 2010 is shaping up very nicely. The performance is something that they are still working on, so don’t expect amazing results in the Beta version, but Microsoft are already using this for all of their employees so the dogfooding will make sure that this is given all the attention that it needs!

Securing SharePoint 2010 Web Servers

 

This was one of the best topics I’ve seen so far at the conference. The amount of concrete information was impressive (and to be honest a bit too much to post here) but there was some great information on how to harden your Web Servers.

 

SharePoint 2010 Security Features

There are a whole load of new features and changes to the SharePoint 2010 product for security.

 

  • ASPX Pages are gone for contributors. You can no longer upload ASPX pages into document libraries unless you have "Designer" permissions! The main reason this becomes possible is because the new Wiki Pages are so much more extensible than they were.
  • Anonymous Users Lockdown feature  now works for Web Services and WSS (SharePoint Foundation 2010)!
  • PowerShell Access – you can now delegate remote scripting rights through PowerShell, so you no longer need the Setup account to perform PowerShell commands. This can be delegated to farm administrators!
  • XSS (Cross Site Scripting) protection is now in place through the headers (although you can turn it off). This can be even be locked down to individual web part properties (through development)!
  • Application Page settings can now be controlled more granularly, so that you can set the master pages used and even swap out individual pages (such as the Error Page). This makes lock downs and branding of these far easier, without breaking the supported state of your environment, and without extensive development!

 

There was then whole load of recommendations for hardening your environments. It’s a bit of a list so apologies for that, but a lot of information to get through:

 

Hardening your Web Application

  • Place your web application directories on a non-system volume. If you have any issues with logging or file access then the I/O operations (or even disk space requirements) could damage the Operating  System!
  • Change the IIS header. By default this will include the SharePoint version number (which means any attacker knows which service packs and critical patches you have installed!). Removing this reduces your public footprint

 

Hardening your Web Servers

Windows Server 2008 takes care of most of the previous recommendations for hardening automatically, but there are still some things that you should do:

 

  • Restrict remote administration of the Registry (no-brainer, but a lot of people forget to do this)
  • Rename Administrator account
  • Delete / Disable unused accounts (again, make sure your dev and test accounts don’t hang around on the web front ends)
  • Use the IUSR instead of IUSR_<serverName>

The IUSR account is a "built in" account so therefore it doesn’t have a password and no-one can login using that account. This makes it much more secure than the Server specific IUSR account that gets created!

 

Hardening SQL 

There’s a whole load about this on the internet. The only one to mention here is change the port number! There are a lot of viruses and malware that will specifically target this port.

 

Hardening your Network

Again, none of this is SharePoint specific, but goes a long way to making sure that your network in general is secure (which is of course best practice for SharePoint systems).

 

Routers:

  • Block unused protocols and ports (see ports required, below)
  • Screen Traffic (e.g. ICMP)
  • Intrusion Detection should be in place

 

Firewall

  • Use packet filtering policies
  • Log your permitted / denied traffic, and make sure those logs are checked (using alerts)
  • Make sure perimeter networks are firewall secured, effectively providing end to end firewall security.

 

Switches

  • Disable any unused services in the switch
  • Do not overly trust VLANS. Just because your traffic is isolated to a VLAN doesn’t mean you shouldn’t still block off the relevant ports and protocols.

 

Ports Required for Web Servers

Note – When SharePoint is installed the communication ports are automatically opened on the Windows Firewall!

 

External:

  • Http 80 / TCP
  • HTTPS 443 / TCP
  • SMTP 25 / TCP

 

Internal*:

  • HTTP 32843/TCP
  • HTTPS 32844 / TCP
  • TCP 32845 / TCP
  • SMB 445 /TCP|UDP

 

* note that "internal" means Web Application –> Service consumtion over WCF. It does not include SQL or inter "server" communications.

« Older Entries Recent Entries »