Creating a Custom Pre-Security Trimmer for SharePoint 2013


What You Will Learn

This blog post will show you how to write your own custom security pre-trimmer for SharePoint 2013. We will take you through the steps of deploying and registering the trimmer before putting the trimmer to work.

Please visit the official MSDN documentation for the overview and
definitive source of documentation of this feature:

http://msdn.microsoft.com/en-us/library/ee819930.aspx

Why Use Pre-Security Trimmers

Pre-trimming refers to pre-query evaluation where the backend rewrites the query adding security information before the index lookup in the search index. Post-trimming refers to post-query evaluation where search results are pruned before they are returned to the user.

We recommend the use of pre-trimming for performance and general correctness; pre-trimming prevents information leakage for refiner data and hit count instances.

Requirements

  • SharePoint 2013 Server
  • Visual Studio 2012
  • A Custom Connector sending claims (see previous post on this blog)

The Trimmer Design

Let’s create a simple pre-security trimmer. A trimmer that reads group membership data from a text file, performs a user lookup for group membership data and then adds claims to the query tree based upon this. In short, the trimmer code needs to figure out which user that is issuing the query, then perform a group membership lookup on that user and then add claims for that user, depending on the group membership.

The Code

This MSDN article offers useful starting tips on creating the security pre-trimmer project in Visual Studio, by adding references to both the Microsoft.Office.Server.Search.dll and the Microsoft.IdentityModel.dll.

Add the following to the using directives at the beginning of the class file, SearchPreTrimmer.cs:

    using System.Security.Principal;
    using Microsoft.IdentityModel.Claims;
    using Microsoft.Office.Server.Search.Administration;
    using Microsoft.Office.Server.Search.Query;

We then define the class as implementing the ISecurityTrimmerPre interface in the class declaration:

public class XmlContentSourcePreTrimmer : ISecurityTrimmerPre

We have to define a few constants at the beginning of the class. These variables may be altered by the static properties given when the trimmer is registered with SharePoint.

    private string _claimType = "http://surface.microsoft.com/security/acl";
    private string _claimIssuer = "customtrimmer";
    private string _claimValueType = ClaimValueTypes.String;
    private string _lookupFilePath = "datafile.txt";

The initialization method of the trimmer may modify a few “constant variables”, primarily claim type and issuer along with the file path to the input data of this trimmer’s group membership data:

    /// <summary>
    /// Initialize the pre-trimmer.
    /// </summary>
    /// <param name="staticProperties">Static properties configured for the trimmer.</param>
    /// <param name="searchApplication">Search Service Application object</param>
    public void Initialize(NameValueCollection staticProperties, SearchServiceApplication searchApplication)
    {
        if (staticProperties.Get("claimtype") != null)
        {
            _claimType = staticProperties.Get("claimtype");
        }

        if (staticProperties.Get("claimissuer") != null)
        {
            _claimIssuer = staticProperties.Get("claimissuer");
        }

        if (staticProperties.Get("datafile") != null)
        {
            _lookupFilePath = staticProperties.Get("datafile");
        }

        RefreshDataFile();
    }    

The AddAccess method of the trimmer is responsible for returning claims to be added to the query tree. We will refresh the group membership data if needed and figure out the user id for key lookup into the group membership structure from the text file. 

    /// <summary> 
    /// Add custom claims to the query tree 
    /// </summary>
    /// <param name="sessionProperties">Session properties collection</param>
    /// <param name="userIdentity">query user identity</param>
    /// <returns>An enumerable of tuples with claims</returns>
    public IEnumerable<Tuple<Claim, bool>> AddAccess(IDictionary<string, object> sessionProperties, IIdentity userIdentity)
    {

        if (null == userIdentity)
        {
            throw new NullReferenceException("Error: AxdAccess method is called with an invalid user identity parameter");
        }

        RefreshDataFile();
        var claims = new LinkedList<Tuple<Claim, bool>>();
        var membership = GetMembership(GetUserId(userIdentity));
        if (membership != null)
        {
            foreach (var member in membership)
            {
                claims.AddLast(new Tuple<Claim, bool>(new Claim(_claimType, member, _claimValueType, _claimIssuer, _claimIssuer), false));
            }
        }

        return claims;
    }

We need a class to act as a simple wrapper of a Dictionary<string, string>. It should essentially serve as a key-value lookup from a user-ID to its group membership data. Each user’s group membership has the form group1;group2;group3, where “;” is used to separate between each group membership entry. This class is simply called Lookup.

    private string GetUserId(IIdentity userIdentity)
    {
        // Run through all the claims of the claims identity, look for the
        // user logon name claim and primary SID to get the current user:
        //   domain\username
        ...
        ...

        return strUser;
    }

    private string[] GetMembership(string key)
    {
        lock (_lock)
        {
            var value = _lookup.Get(key);
            if (!string.IsNullOrEmpty(value))
            {
                return value.Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
            }

            return null;
        }
    }

    private void RefreshDataFile()
    {
        if ((DateTime.Now - _lookupFileStamp).Seconds > 30)
        {
            lock (_lock)
            {
                _lookupFileStamp = DateTime.Now;
                if (File.Exists(_lookupFilePath))
                {
                    _lookup = Lookup.Load(_lookupFilePath);
                }
            }
        }
    }

Performance Considerations 

Consider the following tips to improve the overall performance with this trimmer as a starting point:

  • Use multiple Lookup classes in a hash-table on a given key.
  • Use a compressible stream and binary serialize the Lookup data.
  • Use IPC (NetPipe) to talk to a local service that holds the more efficient key-value Lookups.

Deploying Trimmer

After you have built the custom security trimmer project, you must deploy it to the global assembly cache on any server in the Query role.

  1. On the Start menu, choose All Programs, choose Microsoft Visual Studio 2010, and then choose Visual Studio Tools and open a Visual Studio command prompt.
  2. To install the SearchPreTrimmer.dll, type the following the command prompt

    gacutil /i <ExtractedFolderPath>\PreTrimmer\bin\Debug\SearchPreTrimmer.dll

  3. As the last step of deploying the trimmer, we need to learn about the token of the DLL. Type the following the command prompt

    gacutil /l SearchPreTrimmer
    Write down the token listed for the newly added DLL.

Registering Trimmer

  1. Open the SharePoint Management Shell.
  2. At the command prompt, type the following command:

    New-SPEnterpriseSearchSecurityTrimmer -SearchApplication “Search Service Application”
    -typeName “CstSearchPreTrimmer.SearchPreTrimmer, CstSearchPreTrimmer,
    Version=1.0.0.0, Culture=neutral, PublicKeyToken=token”

    where token is the text copied from the gacutil /l command above. Example: New-SPEnterpriseSearchSecurityTrimmer -SearchApplication “Search Service Application” -typeName “CstSearchPreTrimmer.SearchPreTrimmer, SearchPreTrimmer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=4ba2b4aceeb50e6d” -Id 2

  3. Restart the Search Service by typing

    net restart sphostcontrollerservice

Testing the Trimmer

Now, you can issue queries and you can see the beauty of the pre-trimmer logic in action for every query evaluated. Try modifying the datafile.txt for different group membership per user (keep in mind that contents of the data file are only loaded every 30 seconds). The 30 second refresh interval is a defined constant in the code. Enjoy!

Acknowledgements

Author: Sveinar Rasmussen (sveinar).

 

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-55-87-Trimming/8357.PreTrimmer.zip


Comments (18)

  1. Hi,

    Thanks for wonderful Article and detailed steps!!!.

    Am not able to debug the Security Trimmer Dll ,but i added few logs in code initially which am able to see log comments but when i added bit extra logic and extra logging,deployed to GAC i still see still old log comments despite of repetitive deployment (around 20 times) πŸ™ ..

    Please let me know how to referring to latest to DLL ,sorry if this too native question..

    Between can we achieve same using rest API (client) of Fast to consume Search Results and see if security trimming works?

  2. Helloooo, debugging the Security Trimmer DLL is done by attaching to the proper NodeRunner.exe binary. To locate the correct binary to attach with a debugger, you could perhaps use ProcExplorer and view the parameters of those processes. Look for the process that has Query in its parameters.

    Just like I do in this Powershell snippit which should kill the correct NodeRunner instance:

    Get-WmiObject win32_process -Filter "name = 'noderunner.exe'" | ForEach-Object { if ($_.CommandLine -like '*Query*') { Stop-Process $_.ProcessId }

    The REST API on the client side is transparent with regards to security trimmers. The trimmer work is already done when this stuff reaches the client side. With pre-trimming, it is not possible to detect that a trimmer has been run πŸ™‚

  3. lior says:

    hi,

    Im looking for a away to return results regardless of the user access rights, for example, writing the loging name of a user on a metadata field and check that field in Custom Pre-Security Trimmer phase but i dont understand how to add such a claim.

    can this be done using Custom Pre-Security Trimmer ?

  4. Lior, you would be looking at adding a claim that everyone can see. It is not possible to disable security trimming in one custom pre-security trimmer. You cannot add a claim to match 'everyone' in a Windows SID terminology (by design, SID claims are not accepted from a pre-security trimmer). That said, it should be possible to add a claim that represents authenticated people in SharePoint. Perhaps you could add a claim of the following type: sharepoint.microsoft.com/…/isauthenticated with value set to true ?

  5. RK says:

    Hi Sveinar,

    Thanks for you reply with specific to debugging of security trimmer πŸ™‚ that  helped me a lot

    we have requirement for having Custom Security Groups(non-AD security groups) which are local to external source and those groups are not a part of AD .

    Below is what are looking to implement .please let me know how can we achieved ?

    1)SP crawler has to index External source which is outside of share point that has it own security groups.

    2)While crawling the document i want to set this security group which is local to that external source .

    3)while user logs in for searching content,we want to trim the results based on logged in user custom security group (assuming we have those details from http header for that logged in user) .

    Regards,

    RK.

  6. Sveinar Rasmussen (Principal SDE) says:

    While I haven’t set up a similar case with multiple SharePoint farms where each farm has its own security groups, I do believe that one obstacle here to overcome is the ACLs that will be associated with the external source contents (as seen from the SP web-crawl). The general rule is to use the BCS framework to set the SecurityDescriptor with custom claims, where the custom claims are those other external (and different) security groups. Similarly, when the user logs on to query for content, a Pre-trimmer can be used here to perform an analogous mapping from the local user to its corresponding external security group memberships. Thus, this should be possible but it requires a custom BCS connector to supply the proper claims/ACLs on the content and a pre-trimmer to perform a successful secure lookup. With the SP Crawler, I do believe that it is not possible to customize the ACLs that it puts on the content crawled.

    For additional information, be sure to check out the blog post on β€œCreating Custom Connector Sending Claims with SharePoint 2013”.

  7. Anonymous says:

    Hi Sveinar,

    what actual value we can pass in pre-security trimming member value of AddAccess method. I have two domain A, and B.  so I want to add domain A users claim with the search query in presecurity trimming for the items where domain B have permissions only. Can it be done through this?. what actual value we need to pass in member field(group or loginname of Domain A user) i.e Can we add like domainAuser1 as member value in AddAccess method.

  8. Hello Atul, the trimmer's AddAccess returns a list of custom claims, preferrably. For instance:

    claims.AddLast(new Tuple<Claim, bool>(

               new Claim("schemas.happy.bdc.microsoft.com/…/acl", loginDomainUser), false));

    Please note that we prevent system accounts we we use SIDs or similar to be returned. This is by design since giving search access to content should be really explicit. The way to do that with domain users, is to wrap the loginname of the domain user here. It requires that the same property claim is sent off by the BCS indexing input.

  9. Anonymous says:

    So if I Pass this like

    claims.AddLast(new Tuple<Claim, bool>(

              new Claim("schemas.happy.bdc.microsoft.com/…/acl", loginDomainUser), false));

    where "schemas.happy.bdc.microsoft.com/…/acl" as same which you have written and I only change the logindomainuser value as domainAatul, then I would be able to see the document, or do in need to change the claim type url "schemas.happy.bdc.microsoft.com/…/acl" also to map with our environment(Dev, UAT, production)…

  10. Atul, a claim is just a statement where the "schemas.happy.bdc.microsoft.com" is just a claim type identifier together with a claim value (group or domain name for instance). For Windows security identifiers like SIDs, these are not allowed to pass through the pre-trimmer for security reasons (these would be of the claim type "schemas.microsoft.com/…/primarysid"). If that was allowed, it would enable pre-trimmers to unlock content without changing the crawler source, if the latter was submitting SID security ACEs. Instead, the work must be symmetrical for this to work. E.g. you can define a custom claim type of your own like "https://atuls.happy.claim.type.here.com&quot; with the value of DomainA/atul in the pre-trimmer. Similarly, you need to ensure that the SecurityDescriptor from the BCS framework on the crawler/connector side match up to this same custom claim type. For more details on claims: technet.microsoft.com/…/ee913589.aspx

  11. Claims not working says:

    I am searching with user domanuser1, but user1 dont have permission on one file. Atul has permission on file so I am passing this like

    claims.AddLast(new Tuple<Claim, bool>(new Claim(_claimType,  "domainatul" _claimValueType, _claimIssuer, _claimIssuer), false));

    I have following configuration

    private string _claimType = "surface.microsoft.com/…/acl";

           private string _claimIssuer = "customtrimmer";

           private string _claimValueType = ClaimValueTypes.String;

           private string _lookupFilePath = @"c:membershipdatafile.txt";

    During query time page is htting the code in vs2013, means trimmer is registered perfectly.

    But this is not returning results based on "domaintatul". Do I need to change anything.

  12. Hello "Claims not working" πŸ™‚ A few comments to help you on your way…

    With custom claims, you also need to tag the claim to the documents using the BCS framework. That is, you need to submit documents with the proper claim ACL to the index. Did you follow the steps of blogs.msdn.com/…/creating-custom-connector-sending-claims-to-sharepoint-2013.aspx for that part?

    On a minor note, I am assuming that the custom connector is in place and that the claim type naturally does not contain any "…" but instead _claimType = "surface.microsoft.com/…/acl" (not "surface.microsoft.com/…/acl") and that the key for 'domainatul' results in a few group membership entries in the datafile.txt.

  13. Add claims in OOTB sharepoint connector says:

    Thanks Sveinar for the clarification.

    But If  we write my own BCS connector then we can handle the user permission. I was thinking to send custom claims of a new user in OOTB indexed document while searching. We have some content on which some users dont have permissions. After crawling using OOTB fileshare Sharepoint connector, i want to add the claim of a users during search.

  14. Indeed, the SharePoint Pre-Trimmer is prevented from allowing SID claims to be processed by design. This was a security concern during development, and we can reassess if this concern is still valid. Can you file a feature request perhaps to lift the restriction to allow pre-trimmers to emit any claims possible? That way, you can then "modify" OOTB indexed document permissions from the SharePoint connector etc. An argument is that pre-trimmers are considered trusted code deployed by trusted parties only with admin access. For more details on this, see my post on 22 Jun 2015 1:21 PM.

  15. Saber Ghanmi says:

    Hello Thanks fo this article.

    I need to create SharePoint 2013 search service that return all documents in SharePoints (Anythings ). I created the SearchPreTrimmer and in the method: public IEnumerable<Tuple<Claim, bool>> AddAccess I added this code

    claims.AddLast(new Tuple<Claim, bool>(new Claim(_claimType, "domain\Administrator"), false));

    I use local Active Directory. I need this Trimer for the documents in SharePoint.

    But is not working.

  16. Saber, the Security Pre-Trimmer is not given the permission to override the SIDs with having a claim with values like domainusername. The O365 Security and Threat models deemed that far too risky back in 2012. One could argue that, given special permissions, a custom security trimmer has system access and should be able to do anything to expose every document from any of the out-of-the-box crawlers. Please see my comments from August 2015 on this above. As such, I would urge you to submit a feature request on this to weigh in on the security discussion in SharePoint 2016 for instance… No promises though as this has to go through proper O365 security review if even accepted.

  17. Saber Ghanmi says:

    Hi Sveinar, thanks for the reply.

    Do you have any suggestion to do what I expected. I overrieded the Search WebPart and runed it with runwithelevatedprivileges with an admin account bu it didn't work.

    Thanks you for your help

  18. Saber, overriding the SearchWebPart with elevated privileges will not have any affect on the behavior here. Sorry. The logic in the claims encoding prevents claims like these SID claims from being encoded in the first place. This translates to no-match for the search engine term lookup – thus, by design, your documents will not be surfaced. I would recommend filing a feature request or use the BCS framework to supply your own custom claims for the documents submitted.