tirsdag den 16. juli 2013

MongoDB,follow up

While venturing into the world of MongoDB I had a lot of “aha” experiences. I can’t even remember all of them and write about them, but a few stands out.

One common way to start on a new .NET/MS SQL project is to start at the database layer. weather your sitting in Enterprise manager creating tables and relations, or coding up your classes in a POCO model, your tend to think more about relations of your data, and ways your application would need to “fetch” data, and worry about performance later.And, god knows why, for some reason in my mind set, the less data you would need to get per dataset, the faster it had to be, right ?

But for me, working with MongoDB had me thinking more about performance first, ways to insert/update, and THEN how the application would access this. A lot of the performance gains can be archived though using $INC instead of updating complete documents. That can be quite hard to implement if you got a documents that is not just a few strings and integers.

So to challenge my self I set my self the goal to create a browser based RPG game. My goal was no page would take more than 20 milliseconds to generate. As I progressed and the game (and database) got more complex I found my self doing more and more “database” stuff directly in the webpages than in a “prober” DAL layer, to get that “extras” performance. While load testing I would also start getting issues with concurrency ( 2 webpages updating the same document )

So after a few weeks I deleted everything and started over, but this time, wiser from experience, I started with another mind set, that would include measuring each document and field in regard to concurrency. ( Do a Google search on Atomic Updates and concurrency in mongo dB if your interested in this. ) To avoid having tons of small “update” statements placed all over my code, I started thinking in two ways to handle updates. One would be the traditionally “entity framework” thinking where all my classes would have some kind of “tracking” ability to detect updated. I still believe this is the best way, but it is a LOT of work to implement. I also started wondering if you couldn’t just automated all this, and while goggling that, I came across UpdateDocumentBuilder .

This class is really cool, but there are 2 big issues with this class. The first issue comes from Mongo dB. You cannot pop/push multiple classes from an array in one update statement. Sure you could extend this class to support splitting up updates into many updates, but I will leave that up to someone else. The other issue, is the fact it uses $SET on all updates, and not embracing the whole “we don’t care in what order updates gets done” thinking. So I created a new version of this class, that uses $INC on all number values. To ensure support for atomic updates I add an lastupdatedon and use that in my query incase if needed.

Odata. There are so many skilled people out there that would be much better at creating this than me, but until that happens I had to work with my own implementation. I found a ton of issues in my last post, so I’ve uploaded a new version that has less errors and better support for Microsoft OData client ( adding a Service reference to your odata feed ). Beware with that. It doesn’t support properties starting with underscoore  _. Also if you want to expose _t use a different name, add bsonignore to avoid double updates, and then use my filterinformation attribute to support querys on it ( see jira 742 )
Basicly just use the new BSONFilter2 from your web api controller, You can test different querys by running then TestBSONFilter project.

You can see the updated filter, and my updated UpdateDocumentBuilder in this test project.

torsdag den 25. april 2013

MongoDB and OData

updated: 01-05-2013: small bug fixes in source to JSONFilter class.
updated: 01-05-2013: add support for resolving bson attributes.
updated: 16-07-2013: This is outdated, see new post here

I decided to try and spend some time playing with MongoDB. It didn’t take more than a few hours before I feel completely in love with MongoDB, so to put it to good use I decided to wrap OData on top of it. One of the big problems using OData and entity framework is the fact it will try and save as much as possible in memory. That is fine if you got a small database of a few MB/GB but if you got millions and millions of rows, that starts to get troublesome. So it felt important to find a solution that would send all database query's to the database
At first I tried MongOData. At first it looked really promising but what makes OData strong is the ability to filter data using QueryInterceptor and ChangeInterceptor and that doesn't seem to work with that solution ( The developers added this feature within a few days, after asking about this feature. ). Next I tried playing around with Web Api OData. The information out on the “Google web” is really “conflicting”, since 80% of the code samples you will find, doesn’t work but once you get something working it looks really promising. Until you decide to something simple like . Simple comparison works but searching arrays, or doing text search just doesn’t work. Than I came across LinqToQuerystring . That looked really good, they even have a special test project for testing against MongoDB, but after realizing you cannot query on arrays and collections I quickly ruled that out again. ( Apparently Linq To Query now supports any query's, but at the time of writing this, it didn't. Thank you for that information Pete Smith :-) )
After a few days of testing different things I decided to throw my love at Web Api OData. ( I went back to MongOData again, but more about that later )

So, for each step, try out the following queries. Replace /odata/ with /api/ in the first sample, since the odata route haven't been added yet.

So lets first try and have quick look at how to expose a Collection (table in MongoDB ) as OData using Web Api OData.

http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=name eq 'Cruz Roudabush'
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=(substringof('Vaill', name) eq true)
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=skills/any(x: x eq 'VB' )
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=skills/any(x: (x eq 'VB') or (x eq 'XML'))
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=skills/any(x: (x eq 'VB') and (x eq 'MongoDB'))
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=skills/any(x: (substringof('L', x) eq true))

http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=addresses/any(x: x/city eq 'Anchorage' )
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=addresses/any(x: x/city eq 'New York' )
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=addresses/any(x: x/city eq 'New York' and x/zip eq '10018' )
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=addresses/any(x: x/city eq 'New York' or x/zip eq '99515' )
http://localhost/webapiodatatest/odata/Persons?$top=5&$filter=addresses/any(x: (substringof('York', x/city) eq true) )

Create an empty MVC 4 project, open package manager and install MongoDB C# Driver and WebApi.OData … just to be safe, update all NuGet packages by issuing Update-Package in the Package Manager Console.

The easy and simple way, would be to use the old syntax.
Add an empty API controller, add a GET function and decorate it with Queryable, and return an IQueryable(of person)


Imports MongoDB.Driver.Linq

Public Class PersonsController
Inherits ApiController

Public Function [get]() As IQueryable(Of person)
Return repository.Persons.AsQueryable
End Function

End Class

But, to embrace MVC a bit more, and open up for more advanced queries. First we add an OData route in WebApiConfig.vb , then we change the controller to an EntitySetController. You can either use Queryable decoration on all functions you want to allow it for, or just generally open up for this on all functions by adding EnableQuerySupport here

Public Class WebApiConfig
Public Shared Sub Register(ByVal config As HttpConfiguration)
Dim model = repository.getModel()
config.Routes.MapODataRoute(routeName:="odata", routePrefix:="odata", model:=model)

Public Class PersonsController
Inherits OData.EntitySetController(Of person, String)

Protected Overrides Function GetEntityByKey(key As String) As person
Return repository.Persons.AsQueryable.Where(Function(x) x.id = key).FirstOrDefault
End Function

Public Overrides Function [get]() As IQueryable(Of person)
Return repository.Persons.AsQueryable
End Function

End Class

But there is a slight problem with the 2 above solutions.
The Mongo DB C# driver is very limited in what queries it understands/supports. Simple stuff like “name eq” and “age gt 10” works.  But if you want to search a string, or search lists or sub documents you will get all kinds of errors.
We can, however, try and process the odata query our self. One “quick” way of doing that would be to look at LinqToQuerystring but that doesn’t support any queries, so that wouldn’t fit my needs. So I started looking at what it would take to “parse” the query manually and translate it to something the Mongo DB c# driver would understand.

If we look at QueryOptions ( a property you get access too when inheriting EntitySetController or add as parameter when inheriting from ODataController, we get easy access to the “query tree”, I think we can call it and so something like this

Public Class PersonsController
Inherits OData.EntitySetController(Of person, String)

Protected Overrides Function GetEntityByKey(key As String) As person
Return repository.Persons.AsQueryable.Where(Function(x) x.id = key).FirstOrDefault
End Function

Public Overrides Function [get]() As IQueryable(Of person)
Dim cursor = JSONFilter.QueryOptionsToCursor(Of person)(repository.Persons, QueryOptions, 5)
Return cursor.AsQueryable
End Function

End Class

We have 3 options, when we want to parse the OData query.

1) We can do like Microsoft and turn it into LINQ but “turn down” some of their aggressive “isnullable” expressions to make expressions more eatable by the Mongo DB Driver. that works some of the way, but once you start trying to do Any queries (Searching lists and collections) you get into all kinds of problems, most of them with the Mongo DB driver.

2) we can parse the queries one by one, and try and express the as an IMongoQuery . That is actually pretty easy with all the simple stuff, but as soon as you get to functions, any/all queries and such, it just gets very complicated, and hard to rewrite.

3) we can parse the queries to the JSON queries language mongo dB speaks, and then load that as an Query Document and execute that.

I began writing code for all 3 solutions and got furthest using option 3.

So in the end we end up with a controller that looks like this

Public Class PersonsController
Inherits OData.ODataController

Function [Get](key As String) As person
Return repository.Persons.FindOne(MongoDB.Driver.Builders.Query.EQ("id", New MongoDB.Bson.BsonString(key)))
End Function

Public Function [Get](QueryOptions As Web.Http.OData.Query.ODataQueryOptions) As OData.PageResult(Of person)
Dim inlinecount As Integer = -1
If QueryOptions.InlineCount IsNot Nothing Then
If QueryOptions.InlineCount.Value = Web.Http.OData.Query.InlineCountValue.AllPages Then
' we cannot use cursor twice, so need to open a new one, and do the count on that
Dim countcursor = JSONFilter.QueryOptionsToCursor(Of person)(repository.Persons, QueryOptions)
inlinecount = countcursor.Count
End If
End If

Dim cursor = JSONFilter.QueryOptionsToCursor(Of person)(repository.Persons, QueryOptions, 2)
' yeah, yeah ... sue me, for being creative here
Dim NextPageLink As String = Request.RequestUri.AbsolutePath
If QueryOptions.Skip Is Nothing Then
NextPageLink = Request.RequestUri.AbsoluteUri & "&$skip=" & cursor.Limit
NextPageLink = Request.RequestUri.AbsoluteUri
NextPageLink = NextPageLink.Replace("$skip=" & cursor.Skip, "$skip=" & cursor.Skip + cursor.Limit)
End If

If inlinecount > -1 Then
Return New OData.PageResult(Of person)(cursor, New Uri(NextPageLink), inlinecount)
Return New OData.PageResult(Of person)(cursor, New Uri(NextPageLink), Nothing)
End If
End Function

End Class

And JSONFilter

' filters explained
' http://blogs.msdn.com/b/alexj/archive/2012/12/06/parsing-filter-and-orderby-using-the-odatauriparser.aspx

' inspiration
' http://www.symbolsource.org/MyGet/Metadata/aspnetwebstacknightly/Project/Microsoft.AspNet.WebApi.OData/4.0.0-rtm-130106/Release/Default/System.Web.Http.OData/System.Web.Http.OData/System.Web.Http.OData/OData/Query/Expressions/FilterBinder.cs?ImageName=System.Web.Http.OData

Imports MongoDB.Driver
Imports MongoDB.Bson.BsonExtensionMethods

Imports System.Web.Http.OData.Query
Imports Microsoft.Data.Edm
Imports Microsoft.Data.OData.Query.SemanticAst
Imports Microsoft.Data.OData.Query
Imports MongoDB

Imports System.Reflection

Public Class JSONFilter
Private _model As IEdmModel
Protected Sub New(model As IEdmModel)
_model = model
End Sub

Public Shared Function QueryOptionsToCursor(Of T)(collection As MongoCollection, options As ODataQueryOptions,
Optional defaultPagesize As Integer = 2) As MongoDB.Driver.MongoCursor(Of T)
Dim _t As Type = GetType(T)
Dim q = Builders.Query.EQ("_t", New Bson.BsonString(_t.Name))
Return QueryOptionsToCursor(Of T)(collection, options, q, defaultPagesize)
End Function
Public Shared Function QueryOptionsToCursor(Of T)(collection As MongoCollection, options As ODataQueryOptions,
basequery As IMongoQuery, Optional defaultPagesize As Integer = 2) As MongoDB.Driver.MongoCursor(Of T)
Dim query As MongoDB.Driver.IMongoQuery = Nothing
Dim cursor As MongoDB.Driver.MongoCursor(Of T)
Dim querylimit As Integer = defaultPagesize
Dim queryskip As Integer = 0
If (options.Top IsNot Nothing) Then querylimit = options.Top.Value
If (options.Skip IsNot Nothing) Then queryskip = options.Skip.Value

If options.Filter IsNot Nothing Then
Dim jsonQuery As String = JSONFilter.BindFilterQueryOption(options.Filter)
Debug.WriteLine("db." & collection.Name & ".find( " & jsonQuery & ");")

If basequery IsNot Nothing Then
jsonQuery = "{$and : [" & jsonQuery & "," & basequery.ToJson & "]}"
End If
Dim doc As Bson.BsonDocument = MongoDB.Bson.Serialization.BsonSerializer.Deserialize(Of Bson.BsonDocument)(jsonQuery)
Dim queryDoc = Bson.Serialization.BsonSerializer.Deserialize(Of Bson.BsonDocument)(jsonQuery)
cursor = collection.FindAs(Of T)(New QueryDocument(queryDoc))
If basequery IsNot Nothing Then
cursor = collection.FindAs(Of T)(basequery)
cursor = collection.FindAllAs(Of T)()
End If
End If
If queryskip > 0 Then cursor.SetSkip(queryskip)

If options.OrderBy IsNot Nothing Then
For Each orderby As System.Web.Http.OData.Query.OrderByPropertyNode In options.OrderBy.OrderByNodes
If orderby.Direction = Microsoft.Data.OData.Query.OrderByDirection.Ascending Then
End If
End If

Return cursor
End Function

Public Shared Function BindFilterQueryOption(filterQuery As FilterQueryOption) As String
If filterQuery IsNot Nothing Then
Dim binder As New JSONFilter(filterQuery.Context.Model)
Return "{" & binder.Bind(filterQuery.FilterClause.Expression) & "}"
End If
Return ""
End Function

Protected Function Bind(node As QueryNode) As String
Dim collectionNode As CollectionNode = TryCast(node, CollectionNode)
Dim singleValueNode As SingleValueNode = TryCast(node, SingleValueNode)

If collectionNode IsNot Nothing Then
Select Case node.Kind
Case QueryNodeKind.CollectionNavigationNode
Dim navigationNode As CollectionNavigationNode = TryCast(node, CollectionNavigationNode)
Return BindNavigationPropertyNode(navigationNode.Source, navigationNode.NavigationProperty)

Case QueryNodeKind.CollectionPropertyAccess
Return BindCollectionPropertyAccessNode(TryCast(node, CollectionPropertyAccessNode))
End Select
ElseIf singleValueNode IsNot Nothing Then
Select Case node.Kind
Case QueryNodeKind.BinaryOperator
Return BindBinaryOperatorNode(TryCast(node, BinaryOperatorNode))

Case QueryNodeKind.Constant
Return BindConstantNode(TryCast(node, ConstantNode))

Case QueryNodeKind.Convert
Return BindConvertNode(TryCast(node, ConvertNode))

Case QueryNodeKind.EntityRangeVariableReference
Return BindRangeVariable(TryCast(node, EntityRangeVariableReferenceNode).RangeVariable)

Case QueryNodeKind.NonentityRangeVariableReference
Return BindRangeVariable(TryCast(node, NonentityRangeVariableReferenceNode).RangeVariable)

Case QueryNodeKind.SingleValuePropertyAccess
Return BindPropertyAccessQueryNode(TryCast(node, SingleValuePropertyAccessNode))

Case QueryNodeKind.UnaryOperator
Return BindUnaryOperatorNode(TryCast(node, UnaryOperatorNode))

Case QueryNodeKind.SingleValueFunctionCall
Return BindSingleValueFunctionCallNode(TryCast(node, SingleValueFunctionCallNode))

Case QueryNodeKind.SingleNavigationNode
Dim navigationNode As SingleNavigationNode = TryCast(node, SingleNavigationNode)
Return BindNavigationPropertyNode(navigationNode.Source, navigationNode.NavigationProperty)

Case QueryNodeKind.Any
Return BindAnyNode(TryCast(node, AnyNode))

Case QueryNodeKind.All
Return BindAllNode(TryCast(node, AllNode))
End Select
End If

Throw New NotSupportedException([String].Format("Nodes of type {0} are not supported", node.Kind))
End Function

Private Function findType(name As String, fullname As String) As Type
Dim res = From assembly In AppDomain.CurrentDomain.GetAssemblies()
From type In assembly.GetTypes()
Where type.Name = name
For Each t In res
If t.type.FullName = fullname Then Return t.type
Return Nothing
End Function

Private Function BSONPropertyNode(sp As Library.EdmStructuralProperty) As String
Dim PropertyName As String = sp.Name
'Dim et As Microsoft.Data.Edm.Library.EdmEntityType = sp.DeclaringType
'Dim et As Microsoft.Data.Edm.Library.EdmStructuredType = sp.DeclaringType

Dim ClassName As String
Dim PropertyClassType As Type = Nothing
If TypeOf sp.DeclaringType Is Microsoft.Data.Edm.Library.EdmEntityType Then
Dim et As Microsoft.Data.Edm.Library.EdmEntityType = sp.DeclaringType
ClassName = et.Namespace & "." & et.Name
PropertyClassType = findType(et.Name, ClassName)

ElseIf TypeOf sp.DeclaringType Is Microsoft.Data.Edm.Library.EdmComplexType Then
Dim et As Microsoft.Data.Edm.Library.EdmComplexType = sp.DeclaringType
ClassName = et.Namespace & "." & et.Name
PropertyClassType = findType(et.Name, ClassName)
End If
If PropertyClassType IsNot Nothing Then
Dim pdc As ComponentModel.PropertyDescriptorCollection = ComponentModel.TypeDescriptor.GetProperties(PropertyClassType)
Dim prop As ComponentModel.PropertyDescriptor = pdc.Find(PropertyName, False)

Dim BsonElement As MongoDB.Bson.Serialization.Attributes.BsonElementAttribute = prop.Attributes.Item(GetType(MongoDB.Bson.Serialization.Attributes.BsonElementAttribute))
Dim BsonId As MongoDB.Bson.Serialization.Attributes.BsonIdAttribute = prop.Attributes.Item(GetType(MongoDB.Bson.Serialization.Attributes.BsonIdAttribute))
If BsonElement IsNot Nothing Then
If Not String.IsNullOrEmpty(BsonElement.ElementName) Then
Return Convert.ToString(BsonElement.ElementName)
End If
ElseIf Not BsonId Is Nothing Then
Return Convert.ToString("_id")
End If
End If
Return PropertyName
End Function

Private Function BindCollectionPropertyAccessNode(node As CollectionPropertyAccessNode) As String
Return BSONPropertyNode(node.Property)
'Return Convert.ToString(node.[Property].Name)
'Return Bind(collectionPropertyAccessNode.Source) & "." & Convert.ToString(collectionPropertyAccessNode.[Property].Name)
End Function

Private Function BindNavigationPropertyNode(singleValueNode As SingleValueNode, edmNavigationProperty As IEdmNavigationProperty) As String
Return Convert.ToString(edmNavigationProperty.Name)
'Return Bind(singleValueNode) & "." & Convert.ToString(edmNavigationProperty.Name)
End Function

Private Function BindAllNode(allNode As AllNode) As String
Dim isInList As String = "[]"
Dim anylist As String = Bind(allNode.Body)
anylist = anylist.Replace("""" & allNode.RangeVariables.First().Name & """ : ", "")
Dim result As String = Bind(allNode.Source) & " : {$all : [" & anylist & "]}"
Return result
End Function

Private Function BindAnyNode(Node As AnyNode) As String
Dim source = Bind(Node.Source)
Dim anylist As String = Bind(Node.Body)
Dim var = Node.RangeVariables.First.Name

If TypeOf Node.Source.ItemType Is Library.EdmComplexTypeReference Then

Dim result As String = Bind(Node.Source) & " : {$elemMatch: {" & anylist & "} }"
' list of class ?
'Throw New Exception("any on complex types not supported")
Return result
' list of values
'Dim isInList As String = "[]"
'anylist = anylist.Replace("""" & Node.RangeVariables.First().Name & """ : ", "")
'Dim result As String = Bind(Node.Source) & " : {$in : [" & anylist & "]}"
anylist = anylist.Replace("""" & Node.RangeVariables.First().Name & """", """" & Bind(Node.Source) & """")
Dim result As String = anylist
Return result

End If
End Function

Private Function BindNavigationPropertyNode(singleEntityNode As SingleEntityNode, edmNavigationProperty As IEdmNavigationProperty) As String
Return Convert.ToString(edmNavigationProperty.Name)
'Return Bind(singleEntityNode) & "." & Convert.ToString(edmNavigationProperty.Name)
End Function

Private Function BindSingleValueFunctionCallNode(singleValueFunctionCallNode As SingleValueFunctionCallNode) As String
Dim arguments = singleValueFunctionCallNode.Arguments.ToList()
Select Case singleValueFunctionCallNode.Name
'Case "concat"
' Return Convert.ToString(singleValueFunctionCallNode.Name) & "(" & Bind(arguments(0)) & "," & Bind(arguments(1)) & ")"
'Case "length", "trim", "year", "years", "month", "months", _
' "day", "days", "hour", "hours", "minute", "minutes", _
' "second", "seconds", "round", "floor", "ceiling"
' Return Convert.ToString(singleValueFunctionCallNode.Name) & "(" & Bind(arguments(0)) & ")"

Case "substringof"
Dim val As String = Bind(arguments(0))
If val.StartsWith("'") And val.EndsWith("'") Then
val = val.Substring(1, val.Length - 2)
End If
Return """" & Bind(arguments(1)) & """ : /" & val & "/i"
Case Else
Throw New NotImplementedException()
End Select
End Function

Private Function BindUnaryOperatorNode(unaryOperatorNode As UnaryOperatorNode) As String
Return ToString(unaryOperatorNode.OperatorKind) & "(" & Bind(unaryOperatorNode.Operand) & ")"
End Function

Private Function BindPropertyAccessQueryNode(node As SingleValuePropertyAccessNode) As String
Dim source = Bind(node.Source)
If source = "$it" Then
Return BSONPropertyNode(node.Property)
'Return Convert.ToString(node.[Property].Name)
Return source & "." & BSONPropertyNode(node.Property)
'Return Convert.ToString(source & "." & node.[Property].Name)
End If
'Return Bind(singleValuePropertyAccessNode.Source) & "." & Convert.ToString(singleValuePropertyAccessNode.[Property].Name)
End Function

Private Function BindRangeVariable(nonentityRangeVariable As NonentityRangeVariable) As String
Return nonentityRangeVariable.Name.ToString()
End Function

Private Function BindRangeVariable(entityRangeVariable As EntityRangeVariable) As String
Return entityRangeVariable.Name.ToString()
End Function

Private Function BindConvertNode(convertNode As ConvertNode) As String
Return Bind(convertNode.Source)
End Function

Private Function BindConstantNode(constantNode As ConstantNode) As String
If TypeOf constantNode.Value Is String Then
Return [String].Format("'{0}'", constantNode.Value)
ElseIf TypeOf constantNode.Value Is DateTime Then
Dim t1 = New MongoDB.Bson.BsonDateTime(DirectCast(constantNode.Value, DateTime))
Dim t2 = "ISODate(""" & t1.AsBsonValue.ToString & """)"
Return t2
ElseIf TypeOf constantNode.Value Is Integer Then
Return constantNode.Value.ToString
End If
Return constantNode.Value.ToString()
End Function

Private Function BindBinaryOperatorNode(binaryOperatorNode As BinaryOperatorNode) As String
Dim left = Bind(binaryOperatorNode.Left)
Dim right = Bind(binaryOperatorNode.Right)
If binaryOperatorNode.Left.Kind = QueryNodeKind.SingleValueFunctionCall Then
Return left
ElseIf binaryOperatorNode.OperatorKind = BinaryOperatorKind.Equal Then
Return """" & left & """ : " & right
ElseIf binaryOperatorNode.OperatorKind = BinaryOperatorKind.Or Then
Return "$or : [ {" & left & "}, {" & right & "} ]"
ElseIf binaryOperatorNode.OperatorKind = BinaryOperatorKind.And Then
Return "$and : [ {" & left & "}, {" & right & "} ]"
Return """" & left & """ : {" & ToString(binaryOperatorNode.OperatorKind) & " : " & right & "}"
End If

End Function

Private Overloads Function ToString(binaryOpertor As BinaryOperatorKind) As String
Select Case binaryOpertor
Case BinaryOperatorKind.Add
Return "$inc"
Case BinaryOperatorKind.[And]
Return "$and"
'Case BinaryOperatorKind.Divide
' Return "/"
Case BinaryOperatorKind.Equal
Return "$eq"
Case BinaryOperatorKind.GreaterThan
Return "$gt"
Case BinaryOperatorKind.GreaterThanOrEqual
Return "$gte"
Case BinaryOperatorKind.LessThan
Return "$lt"
Case BinaryOperatorKind.LessThanOrEqual
Return "$lte"
Case BinaryOperatorKind.Modulo
Return "$mod"
'Case BinaryOperatorKind.Multiply
' Return "*"
Case BinaryOperatorKind.NotEqual
Return "$ne"
Case BinaryOperatorKind.[Or]
Return "$or"
'Case BinaryOperatorKind.Subtract
'Return "-"
Case Else
Throw New NotSupportedException([String].Format("Opdaterator of type '{0}' are not supported", binaryOpertor))
'Return Nothing
End Select
End Function

Private Overloads Function ToString(unaryOperator As UnaryOperatorKind) As String
Select Case unaryOperator
Case UnaryOperatorKind.Negate
Return "!"
Case UnaryOperatorKind.[Not]
Return "NOT"
Case Else
Return Nothing
End Select
End Function

End Class

JSONFilter being one of 3 classes that can parse the ODataQueryOptions into something mongo db will understand, you can download a test project here.

mandag den 25. februar 2013

ASP.NET and/or MVC websites and Claims based Authentication

Implementing WIF on a website is pretty straight forward. There are ton’s of guide out there on how to do it …. By doing a bit of magic in web.config you can make any website support claims based authentication. ( even Exchange 2010 and Citrix  )

If you are a developer, implementing WIF on your website is just as easy. You install WIF SDK and run FedUtil.exe who will handle configuring your web.config for you …

But what if you want absolute control over the login process. You often end up, with a mix of stuff in web.config and some in code. So I decided to figure out, what it would take, to support getting a user signed on, without ever touching web.config.

You will need to handle 4 actions. Requesting login, Handle login, Handle sign-out and handling Cleanup. So lets start with requesting login. But before we can do that, we need to have a basic setup of WIF. WIF normally do most of it’s magic though 3-4 modules, that get’s its configuration though web.config, and is saved in a static (shared) variable you can access simply by typing

Dim fam As Microsoft.IdentityModel.Web.WSFederationAuthenticationModule = Microsoft.IdentityModel.Web.FederatedAuthentication.WSFederationAuthenticationModule

But we want to control. To make things simple, we function like this

Public Function GetAuthenticationModule() As WSFederationAuthenticationModule
Dim fam As New WSFederationAuthenticationModule
fam.ServiceConfiguration = New Microsoft.IdentityModel.Configuration.ServiceConfiguration()
' use this, to allow any realm
' fam.ServiceConfiguration.AudienceRestriction.AudienceMode = IdentityModel.Selectors.AudienceUriMode.Never
' Add known Audience's ( Realms )
' This would proberly come from a database or what ever
For Each Realm in my.settings.Realms
Dim AllowedUri = fam.ServiceConfiguration.AudienceRestriction.AllowedAudienceUris.Where(Function(x) x.AbsoluteUri = IDP.TargetRealm).FirstOrDefault()
If AllowedUri Is Nothing Then
fam.ServiceConfiguration.AudienceRestriction.AllowedAudienceUris.Add(New Uri(IDP.TargetRealm))
End If
fam.ServiceConfiguration.IssuerNameRegistry = New CustomTrustedIssuerNameRegistry()

Dim certificate As System.Security.Cryptography.X509Certificates.X509Certificate2 = CustomSecurityTokenServiceConfiguration.Current.EncryptingCertificate
AttachCert(fam.ServiceConfiguration, certificate)
' If you later decide to let WIF set the identity and save a FedAuth cookie, use thise to allow it to work incase you got
' your website hosted in Azure. The reason is Azure's Loadbalencing are using true round robin and doesnt support Sticky sessions.
' So in order to make all websites able to read the cookie, you need to a "common" ground for reading the encrypted token
'Dim sessionTransforms As New List(Of CookieTransform)(New CookieTransform() {New DeflateCookieTransform(), New RsaEncryptionCookieTransform(certificate),
' New RsaSignatureCookieTransform(certificate)})
'Dim sessionHandler As New Microsoft.IdentityModel.Tokens.SessionSecurityTokenHandler(sessionTransforms.AsReadOnly())
Return fam
End Function

Private Function InlineAssignHelper(Of T)(ByRef target As T, value As T) As T
target = value
Return value
End Function
Sub AttachCert(configuration As Microsoft.IdentityModel.Configuration.ServiceConfiguration, certificate As System.Security.Cryptography.X509Certificates.X509Certificate2)
configuration.ServiceCertificate = certificate
Dim certificates = New List(Of System.IdentityModel.Tokens.SecurityToken)() From { _
New System.IdentityModel.Tokens.X509SecurityToken(Configuration.ServiceCertificate) _
Dim encryptedSecurityTokenHandler = TryCast((From handler In Configuration.SecurityTokenHandlers Where TypeOf handler Is Microsoft.IdentityModel.Tokens.EncryptedSecurityTokenHandler).First(), Microsoft.IdentityModel.Tokens.EncryptedSecurityTokenHandler)
configuration.ServiceTokenResolver = InlineAssignHelper(encryptedSecurityTokenHandler.Configuration.ServiceTokenResolver, System.IdentityModel.Selectors.SecurityTokenResolver.CreateDefaultSecurityTokenResolver(certificates.AsReadOnly(), False))
End Sub

We need to tell WIF the realm(s) we are running under right now, called Audience Uri.
We need to tell WIF what signing certificate's to allow tokens from, called Issuer Name Registry.
CustomTrustedIssuerNameRegistry is a class that will get “fed” the signing certificates when WIF is validating the tokens, and should return a name (any name) if the certificate was allowed. Code for that is further down

Perfect, so now we are ready to create a sign request ( what happens when a user clicks the “login” button or what ever)

Public Function CreateSignInRequestMessage(Issuer As String, Realm As String, Reply As String) As SignInRequestMessage
Dim fam = GetAuthenticationModule()
fam.Issuer = Issuer
fam.Realm = Realm
Dim signInRequest = New SignInRequestMessage(New Uri(fam.Issuer), fam.Realm) With { _
.AuthenticationType = fam.AuthenticationType, _
.Freshness = fam.Freshness, .Reply = Reply
Return signInRequest
End Function

So in the code for your “login button” you do something down the line of this

Dim STSLoginPage as string = "https://adfs.wingu.dk/adfs/ls/"
' This webpage, must be in the AllowedAudienceUris
Dim TargetRealm as string = "https://domain.com/MyAwsomeWebapp"
Dim returnURL as string = "https://domain.com/MyAwsomeWebapp/login.aspx"
' or if using MVC
Dim returnURL as string = "https://domain.com/MyAwsomeWebapp/login/signin/"

Dim signInRequest = WIFHelper.CreateSignInRequestMessage(STSLoginPage, TargetRealm, returnURL)
Dim redirURL As String = signInRequest.WriteQueryString()

That will kick the user over to the identity provider, and if all goes well, he will then return with a POST to the return URL, where we can now pickup he’s SAML token.
So on the login page (or MVC controller) its now time to read the token, we add a function to our module, than handles that part

Private Function GetSignInResponseMessage() As SignInResponseMessage
Dim ctx = System.Web.HttpContext.Current
Dim req = ctx.Request
Dim message As SignInResponseMessage = WSFederationMessage.CreateFromFormPost(req) 'as SignInResponseMessage
Return message
End Function

Public Function GetWIFPrincipal(ByRef Issuer As String) As Microsoft.IdentityModel.Claims.ClaimsPrincipal 'System.Security.Principal.IPrincipal
Dim fam = GetAuthenticationModule()
Dim message As SignInResponseMessage = GetSignInResponseMessage()
'Dim token As System.IdentityModel.Tokens.SamlSecurityToken = fam.GetSecurityToken(message)
Dim token = fam.GetSecurityToken(message)
If token IsNot Nothing Then
Dim claims As Microsoft.IdentityModel.Claims.ClaimsIdentityCollection = fam.ServiceConfiguration.SecurityTokenHandlers.ValidateToken(token)
Dim principal As System.Security.Principal.IPrincipal = New Microsoft.IdentityModel.Claims.ClaimsPrincipal(claims)
If TypeOf token Is System.IdentityModel.Tokens.SamlSecurityToken Then
Dim t As System.IdentityModel.Tokens.SamlSecurityToken = token
Issuer = t.Assertion.Issuer
ElseIf TypeOf token Is Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityToken Then
Dim t As Microsoft.IdentityModel.Tokens.Saml2.Saml2SecurityToken = token
Issuer = t.Assertion.Issuer.Value
Throw New Exception("Failed resolving SecurityToken to a known type '" & token.GetType.Name & "'")
End If
Return principal
End If
Return Nothing
End Function

And in the login page/controller you, can now either set the current principal, or parse the claims and sign in the user with formsbased authentication or what ever makes you tick.

Dim Issuer As String = Nothing
Dim Principal As Microsoft.IdentityModel.Claims.ClaimsPrincipal = GetWIFPrincipal(Issuer)
If Principal Is Nothing Then Throw New Exception("Failed parsing sign message to new Principal")
Dim Identity As Microsoft.IdentityModel.Claims.ClaimsIdentity = Principal.Identity
If Identity Is Nothing Then Throw New Exception("Failed parsing sign message to new Principal")
Issuer = Issuer.ToLower

Lastly, to make everything work correctly, we need to make our “login” page support at least 3 of the WS-Federation commands ( I guess that makes “login” page a wrong name then ? ) . So to wrap it all up, you will get something like this

Dim action as string = Request.QueryString(WSFederationConstants.Parameters.Action)
Select Case action
Case WSFederationConstants.Actions.SignIn
' Send user to the identity provider
Dim STSLoginPage As String = "https://adfs.wingu.dk/adfs/ls/"
' This webpage, must be in the AllowedAudienceUris
Dim TargetRealm As String = "https://domain.com/MyAwsomeWebapp"
Dim returnURL As String = "https://domain.com/MyAwsomeWebapp/login.aspx"
' or if using MVC
returnURL = "https://domain.com/MyAwsomeWebapp/login/signin/"
Case WSFederationConstants.Actions.SignOut
Dim requestMessage As SignOutRequestMessage = Nothing
' This will fail, if a "sharepoint" login message
requestMessage = WSFederationMessage.CreateFromUri(Request.Url)
Catch ex As Exception
' ignore
End Try
' Return user to front page
model.RedirectURL = Request.Url.Scheme & "://" & Request.Url.Host & Request.ApplicationPath
If Not requestMessage Is Nothing Then
If Not String.IsNullOrEmpty(requestMessage.Reply) Then
' unless they logged out from another site, then send them back there
model.RedirectURL = requestMessage.Reply
End If
End If

' clear authentication cookie
Dim cookie1 As HttpCookie = New HttpCookie(FormsAuthentication.FormsCookieName, "")
cookie1.Expires = DateTime.Now.AddYears(-1)

' clear session cookie
Dim cookie2 As HttpCookie = New HttpCookie("ASP.NET_SessionId", "")
cookie2.Expires = DateTime.Now.AddYears(-1)
Case WSFederationConstants.Actions.SignOutCleanup
' Do the same as above, but send an "ok" image back
End Select
And lastly, we need to create class that validates singing certificates. 
Imports Microsoft.IdentityModel.Tokens
Imports System.IdentityModel.Tokens

Public Class CustomTrustedIssuerNameRegistry
Inherits IssuerNameRegistry
'Inherits Microsoft.IdentityModel.Tokens.ConfigurationBasedIssuerNameRegistry

Public Sub New()
End Sub
'Public Sub New(customConfiguration As System.Xml.XmlNodeList)
' MyBase.New(customConfiguration)
'End Sub

Public Overrides Function GetWindowsIssuerName() As String
Return MyBase.GetWindowsIssuerName()
End Function

Public Overloads Overrides Function GetIssuerName(securityToken As IdentityModel.Tokens.SecurityToken) As String
Dim x509Token As X509SecurityToken = TryCast(securityToken, X509SecurityToken)
If x509Token IsNot Nothing Then

For Each Thumbprint in My.Settings.AllowedSigningThumbprints
If [String].Equals(x509Token.Certificate.Thumbprint, Thumbprint) Then
Return x509Token.Certificate.SubjectName.Name
End If
End If
Throw New SecurityTokenException("Untrusted issuer.")
End Function

End Class

lørdag den 23. februar 2013

SharePoint/CRM/Exchange and other Claims based websites with Social Sign in

The last week I have been working on a project, regarding combining Claims based authentication and login using Social Providers. In the cloud control panel that I normal work on, I have implemented support for signing in using Office 365, Azure AD, pass-through using Azure ACS, and the mandatory Open ID, Google ID, Windows Live ID, LinkedIn, Facebook, and Twitter. But for reasons I don’t want go into, a need came up, to create a simple STS (Secure Token Service), than would “tie” together one or more identities from social network, and offer those to websites and applications supporting WIF/Claims based authentication.

Why a new STS
If you implemented Social Network, either directly, or indirectly though ACS you probably noticed that if a user sign in using Google ID they will not be the same user, if they later decide to sign in using Facebook. That is relatively easy to “handle” if you wrote the application your self, and Microsoft did a decent job, tying it all up, using the membership provider together with DotNetOpenAuth in MVC 4, but if you don’t want to mess with wrapping up your own customer member ship provider, or if your facilitating WIF in Exchange/CRM/SharePoint/Navision or what ever, wouldn’t it just be awesome if you could just get it all “out of the box” some where. And what if that Service would also allow you to setup your own Identity Providers, so you could implement sign in using Office 365/Azure AD, your own ADFS server, or any other Secure Token Service you might have. You can now

The basic setup
We are still figuring out license models and such, but for now were thinking either a hosted solution, and one where you get the website to run for your self, lets image a hosted setup, and after that you get billed a small amount for each 100k users or something like that. So let’s imaging you want to implement this with Microsoft CRM 2011.

  • First signup for an account, you then get access to a Manage menu. But before you go there, head to CRM and start the “Configure Claims-Based Authentication” wizard
    Type in the URL of the Secure Token Service
    Select a certificate to use, when signing the Claims inside the tokens (this certificate needs to exists on all your CRM servers if you got more than one. It can be a self signed, from your internal CA, or one you bought from a third party
    Go though the rest. Go though the “Configure Internet-Facing Deployment” if you haven't don’t so already, and at the end note the URL to the CRM servers metadata.xml file
  • Go to the STS, click Relays, past in the URL from the “Create new relying part” field, and press Create. By default all Social connectors will be allowed, but if you only want for instance Live ID, select that under Identity Providers, and hit Save.

And that is that. SharePoint how ever doesn’t have these cool metadata.xml files, so configuring that, will take a tiny bit more work, but not to worry, that is why we got PowerShell.

  • Open a PowerShell prompt locally on any of the servers in your farm. And run the following PowerShell Script . ( there is a link on the Relaying Party page too ) Make sure you modify the variables at the top to fit your installation
  • Head to the STS, click management, type a meaningful name into “Create new relying part” field, and press Create
    If you went with the defaults in the script, SharePoint should be sending a reply URL, so we can leave PassiveRequestorURI blank.
    SharePoint (default) doesn’t want the claims to be encrypted so we don’t need to add a certificate.
    Under “Audience Uris” add each of the URN’s that was mapped for each of the websites, show in the end of the script.

And you are all done.

Adding Azure AD, office 365, Azure ACS or an ADFS server
That’s all nice, but what about all your local users, or all your Office 365 users ?
Not a problem, head to the “Providers” page, and type in the URL to the metadata file.

If you are using Office 365 or Azure AD the URL will look something like this

If your using Microsoft ACS, your URL will look something like this
(note, someone at Microsoft fucked this up, so it doesn’t sent a Signing Certificate, but it does doing Office 365 … it’s the same certificate thou, so is easy to grab )

If your using ADFS, the URL will look something like

If your using ACS or ADFS, you don’t need to do anything else than going to the ADFS Server/Azure ACS/Azure AD website and add https://STS/FederationMetadata.xml and it will all configure it self. If how ever your using Office 365, you need to add a ServicePrincipal representing the STS. At the top of the Provider screen is a link to a powershell scrip that will set this up. Once the script is run, past in the “spn:<guid>” from the output of the script into the Target Realm field of the new Provider.

Lastly, go back to your Relying Party, and you can now also select your new Office 365 or what ever Provider you added, and let your users sign in using that

SharePoint 2010 and claims based authentication made a little bit easier

updated 05/03-2013: Added support for extracting root cert from signing certificate
So you have ADFS 2.0 installed. Or your reading some guide about how to configure SharePoint to allow login in using Office 365 or Azure AD or any other kind of Secure Token Service. And 99,9% of them will tell you to copy 1 or more certificates, and run a lot of weird PowerShell command, and it all ends up with you being able to setup some awesome new login page here


Well, fear not. If your STS support generating a FederationMetadata.xml URL, you can now use this PowerShell Script to quickly setup your SPTrustedIdentityTokenIssuer . The script can also setup Uri mappings and configure the websites, but I do not recommend using that part in other than development/test environments. Make sure to modify the variables at the top.

I’ve updated the script. If your STS was using a self issued certificate, or a certificate from an internal CA, it wouldn’t be trusted by SharePoint, so added a bit of code to traverse the certificate chain, and then add the root certificate to the Trusted store and also add the root to the list of trusted certificates inside SharePoint

# URL for ADFS / ACS / Office 365 / SimpleSTS / Wingu Cloud ControlPanel / what ever

# $metadataurl = 'https://adfs.wingu.dk/FederationMetadata/2007-06/federationmetadata.xml' # ADFS
# $metadataurl = 'https://accounts.accesscontrol.windows.net/skadefro.onmicrosoft.com/FederationMetadata/2007-06/FederationMetadata.xml' # Azure AD
# $metadataurl = 'https://skadefro.accesscontrol.windows.net/FederationMetadata/2007-06/FederationMetadata.xml' # Azure ACS - Wont work, apperently the guy who made this, forgot to add the signing certificat .. :-(
# $metadataurl = 'https://simplests.wingu.dk/FederationMetadata.xml' # Simple STS
# $metadataurl = 'http://admin.wingu.dk/sts2/FederationMetadata/2007-06/federationmetadata.xml' # Wingu Cloud Control Panel
$metadataurl = 'https://simplests.wingu.dk/FederationMetadata.xml'

# This should really be NameID, but often people will use emails or UPN. choose what fits your needs
$IdentifierClaim = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier"
# What claims do you want mapped. All will be exposed in the ClaimsToken, but what will show up in the people picker ?
$mapClaims = @('UPN', 'Role')

# Default realm. If you only have 1 website, you can add that here, but more correctly would be adding
# Each URI accessible on your farm, mapping it to a URN
# If your using Wingu Cloud Control Panal or SimpleSTS, urn:sps will automaticly be converte to a SharePoint signin URI
# urn:sps:http://spfarm01.cloudapp.net becomes http://spfarm01.cloudapp.net/_trust/
# if not, just use the full uri, like http://spfarm01.cloudapp.net/_trust/
# if you want several sharepoint sites to allow the user access without prompting for login each time,
# use one shared urn (not URI) for all those sites. There are certain restrictions, on how/where you can do that
$defaultrealm = 'urn:sps:https://sharepoint.domain.com'

# Be carefull with this. Not all STS' understand/support's this, and often the above is a simpler solution
# set this to true, will go though the list of sites in the farm and add an "urn:sps:URI" entry for each site
$autoGenerateProviderRealms = $false

# Dont, Dont, DONT ... D-O-N-T, use this
# This will add Windows and ALL identityProviders to all WebApplication's (except Central Admin) for the "default" zone.
$autoAddSTS = $false

$stsname = 'SimpleSTS'
$name = $stsname
$realm = $null
$stsd = $null
$asd = $null

$snapin = Get-PSSnapin | where {$_.name -eq 'Microsoft.SharePoint.PowerShell'}
if($snapin -eq $null){ Add-PSSnapin Microsoft.SharePoint.PowerShell }

function Get-medadata ($metadataurl) {
[void] [Reflection.Assembly]::LoadWithPartialName("Microsoft.IdentityModel")
$STSReader = New-Object Microsoft.IdentityModel.Protocols.WSFederation.Metadata.MetadataSerializer
$req = [System.Net.WebRequest]::Create($metadataurl)
$xmlreader = [System.Xml.XmlReader]::Create($req.GetResponse().GetResponseStream())
$metadata = $STSReader.ReadMetadata([System.Xml.XmlReader]$xmlreader)

function Get-STS ($metadata) {
foreach($RoleDescriptor in $metadata.RoleDescriptors){
if($RoleDescriptor.GetType().FullName -eq 'Microsoft.IdentityModel.Protocols.WSFederation.Metadata.SecurityTokenServiceDescriptor'){
# metadata have a STS
$stsd = $RoleDescriptor
} elseif($RoleDescriptor.GetType().FullName -eq 'Microsoft.IdentityModel.Protocols.WSFederation.Metadata.ApplicationServiceDescriptor'){
# metadata have a Relying Party
$asd = $RoleDescriptor
function Get-RelyingParty ($metadata) {
foreach($RoleDescriptor in $metadata.RoleDescriptors){
if($RoleDescriptor.GetType().FullName -eq 'Microsoft.IdentityModel.Protocols.WSFederation.Metadata.SecurityTokenServiceDescriptor'){
# metadata have a STS
$stsd = $RoleDescriptor
} elseif($RoleDescriptor.GetType().FullName -eq 'Microsoft.IdentityModel.Protocols.WSFederation.Metadata.ApplicationServiceDescriptor'){
# metadata have a Relying Party
$asd = $RoleDescriptor

function Get-GetCertificate ($stsd, $certificatetype) {
if( ($certificatetype -ne 'Signing') -and ($certificatetype -ne 'Encryption')){
Write-Error "CertificateType should be 'Signing' or 'Encryption'"
$has40 = [Reflection.Assembly]::LoadWithPartialName("System.IdentityModel")

$SigningCertificate = $null
foreach($key in $stsd.Keys){
if($key.Use -eq $certificatetype){
if($has40.GetName().Version.Major -eq 4){
#if .net 4.0 is install we can just do this, when System.IdentityModel v4.0.30319 is loaded
$bytes = $key.Keyinfo.GetX509RawData()
} else {
#but default, sharepoint 2010 only have version v2 loaded so, lets do it, the ugly way
$str = $key.Keyinfo.ToString()
$str = $str.Substring( $str.IndexOf('RawData = ') + 10)
$str = $str.Substring(0, $str.IndexOf(')'))

$bytes = ([system.Text.Encoding]::UTF8).GetBytes($str)
$SigningCertificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]([byte[]]$bytes)

Function Ensure-SPProvider($name, $defaultrealm, $SigningCertificate, $SigninURL, $claims, $IdentifierClaim, $mapClaims){
$certificatename = "$($name) Signing Certificate"
$cert = get-SPTrustedRootAuthority | ?{($_.Certificate.Thumbprint -eq $SigningCertificate.Thumbprint) -or ($_.Name -eq $certificatename)}
Write-Host "Adding $($SigningCertificate.Thumbprint) as a Trust Certificate"
$cert = New-SPTrustedRootAuthority -name $certificatename -Certificate $SigningCertificate
} else {
$cert | Set-SPTrustedRootAuthority -Certificate $SigningCertificate
$chain = New-Object System.Security.Cryptography.X509Certificates.X509Chain
$SigningCertificateRoot = $chain.ChainElements[$chain.ChainElements.Count -1].Certificate

# Ensure Root Cert has been added to Trusted Store
$store = get-item cert:\LocalMachine\Root
$store.Add($SigningCertificateRoot )

# And now, add it to SharePoint too
$rootcert = get-SPTrustedRootAuthority | ?{($_.Certificate.Thumbprint -eq $SigningCertificateRoot.Thumbprint)}
Write-Host "Adding $($SigningCertificateRoot.Thumbprint) as a Trust Certificate"
$rootcert = New-SPTrustedRootAuthority -name $SigningCertificateRoot.Subject -Certificate $SigningCertificateRoot

$IDClaim = $null
$claimsmappings = @()
foreach($claim in $claims) {
# http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name is reserved inside SharePoint (??? Why ?... really, WTF !!!!)
# http://schemas.microsoft.com/ws/2008/06/identity/claims/primarysid is only used in SharePoint 2013 ( not tested yet )
# its SharePoints internal STS job to handle the requiered fields
# http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationmethod
# http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationinstant

if($claim.ClaimType -eq 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name'){
$claimsmappings += New-SPClaimTypeMapping $claim.ClaimType -IncomingClaimTypeDisplayName $claim.DisplayTag -LocalClaimType 'http://simplests.wingu.dk/identity/claims/name'
} elseif($claim.ClaimType -eq 'http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationmethod'){
# Skip
} elseif($claim.ClaimType -eq 'http://schemas.microsoft.com/ws/2008/06/identity/claims/authenticationinstant'){
# Skip
} elseif($claim.ClaimType -eq 'http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier'){
$claimsmappings += New-SPClaimTypeMapping $claim.ClaimType -IncomingClaimTypeDisplayName $claim.DisplayTag -LocalClaimType 'http://simplests.wingu.dk/identity/claims/nameidentifier'
} else {
$claimsmappings += New-SPClaimTypeMapping -IncomingClaimType $claim.ClaimType -IncomingClaimTypeDisplayName $claim.DisplayTag -SameAsIncoming
if($IdentifierClaim -eq $claim.ClaimType){
$IDClaim = $claim.ClaimType

$ap = Get-SPTrustedIdentityTokenIssuer | ?{$_.name -eq $name}
$ap = New-SPTrustedIdentityTokenIssuer -Name $name -Description $name -Realm $defaultrealm -ImportTrustCertificate $SigningCertificate -ClaimsMappings $claimsmappings -SignInUrl $SigninURL -IdentifierClaim $IDClaim
# Ensure all claim mappins are there
#foreach($claim in $claims) {
# $hasClaim = $ap.ClaimTypes | ?{$_ -eq $claim.ClaimType}
# if(!$hasClaim) {
# Write-Host "adding ClaimType $($claim.DisplayTag) to $name"
# $ap.ClaimTypes.Add($claim.ClaimType)
# }

#Add-SPClaimTypeMapping -Identity $claim -TrustedIdentityTokenIssuer $ap
#$ap = Get-SPTrustedIdentityTokenIssuer | ?{$_.name -eq $name}

# if you only have 1 sharepoint site, and dont want to "bother" setting up
# Reply URL on your STS ( adfs - AD FS 2.0 Management -> Trust Releationships ->
# Relying Party Trusts -> Properties on RP Go to "Endpoints" tab -> WS-Federation Passive Endpoint
# Or SimpleSTS -> Management -> Relying Party -> Manage -> PassiveRequestorURI
# you can just override the $ap.DefaultProviderRealm
# but the correct way would be to list each unique URI with an associated URN in $ap.ProviderRealms
$ap.DefaultProviderRealm = $defaultrealm

$ap.ProviderUri = $SigninURL
#$ap.Name = $name
#$ap.DisplayName = $name

# If WReply is not set, sharepoint will not add return URL when sending user to login page
# if DefaultProviderRealm/$ap.ProviderRealms URN is not listed with a Return URL in the STS
# the STS will not know where to send the user. What's "correct" ? dont know.
# if your paranoid, leacing WReply $false, using URN's all the places, and keeping Passive Federation
# URL's in you STS would proberly be best, but let's ensure we can have both unique URI's, prober return URI's
# and unique'nes by using $ap.ProviderRealms
$ap.UseWReplyParameter = $true

# HomeRealm tells the STS how the user want to to login instead of prompting.
# ADFS will ask and save a cookie, for 3 months, most custom solutions, will prompt each time
# for instance SimpleSTS will allow you to choose each time from a list of all allowed providers
# Sharepoint will NEVER know what those homerealms could be.
# SharePoint's internet STS will normaly "strip" off HomeRealm ( WHR= ) but with the Juni RU this new
# parameter was added to allow adding HomeRealm in custom code (modules or custom login page) without loosing the WHR parameter
# http://blogs.msdn.com/b/chriskeyser/archive/2011/10/02/sharepoint-2010-claims-and-home-realm-discover-passing-whr-on-the-url-to-sharepoint.aspx
try {
$ap.UseWHomeRealmParameter = $true
} catch {
Write-Warning "SharePoint is not update, and wont allow setting UseWHomeRealmParameter, make sure you use uri:sp: as realms/AudienceURIs, or set PassiveRequestorURI to [SharePointURI]/_trust/"

$metadata = Get-medadata $metadataurl
$stsrealm = $metadata.EntityId.Id
$stsd = Get-STS $metadata
#$asd = Get-RelyingParty $metadata
$SigningCertificate = Get-GetCertificate $stsd 'Signing'
#$encryptionCertificate = Get-GetCertificate $asd 'Encryption'
$claims = $stsd.ClaimTypesOffered

foreach($PassiveRequestorEndpoint in $stsd.PassiveRequestorEndpoints){
$SigninURL = $PassiveRequestorEndpoint.Uri.AbsoluteUri

Ensure-SPProvider $stsname $defaultrealm $SigningCertificate $SigninURL $claims $IdentifierClaim $mapClaims

if($autoGenerateProviderRealms -eq $true){
# We could use Get-SPWebApplication, and then Get-SPAlternateURL on each site
# we could also filter out only "external" sites, etc ....

$ap = Get-SPTrustedIdentityTokenIssuer | ?{$_.name -eq $stsname}
foreach($wa in Get-SPWebApplication){
$pr = $ap.ProviderRealms.keys | where {$_.AbsoluteUri -eq $wa.Url}
if(!$pr){ $pr = $ap.ProviderRealms.Add([uri]($wa.Url), "urn:sps:$($wa.Url)") }
if($autoAddSTS -eq $true){
$windowsap = New-SPAuthenticationProvider
#$aps = Get-SPAuthenticationProvider -Zone "Default" -WebApplication $wa
$aps = @()
$aps += $windowsap
foreach($ap in Get-SPTrustedIdentityTokenIssuer){
$aps += $ap
foreach($wa in Get-SPWebApplication){
Set-SPWebApplication -Identity $wa -Zone "Default" -AuthenticationProvider $aps # -Force

$ap = Get-SPTrustedIdentityTokenIssuer | ?{$_.name -eq $stsname}
Write-Host "ensure '$($defaultrealm) and each 'Value' is added to the Relying Party in the STS"