Wednesday, March 14, 2012

Setup Tortoise Git and Winmerge


Winmerge has been my favorite diff and merge tool for a long time. I always use Winmerge for those purposes so when I use Tortoise softwares like TortoiseSVN, TortoiseHG, Winmerge always is my first option. Now I prefer same thing with GIT. Unlike TortoiseHG which helps you integrate with Winmerge by 1 single click. TortoiseGIT requires some more steps:

1. For diff tool:

  • Go to Settings
  • Click on Diff Viewer node on the left
  • Enter following line to the textboxes like the picture below. Remember to choose External for the diff tool:

C:\Program Files (x86)\WinMerge\WinMergeU.exe /s /e /x /u /wl /maximize /dl %bname /dr %yname %base %mine

Diff Settings

2. For merge tool:

  • Click on Merge Tool node on the left
  • Enter following line to the textbox like the picture below. Remember to choose External for the merge tool:

C:\Program Files (x86)\WinMerge\WinMergeU.exe /maximize /wl /u /s /dl [%tname]-%theirs /dr [%yname]-%mine %theirs %mine %merged

The parameters can be used are from here: https://tortoisegit.org/docs/tortoisegit/tgit-dug-settings.html

Merge Settings

Left panel is readonly. For the merge, any write to the right panel will be applied the merged file.
If you're using GitExtensions, you can config WinMerge as the diff tool as well:
"C:/Program Files (x86)/WinMerge/WinMergeU.exe" -s -e -x -u -wl -wr -maximize -dl remote:"$REMOTE" -dr local:"$LOCAL" "$REMOTE" "$LOCAL"
Now, enjoy GIT

Cheers

Tuesday, March 6, 2012

Consume Twitter Stream with OAuth in .NET

- Twitter has announced that they're going to abandon the basic username/password authentication. OAuth seems to be the widely supported authentication standard among many big websites. I have dealed with Yahoo OAuth in the past and to be honest, their documentation is not as good as Twitters. Anyway, in my use case, I won't need to build any fancy UI to ask the users whether they allow my application works on their behalf to access their twitter stream. In other words, I won't need to obtain access tokens because I just want to authorize all my stream requests using OAuth and the access tokens are easily obtained from Twitter dev site. Really good job, Twitter.
Now, given that I have got the following tokens from twitter dev site:
  • twitterConsumerKey
  • twitterConsumerSecret
  • twitterAccessToken
  • twitterAccessTokenSecret
What I need to do next is following their awesome document to create the "Authentication" header. And the most important step to have that is generating the oauth_signature.
They have said very clear about the parameters included in the request. I want to say another point about these parameters: If you're sending a post request, all the parameter value also have to be url encoded. My case is quite simple, I'm going to use HttpWebRequest object to send OAuth request to Twitter. POST data will be url encoded and sent once the HttpWebRequest object is created. So I need something to authorise the request, something look like:
public interface IRequestAuthorizer
{
    void Authorize(HttpWebRequest request, string postData);
}

And here is the implementation:
public class OAuthRequestAuthorizer : OAuthBase, IRequestAuthorizer
{
    private readonly string _twitterConsumerKey;
    private readonly string _twitterConsumerSecret;
    private readonly string _twitterAccessToken;
    private readonly string _twitterAccessTokenSecret;

    public OAuthRequestAuthorizer(string twitterConsumerKey, string twitterConsumerSecret, string twitterAccessToken, string twitterAccessTokenSecret)
    {
        _twitterConsumerKey = twitterConsumerKey;
        _twitterConsumerSecret = twitterConsumerSecret;
        _twitterAccessToken = twitterAccessToken;
        _twitterAccessTokenSecret = twitterAccessTokenSecret;
    }

    public override string GenerateNonce()
    {
        return new Nonce().Value;
    }

    private string BuildHeader(HttpWebRequest request, Uri uri)
    {
        var nonce = GenerateNonce();
        var timeStamp = GenerateTimeStamp();
        
        string normalizedUrl;
        string normalizedRequestParameters;
        
        var httpMethod = request.Method;
        var signature = GenerateSignature(uri, _twitterConsumerKey, _twitterConsumerSecret, _twitterAccessToken, _twitterAccessTokenSecret,
                                          httpMethod, timeStamp, nonce, out normalizedUrl,
                                          out normalizedRequestParameters);

        // https://dev.twitter.com/docs/auth/authorizing-request
        return
            string.Format(
                "OAuth oauth_consumer_key=\"{0}\", " +
                "oauth_nonce=\"{1}\", " +
                "oauth_signature=\"{2}\", " +
                "oauth_signature_method=\"HMAC-SHA1\", " +
                "oauth_timestamp=\"{3}\", " +
                "oauth_token=\"{4}\", " +
                "oauth_version=\"1.0\"", 
                
                UrlEncoder.Encode(_twitterConsumerKey), 
                UrlEncoder.Encode(nonce), 
                UrlEncoder.Encode(signature), 
                UrlEncoder.Encode(timeStamp), 
                UrlEncoder.Encode(_twitterAccessToken));
    }

    public void Authorize(HttpWebRequest request, string postData)
    {
        //NOTE: It's a must to collect all param either in the header/querystring or post body
        var baseUri = string.IsNullOrEmpty(postData) || request.Method.ToUpper() == "GET"
                    ? request.RequestUri
                    : new Uri(string.Format("{0}?{1}", request.RequestUri.AbsoluteUri, postData));

        request.Headers["Authorization"] = BuildHeader(request, baseUri);
    }
}

Nonce is a custom class I created to generate the unique nonce. You can make your self this unique value like a GUID or a DateTime tick. OAuthBase is the base class I got from oauth.googlecode.com. This class did a good job to generate the signature based on what you provide. So utilizing this object will give you the "Authorization" value to put to the request header. UrlEncoder static class is extracted from the UrlEncode method in OAuthBase class. If you encode the Post data as mentioned above, Twitter stream and API is your oyster like they said :D.

Here is the demo code to consume the twitter stream:
var postData = "track=" + UrlEncoder.Encode("IPhone,IPad");
var requestContentBuffer = Encoding.UTF8.GetBytes(postData);

var request = (HttpWebRequest)WebRequest.Create(new Uri("https://stream.twitter.com/1/statuses/filter.json"));
request.Method = "POST";
request.ContentLength = requestContentBuffer.Length;
request.ContentType = "application/x-www-form-urlencoded";

request.UserAgent = "OAuthTwitterStream";
request.Headers["Accept-Encoding"] = "deflate, gzip";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

_requestAuthorizer.Authorize(request, postData);

using (var dataStream = request.GetRequestStream())
{
    dataStream.Write(requestContentBuffer, 0, requestContentBuffer.Length);
}

var apiResponse = (HttpWebResponse) apiRequest.GetResponse();
apiResponseStream = apiResponse.GetResponseStream();

while(true)
{
    var bytesReadCount = apiResponseStream.Read(readBuffer, 0, readBufferSize);
    // Parse Json content
}

I didn't expect the job would be as simple as this because lastime when I tried to consume Yahoo OAuth service, It took me a while to debug since a small mistake while encoding for example would give you the no clue Unauthorized error :D.

Cheers

Friday, January 27, 2012

Validation in WCF application with NValidator

I have been working on a heavy validation messaging project for 3 months. Basically, my company had built a workflow based system to process incoming WCF messages from other business partners. These WCF messages are supposed to be validated before being queued in a RabbitMQ queue store. And I was responsible for implementing the validation stuff for the application. As mentioned in previous post, I wanted to have something similar to ASP.NET MVC such as Validattion Provider, Validation Factory and using DataAnnotation for message contracts. But after a first draft implementation I has seen some potential issues:

1/ It's hard to have a clean contract DLL if using DataAnnotation. The validation logic for the messaging application is quite complicated. We just perform field validation without touching database at this point. However, the rules are so many and quite complicated. So some basic DataAnnotation attributes cannot fit all requirements. Many custom DataAnnotation attributes have been implemented and I don't want to embedded them in the same assembly with the Contracts. As a result, the contract dll had to reference to 1 or 2 other assemblies which contains the custom DataAnnotation classes. I hate this drawback since the contract we built will be provided to other internal teams, even our business partners.

2/ It's possible but quite ugly to perform cross field validations Using DataAnnotation, we can only perform validation rules on the "reader property" of a contract. Believe me, it's not gonna work for public field since the TypeDescriptor .NET class couldn't reflect the fields. To perform validation on crossfields, either we implement an attribute and decorate on the class it self, or implement interface IValidatableObject. I don't like both approaches since the first choice seems to be so complicated and not a straightforward way. The second choice pollutes my POCO classes. So, it's the second drawback to give up DataAnnotation

3/ Validation rules could be different between different trading partners Using DataAnnotation, the validation logics are tight to the contract classes so it's almost impossible to changes the rules since the data annotation attributes are decorated on the contract's properties or the class itself.

4/ And similar to the above reason, validation rules should support versioning. DataAnnotation makes it impossible to deploy new validaton logic without compilation the application eventhough we're hardly need this level of support.

Based on above reasons, separating the validation logics to different assembly seems to be a proper decision. I implemented NValidator and changed my code to using NValidator and it works perfectly. The validation errors are so meanningfull and easy to debug. After applying the library a few days, I got 90 bugs regarding these validations but believe me or not, I fixed all of them just in 1 day.

<a:Result>
    <a:MessageCreatedDT>20120105 12170001</a:MessageCreatedDT>
    <a:MessageReceivedDT>20120127 13250974</a:MessageReceivedDT>
    <a:MessageSuccess>Fail</a:MessageSuccess>
    <a:ProcessingErrors>
        <a:ProcessingError>   
            <a:ErrorDataField>CreateProfile.ApplicationNumber</a:ErrorDataField>
            <a:ErrorDescription>ApplicationNumber is not a valid application number. Received value: "abcdef"</a:ErrorDescription>
            <a:ErrorUniqueID>2d31f33f-44df-48aa-a682-61e3ca7d5566</a:ErrorUniqueID>
        </a:ProcessingError>
        <a:ProcessingError>   
            <a:ErrorDataField>CreateProfile.Entities[0].CompanyName</a:ErrorDataField>
            <a:ErrorDescription>CompanyName must be null because ‘Entity Type’ = “Person”. Received value: ""</a:ErrorDescription>
            <a:ErrorUniqueID>854e68c1-7604-412f-991c-20a6fc31b611</a:ErrorUniqueID>
        </a:ProcessingError>
        <a:ProcessingError>   
            <a:ErrorDataField>CreateProfile.Entities[1].CompanyName</a:ErrorDataField>
            <a:ErrorDescription>Mandatory because ‘Entity Type’ = “Company”. Received value: ""</a:ErrorDescription>
            <a:ErrorUniqueID>cca94d9d-6ea5-4846-8109-091285ae03e4</a:ErrorUniqueID>
        </a:ProcessingError>
    </a:ProcessingErrors>
</a:Result>

Above is a sample message response when validation fails. The 2 last erros are examples of cross field validation. If EntityType = "Person" the CompanyName must be null. CompanyName is mandatory if EntityType = "Company".
Reading enough? Here are the steps for using NValidator for validation on WCF service:
  1. Assume that you have your contract defined in an assembly, add another class library project just for validation rules. Using Nuget to get latest NValidator and add your validator classes.
  2. Register your validator classes in the application bootstrapper or anywhere of your app where the code is executed before other stuff.
  3. Create a validation attribute to decorate on your WCF service. The method is pretty similar to this post.

And here is the code. In this demo, I want to print out the value of the validated object so I create a custom Validation result as below. You don't have to do this if you don't want this function. The default ValidationResult and ValidationBuilder work just fine
public class CustomValidationResult : FormattableMessageResult
{
    public CustomValidationResult() : base(new Dictionary<string, object>())
    {
    }

    public CustomValidationResult(ValidationResult originalResult)
        : base(originalResult is FormattableMessageResult
                ? (originalResult as FormattableMessageResult).Params
                : new Dictionary<string, object>())
    {
        CopyValues(originalResult);
    }

    private void CopyValues(ValidationResult originalResult)
    {
        var customValidationResult = originalResult as CustomValidationResult;
        MemberName = originalResult.MemberName;
        Message = originalResult.Message;
        PropertyName = originalResult.PropertyName;

        if (customValidationResult != null && customValidationResult.ValidatedValueWasSet)
        {
            ValidatedValue = customValidationResult.ValidatedValue;
        }
    }

    public bool ValidatedValueWasSet { get; private set; }

    private object _validatedValue;
    public object ValidatedValue
    {
        get
        {
            return _validatedValue;
        }
        set
        {
            if (!ValidatedValueWasSet)
            {
                _validatedValue = value;
                ValidatedValueWasSet = true;
            }
        }
    }
}

I also have to make a customization of ValidationBuilder to make it return my custom ValidationResult:
/// <summary>
/// This class customizes the behavior of ValidationBuilder and return CustomValidationResult
/// </summary>
/// <typeparam name="T"></typeparam>
/// <typeparam name="TProperty">The type of the property.</typeparam>
public class CustomValidationBuilder<T, TProperty> : ValidationBuilder<T, TProperty>
{
    private object _validatedObject;
    public CustomValidationBuilder(Expression<Func<T, TProperty>> expression)
        : base(expression)
    {
    }

    protected override NValidator.ValidationResult FormatValidationResult(NValidator.ValidationResult result, string propertyChain)
    {
        var baseResult = base.FormatValidationResult(result, propertyChain);
        var customResult = new CustomValidationResult(baseResult);
        if (!customResult.ValidatedValueWasSet)
        {
            customResult.ValidatedValue = _validatedObject;
        }
        return customResult;
    }

    protected override object GetObjectToValidate(T value)
    {
        _validatedObject = base.GetObjectToValidate(value);
        return _validatedObject;
    }
}

And here is the ParameterValidatorBehavior which invokes validation on every WCF message call:
public class ParameterValidatorBehavior : IParameterInspector
{
    public void AfterCall(string operationName, object[] outputs, object returnValue, object correlationState)
    {
    }

    public object BeforeCall(string operationName, object[] inputs)
    {
        // validate parameters before call
        var validationResults = new List<ValidationResult>();
        var serviceIntance = OperationContext.Current.InstanceContext.GetServiceInstance();
        if (serviceIntance != null)
        {
            foreach (var input in inputs)
            {
                if (input != null)
                {
                    var validator = ValidatorFactory.Current.GetValidatorFor(input.GetType());
                    if (validator == null)
                    {
                        ColorConsole.WriteLine(ConsoleColor.Yellow, "Validator for {0} not found.", input.GetType().Name);
                        continue;
                    }
                    validationResults.AddRange(validator.GetValidationResult(input));
                }
            }
            if (validationResults.Count > 0)
            {
                throw new FaultException<ValidationFault>(new ValidationFault(validationResults), "Validation error");
            }
        }
        return null;
    }
}

As mentioned in the NValidator topic, the above CustomValidationBuilder has to be registered in BootStrapper:
internal class BootStrapper
{
    public static void Start()
    {
        ValidatorFactory.Config.DefaultValidationBuilderType = typeof (CustomValidationBuilder<,>);

        ValidatorFactory.Current.Register<OrderValidator>(true);
        ValidatorFactory.Current.Register<OrderDetailValidator>(true);
    }
}

That cannot be easier. Again, the full source code can be check out here at GitHub. Cheers.

Thursday, January 19, 2012

Is Repository old skool?

I used to be a fan of Repository pattern, DAO, DAL whatever they're called. I have tried many different approaches for this pattern and normally end up at a DAO object rather than a pure generic Repository. However, I still fooled myself that it was my bad design of Repository pattern and there must be a better design somewhere. More than two years ago, I saw code of my elder colleague about his design of database abstraction layer. He used Specificication pattern together with the Generic Repository on top of Entity Framework. I talked to myself, what the hell is specification and why on earth would we want to use it when we have LinQ built in in the language. Hmm, I might be wrong at time time and now I know I was properly wrong. Recently, I'm watching a interesting argument among many blog readers on a topic of Oren's blog. Basically, they're arguing about whether we should use Repository Pattern. I'm confused about the trend of community on this topic so I decided to ask my 2 experienced colleagues. Here are their responses:


I don’t tend to use them these days ...

A Repository in concept is “an in-memory list” for all intents and purposes ... the problem is they are almost always written as “a generic data access layer”. The main problem is that Repositories should only *ever* be for an Aggregate Root, and therefore handle all data access for all child objects too. Ours are a nasty generic data access layer pretending to be a repository.

Personally I usually favour QueryObjects for querying and CommandObjects for writing

In this case a QueryObject knows how to do one query or projection (which may involve database calls), and returns either DTOs or domain entities

A CommandObject knows how to do one single operation, which may involve multiple database calls), and returns Void



Or as I think Oren advocates later on, I would just let the NH objects into the domain these days – nobody ever replaced a data layer

Jak Charlton


I usually, but not always, tend to judge the repository pattern as an anti-pattern and an unnecessary additional layer of abstraction, complexity and maintenance. It's quite useful if you're going to need to swap out your storage platform at some point, say SQL Server for Oracle or MongoDB, but even then there are often underlying specific implementation differences between platforms that break the abstraction compatibility, so you end up needing to redesign the repository's API contract and break compatibility. A generic repository that works across several storage platforms must be very generalised, and isn't able to leverage specific features of a particular technology.

This only applies to a generic repository, I see less harm in it when applied more specifically, like EF which provides a good storage abstraction and API over relational databases.

As always, it depends. I currently tend to design for a service layer that directly uses a mirco-ORM for SQL Server, and other data access libraries specific to their storage type, like the .NET MongoDB driver.

Chris Fulstow


Well, clear enough. These comments plus people's comments on Oren's post tell me that I'm at least 3 years out of date. Repositories are nolonger hot fashion even though I believe there are still more guys like the old me still see the value of Repositories. I had seen enough problems using Repositories as a generic Repository can easily become a DAO at some point otherwise it would lose many features of the ORM technology abstracted behind it. Properly QueryObjects are the one to go for me next time :D.

Wednesday, December 28, 2011

Introduce NValidator


I had spent 2 weeks before Christmas to develop a small library to do the validation. It's a lot similar to FluentValidation in syntax. Actually, the implementation and fluent syntax are inspired from this well-known library Fluent Validation. However, I implemented it using Chain of Reponsibility pattern and made it support Dependency Injection for validator classes. Here is a simple example that can help you to start:
public class UserValidator : TypeValidator<User>
{
    public UserValidator()
    {
        RuleFor(user => user.Id)
            .NotEmpty()
            .NotNull();

        RuleFor(user => user.UserName)
            .NotNull()
            .Length(6, 10)
            .Must(x => !x.Contains(" ")).WithMessage("@PropertyName cannot contain empty string.");

        RuleFor(user => user.Password)
            .StopOnFirstError()
            .NotEmpty()
            .Length(6, 20)
            .Must(x => x.Any(c => c >= 'a' && c <= 'z') &&
                       x.Any(c => c >= '0' && c <= '9') &&
                       x.Any(c => c >= 'A' && c <= 'Z')).WithMessage("@PropertyName must contain both letter and digit.");

        RuleFor(user => user.Title)
            .In("MR", "MS", "MRS", "DR", "PROF", "REV", "OTHER");

        RuleFor(user => user.FirstName)
            .NotEmpty().WithMessage("Please specify the first name.");

        RuleFor(user => user.LastName)
            .NotEmpty().WithMessage("Please specify the last name.");
        
        RuleFor(user => user.Email)
            .Email();

        RuleFor(user => user.Address)
            .SetValidator<AddressValidator>()
            .When(x => x != null);
    }
}
I'm currently using this library for a heavy server side validation project and so far so good. You may ask me why don't I just use FluentValidation instead of reinventing the wheel? My answer is I want to keep myself busy. I have a passionate on Fluent interfaces for a while and this is a good opportunity for me to "do something" :D Anyway, the source code repository for this project is on GitHub. The NuGet package "NValidator" is also available. Please checkout the document and other topics to get started.
Happy New Year.