TomHarris.co.uk >> Blog

Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts

Wednesday, 25 January 2017

Specflow, Selenium and TeamCity - Screenshots in TeamCity results

Trying to replicate automation test can be a real pain. Especially for those flaky test that fail from time to time.

As a result my team started to investigate how we could see what the test actually did while running.

Solution 1 - BrowserStack Automate
We started looking at BrowserStack Automate. The tool is impressive and integrates using the "RemoteWebDriver". You can then log into BrowserStack and watch videos of the test runs.

This was awesome but had some draw back. Specifically speed was an issue. Running the remote browser even in Parallel added about a 3rd to our run time. We run our automation test in TeamCity having yet another tool to monitor results was going to be tedious.

I recommend setting this up if only for the multiple browser support. https://www.browserstack.com/automate/specflow

Solution 2 - WebDriver Screenshot
We then decided to start taking screenshots of the state throughout the journey of the test. You can easily use the WebDriver to take a screenshots. We then would place this image on the Team City Agent.

This worked fast but still accessing the files and associating them to the test was not easy.

Solution 3 - TeamCity static content
Finally we have found a solution that worked for us. Making some tweaks to the screenshot solution we started to upload our images to S3. This meant we had friendly URLs we could spit out in our output logs.

Then using some hacky code we processed the logs and pulled the image content into the TeamCity stacktrace. Each image from the logs is pulled in order and appended to a list showing the journey as per our interactions. This lets us quickly build a view of what happened and spot obvious mistake.

Below is an example of how the screenshot is added to the page. The image shows on the expanded test failure.



How to : https://gist.github.com/tharris29/8c9b4aca6ae0b2bc2e3ce1b3ac406314

Excuse the hacky code and html. I will tidy it up one day. Fill free to improve it.

Saturday, 25 October 2014

Testing WebApi with Owin Hosting

Lately I have been working with the OWIN specification; specifically OWIN hosting, known as Katana, which is Microsoft's implementation of the OWIN specification. During this time I have come across some tools that have helped me create my project. One in particular stood out so I thought I would share.

Brief introduction to Owin hosting

Here is a brief example of OWIN hosting Startup class initializing WebApi, full solution here.

using Owin;
using System.Web.Http;

namespace WebApi
{
    public class Startup
    {
        public void Configuration(IAppBuilder builder)
        {
            HttpConfiguration config = new HttpConfiguration();
            config.Routes.MapHttpRoute("Default", "{controller}/{customerID}", new { controller = "Customer", customerID = RouteParameter.Optional });

            builder.UseWebApi(config);
        }
    }
}

This is the entry point to your API, regardless of being hosted in IIS, Windows Service or even a command window. This is the "first" part of  the code that will be touched in the solution, meaning it's the first place you could break your site unless you have some test covering it.

To learn more about OWIN Katana, see these examples http://www.asp.net/aspnet/samples/owin-katana.

Why is this important?

This pattern of hosting is the future of ASP.net. This is proven with the latest version of vNext using this method.

Testing boundaries

The current trends on testing seem to be in a state of flux. Many people have different views on what the best way of testing is, an argument I'm not trying to take part in here. However, one of these views is to test from the boundaries, not unit testing. In our API's case, the boundaries can be considered as the Startup.cs and likely a database or some external resource. These are the first and last entry points of our solution.

If you want to know more about boundaries, check out Ian Cooper's presentation on Hexagonal Architecture, I saw it at DDD East Anglia and it clarifies a few points of confusion.

Why does the OWIN host make this easy?

One of the NuGets that has come out with the introduction of OWIN host is the Microsoft.Owin.Testing library. With this library we have the ability to call our API from the outside, from the first boundary point of our code.

When using the Mircosoft.Owin.Testing library, the tests are started in an in-memory host. Any calls to the host will now be routed via your "Startup.Configuration" which in our example will pass the calls to the WebApi framework.

This means no setup of a host (IIS, Windows Service...) before you run your test. The tests are completely contained.

Creating a test

  1. Create your API using Owin Hosting, as shown earlier
  2. Create a Unit Test project,
  3. Add a reference to your API project. This allows us to reference the Startup class.
  4. Add the "Microsoft.Owin.Testing" NuGet
  5. Add the following code in your test. Changing the URL to your API url.

var server = TestServer.Create<Startup>();

var response = server.HttpClient.GetAsync("/hello/world").Result;
var result = response.Content.ReadAsAsync<IEnumerable<string>>().Result;

Assert.AreEqual("helloworld", result.First());

Now you can run your test knowing that all your routing and configuration is correct, without setting up any IIS or Service to host your solution.

For some other example see https://github.com/justeat/openrasta-hosting-owin/tree/master/src/OpenRasta.Owin.Test

Not just WebApi

This doesn't just work with WebApi! Any framework that has an OWIN hosting can run within the Microsoft.Owin.Testing in-memory host, here are a couple of frameworks that now allow this.

Conclusion

Whether you want to do integration testing, boundary testing or feature testing I believe that you can use the OWIN in-memory host to achieve this. Removing any prior configuration of a server before running the test also allows for easier collaboration on different machines. With Microsoft following the same patterns in vNext it seems like a promising road map to be on.

**UPDATE**

Since releasing this post I have now been working with JustFakeIt. This is a nice wrapper around a in process host using similar methods as above. The benefit is that we can use it like many mock frameworks with expected results and ignoring parameters. Check it out https://github.com/justeat/JustFakeIt.


Sunday, 27 October 2013

JavaScript testing

JavaScript testing has got to be one of the most overlooked areas of testing since practices like TDD and Unit testing begun. It is often seen as a code to just add events and load data, these days with HTML5 bringing new functionality I have seen more and more people use client side coding. People are seeing the benefit of client side processing and with the introduction of JavaScript workers in new browsers this is only going to continue. This code clearly has a place and therefore we need to start thinking about how we test this.

I have been working on web applications for ten years and in the past 3 years I have really bought into testing be it TDD or just unit test developed after. I believe it’s the stable to any long lasting project. The question I had to ask myself about a year ago was how I can ensure that my client side code is as stable as my server side code. Firstly it was a case of replacing all old legacy code with standard libraries like JQuery, this then left the community to support these areas. Also worth pointing out JQuery is tested.

That alone wasn't enough to get me coverage. I work on a system with complex calculations that process for nutrition and recipe data on the client. This data is critical to business process so has to be correct. We needed to make sure these and calculations similar where covered to our server side code. So I took on the task of testing our Client side code.

Now the idea of testing JavaScript isn't hard. Many solutions are out there to help you on this path, below are some examples. But the concept of testing something like JavaScript is the hard one. Below are some of the issues and solutions and my conclusion so far to how you can start testing.


DOM Interaction

So we have all inherited someone's code or even written sloppy code when we where juniors. Well often one of the key mistakes when starting out is the crossing over of JavaScript and the DOM. Now this is perfectly fine when you are working with a DOM manipulating library but when you come across a solution that does calculations and each time it gets the value from the DOM input this is a big "no no". When it comes to JavaScript testing, and really any testing, this has to stop. What we should be trying to do is pass a model to our method or parameters and get the result. With the result you can then bind this to the input DOM. This way when you run you test it can run without the DOM in place.

Server Calls

Next on my list of changes was the methods which hit the server. In the test environment I don't have the data set-up to ensure that my test pass. If I do have the data I cannot be sure it's the same each time. So what I need is to be able to simulate the server being hit, I need to mock the server. This is common practice in .NET development but maybe not obvious in JavaScript. The article below explains in detail the different libaries that are available and

http://testdrivenwebsites.com/2010/05/06/java-script-mock-frameworks-comparison/

Integrated builds

With any testing framework what we want is for it to be automated. Ideally this should happen on each check-in.  The below blog explains some of the more popular frameworks and getting them running in a automated fashion.

http://blog.danmerino.com/continuos-integration-ci-for-javascript-jasmine-and-teamcity/


Conclusion So Far
Again to point out this is my feelings so far, 6 months in. I'm sure as I move forward and so does the concept of JavaScript testing the decisions and problems will changes. I will try to keep this post updated with the latest.

For me JavaScript testing although it is firmly out there is still not at the maturity that allows a clear decision. Many libraries are available and its a constantly changing environment. It also depends on the set-up you have and the requirements.

If you are learning about testing and how to structure your code. I would start with QUnit its simple to setup and get going. You will however outgrow QUnit fairly quickly if you are working with multiple developers.

If you want the full feature set then go for Jasmine today it's more of a framework for testing than QUnit. It also appears to be a favourite with many developers. And some large libraries like AngularJS are being tested with this today.